2020-11-20T11:14:13.714 INFO:root:teuthology version: 1.0.0-357fd4a 2020-11-20T11:14:13.719 DEBUG:teuthology.run:Teuthology command: teuthology --name pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi --verbose --archive /home/teuthworker/archive/pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi/5640259 --owner scheduled_pdonnell@teuthology --description fs/thrash/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/ffsb}} --lock --block -- /tmp/teuthology-worker.xlxui63i.tmp 2020-11-20T11:14:13.774 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-20T11:14:13.876 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi/5640259 branch: wip-pdonnell-testing-20201120.053228 description: fs/thrash/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/ffsb}} email: pdonnell@redhat.com first_in_suite: false job_id: '5640259' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.3' overrides: admin_socket: branch: wip-pdonnell-testing-20201120.053228 ceph: cephfs: session_timeout: 300 cephfs_ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 15m rados osd op timeout: 15m mds: debug mds: 20 debug ms: 1 mds bal frag: true mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 15m rados osd op timeout: 15m mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 debug rocksdb: 10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - overall HEALTH_ - \(MON_DOWN\) - SLOW_OPS - slow request log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: b86e691bf191566922a8f08e2a5abf64fde8cfcb ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: sha1: b86e691bf191566922a8f08e2a5abf64fde8cfcb selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20201120.053228 sha1: b86e691bf191566922a8f08e2a5abf64fde8cfcb owner: scheduled_pdonnell@teuthology priority: 95 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - osd.4 - osd.5 - osd.6 - osd.7 sha1: b86e691bf191566922a8f08e2a5abf64fde8cfcb sleep_before_teardown: 0 suite: fs suite_branch: wip-pdonnell-testing-20201120.053228 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20201120.053228/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: b86e691bf191566922a8f08e2a5abf64fde8cfcb tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell rpm: - python3-cephfs extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - ceph-fuse: null - mon_thrash: check_mds_failover: true revive_delay: 20 thrash_delay: 10 - workunit: clients: all: - suites/ffsb.sh teuthology_branch: master timestamp: 2020-11-20_08:39:42 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.3615 2020-11-20T11:14:13.877 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20201120.053228/qa; will attempt to use it 2020-11-20T11:14:13.885 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20201120.053228/qa/tasks 2020-11-20T11:14:13.886 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-11-20T11:14:13.888 INFO:teuthology.task.internal:Checking packages... 2020-11-20T11:14:13.931 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'basic' and ceph hash 'b86e691bf191566922a8f08e2a5abf64fde8cfcb' 2020-11-20T11:14:13.931 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-11-20T11:14:13.932 INFO:teuthology.packaging:ref: None 2020-11-20T11:14:13.932 INFO:teuthology.packaging:tag: None 2020-11-20T11:14:13.933 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20201120.053228 2020-11-20T11:14:13.933 INFO:teuthology.packaging:sha1: b86e691bf191566922a8f08e2a5abf64fde8cfcb 2020-11-20T11:14:13.934 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-pdonnell-testing-20201120.053228 2020-11-20T11:14:14.178 INFO:teuthology.task.internal:Found packages for ceph version 16.0.0-7397.gb86e691b 2020-11-20T11:14:14.179 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-11-20T11:14:14.191 INFO:teuthology.task.internal:no buildpackages task found 2020-11-20T11:14:14.192 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-11-20T11:14:14.234 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-11-20T11:14:14.237 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-20T11:14:14.334 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:14:24.350 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:14:34.372 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:14:44.390 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:14:54.413 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:15:04.884 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:15:14.907 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:15:24.923 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:15:34.939 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:15:44.957 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:15:54.976 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:16:04.993 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:16:15.009 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:16:25.029 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:16:35.048 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:16:45.065 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:16:55.077 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:17:05.094 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:17:15.109 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:17:25.125 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:17:35.146 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:17:45.161 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:17:55.178 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:18:05.196 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:18:15.214 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:18:25.228 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:18:35.245 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:18:45.262 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:18:55.280 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:19:05.300 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:19:15.331 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:19:25.349 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:19:35.364 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:19:45.380 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:19:55.392 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:20:05.413 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-11-20T11:20:15.432 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:20:25.453 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:20:35.468 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-11-20T11:20:45.484 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_pdonnell@teuthology', 'count': 2, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi/5640259'} 2020-11-20T11:20:45.505 DEBUG:teuthology.lock.ops:locked ubuntu@smithi023.front.sepia.ceph.com, ubuntu@smithi155.front.sepia.ceph.com 2020-11-20T11:20:45.555 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4aa86f31e0>, signals=[15]) 2020-11-20T11:20:45.761 INFO:teuthology.provision.fog.smithi155:Scheduling deploy of rhel 8.3 2020-11-20T11:20:45.764 INFO:teuthology.provision.fog.smithi023:Scheduling deploy of rhel 8.3 2020-11-20T11:20:46.062 INFO:teuthology.orchestra.console:Power off smithi023 2020-11-20T11:20:46.063 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-20T11:20:46.086 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:20:46.101 INFO:teuthology.orchestra.console:Power off smithi155 2020-11-20T11:20:46.101 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-11-20T11:20:46.124 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:20:50.113 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:20:50.145 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:20:54.226 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:20:54.256 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:20:58.337 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:20:58.367 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:20:58.548 INFO:teuthology.orchestra.console:Power off for smithi023 completed 2020-11-20T11:20:58.576 INFO:teuthology.orchestra.console:Power off for smithi155 completed 2020-11-20T11:20:58.649 INFO:teuthology.orchestra.console:Power on smithi023 2020-11-20T11:20:58.650 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-11-20T11:20:58.671 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:20:58.679 INFO:teuthology.orchestra.console:Power on smithi155 2020-11-20T11:20:58.680 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-11-20T11:20:58.701 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:21:02.697 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:21:02.723 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-11-20T11:21:02.907 INFO:teuthology.orchestra.console:Power on for smithi023 completed 2020-11-20T11:21:02.932 INFO:teuthology.orchestra.console:Power on for smithi155 completed 2020-11-20T11:21:03.008 INFO:teuthology.provision.fog.smithi023:Waiting for deploy to finish 2020-11-20T11:21:03.033 INFO:teuthology.provision.fog.smithi155:Waiting for deploy to finish 2020-11-20T11:23:49.503 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:23:58.893 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:24:04.578 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:24:07.974 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:24:13.638 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:24:17.030 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:24:22.694 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:24:26.087 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:24:31.757 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:24:35.146 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:24:40.843 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:24:44.202 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:24:49.899 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:24:53.258 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:24:58.960 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:25:08.045 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:25:17.133 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:25:41.613 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:25:50.698 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:25:52.029 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-20T11:25:52.031 INFO:teuthology.orchestra.run.smithi023:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-20T11:25:59.261 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:25:59.510 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-11-20T11:25:59.510 INFO:teuthology.orchestra.run.smithi155:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-11-20T11:25:59.827 INFO:teuthology.provision.fog.smithi155:Node is ready 2020-11-20T11:25:59.828 INFO:teuthology.orchestra.run.smithi155:> hostname 2020-11-20T11:25:59.852 INFO:teuthology.orchestra.run.smithi155.stdout:smithi155.front.sepia.ceph.com 2020-11-20T11:25:59.853 INFO:teuthology.orchestra.run.smithi155:> grep smithi155.front.sepia.ceph.com /etc/hosts 2020-11-20T11:25:59.912 INFO:teuthology.orchestra.run.smithi155.stdout:172.21.15.155 smithi155.front.sepia.ceph.com smithi155 2020-11-20T11:25:59.914 INFO:teuthology.orchestra.run.smithi155:> sudo hostname smithi155 2020-11-20T11:26:00.021 INFO:teuthology.orchestra.run.smithi155:> sudo sed -i -e 's/smithi155.front.sepia.ceph.com/smithi155/g' /etc/hosts 2020-11-20T11:26:00.057 INFO:teuthology.orchestra.run.smithi155:> sudo sed -i -e 's/172.21.15.155/172.21.15.155/g' /etc/hosts 2020-11-20T11:26:00.130 INFO:teuthology.orchestra.run.smithi155:> sudo sed -i -e 's/smithi155.front.sepia.ceph.com/smithi155/g' /etc/hostname 2020-11-20T11:26:00.204 INFO:teuthology.orchestra.run.smithi155:> sudo hostname smithi155 2020-11-20T11:26:00.279 INFO:teuthology.provision.fog.smithi155:Deploy complete! 2020-11-20T11:26:00.280 DEBUG:teuthology.parallel:result is None 2020-11-20T11:26:07.428 INFO:teuthology.provision.fog.smithi023:Node is ready 2020-11-20T11:26:07.428 INFO:teuthology.orchestra.run.smithi023:> hostname 2020-11-20T11:26:07.447 INFO:teuthology.orchestra.run.smithi023.stdout:smithi023.front.sepia.ceph.com 2020-11-20T11:26:07.448 INFO:teuthology.orchestra.run.smithi023:> grep smithi023.front.sepia.ceph.com /etc/hosts 2020-11-20T11:26:07.505 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23 smithi023.front.sepia.ceph.com smithi023 2020-11-20T11:26:07.505 INFO:teuthology.orchestra.run.smithi023:> sudo hostname smithi023 2020-11-20T11:26:07.635 INFO:teuthology.orchestra.run.smithi023:> sudo sed -i -e 's/smithi023.front.sepia.ceph.com/smithi023/g' /etc/hosts 2020-11-20T11:26:07.664 INFO:teuthology.orchestra.run.smithi023:> sudo sed -i -e 's/172.21.15.23/172.21.15.23/g' /etc/hosts 2020-11-20T11:26:07.733 INFO:teuthology.orchestra.run.smithi023:> sudo sed -i -e 's/smithi023.front.sepia.ceph.com/smithi023/g' /etc/hostname 2020-11-20T11:26:07.800 INFO:teuthology.orchestra.run.smithi023:> sudo hostname smithi023 2020-11-20T11:26:07.866 INFO:teuthology.provision.fog.smithi023:Deploy complete! 2020-11-20T11:26:07.867 DEBUG:teuthology.parallel:result is None 2020-11-20T11:26:07.999 INFO:teuthology.lock.ops:Checking smithi023.front.sepia.ceph.com 2020-11-20T11:26:07.999 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-20T11:26:08.017 INFO:teuthology.lock.ops:Checking smithi155.front.sepia.ceph.com 2020-11-20T11:26:08.017 INFO:teuthology.lock.ops:New key found. Updating... 2020-11-20T11:26:08.033 INFO:teuthology.lock.ops:Updating [smithi023.front.sepia.ceph.com]: set os type and version on server 2020-11-20T11:26:08.034 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-20T11:26:08.034 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:26:08.101 INFO:teuthology.orchestra.run.smithi023:> true 2020-11-20T11:26:08.268 INFO:teuthology.orchestra.run.smithi023:> uname -m 2020-11-20T11:26:08.322 INFO:teuthology.orchestra.run.smithi023.stdout:x86_64 2020-11-20T11:26:08.324 INFO:teuthology.orchestra.run.smithi023:> cat /etc/os-release 2020-11-20T11:26:08.377 INFO:teuthology.orchestra.run.smithi023.stdout:NAME="Red Hat Enterprise Linux" 2020-11-20T11:26:08.378 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION="8.3 (Ootpa)" 2020-11-20T11:26:08.378 INFO:teuthology.orchestra.run.smithi023.stdout:ID="rhel" 2020-11-20T11:26:08.378 INFO:teuthology.orchestra.run.smithi023.stdout:ID_LIKE="fedora" 2020-11-20T11:26:08.379 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION_ID="8.3" 2020-11-20T11:26:08.379 INFO:teuthology.orchestra.run.smithi023.stdout:PLATFORM_ID="platform:el8" 2020-11-20T11:26:08.379 INFO:teuthology.orchestra.run.smithi023.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2020-11-20T11:26:08.379 INFO:teuthology.orchestra.run.smithi023.stdout:ANSI_COLOR="0;31" 2020-11-20T11:26:08.379 INFO:teuthology.orchestra.run.smithi023.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2020-11-20T11:26:08.380 INFO:teuthology.orchestra.run.smithi023.stdout:HOME_URL="https://www.redhat.com/" 2020-11-20T11:26:08.380 INFO:teuthology.orchestra.run.smithi023.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2020-11-20T11:26:08.380 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:26:08.380 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2020-11-20T11:26:08.381 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2020-11-20T11:26:08.381 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2020-11-20T11:26:08.381 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2020-11-20T11:26:08.382 INFO:teuthology.lock.ops:Updating smithi023.front.sepia.ceph.com on lock server 2020-11-20T11:26:08.399 INFO:teuthology.lock.ops:Updating [smithi155.front.sepia.ceph.com]: set os type and version on server 2020-11-20T11:26:08.399 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-20T11:26:08.400 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:26:08.473 INFO:teuthology.orchestra.run.smithi155:> true 2020-11-20T11:26:08.641 INFO:teuthology.orchestra.run.smithi155:> uname -m 2020-11-20T11:26:08.695 INFO:teuthology.orchestra.run.smithi155.stdout:x86_64 2020-11-20T11:26:08.696 INFO:teuthology.orchestra.run.smithi155:> cat /etc/os-release 2020-11-20T11:26:08.750 INFO:teuthology.orchestra.run.smithi155.stdout:NAME="Red Hat Enterprise Linux" 2020-11-20T11:26:08.751 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION="8.3 (Ootpa)" 2020-11-20T11:26:08.751 INFO:teuthology.orchestra.run.smithi155.stdout:ID="rhel" 2020-11-20T11:26:08.751 INFO:teuthology.orchestra.run.smithi155.stdout:ID_LIKE="fedora" 2020-11-20T11:26:08.751 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_ID="8.3" 2020-11-20T11:26:08.752 INFO:teuthology.orchestra.run.smithi155.stdout:PLATFORM_ID="platform:el8" 2020-11-20T11:26:08.752 INFO:teuthology.orchestra.run.smithi155.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2020-11-20T11:26:08.752 INFO:teuthology.orchestra.run.smithi155.stdout:ANSI_COLOR="0;31" 2020-11-20T11:26:08.752 INFO:teuthology.orchestra.run.smithi155.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2020-11-20T11:26:08.753 INFO:teuthology.orchestra.run.smithi155.stdout:HOME_URL="https://www.redhat.com/" 2020-11-20T11:26:08.753 INFO:teuthology.orchestra.run.smithi155.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2020-11-20T11:26:08.753 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:26:08.753 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2020-11-20T11:26:08.753 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2020-11-20T11:26:08.754 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2020-11-20T11:26:08.754 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2020-11-20T11:26:08.755 INFO:teuthology.lock.ops:Updating smithi155.front.sepia.ceph.com on lock server 2020-11-20T11:26:08.776 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2020-11-20T11:26:08.811 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi023.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpDWhJyxHEaXeIe4gkFGgQed9NCWMHEbgBYa8tvID2kLwoa3wcMCUQDaFS53Cl20PLMEVBcJQCbf9gw1ZJm50+waTcO2JDmDl1dsFxX+8/GNWHd/cBHcl1cc7x2YPAFArGh8lMRM1j8sU/45uZY3Tg2C8y41AB8M7TCKNm5rKSUQ0gybty1vM8SjdTKddvkkIwzakXx/vkUtoFktUbqppi1xZJwAbG+FGFqqyI6CZvi3M6+HbAth9Mza8ZP+r4ucId4xpKG8R9BFFKVvpZL9UsJCQcoRG6tC/iV5QzT88jFiZPRo59UpZ2GWLqvZ+UsOnomyHcqsObAN8ga5Q5SJRYJ3KuzDD8R+VZoPCoHEeYnqoYYYcltyCiW9Q0Tjt4zVt2n4FF0AOmws2vmClEPQEKfEUJyNXO5S+evFrgKpGsicy02zDc4d34VLL4+iFjXJ+KH++N5NuwRxW6GRXP5+qK9Acy7JhEptgi5srs7JcamcrfviSGyUXL/AxR6J1lQ2E= smithi155.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBdGF4vMnkzXHeSw7B5wT3VK35OlPEsgWsMpHXmRNzNdf0Zqu/Nhz2GL5CSbEdcMOapXnt7wXZtLXB43JfmQYq+pra+EGsVq3O9z0BfNucyiQOvY7fdkuCaSvEdZ1CKSp3eQWLEIqrb7P0IYPw+WcAZ78ktRpXUHwa4ktLnZGW/YwWBabVsK6bey/9P2bTa58Lt1gl3YLZluLAxH+Xo1BizNjzs1hIC+0KvRKVqggdMpCAAGecVmyjscwH0WwDoXVDw0Dv4rmKYpQx1/j2K9bmMXJMo+x0ua3irZKSLUorPwYkgOBAR4GEsme6ha4mOMc3z+DnzXjx/p5yl0AwvasiqeLBin6KRbNSatCW6YNa4v1Mo3dek89HRUkn6iD/eYTxmsPfU9Q92UaXJFc6bUmSW3KaXB6/xMXnLMSqZHEuS9V1HEjyz0WAODOEcs/G0HCPG8aCjjtT5Z2H8l2z6w0HieM54ohG7Mb8oJ7U/0PTDlAEy3VHsgf3e5MpHqAQrBk= 2020-11-20T11:26:08.812 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-11-20T11:26:08.869 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-11-20T11:26:08.876 INFO:teuthology.task.internal:Saving configuration 2020-11-20T11:26:08.898 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-11-20T11:26:08.908 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-11-20T11:26:08.927 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-20 11:20:45.490272', 'description': '/home/teuthworker/archive/pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi/5640259', 'locked_by': 'scheduled_pdonnell@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNVfyWtpYYdlBPJReuHAQQXpj/a9YAmK7VFfmhepJbDr7b+R9hYmMLkztxyvN2x3cPe3hw2csFs8t+QchsRPXvQ=', 'os_version': '8.3', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi023.front.sepia.ceph.com'} 2020-11-20T11:26:08.952 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-11-20 11:20:45.490293', 'description': '/home/teuthworker/archive/pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi/5640259', 'locked_by': 'scheduled_pdonnell@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKmqzCiQHUIwhiZxsL8OTdBdHcfSvqavuF6BrnMwxZJQxLOHv9rRBkjaAyoHUtJmtewZLp3heBPlQidnUztG2Ro=', 'os_version': '8.3', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi155.front.sepia.ceph.com'} 2020-11-20T11:26:08.952 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-11-20T11:26:08.959 INFO:teuthology.task.internal:roles: ubuntu@smithi023.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2020-11-20T11:26:08.959 INFO:teuthology.task.internal:roles: ubuntu@smithi155.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2020-11-20T11:26:08.959 INFO:teuthology.run_tasks:Running task console_log... 2020-11-20T11:26:09.025 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4aac2a4048>, signals=[15]) 2020-11-20T11:26:09.026 INFO:teuthology.run_tasks:Running task internal.connect... 2020-11-20T11:26:09.036 INFO:teuthology.task.internal:Opening connections... 2020-11-20T11:26:09.036 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi023.front.sepia.ceph.com 2020-11-20T11:26:09.039 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:26:09.111 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi155.front.sepia.ceph.com 2020-11-20T11:26:09.113 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:26:09.181 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-11-20T11:26:09.189 INFO:teuthology.orchestra.run.smithi023:> uname -m 2020-11-20T11:26:09.238 INFO:teuthology.orchestra.run.smithi023.stdout:x86_64 2020-11-20T11:26:09.238 INFO:teuthology.orchestra.run.smithi023:> cat /etc/os-release 2020-11-20T11:26:09.291 INFO:teuthology.orchestra.run.smithi023.stdout:NAME="Red Hat Enterprise Linux" 2020-11-20T11:26:09.292 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION="8.3 (Ootpa)" 2020-11-20T11:26:09.292 INFO:teuthology.orchestra.run.smithi023.stdout:ID="rhel" 2020-11-20T11:26:09.292 INFO:teuthology.orchestra.run.smithi023.stdout:ID_LIKE="fedora" 2020-11-20T11:26:09.292 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION_ID="8.3" 2020-11-20T11:26:09.293 INFO:teuthology.orchestra.run.smithi023.stdout:PLATFORM_ID="platform:el8" 2020-11-20T11:26:09.293 INFO:teuthology.orchestra.run.smithi023.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2020-11-20T11:26:09.293 INFO:teuthology.orchestra.run.smithi023.stdout:ANSI_COLOR="0;31" 2020-11-20T11:26:09.293 INFO:teuthology.orchestra.run.smithi023.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2020-11-20T11:26:09.294 INFO:teuthology.orchestra.run.smithi023.stdout:HOME_URL="https://www.redhat.com/" 2020-11-20T11:26:09.294 INFO:teuthology.orchestra.run.smithi023.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2020-11-20T11:26:09.294 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:26:09.294 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2020-11-20T11:26:09.295 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2020-11-20T11:26:09.295 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2020-11-20T11:26:09.295 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2020-11-20T11:26:09.296 INFO:teuthology.lock.ops:Updating smithi023.front.sepia.ceph.com on lock server 2020-11-20T11:26:09.311 INFO:teuthology.orchestra.run.smithi155:> uname -m 2020-11-20T11:26:09.332 INFO:teuthology.orchestra.run.smithi155.stdout:x86_64 2020-11-20T11:26:09.333 INFO:teuthology.orchestra.run.smithi155:> cat /etc/os-release 2020-11-20T11:26:09.394 INFO:teuthology.orchestra.run.smithi155.stdout:NAME="Red Hat Enterprise Linux" 2020-11-20T11:26:09.394 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION="8.3 (Ootpa)" 2020-11-20T11:26:09.395 INFO:teuthology.orchestra.run.smithi155.stdout:ID="rhel" 2020-11-20T11:26:09.395 INFO:teuthology.orchestra.run.smithi155.stdout:ID_LIKE="fedora" 2020-11-20T11:26:09.396 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_ID="8.3" 2020-11-20T11:26:09.396 INFO:teuthology.orchestra.run.smithi155.stdout:PLATFORM_ID="platform:el8" 2020-11-20T11:26:09.396 INFO:teuthology.orchestra.run.smithi155.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2020-11-20T11:26:09.397 INFO:teuthology.orchestra.run.smithi155.stdout:ANSI_COLOR="0;31" 2020-11-20T11:26:09.397 INFO:teuthology.orchestra.run.smithi155.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2020-11-20T11:26:09.397 INFO:teuthology.orchestra.run.smithi155.stdout:HOME_URL="https://www.redhat.com/" 2020-11-20T11:26:09.398 INFO:teuthology.orchestra.run.smithi155.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2020-11-20T11:26:09.398 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:26:09.399 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2020-11-20T11:26:09.399 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2020-11-20T11:26:09.399 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2020-11-20T11:26:09.400 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2020-11-20T11:26:09.401 INFO:teuthology.lock.ops:Updating smithi155.front.sepia.ceph.com on lock server 2020-11-20T11:26:09.433 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-11-20T11:26:09.880 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-11-20T11:26:09.891 INFO:teuthology.task.internal:Checking for old test directory... 2020-11-20T11:26:09.892 INFO:teuthology.orchestra.run.smithi023:> test '!' -e /home/ubuntu/cephtest 2020-11-20T11:26:09.896 INFO:teuthology.orchestra.run.smithi155:> test '!' -e /home/ubuntu/cephtest 2020-11-20T11:26:09.914 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-11-20T11:26:09.928 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-11-20T11:26:09.928 INFO:teuthology.orchestra.run.smithi023:> test -z $(ls -A /var/lib/ceph) 2020-11-20T11:26:09.953 INFO:teuthology.orchestra.run.smithi155:> test -z $(ls -A /var/lib/ceph) 2020-11-20T11:26:09.974 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-11-20T11:26:10.020 INFO:teuthology.run_tasks:Running task kernel... 2020-11-20T11:26:10.034 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.y': {'kdb': True, 'sha1': 'distro'}, 'mds.a': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'osd.3': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'mds.b': {'kdb': True, 'sha1': 'distro'}, 'osd.4': {'kdb': True, 'sha1': 'distro'}, 'osd.5': {'kdb': True, 'sha1': 'distro'}, 'osd.6': {'kdb': True, 'sha1': 'distro'}, 'osd.7': {'kdb': True, 'sha1': 'distro'}} 2020-11-20T11:26:10.034 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2020-11-20T11:26:10.035 INFO:teuthology.orchestra.run.smithi023:> uname -r 2020-11-20T11:26:10.050 INFO:teuthology.orchestra.run.smithi023.stdout:4.18.0-240.1.1.el8_3.x86_64 2020-11-20T11:26:10.050 INFO:teuthology.task.kernel:Running kernel on smithi023: 4.18.0-240.1.1.el8_3.x86_64 2020-11-20T11:26:10.051 INFO:teuthology.orchestra.run.smithi023:> sudo yum install -y kernel 2020-11-20T11:26:11.646 INFO:teuthology.orchestra.run.smithi023.stdout:Updating Subscription Management repositories. 2020-11-20T11:26:11.649 INFO:teuthology.orchestra.run.smithi023.stdout:Unable to read consumer identity 2020-11-20T11:26:11.748 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:26:11.748 INFO:teuthology.orchestra.run.smithi023.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2020-11-20T11:26:11.749 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:26:13.455 INFO:teuthology.orchestra.run.smithi023.stdout:Extra Packages for Enterprise Linux 25 MB/s | 8.3 MB 00:00 2020-11-20T11:26:15.252 INFO:teuthology.orchestra.run.smithi023.stdout:lab-extras 4.9 MB/s | 23 kB 00:00 2020-11-20T11:26:15.777 INFO:teuthology.orchestra.run.smithi023.stdout:Package kernel-4.18.0-240.1.1.el8_3.x86_64 is already installed. 2020-11-20T11:26:15.778 INFO:teuthology.orchestra.run.smithi023.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2020-11-20T11:26:15.880 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2020-11-20T11:26:17.228 INFO:teuthology.orchestra.run.smithi023.stdout:Nothing to do. 2020-11-20T11:26:17.228 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2020-11-20T11:26:17.229 INFO:teuthology.orchestra.run.smithi023:> echo no | sudo yum reinstall kernel || true 2020-11-20T11:26:17.526 INFO:teuthology.orchestra.run.smithi023.stdout:Updating Subscription Management repositories. 2020-11-20T11:26:17.527 INFO:teuthology.orchestra.run.smithi023.stdout:Unable to read consumer identity 2020-11-20T11:26:17.583 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:26:17.583 INFO:teuthology.orchestra.run.smithi023.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2020-11-20T11:26:17.583 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:26:17.760 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:00:02 ago on Fri 20 Nov 2020 11:26:15 AM UTC. 2020-11-20T11:26:17.887 INFO:teuthology.orchestra.run.smithi023.stdout:Installed package kernel-4.18.0-240.1.1.el8_3.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2020-11-20T11:26:17.887 INFO:teuthology.orchestra.run.smithi023.stdout:Installed package kernel-4.18.0-240.el8.x86_64 (from anaconda) not available. 2020-11-20T11:26:17.918 INFO:teuthology.orchestra.run.smithi023.stderr:Error: No packages marked for reinstall. 2020-11-20T11:26:17.955 INFO:teuthology.orchestra.run.smithi023:> sudo yum reinstall -y kernel || true 2020-11-20T11:26:18.237 INFO:teuthology.orchestra.run.smithi023.stdout:Updating Subscription Management repositories. 2020-11-20T11:26:18.238 INFO:teuthology.orchestra.run.smithi023.stdout:Unable to read consumer identity 2020-11-20T11:26:18.295 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:26:18.295 INFO:teuthology.orchestra.run.smithi023.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2020-11-20T11:26:18.295 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:26:18.475 INFO:teuthology.orchestra.run.smithi023.stdout:Last metadata expiration check: 0:00:03 ago on Fri 20 Nov 2020 11:26:15 AM UTC. 2020-11-20T11:26:18.609 INFO:teuthology.orchestra.run.smithi023.stdout:Installed package kernel-4.18.0-240.1.1.el8_3.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2020-11-20T11:26:18.609 INFO:teuthology.orchestra.run.smithi023.stdout:Installed package kernel-4.18.0-240.el8.x86_64 (from anaconda) not available. 2020-11-20T11:26:18.630 INFO:teuthology.orchestra.run.smithi023.stderr:Error: No packages marked for reinstall. 2020-11-20T11:26:18.671 INFO:teuthology.orchestra.run.smithi023:> rpm -q kernel --last | head -n 1 2020-11-20T11:26:18.757 INFO:teuthology.orchestra.run.smithi023.stdout:kernel-4.18.0-240.1.1.el8_3.x86_64 Tue 10 Nov 2020 09:16:49 PM UTC 2020-11-20T11:26:18.758 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-240.1.1.el8_3.x86_64 2020-11-20T11:26:18.759 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-11-20T11:26:18.759 INFO:teuthology.orchestra.run.smithi155:> uname -r 2020-11-20T11:26:18.776 INFO:teuthology.orchestra.run.smithi155.stdout:4.18.0-240.1.1.el8_3.x86_64 2020-11-20T11:26:18.776 INFO:teuthology.task.kernel:Running kernel on smithi155: 4.18.0-240.1.1.el8_3.x86_64 2020-11-20T11:26:18.777 INFO:teuthology.orchestra.run.smithi155:> sudo yum install -y kernel 2020-11-20T11:26:20.407 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2020-11-20T11:26:20.407 INFO:teuthology.orchestra.run.smithi155.stdout:Unable to read consumer identity 2020-11-20T11:26:20.499 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:26:20.499 INFO:teuthology.orchestra.run.smithi155.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2020-11-20T11:26:20.499 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:26:22.188 INFO:teuthology.orchestra.run.smithi155.stdout:Extra Packages for Enterprise Linux 24 MB/s | 8.3 MB 00:00 2020-11-20T11:26:23.853 INFO:teuthology.orchestra.run.smithi155.stdout:lab-extras 5.1 MB/s | 23 kB 00:00 2020-11-20T11:26:24.356 INFO:teuthology.orchestra.run.smithi155.stdout:Package kernel-4.18.0-240.1.1.el8_3.x86_64 is already installed. 2020-11-20T11:26:24.356 INFO:teuthology.orchestra.run.smithi155.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2020-11-20T11:26:24.455 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2020-11-20T11:26:24.468 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2020-11-20T11:26:24.468 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2020-11-20T11:26:24.567 INFO:teuthology.orchestra.run.smithi155:> echo no | sudo yum reinstall kernel || true 2020-11-20T11:26:24.830 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2020-11-20T11:26:24.830 INFO:teuthology.orchestra.run.smithi155.stdout:Unable to read consumer identity 2020-11-20T11:26:24.882 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:26:24.882 INFO:teuthology.orchestra.run.smithi155.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2020-11-20T11:26:24.883 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:26:25.050 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:02 ago on Fri 20 Nov 2020 11:26:23 AM UTC. 2020-11-20T11:26:25.169 INFO:teuthology.orchestra.run.smithi155.stdout:Installed package kernel-4.18.0-240.1.1.el8_3.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2020-11-20T11:26:25.169 INFO:teuthology.orchestra.run.smithi155.stdout:Installed package kernel-4.18.0-240.el8.x86_64 (from anaconda) not available. 2020-11-20T11:26:25.206 INFO:teuthology.orchestra.run.smithi155.stderr:Error: No packages marked for reinstall. 2020-11-20T11:26:25.243 INFO:teuthology.orchestra.run.smithi155:> sudo yum reinstall -y kernel || true 2020-11-20T11:26:25.531 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2020-11-20T11:26:25.531 INFO:teuthology.orchestra.run.smithi155.stdout:Unable to read consumer identity 2020-11-20T11:26:25.581 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:26:25.582 INFO:teuthology.orchestra.run.smithi155.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2020-11-20T11:26:25.582 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:26:25.747 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:02 ago on Fri 20 Nov 2020 11:26:23 AM UTC. 2020-11-20T11:26:25.868 INFO:teuthology.orchestra.run.smithi155.stdout:Installed package kernel-4.18.0-240.1.1.el8_3.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2020-11-20T11:26:25.869 INFO:teuthology.orchestra.run.smithi155.stdout:Installed package kernel-4.18.0-240.el8.x86_64 (from anaconda) not available. 2020-11-20T11:26:25.898 INFO:teuthology.orchestra.run.smithi155.stderr:Error: No packages marked for reinstall. 2020-11-20T11:26:25.936 INFO:teuthology.orchestra.run.smithi155:> rpm -q kernel --last | head -n 1 2020-11-20T11:26:26.020 INFO:teuthology.orchestra.run.smithi155.stdout:kernel-4.18.0-240.1.1.el8_3.x86_64 Tue 10 Nov 2020 09:16:49 PM UTC 2020-11-20T11:26:26.022 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-240.1.1.el8_3.x86_64 2020-11-20T11:26:26.022 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-11-20T11:26:26.023 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2020-11-20T11:26:26.023 INFO:teuthology.orchestra.run.smithi023:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-11-20T11:26:26.056 INFO:teuthology.orchestra.run.smithi023.stdout:ttyS1 2020-11-20T11:26:26.071 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2020-11-20T11:26:26.072 INFO:teuthology.orchestra.run.smithi155:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-11-20T11:26:26.103 INFO:teuthology.orchestra.run.smithi155.stdout:ttyS1 2020-11-20T11:26:26.115 INFO:teuthology.run_tasks:Running task internal.base... 2020-11-20T11:26:26.130 INFO:teuthology.task.internal:Creating test directory... 2020-11-20T11:26:26.131 INFO:teuthology.orchestra.run.smithi023:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-11-20T11:26:26.133 INFO:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-11-20T11:26:26.170 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-11-20T11:26:26.187 INFO:teuthology.run_tasks:Running task internal.archive... 2020-11-20T11:26:26.200 INFO:teuthology.task.internal:Creating archive directory... 2020-11-20T11:26:26.201 INFO:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-11-20T11:26:26.203 INFO:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-11-20T11:26:26.237 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-11-20T11:26:26.313 INFO:teuthology.task.internal:Enabling coredump saving... 2020-11-20T11:26:26.314 INFO:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-11-20T11:26:26.317 INFO:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-11-20T11:26:26.342 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-20T11:26:26.352 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-20T11:26:26.353 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-20T11:26:26.365 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-11-20T11:26:26.366 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-11-20T11:26:26.373 INFO:teuthology.task.internal:Configuring sudo... 2020-11-20T11:26:26.374 INFO:teuthology.orchestra.run.smithi023:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-11-20T11:26:26.409 INFO:teuthology.orchestra.run.smithi155:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-11-20T11:26:26.435 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-11-20T11:26:26.443 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-11-20T11:26:26.444 INFO:teuthology.orchestra.run.smithi023:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-11-20T11:26:26.477 INFO:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-11-20T11:26:26.492 INFO:teuthology.orchestra.run.smithi023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-20T11:26:26.572 INFO:teuthology.orchestra.run.smithi023:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-20T11:26:26.624 INFO:teuthology.orchestra.run.smithi023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-20T11:26:26.731 INFO:teuthology.orchestra.run.smithi023:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-20T11:26:26.782 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-11-20T11:26:27.016 INFO:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-20T11:26:27.039 INFO:teuthology.orchestra.run.smithi023.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-20T11:26:27.050 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2020-11-20T11:26:27.050 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2020-11-20T11:26:27.051 INFO:teuthology.orchestra.run.smithi023.stderr:152 bytes copied, 3.4001e-05 s, 4.5 MB/s 2020-11-20T11:26:27.052 INFO:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-20T11:26:27.095 INFO:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2020-11-20T11:26:27.176 INFO:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-20T11:26:27.226 INFO:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2020-11-20T11:26:27.261 INFO:teuthology.orchestra.run.smithi155:> set -ex 2020-11-20T11:26:27.262 INFO:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-20T11:26:27.317 INFO:teuthology.orchestra.run.smithi155.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-11-20T11:26:27.327 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records in 2020-11-20T11:26:27.327 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records out 2020-11-20T11:26:27.327 INFO:teuthology.orchestra.run.smithi155.stderr:152 bytes copied, 3.167e-05 s, 4.8 MB/s 2020-11-20T11:26:27.328 INFO:teuthology.orchestra.run.smithi023:> sudo service rsyslog restart 2020-11-20T11:26:27.332 INFO:teuthology.orchestra.run.smithi155:> sudo service rsyslog restart 2020-11-20T11:26:27.400 INFO:teuthology.orchestra.run.smithi023.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2020-11-20T11:26:27.436 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2020-11-20T11:26:27.850 INFO:teuthology.run_tasks:Running task internal.timer... 2020-11-20T11:26:27.863 INFO:teuthology.task.internal:Starting timer... 2020-11-20T11:26:27.864 INFO:teuthology.run_tasks:Running task pcp... 2020-11-20T11:26:27.911 INFO:teuthology.run_tasks:Running task selinux... 2020-11-20T11:26:27.923 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2020-11-20T11:26:27.924 INFO:teuthology.orchestra.run.smithi023:> sudo service auditd rotate 2020-11-20T11:26:27.985 INFO:teuthology.orchestra.run.smithi023.stdout:Rotating logs: [ OK ] 2020-11-20T11:26:27.987 INFO:teuthology.orchestra.run.smithi155:> sudo service auditd rotate 2020-11-20T11:26:28.060 INFO:teuthology.orchestra.run.smithi155.stdout:Rotating logs: [ OK ] 2020-11-20T11:26:28.062 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-11-20T11:26:28.062 INFO:teuthology.orchestra.run.smithi023:> /usr/sbin/getenforce 2020-11-20T11:26:28.081 INFO:teuthology.orchestra.run.smithi023.stdout:Permissive 2020-11-20T11:26:28.082 INFO:teuthology.orchestra.run.smithi155:> /usr/sbin/getenforce 2020-11-20T11:26:28.106 INFO:teuthology.orchestra.run.smithi155.stdout:Permissive 2020-11-20T11:26:28.107 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi023.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi155.front.sepia.ceph.com': 'permissive'} 2020-11-20T11:26:28.107 INFO:teuthology.orchestra.run.smithi023:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' 2020-11-20T11:26:28.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-20T11:26:28.149 INFO:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' 2020-11-20T11:26:28.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-20T11:26:28.175 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-11-20T11:26:28.176 INFO:teuthology.orchestra.run.smithi023:> sudo /usr/sbin/setenforce permissive 2020-11-20T11:26:28.217 INFO:teuthology.orchestra.run.smithi155:> sudo /usr/sbin/setenforce permissive 2020-11-20T11:26:28.243 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-11-20T11:26:28.261 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated; assuming it is current 2020-11-20T11:26:28.262 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-11-20T11:26:28.282 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-11-20T11:26:28.284 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi023.front.sepia.ceph.com,smithi155.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-11-20T11:26:28.689 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-11-20T11:26:29.652 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-11-20T11:26:32.427 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-11-20T11:26:32.591 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:32 +0000 (0:00:00.070) 0:00:00.071 ******* Friday 20 November 2020 11:26:32 +0000 (0:00:00.012) 0:00:00.083 ******* 2020-11-20T11:26:33.980 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi023.front.sepia.ceph.com] 2020-11-20T11:26:34.050 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:34 +0000 (0:00:01.591) 0:00:01.674 ******* 2020-11-20T11:26:34.093 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] 2020-11-20T11:26:34.168 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:34 +0000 (0:00:00.117) 0:00:01.791 ******* 2020-11-20T11:26:34.385 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2020-11-20T11:26:34.386 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-11-20T11:26:34.418 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:34 +0000 (0:00:00.250) 0:00:02.042 ******* 2020-11-20T11:26:34.425 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-11-20T11:26:34.464 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:34 +0000 (0:00:00.046) 0:00:02.088 ******* 2020-11-20T11:26:34.827 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2020-11-20T11:26:34.829 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-11-20T11:26:34.861 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:34 +0000 (0:00:00.396) 0:00:02.485 ******* 2020-11-20T11:26:34.869 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-11-20T11:26:34.902 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:34 +0000 (0:00:00.041) 0:00:02.526 ******* 2020-11-20T11:26:34.908 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2020-11-20T11:26:34.910 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:34.942 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:34 +0000 (0:00:00.039) 0:00:02.566 ******* 2020-11-20T11:26:34.948 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:34.984 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:34 +0000 (0:00:00.041) 0:00:02.608 ******* 2020-11-20T11:26:35.240 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2020-11-20T11:26:35.241 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-20T11:26:35.274 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:35 +0000 (0:00:00.289) 0:00:02.898 ******* 2020-11-20T11:26:35.281 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-20T11:26:35.316 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:35 +0000 (0:00:00.042) 0:00:02.940 ******* 2020-11-20T11:26:36.200 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-11-20T11:26:36.201 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1605871595.2880898-1368166011783/source", "state": "file", "uid": 0} 2020-11-20T11:26:36.235 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:36 +0000 (0:00:00.918) 0:00:03.859 ******* 2020-11-20T11:26:36.241 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1605871595.3288257-251276112083277/source", "state": "file", "uid": 0} 2020-11-20T11:26:36.278 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:36 +0000 (0:00:00.042) 0:00:03.902 ******* 2020-11-20T11:26:36.722 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2020-11-20T11:26:36.724 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-11-20T11:26:36.725 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-11-20T11:26:36.756 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-11-20T11:26:36.757 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-11-20T11:26:36.866 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2020-11-20T11:26:36.874 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-11-20T11:26:37.019 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:37 +0000 (0:00:00.740) 0:00:04.643 ******* 2020-11-20T11:26:37.052 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:37 +0000 (0:00:00.032) 0:00:04.676 ******* 2020-11-20T11:26:37.656 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2020-11-20T11:26:37.658 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] 2020-11-20T11:26:37.700 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] 2020-11-20T11:26:37.743 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2020-11-20T11:26:37.744 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:37 +0000 (0:00:00.691) 0:00:05.367 ******* 2020-11-20T11:26:37.811 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:37 +0000 (0:00:00.067) 0:00:05.435 ******* 2020-11-20T11:26:38.199 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** changed: [smithi023.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.151971", "end": "2020-11-20 11:26:38.175436", "rc": 0, "start": "2020-11-20 11:26:38.023465", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2020-11-20T11:26:38.234 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:38 +0000 (0:00:00.422) 0:00:05.858 ******* 2020-11-20T11:26:38.240 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.155049", "end": "2020-11-20 11:26:38.178050", "rc": 0, "start": "2020-11-20 11:26:38.023001", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2020-11-20T11:26:38.274 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:38 +0000 (0:00:00.040) 0:00:05.898 ******* 2020-11-20T11:26:38.439 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-11-20T11:26:38.441 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.012959", "end": "2020-11-20 11:26:38.411594", "failed_when_result": false, "rc": 0, "start": "2020-11-20 11:26:38.398635", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2020-11-20T11:26:38.480 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:38 +0000 (0:00:00.205) 0:00:06.104 ******* 2020-11-20T11:26:38.487 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.013873", "end": "2020-11-20 11:26:38.447465", "failed_when_result": false, "rc": 0, "start": "2020-11-20 11:26:38.433592", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2020-11-20T11:26:38.524 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:38 +0000 (0:00:00.043) 0:00:06.148 ******* 2020-11-20T11:26:38.530 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-11-20T11:26:38.531 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-11-20T11:26:38.565 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:38 +0000 (0:00:00.040) 0:00:06.189 ******* 2020-11-20T11:26:38.571 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-11-20T11:26:38.605 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:38 +0000 (0:00:00.040) 0:00:06.229 ******* 2020-11-20T11:26:38.611 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-11-20T11:26:38.613 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:38.647 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:38 +0000 (0:00:00.040) 0:00:06.270 ******* 2020-11-20T11:26:38.776 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 11:26:38 +0000 (0:00:00.055) 0:00:06.326 ******* TASK [users : Remove managed_admin_users from managed_users] ******************* skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:38.779 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:38 +0000 (0:00:00.074) 0:00:06.401 ******* 2020-11-20T11:26:38.794 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:38.840 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:38 +0000 (0:00:00.063) 0:00:06.464 ******* 2020-11-20T11:26:38.848 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-11-20T11:26:38.850 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:38.890 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:38 +0000 (0:00:00.048) 0:00:06.513 ******* 2020-11-20T11:26:38.895 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:38.934 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:38 +0000 (0:00:00.044) 0:00:06.558 ******* 2020-11-20T11:26:38.940 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-11-20T11:26:38.942 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:38.976 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:38 +0000 (0:00:00.042) 0:00:06.600 ******* 2020-11-20T11:26:38.983 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.017 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:39 +0000 (0:00:00.040) 0:00:06.641 ******* 2020-11-20T11:26:39.023 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-11-20T11:26:39.024 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.058 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:39 +0000 (0:00:00.040) 0:00:06.681 ******* 2020-11-20T11:26:39.064 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.100 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:39 +0000 (0:00:00.042) 0:00:06.724 ******* 2020-11-20T11:26:39.106 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-11-20T11:26:39.107 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.108 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.119 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.121 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.133 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.142 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.144 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.156 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.158 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjam 2020-11-20T11:26:39.160 INFO:teuthology.task.ansible.out:in@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.170 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.172 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.173 INFO:teuthology.task.ansible.out: skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1Eci 2020-11-20T11:26:39.176 INFO:teuthology.task.ansible.out:VNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.185 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.188 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841e 2020-11-20T11:26:39.190 INFO:teuthology.task.ansible.out:e62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.198 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.200 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front. 2020-11-20T11:26:39.201 INFO:teuthology.task.ansible.out:sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.215 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f30 2020-11-20T11:26:39.217 INFO:teuthology.task.ansible.out:09adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.230 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pk 2020-11-20T11:26:39.232 INFO:teuthology.task.ansible.out:dQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.242 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.244 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef 2020-11-20T11:26:39.245 INFO:teuthology.task.ansible.out:168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ 2020-11-20T11:26:39.247 INFO:teuthology.task.ansible.out:ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.256 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.268 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.269 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.279 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.281 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.292 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.294 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 2020-11-20T11:26:39.296 INFO:teuthology.task.ansible.out: 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115 2020-11-20T11:26:39.297 INFO:teuthology.task.ansible.out:c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.306 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.309 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.318 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.319 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.319 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.319 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.321 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.332 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.335 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-des 2020-11-20T11:26:39.337 INFO:teuthology.task.ansible.out:ktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.346 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.348 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.359 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.360 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.371 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.374 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.384 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.386 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.398 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"na 2020-11-20T11:26:39.400 INFO:teuthology.task.ansible.out:me": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.410 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.412 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.423 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.425 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.426 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "i 2020-11-20T11:26:39.427 INFO:teuthology.task.ansible.out:tem", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.437 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.439 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.449 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.451 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.462 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.464 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.475 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.487 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.498 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.500 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.503 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.511 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.514 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.525 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.527 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.544 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.546 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.547 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.547 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed 2020-11-20T11:26:39.550 INFO:teuthology.task.ansible.out:": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.559 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.562 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.562 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.574 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.574 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.586 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.590 INFO:teuthology.task.ansible.out: 2020-11-20T11:26:39.592 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.595 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.605 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.607 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.609 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.611 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.625 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.628 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.631 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.640 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.642 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.643 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.645 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.645 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.646 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.658 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.660 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.663 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.674 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.676 INFO:teuthology.task.ansible.out: 2020-11-20T11:26:39.687 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.688 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.689 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.690 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.691 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.693 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.705 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.706 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.707 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.718 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.720 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.720 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.722 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.730 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.736 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.737 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.739 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.740 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.751 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.753 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.754 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.765 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.766 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.767 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.768 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.768 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.781 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.783 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.784 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.786 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.798 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.799 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.800 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.801 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.801 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus 2020-11-20T11:26:39.803 INFO:teuthology.task.ansible.out:HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.813 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.813 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.815 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.815 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.827 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.828 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.830 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.842 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.844 INFO:teuthology.task.ansible.out: skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.844 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.845 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.846 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmun 2020-11-20T11:26:39.847 INFO:teuthology.task.ansible.out:julu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.857 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.860 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.876 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.878 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.919 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:39 +0000 (0:00:00.819) 0:00:07.543 ******* 2020-11-20T11:26:39.926 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.928 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.929 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBo 2020-11-20T11:26:39.930 INFO:teuthology.task.ansible.out:x 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2 2020-11-20T11:26:39.932 INFO:teuthology.task.ansible.out:e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.942 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.945 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} TASK [users : Create all users without sudo access.] *************************** TASK [users : Create all admin users with sudo access.] ************************ skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.989 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:39 +0000 (0:00:00.069) 0:00:07.613 ******* 2020-11-20T11:26:39.996 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:39.998 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.002 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "ite 2020-11-20T11:26:40.004 INFO:teuthology.task.ansible.out:m", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.027 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.074 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:40 +0000 (0:00:00.085) 0:00:07.698 ******* 2020-11-20T11:26:40.111 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:40 +0000 (0:00:00.037) 0:00:07.735 ******* 2020-11-20T11:26:40.120 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.154 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:40 +0000 (0:00:00.041) 0:00:07.777 ******* 2020-11-20T11:26:40.159 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-11-20T11:26:40.194 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:40 +0000 (0:00:00.041) 0:00:07.818 ******* 2020-11-20T11:26:40.201 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-11-20T11:26:40.203 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.236 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:40 +0000 (0:00:00.041) 0:00:07.860 ******* 2020-11-20T11:26:40.242 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-11-20T11:26:40.244 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.281 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:40 +0000 (0:00:00.045) 0:00:07.905 ******* 2020-11-20T11:26:40.287 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-11-20T11:26:40.289 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.322 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:40 +0000 (0:00:00.040) 0:00:07.946 ******* 2020-11-20T11:26:40.328 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-11-20T11:26:40.330 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.364 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:40 +0000 (0:00:00.041) 0:00:07.988 ******* 2020-11-20T11:26:40.370 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2020-11-20T11:26:40.371 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.405 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:40 +0000 (0:00:00.041) 0:00:08.029 ******* 2020-11-20T11:26:40.411 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-11-20T11:26:40.413 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.448 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:40 +0000 (0:00:00.042) 0:00:08.072 ******* 2020-11-20T11:26:40.454 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-20T11:26:40.456 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.456 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.468 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.470 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.479 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.514 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:40 +0000 (0:00:00.065) 0:00:08.137 ******* 2020-11-20T11:26:40.519 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-11-20T11:26:40.521 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.521 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.522 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.534 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** skipping: [smithi023.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.570 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:40 +0000 (0:00:00.055) 0:00:08.193 ******* 2020-11-20T11:26:40.575 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.577 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.578 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.578 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.580 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.591 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.602 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.604 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.619 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.621 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.621 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.622 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.633 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.635 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.637 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.637 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.648 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.650 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.663 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.664 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.674 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.676 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.686 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.688 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.698 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.701 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.714 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.729 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.734 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.736 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.746 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.747 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.751 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.753 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.792 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:40 +0000 (0:00:00.222) 0:00:08.416 ******* 2020-11-20T11:26:40.800 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.802 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.803 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.814 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.815 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.826 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.837 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.839 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.850 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "skip_reason": "Conditional result was False"} 2020-11-20T11:26:40.897 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:40 +0000 (0:00:00.103) 0:00:08.520 ******* 2020-11-20T11:26:41.490 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-11-20T11:26:41.492 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "eefb312a9cf82af5b9a46fb88fa640babe5fe20f", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "state": "file", "uid": 0} 2020-11-20T11:26:41.524 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:41 +0000 (0:00:00.628) 0:00:09.148 ******* 2020-11-20T11:26:41.531 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "eefb312a9cf82af5b9a46fb88fa640babe5fe20f", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "state": "file", "uid": 0} 2020-11-20T11:26:41.566 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:41 +0000 (0:00:00.041) 0:00:09.190 ******* 2020-11-20T11:26:41.572 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-11-20T11:26:41.574 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => { "msg": "Host smithi155.front.sepia.ceph.com is running RedHat 8.3 (Ootpa)" } 2020-11-20T11:26:41.608 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:41 +0000 (0:00:00.041) 0:00:09.232 ******* 2020-11-20T11:26:41.615 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => { "msg": "Host smithi023.front.sepia.ceph.com is running RedHat 8.3 (Ootpa)" } 2020-11-20T11:26:41.657 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:41 +0000 (0:00:00.049) 0:00:09.281 ******* 2020-11-20T11:26:41.666 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-11-20T11:26:41.668 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml) => {"ansible_facts": {"container_mirror_cert_path": "/etc/containers/certs.d/{{ container_mirror }}", "nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"], "rhsm_repos": ["rhel-8-for-x86_64-baseos-rpms", "rhel-8-for-x86_64-appstream-rpms"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"} 2020-11-20T11:26:41.701 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:41 +0000 (0:00:00.043) 0:00:09.325 ******* 2020-11-20T11:26:41.707 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml) => {"ansible_facts": {"container_mirror_cert_path": "/etc/containers/certs.d/{{ container_mirror }}", "nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"], "rhsm_repos": ["rhel-8-for-x86_64-baseos-rpms", "rhel-8-for-x86_64-appstream-rpms"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"} 2020-11-20T11:26:41.743 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:41 +0000 (0:00:00.041) 0:00:09.367 ******* 2020-11-20T11:26:41.749 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-11-20T11:26:41.750 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:41.783 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:41 +0000 (0:00:00.040) 0:00:09.407 ******* 2020-11-20T11:26:41.789 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:41.825 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:41 +0000 (0:00:00.042) 0:00:09.449 ******* 2020-11-20T11:26:42.270 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-11-20T11:26:42.271 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.308884", "end": "2020-11-20 11:26:42.250191", "rc": 0, "start": "2020-11-20 11:26:41.941307", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-11-20T11:26:42.305 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:42 +0000 (0:00:00.479) 0:00:09.929 ******* 2020-11-20T11:26:42.331 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-11-20T11:26:42.333 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:42.367 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:42 +0000 (0:00:00.061) 0:00:09.991 ******* 2020-11-20T11:26:42.373 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-11-20T11:26:42.375 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.339687", "end": "2020-11-20 11:26:42.333015", "rc": 0, "start": "2020-11-20 11:26:41.993328", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-11-20T11:26:42.408 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:42 +0000 (0:00:00.041) 0:00:10.032 ******* 2020-11-20T11:26:42.420 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-11-20T11:26:42.422 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:42.454 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:42 +0000 (0:00:00.045) 0:00:10.078 ******* 2020-11-20T11:26:42.461 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:42.495 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:42 +0000 (0:00:00.041) 0:00:10.119 ******* 2020-11-20T11:26:42.502 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-11-20T11:26:42.503 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:42.537 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:42 +0000 (0:00:00.041) 0:00:10.160 ******* 2020-11-20T11:26:42.543 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-11-20T11:26:42.544 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:42.582 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:42 +0000 (0:00:00.045) 0:00:10.205 ******* 2020-11-20T11:26:42.588 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-11-20T11:26:42.590 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:42.624 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:42 +0000 (0:00:00.041) 0:00:10.247 ******* 2020-11-20T11:26:42.629 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-11-20T11:26:42.631 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:42.664 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:42 +0000 (0:00:00.040) 0:00:10.288 ******* 2020-11-20T11:26:42.670 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-11-20T11:26:42.672 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:42.705 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:42 +0000 (0:00:00.041) 0:00:10.329 ******* 2020-11-20T11:26:42.711 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-11-20T11:26:42.713 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:42.747 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:42 +0000 (0:00:00.041) 0:00:10.371 ******* 2020-11-20T11:26:42.753 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-11-20T11:26:42.755 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"entitlements_path": "/etc/ansible/secrets/entitlements.yml"}, "changed": false} 2020-11-20T11:26:42.789 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:42 +0000 (0:00:00.040) 0:00:10.412 ******* 2020-11-20T11:26:42.794 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-11-20T11:26:42.796 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:26:42.830 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:42 +0000 (0:00:00.041) 0:00:10.454 ******* 2020-11-20T11:26:42.846 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-11-20T11:26:42.848 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ok: [smithi155.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T11:26:42.880 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:42 +0000 (0:00:00.050) 0:00:10.504 ******* 2020-11-20T11:26:42.885 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-11-20T11:26:42.887 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"entitlements_path": "/etc/ansible/secrets/entitlements.yml"}, "changed": false} 2020-11-20T11:26:42.921 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:42 +0000 (0:00:00.040) 0:00:10.545 ******* 2020-11-20T11:26:42.927 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-11-20T11:26:42.928 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"have_entitlements": true}, "changed": false} 2020-11-20T11:26:42.962 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:42 +0000 (0:00:00.040) 0:00:10.586 ******* 2020-11-20T11:26:42.967 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-11-20T11:26:42.968 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ok: [smithi023.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T11:26:43.002 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:42 +0000 (0:00:00.040) 0:00:10.626 ******* 2020-11-20T11:26:43.039 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-11-20T11:26:43.041 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"have_entitlements": true}, "changed": false} 2020-11-20T11:26:43.073 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:43 +0000 (0:00:00.070) 0:00:10.697 ******* 2020-11-20T11:26:44.106 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-11-20T11:26:44.295 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed /root/.ansible/tmp/ansible-moduletmp-1605871603.9765368-4q07rwlr/katello-ca-consumer-latest.noarchasdutupf.rpm"]} Friday 20 November 2020 11:26:44 +0000 (0:00:01.067) 0:00:11.764 ******* ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed /root/.ansible/tmp/ansible-moduletmp-1605871604.0243747-nk1tyt7t/katello-ca-consumer-latest.noarchlwyok51y.rpm"]} Friday 20 November 2020 11:26:44 +0000 (0:00:00.051) 0:00:11.816 ******* 2020-11-20T11:26:44.736 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-11-20T11:26:44.738 INFO:teuthology.task.ansible.out:fatal: [smithi155.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ...ignoring 2020-11-20T11:26:44.774 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:44 +0000 (0:00:00.582) 0:00:12.398 ******* 2020-11-20T11:26:44.784 INFO:teuthology.task.ansible.out:fatal: [smithi023.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ...ignoring 2020-11-20T11:26:44.835 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:44 +0000 (0:00:00.060) 0:00:12.459 ******* 2020-11-20T11:26:44.845 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": false}, "changed": false} 2020-11-20T11:26:44.885 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:44 +0000 (0:00:00.049) 0:00:12.509 ******* 2020-11-20T11:26:44.891 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": false}, "changed": false} 2020-11-20T11:26:44.927 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:26:44 +0000 (0:00:00.042) 0:00:12.551 ******* 2020-11-20T11:27:00.126 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-11-20T11:27:00.127 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"attempts": 1, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2020-11-20T11:27:00.160 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:00 +0000 (0:00:15.232) 0:00:27.784 ******* 2020-11-20T11:27:00.197 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": true}, "changed": false} 2020-11-20T11:27:00.231 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:00 +0000 (0:00:00.071) 0:00:27.855 ******* 2020-11-20T11:27:00.826 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-11-20T11:27:00.828 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager release --list | grep -E '[0-9]'", "delta": "0:00:00.395228", "end": "2020-11-20 11:27:00.792946", "failed_when_result": false, "rc": 0, "start": "2020-11-20 11:27:00.397718", "stderr": "", "stderr_lines": [], "stdout": "7.4\n7.5\n7.6\n7.7\n7.8\n7.9\n7Server\n8\n8.0\n8.1\n8.2\n8.3", "stdout_lines": ["7.4", "7.5", "7.6", "7.7", "7.8", "7.9", "7Server", "8", "8.0", "8.1", "8.2", "8.3"]} 2020-11-20T11:27:00.860 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:00 +0000 (0:00:00.629) 0:00:28.484 ******* 2020-11-20T11:27:00.887 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-11-20T11:27:00.889 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:00.921 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:00 +0000 (0:00:00.060) 0:00:28.545 ******* 2020-11-20T11:27:00.969 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-11-20T11:27:00.971 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:01.004 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:00 +0000 (0:00:00.082) 0:00:28.628 ******* 2020-11-20T11:27:01.031 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-11-20T11:27:01.033 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T11:27:01.066 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:01 +0000 (0:00:00.061) 0:00:28.690 ******* 2020-11-20T11:27:01.092 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-11-20T11:27:01.094 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:01.127 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:01 +0000 (0:00:00.060) 0:00:28.751 ******* 2020-11-20T11:27:03.204 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-11-20T11:27:03.600 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"attempts": 1, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} Friday 20 November 2020 11:27:03 +0000 (0:00:02.111) 0:00:30.862 ******* TASK [common : Set rhsm_registered if we just registered] ********************** ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": true}, "changed": false} Friday 20 November 2020 11:27:03 +0000 (0:00:00.071) 0:00:30.934 ******* 2020-11-20T11:27:03.733 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-11-20T11:27:03.735 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager repos --list | grep -B4 'Enabled:.*1' | grep 'Repo ID:' | sed -e 's/Repo ID:\\s*\\(.*\\)/\\1/' | sort", "delta": "0:00:02.407544", "end": "2020-11-20 11:27:03.704441", "rc": 0, "start": "2020-11-20 11:27:01.296897", "stderr": "", "stderr_lines": [], "stdout": "rhel-8-for-x86_64-appstream-rpms\nrhel-8-for-x86_64-baseos-rpms", "stdout_lines": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]} 2020-11-20T11:27:03.768 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:03 +0000 (0:00:00.457) 0:00:31.392 ******* 2020-11-20T11:27:03.805 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-11-20T11:27:03.807 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"repo_list": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]}, "changed": false} 2020-11-20T11:27:03.839 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:03 +0000 (0:00:00.071) 0:00:31.463 ******* 2020-11-20T11:27:03.876 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-11-20T11:27:03.878 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:03.910 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:03 +0000 (0:00:00.071) 0:00:31.534 ******* 2020-11-20T11:27:03.918 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-11-20T11:27:03.920 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager release --list | grep -E '[0-9]'", "delta": "0:00:00.371841", "end": "2020-11-20 11:27:03.862369", "failed_when_result": false, "rc": 0, "start": "2020-11-20 11:27:03.490528", "stderr": "", "stderr_lines": [], "stdout": "7.4\n7.5\n7.6\n7.7\n7.8\n7.9\n7Server\n8\n8.0\n8.1\n8.2\n8.3", "stdout_lines": ["7.4", "7.5", "7.6", "7.7", "7.8", "7.9", "7Server", "8", "8.0", "8.1", "8.2", "8.3"]} 2020-11-20T11:27:03.960 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:03 +0000 (0:00:00.049) 0:00:31.583 ******* 2020-11-20T11:27:03.966 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-11-20T11:27:03.967 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": false}, "changed": false} 2020-11-20T11:27:04.004 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:03 +0000 (0:00:00.043) 0:00:31.627 ******* 2020-11-20T11:27:04.010 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-11-20T11:27:04.011 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:04.045 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:04 +0000 (0:00:00.042) 0:00:31.669 ******* 2020-11-20T11:27:04.051 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-11-20T11:27:04.053 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2020-11-20T11:27:04.087 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:04 +0000 (0:00:00.041) 0:00:31.710 ******* 2020-11-20T11:27:04.093 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-11-20T11:27:04.095 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:04.131 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:04 +0000 (0:00:00.044) 0:00:31.755 ******* 2020-11-20T11:27:04.159 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-11-20T11:27:04.161 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T11:27:04.192 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:04 +0000 (0:00:00.061) 0:00:31.816 ******* 2020-11-20T11:27:04.219 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:04.252 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:04 +0000 (0:00:00.060) 0:00:31.876 ******* 2020-11-20T11:27:06.715 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-11-20T11:27:06.717 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager repos --list | grep -B4 'Enabled:.*1' | grep 'Repo ID:' | sed -e 's/Repo ID:\\s*\\(.*\\)/\\1/' | sort", "delta": "0:00:02.251039", "end": "2020-11-20 11:27:06.691972", "rc": 0, "start": "2020-11-20 11:27:04.440933", "stderr": "", "stderr_lines": [], "stdout": "rhel-8-for-x86_64-appstream-rpms\nrhel-8-for-x86_64-baseos-rpms", "stdout_lines": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]} 2020-11-20T11:27:06.751 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:06 +0000 (0:00:02.498) 0:00:34.375 ******* 2020-11-20T11:27:06.757 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-11-20T11:27:06.759 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2020-11-20T11:27:06.797 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:06 +0000 (0:00:00.046) 0:00:34.421 ******* 2020-11-20T11:27:06.807 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-11-20T11:27:06.808 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"repo_list": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]}, "changed": false} 2020-11-20T11:27:06.863 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:06 +0000 (0:00:00.065) 0:00:34.487 ******* 2020-11-20T11:27:06.915 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-11-20T11:27:06.917 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:06.951 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:06 +0000 (0:00:00.087) 0:00:34.575 ******* 2020-11-20T11:27:06.990 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-11-20T11:27:06.993 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": false}, "changed": false} 2020-11-20T11:27:07.022 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:07 +0000 (0:00:00.071) 0:00:34.646 ******* 2020-11-20T11:27:07.060 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-11-20T11:27:07.062 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2020-11-20T11:27:07.103 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:07 +0000 (0:00:00.080) 0:00:34.727 ******* 2020-11-20T11:27:09.513 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-11-20T11:27:10.598 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} Friday 20 November 2020 11:27:09 +0000 (0:00:02.450) 0:00:37.178 ******* TASK [common : Enable necessary rhsm repos] ************************************ changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": ["subscription-manager", "repos", "--enable=rhel-8-for-x86_64-baseos-rpms", "--enable=rhel-8-for-x86_64-appstream-rpms"], "delta": "0:00:03.434349", "end": "2020-11-20 11:27:10.407788", "rc": 0, "start": "2020-11-20 11:27:06.973439", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system.\nRepository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system.", "Repository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system."]} Friday 20 November 2020 11:27:10 +0000 (0:00:00.910) 0:00:38.088 ******* 2020-11-20T11:27:10.660 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "path": "/etc/yum.repos.d/cd.repo", "state": "absent"} 2020-11-20T11:27:10.695 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:10 +0000 (0:00:00.231) 0:00:38.319 ******* 2020-11-20T11:27:10.885 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-11-20T11:27:10.887 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-20T11:27:10.918 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:10 +0000 (0:00:00.223) 0:00:38.542 ******* 2020-11-20T11:27:11.387 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-11-20T11:27:11.389 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2020-11-20T11:27:11.819 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2020-11-20T11:27:11.857 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:11 +0000 (0:00:00.938) 0:00:39.481 ******* 2020-11-20T11:27:12.316 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-11-20T11:27:12.318 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2020-11-20T11:27:12.750 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2020-11-20T11:27:12.793 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:12 +0000 (0:00:00.929) 0:00:40.410 ******* 2020-11-20T11:27:12.814 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:12.847 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:12 +0000 (0:00:00.061) 0:00:40.471 ******* 2020-11-20T11:27:12.874 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-11-20T11:27:12.876 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:12.910 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:12 +0000 (0:00:00.062) 0:00:40.534 ******* 2020-11-20T11:27:12.937 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-11-20T11:27:12.939 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:12.971 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:12 +0000 (0:00:00.061) 0:00:40.595 ******* 2020-11-20T11:27:12.998 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-11-20T11:27:12.999 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:13.032 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:13 +0000 (0:00:00.060) 0:00:40.655 ******* 2020-11-20T11:27:13.058 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-11-20T11:27:13.060 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:13.092 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:13 +0000 (0:00:00.060) 0:00:40.716 ******* 2020-11-20T11:27:13.119 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-11-20T11:27:13.120 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:13.153 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:13 +0000 (0:00:00.060) 0:00:40.776 ******* 2020-11-20T11:27:13.179 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-11-20T11:27:13.181 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:13.214 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:13 +0000 (0:00:00.061) 0:00:40.838 ******* 2020-11-20T11:27:13.241 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-11-20T11:27:13.243 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:13.274 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:13 +0000 (0:00:00.060) 0:00:40.898 ******* 2020-11-20T11:27:13.304 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:13.340 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:13 +0000 (0:00:00.065) 0:00:40.963 ******* 2020-11-20T11:27:14.122 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-11-20T11:27:14.124 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": ["subscription-manager", "repos", "--enable=rhel-8-for-x86_64-baseos-rpms", "--enable=rhel-8-for-x86_64-appstream-rpms"], "delta": "0:00:04.357166", "end": "2020-11-20 11:27:14.096681", "rc": 0, "start": "2020-11-20 11:27:09.739515", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system.\nRepository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system.", "Repository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system."]} 2020-11-20T11:27:14.161 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:14 +0000 (0:00:00.820) 0:00:41.784 ******* 2020-11-20T11:27:14.356 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-11-20T11:27:14.358 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "path": "/etc/yum.repos.d/cd.repo", "state": "absent"} 2020-11-20T11:27:14.391 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:14 +0000 (0:00:00.231) 0:00:42.015 ******* 2020-11-20T11:27:14.523 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-11-20T11:27:14.525 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.985340", "end": "2020-11-20 11:27:14.500326", "rc": 0, "start": "2020-11-20 11:27:13.514986", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T11:27:14.561 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:14 +0000 (0:00:00.169) 0:00:42.185 ******* 2020-11-20T11:27:14.579 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-11-20T11:27:14.580 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-20T11:27:14.615 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:14 +0000 (0:00:00.053) 0:00:42.238 ******* 2020-11-20T11:27:15.161 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** ok: [smithi155.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2020-11-20T11:27:15.602 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2020-11-20T11:27:15.638 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:15 +0000 (0:00:01.023) 0:00:43.262 ******* 2020-11-20T11:27:16.161 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-11-20T11:27:16.162 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2020-11-20T11:27:16.641 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2020-11-20T11:27:16.681 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:16 +0000 (0:00:01.042) 0:00:44.305 ******* 2020-11-20T11:27:16.708 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-11-20T11:27:16.710 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:16.762 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:16 +0000 (0:00:00.080) 0:00:44.385 ******* 2020-11-20T11:27:16.791 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-11-20T11:27:16.792 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:16.825 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:16 +0000 (0:00:00.062) 0:00:44.448 ******* 2020-11-20T11:27:16.850 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-11-20T11:27:16.852 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:16.884 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:16 +0000 (0:00:00.059) 0:00:44.508 ******* 2020-11-20T11:27:16.910 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-11-20T11:27:16.912 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:16.944 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:16 +0000 (0:00:00.060) 0:00:44.568 ******* 2020-11-20T11:27:16.971 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:17.012 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:17 +0000 (0:00:00.068) 0:00:44.636 ******* 2020-11-20T11:27:17.043 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-11-20T11:27:17.045 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:17.077 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:17 +0000 (0:00:00.064) 0:00:44.701 ******* 2020-11-20T11:27:17.103 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:17.139 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:17 +0000 (0:00:00.062) 0:00:44.763 ******* 2020-11-20T11:27:17.166 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:17.200 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:17 +0000 (0:00:00.060) 0:00:44.824 ******* 2020-11-20T11:27:17.226 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-11-20T11:27:17.228 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:17.259 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:17 +0000 (0:00:00.059) 0:00:44.883 ******* 2020-11-20T11:27:17.518 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-11-20T11:27:17.520 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.085723", "end": "2020-11-20 11:27:17.500338", "rc": 0, "start": "2020-11-20 11:27:17.414615", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T11:27:17.552 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:17 +0000 (0:00:00.293) 0:00:45.176 ******* 2020-11-20T11:27:17.611 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-11-20T11:27:17.615 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: krb5-workstation"]} 2020-11-20T11:27:17.659 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:17 +0000 (0:00:00.106) 0:00:45.283 ******* 2020-11-20T11:27:17.686 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-11-20T11:27:17.688 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:17.721 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:17 +0000 (0:00:00.060) 0:00:45.344 ******* 2020-11-20T11:27:17.747 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-11-20T11:27:17.749 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:17.780 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:17 +0000 (0:00:00.059) 0:00:45.404 ******* 2020-11-20T11:27:17.808 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-11-20T11:27:17.810 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:17.840 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:17 +0000 (0:00:00.060) 0:00:45.464 ******* 2020-11-20T11:27:17.867 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:17.906 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:17 +0000 (0:00:00.065) 0:00:45.530 ******* 2020-11-20T11:27:18.347 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-11-20T11:27:18.349 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2020-11-20T11:27:18.381 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:18 +0000 (0:00:00.474) 0:00:46.005 ******* 2020-11-20T11:27:20.298 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-11-20T11:27:20.300 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: krb5-workstation"]} 2020-11-20T11:27:20.333 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:20 +0000 (0:00:01.952) 0:00:47.957 ******* 2020-11-20T11:27:20.360 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-11-20T11:27:20.361 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:20.396 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:20 +0000 (0:00:00.062) 0:00:48.019 ******* 2020-11-20T11:27:20.422 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:20.455 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:20 +0000 (0:00:00.059) 0:00:48.079 ******* 2020-11-20T11:27:20.481 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-11-20T11:27:20.483 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:20.515 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:20 +0000 (0:00:00.059) 0:00:48.139 ******* 2020-11-20T11:27:20.541 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-11-20T11:27:20.543 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:20.575 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:20 +0000 (0:00:00.060) 0:00:48.199 ******* 2020-11-20T11:27:20.936 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-11-20T11:27:21.787 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: smartmontools"]} Friday 20 November 2020 11:27:20 +0000 (0:00:00.395) 0:00:48.595 ******* TASK [common : Add krb5 config file] ******************************************* ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} Friday 20 November 2020 11:27:21 +0000 (0:00:00.157) 0:00:48.752 ******* TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** ok: [smithi023.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2020-11-20T11:27:21.970 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2020-11-20T11:27:22.441 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2020-11-20T11:27:22.477 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:22 +0000 (0:00:01.349) 0:00:50.101 ******* 2020-11-20T11:27:22.664 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-11-20T11:27:22.665 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-11-20T11:27:22.698 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:22 +0000 (0:00:00.221) 0:00:50.322 ******* 2020-11-20T11:27:23.158 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-11-20T11:27:23.160 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2020-11-20T11:27:23.480 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-11-20T11:27:23.482 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: smartmontools"]} 2020-11-20T11:27:23.515 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:23 +0000 (0:00:00.816) 0:00:51.139 ******* 2020-11-20T11:27:23.563 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-11-20T11:27:23.565 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2020-11-20T11:27:23.964 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-11-20T11:27:23.965 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2020-11-20T11:27:23.976 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-11-20T11:27:23.978 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2020-11-20T11:27:24.015 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:24 +0000 (0:00:00.499) 0:00:51.638 ******* 2020-11-20T11:27:24.042 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:24.078 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:24 +0000 (0:00:00.063) 0:00:51.702 ******* 2020-11-20T11:27:24.116 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2020-11-20T11:27:24.160 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:24 +0000 (0:00:00.082) 0:00:51.784 ******* 2020-11-20T11:27:24.357 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* changed: [smithi023.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2020-11-20T11:27:24.391 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:24 +0000 (0:00:00.230) 0:00:52.015 ******* 2020-11-20T11:27:24.398 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-11-20T11:27:24.401 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2020-11-20T11:27:24.432 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** skipping: [smithi023.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-20T11:27:24.443 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-20T11:27:24.454 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-20T11:27:24.456 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-20T11:27:24.489 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:24 +0000 (0:00:00.097) 0:00:52.113 ******* 2020-11-20T11:27:24.790 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** ok: [smithi155.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2020-11-20T11:27:24.827 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:24 +0000 (0:00:00.338) 0:00:52.451 ******* 2020-11-20T11:27:25.011 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-11-20T11:27:25.014 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-11-20T11:27:25.048 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:25 +0000 (0:00:00.221) 0:00:52.672 ******* 2020-11-20T11:27:25.539 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-11-20T11:27:25.542 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2020-11-20T11:27:25.989 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2020-11-20T11:27:26.384 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2020-11-20T11:27:26.420 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:26 +0000 (0:00:01.371) 0:00:54.043 ******* 2020-11-20T11:27:26.446 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-11-20T11:27:26.448 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:26.480 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:26 +0000 (0:00:00.060) 0:00:54.104 ******* 2020-11-20T11:27:26.517 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-11-20T11:27:26.518 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2020-11-20T11:27:26.551 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:26 +0000 (0:00:00.070) 0:00:54.175 ******* 2020-11-20T11:27:26.755 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-11-20T11:27:26.757 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2020-11-20T11:27:26.789 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:26 +0000 (0:00:00.238) 0:00:54.413 ******* 2020-11-20T11:27:26.838 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-11-20T11:27:26.840 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-20T11:27:26.860 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-11-20T11:27:27.480 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} Friday 20 November 2020 11:27:26 +0000 (0:00:00.127) 0:00:54.540 ******* TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: nagios-common", "Installed: nrpe", "Installed: nagios-plugins", "Installed: nagios-plugins-load"]} Friday 20 November 2020 11:27:27 +0000 (0:00:00.425) 0:00:54.966 ******* TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 11:27:27 +0000 (0:00:00.060) 0:00:55.026 ******* 2020-11-20T11:27:27.932 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-11-20T11:27:27.934 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2020-11-20T11:27:27.970 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:27 +0000 (0:00:00.567) 0:00:55.594 ******* 2020-11-20T11:27:28.489 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2020-11-20T11:27:28.523 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:28 +0000 (0:00:00.553) 0:00:56.147 ******* 2020-11-20T11:27:28.560 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-11-20T11:27:28.562 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2020-11-20T11:27:28.595 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:28 +0000 (0:00:00.071) 0:00:56.219 ******* 2020-11-20T11:27:28.800 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-11-20T11:27:28.802 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-20T11:27:28.835 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:28 +0000 (0:00:00.240) 0:00:56.459 ******* 2020-11-20T11:27:29.105 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-11-20T11:27:29.107 INFO:teuthology.task.ansible.out:fatal: [smithi023.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2020-11-20T11:27:29.142 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:29 +0000 (0:00:00.306) 0:00:56.766 ******* 2020-11-20T11:27:29.169 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-11-20T11:27:29.171 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:29.205 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:29 +0000 (0:00:00.063) 0:00:56.829 ******* 2020-11-20T11:27:29.351 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-11-20T11:27:29.353 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: nagios-common", "Installed: nrpe", "Installed: nagios-plugins", "Installed: nagios-plugins-load"]} 2020-11-20T11:27:29.389 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:29 +0000 (0:00:00.183) 0:00:57.013 ******* 2020-11-20T11:27:29.427 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-11-20T11:27:29.429 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:29.468 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:29 +0000 (0:00:00.079) 0:00:57.092 ******* 2020-11-20T11:27:29.665 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-11-20T11:27:29.667 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2020-11-20T11:27:29.702 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:29 +0000 (0:00:00.233) 0:00:57.326 ******* 2020-11-20T11:27:29.918 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-11-20T11:27:29.920 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2020-11-20T11:27:29.952 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:29 +0000 (0:00:00.249) 0:00:57.576 ******* 2020-11-20T11:27:30.410 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-11-20T11:27:30.411 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2020-11-20T11:27:30.443 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:30 +0000 (0:00:00.490) 0:00:58.067 ******* 2020-11-20T11:27:30.449 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-11-20T11:27:30.451 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2020-11-20 11:25:47 UTC", "ActiveEnterTimestampMonotonic": "15939530", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "time-sync.target -.mount var-run.mount system.slice systemd-journald.socket sysinit.target systemd-tmpfiles-setup.service local-fs.target nss-lookup.target network.target basic.target tmp.mount", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2020-11-20 11:25:47 UTC", "AssertTimestampMonotonic": "15938268", "Before": "getty@tty1.service multi-user.target plymouth-quit.service xdm.service shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2020-11-20 11:25:47 UTC", "ConditionTimestampMonotonic": "15938268", "ConfigurationDirectoryMode": " 2020-11-20T11:27:30.453 INFO:teuthology.task.ansible.out:0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "982", "ExecMainStartTimestamp": "Fri 2020-11-20 11:25:47 UTC", "ExecMainStartTimestampMonotonic": "15939477", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[Fri 2020-11-20 11:25:47 UTC] ; stop_time=[n/a] ; pid=982 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "running", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2020-11-20 11:25:47 UTC", "InactiveExitTimestampMonotonic": "15939530", "InvocationID": "2fc8eb7332b4404382a12a83d2ca67e1", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infin 2020-11-20T11:27:30.454 INFO:teuthology.task.ansible.out:ity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127237", "LimitNPROCSoft": "127237", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127237", "LimitSIGPENDINGSoft": "127237", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "982", "MemoryAccounting": "yes", "MemoryCurrent": "1425408", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualSta 2020-11-20T11:27:30.456 INFO:teuthology.task.ansible.out:rt": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2020-11-20 11:25:47 UTC", "StateChangeTimestampMonotonic": "15939530", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203580", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2020-11-20 11:25:47 UTC", "WatchdogTimestampMonotonic": "15939529", "WatchdogUSec": "0"}} 2020-11-20T11:27:30.486 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:30 +0000 (0:00:00.043) 0:00:58.110 ******* 2020-11-20T11:27:30.492 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-11-20T11:27:30.494 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2020-11-20T11:27:30.503 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2020-11-20T11:27:30.539 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:30 +0000 (0:00:00.052) 0:00:58.163 ******* 2020-11-20T11:27:30.670 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-11-20T11:27:30.672 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001754", "end": "2020-11-20 11:27:30.646933", "rc": 0, "start": "2020-11-20 11:27:30.645179", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2020-11-20T11:27:30.709 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:30 +0000 (0:00:00.170) 0:00:58.333 ******* 2020-11-20T11:27:30.727 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-11-20T11:27:30.728 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-20T11:27:30.768 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:30 +0000 (0:00:00.058) 0:00:58.391 ******* 2020-11-20T11:27:31.038 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-11-20T11:27:31.039 INFO:teuthology.task.ansible.out:fatal: [smithi155.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2020-11-20T11:27:31.072 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:31 +0000 (0:00:00.304) 0:00:58.696 ******* 2020-11-20T11:27:31.100 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-11-20T11:27:31.101 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:27:31.134 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:31 +0000 (0:00:00.061) 0:00:58.757 ******* 2020-11-20T11:27:31.590 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-11-20T11:27:31.592 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2020-11-20T11:27:31.643 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:31 +0000 (0:00:00.509) 0:00:59.267 ******* 2020-11-20T11:27:31.961 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2020-11-20 11:25:23 UTC", "ActiveEnterTimestampMonotonic": "15187454", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "nss-lookup.target system.slice -.mount systemd-tmpfiles-setup.service local-fs.target systemd-journald.socket var-run.mount basic.target sysinit.target time-sync.target network.target tmp.mount", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2020-11-20 11:25:23 UTC", "AssertTimestampMonotonic": "15186150", "Before": "multi-user.target getty@tty1.service plymouth-quit.service xdm.service shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2020-11-20 11:25 2020-11-20T11:27:31.963 INFO:teuthology.task.ansible.out::23 UTC", "ConditionTimestampMonotonic": "15186150", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "988", "ExecMainStartTimestamp": "Fri 2020-11-20 11:25:23 UTC", "ExecMainStartTimestampMonotonic": "15187413", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[Fri 2020-11-20 11:25:23 UTC] ; stop_time=[n/a] ; pid=988 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "running", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2020-11-20 11:25:23 UTC", "InactiveExitTimestampMonotonic": "15187454", "InvocationID": "f5f9b21992af48eea728a1683b34ca03", "JobR 2020-11-20T11:27:31.965 INFO:teuthology.task.ansible.out:unningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127237", "LimitNPROCSoft": "127237", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127237", "LimitSIGPENDINGSoft": "127237", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "988", "MemoryAccounting": "yes", "MemoryCurrent": "1409024", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelMo 2020-11-20T11:27:31.966 INFO:teuthology.task.ansible.out:dules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/run/nrpe /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2020-11-20 11:25:23 UTC", "StateChangeTimestampMonotonic": "15187454", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203580", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2020-11-20 11:25:23 UTC", "WatchdogTimestampMonotonic": "15187453", "WatchdogUSec": "0"}} 2020-11-20T11:27:31.999 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:31 +0000 (0:00:00.355) 0:00:59.623 ******* 2020-11-20T11:27:32.196 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001709", "end": "2020-11-20 11:27:32.170291", "rc": 0, "start": "2020-11-20 11:27:32.168582", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2020-11-20T11:27:32.255 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:32 +0000 (0:00:00.255) 0:00:59.879 ******* 2020-11-20T11:27:33.281 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2020-11-20T11:27:33.283 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: python3-libsemanage", "Installed: python3-policycoreutils"]} 2020-11-20T11:27:33.328 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:33 +0000 (0:00:01.073) 0:01:00.952 ******* 2020-11-20T11:27:34.660 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: python3-libsemanage", "Installed: python3-policycoreutils"]} 2020-11-20T11:27:34.695 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:34 +0000 (0:00:01.367) 0:01:02.319 ******* 2020-11-20T11:27:36.030 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-11-20T11:27:36.032 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: selinux-policy-targeted"]} 2020-11-20T11:27:36.066 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:36 +0000 (0:00:01.370) 0:01:03.689 ******* 2020-11-20T11:27:36.949 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-11-20T11:27:36.951 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2020-11-20T11:27:36.985 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:36 +0000 (0:00:00.919) 0:01:04.609 ******* 2020-11-20T11:27:37.349 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-11-20T11:27:37.350 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: selinux-policy-targeted"]} 2020-11-20T11:27:37.388 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:37 +0000 (0:00:00.402) 0:01:05.011 ******* 2020-11-20T11:27:37.998 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-11-20T11:27:38.000 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2020-11-20T11:27:38.043 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:38 +0000 (0:00:00.655) 0:01:05.667 ******* 2020-11-20T11:27:54.070 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-11-20T11:27:54.072 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.894796", "end": "2020-11-20 11:27:54.041481", "failed_when_result": false, "rc": 0, "start": "2020-11-20 11:27:37.146685", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2020-11-20T11:27:54.110 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:54 +0000 (0:00:16.066) 0:01:21.734 ******* 2020-11-20T11:27:54.548 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.315256", "end": "2020-11-20 11:27:54.516987", "failed_when_result": false, "rc": 0, "start": "2020-11-20 11:27:38.201731", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2020-11-20T11:27:54.593 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:54 +0000 (0:00:00.482) 0:01:22.216 ******* 2020-11-20T11:27:54.600 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-11-20T11:27:54.602 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2020-11-20T11:27:54.646 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:54 +0000 (0:00:00.052) 0:01:22.269 ******* 2020-11-20T11:27:54.887 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-11-20T11:27:54.889 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.013824", "end": "2020-11-20 11:27:54.865145", "rc": 0, "start": "2020-11-20 11:27:54.851321", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T11:27:54.928 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:54 +0000 (0:00:00.282) 0:01:22.552 ******* 2020-11-20T11:27:55.066 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2020-11-20T11:27:55.104 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:55 +0000 (0:00:00.175) 0:01:22.727 ******* 2020-11-20T11:27:55.121 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-11-20T11:27:55.124 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.002319", "end": "2020-11-20 11:27:55.100332", "rc": 0, "start": "2020-11-20 11:27:55.098013", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T11:27:55.158 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:55 +0000 (0:00:00.054) 0:01:22.782 ******* 2020-11-20T11:27:55.292 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-11-20T11:27:55.295 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.009112", "end": "2020-11-20 11:27:55.273954", "rc": 0, "start": "2020-11-20 11:27:55.264842", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T11:27:55.339 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:55 +0000 (0:00:00.180) 0:01:22.963 ******* 2020-11-20T11:27:55.534 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-11-20T11:27:55.536 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.008142", "end": "2020-11-20 11:27:55.507965", "rc": 0, "start": "2020-11-20 11:27:55.499823", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T11:27:55.567 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:27:55 +0000 (0:00:00.228) 0:01:23.191 ******* 2020-11-20T11:28:11.889 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-11-20T11:28:11.891 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.133373", "end": "2020-11-20 11:28:11.862416", "rc": 0, "start": "2020-11-20 11:27:55.729043", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T11:28:11.923 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:11 +0000 (0:00:16.356) 0:01:39.547 ******* 2020-11-20T11:28:12.119 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-11-20T11:28:12.120 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2020-11-20T11:28:12.152 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:12 +0000 (0:00:00.228) 0:01:39.776 ******* 2020-11-20T11:28:12.179 INFO:teuthology.task.ansible.out: TASK [common : Check for podman] *********************************************** 2020-11-20T11:28:12.181 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : nrpe - Load SELinux policy package] ***************************** changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.830768", "end": "2020-11-20 11:28:12.156081", "rc": 0, "start": "2020-11-20 11:27:55.325313", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T11:28:12.214 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:12 +0000 (0:00:00.061) 0:01:39.838 ******* 2020-11-20T11:28:12.245 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:12 +0000 (0:00:00.031) 0:01:39.869 ******* 2020-11-20T11:28:12.283 INFO:teuthology.task.ansible.out: TASK [common : set_fact] ******************************************************* ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"container_mirror_cert_path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000"}, "changed": false} 2020-11-20T11:28:12.317 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:12 +0000 (0:00:00.071) 0:01:39.941 ******* 2020-11-20T11:28:12.343 INFO:teuthology.task.ansible.out: TASK [common : Check for docker] *********************************************** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:12.377 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:12 +0000 (0:00:00.060) 0:01:40.001 ******* 2020-11-20T11:28:12.404 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2020-11-20T11:28:12.406 INFO:teuthology.task.ansible.out: TASK [common : set_fact] ******************************************************* skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:12.438 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:12 +0000 (0:00:00.061) 0:01:40.062 ******* 2020-11-20T11:28:12.470 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:12 +0000 (0:00:00.031) 0:01:40.094 ******* 2020-11-20T11:28:12.476 INFO:teuthology.task.ansible.out: TASK [common : Check for podman] *********************************************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:12.512 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:12 +0000 (0:00:00.042) 0:01:40.136 ******* 2020-11-20T11:28:12.519 INFO:teuthology.task.ansible.out: TASK [common : Include encrypted variables] ************************************ 2020-11-20T11:28:12.520 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T11:28:12.531 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T11:28:12.583 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:12 +0000 (0:00:00.070) 0:01:40.207 ******* 2020-11-20T11:28:12.589 INFO:teuthology.task.ansible.out: TASK [common : set_fact] ******************************************************* 2020-11-20T11:28:12.592 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"container_mirror_cert_path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000"}, "changed": false} 2020-11-20T11:28:12.629 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:12 +0000 (0:00:00.045) 0:01:40.253 ******* 2020-11-20T11:28:12.657 INFO:teuthology.task.ansible.out: TASK [common : Check for docker] *********************************************** 2020-11-20T11:28:12.658 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:12.691 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:12 +0000 (0:00:00.061) 0:01:40.315 ******* 2020-11-20T11:28:12.718 INFO:teuthology.task.ansible.out: TASK [common : set_fact] ******************************************************* skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:12.753 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:12 +0000 (0:00:00.061) 0:01:40.377 ******* 2020-11-20T11:28:12.791 INFO:teuthology.task.ansible.out: TASK [common : Create /etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000] *** 2020-11-20T11:28:12.792 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-11-20T11:28:12.829 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:12 +0000 (0:00:00.076) 0:01:40.453 ******* 2020-11-20T11:28:12.839 INFO:teuthology.task.ansible.out: TASK [common : Include encrypted variables] ************************************ ok: [smithi023.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ok: [smithi023.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-11-20T11:28:12.878 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:12 +0000 (0:00:00.048) 0:01:40.501 ******* 2020-11-20T11:28:13.096 INFO:teuthology.task.ansible.out: TASK [common : Create /etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000] *** 2020-11-20T11:28:13.098 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-11-20T11:28:13.135 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:13 +0000 (0:00:00.257) 0:01:40.759 ******* 2020-11-20T11:28:13.366 INFO:teuthology.task.ansible.out: TASK [common : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "checksum": "d386a5d594d8bb6c408cf08541a4eb609577a03d", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "md5sum": "e2cf8b6a8ad65ba91ea9fc53d49f1bf4", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2236, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1605871692.8713984-143617625530660/source", "state": "file", "uid": 0} 2020-11-20T11:28:13.401 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:13 +0000 (0:00:00.265) 0:01:41.025 ******* 2020-11-20T11:28:13.427 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-11-20T11:28:13.429 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2020-11-20T11:28:13.461 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:13 +0000 (0:00:00.060) 0:01:41.085 ******* 2020-11-20T11:28:13.499 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* ok: [smithi155.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-11-20T11:28:13.534 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:13 +0000 (0:00:00.072) 0:01:41.158 ******* 2020-11-20T11:28:13.571 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-11-20T11:28:13.573 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "epel_packages": ["dbench"], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["dnf-utils", "git-all", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "podman", "podman-docker", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "autoconf", "gdb", "iozone", "lvm2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"} 2020-11-20T11:28:13.611 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:13 +0000 (0:00:00.077) 0:01:41.235 ******* 2020-11-20T11:28:13.650 INFO:teuthology.task.ansible.out: TASK [common : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2020-11-20T11:28:13.651 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "checksum": "d386a5d594d8bb6c408cf08541a4eb609577a03d", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "md5sum": "e2cf8b6a8ad65ba91ea9fc53d49f1bf4", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2236, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1605871693.1742384-158799907266516/source", "state": "file", "uid": 0} TASK [testnode : Including version specific variables.] ************************ ok: [smithi155.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-11-20T11:28:13.686 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:13 +0000 (0:00:00.074) 0:01:41.310 ******* 2020-11-20T11:28:13.717 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:13 +0000 (0:00:00.031) 0:01:41.341 ******* 2020-11-20T11:28:13.723 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-11-20T11:28:13.725 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2020-11-20T11:28:13.760 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:13 +0000 (0:00:00.043) 0:01:41.384 ******* 2020-11-20T11:28:13.797 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-11-20T11:28:13.799 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-11-20T11:28:13.833 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:13 +0000 (0:00:00.072) 0:01:41.457 ******* 2020-11-20T11:28:13.871 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** ok: [smithi023.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "epel_packages": ["dbench"], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["dnf-utils", "git-all", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "podman", "podman-docker", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "autoconf", "gdb", "iozone", "lvm2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"} 2020-11-20T11:28:13.906 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:13 +0000 (0:00:00.073) 0:01:41.530 ******* 2020-11-20T11:28:13.913 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-11-20T11:28:13.914 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-11-20T11:28:13.950 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:13 +0000 (0:00:00.043) 0:01:41.574 ******* 2020-11-20T11:28:13.957 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-11-20T11:28:13.959 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-11-20T11:28:13.994 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:13 +0000 (0:00:00.043) 0:01:41.617 ******* 2020-11-20T11:28:14.140 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "name": "ubuntu", "state": "present", "system": false} 2020-11-20T11:28:14.193 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:14 +0000 (0:00:00.199) 0:01:41.817 ******* 2020-11-20T11:28:14.201 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-11-20T11:28:14.202 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-11-20T11:28:14.245 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:14 +0000 (0:00:00.051) 0:01:41.868 ******* 2020-11-20T11:28:14.253 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-11-20T11:28:14.255 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:14.289 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:14 +0000 (0:00:00.044) 0:01:41.912 ******* 2020-11-20T11:28:14.316 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-11-20T11:28:14.317 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:14.350 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:14 +0000 (0:00:00.061) 0:01:41.974 ******* 2020-11-20T11:28:14.441 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-11-20T11:28:14.443 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "name": "ubuntu", "state": "present", "system": false} 2020-11-20T11:28:14.478 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:14 +0000 (0:00:00.127) 0:01:42.101 ******* 2020-11-20T11:28:14.506 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-11-20T11:28:14.508 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:14.555 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:14 +0000 (0:00:00.076) 0:01:42.178 ******* 2020-11-20T11:28:14.565 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-11-20T11:28:14.567 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2020-11-20T11:28:14.619 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:14 +0000 (0:00:00.064) 0:01:42.243 ******* 2020-11-20T11:28:14.628 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-11-20T11:28:14.631 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:14.665 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:14 +0000 (0:00:00.045) 0:01:42.289 ******* 2020-11-20T11:28:14.872 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-11-20T11:28:14.874 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2020-11-20T11:28:14.906 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:14 +0000 (0:00:00.241) 0:01:42.530 ******* 2020-11-20T11:28:15.111 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-11-20T11:28:15.113 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2020-11-20T11:28:15.163 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:15 +0000 (0:00:00.256) 0:01:42.787 ******* 2020-11-20T11:28:15.191 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-11-20T11:28:15.193 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi155"}, "changed": false} 2020-11-20T11:28:15.234 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:15 +0000 (0:00:00.071) 0:01:42.858 ******* 2020-11-20T11:28:15.421 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-11-20T11:28:15.423 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2020-11-20T11:28:15.462 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:15 +0000 (0:00:00.227) 0:01:43.085 ******* 2020-11-20T11:28:15.490 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi023"}, "changed": false} 2020-11-20T11:28:15.534 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:15 +0000 (0:00:00.072) 0:01:43.158 ******* 2020-11-20T11:28:16.241 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-11-20T11:28:16.243 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi155", "ansible_hostname": "smithi155", "ansible_nodename": "smithi155"}, "changed": false, "name": "smithi155"} 2020-11-20T11:28:16.275 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:16 +0000 (0:00:00.740) 0:01:43.899 ******* 2020-11-20T11:28:16.292 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi023", "ansible_hostname": "smithi023", "ansible_nodename": "smithi023"}, "changed": false, "name": "smithi023"} 2020-11-20T11:28:16.329 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:16 +0000 (0:00:00.051) 0:01:43.950 ******* 2020-11-20T11:28:16.703 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2020-11-20T11:28:16.705 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2020-11-20T11:28:16.740 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:16 +0000 (0:00:00.413) 0:01:44.364 ******* 2020-11-20T11:28:16.779 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2020-11-20T11:28:16.862 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:16 +0000 (0:00:00.074) 0:01:44.438 ******* 2020-11-20T11:28:17.178 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-11-20T11:28:17.180 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2020-11-20T11:28:17.214 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:17 +0000 (0:00:00.399) 0:01:44.838 ******* 2020-11-20T11:28:17.315 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2020-11-20T11:28:17.352 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:17 +0000 (0:00:00.136) 0:01:44.975 ******* 2020-11-20T11:28:17.878 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-11-20T11:28:17.880 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-11-20T11:28:17.921 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:17 +0000 (0:00:00.570) 0:01:45.545 ******* 2020-11-20T11:28:17.983 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-11-20T11:28:18.017 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:18 +0000 (0:00:00.095) 0:01:45.641 ******* 2020-11-20T11:28:18.367 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2020-11-20T11:28:18.404 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:18 +0000 (0:00:00.387) 0:01:46.028 ******* 2020-11-20T11:28:18.476 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2020-11-20T11:28:18.514 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:18 +0000 (0:00:00.109) 0:01:46.138 ******* 2020-11-20T11:28:18.596 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-11-20T11:28:18.598 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2020-11-20T11:28:18.632 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:18 +0000 (0:00:00.117) 0:01:46.255 ******* 2020-11-20T11:28:18.712 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2020-11-20T11:28:18.747 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:18 +0000 (0:00:00.115) 0:01:46.371 ******* 2020-11-20T11:28:18.821 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-11-20T11:28:18.823 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2020-11-20T11:28:18.856 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:18 +0000 (0:00:00.108) 0:01:46.480 ******* 2020-11-20T11:28:18.952 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2020-11-20T11:28:18.992 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:18 +0000 (0:00:00.136) 0:01:46.616 ******* 2020-11-20T11:28:19.074 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-11-20T11:28:19.639 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1236, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} Friday 20 November 2020 11:28:19 +0000 (0:00:00.130) 0:01:46.746 ******* ok: [smithi023.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1236, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} Friday 20 November 2020 11:28:19 +0000 (0:00:00.129) 0:01:46.876 ******* 2020-11-20T11:28:19.657 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2020-11-20T11:28:19.713 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:19 +0000 (0:00:00.452) 0:01:47.328 ******* 2020-11-20T11:28:19.820 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2020-11-20T11:28:19.854 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:19 +0000 (0:00:00.149) 0:01:47.478 ******* 2020-11-20T11:28:20.231 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-11-20T11:28:20.233 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2020-11-20T11:28:20.267 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:20 +0000 (0:00:00.412) 0:01:47.891 ******* 2020-11-20T11:28:20.358 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2020-11-20T11:28:20.396 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:20 +0000 (0:00:00.128) 0:01:48.019 ******* 2020-11-20T11:28:22.435 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ [WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2020-11-20T11:28:22.437 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.929935", "end": "2020-11-20 11:28:22.400143", "rc": 0, "start": "2020-11-20 11:28:20.470208", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T11:28:22.471 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:22 +0000 (0:00:02.074) 0:01:50.094 ******* 2020-11-20T11:28:22.499 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:22.533 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:22 +0000 (0:00:00.062) 0:01:50.157 ******* 2020-11-20T11:28:22.655 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-11-20T11:28:22.657 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.060530", "end": "2020-11-20 11:28:22.624280", "rc": 0, "start": "2020-11-20 11:28:20.563750", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T11:28:22.691 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:22 +0000 (0:00:00.157) 0:01:50.315 ******* 2020-11-20T11:28:22.721 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:22.755 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:22 +0000 (0:00:00.064) 0:01:50.379 ******* 2020-11-20T11:28:23.006 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-11-20T11:28:23.008 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2020-11-20T11:28:23.050 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:23 +0000 (0:00:00.293) 0:01:50.673 ******* 2020-11-20T11:28:23.076 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-11-20T11:28:23.111 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:23 +0000 (0:00:00.061) 0:01:50.735 ******* 2020-11-20T11:28:23.138 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-11-20T11:28:23.176 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:23 +0000 (0:00:00.064) 0:01:50.800 ******* 2020-11-20T11:28:23.246 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-11-20T11:28:23.339 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2020-11-20T11:28:23.341 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-11-20T11:28:23.343 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:23.374 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:23 +0000 (0:00:00.198) 0:01:50.998 ******* 2020-11-20T11:28:23.406 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:23 +0000 (0:00:00.031) 0:01:51.030 ******* 2020-11-20T11:28:23.412 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-11-20T11:28:23.448 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:23 +0000 (0:00:00.042) 0:01:51.072 ******* 2020-11-20T11:28:23.455 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* skipping: [smithi155.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-11-20T11:28:23.457 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-11-20T11:28:23.490 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:23 +0000 (0:00:00.042) 0:01:51.114 ******* 2020-11-20T11:28:23.496 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-11-20T11:28:23.533 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:23 +0000 (0:00:00.042) 0:01:51.157 ******* 2020-11-20T11:28:23.539 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-11-20T11:28:23.541 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:23.578 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:23 +0000 (0:00:00.045) 0:01:51.202 ******* 2020-11-20T11:28:23.585 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-11-20T11:28:23.587 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:23.621 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:23 +0000 (0:00:00.042) 0:01:51.244 ******* 2020-11-20T11:28:23.627 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T11:28:23.629 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:23.663 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:23 +0000 (0:00:00.042) 0:01:51.287 ******* 2020-11-20T11:28:23.671 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* skipping: [smithi023.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-11-20T11:28:23.705 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:23 +0000 (0:00:00.042) 0:01:51.329 ******* 2020-11-20T11:28:23.712 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-11-20T11:28:23.714 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:23.747 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:23 +0000 (0:00:00.041) 0:01:51.371 ******* 2020-11-20T11:28:23.753 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-11-20T11:28:23.755 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:23.792 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:23 +0000 (0:00:00.044) 0:01:51.415 ******* 2020-11-20T11:28:23.800 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:23.836 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:23 +0000 (0:00:00.044) 0:01:51.460 ******* 2020-11-20T11:28:23.842 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T11:28:23.843 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:23.878 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:23 +0000 (0:00:00.042) 0:01:51.502 ******* 2020-11-20T11:28:23.916 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-11-20T11:28:23.918 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:23.952 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:23 +0000 (0:00:00.074) 0:01:51.576 ******* 2020-11-20T11:28:23.992 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-11-20T11:28:23.994 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:24.027 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:24 +0000 (0:00:00.074) 0:01:51.651 ******* 2020-11-20T11:28:24.045 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-11-20T11:28:24.046 INFO:teuthology.task.ansible.out:fatal: [smithi155.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007288", "end": "2020-11-20 11:28:24.014121", "msg": "non-zero return code", "rc": 1, "start": "2020-11-20 11:28:24.006833", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2020-11-20T11:28:24.086 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:24 +0000 (0:00:00.058) 0:01:51.710 ******* 2020-11-20T11:28:24.124 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-11-20T11:28:24.126 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:24.168 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:24 +0000 (0:00:00.082) 0:01:51.792 ******* 2020-11-20T11:28:24.253 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-11-20T11:28:24.254 INFO:teuthology.task.ansible.out:fatal: [smithi023.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007480", "end": "2020-11-20 11:28:24.228831", "msg": "non-zero return code", "rc": 1, "start": "2020-11-20 11:28:24.221351", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2020-11-20T11:28:24.290 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:24 +0000 (0:00:00.120) 0:01:51.913 ******* 2020-11-20T11:28:24.328 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-11-20T11:28:24.329 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:24.362 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:24 +0000 (0:00:00.072) 0:01:51.986 ******* 2020-11-20T11:28:30.374 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:4.2.0-34.module+el8.3.0+7976+077be4ec.x86_64"]} 2020-11-20T11:28:30.409 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:30 +0000 (0:00:06.046) 0:01:58.033 ******* 2020-11-20T11:28:31.068 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:4.2.0-34.module+el8.3.0+7976+077be4ec.x86_64"]} 2020-11-20T11:28:31.102 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:31 +0000 (0:00:00.692) 0:01:58.725 ******* 2020-11-20T11:28:36.256 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-11-20T11:28:36.258 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2020-11-20T11:28:36.292 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:36 +0000 (0:00:05.190) 0:02:03.915 ******* 2020-11-20T11:28:37.290 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2020-11-20T11:28:37.328 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:37 +0000 (0:00:01.033) 0:02:04.949 ******* 2020-11-20T11:28:43.125 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-20T11:28:43.127 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: dnf-utils", "Installed: git-all", "Installed: sysstat", "Installed: libedit", "Installed: boost-thread", "Installed: xfsprogs", "Installed: gdisk", "Installed: parted", "Installed: libgcrypt", "Installed: fuse-libs", "Installed: openssl", "Installed: libuuid", "Installed: attr", "Installed: ant", "Installed: lsof", "Installed: gettext", "Installed: bc", "Installed: xfsdump", "Installed: blktrace", "Installed: usbredir", "Installed: podman", "Installed: podman-docker", "Installed: libev-devel", "Installed: valgrind", "Installed: nfs-utils", "Installed: ncurses-devel", "Installed: gcc", "Installed: git", "Installed: python3-nose", "Installed: python3-virtualenv", "Installed: genisoimage", "Installed: qemu-img", "Installed: qemu-kvm-core", "Installed: qemu-kvm-block-rbd", "Installed: libacl-devel", "Installed: autoconf", "Installed: gdb", "Installed: iozone", "Installed: lvm2", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:4.2.0-34.module+el8.3.0+7976+077be4ec.x86_64", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64"]} 2020-11-20T11:28:43.160 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:43 +0000 (0:00:05.834) 0:02:10.784 ******* 2020-11-20T11:28:44.697 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: dnf-utils", "Installed: git-all", "Installed: sysstat", "Installed: libedit", "Installed: boost-thread", "Installed: xfsprogs", "Installed: gdisk", "Installed: parted", "Installed: libgcrypt", "Installed: fuse-libs", "Installed: openssl", "Installed: libuuid", "Installed: attr", "Installed: ant", "Installed: lsof", "Installed: gettext", "Installed: bc", "Installed: xfsdump", "Installed: blktrace", "Installed: usbredir", "Installed: podman", "Installed: podman-docker", "Installed: libev-devel", "Installed: valgrind", "Installed: nfs-utils", "Installed: ncurses-devel", "Installed: gcc", "Installed: git", "Installed: python3-nose", "Installed: python3-virtualenv", "Installed: genisoimage", "Installed: qemu-img", "Installed: qemu-kvm-core", "Installed: qemu-kvm-block-rbd", "Installed: libacl-devel", "Installed: autoconf", "Installed: gdb", "Installed: iozone", "Installed: lvm2", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:4.2.0-34.module+el8.3.0+7976+077be4ec.x86_64", "Installed: boost-program-options-1.66.0-10.el8.x86_64"]} 2020-11-20T11:28:44.735 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:44 +0000 (0:00:01.574) 0:02:12.358 ******* 2020-11-20T11:28:45.547 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-11-20T11:28:45.548 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: dbench"]} 2020-11-20T11:28:45.582 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:45 +0000 (0:00:00.847) 0:02:13.206 ******* 2020-11-20T11:28:47.314 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: dbench"]} 2020-11-20T11:28:47.350 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:47 +0000 (0:00:01.767) 0:02:14.974 ******* 2020-11-20T11:28:47.990 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-20T11:28:47.991 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2020-11-20T11:28:48.024 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:48 +0000 (0:00:00.674) 0:02:15.648 ******* 2020-11-20T11:28:48.052 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:48.085 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:48 +0000 (0:00:00.060) 0:02:15.709 ******* 2020-11-20T11:28:48.372 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-11-20T11:28:48.373 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "dbus.socket basic.target polkit.service system.slice sysinit.target dbus.service", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "iptables.service ipset.service shutdown.target ip6tables.service nftables.service ebtables.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "D 2020-11-20T11:28:48.374 INFO:teuthology.task.ansible.out:evicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "Limi 2020-11-20T11:28:48.376 INFO:teuthology.task.ansible.out:tNOFILESoft": "1024", "LimitNPROC": "127237", "LimitNPROCSoft": "127237", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127237", "LimitSIGPENDINGSoft": "127237", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice dbus.socket", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "S 2020-11-20T11:28:48.377 INFO:teuthology.task.ansible.out:tartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203580", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2020-11-20T11:28:48.408 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:48 +0000 (0:00:00.322) 0:02:16.031 ******* 2020-11-20T11:28:48.707 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-11-20T11:28:48.709 INFO:teuthology.task.ansible.out:fatal: [smithi155.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2020-11-20T11:28:48.740 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:48 +0000 (0:00:00.332) 0:02:16.364 ******* 2020-11-20T11:28:49.384 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-11-20T11:28:49.386 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2020-11-20T11:28:49.418 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:49 +0000 (0:00:00.677) 0:02:17.042 ******* 2020-11-20T11:28:49.444 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2020-11-20T11:28:49.446 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:49.477 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:49 +0000 (0:00:00.059) 0:02:17.101 ******* 2020-11-20T11:28:49.503 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2020-11-20T11:28:49.505 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:49.537 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:49 +0000 (0:00:00.059) 0:02:17.161 ******* 2020-11-20T11:28:49.563 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2020-11-20T11:28:49.565 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:49.597 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:49 +0000 (0:00:00.059) 0:02:17.220 ******* 2020-11-20T11:28:49.635 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:49.667 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:49 +0000 (0:00:00.070) 0:02:17.291 ******* 2020-11-20T11:28:49.693 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2020-11-20T11:28:49.695 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:49.732 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:49 +0000 (0:00:00.064) 0:02:17.356 ******* 2020-11-20T11:28:49.758 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-11-20T11:28:49.760 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:49.792 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:49 +0000 (0:00:00.060) 0:02:17.416 ******* 2020-11-20T11:28:49.818 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-11-20T11:28:49.852 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:49 +0000 (0:00:00.059) 0:02:17.476 ******* 2020-11-20T11:28:49.878 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:49.912 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:49 +0000 (0:00:00.060) 0:02:17.536 ******* 2020-11-20T11:28:49.939 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:49.972 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:49 +0000 (0:00:00.059) 0:02:17.596 ******* 2020-11-20T11:28:50.000 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-11-20T11:28:50.004 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-11-20T11:28:50.012 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-11-20T11:28:50.045 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:50 +0000 (0:00:00.073) 0:02:17.669 ******* 2020-11-20T11:28:50.051 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-20T11:28:50.053 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2020-11-20T11:28:50.086 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:50 +0000 (0:00:00.041) 0:02:17.710 ******* 2020-11-20T11:28:50.092 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-11-20T11:28:50.094 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:50.128 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:50 +0000 (0:00:00.042) 0:02:17.752 ******* 2020-11-20T11:28:50.134 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-20T11:28:50.136 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:50.169 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:50 +0000 (0:00:00.040) 0:02:17.793 ******* 2020-11-20T11:28:50.177 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-11-20T11:28:50.210 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:50 +0000 (0:00:00.041) 0:02:17.834 ******* 2020-11-20T11:28:50.237 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-11-20T11:28:50.239 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:50.272 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:50 +0000 (0:00:00.061) 0:02:17.896 ******* 2020-11-20T11:28:50.298 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:50.333 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:50 +0000 (0:00:00.061) 0:02:17.957 ******* 2020-11-20T11:28:50.371 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-11-20T11:28:50.373 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:50.405 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:50 +0000 (0:00:00.071) 0:02:18.029 ******* 2020-11-20T11:28:50.434 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:50.465 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:50 +0000 (0:00:00.059) 0:02:18.089 ******* 2020-11-20T11:28:50.482 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-11-20T11:28:50.487 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "dbus.service basic.target sysinit.target dbus.socket polkit.service system.slice", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ebtables.service ipset.service nftables.service iptables.service shutdown.target ip6tables.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "D 2020-11-20T11:28:50.489 INFO:teuthology.task.ansible.out:evicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "Limi 2020-11-20T11:28:50.491 INFO:teuthology.task.ansible.out:tNOFILESoft": "1024", "LimitNPROC": "127237", "LimitNPROCSoft": "127237", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127237", "LimitSIGPENDINGSoft": "127237", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "dbus.socket system.slice sysinit.target", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "S 2020-11-20T11:28:50.493 INFO:teuthology.task.ansible.out:tartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203580", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2020-11-20T11:28:50.534 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:50 +0000 (0:00:00.068) 0:02:18.158 ******* 2020-11-20T11:28:50.542 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-11-20T11:28:50.545 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:50.590 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:50 +0000 (0:00:00.056) 0:02:18.214 ******* 2020-11-20T11:28:50.629 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:50.663 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:50 +0000 (0:00:00.072) 0:02:18.287 ******* 2020-11-20T11:28:50.690 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:50.724 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:50 +0000 (0:00:00.060) 0:02:18.348 ******* 2020-11-20T11:28:50.751 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-11-20T11:28:50.754 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:50.784 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:50 +0000 (0:00:00.060) 0:02:18.408 ******* 2020-11-20T11:28:50.811 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-11-20T11:28:50.813 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:50.847 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:50 +0000 (0:00:00.062) 0:02:18.471 ******* 2020-11-20T11:28:50.853 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-11-20T11:28:50.855 INFO:teuthology.task.ansible.out:fatal: [smithi023.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2020-11-20T11:28:50.889 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:50 +0000 (0:00:00.041) 0:02:18.513 ******* 2020-11-20T11:28:50.895 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-11-20T11:28:50.897 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:50.931 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:50 +0000 (0:00:00.041) 0:02:18.555 ******* 2020-11-20T11:28:50.958 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-11-20T11:28:50.959 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:50.991 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:50 +0000 (0:00:00.060) 0:02:18.615 ******* 2020-11-20T11:28:51.017 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-11-20T11:28:51.019 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:51.050 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:51 +0000 (0:00:00.059) 0:02:18.674 ******* 2020-11-20T11:28:51.077 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-11-20T11:28:51.079 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:51.122 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:51 +0000 (0:00:00.071) 0:02:18.746 ******* 2020-11-20T11:28:51.150 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-11-20T11:28:51.152 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:51.187 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:51 +0000 (0:00:00.064) 0:02:18.811 ******* 2020-11-20T11:28:51.194 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-11-20T11:28:51.195 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2020-11-20T11:28:51.229 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:51 +0000 (0:00:00.041) 0:02:18.853 ******* 2020-11-20T11:28:51.235 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-11-20T11:28:51.237 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:51.271 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:51 +0000 (0:00:00.042) 0:02:18.895 ******* 2020-11-20T11:28:51.277 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2020-11-20T11:28:51.279 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:51.312 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:51 +0000 (0:00:00.040) 0:02:18.936 ******* 2020-11-20T11:28:51.318 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-11-20T11:28:51.320 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:51.354 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:51 +0000 (0:00:00.042) 0:02:18.978 ******* 2020-11-20T11:28:51.360 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2020-11-20T11:28:51.362 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:51.396 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:51 +0000 (0:00:00.041) 0:02:19.020 ******* 2020-11-20T11:28:51.402 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-11-20T11:28:51.404 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:51.442 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:51 +0000 (0:00:00.046) 0:02:19.066 ******* 2020-11-20T11:28:51.448 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2020-11-20T11:28:51.450 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:51.485 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:51 +0000 (0:00:00.042) 0:02:19.109 ******* 2020-11-20T11:28:51.491 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-11-20T11:28:51.493 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:51.527 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:51 +0000 (0:00:00.042) 0:02:19.151 ******* 2020-11-20T11:28:51.533 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2020-11-20T11:28:51.535 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:51.569 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:51 +0000 (0:00:00.041) 0:02:19.193 ******* 2020-11-20T11:28:51.575 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-11-20T11:28:51.577 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:51.610 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:51 +0000 (0:00:00.041) 0:02:19.234 ******* 2020-11-20T11:28:51.616 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2020-11-20T11:28:51.617 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:51.652 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:51 +0000 (0:00:00.042) 0:02:19.276 ******* 2020-11-20T11:28:51.659 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-11-20T11:28:51.660 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:51.694 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:51 +0000 (0:00:00.041) 0:02:19.318 ******* 2020-11-20T11:28:51.700 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:51.738 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:51 +0000 (0:00:00.043) 0:02:19.362 ******* 2020-11-20T11:28:51.745 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-11-20T11:28:51.747 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:51.780 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:51 +0000 (0:00:00.041) 0:02:19.403 ******* 2020-11-20T11:28:51.786 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-11-20T11:28:51.820 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:51 +0000 (0:00:00.040) 0:02:19.444 ******* 2020-11-20T11:28:51.826 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:51.862 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:51 +0000 (0:00:00.041) 0:02:19.485 ******* 2020-11-20T11:28:51.867 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-11-20T11:28:51.869 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:51.902 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:51 +0000 (0:00:00.040) 0:02:19.526 ******* 2020-11-20T11:28:51.909 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T11:28:51.910 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:51.943 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:51 +0000 (0:00:00.039) 0:02:19.566 ******* 2020-11-20T11:28:51.948 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-11-20T11:28:51.950 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:51.983 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:51 +0000 (0:00:00.040) 0:02:19.607 ******* 2020-11-20T11:28:51.989 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-20T11:28:51.991 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:52.026 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:52 +0000 (0:00:00.042) 0:02:19.650 ******* 2020-11-20T11:28:52.033 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* skipping: [smithi023.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-11-20T11:28:52.069 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:52 +0000 (0:00:00.042) 0:02:19.693 ******* 2020-11-20T11:28:52.076 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-20T11:28:52.077 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:52.111 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:52 +0000 (0:00:00.042) 0:02:19.735 ******* 2020-11-20T11:28:52.117 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-11-20T11:28:52.119 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:52.154 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:52 +0000 (0:00:00.042) 0:02:19.777 ******* 2020-11-20T11:28:52.160 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:52.195 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:52 +0000 (0:00:00.041) 0:02:19.818 ******* 2020-11-20T11:28:52.201 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-11-20T11:28:52.240 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:52 +0000 (0:00:00.045) 0:02:19.864 ******* 2020-11-20T11:28:52.254 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:52.286 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:52 +0000 (0:00:00.045) 0:02:19.909 ******* 2020-11-20T11:28:52.291 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-11-20T11:28:52.293 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:52.327 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:52 +0000 (0:00:00.041) 0:02:19.951 ******* 2020-11-20T11:28:52.335 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:52.370 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:52 +0000 (0:00:00.042) 0:02:19.993 ******* 2020-11-20T11:28:52.377 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-11-20T11:28:52.378 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:52.413 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:52 +0000 (0:00:00.043) 0:02:20.037 ******* 2020-11-20T11:28:52.420 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-20T11:28:52.422 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:52.455 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:52 +0000 (0:00:00.041) 0:02:20.079 ******* 2020-11-20T11:28:52.462 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:52.498 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:52 +0000 (0:00:00.042) 0:02:20.122 ******* 2020-11-20T11:28:52.505 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-20T11:28:52.507 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:52.539 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:52 +0000 (0:00:00.041) 0:02:20.163 ******* 2020-11-20T11:28:52.545 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-20T11:28:52.547 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:52.581 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:52 +0000 (0:00:00.042) 0:02:20.205 ******* 2020-11-20T11:28:52.588 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-20T11:28:52.589 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:52.622 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:52 +0000 (0:00:00.041) 0:02:20.246 ******* 2020-11-20T11:28:52.628 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-11-20T11:28:52.630 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:52.668 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:52 +0000 (0:00:00.045) 0:02:20.291 ******* 2020-11-20T11:28:52.675 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-11-20T11:28:52.677 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:52.713 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:52 +0000 (0:00:00.042) 0:02:20.334 ******* 2020-11-20T11:28:52.717 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:52.753 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:52 +0000 (0:00:00.041) 0:02:20.376 ******* 2020-11-20T11:28:52.758 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-11-20T11:28:52.760 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:52.800 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:52 +0000 (0:00:00.047) 0:02:20.423 ******* 2020-11-20T11:28:52.806 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:52.841 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:52 +0000 (0:00:00.041) 0:02:20.465 ******* 2020-11-20T11:28:52.848 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-20T11:28:52.849 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:52.884 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:52 +0000 (0:00:00.041) 0:02:20.507 ******* 2020-11-20T11:28:52.890 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:52.936 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:52 +0000 (0:00:00.053) 0:02:20.560 ******* 2020-11-20T11:28:52.943 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-20T11:28:52.945 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:52.978 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:52 +0000 (0:00:00.041) 0:02:20.602 ******* 2020-11-20T11:28:52.984 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-11-20T11:28:52.987 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:53.019 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:53 +0000 (0:00:00.041) 0:02:20.643 ******* 2020-11-20T11:28:53.026 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-20T11:28:53.028 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:53.061 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:53 +0000 (0:00:00.041) 0:02:20.685 ******* 2020-11-20T11:28:53.075 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:53.109 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:53 +0000 (0:00:00.047) 0:02:20.732 ******* 2020-11-20T11:28:53.115 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-20T11:28:53.117 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:53.150 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:53 +0000 (0:00:00.041) 0:02:20.774 ******* 2020-11-20T11:28:53.157 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:53.191 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:53 +0000 (0:00:00.040) 0:02:20.814 ******* 2020-11-20T11:28:53.197 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:53.235 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:53 +0000 (0:00:00.042) 0:02:20.857 ******* 2020-11-20T11:28:53.241 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-11-20T11:28:53.243 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:53.278 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:53 +0000 (0:00:00.044) 0:02:20.902 ******* 2020-11-20T11:28:53.285 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:53.326 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:53 +0000 (0:00:00.047) 0:02:20.950 ******* 2020-11-20T11:28:53.334 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:53.391 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:53 +0000 (0:00:00.064) 0:02:21.014 ******* 2020-11-20T11:28:53.400 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:53.434 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:53 +0000 (0:00:00.043) 0:02:21.058 ******* 2020-11-20T11:28:53.441 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:53.476 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:53 +0000 (0:00:00.041) 0:02:21.100 ******* 2020-11-20T11:28:53.483 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-11-20T11:28:53.485 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:53.520 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:53 +0000 (0:00:00.043) 0:02:21.143 ******* 2020-11-20T11:28:53.525 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-11-20T11:28:53.527 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:53.562 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:53 +0000 (0:00:00.042) 0:02:21.186 ******* 2020-11-20T11:28:53.572 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:53.617 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:53 +0000 (0:00:00.054) 0:02:21.241 ******* 2020-11-20T11:28:53.623 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-11-20T11:28:53.625 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:53.658 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:53 +0000 (0:00:00.040) 0:02:21.282 ******* 2020-11-20T11:28:53.664 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-11-20T11:28:53.666 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:53.699 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:53 +0000 (0:00:00.041) 0:02:21.323 ******* 2020-11-20T11:28:53.705 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-11-20T11:28:53.706 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:53.741 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:53 +0000 (0:00:00.041) 0:02:21.364 ******* 2020-11-20T11:28:53.747 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-11-20T11:28:53.749 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:53.783 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:53 +0000 (0:00:00.042) 0:02:21.407 ******* 2020-11-20T11:28:53.789 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-11-20T11:28:53.792 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:53.832 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:53 +0000 (0:00:00.049) 0:02:21.456 ******* 2020-11-20T11:28:53.842 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:53.875 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:53 +0000 (0:00:00.042) 0:02:21.499 ******* 2020-11-20T11:28:53.882 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-11-20T11:28:53.884 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:53.919 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:53 +0000 (0:00:00.044) 0:02:21.543 ******* 2020-11-20T11:28:53.926 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-11-20T11:28:53.928 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:53.965 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:53 +0000 (0:00:00.045) 0:02:21.589 ******* 2020-11-20T11:28:53.971 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-11-20T11:28:53.973 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:54.006 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:54 +0000 (0:00:00.041) 0:02:21.630 ******* 2020-11-20T11:28:54.013 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-11-20T11:28:54.015 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:54.051 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:54 +0000 (0:00:00.044) 0:02:21.674 ******* 2020-11-20T11:28:54.058 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:54.092 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:54 +0000 (0:00:00.041) 0:02:21.716 ******* 2020-11-20T11:28:54.098 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-11-20T11:28:54.100 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:54.133 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:54 +0000 (0:00:00.040) 0:02:21.757 ******* 2020-11-20T11:28:54.139 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-11-20T11:28:54.141 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:54.174 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:54 +0000 (0:00:00.041) 0:02:21.798 ******* 2020-11-20T11:28:54.180 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-11-20T11:28:54.182 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:54.214 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:54 +0000 (0:00:00.039) 0:02:21.838 ******* 2020-11-20T11:28:54.219 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T11:28:54.221 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:54.253 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:54 +0000 (0:00:00.039) 0:02:21.877 ******* 2020-11-20T11:28:54.260 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-11-20T11:28:54.262 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:54.295 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:54 +0000 (0:00:00.041) 0:02:21.919 ******* 2020-11-20T11:28:54.301 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-11-20T11:28:54.303 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:54.338 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:54 +0000 (0:00:00.043) 0:02:21.962 ******* 2020-11-20T11:28:54.346 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-11-20T11:28:54.348 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:54.384 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:54 +0000 (0:00:00.045) 0:02:22.007 ******* 2020-11-20T11:28:54.390 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-11-20T11:28:54.392 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:54.431 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:54 +0000 (0:00:00.047) 0:02:22.055 ******* 2020-11-20T11:28:54.437 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-11-20T11:28:54.439 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:54.475 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:54 +0000 (0:00:00.043) 0:02:22.098 ******* 2020-11-20T11:28:54.481 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-11-20T11:28:54.483 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:54.519 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:54 +0000 (0:00:00.044) 0:02:22.142 ******* 2020-11-20T11:28:54.526 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T11:28:54.527 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-11-20T11:28:54.564 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:54 +0000 (0:00:00.045) 0:02:22.188 ******* 2020-11-20T11:28:54.570 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-11-20T11:28:54.572 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:54.613 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:54 +0000 (0:00:00.048) 0:02:22.237 ******* 2020-11-20T11:28:54.620 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-11-20T11:28:54.622 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-11-20T11:28:54.632 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-11-20T11:28:54.634 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:54.670 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:54 +0000 (0:00:00.057) 0:02:22.294 ******* 2020-11-20T11:28:54.677 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-11-20T11:28:54.680 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-11-20T11:28:54.728 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:54 +0000 (0:00:00.057) 0:02:22.352 ******* 2020-11-20T11:28:54.735 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-20T11:28:54.738 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:54.772 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:54 +0000 (0:00:00.043) 0:02:22.396 ******* 2020-11-20T11:28:54.817 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:54.854 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:54 +0000 (0:00:00.081) 0:02:22.478 ******* 2020-11-20T11:28:54.881 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-20T11:28:54.883 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:54.921 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:54 +0000 (0:00:00.065) 0:02:22.543 ******* 2020-11-20T11:28:54.938 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-11-20T11:28:54.940 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003373", "end": "2020-11-20 11:28:54.911033", "rc": 0, "start": "2020-11-20 11:28:54.907660", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-11-20T11:28:54.977 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:54 +0000 (0:00:00.057) 0:02:22.601 ******* 2020-11-20T11:28:54.983 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-11-20T11:28:54.985 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:55.019 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:55 +0000 (0:00:00.042) 0:02:22.643 ******* 2020-11-20T11:28:55.025 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-11-20T11:28:55.027 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:55.066 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:55 +0000 (0:00:00.046) 0:02:22.690 ******* 2020-11-20T11:28:55.072 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-11-20T11:28:55.074 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:55.111 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:55 +0000 (0:00:00.044) 0:02:22.734 ******* 2020-11-20T11:28:55.117 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-11-20T11:28:55.119 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:55.155 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:55 +0000 (0:00:00.044) 0:02:22.779 ******* 2020-11-20T11:28:55.162 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-20T11:28:55.163 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:55.198 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:55 +0000 (0:00:00.043) 0:02:22.822 ******* 2020-11-20T11:28:55.205 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-11-20T11:28:55.206 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:55.239 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:55 +0000 (0:00:00.041) 0:02:22.863 ******* 2020-11-20T11:28:55.246 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-20T11:28:55.248 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:55.282 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:55 +0000 (0:00:00.042) 0:02:22.906 ******* 2020-11-20T11:28:55.288 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-11-20T11:28:55.290 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:55.324 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:55 +0000 (0:00:00.041) 0:02:22.948 ******* 2020-11-20T11:28:55.330 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-20T11:28:55.332 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:55.367 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:55 +0000 (0:00:00.042) 0:02:22.991 ******* 2020-11-20T11:28:55.396 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-11-20T11:28:55.397 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:55.429 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:55 +0000 (0:00:00.062) 0:02:23.053 ******* 2020-11-20T11:28:55.456 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-11-20T11:28:55.457 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:55.490 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:55 +0000 (0:00:00.060) 0:02:23.114 ******* 2020-11-20T11:28:55.516 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-11-20T11:28:55.518 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:55.550 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:55 +0000 (0:00:00.060) 0:02:23.174 ******* 2020-11-20T11:28:55.577 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-11-20T11:28:55.579 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:55.616 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:55 +0000 (0:00:00.065) 0:02:23.240 ******* 2020-11-20T11:28:55.643 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:55.676 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:55 +0000 (0:00:00.059) 0:02:23.300 ******* 2020-11-20T11:28:55.703 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-11-20T11:28:55.705 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:55.736 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:55 +0000 (0:00:00.060) 0:02:23.360 ******* 2020-11-20T11:28:55.763 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-11-20T11:28:55.765 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:55.797 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:55 +0000 (0:00:00.060) 0:02:23.421 ******* 2020-11-20T11:28:55.823 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-11-20T11:28:55.825 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:55.857 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:55 +0000 (0:00:00.060) 0:02:23.481 ******* 2020-11-20T11:28:55.884 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-11-20T11:28:55.886 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:55.934 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:55 +0000 (0:00:00.076) 0:02:23.558 ******* 2020-11-20T11:28:55.960 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-11-20T11:28:55.962 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:55.997 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:55 +0000 (0:00:00.063) 0:02:23.621 ******* 2020-11-20T11:28:56.026 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-11-20T11:28:56.028 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:56.065 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:56 +0000 (0:00:00.067) 0:02:23.689 ******* 2020-11-20T11:28:56.092 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-11-20T11:28:56.094 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:56.144 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:56 +0000 (0:00:00.078) 0:02:23.768 ******* 2020-11-20T11:28:56.174 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-11-20T11:28:56.176 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:56.213 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:56 +0000 (0:00:00.069) 0:02:23.837 ******* 2020-11-20T11:28:56.240 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-11-20T11:28:56.241 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:56.274 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:56 +0000 (0:00:00.060) 0:02:23.897 ******* 2020-11-20T11:28:56.300 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-11-20T11:28:56.302 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:56.334 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:56 +0000 (0:00:00.060) 0:02:23.958 ******* 2020-11-20T11:28:56.360 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-11-20T11:28:56.362 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:56.394 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:56 +0000 (0:00:00.060) 0:02:24.018 ******* 2020-11-20T11:28:56.421 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-11-20T11:28:56.455 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:56 +0000 (0:00:00.060) 0:02:24.079 ******* 2020-11-20T11:28:56.503 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi023.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-11-20T11:28:56.514 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-11-20T11:28:56.525 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-11-20T11:28:56.527 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-11-20T11:28:56.548 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-11-20T11:28:56.549 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-11-20T11:28:56.589 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:56 +0000 (0:00:00.134) 0:02:24.213 ******* 2020-11-20T11:28:56.774 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-11-20T11:28:56.776 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.007411", "end": "2020-11-20 11:28:56.752661", "rc": 0, "start": "2020-11-20 11:28:56.745250", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-11-20T11:28:56.812 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:56 +0000 (0:00:00.223) 0:02:24.436 ******* 2020-11-20T11:28:56.840 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-11-20T11:28:56.841 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:56.877 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:56 +0000 (0:00:00.064) 0:02:24.501 ******* 2020-11-20T11:28:56.904 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:56.937 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:56 +0000 (0:00:00.060) 0:02:24.561 ******* 2020-11-20T11:28:56.965 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-11-20T11:28:56.967 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:56.998 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:56 +0000 (0:00:00.060) 0:02:24.622 ******* 2020-11-20T11:28:57.024 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-11-20T11:28:57.026 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:57.058 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:57 +0000 (0:00:00.060) 0:02:24.682 ******* 2020-11-20T11:28:57.822 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: gdisk", "Installed: device-mapper"]} 2020-11-20T11:28:57.859 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:57 +0000 (0:00:00.800) 0:02:25.483 ******* 2020-11-20T11:28:57.897 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ ok: [smithi155.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930420969472, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227153557, 'block_used': 12941551, 'inode_total': 61054976, 'inode_available': 60986954, 'inode_used': 68022, 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227153557, "block_size": 4096, "block_total": 240095108, "block_used": 12941551, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986954, "inode_total": 61054976, "inode_used": 68022, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930420969472, "size_total": 983429562368, "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}} 2020-11-20T11:28:57.932 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:57 +0000 (0:00:00.073) 0:02:25.556 ******* 2020-11-20T11:28:58.126 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005001", "end": "2020-11-20 11:28:58.097372", "rc": 0, "start": "2020-11-20 11:28:58.092371", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T11:28:58.161 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:58 +0000 (0:00:00.228) 0:02:25.784 ******* 2020-11-20T11:28:58.199 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-11-20T11:28:58.201 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930420969472, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227153557, 'block_used': 12941551, 'inode_total': 61054976, 'inode_available': 60986954, 'inode_used': 68022, 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227153557, "block_size": 4096, "block_total": 240095108, "block_used": 12941551, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986954, "inode_total": 61054976, "inode_used": 68022, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930420969472, "size_total": 983429562368, "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:28:58.235 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:58 +0000 (0:00:00.074) 0:02:25.858 ******* 2020-11-20T11:28:58.262 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-11-20T11:28:58.303 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:58 +0000 (0:00:00.068) 0:02:25.927 ******* 2020-11-20T11:28:59.700 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: gdisk", "Installed: device-mapper"]} 2020-11-20T11:28:59.734 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:59 +0000 (0:00:01.430) 0:02:27.358 ******* 2020-11-20T11:28:59.772 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ ok: [smithi023.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930420920320, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227153545, 'block_used': 12941563, 'inode_total': 61054976, 'inode_available': 60986950, 'inode_used': 68026, 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227153545, "block_size": 4096, "block_total": 240095108, "block_used": 12941563, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986950, "inode_total": 61054976, "inode_used": 68026, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930420920320, "size_total": 983429562368, "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}} 2020-11-20T11:28:59.813 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:28:59 +0000 (0:00:00.078) 0:02:27.437 ******* 2020-11-20T11:28:59.893 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-11-20T11:28:59.894 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-H7golehaj2nqPe16Zr3bjNczCESbsm3kWLXPKLEwjjVDnW0zwc0XuETafGzvzDKC'], 'uuids': ['a59d5cd7-5abb-46d0-8a8b-638640c7cfd2'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.364716", "end": "2020-11-20 11:28:59.849984", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-H7golehaj2nqPe16Zr3bjNczCESbsm3kWLXPKLEwjjVDnW0zwc0XuETafGzvzDKC"], "labels": [], "masters": [], "uuids": ["a59d5cd7-5abb-46d0-8a8b-638640c7cfd2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 11:28:58.485268", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T11:29:00.025 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-11-20T11:29:00.027 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005333", "end": "2020-11-20 11:28:59.996340", "rc": 0, "start": "2020-11-20 11:28:59.991007", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T11:29:00.059 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:00 +0000 (0:00:00.246) 0:02:27.683 ******* 2020-11-20T11:29:00.097 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-11-20T11:29:00.099 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930420920320, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227153545, 'block_used': 12941563, 'inode_total': 61054976, 'inode_available': 60986950, 'inode_used': 68026, 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227153545, "block_size": 4096, "block_total": 240095108, "block_used": 12941563, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986950, "inode_total": 61054976, "inode_used": 68026, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930420920320, "size_total": 983429562368, "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:00.132 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:00 +0000 (0:00:00.072) 0:02:27.756 ******* 2020-11-20T11:29:00.158 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-11-20T11:29:00.193 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:00 +0000 (0:00:00.060) 0:02:27.816 ******* 2020-11-20T11:29:01.175 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-11-20T11:29:01.176 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-xBsVeX-yTfQ-x5gV-9JI3-ct5Q-1QYv-yIYYN6', 'nvme-INTEL_SSDPEDMD400G4_CVFT623300GW400BGN', 'nvme-nvme.8086-43564654363233333030475734303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.086837", "end": "2020-11-20 11:29:01.138020", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-xBsVeX-yTfQ-x5gV-9JI3-ct5Q-1QYv-yIYYN6", "nvme-INTEL_SSDPEDMD400G4_CVFT623300GW400BGN", "nvme-nvme.8086-43564654363233333030475734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 11:29:00.051183", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2020-11-20T11:29:01.178 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T11:29:01.700 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-Jd2RkblnZhSJSPm3p8w3pfxAYoS34lQA22LUlSNxz9Blz9s6UCs45oY1vkXSnd05'], 'uuids': ['24064d49-4469-41e3-aa19-75a5e56dee12'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.285570", "end": "2020-11-20 11:29:01.661951", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-Jd2RkblnZhSJSPm3p8w3pfxAYoS34lQA22LUlSNxz9Blz9s6UCs45oY1vkXSnd05"], "labels": [], "masters": [], "uuids": ["24064d49-4469-41e3-aa19-75a5e56dee12"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 11:29:00.376381", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T11:29:02.441 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-H7golehaj2nqPe16Zr3bjNczCESbsm3k2Hxrm6iNwtM5hAMK2cTETuQBHx1O0oTz'], 'uuids': ['557246cd-1c08-4e99-9ec1-5bf1be4fcd61'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.067176", "end": "2020-11-20 11:29:02.407892", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-H7golehaj2nqPe16Zr3bjNczCESbsm3k2Hxrm6iNwtM5hAMK2cTETuQBHx1O0oTz"], "labels": [], "masters": [], "uuids": ["557246cd-1c08-4e99-9ec1-5bf1be4fcd61"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 11:29:01.340716", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T11:29:02.949 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-0m0sf9-dpDr-OBxr-bP5T-8FG2-8pCQ-YnuiaW', 'nvme-INTEL_SSDPEDKE040T7_BTLE736103B14P0KGN', 'nvme-eui.0100000001000000e4d25c000034df14'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDKE040T7', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '7814037168', 'sectorsize': '512', 'size': '3.64 TB', 'host': 'Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller]', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.056248", "end": "2020-11-20 11:29:02.912134", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller]", "links": {"ids": ["lvm-pv-uuid-0m0sf9-dpDr-OBxr-bP5T-8FG2-8pCQ-YnuiaW", "nvme-INTEL_SSDPEDKE040T7_BTLE736103B14P0KGN", "nvme-eui.0100000001000000e4d25c000034df14"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDKE040T7", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "7814037168", "sectorsize": "512", "size": "3.64 TB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 11:29:01.855886", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_li 2020-11-20T11:29:02.951 INFO:teuthology.task.ansible.out:nes": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T11:29:03.713 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-H7golehaj2nqPe16Zr3bjNczCESbsm3kPfNHnuacSgYeKeJml30iLAZiAFh43PFX'], 'uuids': ['ea067728-18ce-4ae0-80e0-a00cb86c4437'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.089238", "end": "2020-11-20 11:29:03.683811", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-H7golehaj2nqPe16Zr3bjNczCESbsm3kPfNHnuacSgYeKeJml30iLAZiAFh43PFX"], "labels": [], "masters": [], "uuids": ["ea067728-18ce-4ae0-80e0-a00cb86c4437"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 11:29:02.594573", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T11:29:04.757 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-Jd2RkblnZhSJSPm3p8w3pfxAYoS34lQAQW08rs4Hd5ooKcICD9Y4815lt8XP4jUf'], 'uuids': ['39f2fab2-83f5-486f-9106-5859a2148f2c'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.071542", "end": "2020-11-20 11:29:04.193752", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-Jd2RkblnZhSJSPm3p8w3pfxAYoS34lQAQW08rs4Hd5ooKcICD9Y4815lt8XP4jUf"], "labels": [], "masters": [], "uuids": ["39f2fab2-83f5-486f-9106-5859a2148f2c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 11:29:03.122210", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T11:29:04.973 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-H7golehaj2nqPe16Zr3bjNczCESbsm3kDvbsZiZAaMFHAjJEqJETtJieF52eRwlY'], 'uuids': ['1652ba81-8c12-4596-946f-2c738770c0e5'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.061448", "end": "2020-11-20 11:29:04.942415", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-H7golehaj2nqPe16Zr3bjNczCESbsm3kDvbsZiZAaMFHAjJEqJETtJieF52eRwlY"], "labels": [], "masters": [], "uuids": ["1652ba81-8c12-4596-946f-2c738770c0e5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 11:29:03.880967", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T11:29:04.984 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XNRW', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XNRW', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XNRW', 'scsi-35000c5009294102d', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XNRW', 'wwn-0x5000c5009294102d'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XNRW', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c5009294102d', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XNRW-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XNRW-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XNRW-part1', 'scsi-35000c5009294102d-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XNRW-part1', 'wwn-0x5000c5009294102d-part1'], 'uuids': ['782d0384-b480-44b3-9c95-fe95daf886ae'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNRW", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XNRW", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XNRW", "scsi-35000c5009294102d", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XNRW", "wwn-0x5000c5009294102d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNRW-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XNRW-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2020-11-20T11:29:04.986 INFO:teuthology.task.ansible.out:Z1W5XNRW-part1", "scsi-35000c5009294102d-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XNRW-part1", "wwn-0x5000c5009294102d-part1"], "labels": [], "masters": [], "uuids": ["782d0384-b480-44b3-9c95-fe95daf886ae"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XNRW", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009294102d"}}, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:05.469 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-Jd2RkblnZhSJSPm3p8w3pfxAYoS34lQANKsM0yqhMiuqD6hz3ZF0ZgVSYj9kLzHu'], 'uuids': ['b9215572-8a5c-4e53-9868-afced4c3d9ac'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.051126", "end": "2020-11-20 11:29:05.442973", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-Jd2RkblnZhSJSPm3p8w3pfxAYoS34lQANKsM0yqhMiuqD6hz3ZF0ZgVSYj9kLzHu"], "labels": [], "masters": [], "uuids": ["b9215572-8a5c-4e53-9868-afced4c3d9ac"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 11:29:04.391847", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T11:29:06.242 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-H7golehaj2nqPe16Zr3bjNczCESbsm3kRAB3ksbzaC7heeX4YGEVbGOvWAY61K8G'], 'uuids': ['fc765813-56af-4950-96e3-4e62ce325cfc'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.073758", "end": "2020-11-20 11:29:06.209187", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-H7golehaj2nqPe16Zr3bjNczCESbsm3kRAB3ksbzaC7heeX4YGEVbGOvWAY61K8G"], "labels": [], "masters": [], "uuids": ["fc765813-56af-4950-96e3-4e62ce325cfc"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 11:29:05.135429", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T11:29:06.284 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:06 +0000 (0:00:06.090) 0:02:33.907 ******* 2020-11-20T11:29:06.645 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-11-20T11:29:06.646 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-H7golehaj2nqPe16Zr3bjNczCESbsm3kWLXPKLEwjjVDnW0zwc0XuETafGzvzDKC'], 'uuids': ['a59d5cd7-5abb-46d0-8a8b-638640c7cfd2'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.141184", "end": "2020-11-20 11:29:06.614782", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-H7golehaj2nqPe16Zr3bjNczCESbsm3kWLXPKLEwjjVDnW0zwc0XuETafGzvzDKC"], "labels": [], "masters": [], "uuids": ["a59d5cd7-5abb-46d0-8a8b-638640c7cfd2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 11:29:06.473598", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138494 s, 833 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138494 s, 833 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T11:29:06.730 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-11-20T11:29:06.732 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-Jd2RkblnZhSJSPm3p8w3pfxAYoS34lQA6s8xHrAISivDXfsECiJwBiaU9kQNIWcU'], 'uuids': ['3b20ca5c-dd25-4461-86bd-5cf5f0b1ddab'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.064426", "end": "2020-11-20 11:29:06.702750", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-Jd2RkblnZhSJSPm3p8w3pfxAYoS34lQA6s8xHrAISivDXfsECiJwBiaU9kQNIWcU"], "labels": [], "masters": [], "uuids": ["3b20ca5c-dd25-4461-86bd-5cf5f0b1ddab"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 11:29:05.638324", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T11:29:06.743 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HPS3', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPS3', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HPS3', 'scsi-35000c500871a974c', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HPS3', 'wwn-0x5000c500871a974c'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W4HPS3', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500871a974c', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HPS3-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPS3-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HPS3-part1', 'scsi-35000c500871a974c-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HPS3-part1', 'wwn-0x5000c500871a974c-part1'], 'uuids': ['782d0384-b480-44b3-9c95-fe95daf886ae'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPS3", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPS3", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HPS3", "scsi-35000c500871a974c", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HPS3", "wwn-0x5000c500871a974c"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPS3-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPS3-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2020-11-20T11:29:06.745 INFO:teuthology.task.ansible.out:Z1W4HPS3-part1", "scsi-35000c500871a974c-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HPS3-part1", "wwn-0x5000c500871a974c-part1"], "labels": [], "masters": [], "uuids": ["782d0384-b480-44b3-9c95-fe95daf886ae"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HPS3", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a974c"}}, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:06.850 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-11-20T11:29:06.851 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-xBsVeX-yTfQ-x5gV-9JI3-ct5Q-1QYv-yIYYN6', 'nvme-INTEL_SSDPEDMD400G4_CVFT623300GW400BGN', 'nvme-nvme.8086-43564654363233333030475734303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.037903", "end": "2020-11-20 11:29:06.823364", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-xBsVeX-yTfQ-x5gV-9JI3-ct5Q-1QYv-yIYYN6", "nvme-INTEL_SSDPEDMD400G4_CVFT623300GW400BGN", "nvme-nvme.8086-43564654363233333030475734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 11:29:06.785461", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0352437 s, 3.3 GB/s", "std 2020-11-20T11:29:06.853 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0352437 s, 3.3 GB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T11:29:07.186 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-H7golehaj2nqPe16Zr3bjNczCESbsm3k2Hxrm6iNwtM5hAMK2cTETuQBHx1O0oTz'], 'uuids': ['557246cd-1c08-4e99-9ec1-5bf1be4fcd61'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.153035", "end": "2020-11-20 11:29:07.156650", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-H7golehaj2nqPe16Zr3bjNczCESbsm3k2Hxrm6iNwtM5hAMK2cTETuQBHx1O0oTz"], "labels": [], "masters": [], "uuids": ["557246cd-1c08-4e99-9ec1-5bf1be4fcd61"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 11:29:07.003615", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.150408 s, 767 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.150408 s, 767 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T11:29:07.491 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-H7golehaj2nqPe16Zr3bjNczCESbsm3kPfNHnuacSgYeKeJml30iLAZiAFh43PFX'], 'uuids': ['ea067728-18ce-4ae0-80e0-a00cb86c4437'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.139767", "end": "2020-11-20 11:29:07.466363", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-H7golehaj2nqPe16Zr3bjNczCESbsm3kPfNHnuacSgYeKeJml30iLAZiAFh43PFX"], "labels": [], "masters": [], "uuids": ["ea067728-18ce-4ae0-80e0-a00cb86c4437"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 11:29:07.326596", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.13724 s, 840 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.13724 s, 840 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T11:29:07.797 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-H7golehaj2nqPe16Zr3bjNczCESbsm3kDvbsZiZAaMFHAjJEqJETtJieF52eRwlY'], 'uuids': ['1652ba81-8c12-4596-946f-2c738770c0e5'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.141297", "end": "2020-11-20 11:29:07.773809", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-H7golehaj2nqPe16Zr3bjNczCESbsm3kDvbsZiZAaMFHAjJEqJETtJieF52eRwlY"], "labels": [], "masters": [], "uuids": ["1652ba81-8c12-4596-946f-2c738770c0e5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 11:29:07.632512", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138715 s, 832 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138715 s, 832 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T11:29:07.809 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XNRW', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XNRW', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XNRW', 'scsi-35000c5009294102d', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XNRW', 'wwn-0x5000c5009294102d'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XNRW', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c5009294102d', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XNRW-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XNRW-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XNRW-part1', 'scsi-35000c5009294102d-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XNRW-part1', 'wwn-0x5000c5009294102d-part1'], 'uuids': ['782d0384-b480-44b3-9c95-fe95daf886ae'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNRW", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XNRW", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XNRW", "scsi-35000c5009294102d", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XNRW", "wwn-0x5000c5009294102d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNRW-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XNRW-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2020-11-20T11:29:07.810 INFO:teuthology.task.ansible.out:Z1W5XNRW-part1", "scsi-35000c5009294102d-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XNRW-part1", "wwn-0x5000c5009294102d-part1"], "labels": [], "masters": [], "uuids": ["782d0384-b480-44b3-9c95-fe95daf886ae"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XNRW", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009294102d"}}, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:08.007 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-11-20T11:29:08.009 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-Jd2RkblnZhSJSPm3p8w3pfxAYoS34lQAS4tlLJkXQNs6yGNneiEjTqgpxJBzFgut'], 'uuids': ['0ebfad88-c181-4044-87f8-d5f51cf675e5'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.068341", "end": "2020-11-20 11:29:07.972154", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-Jd2RkblnZhSJSPm3p8w3pfxAYoS34lQAS4tlLJkXQNs6yGNneiEjTqgpxJBzFgut"], "labels": [], "masters": [], "uuids": ["0ebfad88-c181-4044-87f8-d5f51cf675e5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 11:29:06.903813", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-11-20T11:29:08.048 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:08 +0000 (0:00:01.764) 0:02:35.672 ******* 2020-11-20T11:29:08.117 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-11-20T11:29:08.119 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-H7golehaj2nqPe16Zr3bjNczCESbsm3kRAB3ksbzaC7heeX4YGEVbGOvWAY61K8G'], 'uuids': ['fc765813-56af-4950-96e3-4e62ce325cfc'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.138784", "end": "2020-11-20 11:29:08.089856", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-H7golehaj2nqPe16Zr3bjNczCESbsm3kRAB3ksbzaC7heeX4YGEVbGOvWAY61K8G"], "labels": [], "masters": [], "uuids": ["fc765813-56af-4950-96e3-4e62ce325cfc"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 11:29:07.951072", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.136247 s, 847 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.136247 s, 847 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T11:29:08.159 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:08 +0000 (0:00:00.111) 0:02:35.783 ******* 2020-11-20T11:29:08.394 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-11-20T11:29:08.396 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.056484", "end": "2020-11-20 11:29:08.368768", "rc": 0, "start": "2020-11-20 11:29:08.312284", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T11:29:08.429 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:08 +0000 (0:00:00.269) 0:02:36.053 ******* 2020-11-20T11:29:08.435 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-11-20T11:29:08.437 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-Jd2RkblnZhSJSPm3p8w3pfxAYoS34lQA22LUlSNxz9Blz9s6UCs45oY1vkXSnd05'], 'uuids': ['24064d49-4469-41e3-aa19-75a5e56dee12'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.162154", "end": "2020-11-20 11:29:08.403918", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-Jd2RkblnZhSJSPm3p8w3pfxAYoS34lQA22LUlSNxz9Blz9s6UCs45oY1vkXSnd05"], "labels": [], "masters": [], "uuids": ["24064d49-4469-41e3-aa19-75a5e56dee12"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 11:29:08.241764", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.159413 s, 724 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.159413 s, 724 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T11:29:08.651 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-11-20T11:29:08.653 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.050222", "end": "2020-11-20 11:29:08.631084", "rc": 0, "start": "2020-11-20 11:29:08.580862", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-0m0sf9-dpDr-OBxr-bP5T-8FG2-8pCQ-YnuiaW', 'nvme-INTEL_SSDPEDKE040T7_BTLE736103B14P0KGN', 'nvme-eui.0100000001000000e4d25c000034df14'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDKE040T7', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '7814037168', 'sectorsize': '512', 'size': '3.64 TB', 'host': 'Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller]', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.036373", "end": "2020-11-20 11:29:08.621808", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller]", "links": {"ids": ["lvm-pv-uuid-0m0sf9-dpDr-OBxr-bP5T-8FG2-8pCQ-YnuiaW", "nvme-INTEL_SSDPEDKE040T7_BTLE736103B14P0KGN", "nvme-eui.0100000001000000e4d25c000034df14"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDKE040T7", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "sche 2020-11-20T11:29:08.655 INFO:teuthology.task.ansible.out:duler_mode": "none", "sectors": "7814037168", "sectorsize": "512", "size": "3.64 TB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 11:29:08.585435", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0337733 s, 3.4 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0337733 s, 3.4 GB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T11:29:08.687 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:08 +0000 (0:00:00.257) 0:02:36.310 ******* 2020-11-20T11:29:08.714 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-11-20T11:29:08.752 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:08 +0000 (0:00:00.064) 0:02:36.375 ******* 2020-11-20T11:29:08.778 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-11-20T11:29:08.781 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:08.827 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:08 +0000 (0:00:00.074) 0:02:36.450 ******* 2020-11-20T11:29:08.855 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:08.889 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:08 +0000 (0:00:00.062) 0:02:36.513 ******* 2020-11-20T11:29:08.916 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-11-20T11:29:08.918 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:08.952 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:08 +0000 (0:00:00.063) 0:02:36.576 ******* 2020-11-20T11:29:08.979 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-11-20T11:29:08.981 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:09.014 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:09 +0000 (0:00:00.061) 0:02:36.637 ******* 2020-11-20T11:29:09.019 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-11-20T11:29:09.020 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-Jd2RkblnZhSJSPm3p8w3pfxAYoS34lQAQW08rs4Hd5ooKcICD9Y4815lt8XP4jUf'], 'uuids': ['39f2fab2-83f5-486f-9106-5859a2148f2c'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.159499", "end": "2020-11-20 11:29:08.969156", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-Jd2RkblnZhSJSPm3p8w3pfxAYoS34lQAQW08rs4Hd5ooKcICD9Y4815lt8XP4jUf"], "labels": [], "masters": [], "uuids": ["39f2fab2-83f5-486f-9106-5859a2148f2c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 11:29:08.809657", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.156833 s, 735 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.156833 s, 735 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T11:29:09.188 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-11-20T11:29:09.189 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-20T11:29:09.222 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:09 +0000 (0:00:00.208) 0:02:36.846 ******* 2020-11-20T11:29:09.259 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-11-20T11:29:09.261 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930420969472, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227153557, 'block_used': 12941551, 'inode_total': 61054976, 'inode_available': 60986954, 'inode_used': 68022, 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227153557, "block_size": 4096, "block_total": 240095108, "block_used": 12941551, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986954, "inode_total": 61054976, "inode_used": 68022, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930420969472, "size_total": 983429562368, "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:09.294 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:09 +0000 (0:00:00.072) 0:02:36.918 ******* 2020-11-20T11:29:09.312 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-Jd2RkblnZhSJSPm3p8w3pfxAYoS34lQANKsM0yqhMiuqD6hz3ZF0ZgVSYj9kLzHu'], 'uuids': ['b9215572-8a5c-4e53-9868-afced4c3d9ac'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.132719", "end": "2020-11-20 11:29:09.275011", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-Jd2RkblnZhSJSPm3p8w3pfxAYoS34lQANKsM0yqhMiuqD6hz3ZF0ZgVSYj9kLzHu"], "labels": [], "masters": [], "uuids": ["b9215572-8a5c-4e53-9868-afced4c3d9ac"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 11:29:09.142292", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.130033 s, 887 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.130033 s, 887 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T11:29:09.323 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:09.357 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:09 +0000 (0:00:00.062) 0:02:36.981 ******* 2020-11-20T11:29:09.384 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:09.419 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:09 +0000 (0:00:00.062) 0:02:37.043 ******* 2020-11-20T11:29:09.446 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T11:29:09.448 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:09.481 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:09 +0000 (0:00:00.061) 0:02:37.105 ******* 2020-11-20T11:29:09.508 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-11-20T11:29:09.510 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:09.543 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:09 +0000 (0:00:00.062) 0:02:37.167 ******* 2020-11-20T11:29:09.570 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-11-20T11:29:09.572 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:09.605 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:09 +0000 (0:00:00.061) 0:02:37.229 ******* 2020-11-20T11:29:09.622 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-11-20T11:29:09.623 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-Jd2RkblnZhSJSPm3p8w3pfxAYoS34lQA6s8xHrAISivDXfsECiJwBiaU9kQNIWcU'], 'uuids': ['3b20ca5c-dd25-4461-86bd-5cf5f0b1ddab'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.130798", "end": "2020-11-20 11:29:09.579301", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-Jd2RkblnZhSJSPm3p8w3pfxAYoS34lQA6s8xHrAISivDXfsECiJwBiaU9kQNIWcU"], "labels": [], "masters": [], "uuids": ["3b20ca5c-dd25-4461-86bd-5cf5f0b1ddab"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 11:29:09.448503", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.128194 s, 900 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.128194 s, 900 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T11:29:09.624 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HPS3', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPS3', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HPS3', 'scsi-35000c500871a974c', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HPS3', 'wwn-0x5000c500871a974c'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_ 2020-11-20T11:29:09.626 INFO:teuthology.task.ansible.out:address': None, 'sas_device_handle': None, 'serial': 'Z1W4HPS3', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500871a974c', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HPS3-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPS3-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HPS3-part1', 'scsi-35000c500871a974c-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HPS3-part1', 'wwn-0x5000c500871a974c-part1'], 'uuids': ['782d0384-b480-44b3-9c95-fe95daf886ae'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPS3", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPS3", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HPS3", "scsi-35000c500871a974c", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HPS3", "wwn-0x5000c500871a974c"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HPS3-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HPS3-part1", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HPS3-part1", "scsi-35000c500871a974c-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HPS3-part1", "wwn-0x5000c500871a974c-part1"], "labels": [], "masters": [], "uuids": ["782d0384-b480-44b3-9c95-fe95daf886ae"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_ha 2020-11-20T11:29:09.627 INFO:teuthology.task.ansible.out:ndle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HPS3", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a974c"}}, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:09.635 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:09.672 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:09 +0000 (0:00:00.067) 0:02:37.296 ******* 2020-11-20T11:29:09.917 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-11-20T11:29:09.919 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-Jd2RkblnZhSJSPm3p8w3pfxAYoS34lQAS4tlLJkXQNs6yGNneiEjTqgpxJBzFgut'], 'uuids': ['0ebfad88-c181-4044-87f8-d5f51cf675e5'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.123657", "end": "2020-11-20 11:29:09.887804", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-Jd2RkblnZhSJSPm3p8w3pfxAYoS34lQAS4tlLJkXQNs6yGNneiEjTqgpxJBzFgut"], "labels": [], "masters": [], "uuids": ["0ebfad88-c181-4044-87f8-d5f51cf675e5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-11-20 11:29:09.764147", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.12102 s, 953 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.12102 s, 953 MB/s"], "stdout": "", "stdout_lines": []} 2020-11-20T11:29:09.960 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:09 +0000 (0:00:00.287) 0:02:37.583 ******* 2020-11-20T11:29:10.240 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-11-20T11:29:10.242 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.056432", "end": "2020-11-20 11:29:10.208345", "rc": 0, "start": "2020-11-20 11:29:10.151913", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T11:29:10.274 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:10 +0000 (0:00:00.314) 0:02:37.898 ******* 2020-11-20T11:29:10.473 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-11-20T11:29:10.475 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-11-20T11:29:10.512 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:10 +0000 (0:00:00.237) 0:02:38.136 ******* 2020-11-20T11:29:10.520 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-11-20T11:29:10.522 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.040251", "end": "2020-11-20 11:29:10.491255", "rc": 0, "start": "2020-11-20 11:29:10.451004", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T11:29:10.558 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:10 +0000 (0:00:00.045) 0:02:38.182 ******* 2020-11-20T11:29:10.560 INFO:teuthology.task.ansible.out: 2020-11-20T11:29:10.586 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-11-20T11:29:10.624 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:10 +0000 (0:00:00.066) 0:02:38.248 ******* 2020-11-20T11:29:10.651 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-11-20T11:29:10.653 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:10.688 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:10 +0000 (0:00:00.063) 0:02:38.312 ******* 2020-11-20T11:29:10.715 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-11-20T11:29:10.717 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:10.749 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:10 +0000 (0:00:00.060) 0:02:38.372 ******* 2020-11-20T11:29:10.775 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-11-20T11:29:10.777 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:10.811 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:10 +0000 (0:00:00.062) 0:02:38.434 ******* 2020-11-20T11:29:10.838 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:10.872 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:10 +0000 (0:00:00.061) 0:02:38.496 ******* 2020-11-20T11:29:11.056 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ ok: [smithi023.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-11-20T11:29:11.089 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:11 +0000 (0:00:00.217) 0:02:38.713 ******* 2020-11-20T11:29:11.091 INFO:teuthology.task.ansible.out: 2020-11-20T11:29:11.107 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-11-20T11:29:11.108 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-20T11:29:11.129 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-11-20T11:29:11.131 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930420920320, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227153545, 'block_used': 12941563, 'inode_total': 61054976, 'inode_available': 60986950, 'inode_used': 68026, 'uuid': '782d0384-b480-44b3-9c95-fe95daf886ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227153545, "block_size": 4096, "block_total": 240095108, "block_used": 12941563, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986950, "inode_total": 61054976, "inode_used": 68026, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930420920320, "size_total": 983429562368, "uuid": "782d0384-b480-44b3-9c95-fe95daf886ae"}, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:11.172 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:11 +0000 (0:00:00.083) 0:02:38.796 ******* 2020-11-20T11:29:11.201 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:11.236 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:11 +0000 (0:00:00.063) 0:02:38.859 ******* 2020-11-20T11:29:11.263 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T11:29:11.264 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:11.299 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:11 +0000 (0:00:00.063) 0:02:38.923 ******* 2020-11-20T11:29:11.326 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:11.362 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:11 +0000 (0:00:00.062) 0:02:38.986 ******* 2020-11-20T11:29:11.389 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-11-20T11:29:11.390 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:11.424 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:11 +0000 (0:00:00.062) 0:02:39.048 ******* 2020-11-20T11:29:11.431 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-11-20T11:29:11.433 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-20T11:29:11.455 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:11.488 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:11 +0000 (0:00:00.063) 0:02:39.112 ******* 2020-11-20T11:29:11.515 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-11-20T11:29:11.516 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:11.548 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:11 +0000 (0:00:00.059) 0:02:39.172 ******* 2020-11-20T11:29:11.735 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi155.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-20T11:29:12.012 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-11-20T11:29:12.428 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} Friday 20 November 2020 11:29:12 +0000 (0:00:00.501) 0:02:39.673 ******* TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi155.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} changed: [smithi155.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} Friday 20 November 2020 11:29:12 +0000 (0:00:00.368) 0:02:40.042 ******* 2020-11-20T11:29:12.435 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-20T11:29:12.603 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-11-20T11:29:12.604 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-11-20T11:29:12.637 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:12 +0000 (0:00:00.218) 0:02:40.261 ******* 2020-11-20T11:29:12.758 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-11-20T11:29:12.760 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-20T11:29:12.833 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-11-20T11:29:12.835 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-20T11:29:13.004 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-20T11:29:13.109 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-11-20T11:29:13.111 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-20T11:29:13.175 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-11-20T11:29:13.176 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-20T11:29:13.345 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-20T11:29:13.362 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:13.410 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:13 +0000 (0:00:00.772) 0:02:41.034 ******* 2020-11-20T11:29:13.442 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-20T11:29:13.609 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-11-20T11:29:13.643 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:13 +0000 (0:00:00.233) 0:02:41.267 ******* 2020-11-20T11:29:13.670 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:13.706 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:13 +0000 (0:00:00.062) 0:02:41.329 ******* 2020-11-20T11:29:13.775 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-11-20T11:29:13.777 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-11-20T11:29:13.813 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:13 +0000 (0:00:00.107) 0:02:41.437 ******* 2020-11-20T11:29:14.010 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-11-20T11:29:14.011 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-11-20T11:29:14.045 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:14 +0000 (0:00:00.231) 0:02:41.668 ******* 2020-11-20T11:29:14.238 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-11-20T11:29:14.240 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true} 2020-11-20T11:29:14.275 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:14 +0000 (0:00:00.230) 0:02:41.899 ******* 2020-11-20T11:29:14.282 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-11-20T11:29:14.283 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-20T11:29:14.421 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-20T11:29:14.602 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-20T11:29:14.813 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-11-20T11:29:14.825 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:14.862 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:14 +0000 (0:00:00.586) 0:02:42.486 ******* 2020-11-20T11:29:14.868 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-11-20T11:29:14.870 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-11-20T11:29:14.906 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:14 +0000 (0:00:00.043) 0:02:42.529 ******* 2020-11-20T11:29:14.932 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-11-20T11:29:14.933 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:14.965 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:14 +0000 (0:00:00.060) 0:02:42.589 ******* 2020-11-20T11:29:14.992 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-11-20T11:29:14.994 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:15.026 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:15 +0000 (0:00:00.060) 0:02:42.650 ******* 2020-11-20T11:29:15.063 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:15.101 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:15 +0000 (0:00:00.074) 0:02:42.725 ******* 2020-11-20T11:29:15.107 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-11-20T11:29:15.109 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-11-20T11:29:15.146 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:15 +0000 (0:00:00.044) 0:02:42.770 ******* 2020-11-20T11:29:15.154 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-11-20T11:29:15.156 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:15.192 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:15 +0000 (0:00:00.045) 0:02:42.816 ******* 2020-11-20T11:29:15.198 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-11-20T11:29:15.201 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:15.234 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:15 +0000 (0:00:00.042) 0:02:42.858 ******* 2020-11-20T11:29:15.244 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:15.278 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:15 +0000 (0:00:00.043) 0:02:42.902 ******* 2020-11-20T11:29:15.306 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-11-20T11:29:15.307 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:15.341 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:15 +0000 (0:00:00.062) 0:02:42.965 ******* 2020-11-20T11:29:15.370 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-11-20T11:29:15.371 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:15.406 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:15 +0000 (0:00:00.063) 0:02:43.029 ******* 2020-11-20T11:29:15.434 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-11-20T11:29:15.436 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:15.468 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:15 +0000 (0:00:00.062) 0:02:43.092 ******* 2020-11-20T11:29:15.494 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-11-20T11:29:15.496 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:15.529 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:15 +0000 (0:00:00.061) 0:02:43.153 ******* 2020-11-20T11:29:15.556 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-11-20T11:29:15.558 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:15.591 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:15 +0000 (0:00:00.061) 0:02:43.215 ******* 2020-11-20T11:29:15.630 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-11-20T11:29:15.634 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:15.665 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:15 +0000 (0:00:00.073) 0:02:43.288 ******* 2020-11-20T11:29:16.122 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-11-20T11:29:16.124 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2020-11-20T11:29:16.157 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:16 +0000 (0:00:00.492) 0:02:43.781 ******* 2020-11-20T11:29:16.435 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-11-20T11:29:16.437 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2020-11-20 11:25:23 UTC", "ActiveEnterTimestampMonotonic": "15179457", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "basic.target ntpd.service -.mount systemd-tmpfiles-setup.service tmp.mount ntpdate.service system.slice sysinit.target sntp.service systemd-journald.socket", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2020-11-20 11:25:22 UTC", "AssertTimestampMonotonic": "13917295", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2020-11-20 11:25:22 UTC", "ConditionTimestampMonotonic": "13917248", "ConfigurationDirectoryMode": "0755", "Conflicts": "ntpd.service shutdown.target systemd-timesyncd.service", "ControlGro 2020-11-20T11:29:16.438 INFO:teuthology.task.ansible.out:up": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "894", "ExecMainStartTimestamp": "Fri 2020-11-20 11:25:22 UTC", "ExecMainStartTimestampMonotonic": "14156683", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[Fri 2020-11-20 11:25:22 UTC] ; stop_time=[Fri 2020-11-20 11:25:22 UTC] ; pid=878 ; code=exited ; status=0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[Fri 2020-11-20 11:25:22 UTC] ; stop_time=[Fri 2020-11-20 11:25:23 UTC] ; pid=900 ; code=exited ; status=0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2020-11-20 11:25:22 UTC", "InactiveExitTimestampMonotonic": "13946186", "InvocationID": "a48c82693f5041df92ef295b10b1d72d", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "Limit 2020-11-20T11:29:16.440 INFO:teuthology.task.ansible.out:AS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127237", "LimitNPROCSoft": "127237", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127237", "LimitSIGPENDINGSoft": "127237", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "894", "MemoryAccounting": "yes", "MemoryCurrent": "3231744", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "R 2020-11-20T11:29:16.442 INFO:teuthology.task.ansible.out:equires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2020-11-20 11:25:23 UTC", "StateChangeTimestampMonotonic": "15179457", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203580", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2020-11-20 11:25:22 UTC", "WatchdogTimestampMonotonic": "14156700", "WatchdogUSec": "0"}} 2020-11-20T11:29:16.471 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:16 +0000 (0:00:00.313) 0:02:44.095 ******* 2020-11-20T11:29:16.498 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-11-20T11:29:16.531 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:16 +0000 (0:00:00.060) 0:02:44.155 ******* 2020-11-20T11:29:16.572 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T11:29:16.574 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2020-11-20T11:29:16.605 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:16 +0000 (0:00:00.074) 0:02:44.229 ******* 2020-11-20T11:29:16.631 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-11-20T11:29:16.633 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:16.666 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:16 +0000 (0:00:00.060) 0:02:44.290 ******* 2020-11-20T11:29:16.692 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:16.727 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:16 +0000 (0:00:00.061) 0:02:44.351 ******* 2020-11-20T11:29:16.733 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* changed: [smithi023.front.sepia.ceph.com] => {"changed": true} 2020-11-20T11:29:16.769 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:16 +0000 (0:00:00.041) 0:02:44.393 ******* 2020-11-20T11:29:16.775 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-11-20T11:29:16.778 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:16.812 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:16 +0000 (0:00:00.042) 0:02:44.435 ******* 2020-11-20T11:29:16.998 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-11-20T11:29:16.999 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2020-11-20T11:29:17.033 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:17 +0000 (0:00:00.221) 0:02:44.657 ******* 2020-11-20T11:29:17.092 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-11-20T11:29:17.127 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:17 +0000 (0:00:00.094) 0:02:44.751 ******* 2020-11-20T11:29:17.154 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-11-20T11:29:17.156 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:17.188 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:17 +0000 (0:00:00.060) 0:02:44.812 ******* 2020-11-20T11:29:17.215 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:17.249 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:17 +0000 (0:00:00.060) 0:02:44.873 ******* 2020-11-20T11:29:17.275 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-11-20T11:29:17.277 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:17.313 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:17 +0000 (0:00:00.064) 0:02:44.937 ******* 2020-11-20T11:29:17.340 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-11-20T11:29:17.342 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:17.374 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:17 +0000 (0:00:00.060) 0:02:44.998 ******* 2020-11-20T11:29:17.401 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-11-20T11:29:17.403 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:17.435 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:17 +0000 (0:00:00.061) 0:02:45.059 ******* 2020-11-20T11:29:17.462 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:17.497 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:17 +0000 (0:00:00.061) 0:02:45.120 ******* 2020-11-20T11:29:17.503 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-11-20T11:29:17.505 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2020-11-20T11:29:17.538 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:17 +0000 (0:00:00.041) 0:02:45.162 ******* 2020-11-20T11:29:17.545 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-11-20T11:29:17.546 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:17.584 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:17 +0000 (0:00:00.045) 0:02:45.208 ******* 2020-11-20T11:29:17.611 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-11-20T11:29:17.613 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:17.645 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:17 +0000 (0:00:00.061) 0:02:45.269 ******* 2020-11-20T11:29:17.672 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-11-20T11:29:17.674 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:17.711 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:17 +0000 (0:00:00.065) 0:02:45.335 ******* 2020-11-20T11:29:17.738 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-11-20T11:29:17.740 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:17.772 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:17 +0000 (0:00:00.061) 0:02:45.396 ******* 2020-11-20T11:29:17.799 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-11-20T11:29:17.800 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:17.833 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:17 +0000 (0:00:00.060) 0:02:45.456 ******* 2020-11-20T11:29:18.285 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2020-11-20T11:29:18.318 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:18 +0000 (0:00:00.485) 0:02:45.942 ******* 2020-11-20T11:29:18.615 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-11-20T11:29:18.618 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2020-11-20 11:25:47 UTC", "ActiveEnterTimestampMonotonic": "15778029", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sntp.service system.slice ntpd.service basic.target sysinit.target systemd-journald.socket systemd-tmpfiles-setup.service -.mount tmp.mount ntpdate.service", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2020-11-20 11:25:46 UTC", "AssertTimestampMonotonic": "14641993", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2020-11-20 11:25:46 UTC", "ConditionTimestampMonotonic": "14641957", "ConfigurationDirectoryMode": "0755", "Conflicts": "ntpd.service shutdown.target systemd-timesyncd.service", "ControlGro 2020-11-20T11:29:18.622 INFO:teuthology.task.ansible.out:up": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "898", "ExecMainStartTimestamp": "Fri 2020-11-20 11:25:46 UTC", "ExecMainStartTimestampMonotonic": "14891166", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[Fri 2020-11-20 11:25:46 UTC] ; stop_time=[Fri 2020-11-20 11:25:46 UTC] ; pid=879 ; code=exited ; status=0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[Fri 2020-11-20 11:25:46 UTC] ; stop_time=[Fri 2020-11-20 11:25:47 UTC] ; pid=913 ; code=exited ; status=0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2020-11-20 11:25:46 UTC", "InactiveExitTimestampMonotonic": "14655731", "InvocationID": "5b20e99005aa48b99112b5c187b31204", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "Limit 2020-11-20T11:29:18.625 INFO:teuthology.task.ansible.out:AS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127237", "LimitNPROCSoft": "127237", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127237", "LimitSIGPENDINGSoft": "127237", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "898", "MemoryAccounting": "yes", "MemoryCurrent": "3194880", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "R 2020-11-20T11:29:18.626 INFO:teuthology.task.ansible.out:equires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2020-11-20 11:25:47 UTC", "StateChangeTimestampMonotonic": "15778029", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203580", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2020-11-20 11:25:46 UTC", "WatchdogTimestampMonotonic": "14891183", "WatchdogUSec": "0"}} 2020-11-20T11:29:18.655 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:18 +0000 (0:00:00.336) 0:02:46.279 ******* 2020-11-20T11:29:18.685 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-11-20T11:29:18.716 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:18 +0000 (0:00:00.061) 0:02:46.340 ******* 2020-11-20T11:29:18.755 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2020-11-20T11:29:18.789 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:18 +0000 (0:00:00.072) 0:02:46.413 ******* 2020-11-20T11:29:18.797 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-11-20T11:29:18.832 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:18 +0000 (0:00:00.042) 0:02:46.456 ******* 2020-11-20T11:29:18.838 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-11-20T11:29:18.840 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:18.873 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:18 +0000 (0:00:00.041) 0:02:46.497 ******* 2020-11-20T11:29:18.879 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-11-20T11:29:18.881 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:18.915 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:18 +0000 (0:00:00.041) 0:02:46.539 ******* 2020-11-20T11:29:18.921 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-11-20T11:29:18.923 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:18.959 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:18 +0000 (0:00:00.043) 0:02:46.582 ******* 2020-11-20T11:29:18.965 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-11-20T11:29:18.966 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-11-20T11:29:19.005 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:18 +0000 (0:00:00.044) 0:02:46.627 ******* 2020-11-20T11:29:19.009 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-11-20T11:29:19.010 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:19.044 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:19 +0000 (0:00:00.041) 0:02:46.668 ******* 2020-11-20T11:29:19.198 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-11-20T11:29:19.200 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2020-11-20T11:29:19.234 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:19 +0000 (0:00:00.189) 0:02:46.858 ******* 2020-11-20T11:29:19.241 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-11-20T11:29:19.243 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2020-11-20T11:29:19.276 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:19 +0000 (0:00:00.042) 0:02:46.900 ******* 2020-11-20T11:29:19.283 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-11-20T11:29:19.285 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:19.319 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:19 +0000 (0:00:00.042) 0:02:46.943 ******* 2020-11-20T11:29:19.346 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-11-20T11:29:19.348 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-11-20T11:29:19.381 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:19 +0000 (0:00:00.061) 0:02:47.005 ******* 2020-11-20T11:29:19.409 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-11-20T11:29:19.411 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:19.443 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:19 +0000 (0:00:00.061) 0:02:47.066 ******* 2020-11-20T11:29:19.627 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-11-20T11:29:19.629 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-11-20T11:29:19.664 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:19 +0000 (0:00:00.221) 0:02:47.288 ******* 2020-11-20T11:29:19.737 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2020-11-20T11:29:19.775 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:19 +0000 (0:00:00.111) 0:02:47.399 ******* 2020-11-20T11:29:19.856 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-11-20T11:29:19.858 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-11-20T11:29:19.891 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:19 +0000 (0:00:00.115) 0:02:47.515 ******* 2020-11-20T11:29:19.918 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-11-20T11:29:19.919 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:19.956 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:19 +0000 (0:00:00.064) 0:02:47.580 ******* 2020-11-20T11:29:20.000 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-11-20T11:29:20.001 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-11-20T11:29:20.011 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-11-20T11:29:20.013 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-11-20T11:29:20.046 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:20 +0000 (0:00:00.090) 0:02:47.670 ******* 2020-11-20T11:29:20.085 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** skipping: [smithi155.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:20.096 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:20.107 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:20.108 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:20.142 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:20 +0000 (0:00:00.094) 0:02:47.765 ******* 2020-11-20T11:29:20.168 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-11-20T11:29:20.170 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:20.205 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:20 +0000 (0:00:00.063) 0:02:47.829 ******* 2020-11-20T11:29:20.247 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:20.279 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:20 +0000 (0:00:00.074) 0:02:47.903 ******* 2020-11-20T11:29:20.317 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-11-20T11:29:20.321 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:20.354 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:20 +0000 (0:00:00.074) 0:02:47.978 ******* 2020-11-20T11:29:20.382 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-11-20T11:29:20.498 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 11:29:20 +0000 (0:00:00.062) 0:02:48.040 ******* TASK [testnode : Set MTU on Ubuntu] ******************************************** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 20 November 2020 11:29:20 +0000 (0:00:00.062) 0:02:48.102 ******* 2020-11-20T11:29:20.506 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:20.543 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:20 +0000 (0:00:00.063) 0:02:48.166 ******* 2020-11-20T11:29:20.572 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:20.614 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:20 +0000 (0:00:00.072) 0:02:48.238 ******* 2020-11-20T11:29:20.642 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:20.679 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:20 +0000 (0:00:00.064) 0:02:48.302 ******* 2020-11-20T11:29:20.833 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-11-20T11:29:20.835 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-11-20T11:29:20.869 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:20 +0000 (0:00:00.189) 0:02:48.492 ******* 2020-11-20T11:29:20.887 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-11-20T11:29:20.889 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005097", "end": "2020-11-20 11:29:20.854242", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-11-20 11:29:20.849145", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T11:29:20.899 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:20.932 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:20 +0000 (0:00:00.063) 0:02:48.556 ******* 2020-11-20T11:29:20.959 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-11-20T11:29:20.961 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2020-11-20T11:29:20.994 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:20 +0000 (0:00:00.061) 0:02:48.618 ******* 2020-11-20T11:29:21.097 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-11-20T11:29:21.099 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.002999", "end": "2020-11-20 11:29:21.044274", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-11-20 11:29:21.041275", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T11:29:21.215 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** ok: [smithi023.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2020-11-20T11:29:21.249 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:21 +0000 (0:00:00.254) 0:02:48.873 ******* 2020-11-20T11:29:21.267 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi155.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002806", "end": "2020-11-20 11:29:21.219348", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-11-20 11:29:21.216542", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2020-11-20T11:29:21.269 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:21.302 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:21 +0000 (0:00:00.052) 0:02:48.925 ******* 2020-11-20T11:29:21.347 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-11-20T11:29:21.348 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-11-20T11:29:21.349 INFO:teuthology.task.ansible.out: 2020-11-20T11:29:21.388 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:21 +0000 (0:00:00.086) 0:02:49.012 ******* 2020-11-20T11:29:21.414 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-11-20T11:29:21.416 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:21.457 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:21 +0000 (0:00:00.068) 0:02:49.080 ******* 2020-11-20T11:29:21.465 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi155.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.014313", "end": "2020-11-20 11:29:21.409073", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-11-20 11:29:21.394760", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T11:29:21.505 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:21 +0000 (0:00:00.047) 0:02:49.128 ******* 2020-11-20T11:29:21.532 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-11-20T11:29:21.569 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:21 +0000 (0:00:00.064) 0:02:49.193 ******* 2020-11-20T11:29:21.661 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-11-20T11:29:21.663 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-11-20T11:29:21.699 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:21 +0000 (0:00:00.128) 0:02:49.322 ******* 2020-11-20T11:29:21.757 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-11-20T11:29:21.759 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2020-11-20T11:29:21.885 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-11-20T11:29:21.887 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-11-20T11:29:21.918 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:21 +0000 (0:00:00.220) 0:02:49.542 ******* 2020-11-20T11:29:21.945 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-11-20T11:29:21.947 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:21.983 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:21 +0000 (0:00:00.064) 0:02:49.607 ******* 2020-11-20T11:29:22.010 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* skipping: [smithi023.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} 2020-11-20T11:29:22.021 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-11-20T11:29:22.023 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} 2020-11-20T11:29:22.033 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-11-20T11:29:22.035 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-11-20T11:29:22.069 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:22 +0000 (0:00:00.085) 0:02:49.692 ******* 2020-11-20T11:29:22.105 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-11-20T11:29:22.107 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:22.125 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi023.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:22.129 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:22.164 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:22 +0000 (0:00:00.095) 0:02:49.788 ******* 2020-11-20T11:29:22.193 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:22.226 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:22 +0000 (0:00:00.062) 0:02:49.850 ******* 2020-11-20T11:29:22.253 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-11-20T11:29:22.255 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:22.287 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:22 +0000 (0:00:00.060) 0:02:49.910 ******* 2020-11-20T11:29:22.313 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-11-20T11:29:22.315 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:22.352 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:22 +0000 (0:00:00.065) 0:02:49.976 ******* 2020-11-20T11:29:22.381 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-11-20T11:29:22.383 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:22.436 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:22 +0000 (0:00:00.084) 0:02:50.060 ******* 2020-11-20T11:29:22.466 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-11-20T11:29:22.468 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:22.502 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:22 +0000 (0:00:00.066) 0:02:50.126 ******* 2020-11-20T11:29:22.529 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-11-20T11:29:22.531 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:22.564 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:22 +0000 (0:00:00.062) 0:02:50.188 ******* 2020-11-20T11:29:22.591 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-11-20T11:29:22.593 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:22.627 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:22 +0000 (0:00:00.062) 0:02:50.251 ******* 2020-11-20T11:29:22.654 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-11-20T11:29:22.657 INFO:teuthology.task.ansible.out:skipping: [smithi023.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-11-20T11:29:22.692 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:22 +0000 (0:00:00.064) 0:02:50.316 ******* 2020-11-20T11:29:22.908 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi023.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005472", "end": "2020-11-20 11:29:22.872453", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-11-20 11:29:22.866981", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T11:29:23.076 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003142", "end": "2020-11-20 11:29:23.047613", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-11-20 11:29:23.044471", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T11:29:23.255 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002896", "end": "2020-11-20 11:29:23.221610", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-11-20 11:29:23.218714", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2020-11-20T11:29:24.450 INFO:teuthology.task.ansible.out:ok: [smithi023.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:01.021792", "end": "2020-11-20 11:29:24.417953", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-11-20 11:29:23.396161", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-11-20T11:29:24.485 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:24 +0000 (0:00:01.793) 0:02:52.109 ******* 2020-11-20T11:29:24.522 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-11-20T11:29:24.557 INFO:teuthology.task.ansible.out:Friday 20 November 2020 11:29:24 +0000 (0:00:00.071) 0:02:52.180 ******* 2020-11-20T11:29:24.752 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-11-20T11:29:24.753 INFO:teuthology.task.ansible.out:changed: [smithi023.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2020-11-20T11:29:24.851 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2020-11-20T11:29:24.855 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2020-11-20T11:29:24.858 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2020-11-20T11:29:24.860 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* smithi023.front.sepia.ceph.com : ok=125 changed=35 unreachable=0 failed=0 skipped=171 rescued=0 ignored=4 smithi155.front.sepia.ceph.com : ok=125 changed=35 unreachable=0 failed=0 skipped=171 rescued=0 ignored=4 Friday 20 November 2020 11:29:24 +0000 (0:00:00.301) 0:02:52.482 ******* =============================================================================== common : nrpe - Load SELinux policy package ---------------------------- 16.36s common : nrpe - Remove SELinux policy package -------------------------- 16.07s common : Register with subscription-manager. --------------------------- 15.23s testnode : Zap all non-root disks --------------------------------------- 6.09s testnode : Ensure ceph packages are not present. ------------------------ 6.05s testnode : Install packages --------------------------------------------- 5.83s testnode : Ensure ceph dependency packages are not present. ------------- 5.19s common : Get list of enabled RHSM repos --------------------------------- 2.50s common : Disable all rhsm repos ----------------------------------------- 2.45s testnode : Rebuild rpmdb ------------------------------------------------ 2.07s testnode : List any leftover Ceph artifacts from previous jobs ---------- 1.79s Gathering Facts --------------------------------------------------------- 1.59s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 1.37s common : nrpe - Ensure SELinux policy is up to date --------------------- 1.37s common : Make sure smartmontools is installed --------------------------- 1.35s common : nrpe - Install SELinux dependencies ---------------------------- 1.07s common : Install CA Cert from Satellite Server -------------------------- 1.07s common : Configure local epel mirrorlists ------------------------------- 1.04s common : Configure epel rep 2020-11-20T11:29:24.861 INFO:teuthology.task.ansible.out:os in /etc/yum.repos.d/ ---------------------- 1.02s ansible-managed : Create the cephlab_sudo sudoers.d file. --------------- 0.92s 2020-11-20T11:29:25.186 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi023.front.sepia.ceph.com'), Remote(name='ubuntu@smithi155.front.sepia.ceph.com')] 2020-11-20T11:29:25.187 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-20T11:29:25.188 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:29:25.258 INFO:teuthology.orchestra.run.smithi023:> true 2020-11-20T11:29:25.353 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-11-20T11:29:25.354 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-11-20T11:29:25.422 INFO:teuthology.orchestra.run.smithi155:> true 2020-11-20T11:29:25.517 INFO:teuthology.run_tasks:Running task clock... 2020-11-20T11:29:25.568 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-11-20T11:29:25.569 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-11-20T11:29:25.569 INFO:teuthology.orchestra.run.smithi023:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-11-20T11:29:25.604 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2020-11-20T11:29:25.619 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2020-11-20T11:29:25.654 INFO:teuthology.orchestra.run.smithi023.stderr:sudo: ntpd: command not found 2020-11-20T11:29:25.666 INFO:teuthology.orchestra.run.smithi023.stdout:506 Cannot talk to daemon 2020-11-20T11:29:25.680 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to start ntp.service: Unit ntp.service not found. 2020-11-20T11:29:25.695 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2020-11-20T11:29:25.773 INFO:teuthology.orchestra.run.smithi023.stderr:bash: ntpq: command not found 2020-11-20T11:29:25.775 INFO:teuthology.orchestra.run.smithi023.stdout:210 Number of sources = 4 2020-11-20T11:29:25.775 INFO:teuthology.orchestra.run.smithi023.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2020-11-20T11:29:25.776 INFO:teuthology.orchestra.run.smithi023.stdout:=============================================================================== 2020-11-20T11:29:25.776 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-11-20T11:29:25.776 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-11-20T11:29:25.777 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-11-20T11:29:25.777 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-11-20T11:29:25.778 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-11-20T11:29:25.778 INFO:teuthology.orchestra.run.smithi155:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-11-20T11:29:25.814 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2020-11-20T11:29:25.829 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2020-11-20T11:29:25.863 INFO:teuthology.orchestra.run.smithi155.stderr:sudo: ntpd: command not found 2020-11-20T11:29:25.875 INFO:teuthology.orchestra.run.smithi155.stdout:506 Cannot talk to daemon 2020-11-20T11:29:25.888 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntp.service: Unit ntp.service not found. 2020-11-20T11:29:25.903 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2020-11-20T11:29:25.962 INFO:teuthology.orchestra.run.smithi155.stderr:bash: ntpq: command not found 2020-11-20T11:29:25.965 INFO:teuthology.orchestra.run.smithi155.stdout:210 Number of sources = 4 2020-11-20T11:29:25.965 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2020-11-20T11:29:25.966 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2020-11-20T11:29:25.966 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-11-20T11:29:25.966 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-11-20T11:29:25.966 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-11-20T11:29:25.967 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-11-20T11:29:25.969 INFO:teuthology.run_tasks:Running task install... 2020-11-20T11:29:25.979 DEBUG:teuthology.task.install:project ceph 2020-11-20T11:29:25.980 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': 'b86e691bf191566922a8f08e2a5abf64fde8cfcb'}} 2020-11-20T11:29:25.980 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'sha1': 'b86e691bf191566922a8f08e2a5abf64fde8cfcb'} 2020-11-20T11:29:25.980 INFO:teuthology.task.install:Using flavor: basic 2020-11-20T11:29:25.987 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-11-20T11:29:25.988 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']} 2020-11-20T11:29:25.988 INFO:teuthology.orchestra.run.smithi023:> sudo lsb_release -is 2020-11-20T11:29:26.023 INFO:teuthology.orchestra.run.smithi023.stdout:RedHatEnterprise 2020-11-20T11:29:26.024 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2020-11-20T11:29:26.024 INFO:teuthology.orchestra.run.smithi155:> sudo lsb_release -is 2020-11-20T11:29:26.025 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b86e691bf191566922a8f08e2a5abf64fde8cfcb', 'tag': None, 'wait_for_package': False} 2020-11-20T11:29:26.025 INFO:teuthology.orchestra.run.smithi023:> sudo lsb_release -is 2020-11-20T11:29:26.058 INFO:teuthology.orchestra.run.smithi155.stdout:RedHatEnterprise 2020-11-20T11:29:26.059 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2020-11-20T11:29:26.060 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b86e691bf191566922a8f08e2a5abf64fde8cfcb', 'tag': None, 'wait_for_package': False} 2020-11-20T11:29:26.060 INFO:teuthology.orchestra.run.smithi155:> sudo lsb_release -is 2020-11-20T11:29:26.097 INFO:teuthology.orchestra.run.smithi023.stdout:RedHatEnterprise 2020-11-20T11:29:26.098 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2020-11-20T11:29:26.099 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b86e691bf191566922a8f08e2a5abf64fde8cfcb 2020-11-20T11:29:26.135 INFO:teuthology.orchestra.run.smithi155.stdout:RedHatEnterprise 2020-11-20T11:29:26.135 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2020-11-20T11:29:26.136 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b86e691bf191566922a8f08e2a5abf64fde8cfcb 2020-11-20T11:29:26.266 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/centos/8/flavors/default/ 2020-11-20T11:29:26.266 INFO:teuthology.task.install.rpm:Package version is 16.0.0-7397.gb86e691b 2020-11-20T11:29:26.295 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/centos/8/flavors/default/ 2020-11-20T11:29:26.295 INFO:teuthology.task.install.rpm:Package version is 16.0.0-7397.gb86e691b 2020-11-20T11:29:26.402 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2020-11-20T11:29:26.403 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-11-20T11:29:26.405 INFO:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/yum.repos.d/ceph.repo 2020-11-20T11:29:26.416 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20201120.053228/b86e691bf191566922a8f08e2a5abf64fde8cfcb/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2020-11-20T11:29:26.417 INFO:teuthology.orchestra.run.smithi155:> set -ex 2020-11-20T11:29:26.417 INFO:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/yum.repos.d/ceph.repo 2020-11-20T11:29:26.419 INFO:teuthology.orchestra.run.smithi023.stderr:+ sudo dd of=/etc/yum.repos.d/ceph.repo 2020-11-20T11:29:26.430 INFO:teuthology.orchestra.run.smithi023.stderr:1+1 records in 2020-11-20T11:29:26.430 INFO:teuthology.orchestra.run.smithi023.stderr:1+1 records out 2020-11-20T11:29:26.430 INFO:teuthology.orchestra.run.smithi023.stderr:676 bytes copied, 4.3147e-05 s, 15.7 MB/s 2020-11-20T11:29:26.431 INFO:teuthology.orchestra.run.smithi155.stderr:+ sudo dd of=/etc/yum.repos.d/ceph.repo 2020-11-20T11:29:26.432 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2020-11-20T11:29:26.432 INFO:teuthology.orchestra.run.smithi023:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b86e691bf191566922a8f08e2a5abf64fde8cfcb/;g' /etc/yum.repos.d/ceph.repo ; fi 2020-11-20T11:29:26.440 INFO:teuthology.orchestra.run.smithi155.stderr:1+1 records in 2020-11-20T11:29:26.440 INFO:teuthology.orchestra.run.smithi155.stderr:1+1 records out 2020-11-20T11:29:26.441 INFO:teuthology.orchestra.run.smithi155.stderr:676 bytes copied, 3.8983e-05 s, 17.3 MB/s 2020-11-20T11:29:26.441 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2020-11-20T11:29:26.442 INFO:teuthology.orchestra.run.smithi155:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b86e691bf191566922a8f08e2a5abf64fde8cfcb/;g' /etc/yum.repos.d/ceph.repo ; fi 2020-11-20T11:29:26.498 INFO:teuthology.orchestra.run.smithi023:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2020-11-20T11:29:26.506 INFO:teuthology.orchestra.run.smithi155:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2020-11-20T11:29:26.573 INFO:teuthology.orchestra.run.smithi023:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2020-11-20T11:29:26.580 INFO:teuthology.orchestra.run.smithi155:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2020-11-20T11:29:26.638 INFO:teuthology.orchestra.run.smithi023.stdout:check_obsoletes = 1 2020-11-20T11:29:26.639 INFO:teuthology.orchestra.run.smithi023:> sudo yum clean all 2020-11-20T11:29:26.642 INFO:teuthology.orchestra.run.smithi155.stdout:check_obsoletes = 1 2020-11-20T11:29:26.644 INFO:teuthology.orchestra.run.smithi155:> sudo yum clean all 2020-11-20T11:29:26.948 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2020-11-20T11:29:26.959 INFO:teuthology.orchestra.run.smithi023.stdout:Updating Subscription Management repositories. 2020-11-20T11:29:27.208 INFO:teuthology.orchestra.run.smithi155.stdout:38 files removed 2020-11-20T11:29:27.241 INFO:teuthology.orchestra.run.smithi023.stdout:38 files removed 2020-11-20T11:29:27.246 INFO:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2020-11-20T11:29:27.283 INFO:teuthology.orchestra.run.smithi023:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2020-11-20T11:29:27.512 INFO:teuthology.orchestra.run.smithi155.stdout:Updating Subscription Management repositories. 2020-11-20T11:29:27.564 INFO:teuthology.orchestra.run.smithi023.stdout:Updating Subscription Management repositories. 2020-11-20T11:29:28.129 INFO:teuthology.orchestra.run.smithi155.stdout:ceph packages for x86_64 292 kB/s | 72 kB 00:00 2020-11-20T11:29:28.217 INFO:teuthology.orchestra.run.smithi023.stdout:ceph packages for x86_64 288 kB/s | 72 kB 00:00 2020-11-20T11:29:28.361 INFO:teuthology.orchestra.run.smithi155.stdout:ceph noarch packages 73 kB/s | 15 kB 00:00 2020-11-20T11:29:28.454 INFO:teuthology.orchestra.run.smithi023.stdout:ceph noarch packages 72 kB/s | 15 kB 00:00 2020-11-20T11:29:28.587 INFO:teuthology.orchestra.run.smithi155.stdout:ceph source packages 7.0 kB/s | 1.4 kB 00:00 2020-11-20T11:29:28.683 INFO:teuthology.orchestra.run.smithi023.stdout:ceph source packages 6.9 kB/s | 1.4 kB 00:00 2020-11-20T11:29:29.007 INFO:teuthology.orchestra.run.smithi155.stdout:Extra Packages for Enterprise Linux 21 MB/s | 8.3 MB 00:00 2020-11-20T11:29:29.291 INFO:teuthology.orchestra.run.smithi023.stdout:Extra Packages for Enterprise Linux 14 MB/s | 8.3 MB 00:00 2020-11-20T11:29:30.735 INFO:teuthology.orchestra.run.smithi155.stdout:lab-extras 4.6 MB/s | 23 kB 00:00 2020-11-20T11:29:31.004 INFO:teuthology.orchestra.run.smithi155.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 109 MB/s | 24 MB 00:00 2020-11-20T11:29:31.101 INFO:teuthology.orchestra.run.smithi023.stdout:lab-extras 5.5 MB/s | 23 kB 00:00 2020-11-20T11:29:31.424 INFO:teuthology.orchestra.run.smithi023.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 93 MB/s | 25 MB 00:00 2020-11-20T11:29:34.847 INFO:teuthology.orchestra.run.smithi155.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 118 MB/s | 25 MB 00:00 2020-11-20T11:29:35.491 INFO:teuthology.orchestra.run.smithi023.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 86 MB/s | 24 MB 00:00 2020-11-20T11:29:40.595 INFO:teuthology.orchestra.run.smithi155.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2020-11-20T11:29:40.595 INFO:teuthology.orchestra.run.smithi155.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2020-11-20T11:29:40.602 INFO:teuthology.orchestra.run.smithi155.stdout:Package NetworkManager-1:1.26.0-8.el8.x86_64 is already installed. 2020-11-20T11:29:40.603 INFO:teuthology.orchestra.run.smithi155.stdout:Package iproute-5.3.0-5.el8.x86_64 is already installed. 2020-11-20T11:29:40.603 INFO:teuthology.orchestra.run.smithi155.stdout:Package util-linux-2.32.1-24.el8.x86_64 is already installed. 2020-11-20T11:29:40.604 INFO:teuthology.orchestra.run.smithi155.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2020-11-20T11:29:40.605 INFO:teuthology.orchestra.run.smithi155.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2020-11-20T11:29:40.607 INFO:teuthology.orchestra.run.smithi155.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2020-11-20T11:29:40.608 INFO:teuthology.orchestra.run.smithi155.stdout:Package xfsprogs-5.0.0-4.el8.x86_64 is already installed. 2020-11-20T11:29:40.921 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2020-11-20T11:29:40.946 INFO:teuthology.orchestra.run.smithi155.stdout:====================================================================================================================== 2020-11-20T11:29:40.946 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2020-11-20T11:29:40.946 INFO:teuthology.orchestra.run.smithi155.stdout:====================================================================================================================== 2020-11-20T11:29:40.946 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2020-11-20T11:29:40.947 INFO:teuthology.orchestra.run.smithi155.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2020-11-20T11:29:40.947 INFO:teuthology.orchestra.run.smithi155.stdout: ceph x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 6.4 k 2020-11-20T11:29:40.947 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 6.0 M 2020-11-20T11:29:40.947 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 632 k 2020-11-20T11:29:40.948 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 204 k 2020-11-20T11:29:40.948 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 1.5 M 2020-11-20T11:29:40.948 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm noarch 2:16.0.0-7397.gb86e691b.el8 ceph-noarch 66 k 2020-11-20T11:29:40.948 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard noarch 2:16.0.0-7397.gb86e691b.el8 ceph-noarch 3.2 M 2020-11-20T11:29:40.948 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local noarch 2:16.0.0-7397.gb86e691b.el8 ceph-noarch 8.6 M 2020-11-20T11:29:40.948 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook noarch 2:16.0.0-7397.gb86e691b.el8 ceph-noarch 27 k 2020-11-20T11:29:40.949 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 12 M 2020-11-20T11:29:40.949 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 48 M 2020-11-20T11:29:40.949 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm noarch 2:16.0.0-7397.gb86e691b.el8 ceph-noarch 57 k 2020-11-20T11:29:40.949 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-libelf-devel x86_64 0.180-1.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2020-11-20T11:29:40.949 INFO:teuthology.orchestra.run.smithi155.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2020-11-20T11:29:40.950 INFO:teuthology.orchestra.run.smithi155.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2020-11-20T11:29:40.950 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 24 k 2020-11-20T11:29:40.950 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2 x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 562 k 2020-11-20T11:29:40.950 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 131 k 2020-11-20T11:29:40.950 INFO:teuthology.orchestra.run.smithi155.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2020-11-20T11:29:40.951 INFO:teuthology.orchestra.run.smithi155.stdout: libuuid-devel x86_64 2.32.1-24.el8 rhel-8-for-x86_64-baseos-rpms 96 k 2020-11-20T11:29:40.951 INFO:teuthology.orchestra.run.smithi155.stdout: openssl-devel x86_64 1:1.1.1g-11.el8 rhel-8-for-x86_64-baseos-rpms 2.3 M 2020-11-20T11:29:40.951 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 204 k 2020-11-20T11:29:40.951 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 375 k 2020-11-20T11:29:40.951 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 353 k 2020-11-20T11:29:40.952 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 113 k 2020-11-20T11:29:40.952 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 88 k 2020-11-20T11:29:40.952 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 3.2 M 2020-11-20T11:29:40.952 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 171 k 2020-11-20T11:29:40.952 INFO:teuthology.orchestra.run.smithi155.stdout: xfsprogs-devel x86_64 5.0.0-4.el8 rhel-8-for-x86_64-baseos-rpms 69 k 2020-11-20T11:29:40.953 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrading: 2020-11-20T11:29:40.953 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager x86_64 1:1.26.0-9.el8_3 rhel-8-for-x86_64-baseos-rpms 2.4 M 2020-11-20T11:29:40.953 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager-libnm x86_64 1:1.26.0-9.el8_3 rhel-8-for-x86_64-baseos-rpms 1.7 M 2020-11-20T11:29:40.953 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager-team x86_64 1:1.26.0-9.el8_3 rhel-8-for-x86_64-baseos-rpms 141 k 2020-11-20T11:29:40.953 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager-tui x86_64 1:1.26.0-9.el8_3 rhel-8-for-x86_64-baseos-rpms 320 k 2020-11-20T11:29:40.954 INFO:teuthology.orchestra.run.smithi155.stdout: librados2 x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 3.7 M 2020-11-20T11:29:40.954 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1 x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 2.4 M 2020-11-20T11:29:40.954 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2020-11-20T11:29:40.954 INFO:teuthology.orchestra.run.smithi155.stdout: automake noarch 1.16.1-6.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2020-11-20T11:29:40.954 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 23 M 2020-11-20T11:29:40.955 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards noarch 2:16.0.0-7397.gb86e691b.el8 ceph-noarch 24 k 2020-11-20T11:29:40.955 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 2.1 M 2020-11-20T11:29:40.955 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core noarch 2:16.0.0-7397.gb86e691b.el8 ceph-noarch 198 k 2020-11-20T11:29:40.955 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 5.0 M 2020-11-20T11:29:40.955 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 18 M 2020-11-20T11:29:40.956 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts noarch 2:16.0.0-7397.gb86e691b.el8 ceph-noarch 8.8 k 2020-11-20T11:29:40.956 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 25 k 2020-11-20T11:29:40.956 INFO:teuthology.orchestra.run.smithi155.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2020-11-20T11:29:40.956 INFO:teuthology.orchestra.run.smithi155.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2020-11-20T11:29:40.956 INFO:teuthology.orchestra.run.smithi155.stdout: fuse-common x86_64 3.2.1-12.el8 rhel-8-for-x86_64-baseos-rpms 21 k 2020-11-20T11:29:40.957 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs x86_64 2.7-6.el8 epel 315 k 2020-11-20T11:29:40.957 INFO:teuthology.orchestra.run.smithi155.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2020-11-20T11:29:40.957 INFO:teuthology.orchestra.run.smithi155.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2020-11-20T11:29:40.957 INFO:teuthology.orchestra.run.smithi155.stdout: krb5-devel x86_64 1.18.2-5.el8 rhel-8-for-x86_64-baseos-rpms 558 k 2020-11-20T11:29:40.957 INFO:teuthology.orchestra.run.smithi155.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2020-11-20T11:29:40.958 INFO:teuthology.orchestra.run.smithi155.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2020-11-20T11:29:40.958 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran x86_64 8.3.1-5.1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2020-11-20T11:29:40.958 INFO:teuthology.orchestra.run.smithi155.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2020-11-20T11:29:40.958 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath x86_64 8.3.1-5.1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2020-11-20T11:29:40.958 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq x86_64 0.9.0-2.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2020-11-20T11:29:40.959 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1 x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 486 k 2020-11-20T11:29:40.959 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2020-11-20T11:29:40.959 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2 x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 3.7 M 2020-11-20T11:29:40.959 INFO:teuthology.orchestra.run.smithi155.stdout: libselinux-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 199 k 2020-11-20T11:29:40.959 INFO:teuthology.orchestra.run.smithi155.stdout: libsepol-devel x86_64 2.9-1.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2020-11-20T11:29:40.959 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2020-11-20T11:29:40.960 INFO:teuthology.orchestra.run.smithi155.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2020-11-20T11:29:40.960 INFO:teuthology.orchestra.run.smithi155.stdout: libxslt x86_64 1.1.32-5.el8 rhel-8-for-x86_64-baseos-rpms 250 k 2020-11-20T11:29:40.960 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2020-11-20T11:29:40.960 INFO:teuthology.orchestra.run.smithi155.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2020-11-20T11:29:40.960 INFO:teuthology.orchestra.run.smithi155.stdout: openblas x86_64 0.3.3-5.el8 rhel-8-for-x86_64-appstream-rpms 4.3 M 2020-11-20T11:29:40.961 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-threads x86_64 0.3.3-5.el8 rhel-8-for-x86_64-appstream-rpms 4.4 M 2020-11-20T11:29:40.961 INFO:teuthology.orchestra.run.smithi155.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2020-11-20T11:29:40.961 INFO:teuthology.orchestra.run.smithi155.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2020-11-20T11:29:40.961 INFO:teuthology.orchestra.run.smithi155.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2020-11-20T11:29:40.961 INFO:teuthology.orchestra.run.smithi155.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2020-11-20T11:29:40.962 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asn1crypto noarch 0.24.0-3.el8 rhel-8-for-x86_64-baseos-rpms 181 k 2020-11-20T11:29:40.962 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2020-11-20T11:29:40.962 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2020-11-20T11:29:40.962 INFO:teuthology.orchestra.run.smithi155.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2020-11-20T11:29:40.962 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2020-11-20T11:29:40.963 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 40 k 2020-11-20T11:29:40.963 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 73 k 2020-11-20T11:29:40.963 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2020-11-20T11:29:40.963 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2020-11-20T11:29:40.963 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot noarch 8.2.1-1.el8 epel 145 k 2020-11-20T11:29:40.964 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2020-11-20T11:29:40.964 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography x86_64 2.3-3.el8 rhel-8-for-x86_64-baseos-rpms 511 k 2020-11-20T11:29:40.964 INFO:teuthology.orchestra.run.smithi155.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2020-11-20T11:29:40.964 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2020-11-20T11:29:40.964 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2020-11-20T11:29:40.964 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2020-11-20T11:29:40.965 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2020-11-20T11:29:40.965 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2020-11-20T11:29:40.965 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2020-11-20T11:29:40.965 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2020-11-20T11:29:40.965 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes noarch 11.0.0-2.el8 epel 1.2 M 2020-11-20T11:29:40.966 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2020-11-20T11:29:40.966 INFO:teuthology.orchestra.run.smithi155.stdout: python3-lxml x86_64 4.2.3-1.el8 rhel-8-for-x86_64-appstream-rpms 1.5 M 2020-11-20T11:29:40.966 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2020-11-20T11:29:40.966 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2020-11-20T11:29:40.966 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2020-11-20T11:29:40.967 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2020-11-20T11:29:40.967 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2020-11-20T11:29:40.967 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2020-11-20T11:29:40.967 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2020-11-20T11:29:40.967 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2020-11-20T11:29:40.968 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2020-11-20T11:29:40.968 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL noarch 18.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2020-11-20T11:29:40.968 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2020-11-20T11:29:40.968 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2020-11-20T11:29:40.968 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2020-11-20T11:29:40.969 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2020-11-20T11:29:40.969 INFO:teuthology.orchestra.run.smithi155.stdout: python3-remoto noarch 1.1.4-4.el8 epel 48 k 2020-11-20T11:29:40.969 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2020-11-20T11:29:40.969 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2020-11-20T11:29:40.969 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2020-11-20T11:29:40.970 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa noarch 3.4.2-15.el8 epel 73 k 2020-11-20T11:29:40.970 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2020-11-20T11:29:40.970 INFO:teuthology.orchestra.run.smithi155.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2020-11-20T11:29:40.970 INFO:teuthology.orchestra.run.smithi155.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2020-11-20T11:29:40.970 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2020-11-20T11:29:40.971 INFO:teuthology.orchestra.run.smithi155.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2020-11-20T11:29:40.971 INFO:teuthology.orchestra.run.smithi155.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2020-11-20T11:29:40.971 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webencodings noarch 0.5.1-6.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2020-11-20T11:29:40.971 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2020-11-20T11:29:40.971 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2020-11-20T11:29:40.972 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2020-11-20T11:29:40.972 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2020-11-20T11:29:40.972 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2020-11-20T11:29:40.972 INFO:teuthology.orchestra.run.smithi155.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2020-11-20T11:29:40.972 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2020-11-20T11:29:40.973 INFO:teuthology.orchestra.run.smithi155.stdout: zlib-devel x86_64 1.2.11-16.el8_2 rhel-8-for-x86_64-baseos-rpms 57 k 2020-11-20T11:29:40.973 INFO:teuthology.orchestra.run.smithi155.stdout:Installing weak dependencies: 2020-11-20T11:29:40.973 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cssselect noarch 0.9.2-10.el8 epel 40 k 2020-11-20T11:29:40.973 INFO:teuthology.orchestra.run.smithi155.stdout: python3-html5lib noarch 1:0.999999999-6.el8 rhel-8-for-x86_64-appstream-rpms 214 k 2020-11-20T11:29:40.973 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:29:40.974 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2020-11-20T11:29:40.974 INFO:teuthology.orchestra.run.smithi155.stdout:====================================================================================================================== 2020-11-20T11:29:40.974 INFO:teuthology.orchestra.run.smithi155.stdout:Install 127 Packages 2020-11-20T11:29:40.974 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrade 6 Packages 2020-11-20T11:29:40.974 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:29:40.975 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 199 M 2020-11-20T11:29:40.975 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2020-11-20T11:29:41.126 INFO:teuthology.orchestra.run.smithi155.stdout:(1/133): ceph-16.0.0-7397.gb86e691b.el8.x86_64. 39 kB/s | 6.4 kB 00:00 2020-11-20T11:29:41.797 INFO:teuthology.orchestra.run.smithi023.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2020-11-20T11:29:41.798 INFO:teuthology.orchestra.run.smithi023.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2020-11-20T11:29:41.805 INFO:teuthology.orchestra.run.smithi023.stdout:Package NetworkManager-1:1.26.0-8.el8.x86_64 is already installed. 2020-11-20T11:29:41.806 INFO:teuthology.orchestra.run.smithi023.stdout:Package iproute-5.3.0-5.el8.x86_64 is already installed. 2020-11-20T11:29:41.808 INFO:teuthology.orchestra.run.smithi023.stdout:Package util-linux-2.32.1-24.el8.x86_64 is already installed. 2020-11-20T11:29:41.809 INFO:teuthology.orchestra.run.smithi023.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2020-11-20T11:29:41.809 INFO:teuthology.orchestra.run.smithi023.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2020-11-20T11:29:41.810 INFO:teuthology.orchestra.run.smithi023.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2020-11-20T11:29:41.812 INFO:teuthology.orchestra.run.smithi023.stdout:Package xfsprogs-5.0.0-4.el8.x86_64 is already installed. 2020-11-20T11:29:41.823 INFO:teuthology.orchestra.run.smithi155.stdout:(2/133): ceph-fuse-16.0.0-7397.gb86e691b.el8.x8 910 kB/s | 632 kB 00:00 2020-11-20T11:29:41.966 INFO:teuthology.orchestra.run.smithi155.stdout:(3/133): ceph-immutable-object-cache-16.0.0-739 1.4 MB/s | 204 kB 00:00 2020-11-20T11:29:42.144 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2020-11-20T11:29:42.165 INFO:teuthology.orchestra.run.smithi023.stdout:====================================================================================================================== 2020-11-20T11:29:42.165 INFO:teuthology.orchestra.run.smithi023.stdout: Package Arch Version Repository Size 2020-11-20T11:29:42.165 INFO:teuthology.orchestra.run.smithi023.stdout:====================================================================================================================== 2020-11-20T11:29:42.166 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2020-11-20T11:29:42.166 INFO:teuthology.orchestra.run.smithi023.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2020-11-20T11:29:42.167 INFO:teuthology.orchestra.run.smithi023.stdout: ceph x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 6.4 k 2020-11-20T11:29:42.167 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-base x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 6.0 M 2020-11-20T11:29:42.167 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-fuse x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 632 k 2020-11-20T11:29:42.167 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-immutable-object-cache x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 204 k 2020-11-20T11:29:42.168 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 1.5 M 2020-11-20T11:29:42.168 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-cephadm noarch 2:16.0.0-7397.gb86e691b.el8 ceph-noarch 66 k 2020-11-20T11:29:42.168 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-dashboard noarch 2:16.0.0-7397.gb86e691b.el8 ceph-noarch 3.2 M 2020-11-20T11:29:42.168 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-diskprediction-local noarch 2:16.0.0-7397.gb86e691b.el8 ceph-noarch 8.6 M 2020-11-20T11:29:42.168 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-rook noarch 2:16.0.0-7397.gb86e691b.el8 ceph-noarch 27 k 2020-11-20T11:29:42.169 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-radosgw x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 12 M 2020-11-20T11:29:42.169 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-test x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 48 M 2020-11-20T11:29:42.169 INFO:teuthology.orchestra.run.smithi023.stdout: cephadm noarch 2:16.0.0-7397.gb86e691b.el8 ceph-noarch 57 k 2020-11-20T11:29:42.169 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libelf-devel x86_64 0.180-1.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2020-11-20T11:29:42.170 INFO:teuthology.orchestra.run.smithi023.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2020-11-20T11:29:42.170 INFO:teuthology.orchestra.run.smithi023.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2020-11-20T11:29:42.170 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs-devel x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 24 k 2020-11-20T11:29:42.170 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs2 x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 562 k 2020-11-20T11:29:42.171 INFO:teuthology.orchestra.run.smithi023.stdout: librados-devel x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 131 k 2020-11-20T11:29:42.171 INFO:teuthology.orchestra.run.smithi023.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2020-11-20T11:29:42.171 INFO:teuthology.orchestra.run.smithi023.stdout: libuuid-devel x86_64 2.32.1-24.el8 rhel-8-for-x86_64-baseos-rpms 96 k 2020-11-20T11:29:42.171 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-devel x86_64 1:1.1.1g-11.el8 rhel-8-for-x86_64-baseos-rpms 2.3 M 2020-11-20T11:29:42.171 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cephfs x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 204 k 2020-11-20T11:29:42.172 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rados x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 375 k 2020-11-20T11:29:42.172 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rbd x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 353 k 2020-11-20T11:29:42.172 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rgw x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 113 k 2020-11-20T11:29:42.172 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-fuse x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 88 k 2020-11-20T11:29:42.172 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-mirror x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 3.2 M 2020-11-20T11:29:42.173 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-nbd x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 171 k 2020-11-20T11:29:42.173 INFO:teuthology.orchestra.run.smithi023.stdout: xfsprogs-devel x86_64 5.0.0-4.el8 rhel-8-for-x86_64-baseos-rpms 69 k 2020-11-20T11:29:42.173 INFO:teuthology.orchestra.run.smithi023.stdout:Upgrading: 2020-11-20T11:29:42.173 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager x86_64 1:1.26.0-9.el8_3 rhel-8-for-x86_64-baseos-rpms 2.4 M 2020-11-20T11:29:42.174 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-libnm x86_64 1:1.26.0-9.el8_3 rhel-8-for-x86_64-baseos-rpms 1.7 M 2020-11-20T11:29:42.174 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-team x86_64 1:1.26.0-9.el8_3 rhel-8-for-x86_64-baseos-rpms 141 k 2020-11-20T11:29:42.174 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-tui x86_64 1:1.26.0-9.el8_3 rhel-8-for-x86_64-baseos-rpms 320 k 2020-11-20T11:29:42.174 INFO:teuthology.orchestra.run.smithi023.stdout: librados2 x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 3.7 M 2020-11-20T11:29:42.174 INFO:teuthology.orchestra.run.smithi023.stdout: librbd1 x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 2.4 M 2020-11-20T11:29:42.175 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2020-11-20T11:29:42.175 INFO:teuthology.orchestra.run.smithi023.stdout: automake noarch 1.16.1-6.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2020-11-20T11:29:42.175 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-common x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 23 M 2020-11-20T11:29:42.175 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-grafana-dashboards noarch 2:16.0.0-7397.gb86e691b.el8 ceph-noarch 24 k 2020-11-20T11:29:42.176 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mds x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 2.1 M 2020-11-20T11:29:42.176 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-modules-core noarch 2:16.0.0-7397.gb86e691b.el8 ceph-noarch 198 k 2020-11-20T11:29:42.176 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mon x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 5.0 M 2020-11-20T11:29:42.176 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-osd x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 18 M 2020-11-20T11:29:42.176 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-prometheus-alerts noarch 2:16.0.0-7397.gb86e691b.el8 ceph-noarch 8.8 k 2020-11-20T11:29:42.177 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-selinux x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 25 k 2020-11-20T11:29:42.177 INFO:teuthology.orchestra.run.smithi023.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2020-11-20T11:29:42.177 INFO:teuthology.orchestra.run.smithi023.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2020-11-20T11:29:42.177 INFO:teuthology.orchestra.run.smithi023.stdout: fuse-common x86_64 3.2.1-12.el8 rhel-8-for-x86_64-baseos-rpms 21 k 2020-11-20T11:29:42.177 INFO:teuthology.orchestra.run.smithi023.stdout: gperftools-libs x86_64 2.7-6.el8 epel 315 k 2020-11-20T11:29:42.178 INFO:teuthology.orchestra.run.smithi023.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2020-11-20T11:29:42.178 INFO:teuthology.orchestra.run.smithi023.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2020-11-20T11:29:42.178 INFO:teuthology.orchestra.run.smithi023.stdout: krb5-devel x86_64 1.18.2-5.el8 rhel-8-for-x86_64-baseos-rpms 558 k 2020-11-20T11:29:42.178 INFO:teuthology.orchestra.run.smithi023.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2020-11-20T11:29:42.178 INFO:teuthology.orchestra.run.smithi023.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2020-11-20T11:29:42.179 INFO:teuthology.orchestra.run.smithi023.stdout: libgfortran x86_64 8.3.1-5.1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2020-11-20T11:29:42.179 INFO:teuthology.orchestra.run.smithi023.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2020-11-20T11:29:42.179 INFO:teuthology.orchestra.run.smithi023.stdout: libquadmath x86_64 8.3.1-5.1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2020-11-20T11:29:42.179 INFO:teuthology.orchestra.run.smithi023.stdout: librabbitmq x86_64 0.9.0-2.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2020-11-20T11:29:42.179 INFO:teuthology.orchestra.run.smithi023.stdout: libradosstriper1 x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 486 k 2020-11-20T11:29:42.180 INFO:teuthology.orchestra.run.smithi023.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2020-11-20T11:29:42.180 INFO:teuthology.orchestra.run.smithi023.stdout: librgw2 x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 3.7 M 2020-11-20T11:29:42.180 INFO:teuthology.orchestra.run.smithi023.stdout: libselinux-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 199 k 2020-11-20T11:29:42.180 INFO:teuthology.orchestra.run.smithi023.stdout: libsepol-devel x86_64 2.9-1.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2020-11-20T11:29:42.180 INFO:teuthology.orchestra.run.smithi023.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2020-11-20T11:29:42.181 INFO:teuthology.orchestra.run.smithi023.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2020-11-20T11:29:42.181 INFO:teuthology.orchestra.run.smithi023.stdout: libxslt x86_64 1.1.32-5.el8 rhel-8-for-x86_64-baseos-rpms 250 k 2020-11-20T11:29:42.181 INFO:teuthology.orchestra.run.smithi023.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2020-11-20T11:29:42.182 INFO:teuthology.orchestra.run.smithi023.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2020-11-20T11:29:42.182 INFO:teuthology.orchestra.run.smithi023.stdout: openblas x86_64 0.3.3-5.el8 rhel-8-for-x86_64-appstream-rpms 4.3 M 2020-11-20T11:29:42.182 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-threads x86_64 0.3.3-5.el8 rhel-8-for-x86_64-appstream-rpms 4.4 M 2020-11-20T11:29:42.182 INFO:teuthology.orchestra.run.smithi023.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2020-11-20T11:29:42.182 INFO:teuthology.orchestra.run.smithi023.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2020-11-20T11:29:42.183 INFO:teuthology.orchestra.run.smithi023.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2020-11-20T11:29:42.183 INFO:teuthology.orchestra.run.smithi023.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2020-11-20T11:29:42.183 INFO:teuthology.orchestra.run.smithi023.stdout: python3-asn1crypto noarch 0.24.0-3.el8 rhel-8-for-x86_64-baseos-rpms 181 k 2020-11-20T11:29:42.183 INFO:teuthology.orchestra.run.smithi023.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2020-11-20T11:29:42.184 INFO:teuthology.orchestra.run.smithi023.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2020-11-20T11:29:42.184 INFO:teuthology.orchestra.run.smithi023.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2020-11-20T11:29:42.184 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2020-11-20T11:29:42.184 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-argparse x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 40 k 2020-11-20T11:29:42.185 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-common x86_64 2:16.0.0-7397.gb86e691b.el8 ceph 73 k 2020-11-20T11:29:42.185 INFO:teuthology.orchestra.run.smithi023.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2020-11-20T11:29:42.185 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2020-11-20T11:29:42.185 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cheroot noarch 8.2.1-1.el8 epel 145 k 2020-11-20T11:29:42.185 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2020-11-20T11:29:42.186 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cryptography x86_64 2.3-3.el8 rhel-8-for-x86_64-baseos-rpms 511 k 2020-11-20T11:29:42.186 INFO:teuthology.orchestra.run.smithi023.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2020-11-20T11:29:42.186 INFO:teuthology.orchestra.run.smithi023.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2020-11-20T11:29:42.186 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2020-11-20T11:29:42.186 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2020-11-20T11:29:42.187 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2020-11-20T11:29:42.187 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2020-11-20T11:29:42.187 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2020-11-20T11:29:42.187 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2020-11-20T11:29:42.187 INFO:teuthology.orchestra.run.smithi023.stdout: python3-kubernetes noarch 11.0.0-2.el8 epel 1.2 M 2020-11-20T11:29:42.188 INFO:teuthology.orchestra.run.smithi023.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2020-11-20T11:29:42.188 INFO:teuthology.orchestra.run.smithi023.stdout: python3-lxml x86_64 4.2.3-1.el8 rhel-8-for-x86_64-appstream-rpms 1.5 M 2020-11-20T11:29:42.188 INFO:teuthology.orchestra.run.smithi023.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2020-11-20T11:29:42.188 INFO:teuthology.orchestra.run.smithi023.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2020-11-20T11:29:42.188 INFO:teuthology.orchestra.run.smithi023.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2020-11-20T11:29:42.189 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2020-11-20T11:29:42.189 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2020-11-20T11:29:42.189 INFO:teuthology.orchestra.run.smithi023.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2020-11-20T11:29:42.189 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2020-11-20T11:29:42.189 INFO:teuthology.orchestra.run.smithi023.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2020-11-20T11:29:42.190 INFO:teuthology.orchestra.run.smithi023.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2020-11-20T11:29:42.190 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyOpenSSL noarch 18.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2020-11-20T11:29:42.190 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2020-11-20T11:29:42.190 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2020-11-20T11:29:42.190 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2020-11-20T11:29:42.191 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2020-11-20T11:29:42.191 INFO:teuthology.orchestra.run.smithi023.stdout: python3-remoto noarch 1.1.4-4.el8 epel 48 k 2020-11-20T11:29:42.191 INFO:teuthology.orchestra.run.smithi023.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2020-11-20T11:29:42.191 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2020-11-20T11:29:42.191 INFO:teuthology.orchestra.run.smithi023.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2020-11-20T11:29:42.192 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rsa noarch 3.4.2-15.el8 epel 73 k 2020-11-20T11:29:42.192 INFO:teuthology.orchestra.run.smithi023.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2020-11-20T11:29:42.192 INFO:teuthology.orchestra.run.smithi023.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2020-11-20T11:29:42.192 INFO:teuthology.orchestra.run.smithi023.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2020-11-20T11:29:42.193 INFO:teuthology.orchestra.run.smithi023.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2020-11-20T11:29:42.193 INFO:teuthology.orchestra.run.smithi023.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2020-11-20T11:29:42.193 INFO:teuthology.orchestra.run.smithi023.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2020-11-20T11:29:42.193 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webencodings noarch 0.5.1-6.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2020-11-20T11:29:42.194 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2020-11-20T11:29:42.194 INFO:teuthology.orchestra.run.smithi023.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2020-11-20T11:29:42.194 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2020-11-20T11:29:42.194 INFO:teuthology.orchestra.run.smithi023.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2020-11-20T11:29:42.194 INFO:teuthology.orchestra.run.smithi023.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2020-11-20T11:29:42.195 INFO:teuthology.orchestra.run.smithi023.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2020-11-20T11:29:42.195 INFO:teuthology.orchestra.run.smithi023.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2020-11-20T11:29:42.195 INFO:teuthology.orchestra.run.smithi023.stdout: zlib-devel x86_64 1.2.11-16.el8_2 rhel-8-for-x86_64-baseos-rpms 57 k 2020-11-20T11:29:42.195 INFO:teuthology.orchestra.run.smithi023.stdout:Installing weak dependencies: 2020-11-20T11:29:42.195 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cssselect noarch 0.9.2-10.el8 epel 40 k 2020-11-20T11:29:42.196 INFO:teuthology.orchestra.run.smithi023.stdout: python3-html5lib noarch 1:0.999999999-6.el8 rhel-8-for-x86_64-appstream-rpms 214 k 2020-11-20T11:29:42.196 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:29:42.196 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2020-11-20T11:29:42.196 INFO:teuthology.orchestra.run.smithi023.stdout:====================================================================================================================== 2020-11-20T11:29:42.196 INFO:teuthology.orchestra.run.smithi023.stdout:Install 127 Packages 2020-11-20T11:29:42.197 INFO:teuthology.orchestra.run.smithi023.stdout:Upgrade 6 Packages 2020-11-20T11:29:42.197 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:29:42.198 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 199 M 2020-11-20T11:29:42.198 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2020-11-20T11:29:42.328 INFO:teuthology.orchestra.run.smithi023.stdout:(1/133): ceph-16.0.0-7397.gb86e691b.el8.x86_64. 44 kB/s | 6.4 kB 00:00 2020-11-20T11:29:42.584 INFO:teuthology.orchestra.run.smithi155.stdout:(4/133): ceph-mds-16.0.0-7397.gb86e691b.el8.x86 3.5 MB/s | 2.1 MB 00:00 2020-11-20T11:29:42.918 INFO:teuthology.orchestra.run.smithi023.stdout:(2/133): ceph-fuse-16.0.0-7397.gb86e691b.el8.x8 1.1 MB/s | 632 kB 00:00 2020-11-20T11:29:42.985 INFO:teuthology.orchestra.run.smithi155.stdout:(5/133): ceph-mgr-16.0.0-7397.gb86e691b.el8.x86 3.8 MB/s | 1.5 MB 00:00 2020-11-20T11:29:43.099 INFO:teuthology.orchestra.run.smithi023.stdout:(3/133): ceph-immutable-object-cache-16.0.0-739 1.1 MB/s | 204 kB 00:00 2020-11-20T11:29:43.150 INFO:teuthology.orchestra.run.smithi155.stdout:(6/133): ceph-base-16.0.0-7397.gb86e691b.el8.x8 2.7 MB/s | 6.0 MB 00:02 2020-11-20T11:29:44.088 INFO:teuthology.orchestra.run.smithi155.stdout:(7/133): ceph-mon-16.0.0-7397.gb86e691b.el8.x86 4.5 MB/s | 5.0 MB 00:01 2020-11-20T11:29:44.617 INFO:teuthology.orchestra.run.smithi023.stdout:(4/133): ceph-mds-16.0.0-7397.gb86e691b.el8.x86 1.4 MB/s | 2.1 MB 00:01 2020-11-20T11:29:45.205 INFO:teuthology.orchestra.run.smithi023.stdout:(5/133): ceph-base-16.0.0-7397.gb86e691b.el8.x8 2.0 MB/s | 6.0 MB 00:03 2020-11-20T11:29:45.642 INFO:teuthology.orchestra.run.smithi023.stdout:(6/133): ceph-mgr-16.0.0-7397.gb86e691b.el8.x86 1.5 MB/s | 1.5 MB 00:01 2020-11-20T11:29:46.278 INFO:teuthology.orchestra.run.smithi155.stdout:(8/133): ceph-common-16.0.0-7397.gb86e691b.el8. 4.3 MB/s | 23 MB 00:05 2020-11-20T11:29:46.349 INFO:teuthology.orchestra.run.smithi155.stdout:(9/133): ceph-selinux-16.0.0-7397.gb86e691b.el8 348 kB/s | 25 kB 00:00 2020-11-20T11:29:46.687 INFO:teuthology.orchestra.run.smithi155.stdout:(10/133): ceph-osd-16.0.0-7397.gb86e691b.el8.x8 5.1 MB/s | 18 MB 00:03 2020-11-20T11:29:46.790 INFO:teuthology.orchestra.run.smithi155.stdout:(11/133): ceph-radosgw-16.0.0-7397.gb86e691b.el 4.4 MB/s | 12 MB 00:02 2020-11-20T11:29:46.790 INFO:teuthology.orchestra.run.smithi155.stdout:(12/133): libcephfs-devel-16.0.0-7397.gb86e691b 229 kB/s | 24 kB 00:00 2020-11-20T11:29:46.855 INFO:teuthology.orchestra.run.smithi155.stdout:(13/133): librados-devel-16.0.0-7397.gb86e691b. 2.0 MB/s | 131 kB 00:00 2020-11-20T11:29:46.905 INFO:teuthology.orchestra.run.smithi155.stdout:(14/133): libcephfs2-16.0.0-7397.gb86e691b.el8. 4.8 MB/s | 562 kB 00:00 2020-11-20T11:29:46.977 INFO:teuthology.orchestra.run.smithi155.stdout:(15/133): libradosstriper1-16.0.0-7397.gb86e691 3.9 MB/s | 486 kB 00:00 2020-11-20T11:29:47.030 INFO:teuthology.orchestra.run.smithi155.stdout:(16/133): python3-ceph-argparse-16.0.0-7397.gb8 755 kB/s | 40 kB 00:00 2020-11-20T11:29:47.083 INFO:teuthology.orchestra.run.smithi155.stdout:(17/133): python3-ceph-common-16.0.0-7397.gb86e 1.3 MB/s | 73 kB 00:00 2020-11-20T11:29:47.109 INFO:teuthology.orchestra.run.smithi023.stdout:(7/133): ceph-mon-16.0.0-7397.gb86e691b.el8.x86 2.6 MB/s | 5.0 MB 00:01 2020-11-20T11:29:47.141 INFO:teuthology.orchestra.run.smithi155.stdout:(18/133): python3-cephfs-16.0.0-7397.gb86e691b. 3.4 MB/s | 204 kB 00:00 2020-11-20T11:29:47.211 INFO:teuthology.orchestra.run.smithi155.stdout:(19/133): python3-rados-16.0.0-7397.gb86e691b.e 5.2 MB/s | 375 kB 00:00 2020-11-20T11:29:47.285 INFO:teuthology.orchestra.run.smithi155.stdout:(20/133): python3-rbd-16.0.0-7397.gb86e691b.el8 4.7 MB/s | 353 kB 00:00 2020-11-20T11:29:47.343 INFO:teuthology.orchestra.run.smithi155.stdout:(21/133): python3-rgw-16.0.0-7397.gb86e691b.el8 1.9 MB/s | 113 kB 00:00 2020-11-20T11:29:47.407 INFO:teuthology.orchestra.run.smithi155.stdout:(22/133): rbd-fuse-16.0.0-7397.gb86e691b.el8.x8 1.4 MB/s | 88 kB 00:00 2020-11-20T11:29:47.519 INFO:teuthology.orchestra.run.smithi155.stdout:(23/133): librgw2-16.0.0-7397.gb86e691b.el8.x86 6.0 MB/s | 3.7 MB 00:00 2020-11-20T11:29:47.586 INFO:teuthology.orchestra.run.smithi155.stdout:(24/133): rbd-nbd-16.0.0-7397.gb86e691b.el8.x86 2.5 MB/s | 171 kB 00:00 2020-11-20T11:29:47.645 INFO:teuthology.orchestra.run.smithi155.stdout:(25/133): ceph-grafana-dashboards-16.0.0-7397.g 404 kB/s | 24 kB 00:00 2020-11-20T11:29:47.711 INFO:teuthology.orchestra.run.smithi155.stdout:(26/133): ceph-mgr-cephadm-16.0.0-7397.gb86e691 1.0 MB/s | 66 kB 00:00 2020-11-20T11:29:47.931 INFO:teuthology.orchestra.run.smithi155.stdout:(27/133): rbd-mirror-16.0.0-7397.gb86e691b.el8. 6.1 MB/s | 3.2 MB 00:00 2020-11-20T11:29:48.314 INFO:teuthology.orchestra.run.smithi155.stdout:(28/133): ceph-mgr-dashboard-16.0.0-7397.gb86e6 5.2 MB/s | 3.2 MB 00:00 2020-11-20T11:29:48.388 INFO:teuthology.orchestra.run.smithi155.stdout:(29/133): ceph-mgr-modules-core-16.0.0-7397.gb8 2.6 MB/s | 198 kB 00:00 2020-11-20T11:29:48.456 INFO:teuthology.orchestra.run.smithi155.stdout:(30/133): ceph-mgr-rook-16.0.0-7397.gb86e691b.e 403 kB/s | 27 kB 00:00 2020-11-20T11:29:48.523 INFO:teuthology.orchestra.run.smithi155.stdout:(31/133): ceph-prometheus-alerts-16.0.0-7397.gb 132 kB/s | 8.8 kB 00:00 2020-11-20T11:29:48.593 INFO:teuthology.orchestra.run.smithi155.stdout:(32/133): cephadm-16.0.0-7397.gb86e691b.el8.noa 822 kB/s | 57 kB 00:00 2020-11-20T11:29:48.674 INFO:teuthology.orchestra.run.smithi155.stdout:(33/133): fmt-6.2.1-1.el8.x86_64.rpm 1.4 MB/s | 116 kB 00:00 2020-11-20T11:29:48.723 INFO:teuthology.orchestra.run.smithi155.stdout:(34/133): gperftools-libs-2.7-6.el8.x86_64.rpm 6.3 MB/s | 315 kB 00:00 2020-11-20T11:29:48.740 INFO:teuthology.orchestra.run.smithi155.stdout:(35/133): leveldb-1.22-1.el8.x86_64.rpm 10 MB/s | 181 kB 00:00 2020-11-20T11:29:48.757 INFO:teuthology.orchestra.run.smithi155.stdout:(36/133): liboath-2.6.2-3.el8.x86_64.rpm 3.5 MB/s | 59 kB 00:00 2020-11-20T11:29:48.774 INFO:teuthology.orchestra.run.smithi155.stdout:(37/133): libunwind-1.3.1-3.el8.x86_64.rpm 4.4 MB/s | 75 kB 00:00 2020-11-20T11:29:48.790 INFO:teuthology.orchestra.run.smithi155.stdout:(38/133): python3-apipkg-1.5-6.el8.noarch.rpm 1.3 MB/s | 21 kB 00:00 2020-11-20T11:29:48.807 INFO:teuthology.orchestra.run.smithi155.stdout:(39/133): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 2.5 MB/s | 44 kB 00:00 2020-11-20T11:29:48.825 INFO:teuthology.orchestra.run.smithi155.stdout:(40/133): python3-beautifulsoup4-4.6.3-2.el8.1. 10 MB/s | 185 kB 00:00 2020-11-20T11:29:48.841 INFO:teuthology.orchestra.run.smithi155.stdout:(41/133): python3-cachetools-3.1.1-4.el8.noarch 2.0 MB/s | 33 kB 00:00 2020-11-20T11:29:48.858 INFO:teuthology.orchestra.run.smithi155.stdout:(42/133): python3-certifi-2018.10.15-7.el8.noar 998 kB/s | 16 kB 00:00 2020-11-20T11:29:48.876 INFO:teuthology.orchestra.run.smithi155.stdout:(43/133): python3-cheroot-8.2.1-1.el8.noarch.rp 7.9 MB/s | 145 kB 00:00 2020-11-20T11:29:48.909 INFO:teuthology.orchestra.run.smithi155.stdout:(44/133): python3-cherrypy-18.4.0-1.el8.noarch. 11 MB/s | 384 kB 00:00 2020-11-20T11:29:48.926 INFO:teuthology.orchestra.run.smithi155.stdout:(45/133): python3-cssselect-0.9.2-10.el8.noarch 2.4 MB/s | 40 kB 00:00 2020-11-20T11:29:48.959 INFO:teuthology.orchestra.run.smithi155.stdout:(46/133): python3-execnet-1.7.1-1.el8.noarch.rp 11 MB/s | 355 kB 00:00 2020-11-20T11:29:48.976 INFO:teuthology.orchestra.run.smithi155.stdout:(47/133): python3-google-auth-1.1.1-10.el8.noar 4.7 MB/s | 81 kB 00:00 2020-11-20T11:29:48.992 INFO:teuthology.orchestra.run.smithi155.stdout:(48/133): python3-jaraco-6.2-6.el8.noarch.rpm 687 kB/s | 11 kB 00:00 2020-11-20T11:29:49.009 INFO:teuthology.orchestra.run.smithi155.stdout:(49/133): python3-jaraco-functools-2.0-4.el8.no 1.0 MB/s | 18 kB 00:00 2020-11-20T11:29:49.076 INFO:teuthology.orchestra.run.smithi155.stdout:(50/133): python3-kubernetes-11.0.0-2.el8.noarc 17 MB/s | 1.2 MB 00:00 2020-11-20T11:29:49.093 INFO:teuthology.orchestra.run.smithi155.stdout:(51/133): python3-logutils-0.3.5-11.el8.noarch. 2.9 MB/s | 49 kB 00:00 2020-11-20T11:29:49.110 INFO:teuthology.orchestra.run.smithi155.stdout:(52/133): python3-more-itertools-7.2.0-3.el8.no 3.5 MB/s | 59 kB 00:00 2020-11-20T11:29:49.128 INFO:teuthology.orchestra.run.smithi155.stdout:(53/133): python3-pecan-1.3.2-9.el8.noarch.rpm 15 MB/s | 283 kB 00:00 2020-11-20T11:29:49.145 INFO:teuthology.orchestra.run.smithi155.stdout:(54/133): python3-portend-2.6-1.el8.noarch.rpm 957 kB/s | 16 kB 00:00 2020-11-20T11:29:49.161 INFO:teuthology.orchestra.run.smithi155.stdout:(55/133): python3-remoto-1.1.4-4.el8.noarch.rpm 2.8 MB/s | 48 kB 00:00 2020-11-20T11:29:49.178 INFO:teuthology.orchestra.run.smithi155.stdout:(56/133): python3-repoze-lru-0.7-6.el8.noarch.r 2.0 MB/s | 34 kB 00:00 2020-11-20T11:29:49.196 INFO:teuthology.orchestra.run.smithi155.stdout:(57/133): python3-routes-2.4.1-12.el8.noarch.rp 11 MB/s | 196 kB 00:00 2020-11-20T11:29:49.213 INFO:teuthology.orchestra.run.smithi155.stdout:(58/133): python3-rsa-3.4.2-15.el8.noarch.rpm 4.2 MB/s | 73 kB 00:00 2020-11-20T11:29:49.229 INFO:teuthology.orchestra.run.smithi155.stdout:(59/133): python3-simplegeneric-0.8.1-17.el8.no 1.1 MB/s | 19 kB 00:00 2020-11-20T11:29:49.246 INFO:teuthology.orchestra.run.smithi155.stdout:(60/133): python3-singledispatch-3.4.0.3-18.el8 1.4 MB/s | 24 kB 00:00 2020-11-20T11:29:49.271 INFO:teuthology.orchestra.run.smithi155.stdout:(61/133): ceph-mgr-diskprediction-local-16.0.0- 6.4 MB/s | 8.6 MB 00:01 2020-11-20T11:29:49.271 INFO:teuthology.orchestra.run.smithi155.stdout:(62/133): python3-tempora-1.14.1-5.el8.noarch.r 1.1 MB/s | 29 kB 00:00 2020-11-20T11:29:49.290 INFO:teuthology.orchestra.run.smithi155.stdout:(63/133): python3-waitress-1.2.1-2.el8.1.noarch 13 MB/s | 241 kB 00:00 2020-11-20T11:29:49.308 INFO:teuthology.orchestra.run.smithi155.stdout:(64/133): python3-webob-1.8.5-1.el8.1.noarch.rp 14 MB/s | 251 kB 00:00 2020-11-20T11:29:49.325 INFO:teuthology.orchestra.run.smithi155.stdout:(65/133): python3-websocket-client-0.56.0-5.el8 3.6 MB/s | 61 kB 00:00 2020-11-20T11:29:49.329 INFO:teuthology.orchestra.run.smithi155.stdout:(66/133): python3-trustme-0.6.0-4.el8.noarch.rp 471 kB/s | 27 kB 00:00 2020-11-20T11:29:49.341 INFO:teuthology.orchestra.run.smithi155.stdout:(67/133): python3-webtest-2.0.33-1.el8.noarch.r 5.0 MB/s | 85 kB 00:00 2020-11-20T11:29:49.349 INFO:teuthology.orchestra.run.smithi155.stdout:(68/133): python3-zc-lockfile-2.0-2.el8.noarch. 1.1 MB/s | 23 kB 00:00 2020-11-20T11:29:49.360 INFO:teuthology.orchestra.run.smithi155.stdout:(69/133): xmlstarlet-1.6.1-11.el8.x86_64.rpm 3.7 MB/s | 69 kB 00:00 2020-11-20T11:29:49.389 INFO:teuthology.orchestra.run.smithi155.stdout:(70/133): automake-1.16.1-6.el8.noarch.rpm 18 MB/s | 713 kB 00:00 2020-11-20T11:29:49.407 INFO:teuthology.orchestra.run.smithi155.stdout:(71/133): python3-webencodings-0.5.1-6.el8.noar 577 kB/s | 27 kB 00:00 2020-11-20T11:29:49.427 INFO:teuthology.orchestra.run.smithi155.stdout:(72/133): python3-lxml-4.2.3-1.el8.x86_64.rpm 45 MB/s | 1.5 MB 00:00 2020-11-20T11:29:49.428 INFO:teuthology.orchestra.run.smithi155.stdout:(73/133): jq-1.5-12.el8.x86_64.rpm 7.2 MB/s | 161 kB 00:00 2020-11-20T11:29:49.452 INFO:teuthology.orchestra.run.smithi155.stdout:(74/133): python3-pyOpenSSL-18.0.0-1.el8.noarch 4.1 MB/s | 103 kB 00:00 2020-11-20T11:29:49.461 INFO:teuthology.orchestra.run.smithi155.stdout:(75/133): python3-werkzeug-0.12.2-4.el8.noarch. 14 MB/s | 457 kB 00:00 2020-11-20T11:29:49.474 INFO:teuthology.orchestra.run.smithi155.stdout:(76/133): python3-mako-1.0.6-13.el8.noarch.rpm 6.8 MB/s | 157 kB 00:00 2020-11-20T11:29:49.485 INFO:teuthology.orchestra.run.smithi155.stdout:(77/133): python3-jsonpatch-1.21-2.el8.noarch.r 1.3 MB/s | 27 kB 00:00 2020-11-20T11:29:49.503 INFO:teuthology.orchestra.run.smithi155.stdout:(78/133): python3-jsonpointer-1.10-11.el8.noarc 711 kB/s | 20 kB 00:00 2020-11-20T11:29:49.511 INFO:teuthology.orchestra.run.smithi155.stdout:(79/133): python3-html5lib-0.999999999-6.el8.no 8.2 MB/s | 214 kB 00:00 2020-11-20T11:29:49.528 INFO:teuthology.orchestra.run.smithi155.stdout:(80/133): python3-pyasn1-0.3.7-6.el8.noarch.rpm 4.9 MB/s | 126 kB 00:00 2020-11-20T11:29:49.535 INFO:teuthology.orchestra.run.smithi155.stdout:(81/133): python3-pytz-2017.2-9.el8.noarch.rpm 2.2 MB/s | 54 kB 00:00 2020-11-20T11:29:49.553 INFO:teuthology.orchestra.run.smithi155.stdout:(82/133): python3-pyasn1-modules-0.3.7-6.el8.no 4.3 MB/s | 110 kB 00:00 2020-11-20T11:29:49.567 INFO:teuthology.orchestra.run.smithi155.stdout:(83/133): python3-prettytable-0.7.2-14.el8.noar 1.3 MB/s | 44 kB 00:00 2020-11-20T11:29:49.575 INFO:teuthology.orchestra.run.smithi155.stdout:(84/133): python3-markupsafe-0.23-19.el8.x86_64 1.7 MB/s | 39 kB 00:00 2020-11-20T11:29:49.595 INFO:teuthology.orchestra.run.smithi155.stdout:(85/133): flex-2.6.1-9.el8.x86_64.rpm 11 MB/s | 320 kB 00:00 2020-11-20T11:29:49.604 INFO:teuthology.orchestra.run.smithi155.stdout:(86/133): librdkafka-0.11.4-1.el8.x86_64.rpm 12 MB/s | 353 kB 00:00 2020-11-20T11:29:49.623 INFO:teuthology.orchestra.run.smithi155.stdout:(87/133): bison-3.0.4-10.el8.x86_64.rpm 24 MB/s | 688 kB 00:00 2020-11-20T11:29:49.639 INFO:teuthology.orchestra.run.smithi155.stdout:(88/133): libtool-2.4.6-25.el8.x86_64.rpm 23 MB/s | 709 kB 00:00 2020-11-20T11:29:49.654 INFO:teuthology.orchestra.run.smithi155.stdout:(89/133): python3-jinja2-2.10.1-2.el8_0.noarch. 17 MB/s | 538 kB 00:00 2020-11-20T11:29:49.680 INFO:teuthology.orchestra.run.smithi155.stdout:(90/133): python3-numpy-f2py-1.14.3-9.el8.x86_6 8.7 MB/s | 225 kB 00:00 2020-11-20T11:29:49.742 INFO:teuthology.orchestra.run.smithi155.stdout:(91/133): python3-babel-2.5.1-5.el8.noarch.rpm 77 MB/s | 4.8 MB 00:00 2020-11-20T11:29:49.764 INFO:teuthology.orchestra.run.smithi155.stdout:(92/133): lttng-ust-2.8.1-11.el8.x86_64.rpm 12 MB/s | 259 kB 00:00 2020-11-20T11:29:49.866 INFO:teuthology.orchestra.run.smithi155.stdout:(93/133): python3-scipy-1.0.0-20.module+el8.1.0 64 MB/s | 15 MB 00:00 2020-11-20T11:29:49.916 INFO:teuthology.orchestra.run.smithi155.stdout:(94/133): python3-numpy-1.14.3-9.el8.x86_64.rpm 25 MB/s | 3.7 MB 00:00 2020-11-20T11:29:49.945 INFO:teuthology.orchestra.run.smithi155.stdout:(95/133): openblas-threads-0.3.3-5.el8.x86_64.r 59 MB/s | 4.4 MB 00:00 2020-11-20T11:29:49.971 INFO:teuthology.orchestra.run.smithi155.stdout:(96/133): socat-1.7.3.3-2.el8.x86_64.rpm 13 MB/s | 302 kB 00:00 2020-11-20T11:29:50.481 INFO:teuthology.orchestra.run.smithi155.stdout:(97/133): openblas-0.3.3-5.el8.x86_64.rpm 61 MB/s | 4.3 MB 00:00 2020-11-20T11:29:50.482 INFO:teuthology.orchestra.run.smithi155.stdout:(98/133): oniguruma-6.8.2-2.el8.x86_64.rpm 5.9 MB/s | 187 kB 00:00 2020-11-20T11:29:50.482 INFO:teuthology.orchestra.run.smithi155.stdout:(99/133): python3-cffi-1.11.5-5.el8.x86_64.rpm 8.0 MB/s | 238 kB 00:00 2020-11-20T11:29:50.482 INFO:teuthology.orchestra.run.smithi155.stdout:(100/133): libverto-devel-0.3.0-5.el8.x86_64.rp 672 kB/s | 18 kB 00:00 2020-11-20T11:29:50.482 INFO:teuthology.orchestra.run.smithi155.stdout:(101/133): python3-asn1crypto-0.24.0-3.el8.noar 6.9 MB/s | 181 kB 00:00 2020-11-20T11:29:50.483 INFO:teuthology.orchestra.run.smithi155.stdout:(102/133): python3-requests-oauthlib-1.0.0-1.el 1.6 MB/s | 43 kB 00:00 2020-11-20T11:29:50.483 INFO:teuthology.orchestra.run.smithi155.stdout:(103/133): fuse-2.9.7-12.el8.x86_64.rpm 4.1 MB/s | 83 kB 00:00 2020-11-20T11:29:50.483 INFO:teuthology.orchestra.run.smithi155.stdout:(104/133): python3-pycparser-2.14-14.el8.noarch 4.3 MB/s | 109 kB 00:00 2020-11-20T11:29:50.483 INFO:teuthology.orchestra.run.smithi155.stdout:(105/133): keyutils-libs-devel-1.5.10-6.el8.x86 2.4 MB/s | 48 kB 00:00 2020-11-20T11:29:50.483 INFO:teuthology.orchestra.run.smithi155.stdout:(106/133): python3-jwt-1.6.1-2.el8.noarch.rpm 1.8 MB/s | 43 kB 00:00 2020-11-20T11:29:50.484 INFO:teuthology.orchestra.run.smithi155.stdout:(107/133): python3-oauthlib-2.1.0-1.el8.noarch. 6.2 MB/s | 155 kB 00:00 2020-11-20T11:29:50.484 INFO:teuthology.orchestra.run.smithi155.stdout:(108/133): fuse-common-3.2.1-12.el8.x86_64.rpm 890 kB/s | 21 kB 00:00 2020-11-20T11:29:50.484 INFO:teuthology.orchestra.run.smithi155.stdout:(109/133): libsepol-devel-2.9-1.el8.x86_64.rpm 3.4 MB/s | 86 kB 00:00 2020-11-20T11:29:50.484 INFO:teuthology.orchestra.run.smithi155.stdout:(110/133): libaio-devel-0.3.112-1.el8.x86_64.rp 749 kB/s | 19 kB 00:00 2020-11-20T11:29:50.484 INFO:teuthology.orchestra.run.smithi155.stdout:(111/133): libselinux-devel-2.9-3.el8.x86_64.rp 7.8 MB/s | 199 kB 00:00 2020-11-20T11:29:50.485 INFO:teuthology.orchestra.run.smithi155.stdout:(112/133): python3-cryptography-2.3-3.el8.x86_6 18 MB/s | 511 kB 00:00 2020-11-20T11:29:50.485 INFO:teuthology.orchestra.run.smithi155.stdout:(113/133): zlib-devel-1.2.11-16.el8_2.x86_64.rp 2.8 MB/s | 57 kB 00:00 2020-11-20T11:29:50.485 INFO:teuthology.orchestra.run.smithi155.stdout:(114/133): libuuid-devel-2.32.1-24.el8.x86_64.r 4.8 MB/s | 96 kB 00:00 2020-11-20T11:29:50.485 INFO:teuthology.orchestra.run.smithi155.stdout:(115/133): libxslt-1.1.32-5.el8.x86_64.rpm 9.7 MB/s | 250 kB 00:00 2020-11-20T11:29:50.485 INFO:teuthology.orchestra.run.smithi155.stdout:(116/133): libcom_err-devel-1.45.6-1.el8.x86_64 1.9 MB/s | 38 kB 00:00 2020-11-20T11:29:50.486 INFO:teuthology.orchestra.run.smithi155.stdout:(117/133): libquadmath-8.3.1-5.1.el8.x86_64.rpm 6.0 MB/s | 169 kB 00:00 2020-11-20T11:29:50.486 INFO:teuthology.orchestra.run.smithi155.stdout:(118/133): pcre2-devel-10.32-2.el8.x86_64.rpm 19 MB/s | 605 kB 00:00 2020-11-20T11:29:50.486 INFO:teuthology.orchestra.run.smithi155.stdout:(119/133): libgfortran-8.3.1-5.1.el8.x86_64.rpm 21 MB/s | 639 kB 00:00 2020-11-20T11:29:50.486 INFO:teuthology.orchestra.run.smithi155.stdout:(120/133): elfutils-libelf-devel-0.180-1.el8.x8 2.6 MB/s | 58 kB 00:00 2020-11-20T11:29:50.486 INFO:teuthology.orchestra.run.smithi155.stdout:(121/133): pcre2-utf32-10.32-2.el8.x86_64.rpm 8.5 MB/s | 220 kB 00:00 2020-11-20T11:29:50.487 INFO:teuthology.orchestra.run.smithi155.stdout:(122/133): pcre2-utf16-10.32-2.el8.x86_64.rpm 8.4 MB/s | 229 kB 00:00 2020-11-20T11:29:50.487 INFO:teuthology.orchestra.run.smithi155.stdout:(123/133): krb5-devel-1.18.2-5.el8.x86_64.rpm 20 MB/s | 558 kB 00:00 2020-11-20T11:29:50.487 INFO:teuthology.orchestra.run.smithi155.stdout:(124/133): xfsprogs-devel-5.0.0-4.el8.x86_64.rp 2.7 MB/s | 69 kB 00:00 2020-11-20T11:29:50.487 INFO:teuthology.orchestra.run.smithi155.stdout:(125/133): librabbitmq-0.9.0-2.el8.x86_64.rpm 2.0 MB/s | 47 kB 00:00 2020-11-20T11:29:50.487 INFO:teuthology.orchestra.run.smithi155.stdout:(126/133): openssl-devel-1.1.1g-11.el8.x86_64.r 62 MB/s | 2.3 MB 00:00 2020-11-20T11:29:50.881 INFO:teuthology.orchestra.run.smithi155.stdout:(127/133): librbd1-16.0.0-7397.gb86e691b.el8.x8 4.7 MB/s | 2.4 MB 00:00 2020-11-20T11:29:50.914 INFO:teuthology.orchestra.run.smithi155.stdout:(128/133): NetworkManager-libnm-1.26.0-9.el8_3. 54 MB/s | 1.7 MB 00:00 2020-11-20T11:29:50.972 INFO:teuthology.orchestra.run.smithi155.stdout:(129/133): NetworkManager-tui-1.26.0-9.el8_3.x8 5.4 MB/s | 320 kB 00:00 2020-11-20T11:29:50.992 INFO:teuthology.orchestra.run.smithi155.stdout:(130/133): NetworkManager-team-1.26.0-9.el8_3.x 6.9 MB/s | 141 kB 00:00 2020-11-20T11:29:51.029 INFO:teuthology.orchestra.run.smithi155.stdout:(131/133): NetworkManager-1.26.0-9.el8_3.x86_64 66 MB/s | 2.4 MB 00:00 2020-11-20T11:29:51.123 INFO:teuthology.orchestra.run.smithi155.stdout:(132/133): librados2-16.0.0-7397.gb86e691b.el8. 4.8 MB/s | 3.7 MB 00:00 2020-11-20T11:29:51.145 INFO:teuthology.orchestra.run.smithi023.stdout:(8/133): ceph-common-16.0.0-7397.gb86e691b.el8. 2.6 MB/s | 23 MB 00:08 2020-11-20T11:29:51.187 INFO:teuthology.orchestra.run.smithi023.stdout:(9/133): ceph-selinux-16.0.0-7397.gb86e691b.el8 606 kB/s | 25 kB 00:00 2020-11-20T11:29:51.237 INFO:teuthology.orchestra.run.smithi023.stdout:(10/133): ceph-radosgw-16.0.0-7397.gb86e691b.el 2.9 MB/s | 12 MB 00:04 2020-11-20T11:29:51.264 INFO:teuthology.orchestra.run.smithi023.stdout:(11/133): libcephfs-devel-16.0.0-7397.gb86e691b 869 kB/s | 24 kB 00:00 2020-11-20T11:29:51.407 INFO:teuthology.orchestra.run.smithi023.stdout:(12/133): libcephfs2-16.0.0-7397.gb86e691b.el8. 3.9 MB/s | 562 kB 00:00 2020-11-20T11:29:51.444 INFO:teuthology.orchestra.run.smithi023.stdout:(13/133): librados-devel-16.0.0-7397.gb86e691b. 3.5 MB/s | 131 kB 00:00 2020-11-20T11:29:51.559 INFO:teuthology.orchestra.run.smithi023.stdout:(14/133): libradosstriper1-16.0.0-7397.gb86e691 4.1 MB/s | 486 kB 00:00 2020-11-20T11:29:52.242 INFO:teuthology.orchestra.run.smithi023.stdout:(15/133): librgw2-16.0.0-7397.gb86e691b.el8.x86 5.4 MB/s | 3.7 MB 00:00 2020-11-20T11:29:52.270 INFO:teuthology.orchestra.run.smithi023.stdout:(16/133): python3-ceph-argparse-16.0.0-7397.gb8 1.4 MB/s | 40 kB 00:00 2020-11-20T11:29:52.300 INFO:teuthology.orchestra.run.smithi023.stdout:(17/133): python3-ceph-common-16.0.0-7397.gb86e 2.4 MB/s | 73 kB 00:00 2020-11-20T11:29:52.335 INFO:teuthology.orchestra.run.smithi023.stdout:(18/133): python3-cephfs-16.0.0-7397.gb86e691b. 5.9 MB/s | 204 kB 00:00 2020-11-20T11:29:52.402 INFO:teuthology.orchestra.run.smithi023.stdout:(19/133): python3-rados-16.0.0-7397.gb86e691b.e 5.4 MB/s | 375 kB 00:00 2020-11-20T11:29:52.469 INFO:teuthology.orchestra.run.smithi023.stdout:(20/133): python3-rbd-16.0.0-7397.gb86e691b.el8 5.2 MB/s | 353 kB 00:00 2020-11-20T11:29:52.500 INFO:teuthology.orchestra.run.smithi023.stdout:(21/133): python3-rgw-16.0.0-7397.gb86e691b.el8 3.5 MB/s | 113 kB 00:00 2020-11-20T11:29:52.531 INFO:teuthology.orchestra.run.smithi023.stdout:(22/133): rbd-fuse-16.0.0-7397.gb86e691b.el8.x8 2.9 MB/s | 88 kB 00:00 2020-11-20T11:29:52.793 INFO:teuthology.orchestra.run.smithi023.stdout:(23/133): ceph-osd-16.0.0-7397.gb86e691b.el8.x8 2.5 MB/s | 18 MB 00:07 2020-11-20T11:29:52.823 INFO:teuthology.orchestra.run.smithi023.stdout:(24/133): rbd-nbd-16.0.0-7397.gb86e691b.el8.x86 5.4 MB/s | 171 kB 00:00 2020-11-20T11:29:52.847 INFO:teuthology.orchestra.run.smithi023.stdout:(25/133): ceph-grafana-dashboards-16.0.0-7397.g 1.0 MB/s | 24 kB 00:00 2020-11-20T11:29:52.871 INFO:teuthology.orchestra.run.smithi023.stdout:(26/133): ceph-mgr-cephadm-16.0.0-7397.gb86e691 2.7 MB/s | 66 kB 00:00 2020-11-20T11:29:53.010 INFO:teuthology.orchestra.run.smithi023.stdout:(27/133): rbd-mirror-16.0.0-7397.gb86e691b.el8. 6.6 MB/s | 3.2 MB 00:00 2020-11-20T11:29:53.078 INFO:teuthology.orchestra.run.smithi155.stdout:(133/133): ceph-test-16.0.0-7397.gb86e691b.el8. 7.1 MB/s | 48 MB 00:06 2020-11-20T11:29:53.082 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2020-11-20T11:29:53.083 INFO:teuthology.orchestra.run.smithi155.stdout:Total 16 MB/s | 199 MB 00:12 2020-11-20T11:29:53.147 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2020-11-20T11:29:53.281 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2020-11-20T11:29:53.282 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2020-11-20T11:29:53.471 INFO:teuthology.orchestra.run.smithi023.stdout:(28/133): ceph-mgr-dashboard-16.0.0-7397.gb86e6 5.3 MB/s | 3.2 MB 00:00 2020-11-20T11:29:53.553 INFO:teuthology.orchestra.run.smithi023.stdout:(29/133): ceph-mgr-modules-core-16.0.0-7397.gb8 2.4 MB/s | 198 kB 00:00 2020-11-20T11:29:53.593 INFO:teuthology.orchestra.run.smithi023.stdout:(30/133): ceph-mgr-rook-16.0.0-7397.gb86e691b.e 675 kB/s | 27 kB 00:00 2020-11-20T11:29:53.634 INFO:teuthology.orchestra.run.smithi023.stdout:(31/133): ceph-prometheus-alerts-16.0.0-7397.gb 218 kB/s | 8.8 kB 00:00 2020-11-20T11:29:53.677 INFO:teuthology.orchestra.run.smithi023.stdout:(32/133): cephadm-16.0.0-7397.gb86e691b.el8.noa 1.3 MB/s | 57 kB 00:00 2020-11-20T11:29:53.775 INFO:teuthology.orchestra.run.smithi023.stdout:(33/133): fmt-6.2.1-1.el8.x86_64.rpm 1.2 MB/s | 116 kB 00:00 2020-11-20T11:29:53.815 INFO:teuthology.orchestra.run.smithi023.stdout:(34/133): gperftools-libs-2.7-6.el8.x86_64.rpm 7.7 MB/s | 315 kB 00:00 2020-11-20T11:29:53.836 INFO:teuthology.orchestra.run.smithi023.stdout:(35/133): leveldb-1.22-1.el8.x86_64.rpm 8.5 MB/s | 181 kB 00:00 2020-11-20T11:29:53.856 INFO:teuthology.orchestra.run.smithi023.stdout:(36/133): liboath-2.6.2-3.el8.x86_64.rpm 2.9 MB/s | 59 kB 00:00 2020-11-20T11:29:53.877 INFO:teuthology.orchestra.run.smithi023.stdout:(37/133): libunwind-1.3.1-3.el8.x86_64.rpm 3.6 MB/s | 75 kB 00:00 2020-11-20T11:29:53.897 INFO:teuthology.orchestra.run.smithi023.stdout:(38/133): python3-apipkg-1.5-6.el8.noarch.rpm 1.1 MB/s | 21 kB 00:00 2020-11-20T11:29:53.918 INFO:teuthology.orchestra.run.smithi023.stdout:(39/133): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 2.1 MB/s | 44 kB 00:00 2020-11-20T11:29:53.939 INFO:teuthology.orchestra.run.smithi023.stdout:(40/133): python3-beautifulsoup4-4.6.3-2.el8.1. 8.7 MB/s | 185 kB 00:00 2020-11-20T11:29:53.958 INFO:teuthology.orchestra.run.smithi023.stdout:(41/133): python3-cachetools-3.1.1-4.el8.noarch 1.6 MB/s | 33 kB 00:00 2020-11-20T11:29:53.978 INFO:teuthology.orchestra.run.smithi023.stdout:(42/133): python3-certifi-2018.10.15-7.el8.noar 833 kB/s | 16 kB 00:00 2020-11-20T11:29:53.999 INFO:teuthology.orchestra.run.smithi023.stdout:(43/133): python3-cheroot-8.2.1-1.el8.noarch.rp 6.9 MB/s | 145 kB 00:00 2020-11-20T11:29:54.039 INFO:teuthology.orchestra.run.smithi023.stdout:(44/133): python3-cherrypy-18.4.0-1.el8.noarch. 9.5 MB/s | 384 kB 00:00 2020-11-20T11:29:54.059 INFO:teuthology.orchestra.run.smithi023.stdout:(45/133): python3-cssselect-0.9.2-10.el8.noarch 2.0 MB/s | 40 kB 00:00 2020-11-20T11:29:54.098 INFO:teuthology.orchestra.run.smithi023.stdout:(46/133): python3-execnet-1.7.1-1.el8.noarch.rp 8.8 MB/s | 355 kB 00:00 2020-11-20T11:29:54.119 INFO:teuthology.orchestra.run.smithi023.stdout:(47/133): python3-google-auth-1.1.1-10.el8.noar 3.9 MB/s | 81 kB 00:00 2020-11-20T11:29:54.139 INFO:teuthology.orchestra.run.smithi023.stdout:(48/133): python3-jaraco-6.2-6.el8.noarch.rpm 540 kB/s | 11 kB 00:00 2020-11-20T11:29:54.159 INFO:teuthology.orchestra.run.smithi023.stdout:(49/133): python3-jaraco-functools-2.0-4.el8.no 897 kB/s | 18 kB 00:00 2020-11-20T11:29:54.240 INFO:teuthology.orchestra.run.smithi023.stdout:(50/133): python3-kubernetes-11.0.0-2.el8.noarc 14 MB/s | 1.2 MB 00:00 2020-11-20T11:29:54.260 INFO:teuthology.orchestra.run.smithi023.stdout:(51/133): python3-logutils-0.3.5-11.el8.noarch. 2.4 MB/s | 49 kB 00:00 2020-11-20T11:29:54.280 INFO:teuthology.orchestra.run.smithi023.stdout:(52/133): python3-more-itertools-7.2.0-3.el8.no 2.9 MB/s | 59 kB 00:00 2020-11-20T11:29:54.301 INFO:teuthology.orchestra.run.smithi023.stdout:(53/133): python3-pecan-1.3.2-9.el8.noarch.rpm 13 MB/s | 283 kB 00:00 2020-11-20T11:29:54.322 INFO:teuthology.orchestra.run.smithi023.stdout:(54/133): python3-portend-2.6-1.el8.noarch.rpm 790 kB/s | 16 kB 00:00 2020-11-20T11:29:54.341 INFO:teuthology.orchestra.run.smithi023.stdout:(55/133): python3-remoto-1.1.4-4.el8.noarch.rpm 2.4 MB/s | 48 kB 00:00 2020-11-20T11:29:54.361 INFO:teuthology.orchestra.run.smithi023.stdout:(56/133): python3-repoze-lru-0.7-6.el8.noarch.r 1.6 MB/s | 34 kB 00:00 2020-11-20T11:29:54.383 INFO:teuthology.orchestra.run.smithi023.stdout:(57/133): python3-routes-2.4.1-12.el8.noarch.rp 9.1 MB/s | 196 kB 00:00 2020-11-20T11:29:54.403 INFO:teuthology.orchestra.run.smithi023.stdout:(58/133): python3-rsa-3.4.2-15.el8.noarch.rpm 3.6 MB/s | 73 kB 00:00 2020-11-20T11:29:54.422 INFO:teuthology.orchestra.run.smithi023.stdout:(59/133): python3-simplegeneric-0.8.1-17.el8.no 952 kB/s | 19 kB 00:00 2020-11-20T11:29:54.442 INFO:teuthology.orchestra.run.smithi023.stdout:(60/133): python3-singledispatch-3.4.0.3-18.el8 1.2 MB/s | 24 kB 00:00 2020-11-20T11:29:54.462 INFO:teuthology.orchestra.run.smithi023.stdout:(61/133): python3-tempora-1.14.1-5.el8.noarch.r 1.4 MB/s | 29 kB 00:00 2020-11-20T11:29:54.482 INFO:teuthology.orchestra.run.smithi023.stdout:(62/133): python3-trustme-0.6.0-4.el8.noarch.rp 1.4 MB/s | 27 kB 00:00 2020-11-20T11:29:54.504 INFO:teuthology.orchestra.run.smithi023.stdout:(63/133): python3-waitress-1.2.1-2.el8.1.noarch 11 MB/s | 241 kB 00:00 2020-11-20T11:29:54.525 INFO:teuthology.orchestra.run.smithi023.stdout:(64/133): python3-webob-1.8.5-1.el8.1.noarch.rp 12 MB/s | 251 kB 00:00 2020-11-20T11:29:54.545 INFO:teuthology.orchestra.run.smithi023.stdout:(65/133): python3-websocket-client-0.56.0-5.el8 3.0 MB/s | 61 kB 00:00 2020-11-20T11:29:54.566 INFO:teuthology.orchestra.run.smithi023.stdout:(66/133): python3-webtest-2.0.33-1.el8.noarch.r 4.1 MB/s | 85 kB 00:00 2020-11-20T11:29:54.585 INFO:teuthology.orchestra.run.smithi023.stdout:(67/133): python3-zc-lockfile-2.0-2.el8.noarch. 1.1 MB/s | 23 kB 00:00 2020-11-20T11:29:54.605 INFO:teuthology.orchestra.run.smithi023.stdout:(68/133): xmlstarlet-1.6.1-11.el8.x86_64.rpm 3.4 MB/s | 69 kB 00:00 2020-11-20T11:29:54.646 INFO:teuthology.orchestra.run.smithi023.stdout:(69/133): python3-cffi-1.11.5-5.el8.x86_64.rpm 5.8 MB/s | 238 kB 00:00 2020-11-20T11:29:54.672 INFO:teuthology.orchestra.run.smithi023.stdout:(70/133): libverto-devel-0.3.0-5.el8.x86_64.rpm 691 kB/s | 18 kB 00:00 2020-11-20T11:29:54.698 INFO:teuthology.orchestra.run.smithi023.stdout:(71/133): python3-asn1crypto-0.24.0-3.el8.noarc 6.7 MB/s | 181 kB 00:00 2020-11-20T11:29:54.727 INFO:teuthology.orchestra.run.smithi023.stdout:(72/133): python3-requests-oauthlib-1.0.0-1.el8 1.6 MB/s | 43 kB 00:00 2020-11-20T11:29:54.751 INFO:teuthology.orchestra.run.smithi023.stdout:(73/133): fuse-2.9.7-12.el8.x86_64.rpm 3.1 MB/s | 83 kB 00:00 2020-11-20T11:29:54.777 INFO:teuthology.orchestra.run.smithi023.stdout:(74/133): python3-pycparser-2.14-14.el8.noarch. 4.2 MB/s | 109 kB 00:00 2020-11-20T11:29:54.823 INFO:teuthology.orchestra.run.smithi023.stdout:(75/133): ceph-mgr-diskprediction-local-16.0.0- 4.7 MB/s | 8.6 MB 00:01 2020-11-20T11:29:54.824 INFO:teuthology.orchestra.run.smithi023.stdout:(76/133): keyutils-libs-devel-1.5.10-6.el8.x86_ 1.0 MB/s | 48 kB 00:00 2020-11-20T11:29:54.866 INFO:teuthology.orchestra.run.smithi023.stdout:(77/133): python3-oauthlib-2.1.0-1.el8.noarch.r 3.7 MB/s | 155 kB 00:00 2020-11-20T11:29:54.876 INFO:teuthology.orchestra.run.smithi023.stdout:(78/133): python3-jwt-1.6.1-2.el8.noarch.rpm 818 kB/s | 43 kB 00:00 2020-11-20T11:29:54.892 INFO:teuthology.orchestra.run.smithi023.stdout:(79/133): fuse-common-3.2.1-12.el8.x86_64.rpm 814 kB/s | 21 kB 00:00 2020-11-20T11:29:54.907 INFO:teuthology.orchestra.run.smithi023.stdout:(80/133): libsepol-devel-2.9-1.el8.x86_64.rpm 2.6 MB/s | 86 kB 00:00 2020-11-20T11:29:54.914 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2020-11-20T11:29:54.918 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2020-11-20T11:29:54.923 INFO:teuthology.orchestra.run.smithi023.stdout:(81/133): libaio-devel-0.3.112-1.el8.x86_64.rpm 594 kB/s | 19 kB 00:00 2020-11-20T11:29:54.949 INFO:teuthology.orchestra.run.smithi023.stdout:(82/133): libselinux-devel-2.9-3.el8.x86_64.rpm 4.7 MB/s | 199 kB 00:00 2020-11-20T11:29:54.962 INFO:teuthology.orchestra.run.smithi023.stdout:(83/133): python3-cryptography-2.3-3.el8.x86_64 13 MB/s | 511 kB 00:00 2020-11-20T11:29:54.978 INFO:teuthology.orchestra.run.smithi023.stdout:(84/133): zlib-devel-1.2.11-16.el8_2.x86_64.rpm 1.9 MB/s | 57 kB 00:00 2020-11-20T11:29:55.004 INFO:teuthology.orchestra.run.smithi023.stdout:(85/133): libuuid-devel-2.32.1-24.el8.x86_64.rp 2.6 MB/s | 96 kB 00:00 2020-11-20T11:29:55.016 INFO:teuthology.orchestra.run.smithi023.stdout:(86/133): libxslt-1.1.32-5.el8.x86_64.rpm 6.4 MB/s | 250 kB 00:00 2020-11-20T11:29:55.033 INFO:teuthology.orchestra.run.smithi023.stdout:(87/133): libcom_err-devel-1.45.6-1.el8.x86_64. 1.3 MB/s | 38 kB 00:00 2020-11-20T11:29:55.050 INFO:teuthology.orchestra.run.smithi023.stdout:(88/133): libquadmath-8.3.1-5.1.el8.x86_64.rpm 4.9 MB/s | 169 kB 00:00 2020-11-20T11:29:55.071 INFO:teuthology.orchestra.run.smithi023.stdout:(89/133): pcre2-devel-10.32-2.el8.x86_64.rpm 15 MB/s | 605 kB 00:00 2020-11-20T11:29:55.099 INFO:teuthology.orchestra.run.smithi023.stdout:(90/133): libgfortran-8.3.1-5.1.el8.x86_64.rpm 13 MB/s | 639 kB 00:00 2020-11-20T11:29:55.108 INFO:teuthology.orchestra.run.smithi023.stdout:(91/133): elfutils-libelf-devel-0.180-1.el8.x86 1.8 MB/s | 58 kB 00:00 2020-11-20T11:29:55.127 INFO:teuthology.orchestra.run.smithi023.stdout:(92/133): pcre2-utf32-10.32-2.el8.x86_64.rpm 7.9 MB/s | 220 kB 00:00 2020-11-20T11:29:55.145 INFO:teuthology.orchestra.run.smithi023.stdout:(93/133): pcre2-utf16-10.32-2.el8.x86_64.rpm 6.2 MB/s | 229 kB 00:00 2020-11-20T11:29:55.171 INFO:teuthology.orchestra.run.smithi023.stdout:(94/133): krb5-devel-1.18.2-5.el8.x86_64.rpm 12 MB/s | 558 kB 00:00 2020-11-20T11:29:55.175 INFO:teuthology.orchestra.run.smithi023.stdout:(95/133): xfsprogs-devel-5.0.0-4.el8.x86_64.rpm 2.2 MB/s | 69 kB 00:00 2020-11-20T11:29:55.208 INFO:teuthology.orchestra.run.smithi023.stdout:(96/133): librabbitmq-0.9.0-2.el8.x86_64.rpm 1.2 MB/s | 47 kB 00:00 2020-11-20T11:29:55.243 INFO:teuthology.orchestra.run.smithi023.stdout:(97/133): openssl-devel-1.1.1g-11.el8.x86_64.rp 34 MB/s | 2.3 MB 00:00 2020-11-20T11:29:55.256 INFO:teuthology.orchestra.run.smithi023.stdout:(98/133): automake-1.16.1-6.el8.noarch.rpm 15 MB/s | 713 kB 00:00 2020-11-20T11:29:55.273 INFO:teuthology.orchestra.run.smithi023.stdout:(99/133): python3-webencodings-0.5.1-6.el8.noar 898 kB/s | 27 kB 00:00 2020-11-20T11:29:55.310 INFO:teuthology.orchestra.run.smithi023.stdout:(100/133): python3-lxml-4.2.3-1.el8.x86_64.rpm 28 MB/s | 1.5 MB 00:00 2020-11-20T11:29:55.311 INFO:teuthology.orchestra.run.smithi023.stdout:(101/133): jq-1.5-12.el8.x86_64.rpm 4.2 MB/s | 161 kB 00:00 2020-11-20T11:29:55.345 INFO:teuthology.orchestra.run.smithi023.stdout:(102/133): python3-pyOpenSSL-18.0.0-1.el8.noarc 2.9 MB/s | 103 kB 00:00 2020-11-20T11:29:55.364 INFO:teuthology.orchestra.run.smithi023.stdout:(103/133): python3-werkzeug-0.12.2-4.el8.noarch 8.5 MB/s | 457 kB 00:00 2020-11-20T11:29:55.375 INFO:teuthology.orchestra.run.smithi023.stdout:(104/133): python3-mako-1.0.6-13.el8.noarch.rpm 5.1 MB/s | 157 kB 00:00 2020-11-20T11:29:55.390 INFO:teuthology.orchestra.run.smithi023.stdout:(105/133): python3-jsonpatch-1.21-2.el8.noarch. 1.0 MB/s | 27 kB 00:00 2020-11-20T11:29:55.406 INFO:teuthology.orchestra.run.smithi023.stdout:(106/133): python3-jsonpointer-1.10-11.el8.noar 646 kB/s | 20 kB 00:00 2020-11-20T11:29:55.424 INFO:teuthology.orchestra.run.smithi023.stdout:(107/133): python3-html5lib-0.999999999-6.el8.n 6.3 MB/s | 214 kB 00:00 2020-11-20T11:29:55.441 INFO:teuthology.orchestra.run.smithi023.stdout:(108/133): python3-pyasn1-0.3.7-6.el8.noarch.rp 3.5 MB/s | 126 kB 00:00 2020-11-20T11:29:55.457 INFO:teuthology.orchestra.run.smithi023.stdout:(109/133): python3-pytz-2017.2-9.el8.noarch.rpm 1.6 MB/s | 54 kB 00:00 2020-11-20T11:29:55.473 INFO:teuthology.orchestra.run.smithi023.stdout:(110/133): python3-pyasn1-modules-0.3.7-6.el8.n 3.4 MB/s | 110 kB 00:00 2020-11-20T11:29:55.490 INFO:teuthology.orchestra.run.smithi023.stdout:(111/133): python3-prettytable-0.7.2-14.el8.noa 1.3 MB/s | 44 kB 00:00 2020-11-20T11:29:55.505 INFO:teuthology.orchestra.run.smithi023.stdout:(112/133): python3-markupsafe-0.23-19.el8.x86_6 1.2 MB/s | 39 kB 00:00 2020-11-20T11:29:55.523 INFO:teuthology.orchestra.run.smithi023.stdout:(113/133): flex-2.6.1-9.el8.x86_64.rpm 9.3 MB/s | 320 kB 00:00 2020-11-20T11:29:55.542 INFO:teuthology.orchestra.run.smithi023.stdout:(114/133): librdkafka-0.11.4-1.el8.x86_64.rpm 9.6 MB/s | 353 kB 00:00 2020-11-20T11:29:55.570 INFO:teuthology.orchestra.run.smithi023.stdout:(115/133): bison-3.0.4-10.el8.x86_64.rpm 15 MB/s | 688 kB 00:00 2020-11-20T11:29:55.592 INFO:teuthology.orchestra.run.smithi023.stdout:(116/133): libtool-2.4.6-25.el8.x86_64.rpm 16 MB/s | 709 kB 00:00 2020-11-20T11:29:55.604 INFO:teuthology.orchestra.run.smithi023.stdout:(117/133): python3-jinja2-2.10.1-2.el8_0.noarch 15 MB/s | 538 kB 00:00 2020-11-20T11:29:55.639 INFO:teuthology.orchestra.run.smithi023.stdout:(118/133): python3-numpy-f2py-1.14.3-9.el8.x86_ 7.9 MB/s | 225 kB 00:00 2020-11-20T11:29:55.709 INFO:teuthology.orchestra.run.smithi023.stdout:(119/133): python3-babel-2.5.1-5.el8.noarch.rpm 69 MB/s | 4.8 MB 00:00 2020-11-20T11:29:55.736 INFO:teuthology.orchestra.run.smithi023.stdout:(120/133): lttng-ust-2.8.1-11.el8.x86_64.rpm 9.4 MB/s | 259 kB 00:00 2020-11-20T11:29:55.864 INFO:teuthology.orchestra.run.smithi023.stdout:(121/133): python3-scipy-1.0.0-20.module+el8.1. 53 MB/s | 15 MB 00:00 2020-11-20T11:29:55.905 INFO:teuthology.orchestra.run.smithi023.stdout:(122/133): python3-numpy-1.14.3-9.el8.x86_64.rp 22 MB/s | 3.7 MB 00:00 2020-11-20T11:29:55.946 INFO:teuthology.orchestra.run.smithi023.stdout:(123/133): openblas-threads-0.3.3-5.el8.x86_64. 53 MB/s | 4.4 MB 00:00 2020-11-20T11:29:55.974 INFO:teuthology.orchestra.run.smithi023.stdout:(124/133): socat-1.7.3.3-2.el8.x86_64.rpm 11 MB/s | 302 kB 00:00 2020-11-20T11:29:56.009 INFO:teuthology.orchestra.run.smithi023.stdout:(125/133): openblas-0.3.3-5.el8.x86_64.rpm 42 MB/s | 4.3 MB 00:00 2020-11-20T11:29:56.010 INFO:teuthology.orchestra.run.smithi023.stdout:(126/133): oniguruma-6.8.2-2.el8.x86_64.rpm 5.1 MB/s | 187 kB 00:00 2020-11-20T11:29:56.333 INFO:teuthology.orchestra.run.smithi023.stdout:(127/133): librbd1-16.0.0-7397.gb86e691b.el8.x8 7.5 MB/s | 2.4 MB 00:00 2020-11-20T11:29:56.372 INFO:teuthology.orchestra.run.smithi023.stdout:(128/133): NetworkManager-libnm-1.26.0-9.el8_3. 45 MB/s | 1.7 MB 00:00 2020-11-20T11:29:56.399 INFO:teuthology.orchestra.run.smithi023.stdout:(129/133): NetworkManager-tui-1.26.0-9.el8_3.x8 11 MB/s | 320 kB 00:00 2020-11-20T11:29:56.425 INFO:teuthology.orchestra.run.smithi023.stdout:(130/133): NetworkManager-team-1.26.0-9.el8_3.x 5.3 MB/s | 141 kB 00:00 2020-11-20T11:29:56.468 INFO:teuthology.orchestra.run.smithi023.stdout:(131/133): NetworkManager-1.26.0-9.el8_3.x86_64 57 MB/s | 2.4 MB 00:00 2020-11-20T11:29:56.570 INFO:teuthology.orchestra.run.smithi023.stdout:(132/133): librados2-16.0.0-7397.gb86e691b.el8. 6.6 MB/s | 3.7 MB 00:00 2020-11-20T11:29:57.565 INFO:teuthology.orchestra.run.smithi023.stdout:(133/133): ceph-test-16.0.0-7397.gb86e691b.el8. 7.5 MB/s | 48 MB 00:06 2020-11-20T11:29:57.570 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2020-11-20T11:29:57.570 INFO:teuthology.orchestra.run.smithi023.stdout:Total 13 MB/s | 199 MB 00:15 2020-11-20T11:29:57.642 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2020-11-20T11:29:57.799 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2020-11-20T11:29:57.800 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2020-11-20T11:29:59.564 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2020-11-20T11:29:59.568 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2020-11-20T11:29:59.876 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2020-11-20T11:29:59.926 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: fmt-6.2.1-1.el8.x86_64 1/1 2020-11-20T11:30:00.098 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/139 2020-11-20T11:30:00.242 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/139 2020-11-20T11:30:00.418 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : leveldb-1.22-1.el8.x86_64 3/139 2020-11-20T11:30:00.609 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librabbitmq-0.9.0-2.el8.x86_64 4/139 2020-11-20T11:30:00.780 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/139 2020-11-20T11:30:01.172 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/139 2020-11-20T11:30:01.347 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librados2-2:16.0.0-7397.gb86e691b.el8.x86_64 6/139 2020-11-20T11:30:01.620 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-2:16.0.0-7397.gb86e691b.el8.x86_64 6/139 2020-11-20T11:30:01.822 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librbd1-2:16.0.0-7397.gb86e691b.el8.x86_64 7/139 2020-11-20T11:30:01.930 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-2:16.0.0-7397.gb86e691b.el8.x86_64 7/139 2020-11-20T11:30:02.130 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs2-2:16.0.0-7397.gb86e691b.el8.x86_64 8/139 2020-11-20T11:30:02.211 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephfs2-2:16.0.0-7397.gb86e691b.el8.x86_64 8/139 2020-11-20T11:30:02.398 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rados-2:16.0.0-7397.gb86e691b.el8.x86_64 9/139 2020-11-20T11:30:02.588 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 10/139 2020-11-20T11:30:02.964 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 10/139 2020-11-20T11:30:03.196 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librgw2-2:16.0.0-7397.gb86e691b.el8.x86_64 11/139 2020-11-20T11:30:03.322 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librgw2-2:16.0.0-7397.gb86e691b.el8.x86_64 11/139 2020-11-20T11:30:03.500 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 12/139 2020-11-20T11:30:03.665 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 13/139 2020-11-20T11:30:03.813 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libradosstriper1-2:16.0.0-7397.gb86e691b.el8.x86 14/139 2020-11-20T11:30:04.191 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libradosstriper1-2:16.0.0-7397.gb86e691b.el8.x86 14/139 2020-11-20T11:30:04.438 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : NetworkManager-libnm-1:1.26.0-9.el8_3.x86_64 15/139 2020-11-20T11:30:04.494 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: NetworkManager-libnm-1:1.26.0-9.el8_3.x86_64 15/139 2020-11-20T11:30:04.773 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: NetworkManager-1:1.26.0-9.el8_3.x86_64 16/139 2020-11-20T11:30:04.776 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2020-11-20T11:30:04.824 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: fmt-6.2.1-1.el8.x86_64 1/1 2020-11-20T11:30:05.020 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/139 2020-11-20T11:30:05.046 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : NetworkManager-1:1.26.0-9.el8_3.x86_64 16/139 2020-11-20T11:30:05.181 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/139 2020-11-20T11:30:05.383 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: NetworkManager-1:1.26.0-9.el8_3.x86_64 16/139 2020-11-20T11:30:05.428 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : leveldb-1.22-1.el8.x86_64 3/139 2020-11-20T11:30:05.571 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libquadmath-8.3.1-5.1.el8.x86_64 17/139 2020-11-20T11:30:05.604 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/139 2020-11-20T11:30:05.710 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libquadmath-8.3.1-5.1.el8.x86_64 17/139 2020-11-20T11:30:05.863 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libgfortran-8.3.1-5.1.el8.x86_64 18/139 2020-11-20T11:30:06.528 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/139 2020-11-20T11:30:06.528 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : librados2-2:16.0.0-7397.gb86e691b.el8.x86_64 5/139 2020-11-20T11:30:06.528 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librados2-2:16.0.0-7397.gb86e691b.el8.x86_64 5/139 2020-11-20T11:30:06.529 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libgfortran-8.3.1-5.1.el8.x86_64 18/139 2020-11-20T11:30:06.529 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-threads-0.3.3-5.el8.x86_64 19/139 2020-11-20T11:30:06.754 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : librbd1-2:16.0.0-7397.gb86e691b.el8.x86_64 6/139 2020-11-20T11:30:06.851 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librbd1-2:16.0.0-7397.gb86e691b.el8.x86_64 6/139 2020-11-20T11:30:07.029 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephfs2-2:16.0.0-7397.gb86e691b.el8.x86_64 7/139 2020-11-20T11:30:07.037 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: openblas-threads-0.3.3-5.el8.x86_64 19/139 2020-11-20T11:30:07.114 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libcephfs2-2:16.0.0-7397.gb86e691b.el8.x86_64 7/139 2020-11-20T11:30:07.171 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-0.3.3-5.el8.x86_64 20/139 2020-11-20T11:30:07.339 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rados-2:16.0.0-7397.gb86e691b.el8.x86_64 8/139 2020-11-20T11:30:07.479 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 9/139 2020-11-20T11:30:07.554 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 9/139 2020-11-20T11:30:07.903 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: openblas-0.3.3-5.el8.x86_64 20/139 2020-11-20T11:30:08.097 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librabbitmq-0.9.0-2.el8.x86_64 10/139 2020-11-20T11:30:08.279 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librgw2-2:16.0.0-7397.gb86e691b.el8.x86_64 11/139 2020-11-20T11:30:08.399 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librgw2-2:16.0.0-7397.gb86e691b.el8.x86_64 11/139 2020-11-20T11:30:08.625 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 12/139 2020-11-20T11:30:08.692 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 21/139 2020-11-20T11:30:08.851 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 13/139 2020-11-20T11:30:08.949 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libxslt-1.1.32-5.el8.x86_64 22/139 2020-11-20T11:30:08.987 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libradosstriper1-2:16.0.0-7397.gb86e691b.el8.x86 14/139 2020-11-20T11:30:09.086 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : zlib-devel-1.2.11-16.el8_2.x86_64 23/139 2020-11-20T11:30:09.289 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/139 2020-11-20T11:30:09.509 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 25/139 2020-11-20T11:30:09.792 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 26/139 2020-11-20T11:30:09.872 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libradosstriper1-2:16.0.0-7397.gb86e691b.el8.x86 14/139 2020-11-20T11:30:10.013 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 27/139 2020-11-20T11:30:10.183 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-common-2:16.0.0-7397.gb86e691b.el8. 28/139 2020-11-20T11:30:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : NetworkManager-libnm-1:1.26.0-9.el8_3.x86_64 15/139 2020-11-20T11:30:10.281 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-libnm-1:1.26.0-9.el8_3.x86_64 15/139 2020-11-20T11:30:10.300 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-argparse-2:16.0.0-7397.gb86e691b.el 29/139 2020-11-20T11:30:10.587 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-1:1.26.0-9.el8_3.x86_64 16/139 2020-11-20T11:30:10.854 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : NetworkManager-1:1.26.0-9.el8_3.x86_64 16/139 2020-11-20T11:30:11.189 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cephfs-2:16.0.0-7397.gb86e691b.el8.x86_6 30/139 2020-11-20T11:30:11.206 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-1:1.26.0-9.el8_3.x86_64 16/139 2020-11-20T11:30:11.403 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/139 2020-11-20T11:30:11.498 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 31/139 2020-11-20T11:30:11.709 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/139 2020-11-20T11:30:11.717 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 32/139 2020-11-20T11:30:11.977 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/139 2020-11-20T11:30:11.985 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/139 2020-11-20T11:30:12.132 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libquadmath-8.3.1-5.1.el8.x86_64 20/139 2020-11-20T11:30:12.196 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 34/139 2020-11-20T11:30:12.273 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libquadmath-8.3.1-5.1.el8.x86_64 20/139 2020-11-20T11:30:12.387 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 35/139 2020-11-20T11:30:12.434 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libgfortran-8.3.1-5.1.el8.x86_64 21/139 2020-11-20T11:30:12.963 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libgfortran-8.3.1-5.1.el8.x86_64 21/139 2020-11-20T11:30:13.128 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : openblas-threads-0.3.3-5.el8.x86_64 22/139 2020-11-20T11:30:13.652 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: openblas-threads-0.3.3-5.el8.x86_64 22/139 2020-11-20T11:30:13.778 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : openblas-0.3.3-5.el8.x86_64 23/139 2020-11-20T11:30:14.410 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 35/139 2020-11-20T11:30:14.549 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: openblas-0.3.3-5.el8.x86_64 23/139 2020-11-20T11:30:14.686 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 36/139 2020-11-20T11:30:14.830 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 24/139 2020-11-20T11:30:14.866 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rsa-3.4.2-15.el8.noarch 37/139 2020-11-20T11:30:15.023 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 38/139 2020-11-20T11:30:15.039 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libxslt-1.1.32-5.el8.x86_64 25/139 2020-11-20T11:30:15.236 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rgw-2:16.0.0-7397.gb86e691b.el8.x86_64 39/139 2020-11-20T11:30:15.401 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rbd-2:16.0.0-7397.gb86e691b.el8.x86_64 40/139 2020-11-20T11:30:15.581 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librados-devel-2:16.0.0-7397.gb86e691b.el8.x86_6 41/139 2020-11-20T11:30:15.586 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : zlib-devel-1.2.11-16.el8_2.x86_64 26/139 2020-11-20T11:30:15.780 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/139 2020-11-20T11:30:15.818 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 27/139 2020-11-20T11:30:16.034 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-ceph-common-2:16.0.0-7397.gb86e691b.el8. 28/139 2020-11-20T11:30:16.119 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 43/139 2020-11-20T11:30:16.169 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-ceph-argparse-2:16.0.0-7397.gb86e691b.el 29/139 2020-11-20T11:30:16.291 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 44/139 2020-11-20T11:30:16.368 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cephfs-2:16.0.0-7397.gb86e691b.el8.x86_6 30/139 2020-11-20T11:30:16.460 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 45/139 2020-11-20T11:30:16.593 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libuuid-devel-2.32.1-24.el8.x86_64 46/139 2020-11-20T11:30:16.644 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 31/139 2020-11-20T11:30:16.811 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 32/139 2020-11-20T11:30:16.823 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libsepol-devel-2.9-1.el8.x86_64 47/139 2020-11-20T11:30:16.990 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libselinux-devel-2.9-3.el8.x86_64 48/139 2020-11-20T11:30:16.991 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 49/139 2020-11-20T11:30:16.991 INFO:teuthology.orchestra.run.smithi155.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2020-11-20T11:30:16.991 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:30:17.391 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : fuse-2.9.7-12.el8.x86_64 50/139 2020-11-20T11:30:17.569 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 51/139 2020-11-20T11:30:17.819 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 52/139 2020-11-20T11:30:18.013 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 53/139 2020-11-20T11:30:18.293 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 54/139 2020-11-20T11:30:18.628 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-asn1crypto-0.24.0-3.el8.noarch 55/139 2020-11-20T11:30:18.865 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cryptography-2.3-3.el8.x86_64 56/139 2020-11-20T11:30:18.976 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 32/139 2020-11-20T11:30:19.072 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyOpenSSL-18.0.0-1.el8.noarch 57/139 2020-11-20T11:30:19.328 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 58/139 2020-11-20T11:30:19.558 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 59/139 2020-11-20T11:30:19.703 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 60/139 2020-11-20T11:30:19.888 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 61/139 2020-11-20T11:30:20.032 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 33/139 2020-11-20T11:30:20.078 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 62/139 2020-11-20T11:30:20.323 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : krb5-devel-1.18.2-5.el8.x86_64 63/139 2020-11-20T11:30:20.349 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 34/139 2020-11-20T11:30:20.477 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/139 2020-11-20T11:30:20.563 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/139 2020-11-20T11:30:20.596 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 35/139 2020-11-20T11:30:20.806 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : jq-1.5-12.el8.x86_64 65/139 2020-11-20T11:30:20.835 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 36/139 2020-11-20T11:30:21.010 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 66/139 2020-11-20T11:30:21.060 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rsa-3.4.2-15.el8.noarch 37/139 2020-11-20T11:30:21.209 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 67/139 2020-11-20T11:30:21.256 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 38/139 2020-11-20T11:30:21.360 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 68/139 2020-11-20T11:30:21.445 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rgw-2:16.0.0-7397.gb86e691b.el8.x86_64 39/139 2020-11-20T11:30:21.590 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 69/139 2020-11-20T11:30:21.633 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rbd-2:16.0.0-7397.gb86e691b.el8.x86_64 40/139 2020-11-20T11:30:21.786 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librados-devel-2:16.0.0-7397.gb86e691b.el8.x86_6 41/139 2020-11-20T11:30:21.817 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 70/139 2020-11-20T11:30:21.919 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 42/139 2020-11-20T11:30:21.982 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 42/139 2020-11-20T11:30:22.144 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 71/139 2020-11-20T11:30:22.206 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : jq-1.5-12.el8.x86_64 43/139 2020-11-20T11:30:22.327 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : automake-1.16.1-6.el8.noarch 72/139 2020-11-20T11:30:22.369 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 44/139 2020-11-20T11:30:22.523 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 73/139 2020-11-20T11:30:22.534 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 45/139 2020-11-20T11:30:22.711 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 46/139 2020-11-20T11:30:22.772 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 74/139 2020-11-20T11:30:22.891 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 47/139 2020-11-20T11:30:22.968 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 75/139 2020-11-20T11:30:23.114 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 76/139 2020-11-20T11:30:23.146 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 48/139 2020-11-20T11:30:23.288 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 77/139 2020-11-20T11:30:23.467 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 49/139 2020-11-20T11:30:23.469 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 78/139 2020-11-20T11:30:23.699 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : automake-1.16.1-6.el8.noarch 50/139 2020-11-20T11:30:23.713 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 79/139 2020-11-20T11:30:23.863 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 80/139 2020-11-20T11:30:23.932 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 51/139 2020-11-20T11:30:24.059 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 81/139 2020-11-20T11:30:24.229 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 52/139 2020-11-20T11:30:24.307 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 82/139 2020-11-20T11:30:24.423 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 53/139 2020-11-20T11:30:24.455 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cheroot-8.2.1-1.el8.noarch 83/139 2020-11-20T11:30:24.601 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 54/139 2020-11-20T11:30:24.691 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 84/139 2020-11-20T11:30:24.770 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libuuid-devel-2.32.1-24.el8.x86_64 55/139 2020-11-20T11:30:24.934 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-portend-2.6-1.el8.noarch 85/139 2020-11-20T11:30:25.042 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libsepol-devel-2.9-1.el8.x86_64 56/139 2020-11-20T11:30:25.163 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 86/139 2020-11-20T11:30:25.229 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libselinux-devel-2.9-3.el8.x86_64 57/139 2020-11-20T11:30:25.230 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 58/139 2020-11-20T11:30:25.230 INFO:teuthology.orchestra.run.smithi023.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2020-11-20T11:30:25.230 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:30:25.491 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 87/139 2020-11-20T11:30:25.600 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : fuse-2.9.7-12.el8.x86_64 59/139 2020-11-20T11:30:25.751 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 88/139 2020-11-20T11:30:25.796 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 60/139 2020-11-20T11:30:25.969 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 89/139 2020-11-20T11:30:26.078 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 61/139 2020-11-20T11:30:26.232 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 90/139 2020-11-20T11:30:26.303 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 62/139 2020-11-20T11:30:26.425 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 91/139 2020-11-20T11:30:26.538 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 63/139 2020-11-20T11:30:26.566 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 92/139 2020-11-20T11:30:26.773 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 93/139 2020-11-20T11:30:26.868 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-asn1crypto-0.24.0-3.el8.noarch 64/139 2020-11-20T11:30:27.148 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cryptography-2.3-3.el8.x86_64 65/139 2020-11-20T11:30:27.331 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 66/139 2020-11-20T11:30:27.588 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyOpenSSL-18.0.0-1.el8.noarch 67/139 2020-11-20T11:30:27.825 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 68/139 2020-11-20T11:30:27.862 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 94/139 2020-11-20T11:30:28.066 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-kubernetes-11.0.0-2.el8.noarch 95/139 2020-11-20T11:30:28.097 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 69/139 2020-11-20T11:30:28.237 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 70/139 2020-11-20T11:30:28.289 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 96/139 2020-11-20T11:30:28.438 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 71/139 2020-11-20T11:30:28.489 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 97/139 2020-11-20T11:30:28.661 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : krb5-devel-1.18.2-5.el8.x86_64 72/139 2020-11-20T11:30:28.719 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 98/139 2020-11-20T11:30:28.823 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 73/139 2020-11-20T11:30:28.903 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 99/139 2020-11-20T11:30:29.068 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : gperftools-libs-2.7-6.el8.x86_64 100/139 2020-11-20T11:30:29.107 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 74/139 2020-11-20T11:30:29.317 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 75/139 2020-11-20T11:30:29.505 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 76/139 2020-11-20T11:30:29.654 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 77/139 2020-11-20T11:30:29.846 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 78/139 2020-11-20T11:30:30.014 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 79/139 2020-11-20T11:30:30.179 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 80/139 2020-11-20T11:30:30.293 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 81/139 2020-11-20T11:30:30.538 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 82/139 2020-11-20T11:30:30.756 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cheroot-8.2.1-1.el8.noarch 83/139 2020-11-20T11:30:31.024 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 84/139 2020-11-20T11:30:31.267 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-portend-2.6-1.el8.noarch 85/139 2020-11-20T11:30:31.487 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 86/139 2020-11-20T11:30:31.832 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 87/139 2020-11-20T11:30:32.064 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 88/139 2020-11-20T11:30:32.287 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 89/139 2020-11-20T11:30:32.584 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 90/139 2020-11-20T11:30:32.833 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 91/139 2020-11-20T11:30:33.042 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 92/139 2020-11-20T11:30:33.088 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:16.0.0-7397.gb86e691b.el8.x86_64 101/139 2020-11-20T11:30:33.226 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 93/139 2020-11-20T11:30:33.292 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-common-2:16.0.0-7397.gb86e691b.el8.x86_64 101/139 2020-11-20T11:30:34.026 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:16.0.0-7397.gb86e691b.el8.x86_64 101/139 2020-11-20T11:30:34.410 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 94/139 2020-11-20T11:30:34.633 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-kubernetes-11.0.0-2.el8.noarch 95/139 2020-11-20T11:30:34.877 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 96/139 2020-11-20T11:30:35.092 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-base-2:16.0.0-7397.gb86e691b.el8.x86_64 102/139 2020-11-20T11:30:35.141 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 97/139 2020-11-20T11:30:35.369 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 98/139 2020-11-20T11:30:35.388 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-base-2:16.0.0-7397.gb86e691b.el8.x86_64 102/139 2020-11-20T11:30:35.516 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-selinux-2:16.0.0-7397.gb86e691b.el8.x86_64 103/139 2020-11-20T11:30:35.564 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 99/139 2020-11-20T11:30:35.742 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : gperftools-libs-2.7-6.el8.x86_64 100/139 2020-11-20T11:30:40.256 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-common-2:16.0.0-7397.gb86e691b.el8.x86_64 101/139 2020-11-20T11:30:40.458 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-common-2:16.0.0-7397.gb86e691b.el8.x86_64 101/139 2020-11-20T11:30:41.215 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-common-2:16.0.0-7397.gb86e691b.el8.x86_64 101/139 2020-11-20T11:30:41.514 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-base-2:16.0.0-7397.gb86e691b.el8.x86_64 102/139 2020-11-20T11:30:41.613 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-base-2:16.0.0-7397.gb86e691b.el8.x86_64 102/139 2020-11-20T11:30:42.207 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-selinux-2:16.0.0-7397.gb86e691b.el8.x86_64 103/139 2020-11-20T11:30:52.522 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-selinux-2:16.0.0-7397.gb86e691b.el8.x86_64 103/139 2020-11-20T11:30:52.522 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /sys 2020-11-20T11:30:52.523 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /proc 2020-11-20T11:30:52.523 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /dev 2020-11-20T11:30:52.523 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /run 2020-11-20T11:30:52.523 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /mnt 2020-11-20T11:30:52.523 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /var/tmp 2020-11-20T11:30:52.524 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /home 2020-11-20T11:30:52.524 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /tmp 2020-11-20T11:30:52.524 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /dev 2020-11-20T11:30:52.524 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:30:52.889 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mds-2:16.0.0-7397.gb86e691b.el8.x86_64 104/139 2020-11-20T11:30:53.324 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mds-2:16.0.0-7397.gb86e691b.el8.x86_64 104/139 2020-11-20T11:30:53.466 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mon-2:16.0.0-7397.gb86e691b.el8.x86_64 105/139 2020-11-20T11:30:54.982 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mon-2:16.0.0-7397.gb86e691b.el8.x86_64 105/139 2020-11-20T11:30:55.224 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-osd-2:16.0.0-7397.gb86e691b.el8.x86_64 106/139 2020-11-20T11:30:55.251 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-osd-2:16.0.0-7397.gb86e691b.el8.x86_64 106/139 2020-11-20T11:30:57.650 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephadm-2:16.0.0-7397.gb86e691b.el8.noarch 107/139 2020-11-20T11:30:57.950 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephadm-2:16.0.0-7397.gb86e691b.el8.noarch 107/139 2020-11-20T11:30:58.062 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-prometheus-alerts-2:16.0.0-7397.gb86e691b.e 108/139 2020-11-20T11:30:58.224 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-grafana-dashboards-2:16.0.0-7397.gb86e691b. 109/139 2020-11-20T11:30:58.397 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-cephadm-2:16.0.0-7397.gb86e691b.el8.noa 110/139 2020-11-20T11:30:59.283 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.0.0-7397.gb86e691b.el8.noa 110/139 2020-11-20T11:30:59.540 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-dashboard-2:16.0.0-7397.gb86e691b.el8.n 111/139 2020-11-20T11:31:00.018 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-selinux-2:16.0.0-7397.gb86e691b.el8.x86_64 103/139 2020-11-20T11:31:00.018 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /sys 2020-11-20T11:31:00.018 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /proc 2020-11-20T11:31:00.019 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /dev 2020-11-20T11:31:00.019 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /run 2020-11-20T11:31:00.019 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /mnt 2020-11-20T11:31:00.019 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /var/tmp 2020-11-20T11:31:00.019 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /home 2020-11-20T11:31:00.020 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /tmp 2020-11-20T11:31:00.020 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /dev 2020-11-20T11:31:00.020 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:31:00.364 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mds-2:16.0.0-7397.gb86e691b.el8.x86_64 104/139 2020-11-20T11:31:00.582 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.0.0-7397.gb86e691b.el8.n 111/139 2020-11-20T11:31:00.771 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-diskprediction-local-2:16.0.0-7397.gb86 112/139 2020-11-20T11:31:00.824 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mds-2:16.0.0-7397.gb86e691b.el8.x86_64 104/139 2020-11-20T11:31:00.858 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.0.0-7397.gb86 112/139 2020-11-20T11:31:00.961 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mon-2:16.0.0-7397.gb86e691b.el8.x86_64 105/139 2020-11-20T11:31:01.200 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-modules-core-2:16.0.0-7397.gb86e691b.el 113/139 2020-11-20T11:31:01.390 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-2:16.0.0-7397.gb86e691b.el8.x86_64 114/139 2020-11-20T11:31:01.421 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-2:16.0.0-7397.gb86e691b.el8.x86_64 114/139 2020-11-20T11:31:01.579 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-rook-2:16.0.0-7397.gb86e691b.el8.noarch 115/139 2020-11-20T11:31:01.595 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-rook-2:16.0.0-7397.gb86e691b.el8.noarch 115/139 2020-11-20T11:31:02.572 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mon-2:16.0.0-7397.gb86e691b.el8.x86_64 105/139 2020-11-20T11:31:02.765 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-2:16.0.0-7397.gb86e691b.el8.x86_64 116/139 2020-11-20T11:31:02.803 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-osd-2:16.0.0-7397.gb86e691b.el8.x86_64 106/139 2020-11-20T11:31:02.829 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-osd-2:16.0.0-7397.gb86e691b.el8.x86_64 106/139 2020-11-20T11:31:04.021 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-radosgw-2:16.0.0-7397.gb86e691b.el8.x86_64 117/139 2020-11-20T11:31:04.535 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-radosgw-2:16.0.0-7397.gb86e691b.el8.x86_64 117/139 2020-11-20T11:31:04.696 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-mirror-2:16.0.0-7397.gb86e691b.el8.x86_64 118/139 2020-11-20T11:31:05.354 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: cephadm-2:16.0.0-7397.gb86e691b.el8.noarch 107/139 2020-11-20T11:31:05.570 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephadm-2:16.0.0-7397.gb86e691b.el8.noarch 107/139 2020-11-20T11:31:05.704 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-prometheus-alerts-2:16.0.0-7397.gb86e691b.e 108/139 2020-11-20T11:31:05.884 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-grafana-dashboards-2:16.0.0-7397.gb86e691b. 109/139 2020-11-20T11:31:06.029 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-cephadm-2:16.0.0-7397.gb86e691b.el8.noa 110/139 2020-11-20T11:31:06.977 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.0.0-7397.gb86e691b.el8.noa 110/139 2020-11-20T11:31:07.201 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-dashboard-2:16.0.0-7397.gb86e691b.el8.n 111/139 2020-11-20T11:31:08.313 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.0.0-7397.gb86e691b.el8.n 111/139 2020-11-20T11:31:08.503 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-diskprediction-local-2:16.0.0-7397.gb86 112/139 2020-11-20T11:31:08.594 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.0.0-7397.gb86 112/139 2020-11-20T11:31:08.970 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-modules-core-2:16.0.0-7397.gb86e691b.el 113/139 2020-11-20T11:31:09.102 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-mirror-2:16.0.0-7397.gb86e691b.el8.x86_64 118/139 2020-11-20T11:31:09.118 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-2:16.0.0-7397.gb86e691b.el8.x86_64 114/139 2020-11-20T11:31:09.149 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-2:16.0.0-7397.gb86e691b.el8.x86_64 114/139 2020-11-20T11:31:09.287 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-rook-2:16.0.0-7397.gb86e691b.el8.noarch 115/139 2020-11-20T11:31:09.302 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-rook-2:16.0.0-7397.gb86e691b.el8.noarch 115/139 2020-11-20T11:31:09.386 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-test-2:16.0.0-7397.gb86e691b.el8.x86_64 119/139 2020-11-20T11:31:10.011 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libtool-2.4.6-25.el8.x86_64 120/139 2020-11-20T11:31:10.723 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-2:16.0.0-7397.gb86e691b.el8.x86_64 116/139 2020-11-20T11:31:10.904 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 120/139 2020-11-20T11:31:10.937 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-radosgw-2:16.0.0-7397.gb86e691b.el8.x86_64 117/139 2020-11-20T11:31:11.023 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openssl-devel-1:1.1.1g-11.el8.x86_64 121/139 2020-11-20T11:31:11.210 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-fuse-2:16.0.0-7397.gb86e691b.el8.x86_64 122/139 2020-11-20T11:31:11.335 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-radosgw-2:16.0.0-7397.gb86e691b.el8.x86_64 117/139 2020-11-20T11:31:11.419 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : xfsprogs-devel-5.0.0-4.el8.x86_64 123/139 2020-11-20T11:31:11.512 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-mirror-2:16.0.0-7397.gb86e691b.el8.x86_64 118/139 2020-11-20T11:31:11.557 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs-devel-2:16.0.0-7397.gb86e691b.el8.x86_ 124/139 2020-11-20T11:31:11.739 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : elfutils-libelf-devel-0.180-1.el8.x86_64 125/139 2020-11-20T11:31:11.922 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : NetworkManager-tui-1:1.26.0-9.el8_3.x86_64 126/139 2020-11-20T11:31:12.129 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : NetworkManager-team-1:1.26.0-9.el8_3.x86_64 127/139 2020-11-20T11:31:12.328 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-fuse-2:16.0.0-7397.gb86e691b.el8.x86_64 128/139 2020-11-20T11:31:12.555 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-nbd-2:16.0.0-7397.gb86e691b.el8.x86_64 129/139 2020-11-20T11:31:12.736 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-immutable-object-cache-2:16.0.0-7397.gb86e6 130/139 2020-11-20T11:31:12.766 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.0.0-7397.gb86e6 130/139 2020-11-20T11:31:13.008 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/139 2020-11-20T11:31:13.202 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : bison-3.0.4-10.el8.x86_64 132/139 2020-11-20T11:31:13.263 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 132/139 2020-11-20T11:31:13.469 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flex-2.6.1-9.el8.x86_64 133/139 2020-11-20T11:31:13.491 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 133/139 2020-11-20T11:31:13.595 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : NetworkManager-tui-1:1.26.0-8.el8.x86_64 134/139 2020-11-20T11:31:13.596 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 135/139 2020-11-20T11:31:13.837 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 135/139 2020-11-20T11:31:13.946 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : NetworkManager-team-1:1.26.0-8.el8.x86_64 136/139 2020-11-20T11:31:13.960 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: NetworkManager-1:1.26.0-8.el8.x86_64 137/139 2020-11-20T11:31:13.961 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : NetworkManager-1:1.26.0-8.el8.x86_64 137/139 2020-11-20T11:31:14.414 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: NetworkManager-1:1.26.0-8.el8.x86_64 137/139 2020-11-20T11:31:14.414 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : NetworkManager-libnm-1:1.26.0-8.el8.x86_64 138/139 2020-11-20T11:31:14.680 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: NetworkManager-libnm-1:1.26.0-8.el8.x86_64 138/139 2020-11-20T11:31:14.681 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 139/139 2020-11-20T11:31:15.450 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 139/139 2020-11-20T11:31:15.451 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-2:16.0.0-7397.gb86e691b.el8.x86_64 1/139 2020-11-20T11:31:15.451 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-base-2:16.0.0-7397.gb86e691b.el8.x86_64 2/139 2020-11-20T11:31:15.451 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-common-2:16.0.0-7397.gb86e691b.el8.x86_64 3/139 2020-11-20T11:31:15.452 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-fuse-2:16.0.0-7397.gb86e691b.el8.x86_64 4/139 2020-11-20T11:31:15.452 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-immutable-object-cache-2:16.0.0-7397.gb86e6 5/139 2020-11-20T11:31:15.452 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mds-2:16.0.0-7397.gb86e691b.el8.x86_64 6/139 2020-11-20T11:31:15.452 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-2:16.0.0-7397.gb86e691b.el8.x86_64 7/139 2020-11-20T11:31:15.453 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mon-2:16.0.0-7397.gb86e691b.el8.x86_64 8/139 2020-11-20T11:31:15.453 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-osd-2:16.0.0-7397.gb86e691b.el8.x86_64 9/139 2020-11-20T11:31:15.453 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-radosgw-2:16.0.0-7397.gb86e691b.el8.x86_64 10/139 2020-11-20T11:31:15.453 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-selinux-2:16.0.0-7397.gb86e691b.el8.x86_64 11/139 2020-11-20T11:31:15.453 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-test-2:16.0.0-7397.gb86e691b.el8.x86_64 12/139 2020-11-20T11:31:15.454 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs-devel-2:16.0.0-7397.gb86e691b.el8.x86_ 13/139 2020-11-20T11:31:15.454 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs2-2:16.0.0-7397.gb86e691b.el8.x86_64 14/139 2020-11-20T11:31:15.454 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados-devel-2:16.0.0-7397.gb86e691b.el8.x86_6 15/139 2020-11-20T11:31:15.454 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libradosstriper1-2:16.0.0-7397.gb86e691b.el8.x86 16/139 2020-11-20T11:31:15.454 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librgw2-2:16.0.0-7397.gb86e691b.el8.x86_64 17/139 2020-11-20T11:31:15.455 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-argparse-2:16.0.0-7397.gb86e691b.el 18/139 2020-11-20T11:31:15.455 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-common-2:16.0.0-7397.gb86e691b.el8. 19/139 2020-11-20T11:31:15.455 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cephfs-2:16.0.0-7397.gb86e691b.el8.x86_6 20/139 2020-11-20T11:31:15.455 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rados-2:16.0.0-7397.gb86e691b.el8.x86_64 21/139 2020-11-20T11:31:15.455 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rbd-2:16.0.0-7397.gb86e691b.el8.x86_64 22/139 2020-11-20T11:31:15.456 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rgw-2:16.0.0-7397.gb86e691b.el8.x86_64 23/139 2020-11-20T11:31:15.456 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-fuse-2:16.0.0-7397.gb86e691b.el8.x86_64 24/139 2020-11-20T11:31:15.456 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-mirror-2:16.0.0-7397.gb86e691b.el8.x86_64 25/139 2020-11-20T11:31:15.456 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-nbd-2:16.0.0-7397.gb86e691b.el8.x86_64 26/139 2020-11-20T11:31:15.456 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-grafana-dashboards-2:16.0.0-7397.gb86e691b. 27/139 2020-11-20T11:31:15.457 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-cephadm-2:16.0.0-7397.gb86e691b.el8.noa 28/139 2020-11-20T11:31:15.466 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-dashboard-2:16.0.0-7397.gb86e691b.el8.n 29/139 2020-11-20T11:31:15.466 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.0.0-7397.gb86 30/139 2020-11-20T11:31:15.466 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-modules-core-2:16.0.0-7397.gb86e691b.el 31/139 2020-11-20T11:31:15.466 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-rook-2:16.0.0-7397.gb86e691b.el8.noarch 32/139 2020-11-20T11:31:15.466 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-prometheus-alerts-2:16.0.0-7397.gb86e691b.e 33/139 2020-11-20T11:31:15.467 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephadm-2:16.0.0-7397.gb86e691b.el8.noarch 34/139 2020-11-20T11:31:15.467 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 35/139 2020-11-20T11:31:15.467 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : gperftools-libs-2.7-6.el8.x86_64 36/139 2020-11-20T11:31:15.467 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : leveldb-1.22-1.el8.x86_64 37/139 2020-11-20T11:31:15.467 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 38/139 2020-11-20T11:31:15.468 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 39/139 2020-11-20T11:31:15.468 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 40/139 2020-11-20T11:31:15.468 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 41/139 2020-11-20T11:31:15.468 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 42/139 2020-11-20T11:31:15.469 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 43/139 2020-11-20T11:31:15.469 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 44/139 2020-11-20T11:31:15.469 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cheroot-8.2.1-1.el8.noarch 45/139 2020-11-20T11:31:15.469 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 46/139 2020-11-20T11:31:15.470 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 47/139 2020-11-20T11:31:15.470 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 48/139 2020-11-20T11:31:15.470 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/139 2020-11-20T11:31:15.470 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/139 2020-11-20T11:31:15.470 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/139 2020-11-20T11:31:15.471 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-kubernetes-11.0.0-2.el8.noarch 52/139 2020-11-20T11:31:15.471 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/139 2020-11-20T11:31:15.471 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/139 2020-11-20T11:31:15.471 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 55/139 2020-11-20T11:31:15.471 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-portend-2.6-1.el8.noarch 56/139 2020-11-20T11:31:15.472 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 57/139 2020-11-20T11:31:15.472 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/139 2020-11-20T11:31:15.472 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/139 2020-11-20T11:31:15.472 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rsa-3.4.2-15.el8.noarch 60/139 2020-11-20T11:31:15.472 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/139 2020-11-20T11:31:15.472 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/139 2020-11-20T11:31:15.473 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/139 2020-11-20T11:31:15.473 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/139 2020-11-20T11:31:15.473 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/139 2020-11-20T11:31:15.473 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/139 2020-11-20T11:31:15.473 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/139 2020-11-20T11:31:15.474 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/139 2020-11-20T11:31:15.474 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/139 2020-11-20T11:31:15.474 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 70/139 2020-11-20T11:31:15.474 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : automake-1.16.1-6.el8.noarch 71/139 2020-11-20T11:31:15.474 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 72/139 2020-11-20T11:31:15.475 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 73/139 2020-11-20T11:31:15.475 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : jq-1.5-12.el8.x86_64 74/139 2020-11-20T11:31:15.475 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyOpenSSL-18.0.0-1.el8.noarch 75/139 2020-11-20T11:31:15.475 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 76/139 2020-11-20T11:31:15.475 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 77/139 2020-11-20T11:31:15.476 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 78/139 2020-11-20T11:31:15.476 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 79/139 2020-11-20T11:31:15.476 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 80/139 2020-11-20T11:31:15.476 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 81/139 2020-11-20T11:31:15.477 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 82/139 2020-11-20T11:31:15.477 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 83/139 2020-11-20T11:31:15.477 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 84/139 2020-11-20T11:31:15.477 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 85/139 2020-11-20T11:31:15.478 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flex-2.6.1-9.el8.x86_64 86/139 2020-11-20T11:31:15.478 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 87/139 2020-11-20T11:31:15.478 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : bison-3.0.4-10.el8.x86_64 88/139 2020-11-20T11:31:15.478 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 89/139 2020-11-20T11:31:15.478 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 90/139 2020-11-20T11:31:15.479 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 91/139 2020-11-20T11:31:15.479 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 92/139 2020-11-20T11:31:15.479 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 93/139 2020-11-20T11:31:15.479 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 94/139 2020-11-20T11:31:15.479 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 95/139 2020-11-20T11:31:15.479 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-threads-0.3.3-5.el8.x86_64 96/139 2020-11-20T11:31:15.480 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-0.3.3-5.el8.x86_64 97/139 2020-11-20T11:31:15.480 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 98/139 2020-11-20T11:31:15.480 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 99/139 2020-11-20T11:31:15.480 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 100/139 2020-11-20T11:31:15.480 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 101/139 2020-11-20T11:31:15.481 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-asn1crypto-0.24.0-3.el8.noarch 102/139 2020-11-20T11:31:15.481 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 103/139 2020-11-20T11:31:15.481 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 104/139 2020-11-20T11:31:15.481 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 105/139 2020-11-20T11:31:15.481 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 106/139 2020-11-20T11:31:15.482 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 107/139 2020-11-20T11:31:15.482 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 108/139 2020-11-20T11:31:15.482 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 109/139 2020-11-20T11:31:15.482 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libsepol-devel-2.9-1.el8.x86_64 110/139 2020-11-20T11:31:15.482 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 111/139 2020-11-20T11:31:15.483 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libselinux-devel-2.9-3.el8.x86_64 112/139 2020-11-20T11:31:15.483 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cryptography-2.3-3.el8.x86_64 113/139 2020-11-20T11:31:15.483 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : zlib-devel-1.2.11-16.el8_2.x86_64 114/139 2020-11-20T11:31:15.483 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libuuid-devel-2.32.1-24.el8.x86_64 115/139 2020-11-20T11:31:15.483 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libxslt-1.1.32-5.el8.x86_64 116/139 2020-11-20T11:31:15.484 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 117/139 2020-11-20T11:31:15.484 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libquadmath-8.3.1-5.1.el8.x86_64 118/139 2020-11-20T11:31:15.484 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 119/139 2020-11-20T11:31:15.484 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libgfortran-8.3.1-5.1.el8.x86_64 120/139 2020-11-20T11:31:15.484 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : elfutils-libelf-devel-0.180-1.el8.x86_64 121/139 2020-11-20T11:31:15.485 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 122/139 2020-11-20T11:31:15.485 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 123/139 2020-11-20T11:31:15.485 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : krb5-devel-1.18.2-5.el8.x86_64 124/139 2020-11-20T11:31:15.485 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xfsprogs-devel-5.0.0-4.el8.x86_64 125/139 2020-11-20T11:31:15.485 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librabbitmq-0.9.0-2.el8.x86_64 126/139 2020-11-20T11:31:15.486 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openssl-devel-1:1.1.1g-11.el8.x86_64 127/139 2020-11-20T11:31:15.486 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-2:16.0.0-7397.gb86e691b.el8.x86_64 128/139 2020-11-20T11:31:15.486 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 129/139 2020-11-20T11:31:15.486 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-2:16.0.0-7397.gb86e691b.el8.x86_64 130/139 2020-11-20T11:31:15.487 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 131/139 2020-11-20T11:31:15.487 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-libnm-1:1.26.0-9.el8_3.x86_64 132/139 2020-11-20T11:31:15.487 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-libnm-1:1.26.0-8.el8.x86_64 133/139 2020-11-20T11:31:15.487 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-tui-1:1.26.0-9.el8_3.x86_64 134/139 2020-11-20T11:31:15.487 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-tui-1:1.26.0-8.el8.x86_64 135/139 2020-11-20T11:31:15.488 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-team-1:1.26.0-9.el8_3.x86_64 136/139 2020-11-20T11:31:15.488 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-team-1:1.26.0-8.el8.x86_64 137/139 2020-11-20T11:31:15.488 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-1:1.26.0-9.el8_3.x86_64 138/139 2020-11-20T11:31:16.194 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: rbd-mirror-2:16.0.0-7397.gb86e691b.el8.x86_64 118/139 2020-11-20T11:31:17.325 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-test-2:16.0.0-7397.gb86e691b.el8.x86_64 119/139 2020-11-20T11:31:17.492 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : NetworkManager-1:1.26.0-8.el8.x86_64 139/139 2020-11-20T11:31:17.492 INFO:teuthology.orchestra.run.smithi155.stdout:Installed products updated. 2020-11-20T11:31:17.634 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : openssl-devel-1:1.1.1g-11.el8.x86_64 120/139 2020-11-20T11:31:17.791 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-fuse-2:16.0.0-7397.gb86e691b.el8.x86_64 121/139 2020-11-20T11:31:18.044 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : xfsprogs-devel-5.0.0-4.el8.x86_64 122/139 2020-11-20T11:31:18.250 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libtool-2.4.6-25.el8.x86_64 123/139 2020-11-20T11:31:18.265 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 123/139 2020-11-20T11:31:18.415 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephfs-devel-2:16.0.0-7397.gb86e691b.el8.x86_ 124/139 2020-11-20T11:31:18.623 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : elfutils-libelf-devel-0.180-1.el8.x86_64 125/139 2020-11-20T11:31:18.822 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : NetworkManager-tui-1:1.26.0-9.el8_3.x86_64 126/139 2020-11-20T11:31:18.994 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : NetworkManager-team-1:1.26.0-9.el8_3.x86_64 127/139 2020-11-20T11:31:19.186 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-fuse-2:16.0.0-7397.gb86e691b.el8.x86_64 128/139 2020-11-20T11:31:19.414 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-nbd-2:16.0.0-7397.gb86e691b.el8.x86_64 129/139 2020-11-20T11:31:19.603 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-immutable-object-cache-2:16.0.0-7397.gb86e6 130/139 2020-11-20T11:31:19.726 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.0.0-7397.gb86e6 130/139 2020-11-20T11:31:19.885 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/139 2020-11-20T11:31:19.951 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/139 2020-11-20T11:31:20.152 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : flex-2.6.1-9.el8.x86_64 132/139 2020-11-20T11:31:20.166 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 132/139 2020-11-20T11:31:20.199 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:31:20.199 INFO:teuthology.orchestra.run.smithi155.stdout:Upgraded: 2020-11-20T11:31:20.200 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager-1:1.26.0-9.el8_3.x86_64 2020-11-20T11:31:20.200 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager-libnm-1:1.26.0-9.el8_3.x86_64 2020-11-20T11:31:20.200 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager-team-1:1.26.0-9.el8_3.x86_64 2020-11-20T11:31:20.201 INFO:teuthology.orchestra.run.smithi155.stdout: NetworkManager-tui-1:1.26.0-9.el8_3.x86_64 2020-11-20T11:31:20.201 INFO:teuthology.orchestra.run.smithi155.stdout: librados2-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:20.202 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:20.202 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:31:20.202 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2020-11-20T11:31:20.202 INFO:teuthology.orchestra.run.smithi155.stdout: automake-1.16.1-6.el8.noarch 2020-11-20T11:31:20.203 INFO:teuthology.orchestra.run.smithi155.stdout: bison-3.0.4-10.el8.x86_64 2020-11-20T11:31:20.203 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:20.203 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:20.204 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:20.204 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:20.204 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T11:31:20.205 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:20.205 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:20.205 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:20.206 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T11:31:20.206 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T11:31:20.206 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T11:31:20.207 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T11:31:20.207 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T11:31:20.208 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:20.208 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:20.208 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T11:31:20.209 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:20.209 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:20.209 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:20.210 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T11:31:20.210 INFO:teuthology.orchestra.run.smithi155.stdout: elfutils-libelf-devel-0.180-1.el8.x86_64 2020-11-20T11:31:20.210 INFO:teuthology.orchestra.run.smithi155.stdout: flex-2.6.1-9.el8.x86_64 2020-11-20T11:31:20.211 INFO:teuthology.orchestra.run.smithi155.stdout: fmt-6.2.1-1.el8.x86_64 2020-11-20T11:31:20.211 INFO:teuthology.orchestra.run.smithi155.stdout: fuse-2.9.7-12.el8.x86_64 2020-11-20T11:31:20.212 INFO:teuthology.orchestra.run.smithi155.stdout: fuse-common-3.2.1-12.el8.x86_64 2020-11-20T11:31:20.212 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs-2.7-6.el8.x86_64 2020-11-20T11:31:20.212 INFO:teuthology.orchestra.run.smithi155.stdout: jq-1.5-12.el8.x86_64 2020-11-20T11:31:20.213 INFO:teuthology.orchestra.run.smithi155.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2020-11-20T11:31:20.213 INFO:teuthology.orchestra.run.smithi155.stdout: krb5-devel-1.18.2-5.el8.x86_64 2020-11-20T11:31:20.213 INFO:teuthology.orchestra.run.smithi155.stdout: leveldb-1.22-1.el8.x86_64 2020-11-20T11:31:20.213 INFO:teuthology.orchestra.run.smithi155.stdout: libaio-devel-0.3.112-1.el8.x86_64 2020-11-20T11:31:20.214 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:20.214 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:20.214 INFO:teuthology.orchestra.run.smithi155.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2020-11-20T11:31:20.214 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran-8.3.1-5.1.el8.x86_64 2020-11-20T11:31:20.215 INFO:teuthology.orchestra.run.smithi155.stdout: liboath-2.6.2-3.el8.x86_64 2020-11-20T11:31:20.215 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath-8.3.1-5.1.el8.x86_64 2020-11-20T11:31:20.215 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq-0.9.0-2.el8.x86_64 2020-11-20T11:31:20.215 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:20.216 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:20.216 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka-0.11.4-1.el8.x86_64 2020-11-20T11:31:20.216 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:20.217 INFO:teuthology.orchestra.run.smithi155.stdout: libselinux-devel-2.9-3.el8.x86_64 2020-11-20T11:31:20.217 INFO:teuthology.orchestra.run.smithi155.stdout: libsepol-devel-2.9-1.el8.x86_64 2020-11-20T11:31:20.217 INFO:teuthology.orchestra.run.smithi155.stdout: libtool-2.4.6-25.el8.x86_64 2020-11-20T11:31:20.217 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind-1.3.1-3.el8.x86_64 2020-11-20T11:31:20.217 INFO:teuthology.orchestra.run.smithi155.stdout: libuuid-devel-2.32.1-24.el8.x86_64 2020-11-20T11:31:20.218 INFO:teuthology.orchestra.run.smithi155.stdout: libverto-devel-0.3.0-5.el8.x86_64 2020-11-20T11:31:20.218 INFO:teuthology.orchestra.run.smithi155.stdout: libxslt-1.1.32-5.el8.x86_64 2020-11-20T11:31:20.218 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust-2.8.1-11.el8.x86_64 2020-11-20T11:31:20.218 INFO:teuthology.orchestra.run.smithi155.stdout: oniguruma-6.8.2-2.el8.x86_64 2020-11-20T11:31:20.218 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-0.3.3-5.el8.x86_64 2020-11-20T11:31:20.219 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-threads-0.3.3-5.el8.x86_64 2020-11-20T11:31:20.219 INFO:teuthology.orchestra.run.smithi155.stdout: openssl-devel-1:1.1.1g-11.el8.x86_64 2020-11-20T11:31:20.219 INFO:teuthology.orchestra.run.smithi155.stdout: pcre2-devel-10.32-2.el8.x86_64 2020-11-20T11:31:20.219 INFO:teuthology.orchestra.run.smithi155.stdout: pcre2-utf16-10.32-2.el8.x86_64 2020-11-20T11:31:20.219 INFO:teuthology.orchestra.run.smithi155.stdout: pcre2-utf32-10.32-2.el8.x86_64 2020-11-20T11:31:20.220 INFO:teuthology.orchestra.run.smithi155.stdout: python3-apipkg-1.5-6.el8.noarch 2020-11-20T11:31:20.220 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asn1crypto-0.24.0-3.el8.noarch 2020-11-20T11:31:20.220 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel-2.5.1-5.el8.noarch 2020-11-20T11:31:20.220 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2020-11-20T11:31:20.221 INFO:teuthology.orchestra.run.smithi155.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2020-11-20T11:31:20.221 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools-3.1.1-4.el8.noarch 2020-11-20T11:31:20.221 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:20.221 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:20.222 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:20.222 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi-2018.10.15-7.el8.noarch 2020-11-20T11:31:20.223 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi-1.11.5-5.el8.x86_64 2020-11-20T11:31:20.223 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot-8.2.1-1.el8.noarch 2020-11-20T11:31:20.223 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2020-11-20T11:31:20.223 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography-2.3-3.el8.x86_64 2020-11-20T11:31:20.223 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cssselect-0.9.2-10.el8.noarch 2020-11-20T11:31:20.224 INFO:teuthology.orchestra.run.smithi155.stdout: python3-execnet-1.7.1-1.el8.noarch 2020-11-20T11:31:20.224 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2020-11-20T11:31:20.224 INFO:teuthology.orchestra.run.smithi155.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2020-11-20T11:31:20.224 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-6.2-6.el8.noarch 2020-11-20T11:31:20.224 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2020-11-20T11:31:20.225 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2020-11-20T11:31:20.225 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch-1.21-2.el8.noarch 2020-11-20T11:31:20.225 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer-1.10-11.el8.noarch 2020-11-20T11:31:20.225 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jwt-1.6.1-2.el8.noarch 2020-11-20T11:31:20.226 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes-11.0.0-2.el8.noarch 2020-11-20T11:31:20.226 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils-0.3.5-11.el8.noarch 2020-11-20T11:31:20.226 INFO:teuthology.orchestra.run.smithi155.stdout: python3-lxml-4.2.3-1.el8.x86_64 2020-11-20T11:31:20.226 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako-1.0.6-13.el8.noarch 2020-11-20T11:31:20.227 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe-0.23-19.el8.x86_64 2020-11-20T11:31:20.227 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2020-11-20T11:31:20.227 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2020-11-20T11:31:20.228 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2020-11-20T11:31:20.228 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2020-11-20T11:31:20.228 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan-1.3.2-9.el8.noarch 2020-11-20T11:31:20.228 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend-2.6-1.el8.noarch 2020-11-20T11:31:20.229 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable-0.7.2-14.el8.noarch 2020-11-20T11:31:20.229 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL-18.0.0-1.el8.noarch 2020-11-20T11:31:20.229 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2020-11-20T11:31:20.229 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2020-11-20T11:31:20.230 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser-2.14-14.el8.noarch 2020-11-20T11:31:20.230 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz-2017.2-9.el8.noarch 2020-11-20T11:31:20.230 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:20.230 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:20.231 INFO:teuthology.orchestra.run.smithi155.stdout: python3-remoto-1.1.4-4.el8.noarch 2020-11-20T11:31:20.231 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru-0.7-6.el8.noarch 2020-11-20T11:31:20.231 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2020-11-20T11:31:20.231 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:20.231 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes-2.4.1-12.el8.noarch 2020-11-20T11:31:20.232 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa-3.4.2-15.el8.noarch 2020-11-20T11:31:20.232 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2020-11-20T11:31:20.232 INFO:teuthology.orchestra.run.smithi155.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2020-11-20T11:31:20.233 INFO:teuthology.orchestra.run.smithi155.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2020-11-20T11:31:20.233 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora-1.14.1-5.el8.noarch 2020-11-20T11:31:20.233 INFO:teuthology.orchestra.run.smithi155.stdout: python3-trustme-0.6.0-4.el8.noarch 2020-11-20T11:31:20.233 INFO:teuthology.orchestra.run.smithi155.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2020-11-20T11:31:20.234 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webencodings-0.5.1-6.el8.noarch 2020-11-20T11:31:20.234 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob-1.8.5-1.el8.1.noarch 2020-11-20T11:31:20.234 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2020-11-20T11:31:20.234 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webtest-2.0.33-1.el8.noarch 2020-11-20T11:31:20.234 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2020-11-20T11:31:20.235 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2020-11-20T11:31:20.235 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:20.235 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:20.235 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:20.235 INFO:teuthology.orchestra.run.smithi155.stdout: socat-1.7.3.3-2.el8.x86_64 2020-11-20T11:31:20.236 INFO:teuthology.orchestra.run.smithi155.stdout: xfsprogs-devel-5.0.0-4.el8.x86_64 2020-11-20T11:31:20.236 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2020-11-20T11:31:20.236 INFO:teuthology.orchestra.run.smithi155.stdout: zlib-devel-1.2.11-16.el8_2.x86_64 2020-11-20T11:31:20.236 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:31:20.237 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2020-11-20T11:31:20.339 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 133/139 2020-11-20T11:31:20.374 DEBUG:teuthology.parallel:result is None 2020-11-20T11:31:20.461 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : NetworkManager-tui-1:1.26.0-8.el8.x86_64 134/139 2020-11-20T11:31:20.462 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 135/139 2020-11-20T11:31:20.671 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 135/139 2020-11-20T11:31:20.829 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : NetworkManager-team-1:1.26.0-8.el8.x86_64 136/139 2020-11-20T11:31:20.845 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-1:1.26.0-8.el8.x86_64 137/139 2020-11-20T11:31:20.845 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : NetworkManager-1:1.26.0-8.el8.x86_64 137/139 2020-11-20T11:31:21.290 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-1:1.26.0-8.el8.x86_64 137/139 2020-11-20T11:31:21.290 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : NetworkManager-libnm-1:1.26.0-8.el8.x86_64 138/139 2020-11-20T11:31:21.513 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-libnm-1:1.26.0-8.el8.x86_64 138/139 2020-11-20T11:31:21.514 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 139/139 2020-11-20T11:31:22.366 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 139/139 2020-11-20T11:31:22.366 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-2:16.0.0-7397.gb86e691b.el8.x86_64 1/139 2020-11-20T11:31:22.367 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-base-2:16.0.0-7397.gb86e691b.el8.x86_64 2/139 2020-11-20T11:31:22.367 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-common-2:16.0.0-7397.gb86e691b.el8.x86_64 3/139 2020-11-20T11:31:22.367 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-fuse-2:16.0.0-7397.gb86e691b.el8.x86_64 4/139 2020-11-20T11:31:22.368 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-immutable-object-cache-2:16.0.0-7397.gb86e6 5/139 2020-11-20T11:31:22.368 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mds-2:16.0.0-7397.gb86e691b.el8.x86_64 6/139 2020-11-20T11:31:22.368 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-2:16.0.0-7397.gb86e691b.el8.x86_64 7/139 2020-11-20T11:31:22.369 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mon-2:16.0.0-7397.gb86e691b.el8.x86_64 8/139 2020-11-20T11:31:22.369 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-osd-2:16.0.0-7397.gb86e691b.el8.x86_64 9/139 2020-11-20T11:31:22.369 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-radosgw-2:16.0.0-7397.gb86e691b.el8.x86_64 10/139 2020-11-20T11:31:22.370 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-selinux-2:16.0.0-7397.gb86e691b.el8.x86_64 11/139 2020-11-20T11:31:22.370 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-test-2:16.0.0-7397.gb86e691b.el8.x86_64 12/139 2020-11-20T11:31:22.371 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephfs-devel-2:16.0.0-7397.gb86e691b.el8.x86_ 13/139 2020-11-20T11:31:22.371 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephfs2-2:16.0.0-7397.gb86e691b.el8.x86_64 14/139 2020-11-20T11:31:22.372 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados-devel-2:16.0.0-7397.gb86e691b.el8.x86_6 15/139 2020-11-20T11:31:22.373 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libradosstriper1-2:16.0.0-7397.gb86e691b.el8.x86 16/139 2020-11-20T11:31:22.373 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librgw2-2:16.0.0-7397.gb86e691b.el8.x86_64 17/139 2020-11-20T11:31:22.373 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-ceph-argparse-2:16.0.0-7397.gb86e691b.el 18/139 2020-11-20T11:31:22.374 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-ceph-common-2:16.0.0-7397.gb86e691b.el8. 19/139 2020-11-20T11:31:22.374 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cephfs-2:16.0.0-7397.gb86e691b.el8.x86_6 20/139 2020-11-20T11:31:22.375 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rados-2:16.0.0-7397.gb86e691b.el8.x86_64 21/139 2020-11-20T11:31:22.375 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rbd-2:16.0.0-7397.gb86e691b.el8.x86_64 22/139 2020-11-20T11:31:22.375 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rgw-2:16.0.0-7397.gb86e691b.el8.x86_64 23/139 2020-11-20T11:31:22.375 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-fuse-2:16.0.0-7397.gb86e691b.el8.x86_64 24/139 2020-11-20T11:31:22.376 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-mirror-2:16.0.0-7397.gb86e691b.el8.x86_64 25/139 2020-11-20T11:31:22.376 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-nbd-2:16.0.0-7397.gb86e691b.el8.x86_64 26/139 2020-11-20T11:31:22.376 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-grafana-dashboards-2:16.0.0-7397.gb86e691b. 27/139 2020-11-20T11:31:22.376 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-cephadm-2:16.0.0-7397.gb86e691b.el8.noa 28/139 2020-11-20T11:31:22.376 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-dashboard-2:16.0.0-7397.gb86e691b.el8.n 29/139 2020-11-20T11:31:22.377 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.0.0-7397.gb86 30/139 2020-11-20T11:31:22.377 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-modules-core-2:16.0.0-7397.gb86e691b.el 31/139 2020-11-20T11:31:22.377 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-rook-2:16.0.0-7397.gb86e691b.el8.noarch 32/139 2020-11-20T11:31:22.377 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-prometheus-alerts-2:16.0.0-7397.gb86e691b.e 33/139 2020-11-20T11:31:22.377 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephadm-2:16.0.0-7397.gb86e691b.el8.noarch 34/139 2020-11-20T11:31:22.378 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 35/139 2020-11-20T11:31:22.378 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : gperftools-libs-2.7-6.el8.x86_64 36/139 2020-11-20T11:31:22.378 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : leveldb-1.22-1.el8.x86_64 37/139 2020-11-20T11:31:22.378 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 38/139 2020-11-20T11:31:22.378 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 39/139 2020-11-20T11:31:22.379 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 40/139 2020-11-20T11:31:22.379 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 41/139 2020-11-20T11:31:22.379 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 42/139 2020-11-20T11:31:22.379 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 43/139 2020-11-20T11:31:22.379 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 44/139 2020-11-20T11:31:22.380 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cheroot-8.2.1-1.el8.noarch 45/139 2020-11-20T11:31:22.380 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 46/139 2020-11-20T11:31:22.380 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 47/139 2020-11-20T11:31:22.380 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 48/139 2020-11-20T11:31:22.380 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/139 2020-11-20T11:31:22.381 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/139 2020-11-20T11:31:22.381 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/139 2020-11-20T11:31:22.381 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-kubernetes-11.0.0-2.el8.noarch 52/139 2020-11-20T11:31:22.382 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/139 2020-11-20T11:31:22.382 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/139 2020-11-20T11:31:22.383 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 55/139 2020-11-20T11:31:22.383 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-portend-2.6-1.el8.noarch 56/139 2020-11-20T11:31:22.384 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 57/139 2020-11-20T11:31:22.384 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/139 2020-11-20T11:31:22.384 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/139 2020-11-20T11:31:22.384 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rsa-3.4.2-15.el8.noarch 60/139 2020-11-20T11:31:22.384 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/139 2020-11-20T11:31:22.385 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/139 2020-11-20T11:31:22.385 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/139 2020-11-20T11:31:22.385 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/139 2020-11-20T11:31:22.385 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/139 2020-11-20T11:31:22.385 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/139 2020-11-20T11:31:22.386 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/139 2020-11-20T11:31:22.386 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/139 2020-11-20T11:31:22.386 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/139 2020-11-20T11:31:22.386 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 70/139 2020-11-20T11:31:22.386 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 71/139 2020-11-20T11:31:22.387 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 72/139 2020-11-20T11:31:22.387 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-asn1crypto-0.24.0-3.el8.noarch 73/139 2020-11-20T11:31:22.387 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 74/139 2020-11-20T11:31:22.387 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 75/139 2020-11-20T11:31:22.387 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 76/139 2020-11-20T11:31:22.388 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 77/139 2020-11-20T11:31:22.388 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 78/139 2020-11-20T11:31:22.388 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 79/139 2020-11-20T11:31:22.388 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 80/139 2020-11-20T11:31:22.388 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libsepol-devel-2.9-1.el8.x86_64 81/139 2020-11-20T11:31:22.389 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/139 2020-11-20T11:31:22.389 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libselinux-devel-2.9-3.el8.x86_64 83/139 2020-11-20T11:31:22.389 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cryptography-2.3-3.el8.x86_64 84/139 2020-11-20T11:31:22.389 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : zlib-devel-1.2.11-16.el8_2.x86_64 85/139 2020-11-20T11:31:22.389 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libuuid-devel-2.32.1-24.el8.x86_64 86/139 2020-11-20T11:31:22.390 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libxslt-1.1.32-5.el8.x86_64 87/139 2020-11-20T11:31:22.390 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 88/139 2020-11-20T11:31:22.390 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libquadmath-8.3.1-5.1.el8.x86_64 89/139 2020-11-20T11:31:22.390 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 90/139 2020-11-20T11:31:22.390 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libgfortran-8.3.1-5.1.el8.x86_64 91/139 2020-11-20T11:31:22.391 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libelf-devel-0.180-1.el8.x86_64 92/139 2020-11-20T11:31:22.391 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 93/139 2020-11-20T11:31:22.391 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 94/139 2020-11-20T11:31:22.391 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : krb5-devel-1.18.2-5.el8.x86_64 95/139 2020-11-20T11:31:22.391 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsprogs-devel-5.0.0-4.el8.x86_64 96/139 2020-11-20T11:31:22.392 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librabbitmq-0.9.0-2.el8.x86_64 97/139 2020-11-20T11:31:22.392 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-devel-1:1.1.1g-11.el8.x86_64 98/139 2020-11-20T11:31:22.392 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : automake-1.16.1-6.el8.noarch 99/139 2020-11-20T11:31:22.392 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 100/139 2020-11-20T11:31:22.393 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 101/139 2020-11-20T11:31:22.393 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : jq-1.5-12.el8.x86_64 102/139 2020-11-20T11:31:22.393 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyOpenSSL-18.0.0-1.el8.noarch 103/139 2020-11-20T11:31:22.393 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 104/139 2020-11-20T11:31:22.393 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 105/139 2020-11-20T11:31:22.394 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 106/139 2020-11-20T11:31:22.394 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 107/139 2020-11-20T11:31:22.394 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 108/139 2020-11-20T11:31:22.394 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 109/139 2020-11-20T11:31:22.394 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 110/139 2020-11-20T11:31:22.395 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 111/139 2020-11-20T11:31:22.395 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/139 2020-11-20T11:31:22.395 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 113/139 2020-11-20T11:31:22.395 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : flex-2.6.1-9.el8.x86_64 114/139 2020-11-20T11:31:22.395 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 115/139 2020-11-20T11:31:22.396 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : bison-3.0.4-10.el8.x86_64 116/139 2020-11-20T11:31:22.396 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 117/139 2020-11-20T11:31:22.396 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 118/139 2020-11-20T11:31:22.396 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 119/139 2020-11-20T11:31:22.396 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 120/139 2020-11-20T11:31:22.397 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 121/139 2020-11-20T11:31:22.397 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 122/139 2020-11-20T11:31:22.397 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 123/139 2020-11-20T11:31:22.397 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openblas-threads-0.3.3-5.el8.x86_64 124/139 2020-11-20T11:31:22.397 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openblas-0.3.3-5.el8.x86_64 125/139 2020-11-20T11:31:22.398 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 126/139 2020-11-20T11:31:22.398 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 127/139 2020-11-20T11:31:22.398 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados2-2:16.0.0-7397.gb86e691b.el8.x86_64 128/139 2020-11-20T11:31:22.398 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 129/139 2020-11-20T11:31:22.398 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librbd1-2:16.0.0-7397.gb86e691b.el8.x86_64 130/139 2020-11-20T11:31:22.399 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 131/139 2020-11-20T11:31:22.399 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-libnm-1:1.26.0-9.el8_3.x86_64 132/139 2020-11-20T11:31:22.399 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-libnm-1:1.26.0-8.el8.x86_64 133/139 2020-11-20T11:31:22.399 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-tui-1:1.26.0-9.el8_3.x86_64 134/139 2020-11-20T11:31:22.399 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-tui-1:1.26.0-8.el8.x86_64 135/139 2020-11-20T11:31:22.400 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-team-1:1.26.0-9.el8_3.x86_64 136/139 2020-11-20T11:31:22.400 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-team-1:1.26.0-8.el8.x86_64 137/139 2020-11-20T11:31:22.400 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-1:1.26.0-9.el8_3.x86_64 138/139 2020-11-20T11:31:24.439 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-1:1.26.0-8.el8.x86_64 139/139 2020-11-20T11:31:24.439 INFO:teuthology.orchestra.run.smithi023.stdout:Installed products updated. 2020-11-20T11:31:26.992 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:31:26.992 INFO:teuthology.orchestra.run.smithi023.stdout:Upgraded: 2020-11-20T11:31:26.992 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-1:1.26.0-9.el8_3.x86_64 2020-11-20T11:31:26.992 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-libnm-1:1.26.0-9.el8_3.x86_64 2020-11-20T11:31:26.993 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-team-1:1.26.0-9.el8_3.x86_64 2020-11-20T11:31:26.993 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-tui-1:1.26.0-9.el8_3.x86_64 2020-11-20T11:31:26.993 INFO:teuthology.orchestra.run.smithi023.stdout: librados2-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:26.993 INFO:teuthology.orchestra.run.smithi023.stdout: librbd1-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:26.993 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:31:26.994 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2020-11-20T11:31:26.994 INFO:teuthology.orchestra.run.smithi023.stdout: automake-1.16.1-6.el8.noarch 2020-11-20T11:31:26.994 INFO:teuthology.orchestra.run.smithi023.stdout: bison-3.0.4-10.el8.x86_64 2020-11-20T11:31:26.994 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:26.994 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-base-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:26.995 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-common-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:26.995 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-fuse-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:26.995 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-grafana-dashboards-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T11:31:26.995 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-immutable-object-cache-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:26.995 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mds-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:26.996 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:26.996 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-cephadm-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T11:31:26.996 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-dashboard-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T11:31:26.996 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-diskprediction-local-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T11:31:26.996 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-modules-core-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T11:31:26.997 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-rook-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T11:31:26.997 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mon-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:26.997 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-osd-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:26.997 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-prometheus-alerts-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T11:31:26.998 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-radosgw-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:26.998 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-selinux-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:26.998 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-test-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:26.998 INFO:teuthology.orchestra.run.smithi023.stdout: cephadm-2:16.0.0-7397.gb86e691b.el8.noarch 2020-11-20T11:31:26.998 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libelf-devel-0.180-1.el8.x86_64 2020-11-20T11:31:26.999 INFO:teuthology.orchestra.run.smithi023.stdout: flex-2.6.1-9.el8.x86_64 2020-11-20T11:31:26.999 INFO:teuthology.orchestra.run.smithi023.stdout: fmt-6.2.1-1.el8.x86_64 2020-11-20T11:31:26.999 INFO:teuthology.orchestra.run.smithi023.stdout: fuse-2.9.7-12.el8.x86_64 2020-11-20T11:31:26.999 INFO:teuthology.orchestra.run.smithi023.stdout: fuse-common-3.2.1-12.el8.x86_64 2020-11-20T11:31:26.999 INFO:teuthology.orchestra.run.smithi023.stdout: gperftools-libs-2.7-6.el8.x86_64 2020-11-20T11:31:26.999 INFO:teuthology.orchestra.run.smithi023.stdout: jq-1.5-12.el8.x86_64 2020-11-20T11:31:27.000 INFO:teuthology.orchestra.run.smithi023.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2020-11-20T11:31:27.000 INFO:teuthology.orchestra.run.smithi023.stdout: krb5-devel-1.18.2-5.el8.x86_64 2020-11-20T11:31:27.000 INFO:teuthology.orchestra.run.smithi023.stdout: leveldb-1.22-1.el8.x86_64 2020-11-20T11:31:27.000 INFO:teuthology.orchestra.run.smithi023.stdout: libaio-devel-0.3.112-1.el8.x86_64 2020-11-20T11:31:27.001 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs-devel-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:27.001 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs2-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:27.002 INFO:teuthology.orchestra.run.smithi023.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2020-11-20T11:31:27.002 INFO:teuthology.orchestra.run.smithi023.stdout: libgfortran-8.3.1-5.1.el8.x86_64 2020-11-20T11:31:27.002 INFO:teuthology.orchestra.run.smithi023.stdout: liboath-2.6.2-3.el8.x86_64 2020-11-20T11:31:27.002 INFO:teuthology.orchestra.run.smithi023.stdout: libquadmath-8.3.1-5.1.el8.x86_64 2020-11-20T11:31:27.003 INFO:teuthology.orchestra.run.smithi023.stdout: librabbitmq-0.9.0-2.el8.x86_64 2020-11-20T11:31:27.003 INFO:teuthology.orchestra.run.smithi023.stdout: librados-devel-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:27.003 INFO:teuthology.orchestra.run.smithi023.stdout: libradosstriper1-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:27.003 INFO:teuthology.orchestra.run.smithi023.stdout: librdkafka-0.11.4-1.el8.x86_64 2020-11-20T11:31:27.003 INFO:teuthology.orchestra.run.smithi023.stdout: librgw2-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:27.004 INFO:teuthology.orchestra.run.smithi023.stdout: libselinux-devel-2.9-3.el8.x86_64 2020-11-20T11:31:27.004 INFO:teuthology.orchestra.run.smithi023.stdout: libsepol-devel-2.9-1.el8.x86_64 2020-11-20T11:31:27.004 INFO:teuthology.orchestra.run.smithi023.stdout: libtool-2.4.6-25.el8.x86_64 2020-11-20T11:31:27.004 INFO:teuthology.orchestra.run.smithi023.stdout: libunwind-1.3.1-3.el8.x86_64 2020-11-20T11:31:27.004 INFO:teuthology.orchestra.run.smithi023.stdout: libuuid-devel-2.32.1-24.el8.x86_64 2020-11-20T11:31:27.005 INFO:teuthology.orchestra.run.smithi023.stdout: libverto-devel-0.3.0-5.el8.x86_64 2020-11-20T11:31:27.005 INFO:teuthology.orchestra.run.smithi023.stdout: libxslt-1.1.32-5.el8.x86_64 2020-11-20T11:31:27.005 INFO:teuthology.orchestra.run.smithi023.stdout: lttng-ust-2.8.1-11.el8.x86_64 2020-11-20T11:31:27.005 INFO:teuthology.orchestra.run.smithi023.stdout: oniguruma-6.8.2-2.el8.x86_64 2020-11-20T11:31:27.005 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-0.3.3-5.el8.x86_64 2020-11-20T11:31:27.006 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-threads-0.3.3-5.el8.x86_64 2020-11-20T11:31:27.006 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-devel-1:1.1.1g-11.el8.x86_64 2020-11-20T11:31:27.006 INFO:teuthology.orchestra.run.smithi023.stdout: pcre2-devel-10.32-2.el8.x86_64 2020-11-20T11:31:27.006 INFO:teuthology.orchestra.run.smithi023.stdout: pcre2-utf16-10.32-2.el8.x86_64 2020-11-20T11:31:27.006 INFO:teuthology.orchestra.run.smithi023.stdout: pcre2-utf32-10.32-2.el8.x86_64 2020-11-20T11:31:27.007 INFO:teuthology.orchestra.run.smithi023.stdout: python3-apipkg-1.5-6.el8.noarch 2020-11-20T11:31:27.007 INFO:teuthology.orchestra.run.smithi023.stdout: python3-asn1crypto-0.24.0-3.el8.noarch 2020-11-20T11:31:27.007 INFO:teuthology.orchestra.run.smithi023.stdout: python3-babel-2.5.1-5.el8.noarch 2020-11-20T11:31:27.007 INFO:teuthology.orchestra.run.smithi023.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2020-11-20T11:31:27.007 INFO:teuthology.orchestra.run.smithi023.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2020-11-20T11:31:27.008 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cachetools-3.1.1-4.el8.noarch 2020-11-20T11:31:27.008 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-argparse-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:27.008 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-common-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:27.008 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cephfs-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:27.008 INFO:teuthology.orchestra.run.smithi023.stdout: python3-certifi-2018.10.15-7.el8.noarch 2020-11-20T11:31:27.009 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cffi-1.11.5-5.el8.x86_64 2020-11-20T11:31:27.009 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cheroot-8.2.1-1.el8.noarch 2020-11-20T11:31:27.009 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2020-11-20T11:31:27.009 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cryptography-2.3-3.el8.x86_64 2020-11-20T11:31:27.009 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cssselect-0.9.2-10.el8.noarch 2020-11-20T11:31:27.010 INFO:teuthology.orchestra.run.smithi023.stdout: python3-execnet-1.7.1-1.el8.noarch 2020-11-20T11:31:27.010 INFO:teuthology.orchestra.run.smithi023.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2020-11-20T11:31:27.010 INFO:teuthology.orchestra.run.smithi023.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2020-11-20T11:31:27.010 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-6.2-6.el8.noarch 2020-11-20T11:31:27.010 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2020-11-20T11:31:27.011 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2020-11-20T11:31:27.011 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpatch-1.21-2.el8.noarch 2020-11-20T11:31:27.011 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpointer-1.10-11.el8.noarch 2020-11-20T11:31:27.011 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jwt-1.6.1-2.el8.noarch 2020-11-20T11:31:27.012 INFO:teuthology.orchestra.run.smithi023.stdout: python3-kubernetes-11.0.0-2.el8.noarch 2020-11-20T11:31:27.012 INFO:teuthology.orchestra.run.smithi023.stdout: python3-logutils-0.3.5-11.el8.noarch 2020-11-20T11:31:27.012 INFO:teuthology.orchestra.run.smithi023.stdout: python3-lxml-4.2.3-1.el8.x86_64 2020-11-20T11:31:27.012 INFO:teuthology.orchestra.run.smithi023.stdout: python3-mako-1.0.6-13.el8.noarch 2020-11-20T11:31:27.013 INFO:teuthology.orchestra.run.smithi023.stdout: python3-markupsafe-0.23-19.el8.x86_64 2020-11-20T11:31:27.013 INFO:teuthology.orchestra.run.smithi023.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2020-11-20T11:31:27.013 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2020-11-20T11:31:27.013 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2020-11-20T11:31:27.013 INFO:teuthology.orchestra.run.smithi023.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2020-11-20T11:31:27.014 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pecan-1.3.2-9.el8.noarch 2020-11-20T11:31:27.014 INFO:teuthology.orchestra.run.smithi023.stdout: python3-portend-2.6-1.el8.noarch 2020-11-20T11:31:27.014 INFO:teuthology.orchestra.run.smithi023.stdout: python3-prettytable-0.7.2-14.el8.noarch 2020-11-20T11:31:27.015 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyOpenSSL-18.0.0-1.el8.noarch 2020-11-20T11:31:27.015 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2020-11-20T11:31:27.015 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2020-11-20T11:31:27.016 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pycparser-2.14-14.el8.noarch 2020-11-20T11:31:27.016 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pytz-2017.2-9.el8.noarch 2020-11-20T11:31:27.016 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rados-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:27.016 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rbd-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:27.017 INFO:teuthology.orchestra.run.smithi023.stdout: python3-remoto-1.1.4-4.el8.noarch 2020-11-20T11:31:27.017 INFO:teuthology.orchestra.run.smithi023.stdout: python3-repoze-lru-0.7-6.el8.noarch 2020-11-20T11:31:27.017 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2020-11-20T11:31:27.017 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rgw-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:27.017 INFO:teuthology.orchestra.run.smithi023.stdout: python3-routes-2.4.1-12.el8.noarch 2020-11-20T11:31:27.018 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rsa-3.4.2-15.el8.noarch 2020-11-20T11:31:27.018 INFO:teuthology.orchestra.run.smithi023.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2020-11-20T11:31:27.018 INFO:teuthology.orchestra.run.smithi023.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2020-11-20T11:31:27.018 INFO:teuthology.orchestra.run.smithi023.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2020-11-20T11:31:27.019 INFO:teuthology.orchestra.run.smithi023.stdout: python3-tempora-1.14.1-5.el8.noarch 2020-11-20T11:31:27.019 INFO:teuthology.orchestra.run.smithi023.stdout: python3-trustme-0.6.0-4.el8.noarch 2020-11-20T11:31:27.019 INFO:teuthology.orchestra.run.smithi023.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2020-11-20T11:31:27.019 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webencodings-0.5.1-6.el8.noarch 2020-11-20T11:31:27.019 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webob-1.8.5-1.el8.1.noarch 2020-11-20T11:31:27.020 INFO:teuthology.orchestra.run.smithi023.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2020-11-20T11:31:27.020 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webtest-2.0.33-1.el8.noarch 2020-11-20T11:31:27.020 INFO:teuthology.orchestra.run.smithi023.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2020-11-20T11:31:27.020 INFO:teuthology.orchestra.run.smithi023.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2020-11-20T11:31:27.020 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-fuse-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:27.021 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-mirror-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:27.021 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-nbd-2:16.0.0-7397.gb86e691b.el8.x86_64 2020-11-20T11:31:27.021 INFO:teuthology.orchestra.run.smithi023.stdout: socat-1.7.3.3-2.el8.x86_64 2020-11-20T11:31:27.021 INFO:teuthology.orchestra.run.smithi023.stdout: xfsprogs-devel-5.0.0-4.el8.x86_64 2020-11-20T11:31:27.021 INFO:teuthology.orchestra.run.smithi023.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2020-11-20T11:31:27.022 INFO:teuthology.orchestra.run.smithi023.stdout: zlib-devel-1.2.11-16.el8_2.x86_64 2020-11-20T11:31:27.022 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:31:27.022 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2020-11-20T11:31:27.414 DEBUG:teuthology.parallel:result is None 2020-11-20T11:31:27.415 INFO:teuthology.orchestra.run.smithi023:> sudo lsb_release -is 2020-11-20T11:31:27.462 INFO:teuthology.orchestra.run.smithi023.stdout:RedHatEnterprise 2020-11-20T11:31:27.463 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2020-11-20T11:31:27.464 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b86e691bf191566922a8f08e2a5abf64fde8cfcb 2020-11-20T11:31:27.627 INFO:teuthology.orchestra.run.smithi023:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2020-11-20T11:31:27.647 INFO:teuthology.orchestra.run.smithi023.stdout:16.0.0-7397.gb86e691b.el8 2020-11-20T11:31:27.648 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-7397.gb86e691b.el8 2020-11-20T11:31:27.648 INFO:teuthology.task.install:The correct ceph version 16.0.0-7397.gb86e691b is installed. 2020-11-20T11:31:27.649 INFO:teuthology.orchestra.run.smithi155:> sudo lsb_release -is 2020-11-20T11:31:27.701 INFO:teuthology.orchestra.run.smithi155.stdout:RedHatEnterprise 2020-11-20T11:31:27.702 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2020-11-20T11:31:27.703 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b86e691bf191566922a8f08e2a5abf64fde8cfcb 2020-11-20T11:31:27.865 INFO:teuthology.orchestra.run.smithi155:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2020-11-20T11:31:27.887 INFO:teuthology.orchestra.run.smithi155.stdout:16.0.0-7397.gb86e691b.el8 2020-11-20T11:31:27.888 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-7397.gb86e691b.el8 2020-11-20T11:31:27.888 INFO:teuthology.task.install:The correct ceph version 16.0.0-7397.gb86e691b is installed. 2020-11-20T11:31:27.889 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-11-20T11:31:27.889 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-11-20T11:31:27.890 INFO:teuthology.orchestra.run.smithi023:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-20T11:31:27.904 INFO:teuthology.orchestra.run.smithi023.stderr:+ sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-20T11:31:27.914 INFO:teuthology.orchestra.run.smithi023.stderr:24+1 records in 2020-11-20T11:31:27.914 INFO:teuthology.orchestra.run.smithi023.stderr:24+1 records out 2020-11-20T11:31:27.915 INFO:teuthology.orchestra.run.smithi023.stderr:12667 bytes (13 kB, 12 KiB) copied, 8.9998e-05 s, 141 MB/s 2020-11-20T11:31:27.916 INFO:teuthology.orchestra.run.smithi155:> set -ex 2020-11-20T11:31:27.916 INFO:teuthology.orchestra.run.smithi155:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-20T11:31:27.942 INFO:teuthology.orchestra.run.smithi155.stderr:+ sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-11-20T11:31:27.953 INFO:teuthology.orchestra.run.smithi155.stderr:24+1 records in 2020-11-20T11:31:27.954 INFO:teuthology.orchestra.run.smithi155.stderr:24+1 records out 2020-11-20T11:31:27.954 INFO:teuthology.orchestra.run.smithi155.stderr:12667 bytes (13 kB, 12 KiB) copied, 7.6343e-05 s, 166 MB/s 2020-11-20T11:31:27.955 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-11-20T11:31:27.955 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-11-20T11:31:27.955 INFO:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/daemon-helper 2020-11-20T11:31:27.970 INFO:teuthology.orchestra.run.smithi023.stderr:+ sudo dd of=/usr/bin/daemon-helper 2020-11-20T11:31:27.980 INFO:teuthology.orchestra.run.smithi023.stderr:5+1 records in 2020-11-20T11:31:27.980 INFO:teuthology.orchestra.run.smithi023.stderr:5+1 records out 2020-11-20T11:31:27.980 INFO:teuthology.orchestra.run.smithi023.stderr:3058 bytes (3.1 kB, 3.0 KiB) copied, 4.1299e-05 s, 74.0 MB/s 2020-11-20T11:31:27.981 INFO:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-11-20T11:31:28.046 INFO:teuthology.orchestra.run.smithi155:> set -ex 2020-11-20T11:31:28.046 INFO:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/daemon-helper 2020-11-20T11:31:28.061 INFO:teuthology.orchestra.run.smithi155.stderr:+ sudo dd of=/usr/bin/daemon-helper 2020-11-20T11:31:28.071 INFO:teuthology.orchestra.run.smithi155.stderr:5+1 records in 2020-11-20T11:31:28.071 INFO:teuthology.orchestra.run.smithi155.stderr:5+1 records out 2020-11-20T11:31:28.071 INFO:teuthology.orchestra.run.smithi155.stderr:3058 bytes (3.1 kB, 3.0 KiB) copied, 3.8715e-05 s, 79.0 MB/s 2020-11-20T11:31:28.072 INFO:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-11-20T11:31:28.138 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-11-20T11:31:28.138 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-11-20T11:31:28.139 INFO:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/adjust-ulimits 2020-11-20T11:31:28.152 INFO:teuthology.orchestra.run.smithi023.stderr:+ sudo dd of=/usr/bin/adjust-ulimits 2020-11-20T11:31:28.162 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2020-11-20T11:31:28.162 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2020-11-20T11:31:28.162 INFO:teuthology.orchestra.run.smithi023.stderr:299 bytes copied, 2.76e-05 s, 10.8 MB/s 2020-11-20T11:31:28.163 INFO:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-11-20T11:31:28.228 INFO:teuthology.orchestra.run.smithi155:> set -ex 2020-11-20T11:31:28.229 INFO:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/adjust-ulimits 2020-11-20T11:31:28.243 INFO:teuthology.orchestra.run.smithi155.stderr:+ sudo dd of=/usr/bin/adjust-ulimits 2020-11-20T11:31:28.252 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records in 2020-11-20T11:31:28.253 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records out 2020-11-20T11:31:28.253 INFO:teuthology.orchestra.run.smithi155.stderr:299 bytes copied, 2.655e-05 s, 11.3 MB/s 2020-11-20T11:31:28.254 INFO:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-11-20T11:31:28.319 INFO:teuthology.run_tasks:Running task ceph... 2020-11-20T11:31:28.422 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2020-11-20T11:31:28.423 INFO:teuthology.orchestra.run.smithi023:> sudo chmod 777 /var/log/ceph 2020-11-20T11:31:28.428 INFO:teuthology.orchestra.run.smithi155:> sudo chmod 777 /var/log/ceph 2020-11-20T11:31:28.456 INFO:tasks.ceph:Disabling ceph logrotate... 2020-11-20T11:31:28.456 INFO:teuthology.orchestra.run.smithi023:> sudo rm -f -- /etc/logrotate.d/ceph 2020-11-20T11:31:28.499 INFO:teuthology.orchestra.run.smithi155:> sudo rm -f -- /etc/logrotate.d/ceph 2020-11-20T11:31:28.529 INFO:tasks.ceph:Creating extra log directories... 2020-11-20T11:31:28.529 INFO:teuthology.orchestra.run.smithi023:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-11-20T11:31:28.564 INFO:teuthology.orchestra.run.smithi155:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-11-20T11:31:28.809 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2020-11-20T11:31:28.810 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2020-11-20T11:31:28.810 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2020-11-20T11:31:28.811 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-11-20T11:31:28.812 INFO:teuthology.orchestra.run.smithi023:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2020-11-20T11:31:28.830 INFO:teuthology.orchestra.run.smithi023.stderr:+ dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2020-11-20T11:31:28.830 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2020-11-20T11:31:28.830 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2020-11-20T11:31:28.831 INFO:teuthology.orchestra.run.smithi023.stderr:408 bytes copied, 3.2571e-05 s, 12.5 MB/s 2020-11-20T11:31:28.831 INFO:teuthology.orchestra.run.smithi023:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2020-11-20T11:31:28.939 INFO:teuthology.orchestra.run.smithi023:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2020-11-20T11:31:28.965 INFO:teuthology.orchestra.run.smithi155:> set -ex 2020-11-20T11:31:28.965 INFO:teuthology.orchestra.run.smithi155:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2020-11-20T11:31:28.981 INFO:teuthology.orchestra.run.smithi155.stderr:+ dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2020-11-20T11:31:28.981 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records in 2020-11-20T11:31:28.982 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records out 2020-11-20T11:31:28.982 INFO:teuthology.orchestra.run.smithi155.stderr:408 bytes copied, 2.8618e-05 s, 14.3 MB/s 2020-11-20T11:31:28.982 INFO:teuthology.orchestra.run.smithi155:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2020-11-20T11:31:29.083 INFO:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2020-11-20T11:31:29.113 INFO:tasks.ceph:Creating ceph cluster ceph... 2020-11-20T11:31:29.116 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(MON_DOWN\\)', 'SLOW_OPS', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-11-20T11:31:29.117 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi/5640259', 'branch': 'wip-pdonnell-testing-20201120.053228', 'description': 'fs/thrash/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/ffsb}}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '5640259', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi', 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.3', 'overrides': {'admin_socket': {'branch': 'wip-pdonnell-testing-20201120.053228'}, 'ceph': {'cephfs': {'session_timeout': 300}, 'cephfs_ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(MON_DOWN\\)', 'SLOW_OPS', 'slow request'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'b86e691bf191566922a8f08e2a5abf64fde8cfcb'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'sha1': 'b86e691bf191566922a8f08e2a5abf64fde8cfcb'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-pdonnell-testing-20201120.053228', 'sha1': 'b86e691bf191566922a8f08e2a5abf64fde8cfcb'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 95, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'sha1': 'b86e691bf191566922a8f08e2a5abf64fde8cfcb', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'wip-pdonnell-testing-20201120.053228', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20201120.053228/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': 'b86e691bf191566922a8f08e2a5abf64fde8cfcb', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'sha1': 'b86e691bf191566922a8f08e2a5abf64fde8cfcb'}}, {'ceph': None}, {'ceph-fuse': None}, {'mon_thrash': {'check_mds_failover': True, 'revive_delay': 20, 'thrash_delay': 10}}, {'workunit': {'clients': {'all': ['suites/ffsb.sh']}}}], 'teuthology_branch': 'master', 'timestamp': '2020-11-20_08:39:42', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.3615', 'targets': {'smithi023.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpDWhJyxHEaXeIe4gkFGgQed9NCWMHEbgBYa8tvID2kLwoa3wcMCUQDaFS53Cl20PLMEVBcJQCbf9gw1ZJm50+waTcO2JDmDl1dsFxX+8/GNWHd/cBHcl1cc7x2YPAFArGh8lMRM1j8sU/45uZY3Tg2C8y41AB8M7TCKNm5rKSUQ0gybty1vM8SjdTKddvkkIwzakXx/vkUtoFktUbqppi1xZJwAbG+FGFqqyI6CZvi3M6+HbAth9Mza8ZP+r4ucId4xpKG8R9BFFKVvpZL9UsJCQcoRG6tC/iV5QzT88jFiZPRo59UpZ2GWLqvZ+UsOnomyHcqsObAN8ga5Q5SJRYJ3KuzDD8R+VZoPCoHEeYnqoYYYcltyCiW9Q0Tjt4zVt2n4FF0AOmws2vmClEPQEKfEUJyNXO5S+evFrgKpGsicy02zDc4d34VLL4+iFjXJ+KH++N5NuwRxW6GRXP5+qK9Acy7JhEptgi5srs7JcamcrfviSGyUXL/AxR6J1lQ2E=', 'smithi155.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBdGF4vMnkzXHeSw7B5wT3VK35OlPEsgWsMpHXmRNzNdf0Zqu/Nhz2GL5CSbEdcMOapXnt7wXZtLXB43JfmQYq+pra+EGsVq3O9z0BfNucyiQOvY7fdkuCaSvEdZ1CKSp3eQWLEIqrb7P0IYPw+WcAZ78ktRpXUHwa4ktLnZGW/YwWBabVsK6bey/9P2bTa58Lt1gl3YLZluLAxH+Xo1BizNjzs1hIC+0KvRKVqggdMpCAAGecVmyjscwH0WwDoXVDw0Dv4rmKYpQx1/j2K9bmMXJMo+x0ua3irZKSLUorPwYkgOBAR4GEsme6ha4mOMc3z+DnzXjx/p5yl0AwvasiqeLBin6KRbNSatCW6YNa4v1Mo3dek89HRUkn6iD/eYTxmsPfU9Q92UaXJFc6bUmSW3KaXB6/xMXnLMSqZHEuS9V1HEjyz0WAODOEcs/G0HCPG8aCjjtT5Z2H8l2z6w0HieM54ohG7Mb8oJ7U/0PTDlAEy3VHsgf3e5MpHqAQrBk='}} 2020-11-20T11:31:29.118 INFO:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2020-11-20T11:31:29.120 INFO:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2020-11-20T11:31:29.162 INFO:teuthology.orchestra.run.smithi023:> sudo install -d -m0777 -- /var/run/ceph 2020-11-20T11:31:29.174 INFO:teuthology.orchestra.run.smithi155:> sudo install -d -m0777 -- /var/run/ceph 2020-11-20T11:31:29.225 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-11-20T11:31:29.226 INFO:teuthology.orchestra.run.smithi023:> dd if=/scratch_devs of=/dev/stdout 2020-11-20T11:31:29.251 INFO:teuthology.orchestra.run.smithi023.stderr:+ dd if=/scratch_devs of=/dev/stdout 2020-11-20T11:31:29.251 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2020-11-20T11:31:29.252 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2020-11-20T11:31:29.252 INFO:teuthology.orchestra.run.smithi023.stderr:72 bytes copied, 3.6361e-05 s, 2.0 MB/s 2020-11-20T11:31:29.252 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/vg_nvme/lv_1 2020-11-20T11:31:29.252 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/vg_nvme/lv_2 2020-11-20T11:31:29.253 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/vg_nvme/lv_3 2020-11-20T11:31:29.253 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/vg_nvme/lv_4 2020-11-20T11:31:29.253 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-11-20T11:31:29.254 INFO:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-11-20T11:31:29.306 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-11-20T11:31:29.307 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-20T11:31:29.307 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 6h/6d Inode: 58716 Links: 1 2020-11-20T11:31:29.307 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-20T11:31:29.308 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2020-11-20T11:31:29.308 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2020-11-20 11:31:21.963310087 +0000 2020-11-20T11:31:29.308 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2020-11-20 11:29:12.403703445 +0000 2020-11-20T11:31:29.308 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2020-11-20 11:29:12.403703445 +0000 2020-11-20T11:31:29.308 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: - 2020-11-20T11:31:29.317 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2020-11-20T11:31:29.318 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2020-11-20T11:31:29.319 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000298896 s, 1.7 MB/s 2020-11-20T11:31:29.320 INFO:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-11-20T11:31:29.375 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-11-20T11:31:29.376 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-20T11:31:29.376 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 6h/6d Inode: 59860 Links: 1 2020-11-20T11:31:29.376 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-20T11:31:29.376 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2020-11-20T11:31:29.377 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2020-11-20 11:31:21.963310087 +0000 2020-11-20T11:31:29.377 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2020-11-20 11:29:12.729701517 +0000 2020-11-20T11:31:29.377 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2020-11-20 11:29:12.729701517 +0000 2020-11-20T11:31:29.377 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: - 2020-11-20T11:31:29.385 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2020-11-20T11:31:29.385 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2020-11-20T11:31:29.386 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.00026462 s, 1.9 MB/s 2020-11-20T11:31:29.388 INFO:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-11-20T11:31:29.443 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-11-20T11:31:29.444 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-20T11:31:29.444 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 6h/6d Inode: 60614 Links: 1 2020-11-20T11:31:29.444 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-20T11:31:29.444 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2020-11-20T11:31:29.444 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2020-11-20 11:31:21.963310087 +0000 2020-11-20T11:31:29.445 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2020-11-20 11:29:13.080699442 +0000 2020-11-20T11:31:29.445 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2020-11-20 11:29:13.080699442 +0000 2020-11-20T11:31:29.445 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: - 2020-11-20T11:31:29.453 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2020-11-20T11:31:29.454 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2020-11-20T11:31:29.454 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000288828 s, 1.8 MB/s 2020-11-20T11:31:29.456 INFO:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-11-20T11:31:29.512 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-11-20T11:31:29.513 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-20T11:31:29.513 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 6h/6d Inode: 58749 Links: 1 2020-11-20T11:31:29.514 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-20T11:31:29.514 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2020-11-20T11:31:29.514 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2020-11-20 11:31:21.963310087 +0000 2020-11-20T11:31:29.514 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2020-11-20 11:29:13.404697527 +0000 2020-11-20T11:31:29.514 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2020-11-20 11:29:13.404697527 +0000 2020-11-20T11:31:29.515 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: - 2020-11-20T11:31:29.522 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2020-11-20T11:31:29.522 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2020-11-20T11:31:29.522 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000296081 s, 1.7 MB/s 2020-11-20T11:31:29.524 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2020-11-20T11:31:29.525 INFO:teuthology.orchestra.run.smithi155:> set -ex 2020-11-20T11:31:29.525 INFO:teuthology.orchestra.run.smithi155:> dd if=/scratch_devs of=/dev/stdout 2020-11-20T11:31:29.539 INFO:teuthology.orchestra.run.smithi155.stderr:+ dd if=/scratch_devs of=/dev/stdout 2020-11-20T11:31:29.541 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/vg_nvme/lv_1 2020-11-20T11:31:29.541 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/vg_nvme/lv_2 2020-11-20T11:31:29.541 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/vg_nvme/lv_3 2020-11-20T11:31:29.541 INFO:teuthology.orchestra.run.smithi155.stdout:/dev/vg_nvme/lv_4 2020-11-20T11:31:29.542 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records in 2020-11-20T11:31:29.542 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records out 2020-11-20T11:31:29.542 INFO:teuthology.orchestra.run.smithi155.stderr:72 bytes copied, 2.9371e-05 s, 2.5 MB/s 2020-11-20T11:31:29.542 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-11-20T11:31:29.543 INFO:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-11-20T11:31:29.595 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-11-20T11:31:29.595 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-20T11:31:29.596 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 6h/6d Inode: 58717 Links: 1 2020-11-20T11:31:29.596 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-20T11:31:29.596 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2020-11-20T11:31:29.596 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2020-11-20 11:31:15.087743467 +0000 2020-11-20T11:31:29.596 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2020-11-20 11:29:11.066851491 +0000 2020-11-20T11:31:29.597 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2020-11-20 11:29:11.066851491 +0000 2020-11-20T11:31:29.597 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: - 2020-11-20T11:31:29.605 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2020-11-20T11:31:29.605 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2020-11-20T11:31:29.605 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000223519 s, 2.3 MB/s 2020-11-20T11:31:29.607 INFO:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-11-20T11:31:29.662 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-11-20T11:31:29.663 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-20T11:31:29.663 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 6h/6d Inode: 55166 Links: 1 2020-11-20T11:31:29.663 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-20T11:31:29.664 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2020-11-20T11:31:29.664 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2020-11-20 11:31:15.087743467 +0000 2020-11-20T11:31:29.664 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2020-11-20 11:29:11.393845032 +0000 2020-11-20T11:31:29.664 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2020-11-20 11:29:11.393845032 +0000 2020-11-20T11:31:29.664 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: - 2020-11-20T11:31:29.672 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2020-11-20T11:31:29.672 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2020-11-20T11:31:29.672 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000253995 s, 2.0 MB/s 2020-11-20T11:31:29.675 INFO:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-11-20T11:31:29.729 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-11-20T11:31:29.729 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-20T11:31:29.730 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 6h/6d Inode: 61456 Links: 1 2020-11-20T11:31:29.730 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-20T11:31:29.730 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2020-11-20T11:31:29.730 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2020-11-20 11:31:15.087743467 +0000 2020-11-20T11:31:29.731 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2020-11-20 11:29:11.700838969 +0000 2020-11-20T11:31:29.731 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2020-11-20 11:29:11.700838969 +0000 2020-11-20T11:31:29.731 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: - 2020-11-20T11:31:29.738 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2020-11-20T11:31:29.739 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2020-11-20T11:31:29.739 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000213251 s, 2.4 MB/s 2020-11-20T11:31:29.741 INFO:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-11-20T11:31:29.796 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-11-20T11:31:29.796 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-11-20T11:31:29.797 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 6h/6d Inode: 55206 Links: 1 2020-11-20T11:31:29.797 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-20T11:31:29.797 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2020-11-20T11:31:29.797 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2020-11-20 11:31:15.088743447 +0000 2020-11-20T11:31:29.797 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2020-11-20 11:29:12.029832471 +0000 2020-11-20T11:31:29.798 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2020-11-20 11:29:12.029832471 +0000 2020-11-20T11:31:29.798 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: - 2020-11-20T11:31:29.805 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2020-11-20T11:31:29.806 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2020-11-20T11:31:29.806 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000193633 s, 2.6 MB/s 2020-11-20T11:31:29.808 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2020-11-20T11:31:29.808 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi023.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi155.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2020-11-20T11:31:29.809 INFO:tasks.ceph:Generating config... 2020-11-20T11:31:29.810 INFO:tasks.ceph:[client] client mount timeout = 600 2020-11-20T11:31:29.810 INFO:tasks.ceph:[client] debug client = 20 2020-11-20T11:31:29.811 INFO:tasks.ceph:[client] debug ms = 1 2020-11-20T11:31:29.811 INFO:tasks.ceph:[client] rados mon op timeout = 15m 2020-11-20T11:31:29.811 INFO:tasks.ceph:[client] rados osd op timeout = 15m 2020-11-20T11:31:29.812 INFO:tasks.ceph:[mds] debug mds = 20 2020-11-20T11:31:29.812 INFO:tasks.ceph:[mds] debug ms = 1 2020-11-20T11:31:29.812 INFO:tasks.ceph:[mds] mds bal frag = True 2020-11-20T11:31:29.812 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2020-11-20T11:31:29.813 INFO:tasks.ceph:[mds] mds bal merge size = 5 2020-11-20T11:31:29.813 INFO:tasks.ceph:[mds] mds bal split bits = 3 2020-11-20T11:31:29.813 INFO:tasks.ceph:[mds] mds bal split size = 100 2020-11-20T11:31:29.814 INFO:tasks.ceph:[mds] mds debug frag = True 2020-11-20T11:31:29.814 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2020-11-20T11:31:29.814 INFO:tasks.ceph:[mds] mds op complaint time = 180 2020-11-20T11:31:29.815 INFO:tasks.ceph:[mds] mds verify scatter = True 2020-11-20T11:31:29.815 INFO:tasks.ceph:[mds] osd op complaint time = 180 2020-11-20T11:31:29.815 INFO:tasks.ceph:[mds] rados mon op timeout = 15m 2020-11-20T11:31:29.815 INFO:tasks.ceph:[mds] rados osd op timeout = 15m 2020-11-20T11:31:29.816 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-11-20T11:31:29.816 INFO:tasks.ceph:[mgr] debug ms = 1 2020-11-20T11:31:29.816 INFO:tasks.ceph:[mon] debug mon = 20 2020-11-20T11:31:29.817 INFO:tasks.ceph:[mon] debug ms = 1 2020-11-20T11:31:29.817 INFO:tasks.ceph:[mon] debug paxos = 20 2020-11-20T11:31:29.817 INFO:tasks.ceph:[mon] mon op complaint time = 120 2020-11-20T11:31:29.818 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2020-11-20T11:31:29.818 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2020-11-20T11:31:29.819 INFO:tasks.ceph:[osd] debug bluefs = 20 2020-11-20T11:31:29.819 INFO:tasks.ceph:[osd] debug bluestore = 20 2020-11-20T11:31:29.819 INFO:tasks.ceph:[osd] debug filestore = 20 2020-11-20T11:31:29.819 INFO:tasks.ceph:[osd] debug journal = 20 2020-11-20T11:31:29.820 INFO:tasks.ceph:[osd] debug ms = 20 2020-11-20T11:31:29.820 INFO:tasks.ceph:[osd] debug osd = 25 2020-11-20T11:31:29.820 INFO:tasks.ceph:[osd] debug rocksdb = 10 2020-11-20T11:31:29.820 INFO:tasks.ceph:[osd] filestore flush min = 0 2020-11-20T11:31:29.821 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2020-11-20T11:31:29.821 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2020-11-20T11:31:29.821 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2020-11-20T11:31:29.822 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2020-11-20T11:31:29.822 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2020-11-20T11:31:29.822 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2020-11-20T11:31:29.823 INFO:tasks.ceph:[osd] osd op complaint time = 180 2020-11-20T11:31:29.823 INFO:tasks.ceph:Setting up mon.a... 2020-11-20T11:31:29.823 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2020-11-20T11:31:29.871 INFO:teuthology.orchestra.run.smithi023.stdout:creating /etc/ceph/ceph.keyring 2020-11-20T11:31:29.874 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2020-11-20T11:31:29.913 INFO:teuthology.orchestra.run.smithi023:> sudo chmod 0644 /etc/ceph/ceph.keyring 2020-11-20T11:31:29.978 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.23'), ('mon.b', '172.21.15.155'), ('mon.c', '[v2:172.21.15.155:3301,v1:172.21.15.155:6790]')] 2020-11-20T11:31:29.980 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.23,172.21.15.155,[v2:172.21.15.155:3301,v1:172.21.15.155:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.b': {}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}} 2020-11-20T11:31:29.982 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-11-20T11:31:29.983 INFO:teuthology.orchestra.run.smithi023:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-20T11:31:30.032 INFO:teuthology.orchestra.run.smithi023.stderr:+ dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-11-20T11:31:30.033 INFO:teuthology.orchestra.run.smithi023.stderr:6+1 records in 2020-11-20T11:31:30.034 INFO:teuthology.orchestra.run.smithi023.stderr:6+1 records out 2020-11-20T11:31:30.034 INFO:teuthology.orchestra.run.smithi023.stderr:3473 bytes (3.5 kB, 3.4 KiB) copied, 4.2526e-05 s, 81.7 MB/s 2020-11-20T11:31:30.034 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.23 --add b 172.21.15.155 --addv c '[v2:172.21.15.155:3301,v1:172.21.15.155:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2020-11-20T11:31:30.114 INFO:teuthology.orchestra.run.smithi023.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2020-11-20T11:31:30.114 INFO:teuthology.orchestra.run.smithi023.stdout:monmaptool: generated fsid 4b071d14-8a39-4afe-8acc-4218fa4790c7 2020-11-20T11:31:30.115 INFO:teuthology.orchestra.run.smithi023.stdout:epoch 0 2020-11-20T11:31:30.115 INFO:teuthology.orchestra.run.smithi023.stdout:fsid 4b071d14-8a39-4afe-8acc-4218fa4790c7 2020-11-20T11:31:30.115 INFO:teuthology.orchestra.run.smithi023.stdout:last_changed 2020-11-20T11:31:30.112250+0000 2020-11-20T11:31:30.115 INFO:teuthology.orchestra.run.smithi023.stdout:created 2020-11-20T11:31:30.112250+0000 2020-11-20T11:31:30.116 INFO:teuthology.orchestra.run.smithi023.stdout:min_mon_release 0 (unknown) 2020-11-20T11:31:30.116 INFO:teuthology.orchestra.run.smithi023.stdout:election_strategy: 1 2020-11-20T11:31:30.116 INFO:teuthology.orchestra.run.smithi023.stdout:0: [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] mon.a 2020-11-20T11:31:30.116 INFO:teuthology.orchestra.run.smithi023.stdout:1: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.b 2020-11-20T11:31:30.116 INFO:teuthology.orchestra.run.smithi023.stdout:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2020-11-20T11:31:30.117 INFO:teuthology.orchestra.run.smithi023.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2020-11-20T11:31:30.118 INFO:teuthology.orchestra.run.smithi023:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2020-11-20T11:31:30.172 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 4b071d14-8a39-4afe-8acc-4218fa4790c7... 2020-11-20T11:31:30.173 INFO:teuthology.orchestra.run.smithi023:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T11:31:30.215 INFO:teuthology.orchestra.run.smithi155:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-11-20T11:31:30.256 INFO:teuthology.orchestra.run.smithi023.stdout:[global] 2020-11-20T11:31:30.256 INFO:teuthology.orchestra.run.smithi023.stdout: chdir = "" 2020-11-20T11:31:30.257 INFO:teuthology.orchestra.run.smithi023.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T11:31:30.257 INFO:teuthology.orchestra.run.smithi023.stdout: auth supported = cephx 2020-11-20T11:31:30.257 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:31:30.257 INFO:teuthology.orchestra.run.smithi023.stdout: filestore xattr use omap = true 2020-11-20T11:31:30.257 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:31:30.258 INFO:teuthology.orchestra.run.smithi023.stdout: mon clock drift allowed = 1.000 2020-11-20T11:31:30.258 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:31:30.258 INFO:teuthology.orchestra.run.smithi023.stdout: osd crush chooseleaf type = 0 2020-11-20T11:31:30.258 INFO:teuthology.orchestra.run.smithi023.stdout: auth debug = true 2020-11-20T11:31:30.259 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:31:30.259 INFO:teuthology.orchestra.run.smithi023.stdout: ms die on old message = true 2020-11-20T11:31:30.259 INFO:teuthology.orchestra.run.smithi023.stdout: ms die on bug = true 2020-11-20T11:31:30.259 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:31:30.259 INFO:teuthology.orchestra.run.smithi023.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T11:31:30.260 INFO:teuthology.orchestra.run.smithi023.stdout: mon pg warn max object skew = 0 2020-11-20T11:31:30.260 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:31:30.260 INFO:teuthology.orchestra.run.smithi023.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T11:31:30.261 INFO:teuthology.orchestra.run.smithi023.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T11:31:30.261 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:31:30.261 INFO:teuthology.orchestra.run.smithi023.stdout: osd pool default size = 2 2020-11-20T11:31:30.261 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:31:30.261 INFO:teuthology.orchestra.run.smithi023.stdout: mon osd allow primary affinity = true 2020-11-20T11:31:30.262 INFO:teuthology.orchestra.run.smithi023.stdout: mon osd allow pg remap = true 2020-11-20T11:31:30.262 INFO:teuthology.orchestra.run.smithi023.stdout: mon warn on legacy crush tunables = false 2020-11-20T11:31:30.262 INFO:teuthology.orchestra.run.smithi023.stdout: mon warn on crush straw calc version zero = false 2020-11-20T11:31:30.262 INFO:teuthology.orchestra.run.smithi023.stdout: mon warn on no sortbitwise = false 2020-11-20T11:31:30.262 INFO:teuthology.orchestra.run.smithi023.stdout: mon warn on osd down out interval zero = false 2020-11-20T11:31:30.263 INFO:teuthology.orchestra.run.smithi023.stdout: mon warn on too few osds = false 2020-11-20T11:31:30.263 INFO:teuthology.orchestra.run.smithi023.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T11:31:30.263 INFO:teuthology.orchestra.run.smithi023.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T11:31:30.263 INFO:teuthology.orchestra.run.smithi023.stdout: mon_allow_pool_size_one = true 2020-11-20T11:31:30.263 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:31:30.264 INFO:teuthology.orchestra.run.smithi023.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T11:31:30.264 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:31:30.264 INFO:teuthology.orchestra.run.smithi023.stdout: osd default data pool replay window = 5 2020-11-20T11:31:30.265 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:31:30.265 INFO:teuthology.orchestra.run.smithi023.stdout: mon allow pool delete = true 2020-11-20T11:31:30.265 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:31:30.265 INFO:teuthology.orchestra.run.smithi023.stdout: mon cluster log file level = debug 2020-11-20T11:31:30.265 INFO:teuthology.orchestra.run.smithi023.stdout: debug asserts on shutdown = true 2020-11-20T11:31:30.266 INFO:teuthology.orchestra.run.smithi023.stdout: mon health detail to clog = false 2020-11-20T11:31:30.266 INFO:teuthology.orchestra.run.smithi023.stdout: mon host = "172.21.15.23,172.21.15.155,[v2:172.21.15.155:3301,v1:172.21.15.155:6790]" 2020-11-20T11:31:30.266 INFO:teuthology.orchestra.run.smithi023.stdout: fsid = 4b071d14-8a39-4afe-8acc-4218fa4790c7 2020-11-20T11:31:30.266 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:31:30.266 INFO:teuthology.orchestra.run.smithi023.stdout:[osd] 2020-11-20T11:31:30.267 INFO:teuthology.orchestra.run.smithi023.stdout: osd journal size = 100 2020-11-20T11:31:30.267 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:31:30.267 INFO:teuthology.orchestra.run.smithi023.stdout: osd scrub load threshold = 5.0 2020-11-20T11:31:30.267 INFO:teuthology.orchestra.run.smithi023.stdout: osd scrub max interval = 600 2020-11-20T11:31:30.267 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:31:30.268 INFO:teuthology.orchestra.run.smithi023.stdout: osd recover clone overlap = true 2020-11-20T11:31:30.268 INFO:teuthology.orchestra.run.smithi023.stdout: osd recovery max chunk = 1048576 2020-11-20T11:31:30.268 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:31:30.268 INFO:teuthology.orchestra.run.smithi023.stdout: osd debug shutdown = true 2020-11-20T11:31:30.268 INFO:teuthology.orchestra.run.smithi023.stdout: osd debug op order = true 2020-11-20T11:31:30.269 INFO:teuthology.orchestra.run.smithi023.stdout: osd debug verify stray on activate = true 2020-11-20T11:31:30.269 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:31:30.269 INFO:teuthology.orchestra.run.smithi023.stdout: osd open classes on start = true 2020-11-20T11:31:30.269 INFO:teuthology.orchestra.run.smithi023.stdout: osd debug pg log writeout = true 2020-11-20T11:31:30.270 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:31:30.270 INFO:teuthology.orchestra.run.smithi023.stdout: osd deep scrub update digest min age = 30 2020-11-20T11:31:30.270 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:31:30.270 INFO:teuthology.orchestra.run.smithi023.stdout: osd map max advance = 10 2020-11-20T11:31:30.270 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:31:30.271 INFO:teuthology.orchestra.run.smithi023.stdout: journal zero on create = true 2020-11-20T11:31:30.271 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:31:30.271 INFO:teuthology.orchestra.run.smithi023.stdout: filestore ondisk finisher threads = 3 2020-11-20T11:31:30.271 INFO:teuthology.orchestra.run.smithi023.stdout: filestore apply finisher threads = 3 2020-11-20T11:31:30.271 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:31:30.271 INFO:teuthology.orchestra.run.smithi023.stdout: bdev debug aio = true 2020-11-20T11:31:30.272 INFO:teuthology.orchestra.run.smithi023.stdout: osd debug misdirected ops = true 2020-11-20T11:31:30.272 INFO:teuthology.orchestra.run.smithi023.stdout: bluestore block size = 96636764160 2020-11-20T11:31:30.272 INFO:teuthology.orchestra.run.smithi023.stdout: bluestore fsck on mount = True 2020-11-20T11:31:30.272 INFO:teuthology.orchestra.run.smithi023.stdout: debug bluefs = 20 2020-11-20T11:31:30.272 INFO:teuthology.orchestra.run.smithi023.stdout: debug bluestore = 20 2020-11-20T11:31:30.273 INFO:teuthology.orchestra.run.smithi023.stdout: debug filestore = 20 2020-11-20T11:31:30.273 INFO:teuthology.orchestra.run.smithi023.stdout: debug journal = 20 2020-11-20T11:31:30.273 INFO:teuthology.orchestra.run.smithi023.stdout: debug ms = 20 2020-11-20T11:31:30.273 INFO:teuthology.orchestra.run.smithi023.stdout: debug osd = 25 2020-11-20T11:31:30.273 INFO:teuthology.orchestra.run.smithi023.stdout: debug rocksdb = 10 2020-11-20T11:31:30.274 INFO:teuthology.orchestra.run.smithi023.stdout: filestore flush min = 0 2020-11-20T11:31:30.274 INFO:teuthology.orchestra.run.smithi023.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T11:31:30.274 INFO:teuthology.orchestra.run.smithi023.stdout: mon osd full ratio = 0.9 2020-11-20T11:31:30.274 INFO:teuthology.orchestra.run.smithi023.stdout: mon osd nearfull ratio = 0.8 2020-11-20T11:31:30.274 INFO:teuthology.orchestra.run.smithi023.stdout: osd failsafe full ratio = 0.95 2020-11-20T11:31:30.275 INFO:teuthology.orchestra.run.smithi023.stdout: osd max markdown count = 1000 2020-11-20T11:31:30.275 INFO:teuthology.orchestra.run.smithi023.stdout: osd objectstore = bluestore 2020-11-20T11:31:30.275 INFO:teuthology.orchestra.run.smithi023.stdout: osd op complaint time = 180 2020-11-20T11:31:30.275 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:31:30.275 INFO:teuthology.orchestra.run.smithi023.stdout:[mgr] 2020-11-20T11:31:30.276 INFO:teuthology.orchestra.run.smithi023.stdout: debug ms = 1 2020-11-20T11:31:30.276 INFO:teuthology.orchestra.run.smithi023.stdout: debug mgr = 20 2020-11-20T11:31:30.276 INFO:teuthology.orchestra.run.smithi023.stdout: debug mon = 20 2020-11-20T11:31:30.276 INFO:teuthology.orchestra.run.smithi023.stdout: debug auth = 20 2020-11-20T11:31:30.276 INFO:teuthology.orchestra.run.smithi023.stdout: mon reweight min pgs per osd = 4 2020-11-20T11:31:30.277 INFO:teuthology.orchestra.run.smithi023.stdout: mon reweight min bytes per osd = 10 2020-11-20T11:31:30.277 INFO:teuthology.orchestra.run.smithi023.stdout: mgr/telemetry/nag = false 2020-11-20T11:31:30.277 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:31:30.277 INFO:teuthology.orchestra.run.smithi023.stdout:[mon] 2020-11-20T11:31:30.277 INFO:teuthology.orchestra.run.smithi023.stdout: debug ms = 1 2020-11-20T11:31:30.278 INFO:teuthology.orchestra.run.smithi023.stdout: debug mon = 20 2020-11-20T11:31:30.278 INFO:teuthology.orchestra.run.smithi023.stdout: debug paxos = 20 2020-11-20T11:31:30.278 INFO:teuthology.orchestra.run.smithi023.stdout: debug auth = 20 2020-11-20T11:31:30.278 INFO:teuthology.orchestra.run.smithi023.stdout: mon data avail warn = 5 2020-11-20T11:31:30.278 INFO:teuthology.orchestra.run.smithi023.stdout: mon mgr mkfs grace = 240 2020-11-20T11:31:30.279 INFO:teuthology.orchestra.run.smithi023.stdout: mon reweight min pgs per osd = 4 2020-11-20T11:31:30.279 INFO:teuthology.orchestra.run.smithi023.stdout: mon osd reporter subtree level = osd 2020-11-20T11:31:30.279 INFO:teuthology.orchestra.run.smithi023.stdout: mon osd prime pg temp = true 2020-11-20T11:31:30.279 INFO:teuthology.orchestra.run.smithi023.stdout: mon reweight min bytes per osd = 10 2020-11-20T11:31:30.279 INFO:teuthology.orchestra.run.smithi023.stdout: mon op complaint time = 120 2020-11-20T11:31:30.280 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:31:30.280 INFO:teuthology.orchestra.run.smithi023.stdout:[client] 2020-11-20T11:31:30.280 INFO:teuthology.orchestra.run.smithi023.stdout: rgw cache enabled = true 2020-11-20T11:31:30.280 INFO:teuthology.orchestra.run.smithi023.stdout: rgw enable ops log = true 2020-11-20T11:31:30.280 INFO:teuthology.orchestra.run.smithi023.stdout: rgw enable usage log = true 2020-11-20T11:31:30.281 INFO:teuthology.orchestra.run.smithi023.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T11:31:30.281 INFO:teuthology.orchestra.run.smithi023.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T11:31:30.281 INFO:teuthology.orchestra.run.smithi023.stdout: client mount timeout = 600 2020-11-20T11:31:30.281 INFO:teuthology.orchestra.run.smithi023.stdout: debug client = 20 2020-11-20T11:31:30.282 INFO:teuthology.orchestra.run.smithi023.stdout: debug ms = 1 2020-11-20T11:31:30.282 INFO:teuthology.orchestra.run.smithi023.stdout: rados mon op timeout = 15m 2020-11-20T11:31:30.282 INFO:teuthology.orchestra.run.smithi023.stdout: rados osd op timeout = 15m 2020-11-20T11:31:30.282 INFO:teuthology.orchestra.run.smithi023.stdout:[mon.a] 2020-11-20T11:31:30.283 INFO:teuthology.orchestra.run.smithi023.stdout:[mon.b] 2020-11-20T11:31:30.283 INFO:teuthology.orchestra.run.smithi023.stdout:[mon.c] 2020-11-20T11:31:30.283 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.a] 2020-11-20T11:31:30.283 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.b] 2020-11-20T11:31:30.283 INFO:teuthology.orchestra.run.smithi023.stdout:[mds] 2020-11-20T11:31:30.284 INFO:teuthology.orchestra.run.smithi023.stdout: debug mds = 20 2020-11-20T11:31:30.284 INFO:teuthology.orchestra.run.smithi023.stdout: debug ms = 1 2020-11-20T11:31:30.284 INFO:teuthology.orchestra.run.smithi023.stdout: mds bal frag = True 2020-11-20T11:31:30.284 INFO:teuthology.orchestra.run.smithi023.stdout: mds bal fragment size max = 10000 2020-11-20T11:31:30.284 INFO:teuthology.orchestra.run.smithi023.stdout: mds bal merge size = 5 2020-11-20T11:31:30.285 INFO:teuthology.orchestra.run.smithi023.stdout: mds bal split bits = 3 2020-11-20T11:31:30.285 INFO:teuthology.orchestra.run.smithi023.stdout: mds bal split size = 100 2020-11-20T11:31:30.285 INFO:teuthology.orchestra.run.smithi023.stdout: mds debug frag = True 2020-11-20T11:31:30.285 INFO:teuthology.orchestra.run.smithi023.stdout: mds debug scatterstat = True 2020-11-20T11:31:30.285 INFO:teuthology.orchestra.run.smithi023.stdout: mds op complaint time = 180 2020-11-20T11:31:30.286 INFO:teuthology.orchestra.run.smithi023.stdout: mds verify scatter = True 2020-11-20T11:31:30.286 INFO:teuthology.orchestra.run.smithi023.stdout: osd op complaint time = 180 2020-11-20T11:31:30.286 INFO:teuthology.orchestra.run.smithi023.stdout: rados mon op timeout = 15m 2020-11-20T11:31:30.286 INFO:teuthology.orchestra.run.smithi023.stdout: rados osd op timeout = 15m 2020-11-20T11:31:30.288 INFO:teuthology.orchestra.run.smithi155.stdout:[global] 2020-11-20T11:31:30.288 INFO:teuthology.orchestra.run.smithi155.stdout: chdir = "" 2020-11-20T11:31:30.288 INFO:teuthology.orchestra.run.smithi155.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-11-20T11:31:30.288 INFO:teuthology.orchestra.run.smithi155.stdout: auth supported = cephx 2020-11-20T11:31:30.289 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:31:30.289 INFO:teuthology.orchestra.run.smithi155.stdout: filestore xattr use omap = true 2020-11-20T11:31:30.289 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:31:30.289 INFO:teuthology.orchestra.run.smithi155.stdout: mon clock drift allowed = 1.000 2020-11-20T11:31:30.289 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:31:30.290 INFO:teuthology.orchestra.run.smithi155.stdout: osd crush chooseleaf type = 0 2020-11-20T11:31:30.290 INFO:teuthology.orchestra.run.smithi155.stdout: auth debug = true 2020-11-20T11:31:30.290 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:31:30.290 INFO:teuthology.orchestra.run.smithi155.stdout: ms die on old message = true 2020-11-20T11:31:30.290 INFO:teuthology.orchestra.run.smithi155.stdout: ms die on bug = true 2020-11-20T11:31:30.291 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:31:30.291 INFO:teuthology.orchestra.run.smithi155.stdout: mon max pg per osd = 10000 # >= luminous 2020-11-20T11:31:30.291 INFO:teuthology.orchestra.run.smithi155.stdout: mon pg warn max object skew = 0 2020-11-20T11:31:30.291 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:31:30.292 INFO:teuthology.orchestra.run.smithi155.stdout: # disable pg_autoscaler by default for new pools 2020-11-20T11:31:30.292 INFO:teuthology.orchestra.run.smithi155.stdout: osd_pool_default_pg_autoscale_mode = off 2020-11-20T11:31:30.292 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:31:30.292 INFO:teuthology.orchestra.run.smithi155.stdout: osd pool default size = 2 2020-11-20T11:31:30.292 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:31:30.293 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd allow primary affinity = true 2020-11-20T11:31:30.293 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd allow pg remap = true 2020-11-20T11:31:30.293 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on legacy crush tunables = false 2020-11-20T11:31:30.293 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on crush straw calc version zero = false 2020-11-20T11:31:30.293 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on no sortbitwise = false 2020-11-20T11:31:30.294 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on osd down out interval zero = false 2020-11-20T11:31:30.294 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on too few osds = false 2020-11-20T11:31:30.294 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-11-20T11:31:30.294 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_pool_no_redundancy = false 2020-11-20T11:31:30.294 INFO:teuthology.orchestra.run.smithi155.stdout: mon_allow_pool_size_one = true 2020-11-20T11:31:30.295 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:31:30.295 INFO:teuthology.orchestra.run.smithi155.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-11-20T11:31:30.295 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:31:30.295 INFO:teuthology.orchestra.run.smithi155.stdout: osd default data pool replay window = 5 2020-11-20T11:31:30.295 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:31:30.296 INFO:teuthology.orchestra.run.smithi155.stdout: mon allow pool delete = true 2020-11-20T11:31:30.296 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:31:30.296 INFO:teuthology.orchestra.run.smithi155.stdout: mon cluster log file level = debug 2020-11-20T11:31:30.296 INFO:teuthology.orchestra.run.smithi155.stdout: debug asserts on shutdown = true 2020-11-20T11:31:30.296 INFO:teuthology.orchestra.run.smithi155.stdout: mon health detail to clog = false 2020-11-20T11:31:30.297 INFO:teuthology.orchestra.run.smithi155.stdout: mon host = "172.21.15.23,172.21.15.155,[v2:172.21.15.155:3301,v1:172.21.15.155:6790]" 2020-11-20T11:31:30.297 INFO:teuthology.orchestra.run.smithi155.stdout: fsid = 4b071d14-8a39-4afe-8acc-4218fa4790c7 2020-11-20T11:31:30.297 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:31:30.297 INFO:teuthology.orchestra.run.smithi155.stdout:[osd] 2020-11-20T11:31:30.297 INFO:teuthology.orchestra.run.smithi155.stdout: osd journal size = 100 2020-11-20T11:31:30.298 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:31:30.298 INFO:teuthology.orchestra.run.smithi155.stdout: osd scrub load threshold = 5.0 2020-11-20T11:31:30.298 INFO:teuthology.orchestra.run.smithi155.stdout: osd scrub max interval = 600 2020-11-20T11:31:30.298 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:31:30.298 INFO:teuthology.orchestra.run.smithi155.stdout: osd recover clone overlap = true 2020-11-20T11:31:30.299 INFO:teuthology.orchestra.run.smithi155.stdout: osd recovery max chunk = 1048576 2020-11-20T11:31:30.299 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:31:30.299 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug shutdown = true 2020-11-20T11:31:30.299 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug op order = true 2020-11-20T11:31:30.299 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug verify stray on activate = true 2020-11-20T11:31:30.300 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:31:30.300 INFO:teuthology.orchestra.run.smithi155.stdout: osd open classes on start = true 2020-11-20T11:31:30.300 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug pg log writeout = true 2020-11-20T11:31:30.300 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:31:30.300 INFO:teuthology.orchestra.run.smithi155.stdout: osd deep scrub update digest min age = 30 2020-11-20T11:31:30.301 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:31:30.301 INFO:teuthology.orchestra.run.smithi155.stdout: osd map max advance = 10 2020-11-20T11:31:30.301 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:31:30.301 INFO:teuthology.orchestra.run.smithi155.stdout: journal zero on create = true 2020-11-20T11:31:30.301 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:31:30.302 INFO:teuthology.orchestra.run.smithi155.stdout: filestore ondisk finisher threads = 3 2020-11-20T11:31:30.302 INFO:teuthology.orchestra.run.smithi155.stdout: filestore apply finisher threads = 3 2020-11-20T11:31:30.302 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:31:30.302 INFO:teuthology.orchestra.run.smithi155.stdout: bdev debug aio = true 2020-11-20T11:31:30.302 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug misdirected ops = true 2020-11-20T11:31:30.303 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore block size = 96636764160 2020-11-20T11:31:30.303 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore fsck on mount = True 2020-11-20T11:31:30.303 INFO:teuthology.orchestra.run.smithi155.stdout: debug bluefs = 20 2020-11-20T11:31:30.303 INFO:teuthology.orchestra.run.smithi155.stdout: debug bluestore = 20 2020-11-20T11:31:30.303 INFO:teuthology.orchestra.run.smithi155.stdout: debug filestore = 20 2020-11-20T11:31:30.304 INFO:teuthology.orchestra.run.smithi155.stdout: debug journal = 20 2020-11-20T11:31:30.304 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 20 2020-11-20T11:31:30.304 INFO:teuthology.orchestra.run.smithi155.stdout: debug osd = 25 2020-11-20T11:31:30.304 INFO:teuthology.orchestra.run.smithi155.stdout: debug rocksdb = 10 2020-11-20T11:31:30.304 INFO:teuthology.orchestra.run.smithi155.stdout: filestore flush min = 0 2020-11-20T11:31:30.305 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd backfillfull_ratio = 0.85 2020-11-20T11:31:30.305 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd full ratio = 0.9 2020-11-20T11:31:30.305 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd nearfull ratio = 0.8 2020-11-20T11:31:30.305 INFO:teuthology.orchestra.run.smithi155.stdout: osd failsafe full ratio = 0.95 2020-11-20T11:31:30.305 INFO:teuthology.orchestra.run.smithi155.stdout: osd max markdown count = 1000 2020-11-20T11:31:30.306 INFO:teuthology.orchestra.run.smithi155.stdout: osd objectstore = bluestore 2020-11-20T11:31:30.306 INFO:teuthology.orchestra.run.smithi155.stdout: osd op complaint time = 180 2020-11-20T11:31:30.306 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:31:30.306 INFO:teuthology.orchestra.run.smithi155.stdout:[mgr] 2020-11-20T11:31:30.306 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2020-11-20T11:31:30.307 INFO:teuthology.orchestra.run.smithi155.stdout: debug mgr = 20 2020-11-20T11:31:30.307 INFO:teuthology.orchestra.run.smithi155.stdout: debug mon = 20 2020-11-20T11:31:30.307 INFO:teuthology.orchestra.run.smithi155.stdout: debug auth = 20 2020-11-20T11:31:30.307 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min pgs per osd = 4 2020-11-20T11:31:30.307 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min bytes per osd = 10 2020-11-20T11:31:30.308 INFO:teuthology.orchestra.run.smithi155.stdout: mgr/telemetry/nag = false 2020-11-20T11:31:30.308 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:31:30.308 INFO:teuthology.orchestra.run.smithi155.stdout:[mon] 2020-11-20T11:31:30.308 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2020-11-20T11:31:30.308 INFO:teuthology.orchestra.run.smithi155.stdout: debug mon = 20 2020-11-20T11:31:30.309 INFO:teuthology.orchestra.run.smithi155.stdout: debug paxos = 20 2020-11-20T11:31:30.309 INFO:teuthology.orchestra.run.smithi155.stdout: debug auth = 20 2020-11-20T11:31:30.309 INFO:teuthology.orchestra.run.smithi155.stdout: mon data avail warn = 5 2020-11-20T11:31:30.309 INFO:teuthology.orchestra.run.smithi155.stdout: mon mgr mkfs grace = 240 2020-11-20T11:31:30.309 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min pgs per osd = 4 2020-11-20T11:31:30.310 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd reporter subtree level = osd 2020-11-20T11:31:30.310 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd prime pg temp = true 2020-11-20T11:31:30.310 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min bytes per osd = 10 2020-11-20T11:31:30.310 INFO:teuthology.orchestra.run.smithi155.stdout: mon op complaint time = 120 2020-11-20T11:31:30.310 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-11-20T11:31:30.311 INFO:teuthology.orchestra.run.smithi155.stdout:[client] 2020-11-20T11:31:30.311 INFO:teuthology.orchestra.run.smithi155.stdout: rgw cache enabled = true 2020-11-20T11:31:30.311 INFO:teuthology.orchestra.run.smithi155.stdout: rgw enable ops log = true 2020-11-20T11:31:30.311 INFO:teuthology.orchestra.run.smithi155.stdout: rgw enable usage log = true 2020-11-20T11:31:30.311 INFO:teuthology.orchestra.run.smithi155.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-11-20T11:31:30.312 INFO:teuthology.orchestra.run.smithi155.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-11-20T11:31:30.312 INFO:teuthology.orchestra.run.smithi155.stdout: client mount timeout = 600 2020-11-20T11:31:30.312 INFO:teuthology.orchestra.run.smithi155.stdout: debug client = 20 2020-11-20T11:31:30.312 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2020-11-20T11:31:30.312 INFO:teuthology.orchestra.run.smithi155.stdout: rados mon op timeout = 15m 2020-11-20T11:31:30.313 INFO:teuthology.orchestra.run.smithi155.stdout: rados osd op timeout = 15m 2020-11-20T11:31:30.313 INFO:teuthology.orchestra.run.smithi155.stdout:[mon.a] 2020-11-20T11:31:30.313 INFO:teuthology.orchestra.run.smithi155.stdout:[mon.b] 2020-11-20T11:31:30.313 INFO:teuthology.orchestra.run.smithi155.stdout:[mon.c] 2020-11-20T11:31:30.313 INFO:teuthology.orchestra.run.smithi155.stdout:[mds.a] 2020-11-20T11:31:30.314 INFO:teuthology.orchestra.run.smithi155.stdout:[mds.b] 2020-11-20T11:31:30.314 INFO:teuthology.orchestra.run.smithi155.stdout:[mds] 2020-11-20T11:31:30.314 INFO:teuthology.orchestra.run.smithi155.stdout: debug mds = 20 2020-11-20T11:31:30.314 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2020-11-20T11:31:30.314 INFO:teuthology.orchestra.run.smithi155.stdout: mds bal frag = True 2020-11-20T11:31:30.315 INFO:teuthology.orchestra.run.smithi155.stdout: mds bal fragment size max = 10000 2020-11-20T11:31:30.315 INFO:teuthology.orchestra.run.smithi155.stdout: mds bal merge size = 5 2020-11-20T11:31:30.315 INFO:teuthology.orchestra.run.smithi155.stdout: mds bal split bits = 3 2020-11-20T11:31:30.315 INFO:teuthology.orchestra.run.smithi155.stdout: mds bal split size = 100 2020-11-20T11:31:30.315 INFO:teuthology.orchestra.run.smithi155.stdout: mds debug frag = True 2020-11-20T11:31:30.316 INFO:teuthology.orchestra.run.smithi155.stdout: mds debug scatterstat = True 2020-11-20T11:31:30.316 INFO:teuthology.orchestra.run.smithi155.stdout: mds op complaint time = 180 2020-11-20T11:31:30.316 INFO:teuthology.orchestra.run.smithi155.stdout: mds verify scatter = True 2020-11-20T11:31:30.316 INFO:teuthology.orchestra.run.smithi155.stdout: osd op complaint time = 180 2020-11-20T11:31:30.317 INFO:teuthology.orchestra.run.smithi155.stdout: rados mon op timeout = 15m 2020-11-20T11:31:30.317 INFO:teuthology.orchestra.run.smithi155.stdout: rados osd op timeout = 15m 2020-11-20T11:31:30.317 INFO:tasks.ceph:Creating admin key on mon.a... 2020-11-20T11:31:30.318 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2020-11-20T11:31:30.378 INFO:tasks.ceph:Copying monmap to all nodes... 2020-11-20T11:31:30.379 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-11-20T11:31:30.379 INFO:teuthology.orchestra.run.smithi023:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2020-11-20T11:31:30.398 INFO:teuthology.orchestra.run.smithi023.stderr:+ dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2020-11-20T11:31:30.399 INFO:teuthology.orchestra.run.smithi023.stdout:[mon.] 2020-11-20T11:31:30.399 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQARqbdfpyczNhAA08QD0k92bqlYrrXQAQ3HLA== 2020-11-20T11:31:30.399 INFO:teuthology.orchestra.run.smithi023.stdout:[client.admin] 2020-11-20T11:31:30.400 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQASqbdf7WVSFhAA6IuAz2QW5rlmW86Zu4InGQ== 2020-11-20T11:31:30.400 INFO:teuthology.orchestra.run.smithi023.stdout: caps mds = "allow *" 2020-11-20T11:31:30.400 INFO:teuthology.orchestra.run.smithi023.stdout: caps mgr = "allow *" 2020-11-20T11:31:30.400 INFO:teuthology.orchestra.run.smithi023.stdout: caps mon = "allow *" 2020-11-20T11:31:30.400 INFO:teuthology.orchestra.run.smithi023.stdout: caps osd = "allow *" 2020-11-20T11:31:30.401 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2020-11-20T11:31:30.401 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2020-11-20T11:31:30.401 INFO:teuthology.orchestra.run.smithi023.stderr:206 bytes copied, 3.3777e-05 s, 6.1 MB/s 2020-11-20T11:31:30.402 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-11-20T11:31:30.402 INFO:teuthology.orchestra.run.smithi023:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2020-11-20T11:31:30.469 INFO:teuthology.orchestra.run.smithi023.stderr:+ dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2020-11-20T11:31:30.470 INFO:teuthology.orchestra.run.smithi023.stdout: �K�9J���B�G����_�ΰ��_�ΰ�aXa ���bXb �����cXc �����abc 2020-11-20T11:31:30.471 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2020-11-20T11:31:30.471 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2020-11-20T11:31:30.471 INFO:teuthology.orchestra.run.smithi023.stderr:409 bytes copied, 3.1356e-05 s, 13.0 MB/s 2020-11-20T11:31:30.472 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi023.front.sepia.ceph.com 2020-11-20T11:31:30.472 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-11-20T11:31:30.473 INFO:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/ceph.keyring 2020-11-20T11:31:30.473 INFO:teuthology.orchestra.run.smithi023:> sudo chmod 0644 /etc/ceph/ceph.keyring 2020-11-20T11:31:30.529 INFO:teuthology.orchestra.run.smithi023.stderr:+ sudo dd of=/etc/ceph/ceph.keyring 2020-11-20T11:31:30.543 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2020-11-20T11:31:30.545 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2020-11-20T11:31:30.545 INFO:teuthology.orchestra.run.smithi023.stderr:206 bytes copied, 5.1147e-05 s, 4.0 MB/s 2020-11-20T11:31:30.545 INFO:teuthology.orchestra.run.smithi023.stderr:+ sudo chmod 0644 /etc/ceph/ceph.keyring 2020-11-20T11:31:30.551 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-11-20T11:31:30.552 INFO:teuthology.orchestra.run.smithi023:> dd of=/home/ubuntu/cephtest/ceph.monmap 2020-11-20T11:31:30.611 INFO:teuthology.orchestra.run.smithi023.stderr:+ dd of=/home/ubuntu/cephtest/ceph.monmap 2020-11-20T11:31:30.612 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2020-11-20T11:31:30.612 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2020-11-20T11:31:30.612 INFO:teuthology.orchestra.run.smithi023.stderr:409 bytes copied, 5.7997e-05 s, 7.1 MB/s 2020-11-20T11:31:30.613 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi155.front.sepia.ceph.com 2020-11-20T11:31:30.614 INFO:teuthology.orchestra.run.smithi155:> set -ex 2020-11-20T11:31:30.614 INFO:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/ceph/ceph.keyring 2020-11-20T11:31:30.614 INFO:teuthology.orchestra.run.smithi155:> sudo chmod 0644 /etc/ceph/ceph.keyring 2020-11-20T11:31:30.638 INFO:teuthology.orchestra.run.smithi155.stderr:+ sudo dd of=/etc/ceph/ceph.keyring 2020-11-20T11:31:30.648 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records in 2020-11-20T11:31:30.648 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records out 2020-11-20T11:31:30.649 INFO:teuthology.orchestra.run.smithi155.stderr:206 bytes copied, 2.9644e-05 s, 6.9 MB/s 2020-11-20T11:31:30.649 INFO:teuthology.orchestra.run.smithi155.stderr:+ sudo chmod 0644 /etc/ceph/ceph.keyring 2020-11-20T11:31:30.659 INFO:teuthology.orchestra.run.smithi155:> set -ex 2020-11-20T11:31:30.660 INFO:teuthology.orchestra.run.smithi155:> dd of=/home/ubuntu/cephtest/ceph.monmap 2020-11-20T11:31:30.726 INFO:teuthology.orchestra.run.smithi155.stderr:+ dd of=/home/ubuntu/cephtest/ceph.monmap 2020-11-20T11:31:30.727 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records in 2020-11-20T11:31:30.728 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records out 2020-11-20T11:31:30.729 INFO:teuthology.orchestra.run.smithi155.stderr:409 bytes copied, 0.000277291 s, 1.5 MB/s 2020-11-20T11:31:30.730 INFO:tasks.ceph:Setting up mon nodes... 2020-11-20T11:31:30.730 INFO:tasks.ceph:Setting up mgr nodes... 2020-11-20T11:31:30.730 INFO:teuthology.orchestra.run.smithi023:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2020-11-20T11:31:30.786 INFO:teuthology.orchestra.run.smithi023.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2020-11-20T11:31:30.788 INFO:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2020-11-20T11:31:30.841 INFO:teuthology.orchestra.run.smithi155.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2020-11-20T11:31:30.844 INFO:tasks.ceph:Setting up mds nodes... 2020-11-20T11:31:30.845 INFO:teuthology.orchestra.run.smithi023:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2020-11-20T11:31:30.900 INFO:teuthology.orchestra.run.smithi023.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2020-11-20T11:31:30.903 INFO:teuthology.orchestra.run.smithi023:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2020-11-20T11:31:30.931 INFO:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2020-11-20T11:31:30.980 INFO:teuthology.orchestra.run.smithi155.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2020-11-20T11:31:30.983 INFO:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2020-11-20T11:31:31.010 INFO:tasks.ceph_client:Setting up client nodes... 2020-11-20T11:31:31.011 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2020-11-20T11:31:31.050 INFO:teuthology.orchestra.run.smithi023.stdout:creating /etc/ceph/ceph.client.0.keyring 2020-11-20T11:31:31.062 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-11-20T11:31:31.063 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi023.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi155.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2020-11-20T11:31:31.063 INFO:teuthology.orchestra.run.smithi023:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2020-11-20T11:31:31.127 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2020-11-20T11:31:31.127 INFO:tasks.ceph:role: osd.0 2020-11-20T11:31:31.128 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi023.front.sepia.ceph.com 2020-11-20T11:31:31.128 INFO:teuthology.orchestra.run.smithi023:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-11-20T11:31:31.220 INFO:teuthology.orchestra.run.smithi023.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-11-20T11:31:31.220 INFO:teuthology.orchestra.run.smithi023.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-20T11:31:31.220 INFO:teuthology.orchestra.run.smithi023.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-11-20T11:31:31.221 INFO:teuthology.orchestra.run.smithi023.stdout: = reflink=1 2020-11-20T11:31:31.221 INFO:teuthology.orchestra.run.smithi023.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-20T11:31:31.221 INFO:teuthology.orchestra.run.smithi023.stdout: = sunit=0 swidth=0 blks 2020-11-20T11:31:31.221 INFO:teuthology.orchestra.run.smithi023.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-11-20T11:31:31.221 INFO:teuthology.orchestra.run.smithi023.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-20T11:31:31.222 INFO:teuthology.orchestra.run.smithi023.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-20T11:31:31.222 INFO:teuthology.orchestra.run.smithi023.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-20T11:31:32.479 INFO:teuthology.orchestra.run.smithi023.stdout:Discarding blocks...Done. 2020-11-20T11:31:32.481 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi023.front.sepia.ceph.com -o noatime 2020-11-20T11:31:32.481 INFO:teuthology.orchestra.run.smithi023:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2020-11-20T11:31:32.525 INFO:teuthology.orchestra.run.smithi023:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2020-11-20T11:31:32.597 INFO:teuthology.orchestra.run.smithi023:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2020-11-20T11:31:32.665 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2020-11-20T11:31:32.665 INFO:tasks.ceph:role: osd.1 2020-11-20T11:31:32.665 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi023.front.sepia.ceph.com 2020-11-20T11:31:32.666 INFO:teuthology.orchestra.run.smithi023:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-11-20T11:31:32.759 INFO:teuthology.orchestra.run.smithi023.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-11-20T11:31:33.064 INFO:teuthology.orchestra.run.smithi023.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-20T11:31:33.064 INFO:teuthology.orchestra.run.smithi023.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-11-20T11:31:33.065 INFO:teuthology.orchestra.run.smithi023.stdout: = reflink=1 2020-11-20T11:31:33.065 INFO:teuthology.orchestra.run.smithi023.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-20T11:31:33.065 INFO:teuthology.orchestra.run.smithi023.stdout: = sunit=0 swidth=0 blks 2020-11-20T11:31:33.065 INFO:teuthology.orchestra.run.smithi023.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-11-20T11:31:33.065 INFO:teuthology.orchestra.run.smithi023.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-20T11:31:33.066 INFO:teuthology.orchestra.run.smithi023.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-20T11:31:33.066 INFO:teuthology.orchestra.run.smithi023.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-20T11:31:34.017 INFO:teuthology.orchestra.run.smithi023.stdout:Discarding blocks...Done. 2020-11-20T11:31:34.019 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi023.front.sepia.ceph.com -o noatime 2020-11-20T11:31:34.019 INFO:teuthology.orchestra.run.smithi023:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2020-11-20T11:31:34.069 INFO:teuthology.orchestra.run.smithi023:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2020-11-20T11:31:34.143 INFO:teuthology.orchestra.run.smithi023:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2020-11-20T11:31:34.215 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2020-11-20T11:31:34.216 INFO:tasks.ceph:role: osd.2 2020-11-20T11:31:34.216 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi023.front.sepia.ceph.com 2020-11-20T11:31:34.216 INFO:teuthology.orchestra.run.smithi023:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-11-20T11:31:34.315 INFO:teuthology.orchestra.run.smithi023.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-11-20T11:31:34.315 INFO:teuthology.orchestra.run.smithi023.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-20T11:31:34.316 INFO:teuthology.orchestra.run.smithi023.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-11-20T11:31:34.316 INFO:teuthology.orchestra.run.smithi023.stdout: = reflink=1 2020-11-20T11:31:34.316 INFO:teuthology.orchestra.run.smithi023.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-20T11:31:34.316 INFO:teuthology.orchestra.run.smithi023.stdout: = sunit=0 swidth=0 blks 2020-11-20T11:31:34.316 INFO:teuthology.orchestra.run.smithi023.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-11-20T11:31:34.317 INFO:teuthology.orchestra.run.smithi023.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-20T11:31:34.317 INFO:teuthology.orchestra.run.smithi023.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-20T11:31:34.317 INFO:teuthology.orchestra.run.smithi023.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-20T11:31:35.572 INFO:teuthology.orchestra.run.smithi023.stdout:Discarding blocks...Done. 2020-11-20T11:31:35.574 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi023.front.sepia.ceph.com -o noatime 2020-11-20T11:31:35.574 INFO:teuthology.orchestra.run.smithi023:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2020-11-20T11:31:35.617 INFO:teuthology.orchestra.run.smithi023:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2020-11-20T11:31:35.690 INFO:teuthology.orchestra.run.smithi023:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2020-11-20T11:31:35.758 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2020-11-20T11:31:35.759 INFO:tasks.ceph:role: osd.3 2020-11-20T11:31:35.759 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi023.front.sepia.ceph.com 2020-11-20T11:31:35.759 INFO:teuthology.orchestra.run.smithi023:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2020-11-20T11:31:35.853 INFO:teuthology.orchestra.run.smithi023.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2020-11-20T11:31:35.853 INFO:teuthology.orchestra.run.smithi023.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-20T11:31:35.853 INFO:teuthology.orchestra.run.smithi023.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-11-20T11:31:35.854 INFO:teuthology.orchestra.run.smithi023.stdout: = reflink=1 2020-11-20T11:31:35.854 INFO:teuthology.orchestra.run.smithi023.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-20T11:31:35.854 INFO:teuthology.orchestra.run.smithi023.stdout: = sunit=0 swidth=0 blks 2020-11-20T11:31:35.854 INFO:teuthology.orchestra.run.smithi023.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-11-20T11:31:35.855 INFO:teuthology.orchestra.run.smithi023.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-20T11:31:35.855 INFO:teuthology.orchestra.run.smithi023.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-20T11:31:35.855 INFO:teuthology.orchestra.run.smithi023.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-20T11:31:37.121 INFO:teuthology.orchestra.run.smithi023.stdout:Discarding blocks...Done. 2020-11-20T11:31:37.123 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi023.front.sepia.ceph.com -o noatime 2020-11-20T11:31:37.123 INFO:teuthology.orchestra.run.smithi023:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2020-11-20T11:31:37.166 INFO:teuthology.orchestra.run.smithi023:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2020-11-20T11:31:37.238 INFO:teuthology.orchestra.run.smithi023:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-11-20T11:31:37.324 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:31:37.321+0000 7efd6216ef40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2020-11-20T11:31:37.324 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:31:37.321+0000 7efd6216ef40 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2020-11-20T11:31:37.324 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:31:37.321+0000 7efd6216ef40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2020-11-20T11:31:37.325 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:31:37.321+0000 7efd6216ef40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2020-11-20T11:31:37.325 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:31:37.321+0000 7efd6216ef40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2020-11-20T11:31:37.325 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:31:37.322+0000 7efd6216ef40 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2020-11-20T11:31:40.187 INFO:teuthology.orchestra.run.smithi023:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2020-11-20T11:31:40.216 INFO:teuthology.orchestra.run.smithi023:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-11-20T11:31:40.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:31:40.302+0000 7f427d75bf40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2020-11-20T11:31:40.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:31:40.303+0000 7f427d75bf40 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2020-11-20T11:31:40.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:31:40.303+0000 7f427d75bf40 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2020-11-20T11:31:40.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:31:40.303+0000 7f427d75bf40 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2020-11-20T11:31:40.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:31:40.303+0000 7f427d75bf40 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2020-11-20T11:31:40.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:31:40.304+0000 7f427d75bf40 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2020-11-20T11:31:43.175 INFO:teuthology.orchestra.run.smithi023:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2020-11-20T11:31:43.205 INFO:teuthology.orchestra.run.smithi023:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-11-20T11:31:43.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:31:43.289+0000 7fdf3211af40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2020-11-20T11:31:43.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:31:43.290+0000 7fdf3211af40 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2020-11-20T11:31:43.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:31:43.290+0000 7fdf3211af40 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2020-11-20T11:31:43.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:31:43.290+0000 7fdf3211af40 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2020-11-20T11:31:43.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:31:43.290+0000 7fdf3211af40 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2020-11-20T11:31:43.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:31:43.291+0000 7fdf3211af40 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2020-11-20T11:31:46.161 INFO:teuthology.orchestra.run.smithi023:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2020-11-20T11:31:46.191 INFO:teuthology.orchestra.run.smithi023:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-11-20T11:31:46.279 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:31:46.275+0000 7f3cf7bccf40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2020-11-20T11:31:46.279 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:31:46.275+0000 7f3cf7bccf40 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2020-11-20T11:31:46.279 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:31:46.275+0000 7f3cf7bccf40 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2020-11-20T11:31:46.279 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:31:46.275+0000 7f3cf7bccf40 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2020-11-20T11:31:46.280 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:31:46.275+0000 7f3cf7bccf40 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2020-11-20T11:31:46.280 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:31:46.276+0000 7f3cf7bccf40 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2020-11-20T11:31:49.134 INFO:teuthology.orchestra.run.smithi023:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2020-11-20T11:31:49.166 INFO:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2020-11-20T11:31:49.197 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2020-11-20T11:31:49.197 INFO:tasks.ceph:role: osd.4 2020-11-20T11:31:49.198 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi155.front.sepia.ceph.com 2020-11-20T11:31:49.198 INFO:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-11-20T11:31:49.272 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-11-20T11:31:49.273 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-20T11:31:49.273 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-11-20T11:31:49.273 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 2020-11-20T11:31:49.273 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-20T11:31:49.273 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2020-11-20T11:31:49.274 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-11-20T11:31:49.274 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-20T11:31:49.274 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-20T11:31:49.274 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-20T11:31:49.638 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2020-11-20T11:31:49.640 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2020-11-20T11:31:49.640 INFO:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2020-11-20T11:31:49.756 INFO:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2020-11-20T11:31:49.788 INFO:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2020-11-20T11:31:49.854 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2020-11-20T11:31:49.855 INFO:tasks.ceph:role: osd.5 2020-11-20T11:31:49.855 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi155.front.sepia.ceph.com 2020-11-20T11:31:49.855 INFO:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-11-20T11:31:49.927 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-11-20T11:31:49.928 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-20T11:31:49.928 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-11-20T11:31:49.928 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 2020-11-20T11:31:49.928 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-20T11:31:49.928 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2020-11-20T11:31:49.929 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-11-20T11:31:49.929 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-20T11:31:49.929 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-20T11:31:49.929 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-20T11:31:50.291 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2020-11-20T11:31:50.292 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2020-11-20T11:31:50.293 INFO:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2020-11-20T11:31:50.386 INFO:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2020-11-20T11:31:50.417 INFO:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2020-11-20T11:31:50.483 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2020-11-20T11:31:50.483 INFO:tasks.ceph:role: osd.6 2020-11-20T11:31:50.483 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi155.front.sepia.ceph.com 2020-11-20T11:31:50.484 INFO:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-11-20T11:31:50.555 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-11-20T11:31:50.555 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-20T11:31:50.556 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-11-20T11:31:50.556 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 2020-11-20T11:31:50.556 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-20T11:31:50.556 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2020-11-20T11:31:50.556 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-11-20T11:31:50.557 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-20T11:31:50.557 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-20T11:31:50.557 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-20T11:31:50.921 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2020-11-20T11:31:50.923 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2020-11-20T11:31:50.923 INFO:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2020-11-20T11:31:51.030 INFO:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2020-11-20T11:31:51.060 INFO:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2020-11-20T11:31:51.126 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2020-11-20T11:31:51.127 INFO:tasks.ceph:role: osd.7 2020-11-20T11:31:51.127 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi155.front.sepia.ceph.com 2020-11-20T11:31:51.128 INFO:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2020-11-20T11:31:51.200 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2020-11-20T11:31:51.201 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2020-11-20T11:31:51.201 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-11-20T11:31:51.201 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 2020-11-20T11:31:51.201 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-11-20T11:31:51.201 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2020-11-20T11:31:51.202 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-11-20T11:31:51.202 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-11-20T11:31:51.202 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-11-20T11:31:51.202 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-11-20T11:31:51.575 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2020-11-20T11:31:51.577 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2020-11-20T11:31:51.577 INFO:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2020-11-20T11:31:51.701 INFO:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2020-11-20T11:31:51.731 INFO:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-11-20T11:31:51.818 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:31:51.815+0000 7efc83ee4f40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2020-11-20T11:31:51.818 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:31:51.816+0000 7efc83ee4f40 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2020-11-20T11:31:51.819 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:31:51.816+0000 7efc83ee4f40 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2020-11-20T11:31:51.819 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:31:51.816+0000 7efc83ee4f40 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2020-11-20T11:31:51.819 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:31:51.816+0000 7efc83ee4f40 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2020-11-20T11:31:51.822 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:31:51.820+0000 7efc83ee4f40 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2020-11-20T11:31:54.669 INFO:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2020-11-20T11:31:54.704 INFO:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-11-20T11:31:54.795 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:31:54.792+0000 7fe959389f40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2020-11-20T11:31:54.796 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:31:54.792+0000 7fe959389f40 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2020-11-20T11:31:54.796 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:31:54.792+0000 7fe959389f40 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2020-11-20T11:31:54.796 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:31:54.792+0000 7fe959389f40 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2020-11-20T11:31:54.796 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:31:54.792+0000 7fe959389f40 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2020-11-20T11:31:54.797 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:31:54.793+0000 7fe959389f40 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2020-11-20T11:31:57.641 INFO:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2020-11-20T11:31:57.721 INFO:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-11-20T11:31:57.820 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:31:57.816+0000 7fcc48874f40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2020-11-20T11:31:57.820 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:31:57.816+0000 7fcc48874f40 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2020-11-20T11:31:57.821 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:31:57.816+0000 7fcc48874f40 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2020-11-20T11:31:57.821 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:31:57.816+0000 7fcc48874f40 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2020-11-20T11:31:57.822 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:31:57.816+0000 7fcc48874f40 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2020-11-20T11:31:57.822 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:31:57.816+0000 7fcc48874f40 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2020-11-20T11:31:59.119 INFO:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:31:59.123 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:32:00.664 INFO:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2020-11-20T11:32:00.708 INFO:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-11-20T11:32:00.799 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:00.796+0000 7f0aea4fef40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2020-11-20T11:32:00.800 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:00.796+0000 7f0aea4fef40 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2020-11-20T11:32:00.800 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:00.796+0000 7f0aea4fef40 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2020-11-20T11:32:00.800 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:00.796+0000 7f0aea4fef40 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2020-11-20T11:32:00.801 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:00.796+0000 7f0aea4fef40 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2020-11-20T11:32:00.801 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:00.797+0000 7f0aea4fef40 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2020-11-20T11:32:03.649 INFO:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2020-11-20T11:32:03.718 INFO:tasks.ceph:Reading keys from all nodes... 2020-11-20T11:32:03.719 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-11-20T11:32:03.719 INFO:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2020-11-20T11:32:03.736 INFO:teuthology.orchestra.run.smithi023.stderr:+ sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2020-11-20T11:32:03.747 INFO:teuthology.orchestra.run.smithi023.stdout:[mgr.y] 2020-11-20T11:32:03.748 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQASqbdf0FfJLhAASBTmHtMs/ocNYcjOPuWHCQ== 2020-11-20T11:32:03.748 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2020-11-20T11:32:03.748 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2020-11-20T11:32:03.749 INFO:teuthology.orchestra.run.smithi023.stderr:56 bytes copied, 4.8533e-05 s, 1.2 MB/s 2020-11-20T11:32:03.750 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-11-20T11:32:03.750 INFO:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2020-11-20T11:32:03.805 INFO:teuthology.orchestra.run.smithi023.stderr:+ sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2020-11-20T11:32:03.817 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.a] 2020-11-20T11:32:03.817 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQASqbdfIcqJNRAAB2dkRB0zc/LSHKPw78nGaw== 2020-11-20T11:32:03.817 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2020-11-20T11:32:03.818 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2020-11-20T11:32:03.818 INFO:teuthology.orchestra.run.smithi023.stderr:56 bytes copied, 4.7455e-05 s, 1.2 MB/s 2020-11-20T11:32:03.819 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-11-20T11:32:03.819 INFO:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2020-11-20T11:32:03.874 INFO:teuthology.orchestra.run.smithi023.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2020-11-20T11:32:03.884 INFO:teuthology.orchestra.run.smithi023.stdout:[osd.0] 2020-11-20T11:32:03.885 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQAZqbdfyiAzExAAN0O+njIKtDW6LoQiEoomcQ== 2020-11-20T11:32:03.885 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2020-11-20T11:32:03.885 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2020-11-20T11:32:03.885 INFO:teuthology.orchestra.run.smithi023.stderr:56 bytes copied, 3.51e-05 s, 1.6 MB/s 2020-11-20T11:32:03.886 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-11-20T11:32:03.887 INFO:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2020-11-20T11:32:03.941 INFO:teuthology.orchestra.run.smithi023.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2020-11-20T11:32:03.952 INFO:teuthology.orchestra.run.smithi023.stdout:[osd.1] 2020-11-20T11:32:03.952 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQAcqbdfLFAbEhAA6H9fek0kEnsCh329Rth2Qg== 2020-11-20T11:32:03.952 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2020-11-20T11:32:03.953 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2020-11-20T11:32:03.953 INFO:teuthology.orchestra.run.smithi023.stderr:56 bytes copied, 3.6021e-05 s, 1.6 MB/s 2020-11-20T11:32:03.954 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-11-20T11:32:03.954 INFO:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2020-11-20T11:32:04.008 INFO:teuthology.orchestra.run.smithi023.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2020-11-20T11:32:04.019 INFO:teuthology.orchestra.run.smithi023.stdout:[osd.2] 2020-11-20T11:32:04.019 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQAfqbdfYj9SERAAumOnsjNapgKIPMv/zbsL2g== 2020-11-20T11:32:04.020 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2020-11-20T11:32:04.020 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2020-11-20T11:32:04.020 INFO:teuthology.orchestra.run.smithi023.stderr:56 bytes copied, 3.5661e-05 s, 1.6 MB/s 2020-11-20T11:32:04.021 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-11-20T11:32:04.021 INFO:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2020-11-20T11:32:04.076 INFO:teuthology.orchestra.run.smithi023.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2020-11-20T11:32:04.086 INFO:teuthology.orchestra.run.smithi023.stdout:[osd.3] 2020-11-20T11:32:04.087 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQAiqbdfQHB5EBAAoWkKg53LLVcubouO64AWrg== 2020-11-20T11:32:04.087 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2020-11-20T11:32:04.087 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2020-11-20T11:32:04.087 INFO:teuthology.orchestra.run.smithi023.stderr:56 bytes copied, 3.4594e-05 s, 1.6 MB/s 2020-11-20T11:32:04.088 INFO:teuthology.orchestra.run.smithi155:> set -ex 2020-11-20T11:32:04.089 INFO:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2020-11-20T11:32:04.105 INFO:teuthology.orchestra.run.smithi155.stderr:+ sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2020-11-20T11:32:04.116 INFO:teuthology.orchestra.run.smithi155.stdout:[mgr.x] 2020-11-20T11:32:04.117 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQASqbdfGIcOMhAA+QNT6V30ZxEWIcVnUBBr/A== 2020-11-20T11:32:04.117 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records in 2020-11-20T11:32:04.117 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records out 2020-11-20T11:32:04.117 INFO:teuthology.orchestra.run.smithi155.stderr:56 bytes copied, 4.0038e-05 s, 1.4 MB/s 2020-11-20T11:32:04.118 INFO:teuthology.orchestra.run.smithi155:> set -ex 2020-11-20T11:32:04.119 INFO:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2020-11-20T11:32:04.174 INFO:teuthology.orchestra.run.smithi155.stderr:+ sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2020-11-20T11:32:04.185 INFO:teuthology.orchestra.run.smithi155.stdout:[mds.b] 2020-11-20T11:32:04.186 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQASqbdfm+pYOhAAmTSR3iEn+xKVhorJ6Rls8A== 2020-11-20T11:32:04.186 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records in 2020-11-20T11:32:04.186 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records out 2020-11-20T11:32:04.186 INFO:teuthology.orchestra.run.smithi155.stderr:56 bytes copied, 3.9612e-05 s, 1.4 MB/s 2020-11-20T11:32:04.187 INFO:teuthology.orchestra.run.smithi155:> set -ex 2020-11-20T11:32:04.188 INFO:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2020-11-20T11:32:04.243 INFO:teuthology.orchestra.run.smithi155.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2020-11-20T11:32:04.253 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.4] 2020-11-20T11:32:04.253 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQAnqbdfpFimMBAAk7A2hLUqL1g8ywUDyaSWtQ== 2020-11-20T11:32:04.254 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records in 2020-11-20T11:32:04.254 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records out 2020-11-20T11:32:04.254 INFO:teuthology.orchestra.run.smithi155.stderr:56 bytes copied, 3.0467e-05 s, 1.8 MB/s 2020-11-20T11:32:04.255 INFO:teuthology.orchestra.run.smithi155:> set -ex 2020-11-20T11:32:04.255 INFO:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2020-11-20T11:32:04.309 INFO:teuthology.orchestra.run.smithi155.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2020-11-20T11:32:04.319 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.5] 2020-11-20T11:32:04.320 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQAqqbdfwdZOLxAAiF9hUUZXl8niis3C2yqRKA== 2020-11-20T11:32:04.320 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records in 2020-11-20T11:32:04.320 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records out 2020-11-20T11:32:04.320 INFO:teuthology.orchestra.run.smithi155.stderr:56 bytes copied, 2.9706e-05 s, 1.9 MB/s 2020-11-20T11:32:04.321 INFO:teuthology.orchestra.run.smithi155:> set -ex 2020-11-20T11:32:04.322 INFO:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2020-11-20T11:32:04.376 INFO:teuthology.orchestra.run.smithi155.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2020-11-20T11:32:04.386 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.6] 2020-11-20T11:32:04.386 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQAtqbdf4bS3MBAAir0+b0SX5QzsHEClgfu5Yg== 2020-11-20T11:32:04.386 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records in 2020-11-20T11:32:04.387 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records out 2020-11-20T11:32:04.387 INFO:teuthology.orchestra.run.smithi155.stderr:56 bytes copied, 2.998e-05 s, 1.9 MB/s 2020-11-20T11:32:04.388 INFO:teuthology.orchestra.run.smithi155:> set -ex 2020-11-20T11:32:04.388 INFO:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2020-11-20T11:32:04.442 INFO:teuthology.orchestra.run.smithi155.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2020-11-20T11:32:04.451 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.7] 2020-11-20T11:32:04.452 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQAwqbdfWU6NLxAA2LkNYuGslGZZ/ySIC716mA== 2020-11-20T11:32:04.452 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records in 2020-11-20T11:32:04.452 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records out 2020-11-20T11:32:04.452 INFO:teuthology.orchestra.run.smithi155.stderr:56 bytes copied, 2.7311e-05 s, 2.1 MB/s 2020-11-20T11:32:04.453 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-11-20T11:32:04.455 INFO:teuthology.orchestra.run.smithi023:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2020-11-20T11:32:04.470 INFO:teuthology.orchestra.run.smithi023.stderr:+ dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2020-11-20T11:32:04.471 INFO:teuthology.orchestra.run.smithi023.stdout:[client.0] 2020-11-20T11:32:04.472 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQATqbdfU/7hAhAAA4gTdioQs8bBi0g2WEs6zQ== 2020-11-20T11:32:04.472 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2020-11-20T11:32:04.472 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2020-11-20T11:32:04.472 INFO:teuthology.orchestra.run.smithi023.stderr:59 bytes copied, 3.765e-05 s, 1.6 MB/s 2020-11-20T11:32:04.473 INFO:tasks.ceph:Adding keys to all mons... 2020-11-20T11:32:04.473 INFO:teuthology.orchestra.run.smithi023:> sudo tee -a /etc/ceph/ceph.keyring 2020-11-20T11:32:04.514 INFO:teuthology.orchestra.run.smithi155:> sudo tee -a /etc/ceph/ceph.keyring 2020-11-20T11:32:04.537 INFO:teuthology.orchestra.run.smithi155.stdout:[mgr.y] 2020-11-20T11:32:04.537 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQASqbdf0FfJLhAASBTmHtMs/ocNYcjOPuWHCQ== 2020-11-20T11:32:04.537 INFO:teuthology.orchestra.run.smithi155.stdout:[mds.a] 2020-11-20T11:32:04.538 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQASqbdfIcqJNRAAB2dkRB0zc/LSHKPw78nGaw== 2020-11-20T11:32:04.538 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.0] 2020-11-20T11:32:04.538 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQAZqbdfyiAzExAAN0O+njIKtDW6LoQiEoomcQ== 2020-11-20T11:32:04.538 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.1] 2020-11-20T11:32:04.539 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQAcqbdfLFAbEhAA6H9fek0kEnsCh329Rth2Qg== 2020-11-20T11:32:04.539 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.2] 2020-11-20T11:32:04.539 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQAfqbdfYj9SERAAumOnsjNapgKIPMv/zbsL2g== 2020-11-20T11:32:04.539 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.3] 2020-11-20T11:32:04.539 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQAiqbdfQHB5EBAAoWkKg53LLVcubouO64AWrg== 2020-11-20T11:32:04.540 INFO:teuthology.orchestra.run.smithi155.stdout:[mgr.x] 2020-11-20T11:32:04.540 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQASqbdfGIcOMhAA+QNT6V30ZxEWIcVnUBBr/A== 2020-11-20T11:32:04.540 INFO:teuthology.orchestra.run.smithi155.stdout:[mds.b] 2020-11-20T11:32:04.540 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQASqbdfm+pYOhAAmTSR3iEn+xKVhorJ6Rls8A== 2020-11-20T11:32:04.540 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.4] 2020-11-20T11:32:04.541 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQAnqbdfpFimMBAAk7A2hLUqL1g8ywUDyaSWtQ== 2020-11-20T11:32:04.541 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.5] 2020-11-20T11:32:04.541 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQAqqbdfwdZOLxAAiF9hUUZXl8niis3C2yqRKA== 2020-11-20T11:32:04.541 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.6] 2020-11-20T11:32:04.541 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQAtqbdf4bS3MBAAir0+b0SX5QzsHEClgfu5Yg== 2020-11-20T11:32:04.542 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.7] 2020-11-20T11:32:04.542 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQAwqbdfWU6NLxAA2LkNYuGslGZZ/ySIC716mA== 2020-11-20T11:32:04.542 INFO:teuthology.orchestra.run.smithi155.stdout:[client.0] 2020-11-20T11:32:04.542 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQATqbdfU/7hAhAAA4gTdioQs8bBi0g2WEs6zQ== 2020-11-20T11:32:04.543 INFO:teuthology.orchestra.run.smithi023.stdout:[mgr.y] 2020-11-20T11:32:04.544 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQASqbdf0FfJLhAASBTmHtMs/ocNYcjOPuWHCQ== 2020-11-20T11:32:04.544 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.a] 2020-11-20T11:32:04.544 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQASqbdfIcqJNRAAB2dkRB0zc/LSHKPw78nGaw== 2020-11-20T11:32:04.544 INFO:teuthology.orchestra.run.smithi023.stdout:[osd.0] 2020-11-20T11:32:04.544 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQAZqbdfyiAzExAAN0O+njIKtDW6LoQiEoomcQ== 2020-11-20T11:32:04.545 INFO:teuthology.orchestra.run.smithi023.stdout:[osd.1] 2020-11-20T11:32:04.545 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQAcqbdfLFAbEhAA6H9fek0kEnsCh329Rth2Qg== 2020-11-20T11:32:04.545 INFO:teuthology.orchestra.run.smithi023.stdout:[osd.2] 2020-11-20T11:32:04.545 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQAfqbdfYj9SERAAumOnsjNapgKIPMv/zbsL2g== 2020-11-20T11:32:04.546 INFO:teuthology.orchestra.run.smithi023.stdout:[osd.3] 2020-11-20T11:32:04.546 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQAiqbdfQHB5EBAAoWkKg53LLVcubouO64AWrg== 2020-11-20T11:32:04.546 INFO:teuthology.orchestra.run.smithi023.stdout:[mgr.x] 2020-11-20T11:32:04.546 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQASqbdfGIcOMhAA+QNT6V30ZxEWIcVnUBBr/A== 2020-11-20T11:32:04.546 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.b] 2020-11-20T11:32:04.547 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQASqbdfm+pYOhAAmTSR3iEn+xKVhorJ6Rls8A== 2020-11-20T11:32:04.547 INFO:teuthology.orchestra.run.smithi023.stdout:[osd.4] 2020-11-20T11:32:04.547 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQAnqbdfpFimMBAAk7A2hLUqL1g8ywUDyaSWtQ== 2020-11-20T11:32:04.547 INFO:teuthology.orchestra.run.smithi023.stdout:[osd.5] 2020-11-20T11:32:04.547 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQAqqbdfwdZOLxAAiF9hUUZXl8niis3C2yqRKA== 2020-11-20T11:32:04.548 INFO:teuthology.orchestra.run.smithi023.stdout:[osd.6] 2020-11-20T11:32:04.548 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQAtqbdf4bS3MBAAir0+b0SX5QzsHEClgfu5Yg== 2020-11-20T11:32:04.548 INFO:teuthology.orchestra.run.smithi023.stdout:[osd.7] 2020-11-20T11:32:04.548 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQAwqbdfWU6NLxAA2LkNYuGslGZZ/ySIC716mA== 2020-11-20T11:32:04.548 INFO:teuthology.orchestra.run.smithi023.stdout:[client.0] 2020-11-20T11:32:04.548 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQATqbdfU/7hAhAAA4gTdioQs8bBi0g2WEs6zQ== 2020-11-20T11:32:04.549 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-11-20T11:32:04.586 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-11-20T11:32:04.634 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2020-11-20T11:32:04.637 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2020-11-20T11:32:04.687 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T11:32:04.690 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T11:32:04.735 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T11:32:04.738 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T11:32:04.782 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T11:32:04.785 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T11:32:04.830 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T11:32:04.832 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T11:32:04.877 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-11-20T11:32:04.879 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-11-20T11:32:04.924 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2020-11-20T11:32:04.926 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2020-11-20T11:32:04.974 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T11:32:04.976 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T11:32:05.020 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T11:32:05.023 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T11:32:05.106 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T11:32:05.108 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T11:32:05.191 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T11:32:05.193 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-11-20T11:32:05.238 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-20T11:32:05.240 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-11-20T11:32:05.288 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-11-20T11:32:05.288 INFO:teuthology.orchestra.run.smithi023:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2020-11-20T11:32:05.313 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2020-11-20T11:32:05.414 INFO:teuthology.orchestra.run.smithi023:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2020-11-20T11:32:05.439 INFO:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2020-11-20T11:32:05.464 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2020-11-20T11:32:05.554 INFO:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2020-11-20T11:32:05.578 INFO:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2020-11-20T11:32:05.641 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2020-11-20T11:32:05.730 INFO:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2020-11-20T11:32:05.754 INFO:teuthology.orchestra.run.smithi023:> rm -- /home/ubuntu/cephtest/ceph.monmap 2020-11-20T11:32:05.757 INFO:teuthology.orchestra.run.smithi155:> rm -- /home/ubuntu/cephtest/ceph.monmap 2020-11-20T11:32:05.807 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2020-11-20T11:32:05.808 INFO:tasks.ceph.mon.a:Restarting daemon 2020-11-20T11:32:05.809 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2020-11-20T11:32:05.811 INFO:tasks.ceph.mon.a:Started 2020-11-20T11:32:05.812 INFO:tasks.ceph.mon.b:Restarting daemon 2020-11-20T11:32:05.812 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2020-11-20T11:32:05.850 INFO:tasks.ceph.mon.b:Started 2020-11-20T11:32:05.850 INFO:tasks.ceph.mon.c:Restarting daemon 2020-11-20T11:32:05.851 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2020-11-20T11:32:05.853 INFO:tasks.ceph.mon.c:Started 2020-11-20T11:32:05.853 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2020-11-20T11:32:05.854 INFO:tasks.ceph.mgr.y:Restarting daemon 2020-11-20T11:32:05.854 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2020-11-20T11:32:05.856 INFO:tasks.ceph.mgr.y:Started 2020-11-20T11:32:05.857 INFO:tasks.ceph.mgr.x:Restarting daemon 2020-11-20T11:32:05.857 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2020-11-20T11:32:05.859 INFO:tasks.ceph.mgr.x:Started 2020-11-20T11:32:05.859 INFO:tasks.ceph:Setting crush tunables to default 2020-11-20T11:32:05.860 INFO:teuthology.orchestra.run.smithi023:> sudo ceph --cluster ceph osd crush tunables default 2020-11-20T11:32:05.985 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:05.981+0000 7fc719e17700 1 Processor -- start 2020-11-20T11:32:05.986 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:05.981+0000 7fc719e17700 1 -- start start 2020-11-20T11:32:05.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:05.982+0000 7fc719e17700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc714060ca0 0x7fc7140aa0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:05.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:05.982+0000 7fc719e17700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc714138ff0 0x7fc714139480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:05.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:05.982+0000 7fc719e17700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7140aa590 con 0x7fc7140608c0 2020-11-20T11:32:05.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:05.982+0000 7fc719e17700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7140ab300 con 0x7fc714060ca0 2020-11-20T11:32:05.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:05.982+0000 7fc719e17700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7140aabe0 con 0x7fc714138ff0 2020-11-20T11:32:05.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:05.982+0000 7fc712ffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc714060ca0 0x7fc7140aa0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:05.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:05.982+0000 7fc712ffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc714060ca0 0x7fc7140aa0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40130/0 (socket says 172.21.15.23:40130) 2020-11-20T11:32:05.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:05.982+0000 7fc712ffd700 1 -- 172.21.15.23:0/1440043293 learned_addr learned my addr 172.21.15.23:0/1440043293 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:05.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:05.982+0000 7fc712ffd700 1 -- 172.21.15.23:0/1440043293 >> v2:172.21.15.23:3300/0 conn(0x7fc714060ca0 msgr2=0x7fc7140aa0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:32:05.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:05.982+0000 7fc712ffd700 1 -- 172.21.15.23:0/1440043293 >> v2:172.21.15.23:3300/0 conn(0x7fc714060ca0 msgr2=0x7fc7140aa0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:32:05.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:05.982+0000 7fc712ffd700 1 --2- 172.21.15.23:0/1440043293 >> v2:172.21.15.23:3300/0 conn(0x7fc714060ca0 0x7fc7140aa0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:32:05.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:05.982+0000 7fc712ffd700 1 --2- 172.21.15.23:0/1440043293 >> v2:172.21.15.23:3300/0 conn(0x7fc714060ca0 0x7fc7140aa0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:32:05.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:05.982+0000 7fc713fff700 1 --2- 172.21.15.23:0/1440043293 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc714138ff0 0x7fc714139480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:05.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:05.983+0000 7fc713fff700 1 -- 172.21.15.23:0/1440043293 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc714138ff0 msgr2=0x7fc714139480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:32:05.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:05.983+0000 7fc713fff700 1 -- 172.21.15.23:0/1440043293 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc714138ff0 msgr2=0x7fc714139480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:32:05.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:05.983+0000 7fc713fff700 1 --2- 172.21.15.23:0/1440043293 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc714138ff0 0x7fc714139480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:32:05.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:05.983+0000 7fc713fff700 1 --2- 172.21.15.23:0/1440043293 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc714138ff0 0x7fc714139480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:32:06.152 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.148+0000 7fc7127fc700 1 -- 172.21.15.23:0/1440043293 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027510226 0 0) 0x7fc7140aa590 con 0x7fc7140608c0 2020-11-20T11:32:06.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.149+0000 7fc7127fc700 1 -- 172.21.15.23:0/1440043293 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc700003280 con 0x7fc7140608c0 2020-11-20T11:32:06.159 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.154+0000 7fc7127fc700 1 -- 172.21.15.23:0/1440043293 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2712795994 0 0) 0x7fc700003280 con 0x7fc7140608c0 2020-11-20T11:32:06.160 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.154+0000 7fc7127fc700 1 -- 172.21.15.23:0/1440043293 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc714138ff0 msgr2=0x7fc714139480 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:32:06.160 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.154+0000 7fc7127fc700 1 --2- 172.21.15.23:0/1440043293 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc714138ff0 0x7fc714139480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:06.160 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.154+0000 7fc7127fc700 1 -- 172.21.15.23:0/1440043293 >> v2:172.21.15.23:3300/0 conn(0x7fc714060ca0 msgr2=0x7fc7140aa0a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:32:06.160 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.154+0000 7fc7127fc700 1 --2- 172.21.15.23:0/1440043293 >> v2:172.21.15.23:3300/0 conn(0x7fc714060ca0 0x7fc7140aa0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:06.160 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.154+0000 7fc7127fc700 1 -- 172.21.15.23:0/1440043293 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7140aae80 con 0x7fc7140608c0 2020-11-20T11:32:06.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.154+0000 7fc7127fc700 1 -- 172.21.15.23:0/1440043293 <== mon.1 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7fc704001520 con 0x7fc7140608c0 2020-11-20T11:32:06.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.154+0000 7fc7127fc700 1 -- 172.21.15.23:0/1440043293 >> v1:172.21.15.155:6789/0 conn(0x7fc7140608c0 legacy=0x7fc7140a6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:06.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.154+0000 7fc7127fc700 1 --2- 172.21.15.23:0/1440043293 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc714138ff0 0x7fc7000037f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:06.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.154+0000 7fc7127fc700 1 --2- 172.21.15.23:0/1440043293 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc700003d50 0x7fc700004520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:06.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.154+0000 7fc7127fc700 1 --2- 172.21.15.23:0/1440043293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc700004a10 0x7fc700004e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:06.162 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.154+0000 7fc7127fc700 1 -- 172.21.15.23:0/1440043293 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7140ab300 con 0x7fc700004a10 2020-11-20T11:32:06.162 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.154+0000 7fc7127fc700 1 -- 172.21.15.23:0/1440043293 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7140aabe0 con 0x7fc700003d50 2020-11-20T11:32:06.162 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.154+0000 7fc7127fc700 1 -- 172.21.15.23:0/1440043293 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc700004310 con 0x7fc714138ff0 2020-11-20T11:32:06.162 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.155+0000 7fc713fff700 1 --2- 172.21.15.23:0/1440043293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc700004a10 0x7fc700004e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:06.162 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.155+0000 7fc7137fe700 1 --2- 172.21.15.23:0/1440043293 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc714138ff0 0x7fc7000037f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:06.163 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.155+0000 7fc712ffd700 1 --2- 172.21.15.23:0/1440043293 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc700003d50 0x7fc700004520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:06.163 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.155+0000 7fc7137fe700 1 -- 172.21.15.23:0/1440043293 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc700003d50 msgr2=0x7fc700004520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:06.163 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.155+0000 7fc7137fe700 1 --2- 172.21.15.23:0/1440043293 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc700003d50 0x7fc700004520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:06.163 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.155+0000 7fc7137fe700 1 -- 172.21.15.23:0/1440043293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc700004a10 msgr2=0x7fc700004e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:06.163 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.155+0000 7fc7137fe700 1 --2- 172.21.15.23:0/1440043293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc700004a10 0x7fc700004e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:06.164 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.155+0000 7fc7137fe700 1 -- 172.21.15.23:0/1440043293 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc700003280 con 0x7fc714138ff0 2020-11-20T11:32:06.164 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.155+0000 7fc713fff700 1 --2- 172.21.15.23:0/1440043293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc700004a10 0x7fc700004e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:32:06.164 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.155+0000 7fc7137fe700 1 --2- 172.21.15.23:0/1440043293 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc714138ff0 0x7fc7000037f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fc704007fb0 tx=0x7fc7140ab010).ready entity=mon.2 client_cookie=2d0836f838817d2d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:06.166 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.156+0000 7fc7127fc700 1 -- 172.21.15.23:0/1440043293 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70400cdf0 con 0x7fc714138ff0 2020-11-20T11:32:06.166 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.156+0000 7fc7127fc700 1 -- 172.21.15.23:0/1440043293 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc704011560 con 0x7fc714138ff0 2020-11-20T11:32:06.167 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.156+0000 7fc7127fc700 1 -- 172.21.15.23:0/1440043293 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7040116f0 con 0x7fc714138ff0 2020-11-20T11:32:06.167 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.157+0000 7fc719e17700 1 -- 172.21.15.23:0/1440043293 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc714138ff0 msgr2=0x7fc7000037f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:06.167 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.157+0000 7fc719e17700 1 --2- 172.21.15.23:0/1440043293 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc714138ff0 0x7fc7000037f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fc704007fb0 tx=0x7fc7140ab010).stop 2020-11-20T11:32:06.167 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.157+0000 7fc713fff700 1 -- 172.21.15.23:0/1440043293 reap_dead start 2020-11-20T11:32:06.168 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.157+0000 7fc719e17700 1 -- 172.21.15.23:0/1440043293 shutdown_connections 2020-11-20T11:32:06.168 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.157+0000 7fc719e17700 1 -- 172.21.15.23:0/1440043293 >> 172.21.15.23:0/1440043293 conn(0x7fc714066750 msgr2=0x7fc7140601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:06.168 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.157+0000 7fc719e17700 1 -- 172.21.15.23:0/1440043293 shutdown_connections 2020-11-20T11:32:06.168 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.157+0000 7fc719e17700 1 -- 172.21.15.23:0/1440043293 wait complete. 2020-11-20T11:32:06.168 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.159+0000 7fc719e17700 1 Processor -- start 2020-11-20T11:32:06.169 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.159+0000 7fc719e17700 1 -- start start 2020-11-20T11:32:06.169 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.159+0000 7fc719e17700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc714138ff0 0x7fc714134240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:06.169 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.159+0000 7fc719e17700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc714061560 0x7fc71412fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:06.169 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.159+0000 7fc719e17700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc71412d620 0x7fc71412da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:06.169 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.159+0000 7fc719e17700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc714061d90 con 0x7fc71412d620 2020-11-20T11:32:06.170 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.159+0000 7fc719e17700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7140ad030 con 0x7fc714138ff0 2020-11-20T11:32:06.170 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.159+0000 7fc719e17700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7140ad2d0 con 0x7fc714061560 2020-11-20T11:32:06.170 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.159+0000 7fc713fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc71412d620 0x7fc71412da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:06.170 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.159+0000 7fc713fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc71412d620 0x7fc71412da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40166/0 (socket says 172.21.15.23:40166) 2020-11-20T11:32:06.170 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.159+0000 7fc713fff700 1 -- 172.21.15.23:0/2298562925 learned_addr learned my addr 172.21.15.23:0/2298562925 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:06.171 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.159+0000 7fc7137fe700 1 --2- 172.21.15.23:0/2298562925 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc714138ff0 0x7fc714134240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:06.171 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.160+0000 7fc712ffd700 1 --2- 172.21.15.23:0/2298562925 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc714061560 0x7fc71412fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:06.171 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.160+0000 7fc7137fe700 1 -- 172.21.15.23:0/2298562925 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc714061560 msgr2=0x7fc71412fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:06.171 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.160+0000 7fc7137fe700 1 --2- 172.21.15.23:0/2298562925 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc714061560 0x7fc71412fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:06.171 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.160+0000 7fc7137fe700 1 -- 172.21.15.23:0/2298562925 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc71412d620 msgr2=0x7fc71412da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:06.172 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.160+0000 7fc7137fe700 1 --2- 172.21.15.23:0/2298562925 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc71412d620 0x7fc71412da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:06.172 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.160+0000 7fc7137fe700 1 -- 172.21.15.23:0/2298562925 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7140ad770 con 0x7fc714138ff0 2020-11-20T11:32:06.172 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.160+0000 7fc713fff700 1 --2- 172.21.15.23:0/2298562925 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc71412d620 0x7fc71412da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:32:06.172 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.160+0000 7fc7137fe700 1 --2- 172.21.15.23:0/2298562925 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc714138ff0 0x7fc714134240 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fc70400c1e0 tx=0x7fc704016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:06.173 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.165+0000 7fc6f37fe700 1 -- 172.21.15.23:0/2298562925 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70401f070 con 0x7fc714138ff0 2020-11-20T11:32:06.174 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.165+0000 7fc6f37fe700 1 -- 172.21.15.23:0/2298562925 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc704002a90 con 0x7fc714138ff0 2020-11-20T11:32:06.174 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.165+0000 7fc6f37fe700 1 -- 172.21.15.23:0/2298562925 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc704007040 con 0x7fc714138ff0 2020-11-20T11:32:06.174 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.165+0000 7fc719e17700 1 -- 172.21.15.23:0/2298562925 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7140619e0 con 0x7fc714138ff0 2020-11-20T11:32:06.174 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.165+0000 7fc719e17700 1 -- 172.21.15.23:0/2298562925 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc714134b60 con 0x7fc714138ff0 2020-11-20T11:32:06.174 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.165+0000 7fc6f37fe700 1 -- 172.21.15.23:0/2298562925 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fc704006530 con 0x7fc714138ff0 2020-11-20T11:32:06.175 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.166+0000 7fc6f37fe700 1 -- 172.21.15.23:0/2298562925 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fc70402c7c0 con 0x7fc714138ff0 2020-11-20T11:32:06.175 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.166+0000 7fc719e17700 1 -- 172.21.15.23:0/2298562925 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc714044710 con 0x7fc714138ff0 2020-11-20T11:32:06.175 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.169+0000 7fc6f37fe700 1 -- 172.21.15.23:0/2298562925 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71786 (secure 0 0 0) 0x7fc7040073f0 con 0x7fc714138ff0 2020-11-20T11:32:06.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.241+0000 7fc719e17700 1 -- 172.21.15.23:0/2298562925 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fc71405c280 con 0x7fc714138ff0 2020-11-20T11:32:06.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.250+0000 7fc6f37fe700 1 -- 172.21.15.23:0/2298562925 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc704006900 con 0x7fc714138ff0 2020-11-20T11:32:06.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.250+0000 7fc719e17700 1 -- 172.21.15.23:0/2298562925 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc714138ff0 msgr2=0x7fc714134240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:06.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.250+0000 7fc719e17700 1 --2- 172.21.15.23:0/2298562925 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc714138ff0 0x7fc714134240 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fc70400c1e0 tx=0x7fc704016000).stop 2020-11-20T11:32:06.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.250+0000 7fc719e17700 1 -- 172.21.15.23:0/2298562925 shutdown_connections 2020-11-20T11:32:06.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.250+0000 7fc719e17700 1 --2- 172.21.15.23:0/2298562925 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc71412d620 0x7fc71412da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:06.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.250+0000 7fc719e17700 1 --2- 172.21.15.23:0/2298562925 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc714138ff0 0x7fc714134240 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:06.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.250+0000 7fc719e17700 1 --2- 172.21.15.23:0/2298562925 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc714061560 0x7fc71412fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:06.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.250+0000 7fc719e17700 1 -- 172.21.15.23:0/2298562925 >> 172.21.15.23:0/2298562925 conn(0x7fc714066750 msgr2=0x7fc7140a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:06.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.251+0000 7fc719e17700 1 -- 172.21.15.23:0/2298562925 shutdown_connections 2020-11-20T11:32:06.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:06.251+0000 7fc719e17700 1 -- 172.21.15.23:0/2298562925 wait complete. 2020-11-20T11:32:06.256 INFO:teuthology.orchestra.run.smithi023.stderr:adjusted tunables profile to default 2020-11-20T11:32:06.266 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2020-11-20T11:32:06.266 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-11-20T11:32:06.267 INFO:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2020-11-20T11:32:06.280 INFO:teuthology.orchestra.run.smithi023.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2020-11-20T11:32:06.291 INFO:teuthology.orchestra.run.smithi023.stdout:e9d219f5-e40e-4b2b-9468-deb9c4074a44 2020-11-20T11:32:06.291 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2020-11-20T11:32:06.291 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2020-11-20T11:32:06.292 INFO:teuthology.orchestra.run.smithi023.stderr:37 bytes copied, 3.8684e-05 s, 956 kB/s 2020-11-20T11:32:06.293 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-11-20T11:32:06.293 INFO:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2020-11-20T11:32:06.346 INFO:teuthology.orchestra.run.smithi023.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2020-11-20T11:32:06.356 INFO:teuthology.orchestra.run.smithi023.stdout:2328adde-2108-4fe0-8138-f0d096a496a8 2020-11-20T11:32:06.357 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2020-11-20T11:32:06.357 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2020-11-20T11:32:06.357 INFO:teuthology.orchestra.run.smithi023.stderr:37 bytes copied, 4.6633e-05 s, 793 kB/s 2020-11-20T11:32:06.358 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-11-20T11:32:06.358 INFO:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2020-11-20T11:32:06.412 INFO:teuthology.orchestra.run.smithi023.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2020-11-20T11:32:06.422 INFO:teuthology.orchestra.run.smithi023.stdout:13367708-59a2-4363-8a3a-ab0cb97145ef 2020-11-20T11:32:06.422 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2020-11-20T11:32:06.422 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2020-11-20T11:32:06.422 INFO:teuthology.orchestra.run.smithi023.stderr:37 bytes copied, 3.2044e-05 s, 1.2 MB/s 2020-11-20T11:32:06.424 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-11-20T11:32:06.424 INFO:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2020-11-20T11:32:06.478 INFO:teuthology.orchestra.run.smithi023.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2020-11-20T11:32:06.488 INFO:teuthology.orchestra.run.smithi023.stdout:1f147dd7-9289-47b0-ad1c-4088618d6faa 2020-11-20T11:32:06.489 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2020-11-20T11:32:06.489 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2020-11-20T11:32:06.489 INFO:teuthology.orchestra.run.smithi023.stderr:37 bytes copied, 3.7085e-05 s, 998 kB/s 2020-11-20T11:32:06.490 INFO:teuthology.orchestra.run.smithi155:> set -ex 2020-11-20T11:32:06.490 INFO:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2020-11-20T11:32:06.503 INFO:teuthology.orchestra.run.smithi155.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2020-11-20T11:32:06.514 INFO:teuthology.orchestra.run.smithi155.stdout:d008b039-4bc5-4135-9678-1511e58efc29 2020-11-20T11:32:06.514 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records in 2020-11-20T11:32:06.514 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records out 2020-11-20T11:32:06.514 INFO:teuthology.orchestra.run.smithi155.stderr:37 bytes copied, 4.0054e-05 s, 924 kB/s 2020-11-20T11:32:06.515 INFO:teuthology.orchestra.run.smithi155:> set -ex 2020-11-20T11:32:06.516 INFO:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2020-11-20T11:32:06.569 INFO:teuthology.orchestra.run.smithi155.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2020-11-20T11:32:06.578 INFO:teuthology.orchestra.run.smithi155.stdout:56715e6f-cf6e-44a6-9e39-93ab1224095f 2020-11-20T11:32:06.579 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records in 2020-11-20T11:32:06.579 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records out 2020-11-20T11:32:06.579 INFO:teuthology.orchestra.run.smithi155.stderr:37 bytes copied, 3.5679e-05 s, 1.0 MB/s 2020-11-20T11:32:06.580 INFO:teuthology.orchestra.run.smithi155:> set -ex 2020-11-20T11:32:06.581 INFO:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2020-11-20T11:32:06.635 INFO:teuthology.orchestra.run.smithi155.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2020-11-20T11:32:06.644 INFO:teuthology.orchestra.run.smithi155.stdout:db1e61ca-8e14-4a1f-8607-6e01b148d61c 2020-11-20T11:32:06.645 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records in 2020-11-20T11:32:06.645 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records out 2020-11-20T11:32:06.645 INFO:teuthology.orchestra.run.smithi155.stderr:37 bytes copied, 3.7247e-05 s, 993 kB/s 2020-11-20T11:32:06.646 INFO:teuthology.orchestra.run.smithi155:> set -ex 2020-11-20T11:32:06.647 INFO:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2020-11-20T11:32:06.700 INFO:teuthology.orchestra.run.smithi155.stderr:+ sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2020-11-20T11:32:06.710 INFO:teuthology.orchestra.run.smithi155.stdout:d44419be-ef89-462c-8cf1-3c6db2a6944b 2020-11-20T11:32:06.710 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records in 2020-11-20T11:32:06.710 INFO:teuthology.orchestra.run.smithi155.stderr:0+1 records out 2020-11-20T11:32:06.710 INFO:teuthology.orchestra.run.smithi155.stderr:37 bytes copied, 4.1459e-05 s, 892 kB/s 2020-11-20T11:32:06.711 INFO:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new e9d219f5-e40e-4b2b-9468-deb9c4074a44 0 2020-11-20T11:32:06.862 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.858+0000 7fb532ccc700 1 Processor -- start 2020-11-20T11:32:06.862 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.858+0000 7fb532ccc700 1 -- start start 2020-11-20T11:32:06.864 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.859+0000 7fb532ccc700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb52c067fd0 0x7fb52c13bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:06.865 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.859+0000 7fb532ccc700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb52c0a9080 0x7fb52c0a7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:06.865 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.859+0000 7fb532ccc700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb52c0a8c20 con 0x7fb52c05bc80 2020-11-20T11:32:06.865 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.859+0000 7fb532ccc700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb52c0a9e20 con 0x7fb52c0a9080 2020-11-20T11:32:06.865 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.859+0000 7fb532ccc700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb52c0684e0 con 0x7fb52c067fd0 2020-11-20T11:32:06.866 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.859+0000 7fb531269700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb52c0a9080 0x7fb52c0a7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:06.866 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.859+0000 7fb52bfff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb52c067fd0 0x7fb52c13bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:06.866 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.859+0000 7fb531269700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb52c0a9080 0x7fb52c0a7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:41754/0 (socket says 172.21.15.155:41754) 2020-11-20T11:32:06.866 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.859+0000 7fb52bfff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb52c067fd0 0x7fb52c13bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:58116/0 (socket says 172.21.15.155:58116) 2020-11-20T11:32:06.866 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.859+0000 7fb531269700 1 -- 172.21.15.155:0/411246088 learned_addr learned my addr 172.21.15.155:0/411246088 (peer_addr_for_me v2:172.21.15.155:0/0) 2020-11-20T11:32:06.867 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.860+0000 7fb52bfff700 1 -- 172.21.15.155:0/411246088 >> v2:172.21.15.155:3300/0 conn(0x7fb52c0a9080 msgr2=0x7fb52c0a7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:06.867 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.860+0000 7fb52bfff700 1 --2- 172.21.15.155:0/411246088 >> v2:172.21.15.155:3300/0 conn(0x7fb52c0a9080 0x7fb52c0a7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:06.867 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.860+0000 7fb52bfff700 1 -- 172.21.15.155:0/411246088 >> v1:172.21.15.155:6789/0 conn(0x7fb52c05bc80 legacy=0x7fb52c0a6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:06.867 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.860+0000 7fb52bfff700 1 -- 172.21.15.155:0/411246088 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb52c05c060 con 0x7fb52c067fd0 2020-11-20T11:32:06.868 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.860+0000 7fb52b7fe700 1 -- 172.21.15.155:0/411246088 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 471165940 0 0) 0x7fb52c0a8c20 con 0x7fb52c05bc80 2020-11-20T11:32:06.868 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.860+0000 7fb531269700 1 --2- 172.21.15.155:0/411246088 >> v2:172.21.15.155:3300/0 conn(0x7fb52c0a9080 0x7fb52c0a7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:32:06.868 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.860+0000 7fb52bfff700 1 --2- 172.21.15.155:0/411246088 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb52c067fd0 0x7fb52c13bfa0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fb52c0a8db0 tx=0x7fb518006e30).ready entity=mon.2 client_cookie=76bcce04f002203 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:06.868 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.860+0000 7fb52b7fe700 1 -- 172.21.15.155:0/411246088 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5180097b0 con 0x7fb52c067fd0 2020-11-20T11:32:06.868 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.860+0000 7fb52b7fe700 1 -- 172.21.15.155:0/411246088 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb518000e90 con 0x7fb52c067fd0 2020-11-20T11:32:06.869 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.860+0000 7fb52b7fe700 1 -- 172.21.15.155:0/411246088 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb518009b20 con 0x7fb52c067fd0 2020-11-20T11:32:06.869 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.860+0000 7fb532ccc700 1 -- 172.21.15.155:0/411246088 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb52c067fd0 msgr2=0x7fb52c13bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:06.869 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.860+0000 7fb532ccc700 1 --2- 172.21.15.155:0/411246088 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb52c067fd0 0x7fb52c13bfa0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fb52c0a8db0 tx=0x7fb518006e30).stop 2020-11-20T11:32:06.869 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.860+0000 7fb532ccc700 1 -- 172.21.15.155:0/411246088 shutdown_connections 2020-11-20T11:32:06.869 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.860+0000 7fb532ccc700 1 --2- 172.21.15.155:0/411246088 >> v2:172.21.15.155:3300/0 conn(0x7fb52c0a9080 0x7fb52c0a7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:06.870 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.860+0000 7fb532ccc700 1 --2- 172.21.15.155:0/411246088 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb52c067fd0 0x7fb52c13bfa0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:06.870 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.860+0000 7fb532ccc700 1 -- 172.21.15.155:0/411246088 >> 172.21.15.155:0/411246088 conn(0x7fb52c0ad7a0 msgr2=0x7fb52c0adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:06.870 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.860+0000 7fb532ccc700 1 -- 172.21.15.155:0/411246088 shutdown_connections 2020-11-20T11:32:06.870 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.860+0000 7fb532ccc700 1 -- 172.21.15.155:0/411246088 wait complete. 2020-11-20T11:32:06.870 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.861+0000 7fb532ccc700 1 Processor -- start 2020-11-20T11:32:06.871 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.862+0000 7fb532ccc700 1 -- start start 2020-11-20T11:32:06.873 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.862+0000 7fb532ccc700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb52c05bc80 0x7fb52c0a45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:06.873 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.862+0000 7fb532ccc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb52c067fd0 0x7fb52c0a28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:06.873 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.862+0000 7fb532ccc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb52c0a9080 0x7fb52c147c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:06.873 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.862+0000 7fb532ccc700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb52c0a15b0 con 0x7fb52c067fd0 2020-11-20T11:32:06.874 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.862+0000 7fb532ccc700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb52c0a21c0 con 0x7fb52c0a9080 2020-11-20T11:32:06.874 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.862+0000 7fb532ccc700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb52c0a17e0 con 0x7fb52c05bc80 2020-11-20T11:32:06.874 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.862+0000 7fb530a68700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb52c05bc80 0x7fb52c0a45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:06.874 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.862+0000 7fb531269700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb52c0a9080 0x7fb52c147c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:06.875 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.862+0000 7fb530a68700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb52c05bc80 0x7fb52c0a45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:58120/0 (socket says 172.21.15.155:58120) 2020-11-20T11:32:06.875 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.862+0000 7fb530a68700 1 -- 172.21.15.155:0/625035979 learned_addr learned my addr 172.21.15.155:0/625035979 (peer_addr_for_me v2:172.21.15.155:0/0) 2020-11-20T11:32:06.875 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.862+0000 7fb52bfff700 1 --2- 172.21.15.155:0/625035979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb52c067fd0 0x7fb52c0a28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:06.875 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.863+0000 7fb530a68700 1 -- 172.21.15.155:0/625035979 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb52c0a9080 msgr2=0x7fb52c147c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:06.875 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.863+0000 7fb530a68700 1 --2- 172.21.15.155:0/625035979 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb52c0a9080 0x7fb52c147c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:06.876 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.863+0000 7fb530a68700 1 -- 172.21.15.155:0/625035979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb52c067fd0 msgr2=0x7fb52c0a28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:06.876 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.863+0000 7fb530a68700 1 --2- 172.21.15.155:0/625035979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb52c067fd0 0x7fb52c0a28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:06.876 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.863+0000 7fb530a68700 1 -- 172.21.15.155:0/625035979 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb52c0a1a80 con 0x7fb52c05bc80 2020-11-20T11:32:06.876 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.863+0000 7fb531269700 1 --2- 172.21.15.155:0/625035979 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb52c0a9080 0x7fb52c147c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:32:06.876 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.863+0000 7fb530a68700 1 --2- 172.21.15.155:0/625035979 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb52c05bc80 0x7fb52c0a45f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fb52000c6f0 tx=0x7fb520007b00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:06.877 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.863+0000 7fb528ff9700 1 -- 172.21.15.155:0/625035979 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52000e4a0 con 0x7fb52c05bc80 2020-11-20T11:32:06.877 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.863+0000 7fb528ff9700 1 -- 172.21.15.155:0/625035979 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb52000e620 con 0x7fb52c05bc80 2020-11-20T11:32:06.877 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.863+0000 7fb532ccc700 1 -- 172.21.15.155:0/625035979 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb52c0a1d20 con 0x7fb52c05bc80 2020-11-20T11:32:06.877 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.863+0000 7fb528ff9700 1 -- 172.21.15.155:0/625035979 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52000e9e0 con 0x7fb52c05bc80 2020-11-20T11:32:06.877 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.863+0000 7fb532ccc700 1 -- 172.21.15.155:0/625035979 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb52c09d6e0 con 0x7fb52c05bc80 2020-11-20T11:32:06.878 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.863+0000 7fb528ff9700 1 -- 172.21.15.155:0/625035979 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fb520019030 con 0x7fb52c05bc80 2020-11-20T11:32:06.878 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.863+0000 7fb528ff9700 1 -- 172.21.15.155:0/625035979 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 754+0+0 (secure 0 0 0) 0x7fb52001cd80 con 0x7fb52c05bc80 2020-11-20T11:32:06.878 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.864+0000 7fb532ccc700 1 -- 172.21.15.155:0/625035979 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb52c0a1eb0 con 0x7fb52c05bc80 2020-11-20T11:32:06.878 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.868+0000 7fb528ff9700 1 -- 172.21.15.155:0/625035979 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71786 (secure 0 0 0) 0x7fb520017070 con 0x7fb52c05bc80 2020-11-20T11:32:06.958 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.955+0000 7fb532ccc700 1 -- 172.21.15.155:0/625035979 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e9d219f5-e40e-4b2b-9468-deb9c4074a44", "id": 0} v 0) v1 -- 0x7fb52c044710 con 0x7fb52c05bc80 2020-11-20T11:32:06.968 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.965+0000 7fb528ff9700 1 -- 172.21.15.155:0/625035979 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e9d219f5-e40e-4b2b-9468-deb9c4074a44", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb52001c440 con 0x7fb52c05bc80 2020-11-20T11:32:06.969 INFO:teuthology.orchestra.run.smithi155.stdout:0 2020-11-20T11:32:06.970 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.966+0000 7fb532ccc700 1 -- 172.21.15.155:0/625035979 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb52c05bc80 msgr2=0x7fb52c0a45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:06.970 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.966+0000 7fb532ccc700 1 --2- 172.21.15.155:0/625035979 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb52c05bc80 0x7fb52c0a45f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fb52000c6f0 tx=0x7fb520007b00).stop 2020-11-20T11:32:06.970 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.966+0000 7fb532ccc700 1 -- 172.21.15.155:0/625035979 shutdown_connections 2020-11-20T11:32:06.971 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.966+0000 7fb532ccc700 1 --2- 172.21.15.155:0/625035979 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb52c0a9080 0x7fb52c147c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:06.971 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.966+0000 7fb532ccc700 1 --2- 172.21.15.155:0/625035979 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb52c05bc80 0x7fb52c0a45f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:06.971 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.966+0000 7fb532ccc700 1 --2- 172.21.15.155:0/625035979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb52c067fd0 0x7fb52c0a28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:06.971 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.966+0000 7fb532ccc700 1 -- 172.21.15.155:0/625035979 >> 172.21.15.155:0/625035979 conn(0x7fb52c0ad7a0 msgr2=0x7fb52c138b40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:06.971 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.966+0000 7fb532ccc700 1 -- 172.21.15.155:0/625035979 shutdown_connections 2020-11-20T11:32:06.972 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:06.966+0000 7fb532ccc700 1 -- 172.21.15.155:0/625035979 wait complete. 2020-11-20T11:32:06.981 INFO:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new 2328adde-2108-4fe0-8138-f0d096a496a8 1 2020-11-20T11:32:07.091 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.088+0000 7ff07a69b700 1 Processor -- start 2020-11-20T11:32:07.092 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.088+0000 7ff07a69b700 1 -- start start 2020-11-20T11:32:07.094 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.089+0000 7ff07a69b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff0740aaab0 0x7ff07413ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:07.094 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.089+0000 7ff07a69b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff07413aeb0 0x7ff0740a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:07.095 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.089+0000 7ff07a69b700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0741388b0 con 0x7ff0740aae90 2020-11-20T11:32:07.095 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.089+0000 7ff07a69b700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff07413e600 con 0x7ff0740aaab0 2020-11-20T11:32:07.095 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.089+0000 7ff07a69b700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff07413e8a0 con 0x7ff07413aeb0 2020-11-20T11:32:07.095 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.089+0000 7ff078c38700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff07413aeb0 0x7ff0740a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:07.097 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.089+0000 7ff0737fe700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7ff0740aae90 0x7ff0741381a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:36524/0 (socket says 172.21.15.155:36524) 2020-11-20T11:32:07.098 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.089+0000 7ff0737fe700 1 -- 172.21.15.155:0/2171830410 learned_addr learned my addr 172.21.15.155:0/2171830410 (peer_addr_for_me v1:172.21.15.155:0/0) 2020-11-20T11:32:07.098 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.089+0000 7ff073fff700 1 --2- 172.21.15.155:0/2171830410 >> v2:172.21.15.23:3300/0 conn(0x7ff0740aaab0 0x7ff07413ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:07.098 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.089+0000 7ff072ffd700 1 -- 172.21.15.155:0/2171830410 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1861109133 0 0) 0x7ff0741388b0 con 0x7ff0740aae90 2020-11-20T11:32:07.098 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.089+0000 7ff078c38700 1 -- 172.21.15.155:0/2171830410 >> v2:172.21.15.23:3300/0 conn(0x7ff0740aaab0 msgr2=0x7ff07413ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:07.099 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.089+0000 7ff078c38700 1 --2- 172.21.15.155:0/2171830410 >> v2:172.21.15.23:3300/0 conn(0x7ff0740aaab0 0x7ff07413ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.099 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.089+0000 7ff078c38700 1 -- 172.21.15.155:0/2171830410 >> v1:172.21.15.155:6789/0 conn(0x7ff0740aae90 legacy=0x7ff0741381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:07.099 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.089+0000 7ff078c38700 1 -- 172.21.15.155:0/2171830410 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0740ab6d0 con 0x7ff07413aeb0 2020-11-20T11:32:07.100 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.089+0000 7ff073fff700 1 --2- 172.21.15.155:0/2171830410 >> v2:172.21.15.23:3300/0 conn(0x7ff0740aaab0 0x7ff07413ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:07.100 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.089+0000 7ff078c38700 1 --2- 172.21.15.155:0/2171830410 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff07413aeb0 0x7ff0740a8c60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7ff06c009ce0 tx=0x7ff06c009220).ready entity=mon.2 client_cookie=d1577b0ec745a68f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:07.100 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.089+0000 7ff072ffd700 1 -- 172.21.15.155:0/2171830410 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06c00bbf0 con 0x7ff07413aeb0 2020-11-20T11:32:07.100 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.089+0000 7ff072ffd700 1 -- 172.21.15.155:0/2171830410 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff06c00bd50 con 0x7ff07413aeb0 2020-11-20T11:32:07.101 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.089+0000 7ff072ffd700 1 -- 172.21.15.155:0/2171830410 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06c011610 con 0x7ff07413aeb0 2020-11-20T11:32:07.101 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.090+0000 7ff07a69b700 1 -- 172.21.15.155:0/2171830410 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff07413aeb0 msgr2=0x7ff0740a8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:07.101 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.090+0000 7ff07a69b700 1 --2- 172.21.15.155:0/2171830410 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff07413aeb0 0x7ff0740a8c60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7ff06c009ce0 tx=0x7ff06c009220).stop 2020-11-20T11:32:07.102 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.090+0000 7ff07a69b700 1 -- 172.21.15.155:0/2171830410 shutdown_connections 2020-11-20T11:32:07.102 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.090+0000 7ff07a69b700 1 --2- 172.21.15.155:0/2171830410 >> v2:172.21.15.23:3300/0 conn(0x7ff0740aaab0 0x7ff07413ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.102 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.090+0000 7ff07a69b700 1 --2- 172.21.15.155:0/2171830410 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff07413aeb0 0x7ff0740a8c60 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.102 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.090+0000 7ff07a69b700 1 -- 172.21.15.155:0/2171830410 >> 172.21.15.155:0/2171830410 conn(0x7ff0740678e0 msgr2=0x7ff074067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:07.103 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.090+0000 7ff07a69b700 1 -- 172.21.15.155:0/2171830410 shutdown_connections 2020-11-20T11:32:07.103 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.090+0000 7ff07a69b700 1 -- 172.21.15.155:0/2171830410 wait complete. 2020-11-20T11:32:07.103 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.091+0000 7ff07a69b700 1 Processor -- start 2020-11-20T11:32:07.104 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.091+0000 7ff07a69b700 1 -- start start 2020-11-20T11:32:07.107 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.092+0000 7ff07a69b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff0740aaab0 0x7ff07409df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:07.107 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.092+0000 7ff07a69b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff0740aae90 0x7ff07414d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:07.108 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.092+0000 7ff07a69b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0740a4680 0x7ff0740a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:07.108 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.092+0000 7ff07a69b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07414a4a0 con 0x7ff0740aae90 2020-11-20T11:32:07.108 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.092+0000 7ff07a69b700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07414a740 con 0x7ff0740a4680 2020-11-20T11:32:07.109 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.092+0000 7ff07a69b700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff07414a9e0 con 0x7ff0740aaab0 2020-11-20T11:32:07.109 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.092+0000 7ff073fff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff0740aaab0 0x7ff07409df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:07.109 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.092+0000 7ff073fff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff0740aaab0 0x7ff07409df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:58132/0 (socket says 172.21.15.155:58132) 2020-11-20T11:32:07.109 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.092+0000 7ff073fff700 1 -- 172.21.15.155:0/139880869 learned_addr learned my addr 172.21.15.155:0/139880869 (peer_addr_for_me v2:172.21.15.155:0/0) 2020-11-20T11:32:07.110 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.092+0000 7ff078c38700 1 --2- 172.21.15.155:0/139880869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0740a4680 0x7ff0740a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:07.110 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.092+0000 7ff0737fe700 1 --2- 172.21.15.155:0/139880869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff0740aae90 0x7ff07414d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:07.110 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.092+0000 7ff073fff700 1 -- 172.21.15.155:0/139880869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0740a4680 msgr2=0x7ff0740a4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:07.111 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.092+0000 7ff073fff700 1 --2- 172.21.15.155:0/139880869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0740a4680 0x7ff0740a4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.111 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.092+0000 7ff073fff700 1 -- 172.21.15.155:0/139880869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff0740aae90 msgr2=0x7ff07414d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:07.111 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.092+0000 7ff073fff700 1 --2- 172.21.15.155:0/139880869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff0740aae90 0x7ff07414d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.111 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.092+0000 7ff073fff700 1 -- 172.21.15.155:0/139880869 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0740a5290 con 0x7ff0740aaab0 2020-11-20T11:32:07.112 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.092+0000 7ff073fff700 1 --2- 172.21.15.155:0/139880869 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff0740aaab0 0x7ff07409df70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7ff06800db60 tx=0x7ff068008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:07.112 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.092+0000 7ff053fff700 1 -- 172.21.15.155:0/139880869 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06800de70 con 0x7ff0740aaab0 2020-11-20T11:32:07.112 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.092+0000 7ff053fff700 1 -- 172.21.15.155:0/139880869 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff068009850 con 0x7ff0740aaab0 2020-11-20T11:32:07.113 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.092+0000 7ff07a69b700 1 -- 172.21.15.155:0/139880869 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff07414ae80 con 0x7ff0740aaab0 2020-11-20T11:32:07.113 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.092+0000 7ff053fff700 1 -- 172.21.15.155:0/139880869 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff068009c10 con 0x7ff0740aaab0 2020-11-20T11:32:07.113 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.092+0000 7ff07a69b700 1 -- 172.21.15.155:0/139880869 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0740a6fc0 con 0x7ff0740aaab0 2020-11-20T11:32:07.113 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.093+0000 7ff053fff700 1 -- 172.21.15.155:0/139880869 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7ff068006280 con 0x7ff0740aaab0 2020-11-20T11:32:07.114 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.093+0000 7ff053fff700 1 -- 172.21.15.155:0/139880869 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 869+0+0 (secure 0 0 0) 0x7ff068005de0 con 0x7ff0740aaab0 2020-11-20T11:32:07.114 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.093+0000 7ff07a69b700 1 -- 172.21.15.155:0/139880869 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0740a7180 con 0x7ff0740aaab0 2020-11-20T11:32:07.114 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.095+0000 7ff053fff700 1 -- 172.21.15.155:0/139880869 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71786 (secure 0 0 0) 0x7ff068013550 con 0x7ff0740aaab0 2020-11-20T11:32:07.177 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.174+0000 7ff07a69b700 1 -- 172.21.15.155:0/139880869 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "osd new", "uuid": "2328adde-2108-4fe0-8138-f0d096a496a8", "id": 1} v 0) v1 -- 0x7ff07413ec60 con 0x7ff0740aaab0 2020-11-20T11:32:07.184 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.181+0000 7ff053fff700 1 -- 172.21.15.155:0/139880869 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2328adde-2108-4fe0-8138-f0d096a496a8", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff06802b0b0 con 0x7ff0740aaab0 2020-11-20T11:32:07.185 INFO:teuthology.orchestra.run.smithi155.stdout:1 2020-11-20T11:32:07.186 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.182+0000 7ff07a69b700 1 -- 172.21.15.155:0/139880869 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff0740aaab0 msgr2=0x7ff07409df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:07.187 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.182+0000 7ff07a69b700 1 --2- 172.21.15.155:0/139880869 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff0740aaab0 0x7ff07409df70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7ff06800db60 tx=0x7ff068008040).stop 2020-11-20T11:32:07.187 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.182+0000 7ff07a69b700 1 -- 172.21.15.155:0/139880869 shutdown_connections 2020-11-20T11:32:07.188 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.182+0000 7ff07a69b700 1 --2- 172.21.15.155:0/139880869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff0740a4680 0x7ff0740a4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.188 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.182+0000 7ff07a69b700 1 --2- 172.21.15.155:0/139880869 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff0740aaab0 0x7ff07409df70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.189 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.182+0000 7ff07a69b700 1 --2- 172.21.15.155:0/139880869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff0740aae90 0x7ff07414d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.189 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.182+0000 7ff07a69b700 1 -- 172.21.15.155:0/139880869 >> 172.21.15.155:0/139880869 conn(0x7ff0740678e0 msgr2=0x7ff074139b40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:07.189 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.182+0000 7ff07a69b700 1 -- 172.21.15.155:0/139880869 shutdown_connections 2020-11-20T11:32:07.190 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.182+0000 7ff07a69b700 1 -- 172.21.15.155:0/139880869 wait complete. 2020-11-20T11:32:07.199 INFO:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new 13367708-59a2-4363-8a3a-ab0cb97145ef 2 2020-11-20T11:32:07.347 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.344+0000 7f2764053700 1 Processor -- start 2020-11-20T11:32:07.348 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.344+0000 7f2764053700 1 -- start start 2020-11-20T11:32:07.350 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.345+0000 7f2764053700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f275c0ab2a0 0x7f275c13e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:07.351 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.345+0000 7f2764053700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f275c0a4c20 0x7f275c0a8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:07.351 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.345+0000 7f2764053700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f275c0b1a20 con 0x7f275c0ad8d0 2020-11-20T11:32:07.351 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.345+0000 7f2764053700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f275c13f4a0 con 0x7f275c0ab2a0 2020-11-20T11:32:07.351 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.345+0000 7f2764053700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f275c0aa0e0 con 0x7f275c0a4c20 2020-11-20T11:32:07.352 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.345+0000 7f2761def700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f275c0ab2a0 0x7f275c13e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:07.352 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.345+0000 7f27625f0700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f275c0a4c20 0x7f275c0a8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:07.352 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.345+0000 7f2761def700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f275c0ab2a0 0x7f275c13e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:41776/0 (socket says 172.21.15.155:41776) 2020-11-20T11:32:07.352 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.345+0000 7f2761def700 1 -- 172.21.15.155:0/1477725258 learned_addr learned my addr 172.21.15.155:0/1477725258 (peer_addr_for_me v2:172.21.15.155:0/0) 2020-11-20T11:32:07.353 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.345+0000 7f27625f0700 1 --2- 172.21.15.155:0/1477725258 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f275c0a4c20 0x7f275c0a8fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:58142/0 (socket says 172.21.15.155:58142) 2020-11-20T11:32:07.353 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.345+0000 7f2761def700 1 -- 172.21.15.155:0/1477725258 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f275c0a4c20 msgr2=0x7f275c0a8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:07.353 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.346+0000 7f2761def700 1 --2- 172.21.15.155:0/1477725258 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f275c0a4c20 0x7f275c0a8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.353 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.346+0000 7f2761def700 1 -- 172.21.15.155:0/1477725258 >> v1:172.21.15.23:6789/0 conn(0x7f275c0ad8d0 legacy=0x7f275c0a4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:07.354 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.346+0000 7f2761def700 1 -- 172.21.15.155:0/1477725258 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f275c0a8b90 con 0x7f275c0ab2a0 2020-11-20T11:32:07.354 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.346+0000 7f2761def700 1 --2- 172.21.15.155:0/1477725258 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f275c0ab2a0 0x7f275c13e570 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f274c009a10 tx=0x7f274c006ef0).ready entity=mon.1 client_cookie=f41501734dd45264 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:07.354 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.346+0000 7f2760ded700 1 -- 172.21.15.155:0/1477725258 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274c00b860 con 0x7f275c0ab2a0 2020-11-20T11:32:07.354 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.346+0000 7f2760ded700 1 -- 172.21.15.155:0/1477725258 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f274c000e90 con 0x7f275c0ab2a0 2020-11-20T11:32:07.354 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.346+0000 7f2760ded700 1 -- 172.21.15.155:0/1477725258 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274c00bbd0 con 0x7f275c0ab2a0 2020-11-20T11:32:07.355 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.346+0000 7f2764053700 1 -- 172.21.15.155:0/1477725258 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f275c0ab2a0 msgr2=0x7f275c13e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:07.355 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.346+0000 7f2764053700 1 --2- 172.21.15.155:0/1477725258 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f275c0ab2a0 0x7f275c13e570 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f274c009a10 tx=0x7f274c006ef0).stop 2020-11-20T11:32:07.355 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.346+0000 7f2764053700 1 -- 172.21.15.155:0/1477725258 shutdown_connections 2020-11-20T11:32:07.355 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.346+0000 7f2764053700 1 --2- 172.21.15.155:0/1477725258 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f275c0a4c20 0x7f275c0a8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.356 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.346+0000 7f2764053700 1 --2- 172.21.15.155:0/1477725258 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f275c0ab2a0 0x7f275c13e570 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.356 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.346+0000 7f2764053700 1 -- 172.21.15.155:0/1477725258 >> 172.21.15.155:0/1477725258 conn(0x7f275c0683b0 msgr2=0x7f275c0b1310 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:07.356 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.346+0000 7f2764053700 1 -- 172.21.15.155:0/1477725258 shutdown_connections 2020-11-20T11:32:07.356 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.346+0000 7f2764053700 1 -- 172.21.15.155:0/1477725258 wait complete. 2020-11-20T11:32:07.357 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.347+0000 7f2764053700 1 Processor -- start 2020-11-20T11:32:07.357 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.347+0000 7f2764053700 1 -- start start 2020-11-20T11:32:07.359 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.348+0000 7f2764053700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f275c0a4c20 0x7f275c0a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:07.359 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.348+0000 7f2764053700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f275c0ab2a0 0x7f275c0a28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:07.360 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.348+0000 7f2764053700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f275c0ad8d0 0x7f275c14d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:07.360 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.348+0000 7f2764053700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f275c09f9f0 con 0x7f275c0ad8d0 2020-11-20T11:32:07.360 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.348+0000 7f2764053700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f275c09fc90 con 0x7f275c0a4c20 2020-11-20T11:32:07.360 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.348+0000 7f2764053700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f275c09ff30 con 0x7f275c0ab2a0 2020-11-20T11:32:07.361 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.348+0000 7f27615ee700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f275c0ab2a0 0x7f275c0a28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:07.362 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.348+0000 7f2761def700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f275c0a4c20 0x7f275c0a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:07.362 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.348+0000 7f27615ee700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f275c0ab2a0 0x7f275c0a28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:58144/0 (socket says 172.21.15.155:58144) 2020-11-20T11:32:07.362 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.348+0000 7f27615ee700 1 -- 172.21.15.155:0/2953155217 learned_addr learned my addr 172.21.15.155:0/2953155217 (peer_addr_for_me v2:172.21.15.155:0/0) 2020-11-20T11:32:07.362 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.348+0000 7f2761def700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f275c0a4c20 0x7f275c0a0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:41784/0 (socket says 172.21.15.155:41784) 2020-11-20T11:32:07.363 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.348+0000 7f27625f0700 1 --2- 172.21.15.155:0/2953155217 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f275c0ad8d0 0x7f275c14d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:07.363 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.348+0000 7f27615ee700 1 -- 172.21.15.155:0/2953155217 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f275c0a4c20 msgr2=0x7f275c0a0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:07.363 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.348+0000 7f27615ee700 1 --2- 172.21.15.155:0/2953155217 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f275c0a4c20 0x7f275c0a0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.363 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.348+0000 7f27615ee700 1 -- 172.21.15.155:0/2953155217 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f275c0ad8d0 msgr2=0x7f275c14d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:07.363 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.348+0000 7f27615ee700 1 --2- 172.21.15.155:0/2953155217 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f275c0ad8d0 0x7f275c14d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.364 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.348+0000 7f27615ee700 1 -- 172.21.15.155:0/2953155217 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f275c09ed90 con 0x7f275c0ab2a0 2020-11-20T11:32:07.364 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.348+0000 7f2761def700 1 --2- 172.21.15.155:0/2953155217 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f275c0a4c20 0x7f275c0a0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:32:07.364 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.349+0000 7f27615ee700 1 --2- 172.21.15.155:0/2953155217 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f275c0ab2a0 0x7f275c0a28b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f275800c710 tx=0x7f2758007c90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:07.364 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.349+0000 7f2749ffb700 1 -- 172.21.15.155:0/2953155217 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2758007020 con 0x7f275c0ab2a0 2020-11-20T11:32:07.364 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.349+0000 7f2749ffb700 1 -- 172.21.15.155:0/2953155217 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2758007180 con 0x7f275c0ab2a0 2020-11-20T11:32:07.365 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.349+0000 7f2764053700 1 -- 172.21.15.155:0/2953155217 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f275c0a1350 con 0x7f275c0ab2a0 2020-11-20T11:32:07.365 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.349+0000 7f2749ffb700 1 -- 172.21.15.155:0/2953155217 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2758006530 con 0x7f275c0ab2a0 2020-11-20T11:32:07.365 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.349+0000 7f2749ffb700 1 -- 172.21.15.155:0/2953155217 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f2758010070 con 0x7f275c0ab2a0 2020-11-20T11:32:07.365 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.349+0000 7f2764053700 1 -- 172.21.15.155:0/2953155217 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f275c0a03d0 con 0x7f275c0ab2a0 2020-11-20T11:32:07.365 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.349+0000 7f2749ffb700 1 -- 172.21.15.155:0/2953155217 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 984+0+0 (secure 0 0 0) 0x7f2758006d20 con 0x7f275c0ab2a0 2020-11-20T11:32:07.366 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.350+0000 7f2764053700 1 -- 172.21.15.155:0/2953155217 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f275c0a05f0 con 0x7f275c0ab2a0 2020-11-20T11:32:07.366 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.352+0000 7f2749ffb700 1 -- 172.21.15.155:0/2953155217 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71786 (secure 0 0 0) 0x7f2758016020 con 0x7f275c0ab2a0 2020-11-20T11:32:07.445 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.442+0000 7f2764053700 1 -- 172.21.15.155:0/2953155217 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "osd new", "uuid": "13367708-59a2-4363-8a3a-ab0cb97145ef", "id": 2} v 0) v1 -- 0x7f275c060640 con 0x7f275c0ab2a0 2020-11-20T11:32:07.458 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.452+0000 7f2749ffb700 1 -- 172.21.15.155:0/2953155217 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "13367708-59a2-4363-8a3a-ab0cb97145ef", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f275800f620 con 0x7f275c0ab2a0 2020-11-20T11:32:07.458 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.453+0000 7f2764053700 1 -- 172.21.15.155:0/2953155217 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f275c0ab2a0 msgr2=0x7f275c0a28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:07.458 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.453+0000 7f2764053700 1 --2- 172.21.15.155:0/2953155217 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f275c0ab2a0 0x7f275c0a28b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f275800c710 tx=0x7f2758007c90).stop 2020-11-20T11:32:07.458 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.453+0000 7f2764053700 1 -- 172.21.15.155:0/2953155217 shutdown_connections 2020-11-20T11:32:07.459 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.453+0000 7f2764053700 1 --2- 172.21.15.155:0/2953155217 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f275c0ad8d0 0x7f275c14d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.459 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.453+0000 7f2764053700 1 --2- 172.21.15.155:0/2953155217 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f275c0a4c20 0x7f275c0a0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.459 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.453+0000 7f2764053700 1 --2- 172.21.15.155:0/2953155217 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f275c0ab2a0 0x7f275c0a28b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.459 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.453+0000 7f2764053700 1 -- 172.21.15.155:0/2953155217 >> 172.21.15.155:0/2953155217 conn(0x7f275c0683b0 msgr2=0x7f275c0ae5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:07.459 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.453+0000 7f2764053700 1 -- 172.21.15.155:0/2953155217 shutdown_connections 2020-11-20T11:32:07.460 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.453+0000 7f2764053700 1 -- 172.21.15.155:0/2953155217 wait complete. 2020-11-20T11:32:07.460 INFO:teuthology.orchestra.run.smithi155.stdout:2 2020-11-20T11:32:07.467 INFO:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new 1f147dd7-9289-47b0-ad1c-4088618d6faa 3 2020-11-20T11:32:07.577 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.574+0000 7f14b2acc700 1 Processor -- start 2020-11-20T11:32:07.577 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.574+0000 7f14b2acc700 1 -- start start 2020-11-20T11:32:07.580 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.574+0000 7f14b2acc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f14ac0619d0 0x7f14ac0a9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:07.580 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.574+0000 7f14b2acc700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14ac066170 0x7f14ac137e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:07.580 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.574+0000 7f14b2acc700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14ac0610b0 con 0x7f14ac061db0 2020-11-20T11:32:07.580 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.574+0000 7f14b2acc700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14ac0a9750 con 0x7f14ac0619d0 2020-11-20T11:32:07.581 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.574+0000 7f14b2acc700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac066680 con 0x7f14ac066170 2020-11-20T11:32:07.581 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.574+0000 7f14b1069700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14ac066170 0x7f14ac137e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:07.581 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.574+0000 7f14abfff700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f14ac061db0 0x7f14ac0609a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:36548/0 (socket says 172.21.15.155:36548) 2020-11-20T11:32:07.581 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.574+0000 7f14abfff700 1 -- 172.21.15.155:0/756081298 learned_addr learned my addr 172.21.15.155:0/756081298 (peer_addr_for_me v1:172.21.15.155:0/0) 2020-11-20T11:32:07.582 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.575+0000 7f14b0868700 1 --2- 172.21.15.155:0/756081298 >> v2:172.21.15.23:3300/0 conn(0x7f14ac0619d0 0x7f14ac0a9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:07.582 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.575+0000 7f14ab7fe700 1 -- 172.21.15.155:0/756081298 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340304607 0 0) 0x7f14ac0610b0 con 0x7f14ac061db0 2020-11-20T11:32:07.582 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.575+0000 7f14b1069700 1 -- 172.21.15.155:0/756081298 >> v2:172.21.15.23:3300/0 conn(0x7f14ac0619d0 msgr2=0x7f14ac0a9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:07.582 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.575+0000 7f14b1069700 1 --2- 172.21.15.155:0/756081298 >> v2:172.21.15.23:3300/0 conn(0x7f14ac0619d0 0x7f14ac0a9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.582 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.575+0000 7f14b1069700 1 -- 172.21.15.155:0/756081298 >> v1:172.21.15.155:6789/0 conn(0x7f14ac061db0 legacy=0x7f14ac0609a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:07.583 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.575+0000 7f14b1069700 1 -- 172.21.15.155:0/756081298 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ac066920 con 0x7f14ac066170 2020-11-20T11:32:07.583 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.575+0000 7f14b1069700 1 --2- 172.21.15.155:0/756081298 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14ac066170 0x7f14ac137e90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f14a4009ce0 tx=0x7f14a4009220).ready entity=mon.2 client_cookie=f58722df1febff72 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:07.583 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.575+0000 7f14ab7fe700 1 -- 172.21.15.155:0/756081298 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a400ba40 con 0x7f14ac066170 2020-11-20T11:32:07.583 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.575+0000 7f14ab7fe700 1 -- 172.21.15.155:0/756081298 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a400bba0 con 0x7f14ac066170 2020-11-20T11:32:07.584 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.575+0000 7f14ab7fe700 1 -- 172.21.15.155:0/756081298 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a4011400 con 0x7f14ac066170 2020-11-20T11:32:07.584 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.576+0000 7f14b2acc700 1 -- 172.21.15.155:0/756081298 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14ac066170 msgr2=0x7f14ac137e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:07.584 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.576+0000 7f14b2acc700 1 --2- 172.21.15.155:0/756081298 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14ac066170 0x7f14ac137e90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f14a4009ce0 tx=0x7f14a4009220).stop 2020-11-20T11:32:07.584 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.576+0000 7f14b2acc700 1 -- 172.21.15.155:0/756081298 shutdown_connections 2020-11-20T11:32:07.584 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.576+0000 7f14b2acc700 1 --2- 172.21.15.155:0/756081298 >> v2:172.21.15.23:3300/0 conn(0x7f14ac0619d0 0x7f14ac0a9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.585 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.576+0000 7f14b2acc700 1 --2- 172.21.15.155:0/756081298 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14ac066170 0x7f14ac137e90 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.585 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.576+0000 7f14b2acc700 1 -- 172.21.15.155:0/756081298 >> 172.21.15.155:0/756081298 conn(0x7f14ac063640 msgr2=0x7f14ac0612c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:07.585 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.576+0000 7f14b2acc700 1 -- 172.21.15.155:0/756081298 shutdown_connections 2020-11-20T11:32:07.585 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.576+0000 7f14b2acc700 1 -- 172.21.15.155:0/756081298 wait complete. 2020-11-20T11:32:07.588 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.577+0000 7f14b2acc700 1 Processor -- start 2020-11-20T11:32:07.588 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.577+0000 7f14b2acc700 1 -- start start 2020-11-20T11:32:07.588 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.578+0000 7f14b2acc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14ac0619d0 0x7f14ac0acfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:07.588 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.578+0000 7f14b2acc700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14ac061db0 0x7f14ac0a33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:07.589 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.578+0000 7f14b2acc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14ac066170 0x7f14ac14a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:07.589 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.578+0000 7f14b2acc700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac14a040 con 0x7f14ac066170 2020-11-20T11:32:07.589 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.578+0000 7f14b2acc700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac0a4810 con 0x7f14ac0619d0 2020-11-20T11:32:07.589 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.578+0000 7f14b2acc700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac0a2f00 con 0x7f14ac061db0 2020-11-20T11:32:07.589 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.578+0000 7f14abfff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14ac061db0 0x7f14ac0a33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:07.590 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.578+0000 7f14b0868700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14ac0619d0 0x7f14ac0acfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:07.590 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.578+0000 7f14abfff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14ac061db0 0x7f14ac0a33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:58158/0 (socket says 172.21.15.155:58158) 2020-11-20T11:32:07.590 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.578+0000 7f14abfff700 1 -- 172.21.15.155:0/2582251208 learned_addr learned my addr 172.21.15.155:0/2582251208 (peer_addr_for_me v2:172.21.15.155:0/0) 2020-11-20T11:32:07.590 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.578+0000 7f14b0868700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14ac0619d0 0x7f14ac0acfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:41794/0 (socket says 172.21.15.155:41794) 2020-11-20T11:32:07.590 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.578+0000 7f14b1069700 1 --2- 172.21.15.155:0/2582251208 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14ac066170 0x7f14ac14a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:07.591 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.579+0000 7f14b0868700 1 -- 172.21.15.155:0/2582251208 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14ac061db0 msgr2=0x7f14ac0a33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:07.591 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.579+0000 7f14b0868700 1 --2- 172.21.15.155:0/2582251208 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14ac061db0 0x7f14ac0a33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.591 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.579+0000 7f14b0868700 1 -- 172.21.15.155:0/2582251208 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14ac066170 msgr2=0x7f14ac14a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:07.591 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.579+0000 7f14b0868700 1 --2- 172.21.15.155:0/2582251208 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14ac066170 0x7f14ac14a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.591 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.579+0000 7f14b0868700 1 -- 172.21.15.155:0/2582251208 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ac0a4ab0 con 0x7f14ac0619d0 2020-11-20T11:32:07.592 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.579+0000 7f14abfff700 1 --2- 172.21.15.155:0/2582251208 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14ac061db0 0x7f14ac0a33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:32:07.592 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.579+0000 7f14b0868700 1 --2- 172.21.15.155:0/2582251208 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14ac0619d0 0x7f14ac0acfd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f14a000eba0 tx=0x7f14a0006e30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:07.592 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.579+0000 7f14a8ff9700 1 -- 172.21.15.155:0/2582251208 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a000eeb0 con 0x7f14ac0619d0 2020-11-20T11:32:07.592 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.579+0000 7f14a8ff9700 1 -- 172.21.15.155:0/2582251208 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a000c850 con 0x7f14ac0619d0 2020-11-20T11:32:07.592 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.579+0000 7f14b2acc700 1 -- 172.21.15.155:0/2582251208 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14ac0a4490 con 0x7f14ac0619d0 2020-11-20T11:32:07.593 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.579+0000 7f14a8ff9700 1 -- 172.21.15.155:0/2582251208 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a000cbd0 con 0x7f14ac0619d0 2020-11-20T11:32:07.593 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.579+0000 7f14b2acc700 1 -- 172.21.15.155:0/2582251208 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14ac0a3890 con 0x7f14ac0619d0 2020-11-20T11:32:07.593 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.579+0000 7f14a8ff9700 1 -- 172.21.15.155:0/2582251208 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f14a001c6c0 con 0x7f14ac0619d0 2020-11-20T11:32:07.593 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.579+0000 7f14a8ff9700 1 -- 172.21.15.155:0/2582251208 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1099+0+0 (secure 0 0 0) 0x7f14a00128b0 con 0x7f14ac0619d0 2020-11-20T11:32:07.594 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.580+0000 7f14b2acc700 1 -- 172.21.15.155:0/2582251208 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14ac0a30c0 con 0x7f14ac0619d0 2020-11-20T11:32:07.594 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.582+0000 7f14a8ff9700 1 -- 172.21.15.155:0/2582251208 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71786 (secure 0 0 0) 0x7f14a001cb30 con 0x7f14ac0619d0 2020-11-20T11:32:07.663 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.660+0000 7f14b2acc700 1 -- 172.21.15.155:0/2582251208 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1f147dd7-9289-47b0-ad1c-4088618d6faa", "id": 3} v 0) v1 -- 0x7f14ac0652b0 con 0x7f14ac0619d0 2020-11-20T11:32:07.671 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.668+0000 7f14a8ff9700 1 -- 172.21.15.155:0/2582251208 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1f147dd7-9289-47b0-ad1c-4088618d6faa", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f14a0012b40 con 0x7f14ac0619d0 2020-11-20T11:32:07.672 INFO:teuthology.orchestra.run.smithi155.stdout:3 2020-11-20T11:32:07.673 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.669+0000 7f14b2acc700 1 -- 172.21.15.155:0/2582251208 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14ac0619d0 msgr2=0x7f14ac0acfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:07.673 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.669+0000 7f14b2acc700 1 --2- 172.21.15.155:0/2582251208 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14ac0619d0 0x7f14ac0acfd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f14a000eba0 tx=0x7f14a0006e30).stop 2020-11-20T11:32:07.674 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.669+0000 7f14b2acc700 1 -- 172.21.15.155:0/2582251208 shutdown_connections 2020-11-20T11:32:07.674 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.669+0000 7f14b2acc700 1 --2- 172.21.15.155:0/2582251208 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14ac066170 0x7f14ac14a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.674 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.669+0000 7f14b2acc700 1 --2- 172.21.15.155:0/2582251208 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f14ac0619d0 0x7f14ac0acfd0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.674 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.669+0000 7f14b2acc700 1 --2- 172.21.15.155:0/2582251208 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f14ac061db0 0x7f14ac0a33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.675 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.669+0000 7f14b2acc700 1 -- 172.21.15.155:0/2582251208 >> 172.21.15.155:0/2582251208 conn(0x7f14ac063640 msgr2=0x7f14ac0a86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:07.675 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.669+0000 7f14b2acc700 1 -- 172.21.15.155:0/2582251208 shutdown_connections 2020-11-20T11:32:07.675 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.669+0000 7f14b2acc700 1 -- 172.21.15.155:0/2582251208 wait complete. 2020-11-20T11:32:07.683 INFO:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new d008b039-4bc5-4135-9678-1511e58efc29 4 2020-11-20T11:32:07.794 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.791+0000 7f2b307b5700 1 Processor -- start 2020-11-20T11:32:07.794 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.791+0000 7f2b307b5700 1 -- start start 2020-11-20T11:32:07.797 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.791+0000 7f2b307b5700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f2b280b1a30 0x7f2b280a8b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:07.797 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.791+0000 7f2b307b5700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f2b280b1e10 0x7f2b280abcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:07.797 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.791+0000 7f2b307b5700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b280ab390 con 0x7f2b280ac1b0 2020-11-20T11:32:07.798 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.791+0000 7f2b307b5700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b280ab1a0 con 0x7f2b280b1a30 2020-11-20T11:32:07.798 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.791+0000 7f2b307b5700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b280ab630 con 0x7f2b280b1e10 2020-11-20T11:32:07.798 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.792+0000 7f2b2dd50700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f2b280b1e10 0x7f2b280abcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:07.798 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.792+0000 7f2b2dd50700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f2b280b1e10 0x7f2b280abcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:41800/0 (socket says 172.21.15.155:41800) 2020-11-20T11:32:07.799 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.792+0000 7f2b2dd50700 1 -- 172.21.15.155:0/3909941252 learned_addr learned my addr 172.21.15.155:0/3909941252 (peer_addr_for_me v2:172.21.15.155:0/0) 2020-11-20T11:32:07.799 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.792+0000 7f2b2e551700 1 --2- 172.21.15.155:0/3909941252 >> v2:172.21.15.23:3300/0 conn(0x7f2b280b1a30 0x7f2b280a8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:07.799 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.792+0000 7f2b2dd50700 1 -- 172.21.15.155:0/3909941252 >> v2:172.21.15.23:3300/0 conn(0x7f2b280b1a30 msgr2=0x7f2b280a8b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:07.799 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.792+0000 7f2b2dd50700 1 --2- 172.21.15.155:0/3909941252 >> v2:172.21.15.23:3300/0 conn(0x7f2b280b1a30 0x7f2b280a8b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.799 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.792+0000 7f2b2dd50700 1 -- 172.21.15.155:0/3909941252 >> v1:172.21.15.23:6789/0 conn(0x7f2b280ac1b0 legacy=0x7f2b2813e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:07.800 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.792+0000 7f2b2dd50700 1 -- 172.21.15.155:0/3909941252 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b280a9060 con 0x7f2b280b1e10 2020-11-20T11:32:07.800 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.792+0000 7f2b2dd50700 1 --2- 172.21.15.155:0/3909941252 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b280b1e10 0x7f2b280abcc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f2b2805d0e0 tx=0x7f2b18006ef0).ready entity=mon.1 client_cookie=c65cbed44c79013d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:07.800 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.792+0000 7f2b2d54f700 1 -- 172.21.15.155:0/3909941252 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1800b860 con 0x7f2b280b1e10 2020-11-20T11:32:07.800 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.792+0000 7f2b2d54f700 1 -- 172.21.15.155:0/3909941252 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b18000e90 con 0x7f2b280b1e10 2020-11-20T11:32:07.800 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.792+0000 7f2b2d54f700 1 -- 172.21.15.155:0/3909941252 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1800bbd0 con 0x7f2b280b1e10 2020-11-20T11:32:07.801 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.793+0000 7f2b307b5700 1 -- 172.21.15.155:0/3909941252 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b280b1e10 msgr2=0x7f2b280abcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:07.801 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.793+0000 7f2b307b5700 1 --2- 172.21.15.155:0/3909941252 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b280b1e10 0x7f2b280abcc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f2b2805d0e0 tx=0x7f2b18006ef0).stop 2020-11-20T11:32:07.801 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.793+0000 7f2b307b5700 1 -- 172.21.15.155:0/3909941252 shutdown_connections 2020-11-20T11:32:07.801 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.793+0000 7f2b307b5700 1 --2- 172.21.15.155:0/3909941252 >> v2:172.21.15.23:3300/0 conn(0x7f2b280b1a30 0x7f2b280a8b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.801 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.793+0000 7f2b307b5700 1 --2- 172.21.15.155:0/3909941252 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b280b1e10 0x7f2b280abcc0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.802 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.793+0000 7f2b307b5700 1 -- 172.21.15.155:0/3909941252 >> 172.21.15.155:0/3909941252 conn(0x7f2b28068380 msgr2=0x7f2b280b1320 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:07.802 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.793+0000 7f2b307b5700 1 -- 172.21.15.155:0/3909941252 shutdown_connections 2020-11-20T11:32:07.802 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.793+0000 7f2b307b5700 1 -- 172.21.15.155:0/3909941252 wait complete. 2020-11-20T11:32:07.802 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.794+0000 7f2b307b5700 1 Processor -- start 2020-11-20T11:32:07.802 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.794+0000 7f2b307b5700 1 -- start start 2020-11-20T11:32:07.805 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.794+0000 7f2b307b5700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b280ac1b0 0x7f2b280a3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:07.805 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.794+0000 7f2b307b5700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2b280b1a30 0x7f2b2813fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:07.805 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.794+0000 7f2b307b5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b280b1e10 0x7f2b2813efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:07.805 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.794+0000 7f2b307b5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b281409b0 con 0x7f2b280b1e10 2020-11-20T11:32:07.806 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.794+0000 7f2b307b5700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b28140250 con 0x7f2b280ac1b0 2020-11-20T11:32:07.806 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.794+0000 7f2b307b5700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2813e1a0 con 0x7f2b280b1a30 2020-11-20T11:32:07.806 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.794+0000 7f2b2e551700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b280ac1b0 0x7f2b280a3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:07.806 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.794+0000 7f2b2dd50700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2b280b1a30 0x7f2b2813fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:07.807 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.794+0000 7f2b2e551700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b280ac1b0 0x7f2b280a3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:41804/0 (socket says 172.21.15.155:41804) 2020-11-20T11:32:07.807 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.794+0000 7f2b2dd50700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2b280b1a30 0x7f2b2813fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:58166/0 (socket says 172.21.15.155:58166) 2020-11-20T11:32:07.807 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.794+0000 7f2b2e551700 1 -- 172.21.15.155:0/602175035 learned_addr learned my addr 172.21.15.155:0/602175035 (peer_addr_for_me v2:172.21.15.155:0/0) 2020-11-20T11:32:07.807 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.795+0000 7f2b2ed52700 1 --2- 172.21.15.155:0/602175035 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b280b1e10 0x7f2b2813efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:07.807 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.795+0000 7f2b2dd50700 1 -- 172.21.15.155:0/602175035 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b280ac1b0 msgr2=0x7f2b280a3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:07.808 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.795+0000 7f2b2dd50700 1 --2- 172.21.15.155:0/602175035 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b280ac1b0 0x7f2b280a3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.811 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.795+0000 7f2b2dd50700 1 -- 172.21.15.155:0/602175035 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b280b1e10 msgr2=0x7f2b2813efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:07.811 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.795+0000 7f2b2dd50700 1 --2- 172.21.15.155:0/602175035 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b280b1e10 0x7f2b2813efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.811 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.795+0000 7f2b2dd50700 1 -- 172.21.15.155:0/602175035 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b28140470 con 0x7f2b280b1a30 2020-11-20T11:32:07.811 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.795+0000 7f2b2e551700 1 --2- 172.21.15.155:0/602175035 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b280ac1b0 0x7f2b280a3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:32:07.812 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.795+0000 7f2b2dd50700 1 --2- 172.21.15.155:0/602175035 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2b280b1a30 0x7f2b2813fd60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f2b280ad9e0 tx=0x7f2b18016000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:07.812 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.795+0000 7f2b167fc700 1 -- 172.21.15.155:0/602175035 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b18011a70 con 0x7f2b280b1a30 2020-11-20T11:32:07.812 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.795+0000 7f2b167fc700 1 -- 172.21.15.155:0/602175035 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b18011bd0 con 0x7f2b280b1a30 2020-11-20T11:32:07.812 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.795+0000 7f2b307b5700 1 -- 172.21.15.155:0/602175035 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b28140710 con 0x7f2b280b1a30 2020-11-20T11:32:07.812 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.795+0000 7f2b167fc700 1 -- 172.21.15.155:0/602175035 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1801d450 con 0x7f2b280b1a30 2020-11-20T11:32:07.813 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.795+0000 7f2b307b5700 1 -- 172.21.15.155:0/602175035 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b2813f4b0 con 0x7f2b280b1a30 2020-11-20T11:32:07.813 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.795+0000 7f2b167fc700 1 -- 172.21.15.155:0/602175035 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f2b18017040 con 0x7f2b280b1a30 2020-11-20T11:32:07.813 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.795+0000 7f2b167fc700 1 -- 172.21.15.155:0/602175035 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1214+0+0 (secure 0 0 0) 0x7f2b18019940 con 0x7f2b280b1a30 2020-11-20T11:32:07.813 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.796+0000 7f2b307b5700 1 -- 172.21.15.155:0/602175035 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b28060640 con 0x7f2b280b1a30 2020-11-20T11:32:07.813 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.798+0000 7f2b167fc700 1 -- 172.21.15.155:0/602175035 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71786 (secure 0 0 0) 0x7f2b18021020 con 0x7f2b280b1a30 2020-11-20T11:32:07.878 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.875+0000 7f2b307b5700 1 -- 172.21.15.155:0/602175035 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d008b039-4bc5-4135-9678-1511e58efc29", "id": 4} v 0) v1 -- 0x7f2b2805c7d0 con 0x7f2b280b1a30 2020-11-20T11:32:07.886 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.883+0000 7f2b167fc700 1 -- 172.21.15.155:0/602175035 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d008b039-4bc5-4135-9678-1511e58efc29", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2b18015020 con 0x7f2b280b1a30 2020-11-20T11:32:07.886 INFO:teuthology.orchestra.run.smithi155.stdout:4 2020-11-20T11:32:07.887 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.884+0000 7f2b307b5700 1 -- 172.21.15.155:0/602175035 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2b280b1a30 msgr2=0x7f2b2813fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:07.887 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.884+0000 7f2b307b5700 1 --2- 172.21.15.155:0/602175035 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2b280b1a30 0x7f2b2813fd60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f2b280ad9e0 tx=0x7f2b18016000).stop 2020-11-20T11:32:07.888 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.884+0000 7f2b307b5700 1 -- 172.21.15.155:0/602175035 shutdown_connections 2020-11-20T11:32:07.888 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.884+0000 7f2b307b5700 1 --2- 172.21.15.155:0/602175035 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b280b1e10 0x7f2b2813efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.888 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.884+0000 7f2b307b5700 1 --2- 172.21.15.155:0/602175035 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2b280ac1b0 0x7f2b280a3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.888 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.884+0000 7f2b307b5700 1 --2- 172.21.15.155:0/602175035 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2b280b1a30 0x7f2b2813fd60 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:07.889 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.884+0000 7f2b307b5700 1 -- 172.21.15.155:0/602175035 >> 172.21.15.155:0/602175035 conn(0x7f2b28068380 msgr2=0x7f2b2813a390 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:07.889 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.884+0000 7f2b307b5700 1 -- 172.21.15.155:0/602175035 shutdown_connections 2020-11-20T11:32:07.889 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:07.884+0000 7f2b307b5700 1 -- 172.21.15.155:0/602175035 wait complete. 2020-11-20T11:32:07.898 INFO:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new 56715e6f-cf6e-44a6-9e39-93ab1224095f 5 2020-11-20T11:32:08.011 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.008+0000 7f44f5b69700 1 Processor -- start 2020-11-20T11:32:08.012 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.008+0000 7f44f5b69700 1 -- start start 2020-11-20T11:32:08.016 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.009+0000 7f44f5b69700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44f0137410 0x7f44f0137870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:08.017 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.009+0000 7f44f5b69700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f0062ad0 con 0x7f44f00a6e10 2020-11-20T11:32:08.017 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.009+0000 7f44f5b69700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f00a63e0 con 0x7f44f0133dc0 2020-11-20T11:32:08.017 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.009+0000 7f44f5b69700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44f00a61f0 con 0x7f44f0137410 2020-11-20T11:32:08.017 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.009+0000 7f44eeffd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44f0137410 0x7f44f0137870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:08.018 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.009+0000 7f44effff700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f44f0133dc0 0x7f44f0134230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:36574/0 (socket says 172.21.15.155:36574) 2020-11-20T11:32:08.018 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.009+0000 7f44eeffd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44f0137410 0x7f44f0137870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:58174/0 (socket says 172.21.15.155:58174) 2020-11-20T11:32:08.018 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.009+0000 7f44effff700 1 -- 172.21.15.155:0/3600473663 learned_addr learned my addr 172.21.15.155:0/3600473663 (peer_addr_for_me v1:172.21.15.155:0/0) 2020-11-20T11:32:08.018 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.009+0000 7f44ee7fc700 1 -- 172.21.15.155:0/3600473663 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 548135825 0 0) 0x7f44f00a63e0 con 0x7f44f0133dc0 2020-11-20T11:32:08.019 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.009+0000 7f44eeffd700 1 -- 172.21.15.155:0/3600473663 >> v1:172.21.15.155:6789/0 conn(0x7f44f0133dc0 legacy=0x7f44f0134230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:08.019 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.009+0000 7f44eeffd700 1 -- 172.21.15.155:0/3600473663 >> v1:172.21.15.23:6789/0 conn(0x7f44f00a6e10 legacy=0x7f44f0061850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:08.019 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.009+0000 7f44eeffd700 1 -- 172.21.15.155:0/3600473663 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f00a6680 con 0x7f44f0137410 2020-11-20T11:32:08.020 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.009+0000 7f44eeffd700 1 --2- 172.21.15.155:0/3600473663 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44f0137410 0x7f44f0137870 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f44d800ba00 tx=0x7f44d8006e30).ready entity=mon.2 client_cookie=e811976915943f88 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:08.020 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.009+0000 7f44ee7fc700 1 -- 172.21.15.155:0/3600473663 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d80097e0 con 0x7f44f0137410 2020-11-20T11:32:08.020 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.009+0000 7f44ee7fc700 1 -- 172.21.15.155:0/3600473663 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d8000e90 con 0x7f44f0137410 2020-11-20T11:32:08.020 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.009+0000 7f44ee7fc700 1 -- 172.21.15.155:0/3600473663 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d8009b50 con 0x7f44f0137410 2020-11-20T11:32:08.021 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.010+0000 7f44f5b69700 1 -- 172.21.15.155:0/3600473663 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44f0137410 msgr2=0x7f44f0137870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:08.021 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.010+0000 7f44f5b69700 1 --2- 172.21.15.155:0/3600473663 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44f0137410 0x7f44f0137870 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f44d800ba00 tx=0x7f44d8006e30).stop 2020-11-20T11:32:08.021 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.010+0000 7f44f5b69700 1 -- 172.21.15.155:0/3600473663 shutdown_connections 2020-11-20T11:32:08.021 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.010+0000 7f44f5b69700 1 --2- 172.21.15.155:0/3600473663 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44f0137410 0x7f44f0137870 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:08.022 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.010+0000 7f44f5b69700 1 -- 172.21.15.155:0/3600473663 >> 172.21.15.155:0/3600473663 conn(0x7f44f0060c80 msgr2=0x7f44f00acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:08.022 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.010+0000 7f44f5b69700 1 -- 172.21.15.155:0/3600473663 shutdown_connections 2020-11-20T11:32:08.022 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.010+0000 7f44f5b69700 1 -- 172.21.15.155:0/3600473663 wait complete. 2020-11-20T11:32:08.022 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.011+0000 7f44f5b69700 1 Processor -- start 2020-11-20T11:32:08.023 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.011+0000 7f44f5b69700 1 -- start start 2020-11-20T11:32:08.023 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.011+0000 7f44f5b69700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44f00a6e10 0x7f44f0065b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:08.023 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.011+0000 7f44f5b69700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44f0133dc0 0x7f44f00a99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:08.023 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.012+0000 7f44f5b69700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44f0137410 0x7f44f014d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:08.024 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.012+0000 7f44f5b69700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f00a78a0 con 0x7f44f00a6e10 2020-11-20T11:32:08.024 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.012+0000 7f44f5b69700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f014a2f0 con 0x7f44f0133dc0 2020-11-20T11:32:08.024 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.012+0000 7f44f5b69700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44f014a590 con 0x7f44f0137410 2020-11-20T11:32:08.024 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.012+0000 7f44eeffd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44f0133dc0 0x7f44f00a99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:08.025 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.012+0000 7f44effff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44f0137410 0x7f44f014d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:08.025 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.012+0000 7f44eeffd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44f0133dc0 0x7f44f00a99f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:41818/0 (socket says 172.21.15.155:41818) 2020-11-20T11:32:08.025 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.012+0000 7f44eeffd700 1 -- 172.21.15.155:0/2402137889 learned_addr learned my addr 172.21.15.155:0/2402137889 (peer_addr_for_me v2:172.21.15.155:0/0) 2020-11-20T11:32:08.025 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.012+0000 7f44ef7fe700 1 --2- 172.21.15.155:0/2402137889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44f00a6e10 0x7f44f0065b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:08.026 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.012+0000 7f44eeffd700 1 -- 172.21.15.155:0/2402137889 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44f0137410 msgr2=0x7f44f014d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:08.026 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.012+0000 7f44eeffd700 1 --2- 172.21.15.155:0/2402137889 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44f0137410 0x7f44f014d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:08.026 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.012+0000 7f44eeffd700 1 -- 172.21.15.155:0/2402137889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44f00a6e10 msgr2=0x7f44f0065b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:08.026 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.012+0000 7f44eeffd700 1 --2- 172.21.15.155:0/2402137889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44f00a6e10 0x7f44f0065b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:08.027 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.012+0000 7f44eeffd700 1 -- 172.21.15.155:0/2402137889 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f014a830 con 0x7f44f0133dc0 2020-11-20T11:32:08.027 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.012+0000 7f44eeffd700 1 --2- 172.21.15.155:0/2402137889 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44f0133dc0 0x7f44f00a99f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f44d8003000 tx=0x7f44d8015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:08.027 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.012+0000 7f44e77fe700 1 -- 172.21.15.155:0/2402137889 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d8011c50 con 0x7f44f0133dc0 2020-11-20T11:32:08.027 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.012+0000 7f44e77fe700 1 -- 172.21.15.155:0/2402137889 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d8011db0 con 0x7f44f0133dc0 2020-11-20T11:32:08.028 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.012+0000 7f44f5b69700 1 -- 172.21.15.155:0/2402137889 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44f00a9ee0 con 0x7f44f0133dc0 2020-11-20T11:32:08.028 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.012+0000 7f44e77fe700 1 -- 172.21.15.155:0/2402137889 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d801d660 con 0x7f44f0133dc0 2020-11-20T11:32:08.028 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.012+0000 7f44f5b69700 1 -- 172.21.15.155:0/2402137889 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44f0149f00 con 0x7f44f0133dc0 2020-11-20T11:32:08.028 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.012+0000 7f44e77fe700 1 -- 172.21.15.155:0/2402137889 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f44d800f070 con 0x7f44f0133dc0 2020-11-20T11:32:08.028 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.012+0000 7f44e77fe700 1 -- 172.21.15.155:0/2402137889 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1329+0+0 (secure 0 0 0) 0x7f44d8027a10 con 0x7f44f0133dc0 2020-11-20T11:32:08.029 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.013+0000 7f44f5b69700 1 -- 172.21.15.155:0/2402137889 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44f014a9c0 con 0x7f44f0133dc0 2020-11-20T11:32:08.029 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.015+0000 7f44e77fe700 1 -- 172.21.15.155:0/2402137889 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71786 (secure 0 0 0) 0x7f44d801da10 con 0x7f44f0133dc0 2020-11-20T11:32:08.095 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.092+0000 7f44f5b69700 1 -- 172.21.15.155:0/2402137889 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd new", "uuid": "56715e6f-cf6e-44a6-9e39-93ab1224095f", "id": 5} v 0) v1 -- 0x7f44f014a0f0 con 0x7f44f0133dc0 2020-11-20T11:32:08.099 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.096+0000 7f44e77fe700 1 -- 172.21.15.155:0/2402137889 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "56715e6f-cf6e-44a6-9e39-93ab1224095f", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f44d80360b0 con 0x7f44f0133dc0 2020-11-20T11:32:08.099 INFO:teuthology.orchestra.run.smithi155.stdout:5 2020-11-20T11:32:08.100 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.097+0000 7f44f5b69700 1 -- 172.21.15.155:0/2402137889 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44f0133dc0 msgr2=0x7f44f00a99f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:08.101 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.097+0000 7f44f5b69700 1 --2- 172.21.15.155:0/2402137889 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44f0133dc0 0x7f44f00a99f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f44d8003000 tx=0x7f44d8015000).stop 2020-11-20T11:32:08.101 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.097+0000 7f44f5b69700 1 -- 172.21.15.155:0/2402137889 shutdown_connections 2020-11-20T11:32:08.101 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.097+0000 7f44f5b69700 1 --2- 172.21.15.155:0/2402137889 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44f0137410 0x7f44f014d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:08.101 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.097+0000 7f44f5b69700 1 --2- 172.21.15.155:0/2402137889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44f00a6e10 0x7f44f0065b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:08.102 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.097+0000 7f44f5b69700 1 --2- 172.21.15.155:0/2402137889 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f44f0133dc0 0x7f44f00a99f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:08.102 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.097+0000 7f44f5b69700 1 -- 172.21.15.155:0/2402137889 >> 172.21.15.155:0/2402137889 conn(0x7f44f0060c80 msgr2=0x7f44f0139df0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:08.102 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.097+0000 7f44f5b69700 1 -- 172.21.15.155:0/2402137889 shutdown_connections 2020-11-20T11:32:08.102 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.097+0000 7f44f5b69700 1 -- 172.21.15.155:0/2402137889 wait complete. 2020-11-20T11:32:08.111 INFO:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new db1e61ca-8e14-4a1f-8607-6e01b148d61c 6 2020-11-20T11:32:08.222 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.219+0000 7f65b5a3c700 1 Processor -- start 2020-11-20T11:32:08.222 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.219+0000 7f65b5a3c700 1 -- start start 2020-11-20T11:32:08.224 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.219+0000 7f65b5a3c700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65b00aae90 0x7f65b01381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:08.225 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.219+0000 7f65b5a3c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f65b013aeb0 0x7f65b00a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:08.225 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.219+0000 7f65b5a3c700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b013e600 con 0x7f65b00aaab0 2020-11-20T11:32:08.225 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.219+0000 7f65b5a3c700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65b013e8a0 con 0x7f65b013aeb0 2020-11-20T11:32:08.225 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.219+0000 7f65b5a3c700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b00ab6d0 con 0x7f65b00aae90 2020-11-20T11:32:08.225 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.219+0000 7f65ae7fc700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65b00aae90 0x7f65b01381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:08.226 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.219+0000 7f65aeffd700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f65b00aaab0 0x7f65b013ed40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:36582/0 (socket says 172.21.15.155:36582) 2020-11-20T11:32:08.226 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.219+0000 7f65aeffd700 1 -- 172.21.15.155:0/2814714846 learned_addr learned my addr 172.21.15.155:0/2814714846 (peer_addr_for_me v1:172.21.15.155:0/0) 2020-11-20T11:32:08.226 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.220+0000 7f65af7fe700 1 --2- 172.21.15.155:0/2814714846 >> v2:172.21.15.23:3300/0 conn(0x7f65b013aeb0 0x7f65b00a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:08.226 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.220+0000 7f65adffb700 1 -- 172.21.15.155:0/2814714846 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1982166233 0 0) 0x7f65b013e600 con 0x7f65b00aaab0 2020-11-20T11:32:08.227 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.220+0000 7f65adffb700 1 -- 172.21.15.155:0/2814714846 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f659c003280 con 0x7f65b00aaab0 2020-11-20T11:32:08.227 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.220+0000 7f65ae7fc700 1 -- 172.21.15.155:0/2814714846 >> v2:172.21.15.23:3300/0 conn(0x7f65b013aeb0 msgr2=0x7f65b00a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:08.227 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.220+0000 7f65ae7fc700 1 --2- 172.21.15.155:0/2814714846 >> v2:172.21.15.23:3300/0 conn(0x7f65b013aeb0 0x7f65b00a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:08.227 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.220+0000 7f65ae7fc700 1 -- 172.21.15.155:0/2814714846 >> v1:172.21.15.155:6789/0 conn(0x7f65b00aaab0 legacy=0x7f65b013ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:08.227 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.220+0000 7f65ae7fc700 1 -- 172.21.15.155:0/2814714846 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b0138690 con 0x7f65b00aae90 2020-11-20T11:32:08.227 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.220+0000 7f65ae7fc700 1 --2- 172.21.15.155:0/2814714846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65b00aae90 0x7f65b01381a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f65a0006fd0 tx=0x7f65a0006e30).ready entity=mon.2 client_cookie=f57b44d709dfb916 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:08.228 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.220+0000 7f65adffb700 1 -- 172.21.15.155:0/2814714846 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a0009630 con 0x7f65b00aae90 2020-11-20T11:32:08.228 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.220+0000 7f65adffb700 1 -- 172.21.15.155:0/2814714846 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a0000e90 con 0x7f65b00aae90 2020-11-20T11:32:08.228 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.220+0000 7f65adffb700 1 -- 172.21.15.155:0/2814714846 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a00099a0 con 0x7f65b00aae90 2020-11-20T11:32:08.228 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.221+0000 7f65b5a3c700 1 -- 172.21.15.155:0/2814714846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65b00aae90 msgr2=0x7f65b01381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:08.228 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.221+0000 7f65b5a3c700 1 --2- 172.21.15.155:0/2814714846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65b00aae90 0x7f65b01381a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f65a0006fd0 tx=0x7f65a0006e30).stop 2020-11-20T11:32:08.229 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.221+0000 7f65b5a3c700 1 -- 172.21.15.155:0/2814714846 shutdown_connections 2020-11-20T11:32:08.229 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.221+0000 7f65b5a3c700 1 --2- 172.21.15.155:0/2814714846 >> v2:172.21.15.23:3300/0 conn(0x7f65b013aeb0 0x7f65b00a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:08.229 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.221+0000 7f65b5a3c700 1 --2- 172.21.15.155:0/2814714846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65b00aae90 0x7f65b01381a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:08.229 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.221+0000 7f65b5a3c700 1 -- 172.21.15.155:0/2814714846 >> 172.21.15.155:0/2814714846 conn(0x7f65b00678e0 msgr2=0x7f65b0067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:08.230 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.221+0000 7f65b5a3c700 1 -- 172.21.15.155:0/2814714846 shutdown_connections 2020-11-20T11:32:08.230 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.221+0000 7f65b5a3c700 1 -- 172.21.15.155:0/2814714846 wait complete. 2020-11-20T11:32:08.232 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.222+0000 7f65b5a3c700 1 Processor -- start 2020-11-20T11:32:08.232 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.222+0000 7f65b5a3c700 1 -- start start 2020-11-20T11:32:08.232 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.222+0000 7f65b5a3c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65b00aaab0 0x7f65b0136000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:08.233 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.222+0000 7f65b5a3c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b00aae90 0x7f65b014a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:08.233 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.222+0000 7f65b5a3c700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65b009ecf0 0x7f65b014ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:08.233 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.222+0000 7f65b5a3c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b014b130 con 0x7f65b00aae90 2020-11-20T11:32:08.233 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.222+0000 7f65b5a3c700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b00a7bf0 con 0x7f65b00aaab0 2020-11-20T11:32:08.233 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.222+0000 7f65b5a3c700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b00a76b0 con 0x7f65b009ecf0 2020-11-20T11:32:08.234 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.223+0000 7f65af7fe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65b009ecf0 0x7f65b014ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:08.234 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.223+0000 7f65aeffd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65b00aaab0 0x7f65b0136000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:08.234 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.223+0000 7f65af7fe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65b009ecf0 0x7f65b014ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:58194/0 (socket says 172.21.15.155:58194) 2020-11-20T11:32:08.234 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.223+0000 7f65af7fe700 1 -- 172.21.15.155:0/3361545064 learned_addr learned my addr 172.21.15.155:0/3361545064 (peer_addr_for_me v2:172.21.15.155:0/0) 2020-11-20T11:32:08.235 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.223+0000 7f65ae7fc700 1 --2- 172.21.15.155:0/3361545064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b00aae90 0x7f65b014a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:08.235 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.223+0000 7f65af7fe700 1 -- 172.21.15.155:0/3361545064 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65b00aaab0 msgr2=0x7f65b0136000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:08.235 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.223+0000 7f65af7fe700 1 --2- 172.21.15.155:0/3361545064 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65b00aaab0 0x7f65b0136000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:08.235 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.223+0000 7f65af7fe700 1 -- 172.21.15.155:0/3361545064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b00aae90 msgr2=0x7f65b014a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:08.235 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.223+0000 7f65af7fe700 1 --2- 172.21.15.155:0/3361545064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b00aae90 0x7f65b014a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:08.236 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.223+0000 7f65af7fe700 1 -- 172.21.15.155:0/3361545064 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b00a7950 con 0x7f65b009ecf0 2020-11-20T11:32:08.236 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.223+0000 7f65aeffd700 1 --2- 172.21.15.155:0/3361545064 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65b00aaab0 0x7f65b0136000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:32:08.236 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.223+0000 7f65af7fe700 1 --2- 172.21.15.155:0/3361545064 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65b009ecf0 0x7f65b014ac40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f65a800ed20 tx=0x7f65a8009220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:08.236 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.223+0000 7f6596ffd700 1 -- 172.21.15.155:0/3361545064 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a800c9b0 con 0x7f65b009ecf0 2020-11-20T11:32:08.236 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.223+0000 7f6596ffd700 1 -- 172.21.15.155:0/3361545064 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a800cb10 con 0x7f65b009ecf0 2020-11-20T11:32:08.237 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.223+0000 7f65b5a3c700 1 -- 172.21.15.155:0/3361545064 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b00a5410 con 0x7f65b009ecf0 2020-11-20T11:32:08.237 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.223+0000 7f6596ffd700 1 -- 172.21.15.155:0/3361545064 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a800ceb0 con 0x7f65b009ecf0 2020-11-20T11:32:08.237 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.223+0000 7f65b5a3c700 1 -- 172.21.15.155:0/3361545064 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b00a7440 con 0x7f65b009ecf0 2020-11-20T11:32:08.237 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.223+0000 7f6596ffd700 1 -- 172.21.15.155:0/3361545064 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f65a8022070 con 0x7f65b009ecf0 2020-11-20T11:32:08.237 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.223+0000 7f6596ffd700 1 -- 172.21.15.155:0/3361545064 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1444+0+0 (secure 0 0 0) 0x7f65a801bac0 con 0x7f65b009ecf0 2020-11-20T11:32:08.238 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.224+0000 7f6594ff9700 1 -- 172.21.15.155:0/3361545064 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6580004fd0 con 0x7f65b009ecf0 2020-11-20T11:32:08.238 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.226+0000 7f6596ffd700 1 -- 172.21.15.155:0/3361545064 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71786 (secure 0 0 0) 0x7f65a803a880 con 0x7f65b009ecf0 2020-11-20T11:32:08.308 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.305+0000 7f6594ff9700 1 -- 172.21.15.155:0/3361545064 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "osd new", "uuid": "db1e61ca-8e14-4a1f-8607-6e01b148d61c", "id": 6} v 0) v1 -- 0x7f6580004070 con 0x7f65b009ecf0 2020-11-20T11:32:08.315 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.312+0000 7f6596ffd700 1 -- 172.21.15.155:0/3361545064 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "db1e61ca-8e14-4a1f-8607-6e01b148d61c", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f65a8029030 con 0x7f65b009ecf0 2020-11-20T11:32:08.316 INFO:teuthology.orchestra.run.smithi155.stdout:6 2020-11-20T11:32:08.317 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.313+0000 7f6594ff9700 1 -- 172.21.15.155:0/3361545064 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65b009ecf0 msgr2=0x7f65b014ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:08.317 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.313+0000 7f6594ff9700 1 --2- 172.21.15.155:0/3361545064 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65b009ecf0 0x7f65b014ac40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f65a800ed20 tx=0x7f65a8009220).stop 2020-11-20T11:32:08.317 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.313+0000 7f6594ff9700 1 -- 172.21.15.155:0/3361545064 shutdown_connections 2020-11-20T11:32:08.317 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.313+0000 7f6594ff9700 1 --2- 172.21.15.155:0/3361545064 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65b009ecf0 0x7f65b014ac40 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:08.318 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.313+0000 7f6594ff9700 1 --2- 172.21.15.155:0/3361545064 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65b00aaab0 0x7f65b0136000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:08.318 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.313+0000 7f6594ff9700 1 --2- 172.21.15.155:0/3361545064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65b00aae90 0x7f65b014a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:08.318 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.313+0000 7f6594ff9700 1 -- 172.21.15.155:0/3361545064 >> 172.21.15.155:0/3361545064 conn(0x7f65b00678e0 msgr2=0x7f65b00aba00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:08.318 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.313+0000 7f6594ff9700 1 -- 172.21.15.155:0/3361545064 shutdown_connections 2020-11-20T11:32:08.318 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.313+0000 7f6594ff9700 1 -- 172.21.15.155:0/3361545064 wait complete. 2020-11-20T11:32:08.327 INFO:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd new d44419be-ef89-462c-8cf1-3c6db2a6944b 7 2020-11-20T11:32:08.440 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.437+0000 7fed5f936700 1 Processor -- start 2020-11-20T11:32:08.441 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.437+0000 7fed5f936700 1 -- start start 2020-11-20T11:32:08.443 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.438+0000 7fed5f936700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fed58061380 0x7fed580a6c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:08.443 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.438+0000 7fed5f936700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fed580a9a80 0x7fed58133db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:08.444 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.438+0000 7fed5f936700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed580a51f0 con 0x7fed580a7150 2020-11-20T11:32:08.444 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.438+0000 7fed5f936700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed580a5490 con 0x7fed580a9a80 2020-11-20T11:32:08.444 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.438+0000 7fed5f936700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed580a5730 con 0x7fed58061380 2020-11-20T11:32:08.444 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.438+0000 7fed5d6d2700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fed58061380 0x7fed580a6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:08.444 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.438+0000 7fed5ded3700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fed580a9a80 0x7fed58133db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:08.445 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.438+0000 7fed5d6d2700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fed58061380 0x7fed580a6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:58196/0 (socket says 172.21.15.155:58196) 2020-11-20T11:32:08.445 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.438+0000 7fed5ded3700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fed580a9a80 0x7fed58133db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:41838/0 (socket says 172.21.15.155:41838) 2020-11-20T11:32:08.445 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.438+0000 7fed5d6d2700 1 -- 172.21.15.155:0/3237558823 learned_addr learned my addr 172.21.15.155:0/3237558823 (peer_addr_for_me v2:172.21.15.155:0/0) 2020-11-20T11:32:08.445 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.438+0000 7fed5ced1700 1 --1- 172.21.15.155:0/3237558823 >> v1:172.21.15.155:6789/0 conn(0x7fed580a7150 0x7fed580a9370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:36594/0 (socket says 172.21.15.155:36594) 2020-11-20T11:32:08.445 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.438+0000 7fed47fff700 1 -- 172.21.15.155:0/3237558823 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3150391407 0 0) 0x7fed580a51f0 con 0x7fed580a7150 2020-11-20T11:32:08.446 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.438+0000 7fed47fff700 1 -- 172.21.15.155:0/3237558823 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed40003280 con 0x7fed580a7150 2020-11-20T11:32:08.446 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.439+0000 7fed5d6d2700 1 -- 172.21.15.155:0/3237558823 >> v2:172.21.15.155:3300/0 conn(0x7fed580a9a80 msgr2=0x7fed58133db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:08.446 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.439+0000 7fed5d6d2700 1 --2- 172.21.15.155:0/3237558823 >> v2:172.21.15.155:3300/0 conn(0x7fed580a9a80 0x7fed58133db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:08.446 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.439+0000 7fed5d6d2700 1 -- 172.21.15.155:0/3237558823 >> v1:172.21.15.155:6789/0 conn(0x7fed580a7150 legacy=0x7fed580a9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:08.446 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.439+0000 7fed5d6d2700 1 -- 172.21.15.155:0/3237558823 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed58061760 con 0x7fed58061380 2020-11-20T11:32:08.447 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.439+0000 7fed5d6d2700 1 --2- 172.21.15.155:0/3237558823 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fed58061380 0x7fed580a6c60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fed4800ab40 tx=0x7fed4800ae50).ready entity=mon.2 client_cookie=21dea0753676e6dc server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:08.447 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.439+0000 7fed5ded3700 1 --2- 172.21.15.155:0/3237558823 >> v2:172.21.15.155:3300/0 conn(0x7fed580a9a80 0x7fed58133db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:32:08.447 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.439+0000 7fed47fff700 1 -- 172.21.15.155:0/3237558823 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4800c970 con 0x7fed58061380 2020-11-20T11:32:08.447 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.439+0000 7fed47fff700 1 -- 172.21.15.155:0/3237558823 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed48000e90 con 0x7fed58061380 2020-11-20T11:32:08.447 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.439+0000 7fed47fff700 1 -- 172.21.15.155:0/3237558823 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4800cce0 con 0x7fed58061380 2020-11-20T11:32:08.448 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.439+0000 7fed5f936700 1 -- 172.21.15.155:0/3237558823 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fed58061380 msgr2=0x7fed580a6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:08.448 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.439+0000 7fed5f936700 1 --2- 172.21.15.155:0/3237558823 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fed58061380 0x7fed580a6c60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fed4800ab40 tx=0x7fed4800ae50).stop 2020-11-20T11:32:08.448 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.439+0000 7fed5f936700 1 -- 172.21.15.155:0/3237558823 shutdown_connections 2020-11-20T11:32:08.448 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.439+0000 7fed5f936700 1 --2- 172.21.15.155:0/3237558823 >> v2:172.21.15.155:3300/0 conn(0x7fed580a9a80 0x7fed58133db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:08.448 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.439+0000 7fed5f936700 1 --2- 172.21.15.155:0/3237558823 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fed58061380 0x7fed580a6c60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:08.449 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.440+0000 7fed5f936700 1 -- 172.21.15.155:0/3237558823 >> 172.21.15.155:0/3237558823 conn(0x7fed580ae280 msgr2=0x7fed580aa290 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:08.449 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.440+0000 7fed5f936700 1 -- 172.21.15.155:0/3237558823 shutdown_connections 2020-11-20T11:32:08.449 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.440+0000 7fed5f936700 1 -- 172.21.15.155:0/3237558823 wait complete. 2020-11-20T11:32:08.452 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.441+0000 7fed5f936700 1 Processor -- start 2020-11-20T11:32:08.452 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.441+0000 7fed5f936700 1 -- start start 2020-11-20T11:32:08.452 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.441+0000 7fed5f936700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed58061380 0x7fed580acb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:08.452 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.441+0000 7fed5f936700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fed580a7150 0x7fed5812d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:08.453 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.441+0000 7fed5f936700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed580a9a80 0x7fed5812b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:08.453 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.441+0000 7fed5f936700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed580ad000 con 0x7fed580a9a80 2020-11-20T11:32:08.453 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.441+0000 7fed5f936700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5812b190 con 0x7fed58061380 2020-11-20T11:32:08.453 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.441+0000 7fed5f936700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed5812d670 con 0x7fed580a7150 2020-11-20T11:32:08.454 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.441+0000 7fed5ced1700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fed580a7150 0x7fed5812d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:08.454 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.441+0000 7fed5d6d2700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed58061380 0x7fed580acb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:08.454 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.441+0000 7fed5ced1700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fed580a7150 0x7fed5812d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:58202/0 (socket says 172.21.15.155:58202) 2020-11-20T11:32:08.454 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.441+0000 7fed5d6d2700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed58061380 0x7fed580acb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:41840/0 (socket says 172.21.15.155:41840) 2020-11-20T11:32:08.454 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.441+0000 7fed5ced1700 1 -- 172.21.15.155:0/2671915744 learned_addr learned my addr 172.21.15.155:0/2671915744 (peer_addr_for_me v2:172.21.15.155:0/0) 2020-11-20T11:32:08.455 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.442+0000 7fed5ded3700 1 --2- 172.21.15.155:0/2671915744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed580a9a80 0x7fed5812b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:08.455 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.442+0000 7fed5ced1700 1 -- 172.21.15.155:0/2671915744 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed58061380 msgr2=0x7fed580acb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:08.455 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.442+0000 7fed5ced1700 1 --2- 172.21.15.155:0/2671915744 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed58061380 0x7fed580acb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:08.455 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.442+0000 7fed5ced1700 1 -- 172.21.15.155:0/2671915744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed580a9a80 msgr2=0x7fed5812b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:08.455 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.442+0000 7fed5ced1700 1 --2- 172.21.15.155:0/2671915744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed580a9a80 0x7fed5812b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:08.456 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.442+0000 7fed5ced1700 1 -- 172.21.15.155:0/2671915744 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed58149510 con 0x7fed580a7150 2020-11-20T11:32:08.456 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.442+0000 7fed5ced1700 1 --2- 172.21.15.155:0/2671915744 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fed580a7150 0x7fed5812d180 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fed5400c710 tx=0x7fed54007ae0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:08.456 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.442+0000 7fed457fa700 1 -- 172.21.15.155:0/2671915744 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed540072e0 con 0x7fed580a7150 2020-11-20T11:32:08.456 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.442+0000 7fed457fa700 1 -- 172.21.15.155:0/2671915744 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed54005570 con 0x7fed580a7150 2020-11-20T11:32:08.456 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.442+0000 7fed5f936700 1 -- 172.21.15.155:0/2671915744 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed5812a840 con 0x7fed580a7150 2020-11-20T11:32:08.457 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.442+0000 7fed457fa700 1 -- 172.21.15.155:0/2671915744 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed540058f0 con 0x7fed580a7150 2020-11-20T11:32:08.457 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.442+0000 7fed5f936700 1 -- 172.21.15.155:0/2671915744 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed581388c0 con 0x7fed580a7150 2020-11-20T11:32:08.457 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.442+0000 7fed457fa700 1 -- 172.21.15.155:0/2671915744 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fed5401a060 con 0x7fed580a7150 2020-11-20T11:32:08.457 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.442+0000 7fed457fa700 1 -- 172.21.15.155:0/2671915744 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1559+0+0 (secure 0 0 0) 0x7fed5401c3f0 con 0x7fed580a7150 2020-11-20T11:32:08.457 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.443+0000 7fed5f936700 1 -- 172.21.15.155:0/2671915744 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed58061e10 con 0x7fed580a7150 2020-11-20T11:32:08.458 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.448+0000 7fed457fa700 1 -- 172.21.15.155:0/2671915744 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71786 (secure 0 0 0) 0x7fed54018070 con 0x7fed580a7150 2020-11-20T11:32:08.528 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.525+0000 7fed5f936700 1 -- 172.21.15.155:0/2671915744 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d44419be-ef89-462c-8cf1-3c6db2a6944b", "id": 7} v 0) v1 -- 0x7fed5812a9d0 con 0x7fed580a7150 2020-11-20T11:32:08.532 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.529+0000 7fed457fa700 1 -- 172.21.15.155:0/2671915744 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d44419be-ef89-462c-8cf1-3c6db2a6944b", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fed540137c0 con 0x7fed580a7150 2020-11-20T11:32:08.532 INFO:teuthology.orchestra.run.smithi155.stdout:7 2020-11-20T11:32:08.534 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.530+0000 7fed5f936700 1 -- 172.21.15.155:0/2671915744 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fed580a7150 msgr2=0x7fed5812d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:08.534 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.530+0000 7fed5f936700 1 --2- 172.21.15.155:0/2671915744 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fed580a7150 0x7fed5812d180 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fed5400c710 tx=0x7fed54007ae0).stop 2020-11-20T11:32:08.534 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.530+0000 7fed5f936700 1 -- 172.21.15.155:0/2671915744 shutdown_connections 2020-11-20T11:32:08.534 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.530+0000 7fed5f936700 1 --2- 172.21.15.155:0/2671915744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed580a9a80 0x7fed5812b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:08.535 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.530+0000 7fed5f936700 1 --2- 172.21.15.155:0/2671915744 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed58061380 0x7fed580acb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:08.535 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.530+0000 7fed5f936700 1 --2- 172.21.15.155:0/2671915744 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fed580a7150 0x7fed5812d180 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:08.535 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.530+0000 7fed5f936700 1 -- 172.21.15.155:0/2671915744 >> 172.21.15.155:0/2671915744 conn(0x7fed580ae280 msgr2=0x7fed58137b90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:08.535 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.530+0000 7fed5f936700 1 -- 172.21.15.155:0/2671915744 shutdown_connections 2020-11-20T11:32:08.535 INFO:teuthology.orchestra.run.smithi155.stderr:2020-11-20T11:32:08.530+0000 7fed5f936700 1 -- 172.21.15.155:0/2671915744 wait complete. 2020-11-20T11:32:08.544 INFO:tasks.ceph.osd.0:Restarting daemon 2020-11-20T11:32:08.545 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2020-11-20T11:32:08.547 INFO:tasks.ceph.osd.0:Started 2020-11-20T11:32:08.548 INFO:tasks.ceph.osd.1:Restarting daemon 2020-11-20T11:32:08.548 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2020-11-20T11:32:08.550 INFO:tasks.ceph.osd.1:Started 2020-11-20T11:32:08.550 INFO:tasks.ceph.osd.2:Restarting daemon 2020-11-20T11:32:08.551 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2020-11-20T11:32:08.553 INFO:tasks.ceph.osd.2:Started 2020-11-20T11:32:08.554 INFO:tasks.ceph.osd.3:Restarting daemon 2020-11-20T11:32:08.554 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2020-11-20T11:32:08.556 INFO:tasks.ceph.osd.3:Started 2020-11-20T11:32:08.556 INFO:tasks.ceph.osd.4:Restarting daemon 2020-11-20T11:32:08.557 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2020-11-20T11:32:08.559 INFO:tasks.ceph.osd.4:Started 2020-11-20T11:32:08.559 INFO:tasks.ceph.osd.5:Restarting daemon 2020-11-20T11:32:08.560 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2020-11-20T11:32:08.561 INFO:tasks.ceph.osd.5:Started 2020-11-20T11:32:08.562 INFO:tasks.ceph.osd.6:Restarting daemon 2020-11-20T11:32:08.562 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2020-11-20T11:32:08.564 INFO:tasks.ceph.osd.6:Started 2020-11-20T11:32:08.565 INFO:tasks.ceph.osd.7:Restarting daemon 2020-11-20T11:32:08.565 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2020-11-20T11:32:08.567 INFO:tasks.ceph.osd.7:Started 2020-11-20T11:32:08.567 INFO:tasks.ceph:Waiting for OSDs to come up 2020-11-20T11:32:09.147 INFO:tasks.ceph.osd.0.smithi023.stderr:2020-11-20T11:32:09.143+0000 7f4459398f40 -1 Falling back to public interface 2020-11-20T11:32:09.151 INFO:tasks.ceph.osd.4.smithi155.stderr:2020-11-20T11:32:09.148+0000 7ff562ac0f40 -1 Falling back to public interface 2020-11-20T11:32:09.157 INFO:tasks.ceph.osd.2.smithi023.stderr:2020-11-20T11:32:09.153+0000 7f06a605af40 -1 Falling back to public interface 2020-11-20T11:32:09.163 INFO:tasks.ceph.osd.7.smithi155.stderr:2020-11-20T11:32:09.160+0000 7fb813b2ef40 -1 Falling back to public interface 2020-11-20T11:32:09.168 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:32:09.164+0000 7f5967cddf40 -1 Falling back to public interface 2020-11-20T11:32:09.170 INFO:tasks.ceph.osd.5.smithi155.stderr:2020-11-20T11:32:09.167+0000 7f3fd1798f40 -1 Falling back to public interface 2020-11-20T11:32:09.175 INFO:tasks.ceph.osd.3.smithi023.stderr:2020-11-20T11:32:09.171+0000 7faef1950f40 -1 Falling back to public interface 2020-11-20T11:32:09.177 INFO:tasks.ceph.osd.6.smithi155.stderr:2020-11-20T11:32:09.174+0000 7fd3f887cf40 -1 Falling back to public interface 2020-11-20T11:32:09.375 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2020-11-20T11:32:09.499 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.487+0000 7f38be76e700 1 Processor -- start 2020-11-20T11:32:09.499 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.487+0000 7f38be76e700 1 -- start start 2020-11-20T11:32:09.500 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.488+0000 7f38be76e700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f38b813b9b0 0x7f38b8146760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:09.500 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.488+0000 7f38be76e700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38b8067250 con 0x7f38b80a9b80 2020-11-20T11:32:09.501 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.488+0000 7f38be76e700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38b80ab5b0 con 0x7f38b80a9560 2020-11-20T11:32:09.501 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.488+0000 7f38be76e700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38b813be40 con 0x7f38b813b9b0 2020-11-20T11:32:09.501 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.488+0000 7f38bcd0b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f38b813b9b0 0x7f38b8146760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:09.502 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.488+0000 7f38b7fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f38b80a9b80 0x7f38b8066b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:54444/0 (socket says 172.21.15.23:54444) 2020-11-20T11:32:09.502 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.488+0000 7f38b7fff700 1 -- 172.21.15.23:0/2017059716 learned_addr learned my addr 172.21.15.23:0/2017059716 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:09.503 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.489+0000 7f38bcd0b700 1 -- 172.21.15.23:0/2017059716 >> v1:172.21.15.155:6789/0 conn(0x7f38b80a9560 legacy=0x7f38b813b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:09.503 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.489+0000 7f38bcd0b700 1 -- 172.21.15.23:0/2017059716 >> v1:172.21.15.23:6789/0 conn(0x7f38b80a9b80 legacy=0x7f38b8066b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:09.503 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.489+0000 7f38bcd0b700 1 -- 172.21.15.23:0/2017059716 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b80653c0 con 0x7f38b813b9b0 2020-11-20T11:32:09.504 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.489+0000 7f38bcd0b700 1 --2- 172.21.15.23:0/2017059716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38b813b9b0 0x7f38b8146760 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f38b0009ce0 tx=0x7f38b0009220).ready entity=mon.0 client_cookie=623c197b266afb90 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:09.504 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.489+0000 7f38b6ffd700 1 -- 172.21.15.23:0/2017059716 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b000bbf0 con 0x7f38b813b9b0 2020-11-20T11:32:09.505 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.489+0000 7f38b6ffd700 1 -- 172.21.15.23:0/2017059716 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b000bd50 con 0x7f38b813b9b0 2020-11-20T11:32:09.505 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.489+0000 7f38b6ffd700 1 -- 172.21.15.23:0/2017059716 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b0011610 con 0x7f38b813b9b0 2020-11-20T11:32:09.505 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.489+0000 7f38be76e700 1 -- 172.21.15.23:0/2017059716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38b813b9b0 msgr2=0x7f38b8146760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:09.506 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.489+0000 7f38be76e700 1 --2- 172.21.15.23:0/2017059716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38b813b9b0 0x7f38b8146760 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f38b0009ce0 tx=0x7f38b0009220).stop 2020-11-20T11:32:09.506 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.489+0000 7f38be76e700 1 -- 172.21.15.23:0/2017059716 shutdown_connections 2020-11-20T11:32:09.507 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.489+0000 7f38be76e700 1 --2- 172.21.15.23:0/2017059716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38b813b9b0 0x7f38b8146760 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:09.507 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.489+0000 7f38be76e700 1 -- 172.21.15.23:0/2017059716 >> 172.21.15.23:0/2017059716 conn(0x7f38b80612c0 msgr2=0x7f38b80616d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:09.507 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.490+0000 7f38be76e700 1 -- 172.21.15.23:0/2017059716 shutdown_connections 2020-11-20T11:32:09.508 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.490+0000 7f38be76e700 1 -- 172.21.15.23:0/2017059716 wait complete. 2020-11-20T11:32:09.508 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.490+0000 7f38be76e700 1 Processor -- start 2020-11-20T11:32:09.508 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.491+0000 7f38be76e700 1 -- start start 2020-11-20T11:32:09.509 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.491+0000 7f38be76e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38b80a9560 0x7f38b80ac6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:09.509 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.491+0000 7f38be76e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38b80a9b80 0x7f38b80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:09.510 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.491+0000 7f38be76e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38b813b9b0 0x7f38b809ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:09.510 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.491+0000 7f38be76e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b80a0590 con 0x7f38b813b9b0 2020-11-20T11:32:09.511 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.491+0000 7f38be76e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b809f1c0 con 0x7f38b80a9b80 2020-11-20T11:32:09.511 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.491+0000 7f38be76e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38b813c160 con 0x7f38b80a9560 2020-11-20T11:32:09.511 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.491+0000 7f38bcd0b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38b813b9b0 0x7f38b809ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:09.512 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.491+0000 7f38bcd0b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38b813b9b0 0x7f38b809ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40202/0 (socket says 172.21.15.23:40202) 2020-11-20T11:32:09.512 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.491+0000 7f38bcd0b700 1 -- 172.21.15.23:0/1563915734 learned_addr learned my addr 172.21.15.23:0/1563915734 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:09.512 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.491+0000 7f38b77fe700 1 --2- 172.21.15.23:0/1563915734 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38b80a9b80 0x7f38b80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:09.513 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.492+0000 7f38b7fff700 1 --2- 172.21.15.23:0/1563915734 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38b80a9560 0x7f38b80ac6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:09.513 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.492+0000 7f38bcd0b700 1 -- 172.21.15.23:0/1563915734 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38b80a9560 msgr2=0x7f38b80ac6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:09.514 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.492+0000 7f38bcd0b700 1 --2- 172.21.15.23:0/1563915734 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38b80a9560 0x7f38b80ac6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:09.514 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.492+0000 7f38bcd0b700 1 -- 172.21.15.23:0/1563915734 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38b80a9b80 msgr2=0x7f38b80a46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:09.514 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.492+0000 7f38bcd0b700 1 --2- 172.21.15.23:0/1563915734 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38b80a9b80 0x7f38b80a46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:09.515 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.492+0000 7f38bcd0b700 1 -- 172.21.15.23:0/1563915734 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b809d660 con 0x7f38b813b9b0 2020-11-20T11:32:09.515 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.492+0000 7f38b77fe700 1 --2- 172.21.15.23:0/1563915734 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38b80a9b80 0x7f38b80a46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:09.516 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.492+0000 7f38bcd0b700 1 --2- 172.21.15.23:0/1563915734 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38b813b9b0 0x7f38b809ecd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f38b000b750 tx=0x7f38b0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:09.516 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.492+0000 7f3897fff700 1 -- 172.21.15.23:0/1563915734 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b000bec0 con 0x7f38b813b9b0 2020-11-20T11:32:09.516 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.492+0000 7f3897fff700 1 -- 172.21.15.23:0/1563915734 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b0009e20 con 0x7f38b813b9b0 2020-11-20T11:32:09.517 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.492+0000 7f38be76e700 1 -- 172.21.15.23:0/1563915734 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38b809d900 con 0x7f38b813b9b0 2020-11-20T11:32:09.517 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.492+0000 7f3897fff700 1 -- 172.21.15.23:0/1563915734 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b0010d50 con 0x7f38b813b9b0 2020-11-20T11:32:09.518 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.492+0000 7f38be76e700 1 -- 172.21.15.23:0/1563915734 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38b80a0980 con 0x7f38b813b9b0 2020-11-20T11:32:09.518 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.492+0000 7f3897fff700 1 -- 172.21.15.23:0/1563915734 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f38b0017070 con 0x7f38b813b9b0 2020-11-20T11:32:09.518 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.492+0000 7f3897fff700 1 -- 172.21.15.23:0/1563915734 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1674+0+0 (secure 0 0 0) 0x7f38b0026b60 con 0x7f38b813b9b0 2020-11-20T11:32:09.519 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.494+0000 7f38be76e700 1 -- 172.21.15.23:0/1563915734 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38b8062760 con 0x7f38b813b9b0 2020-11-20T11:32:09.520 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.496+0000 7f3897fff700 1 -- 172.21.15.23:0/1563915734 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71786 (secure 0 0 0) 0x7f38b0015020 con 0x7f38b813b9b0 2020-11-20T11:32:09.520 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.513+0000 7f3897fff700 1 -- 172.21.15.23:0/1563915734 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 2) v1 ==== 33694+0+0 (secure 0 0 0) 0x7f38b000f070 con 0x7f38b813b9b0 2020-11-20T11:32:09.587 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.583+0000 7f38be76e700 1 -- 172.21.15.23:0/1563915734 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38b8044fa0 con 0x7f38b813b9b0 2020-11-20T11:32:09.589 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.584+0000 7f3897fff700 1 -- 172.21.15.23:0/1563915734 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7001 (secure 0 0 0) 0x7f38b0030430 con 0x7f38b813b9b0 2020-11-20T11:32:09.589 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.585+0000 7f38be76e700 1 -- 172.21.15.23:0/1563915734 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38b813b9b0 msgr2=0x7f38b809ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:09.589 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.585+0000 7f38be76e700 1 --2- 172.21.15.23:0/1563915734 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38b813b9b0 0x7f38b809ecd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f38b000b750 tx=0x7f38b0016000).stop 2020-11-20T11:32:09.589 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.585+0000 7f38be76e700 1 -- 172.21.15.23:0/1563915734 shutdown_connections 2020-11-20T11:32:09.590 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.585+0000 7f38be76e700 1 --2- 172.21.15.23:0/1563915734 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38b813b9b0 0x7f38b809ecd0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:09.590 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.585+0000 7f38be76e700 1 --2- 172.21.15.23:0/1563915734 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38b80a9560 0x7f38b80ac6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:09.590 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.585+0000 7f38be76e700 1 --2- 172.21.15.23:0/1563915734 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38b80a9b80 0x7f38b80a46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:09.590 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.585+0000 7f38be76e700 1 -- 172.21.15.23:0/1563915734 >> 172.21.15.23:0/1563915734 conn(0x7f38b80612c0 msgr2=0x7f38b813b070 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:09.591 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.585+0000 7f38be76e700 1 -- 172.21.15.23:0/1563915734 shutdown_connections 2020-11-20T11:32:09.591 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:09.585+0000 7f38be76e700 1 -- 172.21.15.23:0/1563915734 wait complete. 2020-11-20T11:32:09.591 INFO:teuthology.misc.health.smithi023.stdout: 2020-11-20T11:32:09.591 INFO:teuthology.misc.health.smithi023.stdout:{"epoch":10,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","created":"2020-11-20T11:32:06.136679+0000","modified":"2020-11-20T11:32:08.527269+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"e9d219f5-e40e-4b2b-9468-deb9c4074a44","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2328adde-2108-4fe0-8138-f0d096a496a8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"13367708-59a2-4363-8a3a-ab0cb97145ef","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1f147dd7-9289-47b0-ad1c-4088618d6faa","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d008b039-4bc5-4135-9678-1511e58efc29","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"56715e6f-cf6e-44a6-9e39-93ab1224095f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"db1e61ca-8e14-4a1f-8607-6e01b148d61c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d44419be-ef89-462c-8cf1-3c6db2a6944b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T11:32:09.598 DEBUG:teuthology.misc:0 of 8 OSDs are up 2020-11-20T11:32:10.474 INFO:tasks.ceph.osd.4.smithi155.stderr:2020-11-20T11:32:10.471+0000 7ff562ac0f40 -1 osd.4 0 log_to_monitors {default=true} 2020-11-20T11:32:10.497 INFO:tasks.ceph.osd.0.smithi023.stderr:2020-11-20T11:32:10.494+0000 7f4459398f40 -1 osd.0 0 log_to_monitors {default=true} 2020-11-20T11:32:10.501 INFO:tasks.ceph.osd.7.smithi155.stderr:2020-11-20T11:32:10.498+0000 7fb813b2ef40 -1 osd.7 0 log_to_monitors {default=true} 2020-11-20T11:32:10.510 INFO:tasks.ceph.osd.5.smithi155.stderr:2020-11-20T11:32:10.507+0000 7f3fd1798f40 -1 osd.5 0 log_to_monitors {default=true} 2020-11-20T11:32:10.512 INFO:tasks.ceph.osd.2.smithi023.stderr:2020-11-20T11:32:10.509+0000 7f06a605af40 -1 osd.2 0 log_to_monitors {default=true} 2020-11-20T11:32:10.518 INFO:tasks.ceph.osd.6.smithi155.stderr:2020-11-20T11:32:10.515+0000 7fd3f887cf40 -1 osd.6 0 log_to_monitors {default=true} 2020-11-20T11:32:10.520 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:32:10.516+0000 7f5967cddf40 -1 osd.1 0 log_to_monitors {default=true} 2020-11-20T11:32:10.529 INFO:tasks.ceph.osd.3.smithi023.stderr:2020-11-20T11:32:10.526+0000 7faef1950f40 -1 osd.3 0 log_to_monitors {default=true} 2020-11-20T11:32:11.535 INFO:tasks.ceph.osd.0.smithi023.stderr:2020-11-20T11:32:11.531+0000 7f444fb21700 -1 osd.0 0 waiting for initial osdmap 2020-11-20T11:32:11.535 INFO:tasks.ceph.osd.5.smithi155.stderr:2020-11-20T11:32:11.532+0000 7f3fc7f21700 -1 osd.5 0 waiting for initial osdmap 2020-11-20T11:32:11.536 INFO:tasks.ceph.osd.7.smithi155.stderr:2020-11-20T11:32:11.532+0000 7fb80a2b7700 -1 osd.7 0 waiting for initial osdmap 2020-11-20T11:32:11.537 INFO:tasks.ceph.osd.4.smithi155.stderr:2020-11-20T11:32:11.534+0000 7ff559249700 -1 osd.4 0 waiting for initial osdmap 2020-11-20T11:32:12.538 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:32:12.534+0000 7f595e466700 -1 osd.1 0 waiting for initial osdmap 2020-11-20T11:32:12.541 INFO:tasks.ceph.osd.2.smithi023.stderr:2020-11-20T11:32:12.537+0000 7f069c7e3700 -1 osd.2 0 waiting for initial osdmap 2020-11-20T11:32:12.541 INFO:tasks.ceph.osd.6.smithi155.stderr:2020-11-20T11:32:12.538+0000 7fd3ef005700 -1 osd.6 0 waiting for initial osdmap 2020-11-20T11:32:12.542 INFO:tasks.ceph.osd.3.smithi023.stderr:2020-11-20T11:32:12.538+0000 7faee80d9700 -1 osd.3 0 waiting for initial osdmap 2020-11-20T11:32:16.406 INFO:teuthology.orchestra.run.smithi023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2020-11-20T11:32:16.522 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.518+0000 7fbb21d34700 1 Processor -- start 2020-11-20T11:32:16.522 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.518+0000 7fbb21d34700 1 -- start start 2020-11-20T11:32:16.525 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.519+0000 7fbb21d34700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fbb1c0aad90 0x7fbb1c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:16.525 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.519+0000 7fbb21d34700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fbb1c0ab170 0x7fbb1c13ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:16.525 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.519+0000 7fbb21d34700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb1c13b050 con 0x7fbb1c0a87d0 2020-11-20T11:32:16.525 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.519+0000 7fbb21d34700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb1c0abeb0 con 0x7fbb1c0ab170 2020-11-20T11:32:16.525 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.519+0000 7fbb21d34700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb1c0ac150 con 0x7fbb1c0aad90 2020-11-20T11:32:16.526 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.519+0000 7fbb1affd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fbb1c0ab170 0x7fbb1c13ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:16.526 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.519+0000 7fbb1affd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fbb1c0ab170 0x7fbb1c13ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40368/0 (socket says 172.21.15.23:40368) 2020-11-20T11:32:16.526 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.519+0000 7fbb1affd700 1 -- 172.21.15.23:0/1120285479 learned_addr learned my addr 172.21.15.23:0/1120285479 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:16.526 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.519+0000 7fbb1b7fe700 1 --2- 172.21.15.23:0/1120285479 >> v2:172.21.15.155:3300/0 conn(0x7fbb1c0aad90 0x7fbb1c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:16.527 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.519+0000 7fbb1affd700 1 -- 172.21.15.23:0/1120285479 >> v2:172.21.15.155:3300/0 conn(0x7fbb1c0aad90 msgr2=0x7fbb1c0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:16.527 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.519+0000 7fbb1affd700 1 --2- 172.21.15.23:0/1120285479 >> v2:172.21.15.155:3300/0 conn(0x7fbb1c0aad90 0x7fbb1c0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:16.527 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.519+0000 7fbb1affd700 1 -- 172.21.15.23:0/1120285479 >> v1:172.21.15.155:6789/0 conn(0x7fbb1c0a87d0 legacy=0x7fbb1c0a45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:16.527 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.519+0000 7fbb1affd700 1 -- 172.21.15.23:0/1120285479 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb1c0ac3f0 con 0x7fbb1c0ab170 2020-11-20T11:32:16.527 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.520+0000 7fbb1affd700 1 --2- 172.21.15.23:0/1120285479 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb1c0ab170 0x7fbb1c13ab60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fbb1c0b12f0 tx=0x7fbb10006ef0).ready entity=mon.0 client_cookie=ec865e3ca113f7bd server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:16.528 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.520+0000 7fbb1a7fc700 1 -- 172.21.15.23:0/1120285479 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1000b860 con 0x7fbb1c0ab170 2020-11-20T11:32:16.528 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.520+0000 7fbb1a7fc700 1 -- 172.21.15.23:0/1120285479 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb10000e90 con 0x7fbb1c0ab170 2020-11-20T11:32:16.528 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.520+0000 7fbb1a7fc700 1 -- 172.21.15.23:0/1120285479 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1000bbd0 con 0x7fbb1c0ab170 2020-11-20T11:32:16.528 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.520+0000 7fbb21d34700 1 -- 172.21.15.23:0/1120285479 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb1c0ab170 msgr2=0x7fbb1c13ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:16.528 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.520+0000 7fbb21d34700 1 --2- 172.21.15.23:0/1120285479 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb1c0ab170 0x7fbb1c13ab60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fbb1c0b12f0 tx=0x7fbb10006ef0).stop 2020-11-20T11:32:16.529 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.520+0000 7fbb21d34700 1 -- 172.21.15.23:0/1120285479 shutdown_connections 2020-11-20T11:32:16.529 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.520+0000 7fbb21d34700 1 --2- 172.21.15.23:0/1120285479 >> v2:172.21.15.155:3300/0 conn(0x7fbb1c0aad90 0x7fbb1c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:16.529 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.520+0000 7fbb21d34700 1 --2- 172.21.15.23:0/1120285479 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb1c0ab170 0x7fbb1c13ab60 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:16.529 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.520+0000 7fbb21d34700 1 -- 172.21.15.23:0/1120285479 >> 172.21.15.23:0/1120285479 conn(0x7fbb1c0601b0 msgr2=0x7fbb1c0605c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:16.529 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.520+0000 7fbb21d34700 1 -- 172.21.15.23:0/1120285479 shutdown_connections 2020-11-20T11:32:16.530 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.520+0000 7fbb21d34700 1 -- 172.21.15.23:0/1120285479 wait complete. 2020-11-20T11:32:16.532 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.521+0000 7fbb21d34700 1 Processor -- start 2020-11-20T11:32:16.532 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.521+0000 7fbb21d34700 1 -- start start 2020-11-20T11:32:16.533 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.522+0000 7fbb21d34700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb1c0a87d0 0x7fbb1c13caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:16.533 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.522+0000 7fbb21d34700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb1c0aad90 0x7fbb1c14d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:16.533 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.522+0000 7fbb21d34700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb1c0ab170 0x7fbb1c13d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:16.533 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.522+0000 7fbb21d34700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1c134580 con 0x7fbb1c0a87d0 2020-11-20T11:32:16.534 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.522+0000 7fbb21d34700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1c13c3b0 con 0x7fbb1c0ab170 2020-11-20T11:32:16.534 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.522+0000 7fbb21d34700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1c13c650 con 0x7fbb1c0aad90 2020-11-20T11:32:16.534 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.522+0000 7fbb1b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb1c0a87d0 0x7fbb1c13caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:16.534 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.522+0000 7fbb1b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb1c0a87d0 0x7fbb1c13caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40372/0 (socket says 172.21.15.23:40372) 2020-11-20T11:32:16.534 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.522+0000 7fbb1b7fe700 1 -- 172.21.15.23:0/646515831 learned_addr learned my addr 172.21.15.23:0/646515831 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:16.535 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.522+0000 7fbb1bfff700 1 --2- 172.21.15.23:0/646515831 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb1c0ab170 0x7fbb1c13d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:16.535 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.522+0000 7fbb1affd700 1 --2- 172.21.15.23:0/646515831 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb1c0aad90 0x7fbb1c14d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:16.535 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.522+0000 7fbb1b7fe700 1 -- 172.21.15.23:0/646515831 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb1c0aad90 msgr2=0x7fbb1c14d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:16.535 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.522+0000 7fbb1b7fe700 1 --2- 172.21.15.23:0/646515831 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb1c0aad90 0x7fbb1c14d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:16.535 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.522+0000 7fbb1b7fe700 1 -- 172.21.15.23:0/646515831 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb1c0ab170 msgr2=0x7fbb1c13d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:16.536 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.522+0000 7fbb1b7fe700 1 --2- 172.21.15.23:0/646515831 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb1c0ab170 0x7fbb1c13d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:16.536 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.522+0000 7fbb1b7fe700 1 -- 172.21.15.23:0/646515831 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb1c0a79e0 con 0x7fbb1c0a87d0 2020-11-20T11:32:16.536 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.523+0000 7fbb1b7fe700 1 --2- 172.21.15.23:0/646515831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb1c0a87d0 0x7fbb1c13caf0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fbb0c00eba0 tx=0x7fbb0c006e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:16.536 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.523+0000 7fbafb7fe700 1 -- 172.21.15.23:0/646515831 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c00c9d0 con 0x7fbb1c0a87d0 2020-11-20T11:32:16.536 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.523+0000 7fbafb7fe700 1 -- 172.21.15.23:0/646515831 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb0c00cb30 con 0x7fbb1c0a87d0 2020-11-20T11:32:16.537 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.523+0000 7fbb21d34700 1 -- 172.21.15.23:0/646515831 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb1c13f250 con 0x7fbb1c0a87d0 2020-11-20T11:32:16.537 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.523+0000 7fbafb7fe700 1 -- 172.21.15.23:0/646515831 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c0123f0 con 0x7fbb1c0a87d0 2020-11-20T11:32:16.537 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.523+0000 7fbb21d34700 1 -- 172.21.15.23:0/646515831 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb1c13f810 con 0x7fbb1c0a87d0 2020-11-20T11:32:16.537 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.524+0000 7fbb21d34700 1 -- 172.21.15.23:0/646515831 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb1c13fa30 con 0x7fbb1c0a87d0 2020-11-20T11:32:16.537 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.524+0000 7fbafb7fe700 1 -- 172.21.15.23:0/646515831 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbb0c0127e0 con 0x7fbb1c0a87d0 2020-11-20T11:32:16.538 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.524+0000 7fbafb7fe700 1 --2- 172.21.15.23:0/646515831 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbaf4056d60 0x7fbaf4059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:16.538 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.524+0000 7fbafb7fe700 1 -- 172.21.15.23:0/646515831 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4900+0+0 (secure 0 0 0) 0x7fbb0c06fed0 con 0x7fbb1c0a87d0 2020-11-20T11:32:16.538 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.525+0000 7fbb1affd700 1 --2- 172.21.15.23:0/646515831 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbaf4056d60 0x7fbaf4059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:16.538 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.525+0000 7fbb1affd700 1 --2- 172.21.15.23:0/646515831 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbaf4056d60 0x7fbaf4059220 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fbb10006f30 tx=0x7fbb10016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:16.538 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.528+0000 7fbafb7fe700 1 -- 172.21.15.23:0/646515831 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fbb0c0457b0 con 0x7fbb1c0a87d0 2020-11-20T11:32:16.684 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.680+0000 7fbb21d34700 1 -- 172.21.15.23:0/646515831 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb1c044fa0 con 0x7fbb1c0a87d0 2020-11-20T11:32:16.685 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.681+0000 7fbafb7fe700 1 -- 172.21.15.23:0/646515831 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+12152 (secure 0 0 0) 0x7fbb0c048dd0 con 0x7fbb1c0a87d0 2020-11-20T11:32:16.685 INFO:teuthology.misc.health.smithi023.stdout: 2020-11-20T11:32:16.685 INFO:teuthology.misc.health.smithi023.stdout:{"epoch":15,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","created":"2020-11-20T11:32:06.136679+0000","modified":"2020-11-20T11:32:14.552060+0000","last_up_change":"2020-11-20T11:32:13.536194+0000","last_in_change":"2020-11-20T11:32:13.536194+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T11:32:09.559868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"e9d219f5-e40e-4b2b-9468-deb9c4074a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":23083},{"type":"v1","addr":"172.21.15.23:6801","nonce":23083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":23083},{"type":"v1","addr":"172.21.15.23:6803","nonce":23083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":23083},{"type":"v1","addr":"172.21.15.23:6807","nonce":23083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":23083},{"type":"v1","addr":"172.21.15.23:6805","nonce":23083}]},"public_addr":"172.21.15.23:6801/23083","cluster_addr":"172.21.15.23:6803/23083","heartbeat_back_addr":"172.21.15.23:6807/23083","heartbeat_front_addr":"172.21.15.23:6805/23083","state":["exists","up"]},{"osd":1,"uuid":"2328adde-2108-4fe0-8138-f0d096a496a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":23099},{"type":"v1","addr":"172.21.15.23:6817","nonce":23099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":23099},{"type":"v1","addr":"172.21.15.23:6819","nonce":23099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":23099},{"type":"v1","addr":"172.21.15.23:6824","nonce":23099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":23099},{"type":"v1","addr":"172.21.15.23:6821","nonce":23099}]},"public_addr":"172.21.15.23:6817/23099","cluster_addr":"172.21.15.23:6819/23099","heartbeat_back_addr":"172.21.15.23:6824/23099","heartbeat_front_addr":"172.21.15.23:6821/23099","state":["exists","up"]},{"osd":2,"uuid":"13367708-59a2-4363-8a3a-ab0cb97145ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":23084},{"type":"v1","addr":"172.21.15.23:6809","nonce":23084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":23084},{"type":"v1","addr":"172.21.15.23:6811","nonce":23084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":23084},{"type":"v1","addr":"172.21.15.23:6815","nonce":23084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":23084},{"type":"v1","addr":"172.21.15.23:6813","nonce":23084}]},"public_addr":"172.21.15.23:6809/23084","cluster_addr":"172.21.15.23:6811/23084","heartbeat_back_addr":"172.21.15.23:6815/23084","heartbeat_front_addr":"172.21.15.23:6813/23084","state":["exists","up"]},{"osd":3,"uuid":"1f147dd7-9289-47b0-ad1c-4088618d6faa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6823","nonce":23114},{"type":"v1","addr":"172.21.15.23:6825","nonce":23114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":23114},{"type":"v1","addr":"172.21.15.23:6827","nonce":23114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":23114},{"type":"v1","addr":"172.21.15.23:6831","nonce":23114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":23114},{"type":"v1","addr":"172.21.15.23:6829","nonce":23114}]},"public_addr":"172.21.15.23:6825/23114","cluster_addr":"172.21.15.23:6827/23114","heartbeat_back_addr":"172.21.15.23:6831/23114","heartbeat_front_addr":"172.21.15.23:6829/23114","state":["exists","up"]},{"osd":4,"uuid":"d008b039-4bc5-4135-9678-1511e58efc29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":23280},{"type":"v1","addr":"172.21.15.155:6801","nonce":23280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":23280},{"type":"v1","addr":"172.21.15.155:6803","nonce":23280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":23280},{"type":"v1","addr":"172.21.15.155:6807","nonce":23280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":23280},{"type":"v1","addr":"172.21.15.155:6805","nonce":23280}]},"public_addr":"172.21.15.155:6801/23280","cluster_addr":"172.21.15.155:6803/23280","heartbeat_back_addr":"172.21.15.155:6807/23280","heartbeat_front_addr":"172.21.15.155:6805/23280","state":["exists","up"]},{"osd":5,"uuid":"56715e6f-cf6e-44a6-9e39-93ab1224095f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":23298},{"type":"v1","addr":"172.21.15.155:6817","nonce":23298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":23298},{"type":"v1","addr":"172.21.15.155:6819","nonce":23298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":23298},{"type":"v1","addr":"172.21.15.155:6823","nonce":23298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":23298},{"type":"v1","addr":"172.21.15.155:6821","nonce":23298}]},"public_addr":"172.21.15.155:6817/23298","cluster_addr":"172.21.15.155:6819/23298","heartbeat_back_addr":"172.21.15.155:6823/23298","heartbeat_front_addr":"172.21.15.155:6821/23298","state":["exists","up"]},{"osd":6,"uuid":"db1e61ca-8e14-4a1f-8607-6e01b148d61c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":23307},{"type":"v1","addr":"172.21.15.155:6825","nonce":23307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":23307},{"type":"v1","addr":"172.21.15.155:6827","nonce":23307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":23307},{"type":"v1","addr":"172.21.15.155:6831","nonce":23307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":23307},{"type":"v1","addr":"172.21.15.155:6829","nonce":23307}]},"public_addr":"172.21.15.155:6825/23307","cluster_addr":"172.21.15.155:6827/23307","heartbeat_back_addr":"172.21.15.155:6831/23307","heartbeat_front_addr":"172.21.15.155:6829/23307","state":["exists","up"]},{"osd":7,"uuid":"d44419be-ef89-462c-8cf1-3c6db2a6944b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":23281},{"type":"v1","addr":"172.21.15.155:6809","nonce":23281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":23281},{"type":"v1","addr":"172.21.15.155:6811","nonce":23281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":23281},{"type":"v1","addr":"172.21.15.155:6815","nonce":23281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":23281},{"type":"v1","addr":"172.21.15.155:6813","nonce":23281}]},"public_addr":"172.21.15.155:6809/23281","cluster_addr":"172.21.15.155:6811/23281","heartbeat_back_addr":"172.21.15.155:6815/23281","heartbeat_front_addr":"172.21.15.155:6813/23281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.486702+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.509396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.533414+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.509489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.467600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.513068+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.536636+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.531772+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T11:32:16.686 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.682+0000 7fbb21d34700 1 -- 172.21.15.23:0/646515831 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbaf4056d60 msgr2=0x7fbaf4059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:16.687 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.682+0000 7fbb21d34700 1 --2- 172.21.15.23:0/646515831 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbaf4056d60 0x7fbaf4059220 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fbb10006f30 tx=0x7fbb10016000).stop 2020-11-20T11:32:16.687 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.682+0000 7fbb21d34700 1 -- 172.21.15.23:0/646515831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb1c0a87d0 msgr2=0x7fbb1c13caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:16.687 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.682+0000 7fbb21d34700 1 --2- 172.21.15.23:0/646515831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb1c0a87d0 0x7fbb1c13caf0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fbb0c00eba0 tx=0x7fbb0c006e30).stop 2020-11-20T11:32:16.687 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.682+0000 7fbb21d34700 1 -- 172.21.15.23:0/646515831 shutdown_connections 2020-11-20T11:32:16.688 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.682+0000 7fbb21d34700 1 --2- 172.21.15.23:0/646515831 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb1c0ab170 0x7fbb1c13d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:16.688 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.682+0000 7fbb21d34700 1 --2- 172.21.15.23:0/646515831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb1c0a87d0 0x7fbb1c13caf0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:16.688 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.682+0000 7fbb21d34700 1 --2- 172.21.15.23:0/646515831 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbaf4056d60 0x7fbaf4059220 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:16.688 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.682+0000 7fbb21d34700 1 --2- 172.21.15.23:0/646515831 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb1c0aad90 0x7fbb1c14d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:16.688 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.682+0000 7fbb21d34700 1 -- 172.21.15.23:0/646515831 >> 172.21.15.23:0/646515831 conn(0x7fbb1c0601b0 msgr2=0x7fbb1c0ae130 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:16.689 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.682+0000 7fbb21d34700 1 -- 172.21.15.23:0/646515831 shutdown_connections 2020-11-20T11:32:16.689 INFO:teuthology.misc.health.smithi023.stderr:2020-11-20T11:32:16.682+0000 7fbb21d34700 1 -- 172.21.15.23:0/646515831 wait complete. 2020-11-20T11:32:16.696 DEBUG:teuthology.misc:8 of 8 OSDs are up 2020-11-20T11:32:16.697 INFO:tasks.ceph:Creating RBD pool 2020-11-20T11:32:16.697 INFO:teuthology.orchestra.run.smithi023:> sudo ceph --cluster ceph osd pool create rbd 8 2020-11-20T11:32:16.810 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.806+0000 7fdb22f92700 1 Processor -- start 2020-11-20T11:32:16.810 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.806+0000 7fdb22f92700 1 -- start start 2020-11-20T11:32:16.813 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.807+0000 7fdb22f92700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdb1c0aae90 0x7fdb1c1381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:16.813 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.807+0000 7fdb22f92700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fdb1c13aeb0 0x7fdb1c0a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:16.813 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.807+0000 7fdb22f92700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb1c13e600 con 0x7fdb1c0aaab0 2020-11-20T11:32:16.814 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.807+0000 7fdb22f92700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb1c13e8a0 con 0x7fdb1c13aeb0 2020-11-20T11:32:16.814 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.807+0000 7fdb22f92700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1c0ab6d0 con 0x7fdb1c0aae90 2020-11-20T11:32:16.814 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.807+0000 7fdb2152f700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fdb1c13aeb0 0x7fdb1c0a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:16.814 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.807+0000 7fdb2152f700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fdb1c13aeb0 0x7fdb1c0a8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40382/0 (socket says 172.21.15.23:40382) 2020-11-20T11:32:16.814 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.807+0000 7fdb2152f700 1 -- 172.21.15.23:0/3725635677 learned_addr learned my addr 172.21.15.23:0/3725635677 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:16.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.807+0000 7fdb2152f700 1 -- 172.21.15.23:0/3725635677 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdb1c0aae90 msgr2=0x7fdb1c1381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:16.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.807+0000 7fdb1bfff700 1 --2- 172.21.15.23:0/3725635677 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdb1c0aae90 0x7fdb1c1381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:16.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.807+0000 7fdb2152f700 1 --2- 172.21.15.23:0/3725635677 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdb1c0aae90 0x7fdb1c1381a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:16.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.807+0000 7fdb2152f700 1 -- 172.21.15.23:0/3725635677 >> v1:172.21.15.155:6789/0 conn(0x7fdb1c0aaab0 legacy=0x7fdb1c13ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:16.816 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.807+0000 7fdb2152f700 1 -- 172.21.15.23:0/3725635677 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb1c138690 con 0x7fdb1c13aeb0 2020-11-20T11:32:16.816 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.807+0000 7fdb2152f700 1 --2- 172.21.15.23:0/3725635677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb1c13aeb0 0x7fdb1c0a8c60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fdb14009d90 tx=0x7fdb140092d0).ready entity=mon.0 client_cookie=14097270a04afc39 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:16.816 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.808+0000 7fdb1b7fe700 1 -- 172.21.15.23:0/3725635677 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1400bbb0 con 0x7fdb1c13aeb0 2020-11-20T11:32:16.816 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.808+0000 7fdb1b7fe700 1 -- 172.21.15.23:0/3725635677 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb1400bd10 con 0x7fdb1c13aeb0 2020-11-20T11:32:16.816 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.808+0000 7fdb1b7fe700 1 -- 172.21.15.23:0/3725635677 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb14011610 con 0x7fdb1c13aeb0 2020-11-20T11:32:16.817 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.808+0000 7fdb22f92700 1 -- 172.21.15.23:0/3725635677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb1c13aeb0 msgr2=0x7fdb1c0a8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:16.817 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.808+0000 7fdb22f92700 1 --2- 172.21.15.23:0/3725635677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb1c13aeb0 0x7fdb1c0a8c60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fdb14009d90 tx=0x7fdb140092d0).stop 2020-11-20T11:32:16.817 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.808+0000 7fdb22f92700 1 -- 172.21.15.23:0/3725635677 shutdown_connections 2020-11-20T11:32:16.817 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.808+0000 7fdb22f92700 1 --2- 172.21.15.23:0/3725635677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb1c13aeb0 0x7fdb1c0a8c60 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:16.817 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.808+0000 7fdb22f92700 1 --2- 172.21.15.23:0/3725635677 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdb1c0aae90 0x7fdb1c1381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:16.818 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.808+0000 7fdb22f92700 1 -- 172.21.15.23:0/3725635677 >> 172.21.15.23:0/3725635677 conn(0x7fdb1c0678e0 msgr2=0x7fdb1c067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:16.818 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.808+0000 7fdb22f92700 1 -- 172.21.15.23:0/3725635677 shutdown_connections 2020-11-20T11:32:16.818 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.808+0000 7fdb22f92700 1 -- 172.21.15.23:0/3725635677 wait complete. 2020-11-20T11:32:16.818 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.809+0000 7fdb22f92700 1 Processor -- start 2020-11-20T11:32:16.818 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.809+0000 7fdb22f92700 1 -- start start 2020-11-20T11:32:16.821 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.810+0000 7fdb22f92700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdb1c0aaab0 0x7fdb1c0b0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:16.821 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.810+0000 7fdb22f92700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdb1c0aae90 0x7fdb1c0a5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:16.822 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.810+0000 7fdb22f92700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb1c14de10 0x7fdb1c14a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:16.822 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.810+0000 7fdb22f92700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1c136320 con 0x7fdb1c14de10 2020-11-20T11:32:16.822 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.810+0000 7fdb22f92700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1c0a6570 con 0x7fdb1c0aae90 2020-11-20T11:32:16.822 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.810+0000 7fdb22f92700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1c14d6d0 con 0x7fdb1c0aaab0 2020-11-20T11:32:16.822 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.810+0000 7fdb2152f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb1c14de10 0x7fdb1c14a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:16.823 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.810+0000 7fdb2152f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb1c14de10 0x7fdb1c14a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40390/0 (socket says 172.21.15.23:40390) 2020-11-20T11:32:16.823 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.810+0000 7fdb2152f700 1 -- 172.21.15.23:0/72223919 learned_addr learned my addr 172.21.15.23:0/72223919 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:16.823 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.810+0000 7fdb1bfff700 1 --2- 172.21.15.23:0/72223919 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdb1c0aae90 0x7fdb1c0a5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:16.823 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.810+0000 7fdb20d2e700 1 --2- 172.21.15.23:0/72223919 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdb1c0aaab0 0x7fdb1c0b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:16.824 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.810+0000 7fdb2152f700 1 -- 172.21.15.23:0/72223919 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdb1c0aaab0 msgr2=0x7fdb1c0b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:16.824 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.810+0000 7fdb2152f700 1 --2- 172.21.15.23:0/72223919 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdb1c0aaab0 0x7fdb1c0b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:16.824 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.810+0000 7fdb2152f700 1 -- 172.21.15.23:0/72223919 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdb1c0aae90 msgr2=0x7fdb1c0a5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:16.824 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.810+0000 7fdb2152f700 1 --2- 172.21.15.23:0/72223919 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdb1c0aae90 0x7fdb1c0a5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:16.824 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.810+0000 7fdb2152f700 1 -- 172.21.15.23:0/72223919 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb1c14d970 con 0x7fdb1c14de10 2020-11-20T11:32:16.825 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.810+0000 7fdb2152f700 1 --2- 172.21.15.23:0/72223919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb1c14de10 0x7fdb1c14a6d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fdb14009130 tx=0x7fdb14016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:16.825 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.810+0000 7fdb18ff9700 1 -- 172.21.15.23:0/72223919 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb14011e70 con 0x7fdb1c14de10 2020-11-20T11:32:16.825 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.810+0000 7fdb18ff9700 1 -- 172.21.15.23:0/72223919 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb1400be80 con 0x7fdb1c14de10 2020-11-20T11:32:16.825 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.810+0000 7fdb22f92700 1 -- 172.21.15.23:0/72223919 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb1c0a6270 con 0x7fdb1c14de10 2020-11-20T11:32:16.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.810+0000 7fdb18ff9700 1 -- 172.21.15.23:0/72223919 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb14010c70 con 0x7fdb1c14de10 2020-11-20T11:32:16.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.811+0000 7fdb22f92700 1 -- 172.21.15.23:0/72223919 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb1c135610 con 0x7fdb1c14de10 2020-11-20T11:32:16.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.811+0000 7fdb22f92700 1 -- 172.21.15.23:0/72223919 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb1c05c7d0 con 0x7fdb1c14de10 2020-11-20T11:32:16.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.812+0000 7fdb18ff9700 1 -- 172.21.15.23:0/72223919 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fdb14017070 con 0x7fdb1c14de10 2020-11-20T11:32:16.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.812+0000 7fdb18ff9700 1 --2- 172.21.15.23:0/72223919 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fdaf4056d60 0x7fdaf4059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:16.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.812+0000 7fdb18ff9700 1 -- 172.21.15.23:0/72223919 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4900+0+0 (secure 0 0 0) 0x7fdb14071b90 con 0x7fdb1c14de10 2020-11-20T11:32:16.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.812+0000 7fdb20d2e700 1 --2- 172.21.15.23:0/72223919 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fdaf4056d60 0x7fdaf4059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:16.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.812+0000 7fdb20d2e700 1 --2- 172.21.15.23:0/72223919 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fdaf4056d60 0x7fdaf4059220 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fdb10003750 tx=0x7fdb10007bb0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:16.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.816+0000 7fdb18ff9700 1 -- 172.21.15.23:0/72223919 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fdb14047470 con 0x7fdb1c14de10 2020-11-20T11:32:16.968 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:16.964+0000 7fdb22f92700 1 -- 172.21.15.23:0/72223919 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7fdb1c14db00 con 0x7fdb1c14de10 2020-11-20T11:32:17.914 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:17.910+0000 7fdb18ff9700 1 -- 172.21.15.23:0/72223919 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v16) v1 ==== 109+0+0 (secure 0 0 0) 0x7fdb14015070 con 0x7fdb1c14de10 2020-11-20T11:32:17.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:17.913+0000 7fdb22f92700 1 -- 172.21.15.23:0/72223919 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fdaf4056d60 msgr2=0x7fdaf4059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:17.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:17.913+0000 7fdb22f92700 1 --2- 172.21.15.23:0/72223919 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fdaf4056d60 0x7fdaf4059220 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fdb10003750 tx=0x7fdb10007bb0).stop 2020-11-20T11:32:17.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:17.914+0000 7fdb22f92700 1 -- 172.21.15.23:0/72223919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb1c14de10 msgr2=0x7fdb1c14a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:17.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:17.914+0000 7fdb22f92700 1 --2- 172.21.15.23:0/72223919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb1c14de10 0x7fdb1c14a6d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fdb14009130 tx=0x7fdb14016000).stop 2020-11-20T11:32:17.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:17.914+0000 7fdb22f92700 1 -- 172.21.15.23:0/72223919 shutdown_connections 2020-11-20T11:32:17.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:17.914+0000 7fdb22f92700 1 --2- 172.21.15.23:0/72223919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb1c14de10 0x7fdb1c14a6d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:17.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:17.914+0000 7fdb22f92700 1 --2- 172.21.15.23:0/72223919 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fdaf4056d60 0x7fdaf4059220 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:17.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:17.914+0000 7fdb22f92700 1 --2- 172.21.15.23:0/72223919 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdb1c0aaab0 0x7fdb1c0b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:17.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:17.914+0000 7fdb22f92700 1 --2- 172.21.15.23:0/72223919 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdb1c0aae90 0x7fdb1c0a5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:17.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:17.914+0000 7fdb22f92700 1 -- 172.21.15.23:0/72223919 >> 172.21.15.23:0/72223919 conn(0x7fdb1c0678e0 msgr2=0x7fdb1c068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:17.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:17.914+0000 7fdb22f92700 1 -- 172.21.15.23:0/72223919 shutdown_connections 2020-11-20T11:32:17.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:17.914+0000 7fdb22f92700 1 -- 172.21.15.23:0/72223919 wait complete. 2020-11-20T11:32:17.922 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'rbd' created 2020-11-20T11:32:17.932 INFO:teuthology.orchestra.run.smithi023:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2020-11-20T11:32:18.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.058+0000 7f19e5c98700 1 Processor -- start 2020-11-20T11:32:18.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.059+0000 7f19e5c98700 1 -- start start 2020-11-20T11:32:18.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.059+0000 7f19e5c98700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f19e00a61a0 0x7f19e00a69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:18.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.059+0000 7f19e5c98700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f19e013a4f0 0x7f19e013a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:18.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.059+0000 7f19e5c98700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e00a6ea0 con 0x7f19e00a6580 2020-11-20T11:32:18.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.059+0000 7f19e5c98700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19e00a90b0 con 0x7f19e00a61a0 2020-11-20T11:32:18.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.059+0000 7f19e5c98700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19e00a70e0 con 0x7f19e013a4f0 2020-11-20T11:32:18.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.059+0000 7f19df7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f19e00a61a0 0x7f19e00a69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:18.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.059+0000 7f19deffd700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f19e00a6580 0x7f19e00a9ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:54652/0 (socket says 172.21.15.23:54652) 2020-11-20T11:32:18.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.059+0000 7f19deffd700 1 -- 172.21.15.23:0/2274759974 learned_addr learned my addr 172.21.15.23:0/2274759974 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:18.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.060+0000 7f19dffff700 1 --2- 172.21.15.23:0/2274759974 >> v2:172.21.15.155:3300/0 conn(0x7f19e013a4f0 0x7f19e013a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:18.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.060+0000 7f19de7fc700 1 -- 172.21.15.23:0/2274759974 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1893959722 0 0) 0x7f19d4001110 con 0x7f19e00a6580 2020-11-20T11:32:18.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.060+0000 7f19de7fc700 1 -- 172.21.15.23:0/2274759974 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19cc003280 con 0x7f19e00a6580 2020-11-20T11:32:18.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.060+0000 7f19df7fe700 1 -- 172.21.15.23:0/2274759974 >> v2:172.21.15.155:3300/0 conn(0x7f19e013a4f0 msgr2=0x7f19e013a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:18.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.060+0000 7f19df7fe700 1 --2- 172.21.15.23:0/2274759974 >> v2:172.21.15.155:3300/0 conn(0x7f19e013a4f0 0x7f19e013a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:18.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.060+0000 7f19df7fe700 1 -- 172.21.15.23:0/2274759974 >> v1:172.21.15.23:6789/0 conn(0x7f19e00a6580 legacy=0x7f19e00a9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:18.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.060+0000 7f19df7fe700 1 -- 172.21.15.23:0/2274759974 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e00a9390 con 0x7f19e00a61a0 2020-11-20T11:32:18.071 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.060+0000 7f19df7fe700 1 --2- 172.21.15.23:0/2274759974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19e00a61a0 0x7f19e00a69b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f19e00a5570 tx=0x7f19d0006e30).ready entity=mon.0 client_cookie=a34af25b8db195a3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:18.071 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.060+0000 7f19de7fc700 1 -- 172.21.15.23:0/2274759974 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d00097b0 con 0x7f19e00a61a0 2020-11-20T11:32:18.071 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.060+0000 7f19de7fc700 1 -- 172.21.15.23:0/2274759974 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d0000e90 con 0x7f19e00a61a0 2020-11-20T11:32:18.071 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.060+0000 7f19de7fc700 1 -- 172.21.15.23:0/2274759974 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d0009b20 con 0x7f19e00a61a0 2020-11-20T11:32:18.072 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.060+0000 7f19e5c98700 1 -- 172.21.15.23:0/2274759974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19e00a61a0 msgr2=0x7f19e00a69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:18.072 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.060+0000 7f19e5c98700 1 --2- 172.21.15.23:0/2274759974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19e00a61a0 0x7f19e00a69b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f19e00a5570 tx=0x7f19d0006e30).stop 2020-11-20T11:32:18.072 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.061+0000 7f19e5c98700 1 -- 172.21.15.23:0/2274759974 shutdown_connections 2020-11-20T11:32:18.072 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.061+0000 7f19e5c98700 1 --2- 172.21.15.23:0/2274759974 >> v2:172.21.15.155:3300/0 conn(0x7f19e013a4f0 0x7f19e013a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:18.072 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.061+0000 7f19e5c98700 1 --2- 172.21.15.23:0/2274759974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19e00a61a0 0x7f19e00a69b0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:18.073 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.061+0000 7f19e5c98700 1 -- 172.21.15.23:0/2274759974 >> 172.21.15.23:0/2274759974 conn(0x7f19e0062320 msgr2=0x7f19e00601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:18.073 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.061+0000 7f19e5c98700 1 -- 172.21.15.23:0/2274759974 shutdown_connections 2020-11-20T11:32:18.073 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.061+0000 7f19e5c98700 1 -- 172.21.15.23:0/2274759974 wait complete. 2020-11-20T11:32:18.073 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.062+0000 7f19e5c98700 1 Processor -- start 2020-11-20T11:32:18.073 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.062+0000 7f19e5c98700 1 -- start start 2020-11-20T11:32:18.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.062+0000 7f19e5c98700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19e00a61a0 0x7f19e01372a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:18.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.062+0000 7f19e5c98700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19e00a6580 0x7f19e0129190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:18.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.062+0000 7f19e5c98700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19e013a4f0 0x7f19e0129680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:18.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.062+0000 7f19e5c98700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e00ac780 con 0x7f19e013a4f0 2020-11-20T11:32:18.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.062+0000 7f19e5c98700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e012b230 con 0x7f19e00a61a0 2020-11-20T11:32:18.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.062+0000 7f19e5c98700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e012b4d0 con 0x7f19e00a6580 2020-11-20T11:32:18.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.062+0000 7f19dffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19e013a4f0 0x7f19e0129680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:18.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.062+0000 7f19dffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19e013a4f0 0x7f19e0129680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40408/0 (socket says 172.21.15.23:40408) 2020-11-20T11:32:18.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.063+0000 7f19dffff700 1 -- 172.21.15.23:0/3275892196 learned_addr learned my addr 172.21.15.23:0/3275892196 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:18.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.063+0000 7f19df7fe700 1 --2- 172.21.15.23:0/3275892196 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19e00a61a0 0x7f19e01372a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:18.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.063+0000 7f19deffd700 1 --2- 172.21.15.23:0/3275892196 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19e00a6580 0x7f19e0129190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:18.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.063+0000 7f19dffff700 1 -- 172.21.15.23:0/3275892196 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19e00a6580 msgr2=0x7f19e0129190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:18.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.063+0000 7f19dffff700 1 --2- 172.21.15.23:0/3275892196 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19e00a6580 0x7f19e0129190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:18.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.063+0000 7f19dffff700 1 -- 172.21.15.23:0/3275892196 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19e00a61a0 msgr2=0x7f19e01372a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:18.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.063+0000 7f19dffff700 1 --2- 172.21.15.23:0/3275892196 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19e00a61a0 0x7f19e01372a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:18.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.063+0000 7f19dffff700 1 -- 172.21.15.23:0/3275892196 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e012b770 con 0x7f19e013a4f0 2020-11-20T11:32:18.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.063+0000 7f19dffff700 1 --2- 172.21.15.23:0/3275892196 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19e013a4f0 0x7f19e0129680 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f19d800db60 tx=0x7f19d8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:18.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.063+0000 7f19bf7fe700 1 -- 172.21.15.23:0/3275892196 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d8017070 con 0x7f19e013a4f0 2020-11-20T11:32:18.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.063+0000 7f19bf7fe700 1 -- 172.21.15.23:0/3275892196 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d800b7e0 con 0x7f19e013a4f0 2020-11-20T11:32:18.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.063+0000 7f19e5c98700 1 -- 172.21.15.23:0/3275892196 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19e0064e40 con 0x7f19e013a4f0 2020-11-20T11:32:18.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.063+0000 7f19bf7fe700 1 -- 172.21.15.23:0/3275892196 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d800bb80 con 0x7f19e013a4f0 2020-11-20T11:32:18.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.063+0000 7f19e5c98700 1 -- 172.21.15.23:0/3275892196 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19e00678e0 con 0x7f19e013a4f0 2020-11-20T11:32:18.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.064+0000 7f19e5c98700 1 -- 172.21.15.23:0/3275892196 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19e00aa200 con 0x7f19e013a4f0 2020-11-20T11:32:18.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.064+0000 7f19bf7fe700 1 -- 172.21.15.23:0/3275892196 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f19d8013480 con 0x7f19e013a4f0 2020-11-20T11:32:18.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.065+0000 7f19bf7fe700 1 --2- 172.21.15.23:0/3275892196 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f19b8056d60 0x7f19b8059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:18.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.065+0000 7f19bf7fe700 1 -- 172.21.15.23:0/3275892196 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5242+0+0 (secure 0 0 0) 0x7f19d806ff50 con 0x7f19e013a4f0 2020-11-20T11:32:18.079 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.065+0000 7f19df7fe700 1 --2- 172.21.15.23:0/3275892196 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f19b8056d60 0x7f19b8059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:18.079 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.065+0000 7f19df7fe700 1 --2- 172.21.15.23:0/3275892196 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f19b8056d60 0x7f19b8059220 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f19d0006e70 tx=0x7f19d0015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:18.079 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.068+0000 7f19bf7fe700 1 -- 172.21.15.23:0/3275892196 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f19d8045680 con 0x7f19e013a4f0 2020-11-20T11:32:18.229 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.226+0000 7f19e5c98700 1 -- 172.21.15.23:0/3275892196 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f19e0044710 con 0x7f19e013a4f0 2020-11-20T11:32:18.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.916+0000 7f19bf7fe700 1 -- 172.21.15.23:0/3275892196 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v17) v1 ==== 173+0+0 (secure 0 0 0) 0x7f19d8048ca0 con 0x7f19e013a4f0 2020-11-20T11:32:18.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.918+0000 7f19e5c98700 1 -- 172.21.15.23:0/3275892196 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f19b8056d60 msgr2=0x7f19b8059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:18.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.918+0000 7f19e5c98700 1 --2- 172.21.15.23:0/3275892196 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f19b8056d60 0x7f19b8059220 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f19d0006e70 tx=0x7f19d0015000).stop 2020-11-20T11:32:18.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.918+0000 7f19e5c98700 1 -- 172.21.15.23:0/3275892196 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19e013a4f0 msgr2=0x7f19e0129680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:18.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.918+0000 7f19e5c98700 1 --2- 172.21.15.23:0/3275892196 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19e013a4f0 0x7f19e0129680 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f19d800db60 tx=0x7f19d8008040).stop 2020-11-20T11:32:18.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.918+0000 7f19e5c98700 1 -- 172.21.15.23:0/3275892196 shutdown_connections 2020-11-20T11:32:18.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.918+0000 7f19e5c98700 1 --2- 172.21.15.23:0/3275892196 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19e013a4f0 0x7f19e0129680 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:18.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.918+0000 7f19e5c98700 1 --2- 172.21.15.23:0/3275892196 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19e00a61a0 0x7f19e01372a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:18.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.918+0000 7f19e5c98700 1 --2- 172.21.15.23:0/3275892196 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f19b8056d60 0x7f19b8059220 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:18.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.918+0000 7f19e5c98700 1 --2- 172.21.15.23:0/3275892196 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19e00a6580 0x7f19e0129190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:18.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.918+0000 7f19e5c98700 1 -- 172.21.15.23:0/3275892196 >> 172.21.15.23:0/3275892196 conn(0x7f19e0062320 msgr2=0x7f19e00a99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:18.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.918+0000 7f19e5c98700 1 -- 172.21.15.23:0/3275892196 shutdown_connections 2020-11-20T11:32:18.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:18.918+0000 7f19e5c98700 1 -- 172.21.15.23:0/3275892196 wait complete. 2020-11-20T11:32:18.925 INFO:teuthology.orchestra.run.smithi023.stderr:enabled application 'rbd' on pool 'rbd' 2020-11-20T11:32:18.939 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2020-11-20T11:32:18.939 INFO:tasks.ceph.mds.a:Restarting daemon 2020-11-20T11:32:18.940 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2020-11-20T11:32:18.945 INFO:tasks.ceph.mds.a:Started 2020-11-20T11:32:18.945 INFO:tasks.ceph.mds.b:Restarting daemon 2020-11-20T11:32:18.945 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2020-11-20T11:32:18.951 INFO:tasks.ceph.mds.b:Started 2020-11-20T11:32:18.952 INFO:tasks.ceph:Setting up CephFS filesystem... 2020-11-20T11:32:18.953 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T11:32:19.022 INFO:tasks.ceph.mds.a.smithi023.stdout:starting mds.a at 2020-11-20T11:32:19.023 INFO:tasks.ceph.mds.b.smithi155.stdout:starting mds.b at 2020-11-20T11:32:19.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.074+0000 7feecbc42700 1 Processor -- start 2020-11-20T11:32:19.079 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.074+0000 7feecbc42700 1 -- start start 2020-11-20T11:32:19.087 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.075+0000 7feecbc42700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7feec40a9d40 0x7feec40aa1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:19.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.075+0000 7feecbc42700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feec4067bc0 con 0x7feec40abb80 2020-11-20T11:32:19.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.075+0000 7feecbc42700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feec4067e60 con 0x7feec4060a80 2020-11-20T11:32:19.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.075+0000 7feecbc42700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feec4062060 con 0x7feec40a9d40 2020-11-20T11:32:19.089 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.075+0000 7feeca1df700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7feec40abb80 0x7feec40abff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:54718/0 (socket says 172.21.15.23:54718) 2020-11-20T11:32:19.089 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.075+0000 7feeca1df700 1 -- 172.21.15.23:0/829021047 learned_addr learned my addr 172.21.15.23:0/829021047 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:19.089 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.075+0000 7feec91dd700 1 --2- 172.21.15.23:0/829021047 >> v2:172.21.15.155:3300/0 conn(0x7feec40a9d40 0x7feec40aa1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:19.089 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.075+0000 7feec89dc700 1 -- 172.21.15.23:0/829021047 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1903532060 0 0) 0x7feec4067bc0 con 0x7feec40abb80 2020-11-20T11:32:19.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.075+0000 7feec89dc700 1 -- 172.21.15.23:0/829021047 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feeac003280 con 0x7feec40abb80 2020-11-20T11:32:19.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.075+0000 7feec89dc700 1 -- 172.21.15.23:0/829021047 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1921203980 0 0) 0x7feec4067e60 con 0x7feec4060a80 2020-11-20T11:32:19.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.075+0000 7feec89dc700 1 -- 172.21.15.23:0/829021047 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feec4067bc0 con 0x7feec4060a80 2020-11-20T11:32:19.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.075+0000 7feec89dc700 1 -- 172.21.15.23:0/829021047 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1002245808 0 0) 0x7feeac003280 con 0x7feec40abb80 2020-11-20T11:32:19.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.075+0000 7feec91dd700 1 -- 172.21.15.23:0/829021047 >> v1:172.21.15.155:6789/0 conn(0x7feec4060a80 legacy=0x7feec4061950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:19.091 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.075+0000 7feec91dd700 1 -- 172.21.15.23:0/829021047 >> v1:172.21.15.23:6789/0 conn(0x7feec40abb80 legacy=0x7feec40abff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:19.091 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.075+0000 7feec91dd700 1 -- 172.21.15.23:0/829021047 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec4068100 con 0x7feec40a9d40 2020-11-20T11:32:19.091 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.076+0000 7feec91dd700 1 --2- 172.21.15.23:0/829021047 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feec40a9d40 0x7feec40aa1c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7feec0006fd0 tx=0x7feec0006e30).ready entity=mon.1 client_cookie=e834753577280249 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:19.092 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.076+0000 7feec89dc700 1 -- 172.21.15.23:0/829021047 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feebc0018a0 con 0x7feec40a9d40 2020-11-20T11:32:19.092 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.076+0000 7feec89dc700 1 -- 172.21.15.23:0/829021047 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feec0000e90 con 0x7feec40a9d40 2020-11-20T11:32:19.092 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.076+0000 7feec89dc700 1 -- 172.21.15.23:0/829021047 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec0011610 con 0x7feec40a9d40 2020-11-20T11:32:19.093 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.076+0000 7feecbc42700 1 -- 172.21.15.23:0/829021047 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feec40a9d40 msgr2=0x7feec40aa1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:19.093 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.076+0000 7feecbc42700 1 --2- 172.21.15.23:0/829021047 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feec40a9d40 0x7feec40aa1c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7feec0006fd0 tx=0x7feec0006e30).stop 2020-11-20T11:32:19.094 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.076+0000 7feecbc42700 1 -- 172.21.15.23:0/829021047 shutdown_connections 2020-11-20T11:32:19.094 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.076+0000 7feecbc42700 1 --2- 172.21.15.23:0/829021047 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feec40a9d40 0x7feec40aa1c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:19.094 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.076+0000 7feecbc42700 1 -- 172.21.15.23:0/829021047 >> 172.21.15.23:0/829021047 conn(0x7feec4066730 msgr2=0x7feec40601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:19.095 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.076+0000 7feecbc42700 1 -- 172.21.15.23:0/829021047 shutdown_connections 2020-11-20T11:32:19.095 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.076+0000 7feecbc42700 1 -- 172.21.15.23:0/829021047 wait complete. 2020-11-20T11:32:19.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.077+0000 7feecbc42700 1 Processor -- start 2020-11-20T11:32:19.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.077+0000 7feecbc42700 1 -- start start 2020-11-20T11:32:19.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.078+0000 7feecbc42700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feec4060a80 0x7feec413ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:19.097 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.078+0000 7feecbc42700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feec40a9d40 0x7feec4136900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:19.097 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.078+0000 7feecbc42700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feec40abb80 0x7feec40ae780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:19.097 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.078+0000 7feecbc42700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec413d120 con 0x7feec40abb80 2020-11-20T11:32:19.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.078+0000 7feecbc42700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec413d3c0 con 0x7feec40a9d40 2020-11-20T11:32:19.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.078+0000 7feecbc42700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feec413e860 con 0x7feec4060a80 2020-11-20T11:32:19.099 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.078+0000 7feeca1df700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feec40abb80 0x7feec40ae780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:19.099 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.078+0000 7feeca1df700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feec40abb80 0x7feec40ae780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40474/0 (socket says 172.21.15.23:40474) 2020-11-20T11:32:19.099 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.078+0000 7feeca1df700 1 -- 172.21.15.23:0/1923624011 learned_addr learned my addr 172.21.15.23:0/1923624011 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:19.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.078+0000 7feec99de700 1 --2- 172.21.15.23:0/1923624011 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feec4060a80 0x7feec413ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:19.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.078+0000 7feec91dd700 1 --2- 172.21.15.23:0/1923624011 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feec40a9d40 0x7feec4136900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:19.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.078+0000 7feeca1df700 1 -- 172.21.15.23:0/1923624011 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feec4060a80 msgr2=0x7feec413ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:19.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.078+0000 7feeca1df700 1 --2- 172.21.15.23:0/1923624011 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feec4060a80 0x7feec413ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:19.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.078+0000 7feeca1df700 1 -- 172.21.15.23:0/1923624011 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feec40a9d40 msgr2=0x7feec4136900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:19.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.078+0000 7feeca1df700 1 --2- 172.21.15.23:0/1923624011 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feec40a9d40 0x7feec4136900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:19.102 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.078+0000 7feeca1df700 1 -- 172.21.15.23:0/1923624011 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec4138a10 con 0x7feec40abb80 2020-11-20T11:32:19.102 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.078+0000 7feeca1df700 1 --2- 172.21.15.23:0/1923624011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feec40abb80 0x7feec40ae780 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7feebc00ce60 tx=0x7feebc001e80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:19.102 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.079+0000 7feeb1ffb700 1 -- 172.21.15.23:0/1923624011 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feebc004ad0 con 0x7feec40abb80 2020-11-20T11:32:19.102 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.079+0000 7feeb1ffb700 1 -- 172.21.15.23:0/1923624011 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feebc004c30 con 0x7feec40abb80 2020-11-20T11:32:19.103 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.079+0000 7feeb1ffb700 1 -- 172.21.15.23:0/1923624011 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feebc007170 con 0x7feec40abb80 2020-11-20T11:32:19.103 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.079+0000 7feecbc42700 1 -- 172.21.15.23:0/1923624011 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feec41373f0 con 0x7feec40abb80 2020-11-20T11:32:19.103 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.079+0000 7feecbc42700 1 -- 172.21.15.23:0/1923624011 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feec413c3a0 con 0x7feec40abb80 2020-11-20T11:32:19.103 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.080+0000 7feecbc42700 1 -- 172.21.15.23:0/1923624011 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feec40aca60 con 0x7feec40abb80 2020-11-20T11:32:19.103 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.080+0000 7feeb1ffb700 1 -- 172.21.15.23:0/1923624011 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7feebc010070 con 0x7feec40abb80 2020-11-20T11:32:19.104 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.080+0000 7feeb1ffb700 1 --2- 172.21.15.23:0/1923624011 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7feea0056d90 0x7feea0059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:19.104 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.080+0000 7feeb1ffb700 1 -- 172.21.15.23:0/1923624011 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5253+0+0 (secure 0 0 0) 0x7feebc06f990 con 0x7feec40abb80 2020-11-20T11:32:19.104 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.080+0000 7feec99de700 1 --2- 172.21.15.23:0/1923624011 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7feea0056d90 0x7feea0059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:19.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.081+0000 7feec99de700 1 --2- 172.21.15.23:0/1923624011 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7feea0056d90 0x7feea0059250 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7feeb4003710 tx=0x7feeb4007a60).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:19.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.084+0000 7feeb1ffb700 1 -- 172.21.15.23:0/1923624011 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7feebc045110 con 0x7feec40abb80 2020-11-20T11:32:19.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.233+0000 7feecbc42700 1 -- 172.21.15.23:0/1923624011 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feec4061e40 con 0x7feec40abb80 2020-11-20T11:32:19.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.234+0000 7feeb1ffb700 1 -- 172.21.15.23:0/1923624011 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+13685 (secure 0 0 0) 0x7feebc048730 con 0x7feec40abb80 2020-11-20T11:32:19.238 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:32:19.239 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":17,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","created":"2020-11-20T11:32:06.136679+0000","modified":"2020-11-20T11:32:18.909333+0000","last_up_change":"2020-11-20T11:32:13.536194+0000","last_in_change":"2020-11-20T11:32:13.536194+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T11:32:09.559868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T11:32:17.000782+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"e9d219f5-e40e-4b2b-9468-deb9c4074a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":23083},{"type":"v1","addr":"172.21.15.23:6801","nonce":23083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":23083},{"type":"v1","addr":"172.21.15.23:6803","nonce":23083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":23083},{"type":"v1","addr":"172.21.15.23:6807","nonce":23083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":23083},{"type":"v1","addr":"172.21.15.23:6805","nonce":23083}]},"public_addr":"172.21.15.23:6801/23083","cluster_addr":"172.21.15.23:6803/23083","heartbeat_back_addr":"172.21.15.23:6807/23083","heartbeat_front_addr":"172.21.15.23:6805/23083","state":["exists","up"]},{"osd":1,"uuid":"2328adde-2108-4fe0-8138-f0d096a496a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":23099},{"type":"v1","addr":"172.21.15.23:6817","nonce":23099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":23099},{"type":"v1","addr":"172.21.15.23:6819","nonce":23099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":23099},{"type":"v1","addr":"172.21.15.23:6824","nonce":23099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":23099},{"type":"v1","addr":"172.21.15.23:6821","nonce":23099}]},"public_addr":"172.21.15.23:6817/23099","cluster_addr":"172.21.15.23:6819/23099","heartbeat_back_addr":"172.21.15.23:6824/23099","heartbeat_front_addr":"172.21.15.23:6821/23099","state":["exists","up"]},{"osd":2,"uuid":"13367708-59a2-4363-8a3a-ab0cb97145ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":23084},{"type":"v1","addr":"172.21.15.23:6809","nonce":23084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":23084},{"type":"v1","addr":"172.21.15.23:6811","nonce":23084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":23084},{"type":"v1","addr":"172.21.15.23:6815","nonce":23084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":23084},{"type":"v1","addr":"172.21.15.23:6813","nonce":23084}]},"public_addr":"172.21.15.23:6809/23084","cluster_addr":"172.21.15.23:6811/23084","heartbeat_back_addr":"172.21.15.23:6815/23084","heartbeat_front_addr":"172.21.15.23:6813/23084","state":["exists","up"]},{"osd":3,"uuid":"1f147dd7-9289-47b0-ad1c-4088618d6faa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6823","nonce":23114},{"type":"v1","addr":"172.21.15.23:6825","nonce":23114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":23114},{"type":"v1","addr":"172.21.15.23:6827","nonce":23114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":23114},{"type":"v1","addr":"172.21.15.23:6831","nonce":23114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":23114},{"type":"v1","addr":"172.21.15.23:6829","nonce":23114}]},"public_addr":"172.21.15.23:6825/23114","cluster_addr":"172.21.15.23:6827/23114","heartbeat_back_addr":"172.21.15.23:6831/23114","heartbeat_front_addr":"172.21.15.23:6829/23114","state":["exists","up"]},{"osd":4,"uuid":"d008b039-4bc5-4135-9678-1511e58efc29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":23280},{"type":"v1","addr":"172.21.15.155:6801","nonce":23280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":23280},{"type":"v1","addr":"172.21.15.155:6803","nonce":23280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":23280},{"type":"v1","addr":"172.21.15.155:6807","nonce":23280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":23280},{"type":"v1","addr":"172.21.15.155:6805","nonce":23280}]},"public_addr":"172.21.15.155:6801/23280","cluster_addr":"172.21.15.155:6803/23280","heartbeat_back_addr":"172.21.15.155:6807/23280","heartbeat_front_addr":"172.21.15.155:6805/23280","state":["exists","up"]},{"osd":5,"uuid":"56715e6f-cf6e-44a6-9e39-93ab1224095f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":23298},{"type":"v1","addr":"172.21.15.155:6817","nonce":23298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":23298},{"type":"v1","addr":"172.21.15.155:6819","nonce":23298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":23298},{"type":"v1","addr":"172.21.15.155:6823","nonce":23298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":23298},{"type":"v1","addr":"172.21.15.155:6821","nonce":23298}]},"public_addr":"172.21.15.155:6817/23298","cluster_addr":"172.21.15.155:6819/23298","heartbeat_back_addr":"172.21.15.155:6823/23298","heartbeat_front_addr":"172.21.15.155:6821/23298","state":["exists","up"]},{"osd":6,"uuid":"db1e61ca-8e14-4a1f-8607-6e01b148d61c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":23307},{"type":"v1","addr":"172.21.15.155:6825","nonce":23307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":23307},{"type":"v1","addr":"172.21.15.155:6827","nonce":23307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":23307},{"type":"v1","addr":"172.21.15.155:6831","nonce":23307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":23307},{"type":"v1","addr":"172.21.15.155:6829","nonce":23307}]},"public_addr":"172.21.15.155:6825/23307","cluster_addr":"172.21.15.155:6827/23307","heartbeat_back_addr":"172.21.15.155:6831/23307","heartbeat_front_addr":"172.21.15.155:6829/23307","state":["exists","up"]},{"osd":7,"uuid":"d44419be-ef89-462c-8cf1-3c6db2a6944b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":23281},{"type":"v1","addr":"172.21.15.155:6809","nonce":23281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":23281},{"type":"v1","addr":"172.21.15.155:6811","nonce":23281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":23281},{"type":"v1","addr":"172.21.15.155:6815","nonce":23281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":23281},{"type":"v1","addr":"172.21.15.155:6813","nonce":23281}]},"public_addr":"172.21.15.155:6809/23281","cluster_addr":"172.21.15.155:6811/23281","heartbeat_back_addr":"172.21.15.155:6815/23281","heartbeat_front_addr":"172.21.15.155:6813/23281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.486702+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.509396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.533414+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.509489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.467600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.513068+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.536636+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.531772+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T11:32:19.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.236+0000 7feecbc42700 1 -- 172.21.15.23:0/1923624011 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7feea0056d90 msgr2=0x7feea0059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:19.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.236+0000 7feecbc42700 1 --2- 172.21.15.23:0/1923624011 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7feea0056d90 0x7feea0059250 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7feeb4003710 tx=0x7feeb4007a60).stop 2020-11-20T11:32:19.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.236+0000 7feecbc42700 1 -- 172.21.15.23:0/1923624011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feec40abb80 msgr2=0x7feec40ae780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:19.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.236+0000 7feecbc42700 1 --2- 172.21.15.23:0/1923624011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feec40abb80 0x7feec40ae780 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7feebc00ce60 tx=0x7feebc001e80).stop 2020-11-20T11:32:19.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.236+0000 7feecbc42700 1 -- 172.21.15.23:0/1923624011 shutdown_connections 2020-11-20T11:32:19.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.236+0000 7feecbc42700 1 --2- 172.21.15.23:0/1923624011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feec40abb80 0x7feec40ae780 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:19.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.236+0000 7feecbc42700 1 --2- 172.21.15.23:0/1923624011 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7feea0056d90 0x7feea0059250 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:19.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.236+0000 7feecbc42700 1 --2- 172.21.15.23:0/1923624011 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feec4060a80 0x7feec413ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:19.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.236+0000 7feecbc42700 1 --2- 172.21.15.23:0/1923624011 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feec40a9d40 0x7feec4136900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:19.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.236+0000 7feecbc42700 1 -- 172.21.15.23:0/1923624011 >> 172.21.15.23:0/1923624011 conn(0x7feec4066730 msgr2=0x7feec4137890 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:19.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.236+0000 7feecbc42700 1 -- 172.21.15.23:0/1923624011 shutdown_connections 2020-11-20T11:32:19.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.236+0000 7feecbc42700 1 -- 172.21.15.23:0/1923624011 wait complete. 2020-11-20T11:32:19.251 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T11:32:09.559868+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T11:32:17.000782+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-11-20T11:32:19.252 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T11:32:19.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.366+0000 7f1137fd6700 1 Processor -- start 2020-11-20T11:32:19.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.366+0000 7f1137fd6700 1 -- start start 2020-11-20T11:32:19.372 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.367+0000 7f1137fd6700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1130060a80 0x7f1130061950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:19.373 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.367+0000 7f1137fd6700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11300aa1c0 0x7f1130138030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:19.373 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.367+0000 7f1137fd6700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1130067bc0 con 0x7f11300a9d40 2020-11-20T11:32:19.373 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.367+0000 7f1137fd6700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11300aa630 con 0x7f1130060a80 2020-11-20T11:32:19.373 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.367+0000 7f1137fd6700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1130067e60 con 0x7f11300aa1c0 2020-11-20T11:32:19.374 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.367+0000 7f1135d72700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1130060a80 0x7f1130061950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:19.374 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.367+0000 7f1135571700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f11300a9d40 0x7f1130137920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:54730/0 (socket says 172.21.15.23:54730) 2020-11-20T11:32:19.374 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.367+0000 7f1135571700 1 -- 172.21.15.23:0/1408506143 learned_addr learned my addr 172.21.15.23:0/1408506143 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:19.375 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.367+0000 7f1136573700 1 --2- 172.21.15.23:0/1408506143 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11300aa1c0 0x7f1130138030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:19.375 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.367+0000 7f1135d72700 1 -- 172.21.15.23:0/1408506143 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11300aa1c0 msgr2=0x7f1130138030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:19.375 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.367+0000 7f1135d72700 1 --2- 172.21.15.23:0/1408506143 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11300aa1c0 0x7f1130138030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:19.375 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.367+0000 7f1135d72700 1 -- 172.21.15.23:0/1408506143 >> v1:172.21.15.23:6789/0 conn(0x7f11300a9d40 legacy=0x7f1130137920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:19.376 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.367+0000 7f1134d70700 1 -- 172.21.15.23:0/1408506143 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3473617215 0 0) 0x7f1130067bc0 con 0x7f11300a9d40 2020-11-20T11:32:19.376 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.367+0000 7f1135d72700 1 -- 172.21.15.23:0/1408506143 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1130068100 con 0x7f1130060a80 2020-11-20T11:32:19.376 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.367+0000 7f1136573700 1 --2- 172.21.15.23:0/1408506143 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11300aa1c0 0x7f1130138030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:32:19.377 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.367+0000 7f1135d72700 1 --2- 172.21.15.23:0/1408506143 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1130060a80 0x7f1130061950 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f1120006fd0 tx=0x7f1120006e30).ready entity=mon.0 client_cookie=685ac9a39f6ea70c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:19.377 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.367+0000 7f1134d70700 1 -- 172.21.15.23:0/1408506143 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11200097e0 con 0x7f1130060a80 2020-11-20T11:32:19.377 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.367+0000 7f1134d70700 1 -- 172.21.15.23:0/1408506143 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1120000e90 con 0x7f1130060a80 2020-11-20T11:32:19.378 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.367+0000 7f1134d70700 1 -- 172.21.15.23:0/1408506143 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1120009b50 con 0x7f1130060a80 2020-11-20T11:32:19.378 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.368+0000 7f1137fd6700 1 -- 172.21.15.23:0/1408506143 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1130060a80 msgr2=0x7f1130061950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:19.378 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.368+0000 7f1137fd6700 1 --2- 172.21.15.23:0/1408506143 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1130060a80 0x7f1130061950 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f1120006fd0 tx=0x7f1120006e30).stop 2020-11-20T11:32:19.379 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.368+0000 7f1137fd6700 1 -- 172.21.15.23:0/1408506143 shutdown_connections 2020-11-20T11:32:19.379 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.368+0000 7f1137fd6700 1 --2- 172.21.15.23:0/1408506143 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11300aa1c0 0x7f1130138030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:19.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.368+0000 7f1137fd6700 1 --2- 172.21.15.23:0/1408506143 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1130060a80 0x7f1130061950 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:19.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.368+0000 7f1137fd6700 1 -- 172.21.15.23:0/1408506143 >> 172.21.15.23:0/1408506143 conn(0x7f1130066730 msgr2=0x7f11300601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:19.381 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.368+0000 7f1137fd6700 1 -- 172.21.15.23:0/1408506143 shutdown_connections 2020-11-20T11:32:19.381 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.368+0000 7f1137fd6700 1 -- 172.21.15.23:0/1408506143 wait complete. 2020-11-20T11:32:19.382 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.369+0000 7f1137fd6700 1 Processor -- start 2020-11-20T11:32:19.382 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.369+0000 7f1137fd6700 1 -- start start 2020-11-20T11:32:19.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.369+0000 7f1137fd6700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1130060a80 0x7f11300a8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:19.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.369+0000 7f1137fd6700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11300a9d40 0x7f11300a5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:19.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.369+0000 7f1137fd6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11300aa1c0 0x7f1130138160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:19.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.369+0000 7f1137fd6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11300a5220 con 0x7f11300aa1c0 2020-11-20T11:32:19.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.369+0000 7f1137fd6700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11300a5810 con 0x7f1130060a80 2020-11-20T11:32:19.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.369+0000 7f1137fd6700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11300a54c0 con 0x7f11300a9d40 2020-11-20T11:32:19.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.370+0000 7f1136573700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11300aa1c0 0x7f1130138160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:19.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.370+0000 7f1136573700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11300aa1c0 0x7f1130138160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40488/0 (socket says 172.21.15.23:40488) 2020-11-20T11:32:19.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.370+0000 7f1136573700 1 -- 172.21.15.23:0/3567225280 learned_addr learned my addr 172.21.15.23:0/3567225280 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:19.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.370+0000 7f1135d72700 1 --2- 172.21.15.23:0/3567225280 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1130060a80 0x7f11300a8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:19.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.370+0000 7f1135571700 1 --2- 172.21.15.23:0/3567225280 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11300a9d40 0x7f11300a5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:19.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.370+0000 7f1136573700 1 -- 172.21.15.23:0/3567225280 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11300a9d40 msgr2=0x7f11300a5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:19.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.370+0000 7f1136573700 1 --2- 172.21.15.23:0/3567225280 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11300a9d40 0x7f11300a5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:19.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.370+0000 7f1136573700 1 -- 172.21.15.23:0/3567225280 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1130060a80 msgr2=0x7f11300a8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:19.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.370+0000 7f1136573700 1 --2- 172.21.15.23:0/3567225280 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1130060a80 0x7f11300a8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:19.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.370+0000 7f1136573700 1 -- 172.21.15.23:0/3567225280 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11300a78b0 con 0x7f11300aa1c0 2020-11-20T11:32:19.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.370+0000 7f1136573700 1 --2- 172.21.15.23:0/3567225280 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11300aa1c0 0x7f1130138160 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f112800cb20 tx=0x7f1128008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:19.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.370+0000 7f111dffb700 1 -- 172.21.15.23:0/3567225280 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11280077e0 con 0x7f11300aa1c0 2020-11-20T11:32:19.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.370+0000 7f111dffb700 1 -- 172.21.15.23:0/3567225280 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1128007940 con 0x7f11300aa1c0 2020-11-20T11:32:19.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.370+0000 7f1137fd6700 1 -- 172.21.15.23:0/3567225280 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11300a3700 con 0x7f11300aa1c0 2020-11-20T11:32:19.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.370+0000 7f111dffb700 1 -- 172.21.15.23:0/3567225280 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1128007cc0 con 0x7f11300aa1c0 2020-11-20T11:32:19.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.370+0000 7f1137fd6700 1 -- 172.21.15.23:0/3567225280 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11300a4ed0 con 0x7f11300aa1c0 2020-11-20T11:32:19.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.371+0000 7f1137fd6700 1 -- 172.21.15.23:0/3567225280 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11300a59d0 con 0x7f11300aa1c0 2020-11-20T11:32:19.394 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.372+0000 7f111dffb700 1 -- 172.21.15.23:0/3567225280 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1128011620 con 0x7f11300aa1c0 2020-11-20T11:32:19.394 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.372+0000 7f111dffb700 1 --2- 172.21.15.23:0/3567225280 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1108056d60 0x7f1108059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:19.394 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.372+0000 7f111dffb700 1 -- 172.21.15.23:0/3567225280 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f112806fff0 con 0x7f11300aa1c0 2020-11-20T11:32:19.394 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.372+0000 7f1135d72700 1 --2- 172.21.15.23:0/3567225280 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1108056d60 0x7f1108059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:19.394 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.372+0000 7f1135d72700 1 --2- 172.21.15.23:0/3567225280 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1108056d60 0x7f1108059220 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f1120006e70 tx=0x7f1120015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:19.395 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.375+0000 7f111dffb700 1 -- 172.21.15.23:0/3567225280 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1128045770 con 0x7f11300aa1c0 2020-11-20T11:32:19.529 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.525+0000 7f1137fd6700 1 -- 172.21.15.23:0/3567225280 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f1130044710 con 0x7f11300aa1c0 2020-11-20T11:32:19.529 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.525+0000 7f111dffb700 1 -- 172.21.15.23:0/3567225280 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v17) v1 ==== 110+0+10 (secure 0 0 0) 0x7f1128048d90 con 0x7f11300aa1c0 2020-11-20T11:32:19.530 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2020-11-20T11:32:19.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.527+0000 7f1137fd6700 1 -- 172.21.15.23:0/3567225280 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1108056d60 msgr2=0x7f1108059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:19.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.527+0000 7f1137fd6700 1 --2- 172.21.15.23:0/3567225280 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1108056d60 0x7f1108059220 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f1120006e70 tx=0x7f1120015000).stop 2020-11-20T11:32:19.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.527+0000 7f1137fd6700 1 -- 172.21.15.23:0/3567225280 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11300aa1c0 msgr2=0x7f1130138160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:19.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.527+0000 7f1137fd6700 1 --2- 172.21.15.23:0/3567225280 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11300aa1c0 0x7f1130138160 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f112800cb20 tx=0x7f1128008040).stop 2020-11-20T11:32:19.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.527+0000 7f1137fd6700 1 -- 172.21.15.23:0/3567225280 shutdown_connections 2020-11-20T11:32:19.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.527+0000 7f1137fd6700 1 --2- 172.21.15.23:0/3567225280 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11300aa1c0 0x7f1130138160 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:19.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.527+0000 7f1137fd6700 1 --2- 172.21.15.23:0/3567225280 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1130060a80 0x7f11300a8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:19.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.527+0000 7f1137fd6700 1 --2- 172.21.15.23:0/3567225280 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1108056d60 0x7f1108059220 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:19.533 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.527+0000 7f1137fd6700 1 --2- 172.21.15.23:0/3567225280 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11300a9d40 0x7f11300a5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:19.533 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.527+0000 7f1137fd6700 1 -- 172.21.15.23:0/3567225280 >> 172.21.15.23:0/3567225280 conn(0x7f1130066730 msgr2=0x7f1130141580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:19.533 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.527+0000 7f1137fd6700 1 -- 172.21.15.23:0/3567225280 shutdown_connections 2020-11-20T11:32:19.533 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.527+0000 7f1137fd6700 1 -- 172.21.15.23:0/3567225280 wait complete. 2020-11-20T11:32:19.542 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T11:32:19.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.655+0000 7efe0a077700 1 Processor -- start 2020-11-20T11:32:19.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.655+0000 7efe0a077700 1 -- start start 2020-11-20T11:32:19.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.656+0000 7efe0a077700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe040a35f0 0x7efe040a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:19.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.656+0000 7efe0a077700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7efe0413e870 0x7efe0413ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:19.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.656+0000 7efe0a077700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe040a9b00 con 0x7efe040a3210 2020-11-20T11:32:19.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.656+0000 7efe0a077700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe040a1da0 con 0x7efe0413e870 2020-11-20T11:32:19.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.656+0000 7efe0a077700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe040aa710 con 0x7efe040a35f0 2020-11-20T11:32:19.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.656+0000 7efe03fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7efe0413e870 0x7efe0413ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:19.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.656+0000 7efe037fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7efe040a3210 0x7efe040a9f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:54742/0 (socket says 172.21.15.23:54742) 2020-11-20T11:32:19.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.656+0000 7efe037fe700 1 -- 172.21.15.23:0/1823133581 learned_addr learned my addr 172.21.15.23:0/1823133581 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:19.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.656+0000 7efe02ffd700 1 --2- 172.21.15.23:0/1823133581 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe040a35f0 0x7efe040a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:19.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.656+0000 7efe03fff700 1 -- 172.21.15.23:0/1823133581 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe040a35f0 msgr2=0x7efe040a18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:19.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.656+0000 7efe03fff700 1 --2- 172.21.15.23:0/1823133581 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe040a35f0 0x7efe040a18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:19.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.656+0000 7efe03fff700 1 -- 172.21.15.23:0/1823133581 >> v1:172.21.15.23:6789/0 conn(0x7efe040a3210 legacy=0x7efe040a9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:19.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.656+0000 7efe027fc700 1 -- 172.21.15.23:0/1823133581 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3474942351 0 0) 0x7efe040a9b00 con 0x7efe040a3210 2020-11-20T11:32:19.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.656+0000 7efe03fff700 1 -- 172.21.15.23:0/1823133581 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe040aa9b0 con 0x7efe0413e870 2020-11-20T11:32:19.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.656+0000 7efe03fff700 1 --2- 172.21.15.23:0/1823133581 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe0413e870 0x7efe0413ed00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7efdfc009d10 tx=0x7efdfc009220).ready entity=mon.0 client_cookie=7d9fda741d008c48 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:19.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.657+0000 7efe027fc700 1 -- 172.21.15.23:0/1823133581 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdfc00bbb0 con 0x7efe0413e870 2020-11-20T11:32:19.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.657+0000 7efe027fc700 1 -- 172.21.15.23:0/1823133581 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdfc00bd10 con 0x7efe0413e870 2020-11-20T11:32:19.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.657+0000 7efe027fc700 1 -- 172.21.15.23:0/1823133581 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdfc011610 con 0x7efe0413e870 2020-11-20T11:32:19.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.657+0000 7efe0a077700 1 -- 172.21.15.23:0/1823133581 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe0413e870 msgr2=0x7efe0413ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:19.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.657+0000 7efe0a077700 1 --2- 172.21.15.23:0/1823133581 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe0413e870 0x7efe0413ed00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7efdfc009d10 tx=0x7efdfc009220).stop 2020-11-20T11:32:19.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.657+0000 7efe0a077700 1 -- 172.21.15.23:0/1823133581 shutdown_connections 2020-11-20T11:32:19.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.657+0000 7efe0a077700 1 --2- 172.21.15.23:0/1823133581 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe0413e870 0x7efe0413ed00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:19.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.657+0000 7efe0a077700 1 --2- 172.21.15.23:0/1823133581 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe040a35f0 0x7efe040a18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:19.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.657+0000 7efe0a077700 1 -- 172.21.15.23:0/1823133581 >> 172.21.15.23:0/1823133581 conn(0x7efe040601b0 msgr2=0x7efe040605c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:19.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.657+0000 7efe0a077700 1 -- 172.21.15.23:0/1823133581 shutdown_connections 2020-11-20T11:32:19.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.657+0000 7efe0a077700 1 -- 172.21.15.23:0/1823133581 wait complete. 2020-11-20T11:32:19.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.658+0000 7efe0a077700 1 Processor -- start 2020-11-20T11:32:19.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.658+0000 7efe0a077700 1 -- start start 2020-11-20T11:32:19.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.659+0000 7efe0a077700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe040a3210 0x7efe04066110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:19.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.659+0000 7efe0a077700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe040a35f0 0x7efe040a6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:19.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.659+0000 7efe0a077700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe0413e870 0x7efe04149d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:19.675 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.659+0000 7efe0a077700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe040a4c20 con 0x7efe040a35f0 2020-11-20T11:32:19.675 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.659+0000 7efe0a077700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe040a58c0 con 0x7efe0413e870 2020-11-20T11:32:19.675 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.659+0000 7efe0a077700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe040a5b60 con 0x7efe040a3210 2020-11-20T11:32:19.675 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.659+0000 7efe02ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe040a35f0 0x7efe040a6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:19.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.659+0000 7efe02ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe040a35f0 0x7efe040a6d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40498/0 (socket says 172.21.15.23:40498) 2020-11-20T11:32:19.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.659+0000 7efe02ffd700 1 -- 172.21.15.23:0/2834756782 learned_addr learned my addr 172.21.15.23:0/2834756782 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:19.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.659+0000 7efe037fe700 1 --2- 172.21.15.23:0/2834756782 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe040a3210 0x7efe04066110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:19.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.659+0000 7efe03fff700 1 --2- 172.21.15.23:0/2834756782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe0413e870 0x7efe04149d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:19.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.659+0000 7efe02ffd700 1 -- 172.21.15.23:0/2834756782 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe040a3210 msgr2=0x7efe04066110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:19.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.659+0000 7efe02ffd700 1 --2- 172.21.15.23:0/2834756782 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe040a3210 0x7efe04066110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:19.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.659+0000 7efe02ffd700 1 -- 172.21.15.23:0/2834756782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe0413e870 msgr2=0x7efe04149d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:19.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.659+0000 7efe02ffd700 1 --2- 172.21.15.23:0/2834756782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe0413e870 0x7efe04149d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:19.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.659+0000 7efe02ffd700 1 -- 172.21.15.23:0/2834756782 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe040a5e00 con 0x7efe040a35f0 2020-11-20T11:32:19.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.659+0000 7efe03fff700 1 --2- 172.21.15.23:0/2834756782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe0413e870 0x7efe04149d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:19.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.659+0000 7efe02ffd700 1 --2- 172.21.15.23:0/2834756782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe040a35f0 0x7efe040a6d50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7efdf800db30 tx=0x7efdf8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:19.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.659+0000 7efde37fe700 1 -- 172.21.15.23:0/2834756782 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf800be50 con 0x7efe040a35f0 2020-11-20T11:32:19.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.660+0000 7efde37fe700 1 -- 172.21.15.23:0/2834756782 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdf80097c0 con 0x7efe040a35f0 2020-11-20T11:32:19.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.660+0000 7efde37fe700 1 -- 172.21.15.23:0/2834756782 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf8009b40 con 0x7efe040a35f0 2020-11-20T11:32:19.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.660+0000 7efe0a077700 1 -- 172.21.15.23:0/2834756782 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe040a62a0 con 0x7efe040a35f0 2020-11-20T11:32:19.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.660+0000 7efe0a077700 1 -- 172.21.15.23:0/2834756782 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe040a7240 con 0x7efe040a35f0 2020-11-20T11:32:19.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.661+0000 7efe0a077700 1 -- 172.21.15.23:0/2834756782 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe04062c40 con 0x7efe040a35f0 2020-11-20T11:32:19.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.661+0000 7efde37fe700 1 -- 172.21.15.23:0/2834756782 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7efdf8013480 con 0x7efe040a35f0 2020-11-20T11:32:19.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.661+0000 7efde37fe700 1 --2- 172.21.15.23:0/2834756782 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efddc056d60 0x7efddc059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:19.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.661+0000 7efde37fe700 1 -- 172.21.15.23:0/2834756782 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5253+0+0 (secure 0 0 0) 0x7efdf806ff20 con 0x7efe040a35f0 2020-11-20T11:32:19.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.661+0000 7efe037fe700 1 --2- 172.21.15.23:0/2834756782 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efddc056d60 0x7efddc059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:19.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.662+0000 7efe037fe700 1 --2- 172.21.15.23:0/2834756782 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efddc056d60 0x7efddc059220 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7efdf40011e0 tx=0x7efdf4007a40).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:19.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.665+0000 7efde37fe700 1 -- 172.21.15.23:0/2834756782 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7efdf8045640 con 0x7efe040a35f0 2020-11-20T11:32:19.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.812+0000 7efe0a077700 1 -- 172.21.15.23:0/2834756782 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7efe04044710 con 0x7efe040a35f0 2020-11-20T11:32:19.816 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.812+0000 7efde37fe700 1 -- 172.21.15.23:0/2834756782 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v17) v1 ==== 92+0+10 (secure 0 0 0) 0x7efdf8048c60 con 0x7efe040a35f0 2020-11-20T11:32:19.816 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 8 2020-11-20T11:32:19.818 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.813+0000 7efe0a077700 1 -- 172.21.15.23:0/2834756782 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efddc056d60 msgr2=0x7efddc059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:19.818 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.813+0000 7efe0a077700 1 --2- 172.21.15.23:0/2834756782 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efddc056d60 0x7efddc059220 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7efdf40011e0 tx=0x7efdf4007a40).stop 2020-11-20T11:32:19.818 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.813+0000 7efe0a077700 1 -- 172.21.15.23:0/2834756782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe040a35f0 msgr2=0x7efe040a6d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:19.818 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.813+0000 7efe0a077700 1 --2- 172.21.15.23:0/2834756782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe040a35f0 0x7efe040a6d50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7efdf800db30 tx=0x7efdf8008040).stop 2020-11-20T11:32:19.818 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.814+0000 7efe0a077700 1 -- 172.21.15.23:0/2834756782 shutdown_connections 2020-11-20T11:32:19.819 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.814+0000 7efe0a077700 1 --2- 172.21.15.23:0/2834756782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe0413e870 0x7efe04149d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:19.819 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.814+0000 7efe0a077700 1 --2- 172.21.15.23:0/2834756782 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efddc056d60 0x7efddc059220 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:19.819 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.814+0000 7efe0a077700 1 --2- 172.21.15.23:0/2834756782 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe040a3210 0x7efe04066110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:19.819 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.814+0000 7efe0a077700 1 --2- 172.21.15.23:0/2834756782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe040a35f0 0x7efe040a6d50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:19.820 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.814+0000 7efe0a077700 1 -- 172.21.15.23:0/2834756782 >> 172.21.15.23:0/2834756782 conn(0x7efe040601b0 msgr2=0x7efe040ab9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:19.820 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.814+0000 7efe0a077700 1 -- 172.21.15.23:0/2834756782 shutdown_connections 2020-11-20T11:32:19.820 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.814+0000 7efe0a077700 1 -- 172.21.15.23:0/2834756782 wait complete. 2020-11-20T11:32:19.829 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2020-11-20T11:32:19.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.940+0000 7f249e58f700 1 Processor -- start 2020-11-20T11:32:19.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.940+0000 7f249e58f700 1 -- start start 2020-11-20T11:32:19.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.941+0000 7f249e58f700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f24980ad600 0x7f249813af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:19.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.941+0000 7f249e58f700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f24980a6a70 0x7f24980a5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:19.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.941+0000 7f249e58f700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24980a6f10 con 0x7f24980ad9e0 2020-11-20T11:32:19.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.941+0000 7f249e58f700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24980a71b0 con 0x7f24980ad600 2020-11-20T11:32:19.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.941+0000 7f249e58f700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24980a4da0 con 0x7f24980a6a70 2020-11-20T11:32:19.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.941+0000 7f2497fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f24980ad600 0x7f249813af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:19.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.941+0000 7f24977fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f24980ad9e0 0x7f24980a90a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:54758/0 (socket says 172.21.15.23:54758) 2020-11-20T11:32:19.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.941+0000 7f24977fe700 1 -- 172.21.15.23:0/998106223 learned_addr learned my addr 172.21.15.23:0/998106223 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:19.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.941+0000 7f249cb2c700 1 --2- 172.21.15.23:0/998106223 >> v2:172.21.15.155:3300/0 conn(0x7f24980a6a70 0x7f24980a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:19.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.942+0000 7f2497fff700 1 -- 172.21.15.23:0/998106223 >> v2:172.21.15.155:3300/0 conn(0x7f24980a6a70 msgr2=0x7f24980a5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:19.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.942+0000 7f2497fff700 1 --2- 172.21.15.23:0/998106223 >> v2:172.21.15.155:3300/0 conn(0x7f24980a6a70 0x7f24980a5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:19.949 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.942+0000 7f2497fff700 1 -- 172.21.15.23:0/998106223 >> v1:172.21.15.23:6789/0 conn(0x7f24980ad9e0 legacy=0x7f24980a90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:19.949 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.942+0000 7f2497fff700 1 -- 172.21.15.23:0/998106223 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24981352f0 con 0x7f24980ad600 2020-11-20T11:32:19.949 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.942+0000 7f2496ffd700 1 -- 172.21.15.23:0/998106223 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1350692517 0 0) 0x7f24980a6f10 con 0x7f24980ad9e0 2020-11-20T11:32:19.949 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.942+0000 7f2497fff700 1 --2- 172.21.15.23:0/998106223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24980ad600 0x7f249813af90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f24980a8fc0 tx=0x7f2488006e30).ready entity=mon.0 client_cookie=9d1d97a80f2b7cea server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:19.949 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.942+0000 7f2496ffd700 1 -- 172.21.15.23:0/998106223 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24880097b0 con 0x7f24980ad600 2020-11-20T11:32:19.950 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.942+0000 7f2496ffd700 1 -- 172.21.15.23:0/998106223 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2488000e90 con 0x7f24980ad600 2020-11-20T11:32:19.950 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.942+0000 7f2496ffd700 1 -- 172.21.15.23:0/998106223 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2488009b20 con 0x7f24980ad600 2020-11-20T11:32:19.950 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.942+0000 7f249e58f700 1 -- 172.21.15.23:0/998106223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24980ad600 msgr2=0x7f249813af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:19.950 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.942+0000 7f249e58f700 1 --2- 172.21.15.23:0/998106223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24980ad600 0x7f249813af90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f24980a8fc0 tx=0x7f2488006e30).stop 2020-11-20T11:32:19.950 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.942+0000 7f249e58f700 1 -- 172.21.15.23:0/998106223 shutdown_connections 2020-11-20T11:32:19.951 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.942+0000 7f249e58f700 1 --2- 172.21.15.23:0/998106223 >> v2:172.21.15.155:3300/0 conn(0x7f24980a6a70 0x7f24980a5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:19.951 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.942+0000 7f249e58f700 1 --2- 172.21.15.23:0/998106223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24980ad600 0x7f249813af90 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:19.951 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.942+0000 7f249e58f700 1 -- 172.21.15.23:0/998106223 >> 172.21.15.23:0/998106223 conn(0x7f2498060c80 msgr2=0x7f24980acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:19.951 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.942+0000 7f249e58f700 1 -- 172.21.15.23:0/998106223 shutdown_connections 2020-11-20T11:32:19.951 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.942+0000 7f249e58f700 1 -- 172.21.15.23:0/998106223 wait complete. 2020-11-20T11:32:19.954 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.943+0000 7f249e58f700 1 Processor -- start 2020-11-20T11:32:19.954 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.944+0000 7f249e58f700 1 -- start start 2020-11-20T11:32:19.954 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.944+0000 7f249e58f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24980a6a70 0x7f2498061a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:19.954 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.944+0000 7f249e58f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24980ad600 0x7f24980a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:19.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.944+0000 7f249e58f700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f24980ad9e0 0x7f249814de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:19.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.944+0000 7f249e58f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24980a1dc0 con 0x7f24980a6a70 2020-11-20T11:32:19.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.944+0000 7f249e58f700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24980a2060 con 0x7f24980ad600 2020-11-20T11:32:19.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.944+0000 7f249e58f700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24980626e0 con 0x7f24980ad9e0 2020-11-20T11:32:19.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.944+0000 7f2497fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24980a6a70 0x7f2498061a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:19.956 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.944+0000 7f2497fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24980a6a70 0x7f2498061a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40510/0 (socket says 172.21.15.23:40510) 2020-11-20T11:32:19.956 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.944+0000 7f2497fff700 1 -- 172.21.15.23:0/1462760105 learned_addr learned my addr 172.21.15.23:0/1462760105 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:19.956 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.944+0000 7f249cb2c700 1 --2- 172.21.15.23:0/1462760105 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f24980ad9e0 0x7f249814de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:19.956 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.944+0000 7f24977fe700 1 --2- 172.21.15.23:0/1462760105 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24980ad600 0x7f24980a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:19.956 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.944+0000 7f2497fff700 1 -- 172.21.15.23:0/1462760105 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f24980ad9e0 msgr2=0x7f249814de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:19.957 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.944+0000 7f2497fff700 1 --2- 172.21.15.23:0/1462760105 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f24980ad9e0 0x7f249814de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:19.957 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.944+0000 7f2497fff700 1 -- 172.21.15.23:0/1462760105 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24980ad600 msgr2=0x7f24980a2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:19.957 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.944+0000 7f2497fff700 1 --2- 172.21.15.23:0/1462760105 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24980ad600 0x7f24980a2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:19.957 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.944+0000 7f2497fff700 1 -- 172.21.15.23:0/1462760105 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24980a2f20 con 0x7f24980a6a70 2020-11-20T11:32:19.958 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.944+0000 7f24977fe700 1 --2- 172.21.15.23:0/1462760105 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24980ad600 0x7f24980a2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:32:19.958 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.945+0000 7f2497fff700 1 --2- 172.21.15.23:0/1462760105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24980a6a70 0x7f2498061a10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f2488006e70 tx=0x7f2488015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:19.958 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.945+0000 7f2477fff700 1 -- 172.21.15.23:0/1462760105 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2488011c40 con 0x7f24980a6a70 2020-11-20T11:32:19.958 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.945+0000 7f2477fff700 1 -- 172.21.15.23:0/1462760105 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2488011da0 con 0x7f24980a6a70 2020-11-20T11:32:19.958 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.945+0000 7f249e58f700 1 -- 172.21.15.23:0/1462760105 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f249814d6e0 con 0x7f24980a6a70 2020-11-20T11:32:19.959 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.945+0000 7f2477fff700 1 -- 172.21.15.23:0/1462760105 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248801e690 con 0x7f24980a6a70 2020-11-20T11:32:19.959 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.945+0000 7f249e58f700 1 -- 172.21.15.23:0/1462760105 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24980a1a70 con 0x7f24980a6a70 2020-11-20T11:32:19.959 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.946+0000 7f249e58f700 1 -- 172.21.15.23:0/1462760105 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f249805c520 con 0x7f24980a6a70 2020-11-20T11:32:19.959 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.946+0000 7f2477fff700 1 -- 172.21.15.23:0/1462760105 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2488025030 con 0x7f24980a6a70 2020-11-20T11:32:19.959 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.946+0000 7f2477fff700 1 --2- 172.21.15.23:0/1462760105 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2470056d60 0x7f2470059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:19.960 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.946+0000 7f2477fff700 1 -- 172.21.15.23:0/1462760105 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f24880711c0 con 0x7f24980a6a70 2020-11-20T11:32:19.960 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.947+0000 7f24977fe700 1 --2- 172.21.15.23:0/1462760105 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2470056d60 0x7f2470059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:19.960 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.947+0000 7f24977fe700 1 --2- 172.21.15.23:0/1462760105 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2470056d60 0x7f2470059220 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f2498065320 tx=0x7f248c009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:19.960 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:19.950+0000 7f2477fff700 1 -- 172.21.15.23:0/1462760105 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f24880469f0 con 0x7f24980a6a70 2020-11-20T11:32:20.103 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.099+0000 7f249e58f700 1 -- 172.21.15.23:0/1462760105 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2498138950 con 0x7f24980a6a70 2020-11-20T11:32:20.103 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.099+0000 7f2477fff700 1 -- 172.21.15.23:0/1462760105 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v18) v1 ==== 84+0+143 (secure 0 0 0) 0x7f2488023020 con 0x7f24980a6a70 2020-11-20T11:32:20.104 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:32:20.104 INFO:teuthology.orchestra.run.smithi023.stdout:[ 2020-11-20T11:32:20.104 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:20.104 INFO:teuthology.orchestra.run.smithi023.stdout: "poolnum": 1, 2020-11-20T11:32:20.105 INFO:teuthology.orchestra.run.smithi023.stdout: "poolname": "device_health_metrics" 2020-11-20T11:32:20.105 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:20.105 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:20.105 INFO:teuthology.orchestra.run.smithi023.stdout: "poolnum": 2, 2020-11-20T11:32:20.105 INFO:teuthology.orchestra.run.smithi023.stdout: "poolname": "rbd" 2020-11-20T11:32:20.106 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:32:20.106 INFO:teuthology.orchestra.run.smithi023.stdout:] 2020-11-20T11:32:20.107 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.101+0000 7f249e58f700 1 -- 172.21.15.23:0/1462760105 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2470056d60 msgr2=0x7f2470059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:20.107 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.101+0000 7f249e58f700 1 --2- 172.21.15.23:0/1462760105 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2470056d60 0x7f2470059220 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f2498065320 tx=0x7f248c009040).stop 2020-11-20T11:32:20.107 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.101+0000 7f249e58f700 1 -- 172.21.15.23:0/1462760105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24980a6a70 msgr2=0x7f2498061a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:20.107 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.101+0000 7f249e58f700 1 --2- 172.21.15.23:0/1462760105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24980a6a70 0x7f2498061a10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f2488006e70 tx=0x7f2488015000).stop 2020-11-20T11:32:20.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.101+0000 7f249e58f700 1 -- 172.21.15.23:0/1462760105 shutdown_connections 2020-11-20T11:32:20.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.101+0000 7f249e58f700 1 --2- 172.21.15.23:0/1462760105 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2470056d60 0x7f2470059220 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:20.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.101+0000 7f249e58f700 1 --2- 172.21.15.23:0/1462760105 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f24980ad9e0 0x7f249814de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:20.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.101+0000 7f249e58f700 1 --2- 172.21.15.23:0/1462760105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24980a6a70 0x7f2498061a10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:20.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.101+0000 7f249e58f700 1 --2- 172.21.15.23:0/1462760105 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24980ad600 0x7f24980a2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:20.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.101+0000 7f249e58f700 1 -- 172.21.15.23:0/1462760105 >> 172.21.15.23:0/1462760105 conn(0x7f2498060c80 msgr2=0x7f24981345a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:20.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.101+0000 7f249e58f700 1 -- 172.21.15.23:0/1462760105 shutdown_connections 2020-11-20T11:32:20.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.101+0000 7f249e58f700 1 -- 172.21.15.23:0/1462760105 wait complete. 2020-11-20T11:32:20.116 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2020-11-20T11:32:20.116 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2020-11-20T11:32:20.232 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.228+0000 7f38254b7700 1 Processor -- start 2020-11-20T11:32:20.233 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.229+0000 7f38254b7700 1 -- start start 2020-11-20T11:32:20.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.229+0000 7f38254b7700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f38200a18d0 0x7f3820134220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:20.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.229+0000 7f38254b7700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f38200a97e0 0x7f3820134710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:20.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.229+0000 7f38254b7700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3820067440 con 0x7f38200ae010 2020-11-20T11:32:20.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.229+0000 7f38254b7700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38200676e0 con 0x7f38200a97e0 2020-11-20T11:32:20.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.229+0000 7f38254b7700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3820067980 con 0x7f38200a18d0 2020-11-20T11:32:20.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.229+0000 7f381f7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f38200a97e0 0x7f3820134710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:20.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.229+0000 7f381f7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f38200a97e0 0x7f3820134710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40522/0 (socket says 172.21.15.23:40522) 2020-11-20T11:32:20.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.229+0000 7f381f7fe700 1 -- 172.21.15.23:0/2992811209 learned_addr learned my addr 172.21.15.23:0/2992811209 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:20.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.230+0000 7f381e7fc700 1 --2- 172.21.15.23:0/2992811209 >> v2:172.21.15.155:3300/0 conn(0x7f38200a18d0 0x7f3820134220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:20.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.230+0000 7f381f7fe700 1 -- 172.21.15.23:0/2992811209 >> v2:172.21.15.155:3300/0 conn(0x7f38200a18d0 msgr2=0x7f3820134220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:20.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.230+0000 7f381f7fe700 1 --2- 172.21.15.23:0/2992811209 >> v2:172.21.15.155:3300/0 conn(0x7f38200a18d0 0x7f3820134220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:20.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.230+0000 7f381f7fe700 1 -- 172.21.15.23:0/2992811209 >> v1:172.21.15.155:6789/0 conn(0x7f38200ae010 legacy=0x7f38200a90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:20.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.230+0000 7f381f7fe700 1 -- 172.21.15.23:0/2992811209 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3820067c20 con 0x7f38200a97e0 2020-11-20T11:32:20.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.230+0000 7f381f7fe700 1 --2- 172.21.15.23:0/2992811209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38200a97e0 0x7f3820134710 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f38200654b0 tx=0x7f38180092d0).ready entity=mon.0 client_cookie=e0f02e4e6d230569 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:20.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.230+0000 7f381dffb700 1 -- 172.21.15.23:0/2992811209 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381800bbf0 con 0x7f38200a97e0 2020-11-20T11:32:20.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.230+0000 7f381dffb700 1 -- 172.21.15.23:0/2992811209 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f381800bd50 con 0x7f38200a97e0 2020-11-20T11:32:20.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.230+0000 7f381dffb700 1 -- 172.21.15.23:0/2992811209 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3818011610 con 0x7f38200a97e0 2020-11-20T11:32:20.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.230+0000 7f38254b7700 1 -- 172.21.15.23:0/2992811209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38200a97e0 msgr2=0x7f3820134710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:20.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.230+0000 7f38254b7700 1 --2- 172.21.15.23:0/2992811209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38200a97e0 0x7f3820134710 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f38200654b0 tx=0x7f38180092d0).stop 2020-11-20T11:32:20.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.230+0000 7f38254b7700 1 -- 172.21.15.23:0/2992811209 shutdown_connections 2020-11-20T11:32:20.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.230+0000 7f38254b7700 1 --2- 172.21.15.23:0/2992811209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38200a97e0 0x7f3820134710 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:20.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.230+0000 7f38254b7700 1 --2- 172.21.15.23:0/2992811209 >> v2:172.21.15.155:3300/0 conn(0x7f38200a18d0 0x7f3820134220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:20.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.230+0000 7f38254b7700 1 -- 172.21.15.23:0/2992811209 >> 172.21.15.23:0/2992811209 conn(0x7f38200654e0 msgr2=0x7f3820062960 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:20.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.231+0000 7f38254b7700 1 -- 172.21.15.23:0/2992811209 shutdown_connections 2020-11-20T11:32:20.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.231+0000 7f38254b7700 1 -- 172.21.15.23:0/2992811209 wait complete. 2020-11-20T11:32:20.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.232+0000 7f38254b7700 1 Processor -- start 2020-11-20T11:32:20.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.232+0000 7f38254b7700 1 -- start start 2020-11-20T11:32:20.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.232+0000 7f38254b7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38200a18d0 0x7f3820136850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:20.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.232+0000 7f38254b7700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38200a97e0 0x7f38200a4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:20.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.232+0000 7f38254b7700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38200ae010 0x7f38200a3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:20.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.232+0000 7f38254b7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38200a32a0 con 0x7f38200a18d0 2020-11-20T11:32:20.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.232+0000 7f38254b7700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3820147bd0 con 0x7f38200a97e0 2020-11-20T11:32:20.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.232+0000 7f38254b7700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38200a4b70 con 0x7f38200ae010 2020-11-20T11:32:20.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.232+0000 7f381effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38200a18d0 0x7f3820136850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:20.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.232+0000 7f381effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38200a18d0 0x7f3820136850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40524/0 (socket says 172.21.15.23:40524) 2020-11-20T11:32:20.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.232+0000 7f381effd700 1 -- 172.21.15.23:0/669413589 learned_addr learned my addr 172.21.15.23:0/669413589 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:20.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.232+0000 7f381f7fe700 1 --2- 172.21.15.23:0/669413589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38200ae010 0x7f38200a3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:20.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.233+0000 7f381e7fc700 1 --2- 172.21.15.23:0/669413589 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38200a97e0 0x7f38200a4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:20.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.233+0000 7f381effd700 1 -- 172.21.15.23:0/669413589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38200ae010 msgr2=0x7f38200a3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:20.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.233+0000 7f381effd700 1 --2- 172.21.15.23:0/669413589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38200ae010 0x7f38200a3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:20.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.233+0000 7f381effd700 1 -- 172.21.15.23:0/669413589 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38200a97e0 msgr2=0x7f38200a4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:20.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.233+0000 7f381effd700 1 --2- 172.21.15.23:0/669413589 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38200a97e0 0x7f38200a4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:20.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.233+0000 7f381effd700 1 -- 172.21.15.23:0/669413589 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38200a4830 con 0x7f38200a18d0 2020-11-20T11:32:20.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.233+0000 7f381e7fc700 1 --2- 172.21.15.23:0/669413589 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38200a97e0 0x7f38200a4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:32:20.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.233+0000 7f381effd700 1 --2- 172.21.15.23:0/669413589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38200a18d0 0x7f3820136850 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f381400c710 tx=0x7f3814007c90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:20.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.233+0000 7f3806ffd700 1 -- 172.21.15.23:0/669413589 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3814007020 con 0x7f38200a18d0 2020-11-20T11:32:20.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.233+0000 7f3806ffd700 1 -- 172.21.15.23:0/669413589 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3814007180 con 0x7f38200a18d0 2020-11-20T11:32:20.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.233+0000 7f38254b7700 1 -- 172.21.15.23:0/669413589 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38200a4e10 con 0x7f38200a18d0 2020-11-20T11:32:20.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.233+0000 7f3806ffd700 1 -- 172.21.15.23:0/669413589 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3814006530 con 0x7f38200a18d0 2020-11-20T11:32:20.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.233+0000 7f38254b7700 1 -- 172.21.15.23:0/669413589 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f382009ecd0 con 0x7f38200a18d0 2020-11-20T11:32:20.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.234+0000 7f38254b7700 1 -- 172.21.15.23:0/669413589 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38200a8df0 con 0x7f38200a18d0 2020-11-20T11:32:20.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.234+0000 7f3806ffd700 1 -- 172.21.15.23:0/669413589 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3814010070 con 0x7f38200a18d0 2020-11-20T11:32:20.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.234+0000 7f3806ffd700 1 --2- 172.21.15.23:0/669413589 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f37fc056d60 0x7f37fc059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:20.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.234+0000 7f3806ffd700 1 -- 172.21.15.23:0/669413589 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f381406f030 con 0x7f38200a18d0 2020-11-20T11:32:20.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.235+0000 7f381e7fc700 1 --2- 172.21.15.23:0/669413589 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f37fc056d60 0x7f37fc059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:20.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.235+0000 7f381e7fc700 1 --2- 172.21.15.23:0/669413589 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f37fc056d60 0x7f37fc059220 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f38200a3190 tx=0x7f3810008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:20.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.238+0000 7f3806ffd700 1 -- 172.21.15.23:0/669413589 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f38140446a0 con 0x7f38200a18d0 2020-11-20T11:32:20.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:20.386+0000 7f38254b7700 1 -- 172.21.15.23:0/669413589 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f3820134170 con 0x7f38200a18d0 2020-11-20T11:32:21.073 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.070+0000 7f3806ffd700 1 -- 172.21.15.23:0/669413589 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v19) v1 ==== 133+0+0 (secure 0 0 0) 0x7f3814047cc0 con 0x7f38200a18d0 2020-11-20T11:32:21.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.072+0000 7f38254b7700 1 -- 172.21.15.23:0/669413589 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f37fc056d60 msgr2=0x7f37fc059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:21.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.072+0000 7f38254b7700 1 --2- 172.21.15.23:0/669413589 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f37fc056d60 0x7f37fc059220 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f38200a3190 tx=0x7f3810008040).stop 2020-11-20T11:32:21.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.072+0000 7f38254b7700 1 -- 172.21.15.23:0/669413589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38200a18d0 msgr2=0x7f3820136850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:21.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.072+0000 7f38254b7700 1 --2- 172.21.15.23:0/669413589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38200a18d0 0x7f3820136850 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f381400c710 tx=0x7f3814007c90).stop 2020-11-20T11:32:21.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.073+0000 7f38254b7700 1 -- 172.21.15.23:0/669413589 shutdown_connections 2020-11-20T11:32:21.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.073+0000 7f38254b7700 1 --2- 172.21.15.23:0/669413589 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f37fc056d60 0x7f37fc059220 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:21.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.073+0000 7f38254b7700 1 --2- 172.21.15.23:0/669413589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38200ae010 0x7f38200a3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:21.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.073+0000 7f38254b7700 1 --2- 172.21.15.23:0/669413589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38200a18d0 0x7f3820136850 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:21.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.073+0000 7f38254b7700 1 --2- 172.21.15.23:0/669413589 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38200a97e0 0x7f38200a4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:21.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.073+0000 7f38254b7700 1 -- 172.21.15.23:0/669413589 >> 172.21.15.23:0/669413589 conn(0x7f38200654e0 msgr2=0x7f3820061a10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:21.079 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.074+0000 7f38254b7700 1 -- 172.21.15.23:0/669413589 shutdown_connections 2020-11-20T11:32:21.079 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.074+0000 7f38254b7700 1 -- 172.21.15.23:0/669413589 wait complete. 2020-11-20T11:32:21.079 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_metadata' created 2020-11-20T11:32:21.089 INFO:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2020-11-20T11:32:21.090 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data m=2 k=2 crush-failure-domain=osd 2020-11-20T11:32:21.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.205+0000 7f68b54a0700 1 Processor -- start 2020-11-20T11:32:21.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.205+0000 7f68b54a0700 1 -- start start 2020-11-20T11:32:21.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.206+0000 7f68b54a0700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f68b00b1b70 0x7f68b00ab170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:21.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.206+0000 7f68b54a0700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b00a9570 con 0x7f68b00ad5b0 2020-11-20T11:32:21.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.206+0000 7f68b54a0700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b00aaad0 con 0x7f68b00ac350 2020-11-20T11:32:21.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.206+0000 7f68b54a0700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68b00a97d0 con 0x7f68b00b1b70 2020-11-20T11:32:21.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.206+0000 7f68aeffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f68b00b1b70 0x7f68b00ab170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:21.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.206+0000 7f68ae7fc700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f68b00ad5b0 0x7f68b00abc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:54782/0 (socket says 172.21.15.23:54782) 2020-11-20T11:32:21.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.206+0000 7f68ae7fc700 1 -- 172.21.15.23:0/3240159909 learned_addr learned my addr 172.21.15.23:0/3240159909 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:21.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.207+0000 7f68aeffd700 1 -- 172.21.15.23:0/3240159909 >> v1:172.21.15.155:6789/0 conn(0x7f68b00ac350 legacy=0x7f68b01458a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:21.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.207+0000 7f68aeffd700 1 -- 172.21.15.23:0/3240159909 >> v1:172.21.15.23:6789/0 conn(0x7f68b00ad5b0 legacy=0x7f68b00abc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:21.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.207+0000 7f68aeffd700 1 -- 172.21.15.23:0/3240159909 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b00a9a70 con 0x7f68b00b1b70 2020-11-20T11:32:21.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.207+0000 7f68adffb700 1 -- 172.21.15.23:0/3240159909 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637272593 0 0) 0x7f68b00a9570 con 0x7f68b00ad5b0 2020-11-20T11:32:21.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.207+0000 7f68aeffd700 1 --2- 172.21.15.23:0/3240159909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b00b1b70 0x7f68b00ab170 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f68a000b9d0 tx=0x7f68a0006dd0).ready entity=mon.0 client_cookie=3ffac6db524092e0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:21.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.207+0000 7f68adffb700 1 -- 172.21.15.23:0/3240159909 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a0006030 con 0x7f68b00b1b70 2020-11-20T11:32:21.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.207+0000 7f68adffb700 1 -- 172.21.15.23:0/3240159909 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68a0006190 con 0x7f68b00b1b70 2020-11-20T11:32:21.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.207+0000 7f68adffb700 1 -- 172.21.15.23:0/3240159909 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a0006510 con 0x7f68b00b1b70 2020-11-20T11:32:21.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.207+0000 7f68b54a0700 1 -- 172.21.15.23:0/3240159909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b00b1b70 msgr2=0x7f68b00ab170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:21.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.207+0000 7f68b54a0700 1 --2- 172.21.15.23:0/3240159909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b00b1b70 0x7f68b00ab170 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f68a000b9d0 tx=0x7f68a0006dd0).stop 2020-11-20T11:32:21.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.207+0000 7f68b54a0700 1 -- 172.21.15.23:0/3240159909 shutdown_connections 2020-11-20T11:32:21.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.207+0000 7f68b54a0700 1 --2- 172.21.15.23:0/3240159909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b00b1b70 0x7f68b00ab170 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:21.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.207+0000 7f68b54a0700 1 -- 172.21.15.23:0/3240159909 >> 172.21.15.23:0/3240159909 conn(0x7f68b00683b0 msgr2=0x7f68b00b12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:21.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.207+0000 7f68b54a0700 1 -- 172.21.15.23:0/3240159909 shutdown_connections 2020-11-20T11:32:21.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.207+0000 7f68b54a0700 1 -- 172.21.15.23:0/3240159909 wait complete. 2020-11-20T11:32:21.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.208+0000 7f68b54a0700 1 Processor -- start 2020-11-20T11:32:21.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.209+0000 7f68b54a0700 1 -- start start 2020-11-20T11:32:21.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.209+0000 7f68b54a0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b00ac350 0x7f68b00b05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:21.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.209+0000 7f68b54a0700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68b00ad5b0 0x7f68b013d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:21.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.209+0000 7f68b54a0700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68b00b1b70 0x7f68b013d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:21.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.209+0000 7f68b54a0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b013c820 con 0x7f68b00ac350 2020-11-20T11:32:21.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.209+0000 7f68b54a0700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b013cac0 con 0x7f68b00b1b70 2020-11-20T11:32:21.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.209+0000 7f68b54a0700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b013cd60 con 0x7f68b00ad5b0 2020-11-20T11:32:21.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.209+0000 7f68aeffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b00ac350 0x7f68b00b05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:21.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.209+0000 7f68aeffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b00ac350 0x7f68b00b05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40536/0 (socket says 172.21.15.23:40536) 2020-11-20T11:32:21.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.209+0000 7f68aeffd700 1 -- 172.21.15.23:0/2058431721 learned_addr learned my addr 172.21.15.23:0/2058431721 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:21.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.209+0000 7f68ae7fc700 1 --2- 172.21.15.23:0/2058431721 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68b00ad5b0 0x7f68b013d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:21.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.209+0000 7f68af7fe700 1 --2- 172.21.15.23:0/2058431721 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68b00b1b70 0x7f68b013d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:21.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.209+0000 7f68aeffd700 1 -- 172.21.15.23:0/2058431721 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68b00ad5b0 msgr2=0x7f68b013d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:21.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.209+0000 7f68aeffd700 1 --2- 172.21.15.23:0/2058431721 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68b00ad5b0 0x7f68b013d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:21.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.209+0000 7f68aeffd700 1 -- 172.21.15.23:0/2058431721 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68b00b1b70 msgr2=0x7f68b013d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:21.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.209+0000 7f68aeffd700 1 --2- 172.21.15.23:0/2058431721 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68b00b1b70 0x7f68b013d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:21.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.209+0000 7f68aeffd700 1 -- 172.21.15.23:0/2058431721 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b013c510 con 0x7f68b00ac350 2020-11-20T11:32:21.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.209+0000 7f68ae7fc700 1 --2- 172.21.15.23:0/2058431721 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68b00ad5b0 0x7f68b013d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:32:21.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.209+0000 7f68af7fe700 1 --2- 172.21.15.23:0/2058431721 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68b00b1b70 0x7f68b013d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:32:21.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.210+0000 7f68aeffd700 1 --2- 172.21.15.23:0/2058431721 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b00ac350 0x7f68b00b05b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f68a0006fd0 tx=0x7f68a0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:21.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.210+0000 7f6896ffd700 1 -- 172.21.15.23:0/2058431721 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a0005b70 con 0x7f68b00ac350 2020-11-20T11:32:21.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.210+0000 7f6896ffd700 1 -- 172.21.15.23:0/2058431721 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68a0005cd0 con 0x7f68b00ac350 2020-11-20T11:32:21.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.210+0000 7f68b54a0700 1 -- 172.21.15.23:0/2058431721 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68b013c0f0 con 0x7f68b00ac350 2020-11-20T11:32:21.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.210+0000 7f6896ffd700 1 -- 172.21.15.23:0/2058431721 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a001e670 con 0x7f68b00ac350 2020-11-20T11:32:21.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.210+0000 7f68b54a0700 1 -- 172.21.15.23:0/2058431721 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b013f870 con 0x7f68b00ac350 2020-11-20T11:32:21.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.211+0000 7f68b54a0700 1 -- 172.21.15.23:0/2058431721 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68b0060640 con 0x7f68b00ac350 2020-11-20T11:32:21.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.211+0000 7f6896ffd700 1 -- 172.21.15.23:0/2058431721 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f68a0025030 con 0x7f68b00ac350 2020-11-20T11:32:21.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.211+0000 7f6896ffd700 1 --2- 172.21.15.23:0/2058431721 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f688c056d60 0x7f688c059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:21.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.211+0000 7f6896ffd700 1 -- 172.21.15.23:0/2058431721 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f68a0071550 con 0x7f68b00ac350 2020-11-20T11:32:21.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.212+0000 7f68ae7fc700 1 --2- 172.21.15.23:0/2058431721 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f688c056d60 0x7f688c059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:21.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.212+0000 7f68ae7fc700 1 --2- 172.21.15.23:0/2058431721 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f688c056d60 0x7f688c059220 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f68b013a710 tx=0x7f68a4009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:21.225 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.215+0000 7f6896ffd700 1 -- 172.21.15.23:0/2058431721 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f68a0046bc0 con 0x7f68b00ac350 2020-11-20T11:32:21.379 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:21.375+0000 7f68b54a0700 1 -- 172.21.15.23:0/2058431721 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f68b013d210 con 0x7f68b00ac350 2020-11-20T11:32:22.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.081+0000 7f6896ffd700 1 -- 172.21.15.23:0/2058431721 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v20) v1 ==== 154+0+0 (secure 0 0 0) 0x7f68a0023020 con 0x7f68b00ac350 2020-11-20T11:32:22.089 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.085+0000 7f68b54a0700 1 -- 172.21.15.23:0/2058431721 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f688c056d60 msgr2=0x7f688c059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:22.089 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.085+0000 7f68b54a0700 1 --2- 172.21.15.23:0/2058431721 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f688c056d60 0x7f688c059220 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f68b013a710 tx=0x7f68a4009040).stop 2020-11-20T11:32:22.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.085+0000 7f68b54a0700 1 -- 172.21.15.23:0/2058431721 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b00ac350 msgr2=0x7f68b00b05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:22.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.085+0000 7f68b54a0700 1 --2- 172.21.15.23:0/2058431721 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b00ac350 0x7f68b00b05b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f68a0006fd0 tx=0x7f68a0015000).stop 2020-11-20T11:32:22.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.085+0000 7f68b54a0700 1 -- 172.21.15.23:0/2058431721 shutdown_connections 2020-11-20T11:32:22.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.085+0000 7f68b54a0700 1 --2- 172.21.15.23:0/2058431721 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68b00b1b70 0x7f68b013d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:22.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.085+0000 7f68b54a0700 1 --2- 172.21.15.23:0/2058431721 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b00ac350 0x7f68b00b05b0 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0x7f68a0006fd0 tx=0x7f68a0015000).stop 2020-11-20T11:32:22.091 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.085+0000 7f68b54a0700 1 --2- 172.21.15.23:0/2058431721 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f688c056d60 0x7f688c059220 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:22.091 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.085+0000 7f68b54a0700 1 --2- 172.21.15.23:0/2058431721 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68b00ad5b0 0x7f68b013d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:22.091 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.085+0000 7f68b54a0700 1 -- 172.21.15.23:0/2058431721 >> 172.21.15.23:0/2058431721 conn(0x7f68b00683b0 msgr2=0x7f68b0068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:22.092 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.087+0000 7f68b54a0700 1 -- 172.21.15.23:0/2058431721 shutdown_connections 2020-11-20T11:32:22.092 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.087+0000 7f68b54a0700 1 -- 172.21.15.23:0/2058431721 wait complete. 2020-11-20T11:32:22.107 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 erasure cephfs_data 2020-11-20T11:32:22.232 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.229+0000 7f696466c700 1 Processor -- start 2020-11-20T11:32:22.233 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.229+0000 7f696466c700 1 -- start start 2020-11-20T11:32:22.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.229+0000 7f696466c700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f695c0b1b70 0x7f695c0ab170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:22.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.229+0000 7f696466c700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f695c0a9570 con 0x7f695c0ac350 2020-11-20T11:32:22.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.229+0000 7f696466c700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f695c0aaad0 con 0x7f695c0ad5b0 2020-11-20T11:32:22.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.229+0000 7f696466c700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f695c0a97d0 con 0x7f695c0b1b70 2020-11-20T11:32:22.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.230+0000 7f6962c09700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f695c0ac350 0x7f695c1458a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:54798/0 (socket says 172.21.15.23:54798) 2020-11-20T11:32:22.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.230+0000 7f6962c09700 1 -- 172.21.15.23:0/244304524 learned_addr learned my addr 172.21.15.23:0/244304524 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:22.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.230+0000 7f6962408700 1 --2- 172.21.15.23:0/244304524 >> v2:172.21.15.155:3300/0 conn(0x7f695c0b1b70 0x7f695c0ab170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:22.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.230+0000 7f6961406700 1 -- 172.21.15.23:0/244304524 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524837010 0 0) 0x7f695c0a9570 con 0x7f695c0ac350 2020-11-20T11:32:22.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.230+0000 7f6961406700 1 -- 172.21.15.23:0/244304524 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6944003280 con 0x7f695c0ac350 2020-11-20T11:32:22.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.230+0000 7f6961406700 1 -- 172.21.15.23:0/244304524 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4173927574 0 0) 0x7f6944003280 con 0x7f695c0ac350 2020-11-20T11:32:22.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.230+0000 7f6961406700 1 -- 172.21.15.23:0/244304524 >> v2:172.21.15.155:3300/0 conn(0x7f695c0b1b70 msgr2=0x7f695c0ab170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:22.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.230+0000 7f6961406700 1 --2- 172.21.15.23:0/244304524 >> v2:172.21.15.155:3300/0 conn(0x7f695c0b1b70 0x7f695c0ab170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:22.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.230+0000 7f6961406700 1 -- 172.21.15.23:0/244304524 >> v1:172.21.15.155:6789/0 conn(0x7f695c0ad5b0 legacy=0x7f695c0abc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:22.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.230+0000 7f6961406700 1 -- 172.21.15.23:0/244304524 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f695c0a9a70 con 0x7f695c0ac350 2020-11-20T11:32:22.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.230+0000 7f6961406700 1 -- 172.21.15.23:0/244304524 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f69540018a0 con 0x7f695c0ac350 2020-11-20T11:32:22.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.230+0000 7f6962408700 1 --2- 172.21.15.23:0/244304524 >> v2:172.21.15.155:3300/0 conn(0x7f695c0b1b70 0x7f695c0ab170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:32:22.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.230+0000 7f6961406700 1 -- 172.21.15.23:0/244304524 >> v1:172.21.15.23:6789/0 conn(0x7f695c0ac350 legacy=0x7f695c1458a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:22.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.230+0000 7f6961406700 1 --2- 172.21.15.23:0/244304524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6944003770 0x7f6944003b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:22.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.230+0000 7f6961406700 1 --2- 172.21.15.23:0/244304524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6944004520 0x7f69440049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:22.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.230+0000 7f6961406700 1 --2- 172.21.15.23:0/244304524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6944004e90 0x7f694400b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:22.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.230+0000 7f6961406700 1 -- 172.21.15.23:0/244304524 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c0a97d0 con 0x7f6944004520 2020-11-20T11:32:22.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.230+0000 7f6961406700 1 -- 172.21.15.23:0/244304524 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69440042a0 con 0x7f6944004e90 2020-11-20T11:32:22.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.230+0000 7f6961406700 1 -- 172.21.15.23:0/244304524 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f694400bc60 con 0x7f6944003770 2020-11-20T11:32:22.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.230+0000 7f6961c07700 1 --2- 172.21.15.23:0/244304524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6944004520 0x7f69440049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:22.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.230+0000 7f6962408700 1 --2- 172.21.15.23:0/244304524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6944003770 0x7f6944003b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:22.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.231+0000 7f6962c09700 1 --2- 172.21.15.23:0/244304524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6944004e90 0x7f694400b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:22.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.231+0000 7f6961c07700 1 -- 172.21.15.23:0/244304524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6944003770 msgr2=0x7f6944003b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:22.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.231+0000 7f6961c07700 1 --2- 172.21.15.23:0/244304524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6944003770 0x7f6944003b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:22.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.231+0000 7f6961c07700 1 -- 172.21.15.23:0/244304524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6944004e90 msgr2=0x7f694400b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:22.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.231+0000 7f6961c07700 1 --2- 172.21.15.23:0/244304524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6944004e90 0x7f694400b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:22.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.231+0000 7f6962c09700 1 -- 172.21.15.23:0/244304524 reap_dead start 2020-11-20T11:32:22.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.231+0000 7f6961c07700 1 -- 172.21.15.23:0/244304524 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6944003280 con 0x7f6944004520 2020-11-20T11:32:22.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.231+0000 7f6961c07700 1 --2- 172.21.15.23:0/244304524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6944004520 0x7f69440049a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f6958001930 tx=0x7f6958007bd0).ready entity=mon.0 client_cookie=c12a33e1eec2ba4f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:22.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.231+0000 7f6961406700 1 -- 172.21.15.23:0/244304524 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6958007020 con 0x7f6944004520 2020-11-20T11:32:22.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.231+0000 7f6961406700 1 -- 172.21.15.23:0/244304524 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6958007180 con 0x7f6944004520 2020-11-20T11:32:22.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.231+0000 7f6961406700 1 -- 172.21.15.23:0/244304524 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695800c860 con 0x7f6944004520 2020-11-20T11:32:22.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.231+0000 7f696466c700 1 -- 172.21.15.23:0/244304524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6944004520 msgr2=0x7f69440049a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:22.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.231+0000 7f696466c700 1 --2- 172.21.15.23:0/244304524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6944004520 0x7f69440049a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f6958001930 tx=0x7f6958007bd0).stop 2020-11-20T11:32:22.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.231+0000 7f696466c700 1 -- 172.21.15.23:0/244304524 shutdown_connections 2020-11-20T11:32:22.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.231+0000 7f696466c700 1 --2- 172.21.15.23:0/244304524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6944004520 0x7f69440049a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:22.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.231+0000 7f696466c700 1 -- 172.21.15.23:0/244304524 >> 172.21.15.23:0/244304524 conn(0x7f695c0683b0 msgr2=0x7f695c0b12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:22.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.232+0000 7f696466c700 1 -- 172.21.15.23:0/244304524 shutdown_connections 2020-11-20T11:32:22.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.232+0000 7f696466c700 1 -- 172.21.15.23:0/244304524 wait complete. 2020-11-20T11:32:22.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.232+0000 7f696466c700 1 Processor -- start 2020-11-20T11:32:22.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.232+0000 7f696466c700 1 -- start start 2020-11-20T11:32:22.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.233+0000 7f696466c700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6944004520 0x7f695c138d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:22.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.233+0000 7f696466c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f695c13d8c0 0x7f695c13c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:22.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.233+0000 7f696466c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f695c13c8b0 0x7f695c0af0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:22.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.233+0000 7f696466c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c13cd30 con 0x7f695c13c8b0 2020-11-20T11:32:22.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.233+0000 7f696466c700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c09d620 con 0x7f695c13d8c0 2020-11-20T11:32:22.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.233+0000 7f696466c700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f695c09d8c0 con 0x7f6944004520 2020-11-20T11:32:22.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.233+0000 7f6962c09700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f695c13c8b0 0x7f695c0af0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:22.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.233+0000 7f6962c09700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f695c13c8b0 0x7f695c0af0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40560/0 (socket says 172.21.15.23:40560) 2020-11-20T11:32:22.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.233+0000 7f6962c09700 1 -- 172.21.15.23:0/2130840055 learned_addr learned my addr 172.21.15.23:0/2130840055 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:22.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.233+0000 7f6961c07700 1 --2- 172.21.15.23:0/2130840055 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f695c13d8c0 0x7f695c13c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:22.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.233+0000 7f6962408700 1 --2- 172.21.15.23:0/2130840055 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6944004520 0x7f695c138d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:22.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.233+0000 7f6962c09700 1 -- 172.21.15.23:0/2130840055 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6944004520 msgr2=0x7f695c138d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:22.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.234+0000 7f6962c09700 1 --2- 172.21.15.23:0/2130840055 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6944004520 0x7f695c138d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:22.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.234+0000 7f6962c09700 1 -- 172.21.15.23:0/2130840055 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f695c13d8c0 msgr2=0x7f695c13c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:22.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.234+0000 7f6962c09700 1 --2- 172.21.15.23:0/2130840055 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f695c13d8c0 0x7f695c13c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:22.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.234+0000 7f6962c09700 1 -- 172.21.15.23:0/2130840055 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f695c09db60 con 0x7f695c13c8b0 2020-11-20T11:32:22.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.234+0000 7f6962408700 1 --2- 172.21.15.23:0/2130840055 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6944004520 0x7f695c138d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:22.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.234+0000 7f6962c09700 1 --2- 172.21.15.23:0/2130840055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f695c13c8b0 0x7f695c0af0b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f695c062fe0 tx=0x7f695c0aaf00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:22.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.234+0000 7f694a7fc700 1 -- 172.21.15.23:0/2130840055 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695400fca0 con 0x7f695c13c8b0 2020-11-20T11:32:22.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.234+0000 7f694a7fc700 1 -- 172.21.15.23:0/2130840055 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f695400fe00 con 0x7f695c13c8b0 2020-11-20T11:32:22.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.234+0000 7f696466c700 1 -- 172.21.15.23:0/2130840055 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f695c0b04a0 con 0x7f695c13c8b0 2020-11-20T11:32:22.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.234+0000 7f694a7fc700 1 -- 172.21.15.23:0/2130840055 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6954012580 con 0x7f695c13c8b0 2020-11-20T11:32:22.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.234+0000 7f696466c700 1 -- 172.21.15.23:0/2130840055 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f695c13df00 con 0x7f695c13c8b0 2020-11-20T11:32:22.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.235+0000 7f696466c700 1 -- 172.21.15.23:0/2130840055 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f695c060640 con 0x7f695c13c8b0 2020-11-20T11:32:22.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.235+0000 7f694a7fc700 1 -- 172.21.15.23:0/2130840055 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f695401b030 con 0x7f695c13c8b0 2020-11-20T11:32:22.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.235+0000 7f694a7fc700 1 --2- 172.21.15.23:0/2130840055 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6934056d60 0x7f6934059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:22.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.235+0000 7f694a7fc700 1 -- 172.21.15.23:0/2130840055 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5863+0+0 (secure 0 0 0) 0x7f695406f680 con 0x7f695c13c8b0 2020-11-20T11:32:22.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.236+0000 7f6962408700 1 --2- 172.21.15.23:0/2130840055 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6934056d60 0x7f6934059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:22.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.236+0000 7f6962408700 1 --2- 172.21.15.23:0/2130840055 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6934056d60 0x7f6934059220 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f694c006fd0 tx=0x7f694c008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:22.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.239+0000 7f694a7fc700 1 -- 172.21.15.23:0/2130840055 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6954044ba0 con 0x7f695c13c8b0 2020-11-20T11:32:22.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:22.389+0000 7f696466c700 1 -- 172.21.15.23:0/2130840055 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8, "pool_type": "erasure", "erasure_code_profile": "cephfs_data"} v 0) v1 -- 0x7f695c044710 con 0x7f695c13c8b0 2020-11-20T11:32:24.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.097+0000 7f694a7fc700 1 -- 172.21.15.23:0/2130840055 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8, "pool_type": "erasure", "erasure_code_profile": "cephfs_data"}]=0 pool 'cephfs_data' created v22) v1 ==== 188+0+0 (secure 0 0 0) 0x7f6954019020 con 0x7f695c13c8b0 2020-11-20T11:32:24.104 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.100+0000 7f696466c700 1 -- 172.21.15.23:0/2130840055 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6934056d60 msgr2=0x7f6934059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:24.104 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.100+0000 7f696466c700 1 --2- 172.21.15.23:0/2130840055 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6934056d60 0x7f6934059220 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f694c006fd0 tx=0x7f694c008040).stop 2020-11-20T11:32:24.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.100+0000 7f696466c700 1 -- 172.21.15.23:0/2130840055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f695c13c8b0 msgr2=0x7f695c0af0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:24.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.100+0000 7f696466c700 1 --2- 172.21.15.23:0/2130840055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f695c13c8b0 0x7f695c0af0b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f695c062fe0 tx=0x7f695c0aaf00).stop 2020-11-20T11:32:24.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.100+0000 7f696466c700 1 -- 172.21.15.23:0/2130840055 shutdown_connections 2020-11-20T11:32:24.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.100+0000 7f696466c700 1 --2- 172.21.15.23:0/2130840055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f695c13c8b0 0x7f695c0af0b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:24.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.100+0000 7f696466c700 1 --2- 172.21.15.23:0/2130840055 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6934056d60 0x7f6934059220 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:24.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.100+0000 7f696466c700 1 --2- 172.21.15.23:0/2130840055 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6944004520 0x7f695c138d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:24.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.100+0000 7f696466c700 1 --2- 172.21.15.23:0/2130840055 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f695c13d8c0 0x7f695c13c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:24.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.100+0000 7f696466c700 1 -- 172.21.15.23:0/2130840055 >> 172.21.15.23:0/2130840055 conn(0x7f695c0683b0 msgr2=0x7f695c0ac7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:24.107 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.101+0000 7f696466c700 1 -- 172.21.15.23:0/2130840055 shutdown_connections 2020-11-20T11:32:24.107 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.101+0000 7f696466c700 1 -- 172.21.15.23:0/2130840055 wait complete. 2020-11-20T11:32:24.108 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_data' created 2020-11-20T11:32:24.122 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data allow_ec_overwrites true 2020-11-20T11:32:24.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.243+0000 7fa4ca148700 1 Processor -- start 2020-11-20T11:32:24.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.244+0000 7fa4ca148700 1 -- start start 2020-11-20T11:32:24.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.244+0000 7fa4ca148700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa4c40af190 0x7fa4c40aa7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:24.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.244+0000 7fa4ca148700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4c4137a00 0x7fa4c4137e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:24.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.244+0000 7fa4ca148700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4c40a2000 con 0x7fa4c40aace0 2020-11-20T11:32:24.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.244+0000 7fa4ca148700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4c40aa3c0 con 0x7fa4c40af190 2020-11-20T11:32:24.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.244+0000 7fa4ca148700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c40a2230 con 0x7fa4c4137a00 2020-11-20T11:32:24.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.245+0000 7fa4c37fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa4c40af190 0x7fa4c40aa7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:24.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.245+0000 7fa4c3fff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4c4137a00 0x7fa4c4137e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:24.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.245+0000 7fa4c37fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa4c40af190 0x7fa4c40aa7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:33460/0 (socket says 172.21.15.23:33460) 2020-11-20T11:32:24.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.245+0000 7fa4c2ffd700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fa4c40aace0 0x7fa4c4067580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:32790/0 (socket says 172.21.15.23:32790) 2020-11-20T11:32:24.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.245+0000 7fa4c37fe700 1 -- 172.21.15.23:0/3892960265 learned_addr learned my addr 172.21.15.23:0/3892960265 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:24.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.245+0000 7fa4c3fff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4c4137a00 0x7fa4c4137e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:50808/0 (socket says 172.21.15.23:50808) 2020-11-20T11:32:24.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.245+0000 7fa4c3fff700 1 -- 172.21.15.23:0/3892960265 >> v2:172.21.15.155:3300/0 conn(0x7fa4c40af190 msgr2=0x7fa4c40aa7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:24.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.245+0000 7fa4c3fff700 1 --2- 172.21.15.23:0/3892960265 >> v2:172.21.15.155:3300/0 conn(0x7fa4c40af190 0x7fa4c40aa7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:24.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.245+0000 7fa4c3fff700 1 -- 172.21.15.23:0/3892960265 >> v1:172.21.15.155:6789/0 conn(0x7fa4c40aace0 legacy=0x7fa4c4067580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:24.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.245+0000 7fa4c3fff700 1 -- 172.21.15.23:0/3892960265 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c40a24d0 con 0x7fa4c4137a00 2020-11-20T11:32:24.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.245+0000 7fa4c27fc700 1 -- 172.21.15.23:0/3892960265 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020769927 0 0) 0x7fa4c40a2000 con 0x7fa4c40aace0 2020-11-20T11:32:24.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.245+0000 7fa4c37fe700 1 --2- 172.21.15.23:0/3892960265 >> v2:172.21.15.155:3300/0 conn(0x7fa4c40af190 0x7fa4c40aa7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:32:24.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.245+0000 7fa4c3fff700 1 --2- 172.21.15.23:0/3892960265 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4c4137a00 0x7fa4c4137e90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fa4bc009ce0 tx=0x7fa4bc009220).ready entity=mon.2 client_cookie=403260b4d60d9a1f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:24.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.245+0000 7fa4c27fc700 1 -- 172.21.15.23:0/3892960265 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc00ba40 con 0x7fa4c4137a00 2020-11-20T11:32:24.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.245+0000 7fa4c27fc700 1 -- 172.21.15.23:0/3892960265 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4bc00bba0 con 0x7fa4c4137a00 2020-11-20T11:32:24.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.245+0000 7fa4c27fc700 1 -- 172.21.15.23:0/3892960265 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc011400 con 0x7fa4c4137a00 2020-11-20T11:32:24.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.246+0000 7fa4ca148700 1 -- 172.21.15.23:0/3892960265 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4c4137a00 msgr2=0x7fa4c4137e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:24.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.246+0000 7fa4ca148700 1 --2- 172.21.15.23:0/3892960265 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4c4137a00 0x7fa4c4137e90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fa4bc009ce0 tx=0x7fa4bc009220).stop 2020-11-20T11:32:24.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.246+0000 7fa4ca148700 1 -- 172.21.15.23:0/3892960265 shutdown_connections 2020-11-20T11:32:24.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.246+0000 7fa4ca148700 1 --2- 172.21.15.23:0/3892960265 >> v2:172.21.15.155:3300/0 conn(0x7fa4c40af190 0x7fa4c40aa7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:24.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.246+0000 7fa4ca148700 1 --2- 172.21.15.23:0/3892960265 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4c4137a00 0x7fa4c4137e90 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:24.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.246+0000 7fa4ca148700 1 -- 172.21.15.23:0/3892960265 >> 172.21.15.23:0/3892960265 conn(0x7fa4c40666e0 msgr2=0x7fa4c4063d70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:24.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.246+0000 7fa4ca148700 1 -- 172.21.15.23:0/3892960265 shutdown_connections 2020-11-20T11:32:24.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.246+0000 7fa4ca148700 1 -- 172.21.15.23:0/3892960265 wait complete. 2020-11-20T11:32:24.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.247+0000 7fa4ca148700 1 Processor -- start 2020-11-20T11:32:24.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.247+0000 7fa4ca148700 1 -- start start 2020-11-20T11:32:24.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.247+0000 7fa4ca148700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4c40af190 0x7fa4c4149d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:24.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.247+0000 7fa4ca148700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4c4137a00 0x7fa4c40abab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:24.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.247+0000 7fa4ca148700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4c413ec30 0x7fa4c414d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:24.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.247+0000 7fa4ca148700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c413d990 con 0x7fa4c413ec30 2020-11-20T11:32:24.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.247+0000 7fa4ca148700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c414a380 con 0x7fa4c4137a00 2020-11-20T11:32:24.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.247+0000 7fa4ca148700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c414a620 con 0x7fa4c40af190 2020-11-20T11:32:24.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.248+0000 7fa4c3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4c413ec30 0x7fa4c414d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:24.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.248+0000 7fa4c2ffd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4c4137a00 0x7fa4c40abab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:24.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.248+0000 7fa4c3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4c413ec30 0x7fa4c414d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40574/0 (socket says 172.21.15.23:40574) 2020-11-20T11:32:24.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.248+0000 7fa4c3fff700 1 -- 172.21.15.23:0/607618383 learned_addr learned my addr 172.21.15.23:0/607618383 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:24.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.248+0000 7fa4c37fe700 1 --2- 172.21.15.23:0/607618383 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4c40af190 0x7fa4c4149d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:24.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.248+0000 7fa4c3fff700 1 -- 172.21.15.23:0/607618383 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4c40af190 msgr2=0x7fa4c4149d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:24.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.248+0000 7fa4c3fff700 1 --2- 172.21.15.23:0/607618383 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4c40af190 0x7fa4c4149d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:24.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.248+0000 7fa4c3fff700 1 -- 172.21.15.23:0/607618383 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4c4137a00 msgr2=0x7fa4c40abab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:24.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.248+0000 7fa4c3fff700 1 --2- 172.21.15.23:0/607618383 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4c4137a00 0x7fa4c40abab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:24.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.248+0000 7fa4c3fff700 1 -- 172.21.15.23:0/607618383 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c414a8c0 con 0x7fa4c413ec30 2020-11-20T11:32:24.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.248+0000 7fa4c3fff700 1 --2- 172.21.15.23:0/607618383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4c413ec30 0x7fa4c414d2a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fa4bc0090b0 tx=0x7fa4bc016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:24.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.248+0000 7fa4ab7fe700 1 -- 172.21.15.23:0/607618383 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc00b4a0 con 0x7fa4c413ec30 2020-11-20T11:32:24.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.248+0000 7fa4ab7fe700 1 -- 172.21.15.23:0/607618383 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4bc010a50 con 0x7fa4c413ec30 2020-11-20T11:32:24.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.248+0000 7fa4ca148700 1 -- 172.21.15.23:0/607618383 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4c4138dd0 con 0x7fa4c413ec30 2020-11-20T11:32:24.266 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.248+0000 7fa4ab7fe700 1 -- 172.21.15.23:0/607618383 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc010dd0 con 0x7fa4c413ec30 2020-11-20T11:32:24.266 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.248+0000 7fa4ca148700 1 -- 172.21.15.23:0/607618383 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4c414d950 con 0x7fa4c413ec30 2020-11-20T11:32:24.266 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.249+0000 7fa4ca148700 1 -- 172.21.15.23:0/607618383 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4c414aa50 con 0x7fa4c413ec30 2020-11-20T11:32:24.266 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.250+0000 7fa4ab7fe700 1 -- 172.21.15.23:0/607618383 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa4bc02c070 con 0x7fa4c413ec30 2020-11-20T11:32:24.266 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.250+0000 7fa4ab7fe700 1 --2- 172.21.15.23:0/607618383 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa4a0056d90 0x7fa4a0059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:24.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.250+0000 7fa4ab7fe700 1 -- 172.21.15.23:0/607618383 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6315+0+0 (secure 0 0 0) 0x7fa4bc0722a0 con 0x7fa4c413ec30 2020-11-20T11:32:24.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.250+0000 7fa4c37fe700 1 --2- 172.21.15.23:0/607618383 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa4a0056d90 0x7fa4a0059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:24.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.251+0000 7fa4c37fe700 1 --2- 172.21.15.23:0/607618383 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa4a0056d90 0x7fa4a0059250 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fa4b4006a30 tx=0x7fa4b4006ef0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:24.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.254+0000 7fa4ab7fe700 1 -- 172.21.15.23:0/607618383 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa4bc047600 con 0x7fa4c413ec30 2020-11-20T11:32:24.409 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:24.405+0000 7fa4ca148700 1 -- 172.21.15.23:0/607618383 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7fa4c4066af0 con 0x7fa4c413ec30 2020-11-20T11:32:25.118 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.115+0000 7fa4ab7fe700 1 -- 172.21.15.23:0/607618383 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v23) v1 ==== 166+0+0 (secure 0 0 0) 0x7fa4bc02a0c0 con 0x7fa4c413ec30 2020-11-20T11:32:25.125 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.121+0000 7fa4ca148700 1 -- 172.21.15.23:0/607618383 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa4a0056d90 msgr2=0x7fa4a0059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:25.125 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.121+0000 7fa4ca148700 1 --2- 172.21.15.23:0/607618383 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa4a0056d90 0x7fa4a0059250 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fa4b4006a30 tx=0x7fa4b4006ef0).stop 2020-11-20T11:32:25.125 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.121+0000 7fa4ca148700 1 -- 172.21.15.23:0/607618383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4c413ec30 msgr2=0x7fa4c414d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:25.125 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.121+0000 7fa4ca148700 1 --2- 172.21.15.23:0/607618383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4c413ec30 0x7fa4c414d2a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fa4bc0090b0 tx=0x7fa4bc016000).stop 2020-11-20T11:32:25.127 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.123+0000 7fa4ca148700 1 -- 172.21.15.23:0/607618383 shutdown_connections 2020-11-20T11:32:25.127 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.123+0000 7fa4ca148700 1 --2- 172.21.15.23:0/607618383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4c413ec30 0x7fa4c414d2a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:25.128 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.123+0000 7fa4ca148700 1 --2- 172.21.15.23:0/607618383 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa4a0056d90 0x7fa4a0059250 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:25.128 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.123+0000 7fa4ca148700 1 --2- 172.21.15.23:0/607618383 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa4c40af190 0x7fa4c4149d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:25.128 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.123+0000 7fa4ca148700 1 --2- 172.21.15.23:0/607618383 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa4c4137a00 0x7fa4c40abab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:25.128 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.123+0000 7fa4ca148700 1 -- 172.21.15.23:0/607618383 >> 172.21.15.23:0/607618383 conn(0x7fa4c40666e0 msgr2=0x7fa4c4067340 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:25.128 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.124+0000 7fa4ca148700 1 -- 172.21.15.23:0/607618383 shutdown_connections 2020-11-20T11:32:25.129 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.124+0000 7fa4ca148700 1 -- 172.21.15.23:0/607618383 wait complete. 2020-11-20T11:32:25.129 INFO:teuthology.orchestra.run.smithi023.stderr:set pool 4 allow_ec_overwrites to true 2020-11-20T11:32:25.141 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2020-11-20T11:32:25.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.261+0000 7f74778b7700 1 Processor -- start 2020-11-20T11:32:25.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.261+0000 7f74778b7700 1 -- start start 2020-11-20T11:32:25.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.261+0000 7f74778b7700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f74700619d0 0x7f7470137de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:25.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.261+0000 7f74778b7700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7470061db0 0x7f747013e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:25.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.261+0000 7f74778b7700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7470060730 con 0x7f74700a8b20 2020-11-20T11:32:25.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.261+0000 7f74778b7700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7470065120 con 0x7f74700619d0 2020-11-20T11:32:25.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.261+0000 7f74778b7700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7470061110 con 0x7f7470061db0 2020-11-20T11:32:25.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.261+0000 7f7475653700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f74700619d0 0x7f7470137de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:25.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.261+0000 7f7475653700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f74700619d0 0x7f7470137de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40578/0 (socket says 172.21.15.23:40578) 2020-11-20T11:32:25.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.261+0000 7f7475653700 1 -- 172.21.15.23:0/3216379257 learned_addr learned my addr 172.21.15.23:0/3216379257 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:25.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.261+0000 7f7474e52700 1 --2- 172.21.15.23:0/3216379257 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7470061db0 0x7f747013e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:25.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.262+0000 7f7475653700 1 -- 172.21.15.23:0/3216379257 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7470061db0 msgr2=0x7f747013e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:25.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.262+0000 7f7475653700 1 --2- 172.21.15.23:0/3216379257 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7470061db0 0x7f747013e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:25.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.262+0000 7f7475653700 1 -- 172.21.15.23:0/3216379257 >> v1:172.21.15.155:6789/0 conn(0x7f74700a8b20 legacy=0x7f747013aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:25.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.262+0000 7f7475653700 1 -- 172.21.15.23:0/3216379257 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74700609d0 con 0x7f74700619d0 2020-11-20T11:32:25.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.262+0000 7f7474e52700 1 --2- 172.21.15.23:0/3216379257 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7470061db0 0x7f747013e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:32:25.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.262+0000 7f7475653700 1 --2- 172.21.15.23:0/3216379257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74700619d0 0x7f7470137de0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f7464009a40 tx=0x7f7464006ef0).ready entity=mon.0 client_cookie=476429d7e45ef044 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:25.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.262+0000 7f7463fff700 1 -- 172.21.15.23:0/3216379257 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746400b8b0 con 0x7f74700619d0 2020-11-20T11:32:25.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.262+0000 7f7463fff700 1 -- 172.21.15.23:0/3216379257 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7464000e90 con 0x7f74700619d0 2020-11-20T11:32:25.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.262+0000 7f7463fff700 1 -- 172.21.15.23:0/3216379257 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746400bc20 con 0x7f74700619d0 2020-11-20T11:32:25.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.262+0000 7f74778b7700 1 -- 172.21.15.23:0/3216379257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74700619d0 msgr2=0x7f7470137de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:25.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.262+0000 7f74778b7700 1 --2- 172.21.15.23:0/3216379257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74700619d0 0x7f7470137de0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f7464009a40 tx=0x7f7464006ef0).stop 2020-11-20T11:32:25.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.262+0000 7f74778b7700 1 -- 172.21.15.23:0/3216379257 shutdown_connections 2020-11-20T11:32:25.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.262+0000 7f74778b7700 1 --2- 172.21.15.23:0/3216379257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74700619d0 0x7f7470137de0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:25.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.262+0000 7f74778b7700 1 --2- 172.21.15.23:0/3216379257 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7470061db0 0x7f747013e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:25.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.262+0000 7f74778b7700 1 -- 172.21.15.23:0/3216379257 >> 172.21.15.23:0/3216379257 conn(0x7f74700633f0 msgr2=0x7f74700612c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:25.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.262+0000 7f74778b7700 1 -- 172.21.15.23:0/3216379257 shutdown_connections 2020-11-20T11:32:25.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.262+0000 7f74778b7700 1 -- 172.21.15.23:0/3216379257 wait complete. 2020-11-20T11:32:25.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.263+0000 7f74778b7700 1 Processor -- start 2020-11-20T11:32:25.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.263+0000 7f74778b7700 1 -- start start 2020-11-20T11:32:25.275 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.264+0000 7f74778b7700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f74700619d0 0x7f747014a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:25.275 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.264+0000 7f74778b7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7470061db0 0x7f747014a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:25.275 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.264+0000 7f74778b7700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f74700a8b20 0x7f747013afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:25.275 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.264+0000 7f74778b7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747014acf0 con 0x7f7470061db0 2020-11-20T11:32:25.276 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.264+0000 7f74778b7700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74701394b0 con 0x7f74700619d0 2020-11-20T11:32:25.276 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.264+0000 7f74778b7700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74701396e0 con 0x7f74700a8b20 2020-11-20T11:32:25.276 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.264+0000 7f7474e52700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7470061db0 0x7f747014a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:25.276 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.264+0000 7f7474e52700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7470061db0 0x7f747014a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40586/0 (socket says 172.21.15.23:40586) 2020-11-20T11:32:25.276 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.264+0000 7f7474e52700 1 -- 172.21.15.23:0/1020587399 learned_addr learned my addr 172.21.15.23:0/1020587399 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:25.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.264+0000 7f7475653700 1 --2- 172.21.15.23:0/1020587399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f74700619d0 0x7f747014a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:25.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.264+0000 7f7475e54700 1 --2- 172.21.15.23:0/1020587399 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f74700a8b20 0x7f747013afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:25.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.264+0000 7f7474e52700 1 -- 172.21.15.23:0/1020587399 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f74700a8b20 msgr2=0x7f747013afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:25.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.264+0000 7f7474e52700 1 --2- 172.21.15.23:0/1020587399 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f74700a8b20 0x7f747013afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:25.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.264+0000 7f7474e52700 1 -- 172.21.15.23:0/1020587399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f74700619d0 msgr2=0x7f747014a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:25.278 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.264+0000 7f7474e52700 1 --2- 172.21.15.23:0/1020587399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f74700619d0 0x7f747014a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:25.278 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.264+0000 7f7474e52700 1 -- 172.21.15.23:0/1020587399 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74700ad430 con 0x7f7470061db0 2020-11-20T11:32:25.278 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.265+0000 7f7475e54700 1 --2- 172.21.15.23:0/1020587399 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f74700a8b20 0x7f747013afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:25.278 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.265+0000 7f7474e52700 1 --2- 172.21.15.23:0/1020587399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7470061db0 0x7f747014a800 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f746c00cb50 tx=0x7f746c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:25.278 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.265+0000 7f74617fa700 1 -- 172.21.15.23:0/1020587399 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746c015070 con 0x7f7470061db0 2020-11-20T11:32:25.279 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.265+0000 7f74617fa700 1 -- 172.21.15.23:0/1020587399 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f746c006020 con 0x7f7470061db0 2020-11-20T11:32:25.279 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.265+0000 7f74778b7700 1 -- 172.21.15.23:0/1020587399 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7470139980 con 0x7f7470061db0 2020-11-20T11:32:25.279 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.265+0000 7f74617fa700 1 -- 172.21.15.23:0/1020587399 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746c0063a0 con 0x7f7470061db0 2020-11-20T11:32:25.279 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.265+0000 7f74778b7700 1 -- 172.21.15.23:0/1020587399 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f747013b490 con 0x7f7470061db0 2020-11-20T11:32:25.279 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.266+0000 7f74778b7700 1 -- 172.21.15.23:0/1020587399 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74700a50b0 con 0x7f7470061db0 2020-11-20T11:32:25.280 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.266+0000 7f74617fa700 1 -- 172.21.15.23:0/1020587399 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f746c0077d0 con 0x7f7470061db0 2020-11-20T11:32:25.280 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.266+0000 7f74617fa700 1 --2- 172.21.15.23:0/1020587399 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7448056d60 0x7f7448059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:25.280 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.266+0000 7f74617fa700 1 -- 172.21.15.23:0/1020587399 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6315+0+0 (secure 0 0 0) 0x7f746c06fe80 con 0x7f7470061db0 2020-11-20T11:32:25.280 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.267+0000 7f7475653700 1 --2- 172.21.15.23:0/1020587399 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7448056d60 0x7f7448059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:25.280 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.267+0000 7f7475653700 1 --2- 172.21.15.23:0/1020587399 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7448056d60 0x7f7448059220 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f7464006f30 tx=0x7f7464016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:25.281 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.271+0000 7f74617fa700 1 -- 172.21.15.23:0/1020587399 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f746c0451e0 con 0x7f7470061db0 2020-11-20T11:32:25.470 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.466+0000 7f74778b7700 1 -- 172.21.15.23:0/1020587399 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f7470067bb0 con 0x7f7470061db0 2020-11-20T11:32:25.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.473+0000 7f74617fa700 1 -- 172.21.15.23:0/1020587399 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 187+0+0 (secure 0 0 0) 0x7f746c048800 con 0x7f7470061db0 2020-11-20T11:32:25.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.476+0000 7f74778b7700 1 -- 172.21.15.23:0/1020587399 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7448056d60 msgr2=0x7f7448059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:25.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.476+0000 7f74778b7700 1 --2- 172.21.15.23:0/1020587399 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7448056d60 0x7f7448059220 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f7464006f30 tx=0x7f7464016000).stop 2020-11-20T11:32:25.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.476+0000 7f74778b7700 1 -- 172.21.15.23:0/1020587399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7470061db0 msgr2=0x7f747014a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:25.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.476+0000 7f74778b7700 1 --2- 172.21.15.23:0/1020587399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7470061db0 0x7f747014a800 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f746c00cb50 tx=0x7f746c008040).stop 2020-11-20T11:32:25.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.476+0000 7f74778b7700 1 -- 172.21.15.23:0/1020587399 shutdown_connections 2020-11-20T11:32:25.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.476+0000 7f74778b7700 1 --2- 172.21.15.23:0/1020587399 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7448056d60 0x7f7448059220 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:25.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.476+0000 7f74778b7700 1 --2- 172.21.15.23:0/1020587399 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f74700a8b20 0x7f747013afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:25.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.476+0000 7f74778b7700 1 --2- 172.21.15.23:0/1020587399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f74700619d0 0x7f747014a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:25.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.476+0000 7f74778b7700 1 --2- 172.21.15.23:0/1020587399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7470061db0 0x7f747014a800 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:25.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.476+0000 7f74778b7700 1 -- 172.21.15.23:0/1020587399 >> 172.21.15.23:0/1020587399 conn(0x7f74700633f0 msgr2=0x7f74700ae790 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:25.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.476+0000 7f74778b7700 1 -- 172.21.15.23:0/1020587399 shutdown_connections 2020-11-20T11:32:25.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.477+0000 7f74778b7700 1 -- 172.21.15.23:0/1020587399 wait complete. 2020-11-20T11:32:25.485 INFO:teuthology.orchestra.run.smithi023.stderr:new fs with metadata pool 3 and data pool 4 2020-11-20T11:32:25.503 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T11:32:25.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.613+0000 7f1f7c815700 1 Processor -- start 2020-11-20T11:32:25.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.613+0000 7f1f7c815700 1 -- start start 2020-11-20T11:32:25.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.614+0000 7f1f7c815700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1f78061380 0x7f1f780a8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:25.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.614+0000 7f1f7c815700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f780a5000 con 0x7f1f780a7410 2020-11-20T11:32:25.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.614+0000 7f1f7c815700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f7813f6f0 con 0x7f1f780a8d70 2020-11-20T11:32:25.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.614+0000 7f1f7c815700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f780a7890 con 0x7f1f78061380 2020-11-20T11:32:25.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.614+0000 7f1f77fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f1f780a7410 0x7f1f7813efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:54852/0 (socket says 172.21.15.23:54852) 2020-11-20T11:32:25.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.614+0000 7f1f77fff700 1 -- 172.21.15.23:0/2823853543 learned_addr learned my addr 172.21.15.23:0/2823853543 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:25.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.614+0000 7f1f777fe700 1 --2- 172.21.15.23:0/2823853543 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1f78061380 0x7f1f780a8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:25.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.614+0000 7f1f767fc700 1 -- 172.21.15.23:0/2823853543 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1227366941 0 0) 0x7f1f780a5000 con 0x7f1f780a7410 2020-11-20T11:32:25.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.614+0000 7f1f767fc700 1 -- 172.21.15.23:0/2823853543 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f64003280 con 0x7f1f780a7410 2020-11-20T11:32:25.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.614+0000 7f1f767fc700 1 -- 172.21.15.23:0/2823853543 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 112601399 0 0) 0x7f1f64003280 con 0x7f1f780a7410 2020-11-20T11:32:25.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.614+0000 7f1f767fc700 1 -- 172.21.15.23:0/2823853543 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1f78061380 msgr2=0x7f1f780a8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:25.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.614+0000 7f1f767fc700 1 --2- 172.21.15.23:0/2823853543 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1f78061380 0x7f1f780a8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:25.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.614+0000 7f1f767fc700 1 -- 172.21.15.23:0/2823853543 >> v1:172.21.15.155:6789/0 conn(0x7f1f780a8d70 legacy=0x7f1f7813e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:25.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.614+0000 7f1f767fc700 1 -- 172.21.15.23:0/2823853543 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f780a5d30 con 0x7f1f780a7410 2020-11-20T11:32:25.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.614+0000 7f1f767fc700 1 -- 172.21.15.23:0/2823853543 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f1f700018a0 con 0x7f1f780a7410 2020-11-20T11:32:25.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.614+0000 7f1f767fc700 1 -- 172.21.15.23:0/2823853543 >> v1:172.21.15.23:6789/0 conn(0x7f1f780a7410 legacy=0x7f1f7813efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:25.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.614+0000 7f1f767fc700 1 --2- 172.21.15.23:0/2823853543 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f640037a0 0x7f1f64003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:25.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.614+0000 7f1f767fc700 1 --2- 172.21.15.23:0/2823853543 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1f64004520 0x7f1f640049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:25.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.614+0000 7f1f767fc700 1 --2- 172.21.15.23:0/2823853543 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1f78061380 0x7f1f64004e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:25.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.614+0000 7f1f767fc700 1 -- 172.21.15.23:0/2823853543 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f780a7890 con 0x7f1f640037a0 2020-11-20T11:32:25.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.614+0000 7f1f767fc700 1 -- 172.21.15.23:0/2823853543 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f640053d0 con 0x7f1f64004520 2020-11-20T11:32:25.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.614+0000 7f1f767fc700 1 -- 172.21.15.23:0/2823853543 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f64004320 con 0x7f1f78061380 2020-11-20T11:32:25.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.614+0000 7f1f777fe700 1 --2- 172.21.15.23:0/2823853543 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f640037a0 0x7f1f64003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:25.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.615+0000 7f1f77fff700 1 --2- 172.21.15.23:0/2823853543 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1f78061380 0x7f1f64004e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:25.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.615+0000 7f1f76ffd700 1 --2- 172.21.15.23:0/2823853543 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1f64004520 0x7f1f640049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:25.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.615+0000 7f1f777fe700 1 -- 172.21.15.23:0/2823853543 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1f78061380 msgr2=0x7f1f64004e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:25.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.615+0000 7f1f777fe700 1 --2- 172.21.15.23:0/2823853543 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1f78061380 0x7f1f64004e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:25.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.615+0000 7f1f777fe700 1 -- 172.21.15.23:0/2823853543 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1f64004520 msgr2=0x7f1f640049a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:25.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.615+0000 7f1f777fe700 1 --2- 172.21.15.23:0/2823853543 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1f64004520 0x7f1f640049a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:25.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.615+0000 7f1f777fe700 1 -- 172.21.15.23:0/2823853543 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f64003280 con 0x7f1f640037a0 2020-11-20T11:32:25.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.615+0000 7f1f777fe700 1 --2- 172.21.15.23:0/2823853543 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f640037a0 0x7f1f64003b80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f1f6800bc40 tx=0x7f1f68006e30).ready entity=mon.0 client_cookie=371d6b2d7c59ff7c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:25.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.615+0000 7f1f767fc700 1 -- 172.21.15.23:0/2823853543 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6800b350 con 0x7f1f640037a0 2020-11-20T11:32:25.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.615+0000 7f1f767fc700 1 -- 172.21.15.23:0/2823853543 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f6800e9a0 con 0x7f1f640037a0 2020-11-20T11:32:25.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.615+0000 7f1f767fc700 1 -- 172.21.15.23:0/2823853543 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6800ed20 con 0x7f1f640037a0 2020-11-20T11:32:25.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.615+0000 7f1f7c815700 1 -- 172.21.15.23:0/2823853543 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f640037a0 msgr2=0x7f1f64003b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:25.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.615+0000 7f1f7c815700 1 --2- 172.21.15.23:0/2823853543 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f640037a0 0x7f1f64003b80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f1f6800bc40 tx=0x7f1f68006e30).stop 2020-11-20T11:32:25.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.615+0000 7f1f77fff700 1 -- 172.21.15.23:0/2823853543 reap_dead start 2020-11-20T11:32:25.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.615+0000 7f1f7c815700 1 -- 172.21.15.23:0/2823853543 shutdown_connections 2020-11-20T11:32:25.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.615+0000 7f1f7c815700 1 -- 172.21.15.23:0/2823853543 >> 172.21.15.23:0/2823853543 conn(0x7f1f780ae0a0 msgr2=0x7f1f780aa0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:25.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.616+0000 7f1f7c815700 1 -- 172.21.15.23:0/2823853543 shutdown_connections 2020-11-20T11:32:25.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.616+0000 7f1f7c815700 1 -- 172.21.15.23:0/2823853543 wait complete. 2020-11-20T11:32:25.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.617+0000 7f1f7c815700 1 Processor -- start 2020-11-20T11:32:25.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.617+0000 7f1f7c815700 1 -- start start 2020-11-20T11:32:25.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.617+0000 7f1f7c815700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f640037a0 0x7f1f780a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:25.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.617+0000 7f1f7c815700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1f7814dce0 0x7f1f7814e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:25.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.617+0000 7f1f7c815700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1f780a2470 0x7f1f780a28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:25.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.617+0000 7f1f7c815700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f780a2de0 con 0x7f1f640037a0 2020-11-20T11:32:25.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.617+0000 7f1f7c815700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f780a2ff0 con 0x7f1f7814dce0 2020-11-20T11:32:25.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.617+0000 7f1f7c815700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7814d840 con 0x7f1f780a2470 2020-11-20T11:32:25.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.617+0000 7f1f777fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f640037a0 0x7f1f780a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:25.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.617+0000 7f1f777fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f640037a0 0x7f1f780a3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40612/0 (socket says 172.21.15.23:40612) 2020-11-20T11:32:25.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.617+0000 7f1f777fe700 1 -- 172.21.15.23:0/1640833811 learned_addr learned my addr 172.21.15.23:0/1640833811 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:25.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.617+0000 7f1f76ffd700 1 --2- 172.21.15.23:0/1640833811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1f7814dce0 0x7f1f7814e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:25.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.617+0000 7f1f77fff700 1 --2- 172.21.15.23:0/1640833811 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1f780a2470 0x7f1f780a28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:25.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.618+0000 7f1f777fe700 1 -- 172.21.15.23:0/1640833811 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1f780a2470 msgr2=0x7f1f780a28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:25.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.618+0000 7f1f777fe700 1 --2- 172.21.15.23:0/1640833811 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1f780a2470 0x7f1f780a28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:25.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.618+0000 7f1f777fe700 1 -- 172.21.15.23:0/1640833811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1f7814dce0 msgr2=0x7f1f7814e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:25.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.618+0000 7f1f777fe700 1 --2- 172.21.15.23:0/1640833811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1f7814dce0 0x7f1f7814e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:25.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.618+0000 7f1f777fe700 1 -- 172.21.15.23:0/1640833811 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f78138870 con 0x7f1f640037a0 2020-11-20T11:32:25.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.618+0000 7f1f77fff700 1 --2- 172.21.15.23:0/1640833811 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1f780a2470 0x7f1f780a28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:25.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.618+0000 7f1f777fe700 1 --2- 172.21.15.23:0/1640833811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f640037a0 0x7f1f780a3750 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f1f6800e4b0 tx=0x7f1f68019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:25.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.618+0000 7f1f577fe700 1 -- 172.21.15.23:0/1640833811 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f68012db0 con 0x7f1f640037a0 2020-11-20T11:32:25.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.618+0000 7f1f577fe700 1 -- 172.21.15.23:0/1640833811 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f68000e90 con 0x7f1f640037a0 2020-11-20T11:32:25.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.618+0000 7f1f7c815700 1 -- 172.21.15.23:0/1640833811 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f78138a60 con 0x7f1f640037a0 2020-11-20T11:32:25.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.618+0000 7f1f577fe700 1 -- 172.21.15.23:0/1640833811 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6801b650 con 0x7f1f640037a0 2020-11-20T11:32:25.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.618+0000 7f1f7c815700 1 -- 172.21.15.23:0/1640833811 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f78138ed0 con 0x7f1f640037a0 2020-11-20T11:32:25.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.619+0000 7f1f7c815700 1 -- 172.21.15.23:0/1640833811 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f7805cf90 con 0x7f1f640037a0 2020-11-20T11:32:25.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.619+0000 7f1f577fe700 1 -- 172.21.15.23:0/1640833811 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1f6802d070 con 0x7f1f640037a0 2020-11-20T11:32:25.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.619+0000 7f1f577fe700 1 --2- 172.21.15.23:0/1640833811 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1f50056d60 0x7f1f50059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:25.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.620+0000 7f1f577fe700 1 -- 172.21.15.23:0/1640833811 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f1f68071840 con 0x7f1f640037a0 2020-11-20T11:32:25.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.620+0000 7f1f76ffd700 1 --2- 172.21.15.23:0/1640833811 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1f50056d60 0x7f1f50059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:25.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.621+0000 7f1f76ffd700 1 --2- 172.21.15.23:0/1640833811 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1f50056d60 0x7f1f50059220 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f1f6c000c00 tx=0x7f1f6c007bb0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:25.652 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.623+0000 7f1f577fe700 1 -- 172.21.15.23:0/1640833811 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1f68046bb0 con 0x7f1f640037a0 2020-11-20T11:32:25.776 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.772+0000 7f1f7c815700 1 -- 172.21.15.23:0/1640833811 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f7814da00 con 0x7f1f640037a0 2020-11-20T11:32:25.777 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.773+0000 7f1f577fe700 1 -- 172.21.15.23:0/1640833811 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17100 (secure 0 0 0) 0x7f1f6804a1d0 con 0x7f1f640037a0 2020-11-20T11:32:25.777 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:32:25.778 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":24,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","created":"2020-11-20T11:32:06.136679+0000","modified":"2020-11-20T11:32:25.468975+0000","last_up_change":"2020-11-20T11:32:13.536194+0000","last_in_change":"2020-11-20T11:32:13.536194+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T11:32:09.559868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T11:32:17.000782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T11:32:20.423341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T11:32:23.126273+0000","flags":32773,"flags_names":"hashpspool,ec_overwrites,creating","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e9d219f5-e40e-4b2b-9468-deb9c4074a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":23083},{"type":"v1","addr":"172.21.15.23:6801","nonce":23083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":23083},{"type":"v1","addr":"172.21.15.23:6803","nonce":23083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":23083},{"type":"v1","addr":"172.21.15.23:6807","nonce":23083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":23083},{"type":"v1","addr":"172.21.15.23:6805","nonce":23083}]},"public_addr":"172.21.15.23:6801/23083","cluster_addr":"172.21.15.23:6803/23083","heartbeat_back_addr":"172.21.15.23:6807/23083","heartbeat_front_addr":"172.21.15.23:6805/23083","state":["exists","up"]},{"osd":1,"uuid":"2328adde-2108-4fe0-8138-f0d096a496a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":23099},{"type":"v1","addr":"172.21.15.23:6817","nonce":23099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":23099},{"type":"v1","addr":"172.21.15.23:6819","nonce":23099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":23099},{"type":"v1","addr":"172.21.15.23:6824","nonce":23099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":23099},{"type":"v1","addr":"172.21.15.23:6821","nonce":23099}]},"public_addr":"172.21.15.23:6817/23099","cluster_addr":"172.21.15.23:6819/23099","heartbeat_back_addr":"172.21.15.23:6824/23099","heartbeat_front_addr":"172.21.15.23:6821/23099","state":["exists","up"]},{"osd":2,"uuid":"13367708-59a2-4363-8a3a-ab0cb97145ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":23084},{"type":"v1","addr":"172.21.15.23:6809","nonce":23084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":23084},{"type":"v1","addr":"172.21.15.23:6811","nonce":23084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":23084},{"type":"v1","addr":"172.21.15.23:6815","nonce":23084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":23084},{"type":"v1","addr":"172.21.15.23:6813","nonce":23084}]},"public_addr":"172.21.15.23:6809/23084","cluster_addr":"172.21.15.23:6811/23084","heartbeat_back_addr":"172.21.15.23:6815/23084","heartbeat_front_addr":"172.21.15.23:6813/23084","state":["exists","up"]},{"osd":3,"uuid":"1f147dd7-9289-47b0-ad1c-4088618d6faa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6823","nonce":23114},{"type":"v1","addr":"172.21.15.23:6825","nonce":23114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":23114},{"type":"v1","addr":"172.21.15.23:6827","nonce":23114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":23114},{"type":"v1","addr":"172.21.15.23:6831","nonce":23114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":23114},{"type":"v1","addr":"172.21.15.23:6829","nonce":23114}]},"public_addr":"172.21.15.23:6825/23114","cluster_addr":"172.21.15.23:6827/23114","heartbeat_back_addr":"172.21.15.23:6831/23114","heartbeat_front_addr":"172.21.15.23:6829/23114","state":["exists","up"]},{"osd":4,"uuid":"d008b039-4bc5-4135-9678-1511e58efc29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":23280},{"type":"v1","addr":"172.21.15.155:6801","nonce":23280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":23280},{"type":"v1","addr":"172.21.15.155:6803","nonce":23280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":23280},{"type":"v1","addr":"172.21.15.155:6807","nonce":23280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":23280},{"type":"v1","addr":"172.21.15.155:6805","nonce":23280}]},"public_addr":"172.21.15.155:6801/23280","cluster_addr":"172.21.15.155:6803/23280","heartbeat_back_addr":"172.21.15.155:6807/23280","heartbeat_front_addr":"172.21.15.155:6805/23280","state":["exists","up"]},{"osd":5,"uuid":"56715e6f-cf6e-44a6-9e39-93ab1224095f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":23298},{"type":"v1","addr":"172.21.15.155:6817","nonce":23298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":23298},{"type":"v1","addr":"172.21.15.155:6819","nonce":23298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":23298},{"type":"v1","addr":"172.21.15.155:6823","nonce":23298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":23298},{"type":"v1","addr":"172.21.15.155:6821","nonce":23298}]},"public_addr":"172.21.15.155:6817/23298","cluster_addr":"172.21.15.155:6819/23298","heartbeat_back_addr":"172.21.15.155:6823/23298","heartbeat_front_addr":"172.21.15.155:6821/23298","state":["exists","up"]},{"osd":6,"uuid":"db1e61ca-8e14-4a1f-8607-6e01b148d61c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":23307},{"type":"v1","addr":"172.21.15.155:6825","nonce":23307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":23307},{"type":"v1","addr":"172.21.15.155:6827","nonce":23307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":23307},{"type":"v1","addr":"172.21.15.155:6831","nonce":23307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":23307},{"type":"v1","addr":"172.21.15.155:6829","nonce":23307}]},"public_addr":"172.21.15.155:6825/23307","cluster_addr":"172.21.15.155:6827/23307","heartbeat_back_addr":"172.21.15.155:6831/23307","heartbeat_front_addr":"172.21.15.155:6829/23307","state":["exists","up"]},{"osd":7,"uuid":"d44419be-ef89-462c-8cf1-3c6db2a6944b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":23281},{"type":"v1","addr":"172.21.15.155:6809","nonce":23281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":23281},{"type":"v1","addr":"172.21.15.155:6811","nonce":23281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":23281},{"type":"v1","addr":"172.21.15.155:6815","nonce":23281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":23281},{"type":"v1","addr":"172.21.15.155:6813","nonce":23281}]},"public_addr":"172.21.15.155:6809/23281","cluster_addr":"172.21.15.155:6811/23281","heartbeat_back_addr":"172.21.15.155:6815/23281","heartbeat_front_addr":"172.21.15.155:6813/23281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.486702+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.509396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.533414+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.509489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.467600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.513068+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.536636+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.531772+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T11:32:25.779 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.775+0000 7f1f7c815700 1 -- 172.21.15.23:0/1640833811 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1f50056d60 msgr2=0x7f1f50059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:25.779 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.775+0000 7f1f7c815700 1 --2- 172.21.15.23:0/1640833811 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1f50056d60 0x7f1f50059220 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f1f6c000c00 tx=0x7f1f6c007bb0).stop 2020-11-20T11:32:25.779 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.775+0000 7f1f7c815700 1 -- 172.21.15.23:0/1640833811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f640037a0 msgr2=0x7f1f780a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:25.780 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.775+0000 7f1f7c815700 1 --2- 172.21.15.23:0/1640833811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f640037a0 0x7f1f780a3750 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f1f6800e4b0 tx=0x7f1f68019000).stop 2020-11-20T11:32:25.780 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.775+0000 7f1f7c815700 1 -- 172.21.15.23:0/1640833811 shutdown_connections 2020-11-20T11:32:25.780 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.775+0000 7f1f7c815700 1 --2- 172.21.15.23:0/1640833811 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1f50056d60 0x7f1f50059220 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:25.780 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.775+0000 7f1f7c815700 1 --2- 172.21.15.23:0/1640833811 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1f780a2470 0x7f1f780a28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:25.780 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.775+0000 7f1f7c815700 1 --2- 172.21.15.23:0/1640833811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1f640037a0 0x7f1f780a3750 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:25.781 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.775+0000 7f1f7c815700 1 --2- 172.21.15.23:0/1640833811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1f7814dce0 0x7f1f7814e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:25.781 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.775+0000 7f1f7c815700 1 -- 172.21.15.23:0/1640833811 >> 172.21.15.23:0/1640833811 conn(0x7f1f780ae0a0 msgr2=0x7f1f78068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:25.781 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.775+0000 7f1f7c815700 1 -- 172.21.15.23:0/1640833811 shutdown_connections 2020-11-20T11:32:25.781 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.775+0000 7f1f7c815700 1 -- 172.21.15.23:0/1640833811 wait complete. 2020-11-20T11:32:25.790 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T11:32:25.904 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.900+0000 7fee13fd1700 1 Processor -- start 2020-11-20T11:32:25.904 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.900+0000 7fee13fd1700 1 -- start start 2020-11-20T11:32:25.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.901+0000 7fee13fd1700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fee0c067eb0 0x7fee0c066700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:25.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.901+0000 7fee13fd1700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c068430 con 0x7fee0c13ae70 2020-11-20T11:32:25.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.901+0000 7fee13fd1700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c0686d0 con 0x7fee0c05bc80 2020-11-20T11:32:25.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.901+0000 7fee13fd1700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee0c067470 con 0x7fee0c067eb0 2020-11-20T11:32:25.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.901+0000 7fee1256e700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fee0c13ae70 0x7fee0c134cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:54874/0 (socket says 172.21.15.23:54874) 2020-11-20T11:32:25.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.901+0000 7fee1256e700 1 -- 172.21.15.23:0/613576345 learned_addr learned my addr 172.21.15.23:0/613576345 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:25.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.901+0000 7fee1156c700 1 --2- 172.21.15.23:0/613576345 >> v2:172.21.15.155:3300/0 conn(0x7fee0c067eb0 0x7fee0c066700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:25.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.901+0000 7fee10d6b700 1 -- 172.21.15.23:0/613576345 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1477854577 0 0) 0x7fee0c068430 con 0x7fee0c13ae70 2020-11-20T11:32:25.909 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.901+0000 7fee10d6b700 1 -- 172.21.15.23:0/613576345 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedf4003280 con 0x7fee0c13ae70 2020-11-20T11:32:25.909 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.901+0000 7fee1156c700 1 -- 172.21.15.23:0/613576345 >> v1:172.21.15.155:6789/0 conn(0x7fee0c05bc80 legacy=0x7fee0c1345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:25.909 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.901+0000 7fee1156c700 1 -- 172.21.15.23:0/613576345 >> v1:172.21.15.23:6789/0 conn(0x7fee0c13ae70 legacy=0x7fee0c134cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:25.909 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.901+0000 7fee1156c700 1 -- 172.21.15.23:0/613576345 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0c066260 con 0x7fee0c067eb0 2020-11-20T11:32:25.910 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.901+0000 7fee1156c700 1 --2- 172.21.15.23:0/613576345 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee0c067eb0 0x7fee0c066700 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fedfc006fd0 tx=0x7fedfc006e30).ready entity=mon.1 client_cookie=9803782db786044 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:25.910 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.902+0000 7fee10d6b700 1 -- 172.21.15.23:0/613576345 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc0097e0 con 0x7fee0c067eb0 2020-11-20T11:32:25.910 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.902+0000 7fee10d6b700 1 -- 172.21.15.23:0/613576345 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedfc000e90 con 0x7fee0c067eb0 2020-11-20T11:32:25.910 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.902+0000 7fee10d6b700 1 -- 172.21.15.23:0/613576345 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc009b50 con 0x7fee0c067eb0 2020-11-20T11:32:25.910 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.903+0000 7fee13fd1700 1 -- 172.21.15.23:0/613576345 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee0c067eb0 msgr2=0x7fee0c066700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:25.911 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.903+0000 7fee13fd1700 1 --2- 172.21.15.23:0/613576345 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee0c067eb0 0x7fee0c066700 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fedfc006fd0 tx=0x7fedfc006e30).stop 2020-11-20T11:32:25.911 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.903+0000 7fee13fd1700 1 -- 172.21.15.23:0/613576345 shutdown_connections 2020-11-20T11:32:25.911 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.903+0000 7fee13fd1700 1 --2- 172.21.15.23:0/613576345 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee0c067eb0 0x7fee0c066700 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:25.911 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.903+0000 7fee13fd1700 1 -- 172.21.15.23:0/613576345 >> 172.21.15.23:0/613576345 conn(0x7fee0c0ad7a0 msgr2=0x7fee0c0adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:25.911 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.903+0000 7fee13fd1700 1 -- 172.21.15.23:0/613576345 shutdown_connections 2020-11-20T11:32:25.912 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.903+0000 7fee13fd1700 1 -- 172.21.15.23:0/613576345 wait complete. 2020-11-20T11:32:25.915 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.904+0000 7fee13fd1700 1 Processor -- start 2020-11-20T11:32:25.915 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.904+0000 7fee13fd1700 1 -- start start 2020-11-20T11:32:25.915 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.904+0000 7fee13fd1700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fee0c05bc80 0x7fee0c0a20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:25.916 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.904+0000 7fee13fd1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee0c067eb0 0x7fee0c0a3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:25.916 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.904+0000 7fee13fd1700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee0c13ae70 0x7fee0c149090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:25.916 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.904+0000 7fee13fd1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0c13b2e0 con 0x7fee0c067eb0 2020-11-20T11:32:25.916 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.904+0000 7fee13fd1700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0c09fb40 con 0x7fee0c13ae70 2020-11-20T11:32:25.916 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.904+0000 7fee13fd1700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee0c0a0c70 con 0x7fee0c05bc80 2020-11-20T11:32:25.917 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.904+0000 7fee1156c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee0c067eb0 0x7fee0c0a3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:25.917 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.904+0000 7fee1156c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee0c067eb0 0x7fee0c0a3b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40628/0 (socket says 172.21.15.23:40628) 2020-11-20T11:32:25.917 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.904+0000 7fee1156c700 1 -- 172.21.15.23:0/2312730925 learned_addr learned my addr 172.21.15.23:0/2312730925 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:25.917 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.905+0000 7fee11d6d700 1 --2- 172.21.15.23:0/2312730925 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fee0c05bc80 0x7fee0c0a20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:25.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.905+0000 7fee1256e700 1 --2- 172.21.15.23:0/2312730925 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee0c13ae70 0x7fee0c149090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:25.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.905+0000 7fee1156c700 1 -- 172.21.15.23:0/2312730925 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fee0c05bc80 msgr2=0x7fee0c0a20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:25.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.905+0000 7fee1156c700 1 --2- 172.21.15.23:0/2312730925 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fee0c05bc80 0x7fee0c0a20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:25.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.905+0000 7fee1156c700 1 -- 172.21.15.23:0/2312730925 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee0c13ae70 msgr2=0x7fee0c149090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:25.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.905+0000 7fee1156c700 1 --2- 172.21.15.23:0/2312730925 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee0c13ae70 0x7fee0c149090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:25.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.905+0000 7fee1156c700 1 -- 172.21.15.23:0/2312730925 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0c0a0f10 con 0x7fee0c067eb0 2020-11-20T11:32:25.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.905+0000 7fee11d6d700 1 --2- 172.21.15.23:0/2312730925 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fee0c05bc80 0x7fee0c0a20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:25.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.905+0000 7fee1156c700 1 --2- 172.21.15.23:0/2312730925 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee0c067eb0 0x7fee0c0a3b30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fedfc006e70 tx=0x7fedfc015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:25.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.905+0000 7fedf9ffb700 1 -- 172.21.15.23:0/2312730925 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc011c70 con 0x7fee0c067eb0 2020-11-20T11:32:25.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.905+0000 7fedf9ffb700 1 -- 172.21.15.23:0/2312730925 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedfc011dd0 con 0x7fee0c067eb0 2020-11-20T11:32:25.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.905+0000 7fee13fd1700 1 -- 172.21.15.23:0/2312730925 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee0c0a0010 con 0x7fee0c067eb0 2020-11-20T11:32:25.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.905+0000 7fedf9ffb700 1 -- 172.21.15.23:0/2312730925 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedfc01e690 con 0x7fee0c067eb0 2020-11-20T11:32:25.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.905+0000 7fee13fd1700 1 -- 172.21.15.23:0/2312730925 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee0c0a4020 con 0x7fee0c067eb0 2020-11-20T11:32:25.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.906+0000 7fee13fd1700 1 -- 172.21.15.23:0/2312730925 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee0c0a1340 con 0x7fee0c067eb0 2020-11-20T11:32:25.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.906+0000 7fedf9ffb700 1 -- 172.21.15.23:0/2312730925 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fedfc025030 con 0x7fee0c067eb0 2020-11-20T11:32:25.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.907+0000 7fedf9ffb700 1 --2- 172.21.15.23:0/2312730925 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fede4056d60 0x7fede4059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:25.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.907+0000 7fedf9ffb700 1 -- 172.21.15.23:0/2312730925 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fedfc071630 con 0x7fee0c067eb0 2020-11-20T11:32:25.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.907+0000 7fee11d6d700 1 --2- 172.21.15.23:0/2312730925 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fede4056d60 0x7fede4059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:25.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.907+0000 7fee11d6d700 1 --2- 172.21.15.23:0/2312730925 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fede4056d60 0x7fede4059220 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fee0c0ae2c0 tx=0x7fee08009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:25.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:25.910+0000 7fedf9ffb700 1 -- 172.21.15.23:0/2312730925 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fedfc0469d0 con 0x7fee0c067eb0 2020-11-20T11:32:26.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.062+0000 7fee13fd1700 1 -- 172.21.15.23:0/2312730925 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee0c062fe0 con 0x7fee0c067eb0 2020-11-20T11:32:26.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.063+0000 7fedf9ffb700 1 -- 172.21.15.23:0/2312730925 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+17100 (secure 0 0 0) 0x7fedfc0312e0 con 0x7fee0c067eb0 2020-11-20T11:32:26.067 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:32:26.068 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":24,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","created":"2020-11-20T11:32:06.136679+0000","modified":"2020-11-20T11:32:25.468975+0000","last_up_change":"2020-11-20T11:32:13.536194+0000","last_in_change":"2020-11-20T11:32:13.536194+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T11:32:09.559868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T11:32:17.000782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T11:32:20.423341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T11:32:23.126273+0000","flags":32773,"flags_names":"hashpspool,ec_overwrites,creating","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e9d219f5-e40e-4b2b-9468-deb9c4074a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":23083},{"type":"v1","addr":"172.21.15.23:6801","nonce":23083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":23083},{"type":"v1","addr":"172.21.15.23:6803","nonce":23083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":23083},{"type":"v1","addr":"172.21.15.23:6807","nonce":23083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":23083},{"type":"v1","addr":"172.21.15.23:6805","nonce":23083}]},"public_addr":"172.21.15.23:6801/23083","cluster_addr":"172.21.15.23:6803/23083","heartbeat_back_addr":"172.21.15.23:6807/23083","heartbeat_front_addr":"172.21.15.23:6805/23083","state":["exists","up"]},{"osd":1,"uuid":"2328adde-2108-4fe0-8138-f0d096a496a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":23099},{"type":"v1","addr":"172.21.15.23:6817","nonce":23099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":23099},{"type":"v1","addr":"172.21.15.23:6819","nonce":23099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":23099},{"type":"v1","addr":"172.21.15.23:6824","nonce":23099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":23099},{"type":"v1","addr":"172.21.15.23:6821","nonce":23099}]},"public_addr":"172.21.15.23:6817/23099","cluster_addr":"172.21.15.23:6819/23099","heartbeat_back_addr":"172.21.15.23:6824/23099","heartbeat_front_addr":"172.21.15.23:6821/23099","state":["exists","up"]},{"osd":2,"uuid":"13367708-59a2-4363-8a3a-ab0cb97145ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":23084},{"type":"v1","addr":"172.21.15.23:6809","nonce":23084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":23084},{"type":"v1","addr":"172.21.15.23:6811","nonce":23084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":23084},{"type":"v1","addr":"172.21.15.23:6815","nonce":23084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":23084},{"type":"v1","addr":"172.21.15.23:6813","nonce":23084}]},"public_addr":"172.21.15.23:6809/23084","cluster_addr":"172.21.15.23:6811/23084","heartbeat_back_addr":"172.21.15.23:6815/23084","heartbeat_front_addr":"172.21.15.23:6813/23084","state":["exists","up"]},{"osd":3,"uuid":"1f147dd7-9289-47b0-ad1c-4088618d6faa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6823","nonce":23114},{"type":"v1","addr":"172.21.15.23:6825","nonce":23114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":23114},{"type":"v1","addr":"172.21.15.23:6827","nonce":23114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":23114},{"type":"v1","addr":"172.21.15.23:6831","nonce":23114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":23114},{"type":"v1","addr":"172.21.15.23:6829","nonce":23114}]},"public_addr":"172.21.15.23:6825/23114","cluster_addr":"172.21.15.23:6827/23114","heartbeat_back_addr":"172.21.15.23:6831/23114","heartbeat_front_addr":"172.21.15.23:6829/23114","state":["exists","up"]},{"osd":4,"uuid":"d008b039-4bc5-4135-9678-1511e58efc29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":23280},{"type":"v1","addr":"172.21.15.155:6801","nonce":23280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":23280},{"type":"v1","addr":"172.21.15.155:6803","nonce":23280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":23280},{"type":"v1","addr":"172.21.15.155:6807","nonce":23280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":23280},{"type":"v1","addr":"172.21.15.155:6805","nonce":23280}]},"public_addr":"172.21.15.155:6801/23280","cluster_addr":"172.21.15.155:6803/23280","heartbeat_back_addr":"172.21.15.155:6807/23280","heartbeat_front_addr":"172.21.15.155:6805/23280","state":["exists","up"]},{"osd":5,"uuid":"56715e6f-cf6e-44a6-9e39-93ab1224095f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":23298},{"type":"v1","addr":"172.21.15.155:6817","nonce":23298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":23298},{"type":"v1","addr":"172.21.15.155:6819","nonce":23298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":23298},{"type":"v1","addr":"172.21.15.155:6823","nonce":23298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":23298},{"type":"v1","addr":"172.21.15.155:6821","nonce":23298}]},"public_addr":"172.21.15.155:6817/23298","cluster_addr":"172.21.15.155:6819/23298","heartbeat_back_addr":"172.21.15.155:6823/23298","heartbeat_front_addr":"172.21.15.155:6821/23298","state":["exists","up"]},{"osd":6,"uuid":"db1e61ca-8e14-4a1f-8607-6e01b148d61c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":23307},{"type":"v1","addr":"172.21.15.155:6825","nonce":23307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":23307},{"type":"v1","addr":"172.21.15.155:6827","nonce":23307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":23307},{"type":"v1","addr":"172.21.15.155:6831","nonce":23307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":23307},{"type":"v1","addr":"172.21.15.155:6829","nonce":23307}]},"public_addr":"172.21.15.155:6825/23307","cluster_addr":"172.21.15.155:6827/23307","heartbeat_back_addr":"172.21.15.155:6831/23307","heartbeat_front_addr":"172.21.15.155:6829/23307","state":["exists","up"]},{"osd":7,"uuid":"d44419be-ef89-462c-8cf1-3c6db2a6944b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":23281},{"type":"v1","addr":"172.21.15.155:6809","nonce":23281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":23281},{"type":"v1","addr":"172.21.15.155:6811","nonce":23281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":23281},{"type":"v1","addr":"172.21.15.155:6815","nonce":23281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":23281},{"type":"v1","addr":"172.21.15.155:6813","nonce":23281}]},"public_addr":"172.21.15.155:6809/23281","cluster_addr":"172.21.15.155:6811/23281","heartbeat_back_addr":"172.21.15.155:6815/23281","heartbeat_front_addr":"172.21.15.155:6813/23281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.486702+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.509396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.533414+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.509489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.467600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.513068+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.536636+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.531772+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T11:32:26.069 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.064+0000 7fee13fd1700 1 -- 172.21.15.23:0/2312730925 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fede4056d60 msgr2=0x7fede4059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:26.069 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.064+0000 7fee13fd1700 1 --2- 172.21.15.23:0/2312730925 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fede4056d60 0x7fede4059220 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fee0c0ae2c0 tx=0x7fee08009040).stop 2020-11-20T11:32:26.069 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.064+0000 7fee13fd1700 1 -- 172.21.15.23:0/2312730925 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee0c067eb0 msgr2=0x7fee0c0a3b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:26.069 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.064+0000 7fee13fd1700 1 --2- 172.21.15.23:0/2312730925 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee0c067eb0 0x7fee0c0a3b30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fedfc006e70 tx=0x7fedfc015000).stop 2020-11-20T11:32:26.070 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.064+0000 7fee13fd1700 1 -- 172.21.15.23:0/2312730925 shutdown_connections 2020-11-20T11:32:26.070 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.064+0000 7fee13fd1700 1 --2- 172.21.15.23:0/2312730925 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fee0c13ae70 0x7fee0c149090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:26.070 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.064+0000 7fee13fd1700 1 --2- 172.21.15.23:0/2312730925 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fede4056d60 0x7fede4059220 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:26.070 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.064+0000 7fee13fd1700 1 --2- 172.21.15.23:0/2312730925 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fee0c05bc80 0x7fee0c0a20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:26.070 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.064+0000 7fee13fd1700 1 --2- 172.21.15.23:0/2312730925 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee0c067eb0 0x7fee0c0a3b30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:26.071 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.064+0000 7fee13fd1700 1 -- 172.21.15.23:0/2312730925 >> 172.21.15.23:0/2312730925 conn(0x7fee0c0ad7a0 msgr2=0x7fee0c0aa290 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:26.071 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.064+0000 7fee13fd1700 1 -- 172.21.15.23:0/2312730925 shutdown_connections 2020-11-20T11:32:26.071 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.064+0000 7fee13fd1700 1 -- 172.21.15.23:0/2312730925 wait complete. 2020-11-20T11:32:26.080 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2020-11-20T11:32:26.197 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.193+0000 7f8b57593700 1 Processor -- start 2020-11-20T11:32:26.197 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.193+0000 7f8b57593700 1 -- start start 2020-11-20T11:32:26.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.195+0000 7f8b57593700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b500aae90 0x7f8b501381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:26.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.195+0000 7f8b57593700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f8b5013aeb0 0x7f8b500a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:26.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.195+0000 7f8b57593700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b5013e600 con 0x7f8b500aaab0 2020-11-20T11:32:26.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.195+0000 7f8b57593700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b5013e8a0 con 0x7f8b5013aeb0 2020-11-20T11:32:26.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.195+0000 7f8b57593700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b500ab6d0 con 0x7f8b500aae90 2020-11-20T11:32:26.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.195+0000 7f8b54b2e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b500aae90 0x7f8b501381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:26.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.195+0000 7f8b55b30700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f8b5013aeb0 0x7f8b500a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:26.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.195+0000 7f8b5532f700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f8b500aaab0 0x7f8b5013ed40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:32858/0 (socket says 172.21.15.23:32858) 2020-11-20T11:32:26.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.195+0000 7f8b5532f700 1 -- 172.21.15.23:0/2964451407 learned_addr learned my addr 172.21.15.23:0/2964451407 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:26.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.195+0000 7f8b43fff700 1 -- 172.21.15.23:0/2964451407 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648284719 0 0) 0x7f8b5013e600 con 0x7f8b500aaab0 2020-11-20T11:32:26.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.195+0000 7f8b43fff700 1 -- 172.21.15.23:0/2964451407 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b38003280 con 0x7f8b500aaab0 2020-11-20T11:32:26.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.195+0000 7f8b54b2e700 1 -- 172.21.15.23:0/2964451407 >> v2:172.21.15.155:3300/0 conn(0x7f8b5013aeb0 msgr2=0x7f8b500a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:26.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.195+0000 7f8b54b2e700 1 --2- 172.21.15.23:0/2964451407 >> v2:172.21.15.155:3300/0 conn(0x7f8b5013aeb0 0x7f8b500a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:26.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.195+0000 7f8b54b2e700 1 -- 172.21.15.23:0/2964451407 >> v1:172.21.15.155:6789/0 conn(0x7f8b500aaab0 legacy=0x7f8b5013ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:26.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.195+0000 7f8b54b2e700 1 -- 172.21.15.23:0/2964451407 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b50138690 con 0x7f8b500aae90 2020-11-20T11:32:26.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.195+0000 7f8b55b30700 1 --2- 172.21.15.23:0/2964451407 >> v2:172.21.15.155:3300/0 conn(0x7f8b5013aeb0 0x7f8b500a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:32:26.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.195+0000 7f8b54b2e700 1 --2- 172.21.15.23:0/2964451407 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b500aae90 0x7f8b501381a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f8b44006fd0 tx=0x7f8b44006e30).ready entity=mon.2 client_cookie=41b51cedac292f26 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:26.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.195+0000 7f8b43fff700 1 -- 172.21.15.23:0/2964451407 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b44009630 con 0x7f8b500aae90 2020-11-20T11:32:26.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.195+0000 7f8b43fff700 1 -- 172.21.15.23:0/2964451407 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b44000e90 con 0x7f8b500aae90 2020-11-20T11:32:26.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.195+0000 7f8b43fff700 1 -- 172.21.15.23:0/2964451407 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b440099a0 con 0x7f8b500aae90 2020-11-20T11:32:26.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.195+0000 7f8b57593700 1 -- 172.21.15.23:0/2964451407 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b500aae90 msgr2=0x7f8b501381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:26.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.195+0000 7f8b57593700 1 --2- 172.21.15.23:0/2964451407 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b500aae90 0x7f8b501381a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f8b44006fd0 tx=0x7f8b44006e30).stop 2020-11-20T11:32:26.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.195+0000 7f8b57593700 1 -- 172.21.15.23:0/2964451407 shutdown_connections 2020-11-20T11:32:26.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.195+0000 7f8b57593700 1 --2- 172.21.15.23:0/2964451407 >> v2:172.21.15.155:3300/0 conn(0x7f8b5013aeb0 0x7f8b500a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:26.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.195+0000 7f8b57593700 1 --2- 172.21.15.23:0/2964451407 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b500aae90 0x7f8b501381a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:26.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.195+0000 7f8b57593700 1 -- 172.21.15.23:0/2964451407 >> 172.21.15.23:0/2964451407 conn(0x7f8b500678e0 msgr2=0x7f8b50067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:26.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.195+0000 7f8b57593700 1 -- 172.21.15.23:0/2964451407 shutdown_connections 2020-11-20T11:32:26.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.195+0000 7f8b57593700 1 -- 172.21.15.23:0/2964451407 wait complete. 2020-11-20T11:32:26.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.196+0000 7f8b57593700 1 Processor -- start 2020-11-20T11:32:26.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.196+0000 7f8b57593700 1 -- start start 2020-11-20T11:32:26.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.197+0000 7f8b57593700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8b500aaab0 0x7f8b50135d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:26.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.197+0000 7f8b57593700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b500aae90 0x7f8b5013c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:26.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.197+0000 7f8b57593700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b5013dd20 0x7f8b5013cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:26.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.197+0000 7f8b57593700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b500a64c0 con 0x7f8b5013dd20 2020-11-20T11:32:26.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.197+0000 7f8b57593700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b500a6760 con 0x7f8b500aaab0 2020-11-20T11:32:26.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.197+0000 7f8b57593700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5013c350 con 0x7f8b500aae90 2020-11-20T11:32:26.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.197+0000 7f8b55b30700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b5013dd20 0x7f8b5013cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:26.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.197+0000 7f8b55b30700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b5013dd20 0x7f8b5013cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40644/0 (socket says 172.21.15.23:40644) 2020-11-20T11:32:26.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.197+0000 7f8b55b30700 1 -- 172.21.15.23:0/490737217 learned_addr learned my addr 172.21.15.23:0/490737217 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:26.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.197+0000 7f8b54b2e700 1 --2- 172.21.15.23:0/490737217 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b500aae90 0x7f8b5013c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:26.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.197+0000 7f8b5532f700 1 --2- 172.21.15.23:0/490737217 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8b500aaab0 0x7f8b50135d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:26.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.197+0000 7f8b55b30700 1 -- 172.21.15.23:0/490737217 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b500aae90 msgr2=0x7f8b5013c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:26.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.197+0000 7f8b55b30700 1 --2- 172.21.15.23:0/490737217 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b500aae90 0x7f8b5013c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:26.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.197+0000 7f8b55b30700 1 -- 172.21.15.23:0/490737217 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8b500aaab0 msgr2=0x7f8b50135d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:26.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.197+0000 7f8b55b30700 1 --2- 172.21.15.23:0/490737217 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8b500aaab0 0x7f8b50135d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:26.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.197+0000 7f8b55b30700 1 -- 172.21.15.23:0/490737217 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b50132aa0 con 0x7f8b5013dd20 2020-11-20T11:32:26.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.198+0000 7f8b55b30700 1 --2- 172.21.15.23:0/490737217 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b5013dd20 0x7f8b5013cce0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f8b4800ef50 tx=0x7f8b48009220).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:26.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.198+0000 7f8b417fa700 1 -- 172.21.15.23:0/490737217 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4800cc10 con 0x7f8b5013dd20 2020-11-20T11:32:26.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.198+0000 7f8b417fa700 1 -- 172.21.15.23:0/490737217 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b4800cd70 con 0x7f8b5013dd20 2020-11-20T11:32:26.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.198+0000 7f8b417fa700 1 -- 172.21.15.23:0/490737217 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b48012620 con 0x7f8b5013dd20 2020-11-20T11:32:26.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.198+0000 7f8b57593700 1 -- 172.21.15.23:0/490737217 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b500a5f40 con 0x7f8b5013dd20 2020-11-20T11:32:26.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.198+0000 7f8b57593700 1 -- 172.21.15.23:0/490737217 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b5014da90 con 0x7f8b5013dd20 2020-11-20T11:32:26.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.199+0000 7f8b57593700 1 -- 172.21.15.23:0/490737217 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b5005c7d0 con 0x7f8b5013dd20 2020-11-20T11:32:26.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.199+0000 7f8b417fa700 1 -- 172.21.15.23:0/490737217 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8b48022070 con 0x7f8b5013dd20 2020-11-20T11:32:26.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.199+0000 7f8b417fa700 1 --2- 172.21.15.23:0/490737217 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8b28056d90 0x7f8b28059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:26.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.199+0000 7f8b417fa700 1 -- 172.21.15.23:0/490737217 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f8b48070890 con 0x7f8b5013dd20 2020-11-20T11:32:26.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.200+0000 7f8b5532f700 1 --2- 172.21.15.23:0/490737217 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8b28056d90 0x7f8b28059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:26.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.200+0000 7f8b5532f700 1 --2- 172.21.15.23:0/490737217 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8b28056d90 0x7f8b28059250 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f8b4c0055e0 tx=0x7f8b4c009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:26.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.203+0000 7f8b417fa700 1 -- 172.21.15.23:0/490737217 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8b48045b80 con 0x7f8b5013dd20 2020-11-20T11:32:26.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:26.402+0000 7f8b57593700 1 -- 172.21.15.23:0/490737217 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f8b5013c510 con 0x7f8b5013dd20 2020-11-20T11:32:27.117 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.114+0000 7f8b417fa700 1 -- 172.21.15.23:0/490737217 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v6) v1 ==== 118+0+0 (secure 0 0 0) 0x7f8b480160b0 con 0x7f8b5013dd20 2020-11-20T11:32:27.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.117+0000 7f8b57593700 1 -- 172.21.15.23:0/490737217 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8b28056d90 msgr2=0x7f8b28059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:27.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.117+0000 7f8b57593700 1 --2- 172.21.15.23:0/490737217 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8b28056d90 0x7f8b28059250 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f8b4c0055e0 tx=0x7f8b4c009040).stop 2020-11-20T11:32:27.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.117+0000 7f8b57593700 1 -- 172.21.15.23:0/490737217 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b5013dd20 msgr2=0x7f8b5013cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:27.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.117+0000 7f8b57593700 1 --2- 172.21.15.23:0/490737217 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b5013dd20 0x7f8b5013cce0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f8b4800ef50 tx=0x7f8b48009220).stop 2020-11-20T11:32:27.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.117+0000 7f8b57593700 1 -- 172.21.15.23:0/490737217 shutdown_connections 2020-11-20T11:32:27.121 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.117+0000 7f8b57593700 1 --2- 172.21.15.23:0/490737217 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b5013dd20 0x7f8b5013cce0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:27.121 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.117+0000 7f8b57593700 1 --2- 172.21.15.23:0/490737217 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8b500aaab0 0x7f8b50135d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:27.121 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.117+0000 7f8b57593700 1 --2- 172.21.15.23:0/490737217 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8b28056d90 0x7f8b28059250 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:27.121 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.117+0000 7f8b57593700 1 --2- 172.21.15.23:0/490737217 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8b500aae90 0x7f8b5013c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:27.122 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.117+0000 7f8b57593700 1 -- 172.21.15.23:0/490737217 >> 172.21.15.23:0/490737217 conn(0x7f8b500678e0 msgr2=0x7f8b50068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:27.122 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.117+0000 7f8b57593700 1 -- 172.21.15.23:0/490737217 shutdown_connections 2020-11-20T11:32:27.122 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.117+0000 7f8b57593700 1 -- 172.21.15.23:0/490737217 wait complete. 2020-11-20T11:32:27.134 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2020-11-20T11:32:27.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.254+0000 7fe63718d700 1 Processor -- start 2020-11-20T11:32:27.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.254+0000 7fe63718d700 1 -- start start 2020-11-20T11:32:27.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.255+0000 7fe63718d700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe6300aaab0 0x7fe63013ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:27.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.255+0000 7fe63718d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe6300aae90 0x7fe6301381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:27.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.255+0000 7fe63718d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe63013aeb0 0x7fe6300a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:27.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.255+0000 7fe63718d700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe63013e600 con 0x7fe6300aae90 2020-11-20T11:32:27.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.255+0000 7fe63718d700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe63013e8a0 con 0x7fe6300aaab0 2020-11-20T11:32:27.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.255+0000 7fe63718d700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6300ab6d0 con 0x7fe63013aeb0 2020-11-20T11:32:27.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.255+0000 7fe62ffff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe6300aae90 0x7fe6301381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:27.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.255+0000 7fe62ffff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe6300aae90 0x7fe6301381a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40648/0 (socket says 172.21.15.23:40648) 2020-11-20T11:32:27.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.255+0000 7fe62ffff700 1 -- 172.21.15.23:0/425556807 learned_addr learned my addr 172.21.15.23:0/425556807 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:27.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.255+0000 7fe63572a700 1 --2- 172.21.15.23:0/425556807 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe63013aeb0 0x7fe6300a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:27.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.255+0000 7fe634f29700 1 --2- 172.21.15.23:0/425556807 >> v2:172.21.15.155:3300/0 conn(0x7fe6300aaab0 0x7fe63013ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:27.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.255+0000 7fe62ffff700 1 -- 172.21.15.23:0/425556807 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe63013aeb0 msgr2=0x7fe6300a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:27.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.255+0000 7fe62ffff700 1 --2- 172.21.15.23:0/425556807 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe63013aeb0 0x7fe6300a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:27.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.255+0000 7fe62ffff700 1 -- 172.21.15.23:0/425556807 >> v2:172.21.15.155:3300/0 conn(0x7fe6300aaab0 msgr2=0x7fe63013ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:27.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.255+0000 7fe62ffff700 1 --2- 172.21.15.23:0/425556807 >> v2:172.21.15.155:3300/0 conn(0x7fe6300aaab0 0x7fe63013ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:27.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.255+0000 7fe62ffff700 1 -- 172.21.15.23:0/425556807 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe63013f230 con 0x7fe6300aae90 2020-11-20T11:32:27.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.255+0000 7fe62ffff700 1 --2- 172.21.15.23:0/425556807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6300aae90 0x7fe6301381a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fe6300b23d0 tx=0x7fe61c006ef0).ready entity=mon.0 client_cookie=c1dcaf04775cfa7f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:27.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.255+0000 7fe62f7fe700 1 -- 172.21.15.23:0/425556807 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61c00b860 con 0x7fe6300aae90 2020-11-20T11:32:27.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.255+0000 7fe62f7fe700 1 -- 172.21.15.23:0/425556807 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe61c000e90 con 0x7fe6300aae90 2020-11-20T11:32:27.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.255+0000 7fe62f7fe700 1 -- 172.21.15.23:0/425556807 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61c00bbd0 con 0x7fe6300aae90 2020-11-20T11:32:27.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.256+0000 7fe63718d700 1 -- 172.21.15.23:0/425556807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6300aae90 msgr2=0x7fe6301381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:27.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.256+0000 7fe63718d700 1 --2- 172.21.15.23:0/425556807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6300aae90 0x7fe6301381a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fe6300b23d0 tx=0x7fe61c006ef0).stop 2020-11-20T11:32:27.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.256+0000 7fe63718d700 1 -- 172.21.15.23:0/425556807 shutdown_connections 2020-11-20T11:32:27.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.256+0000 7fe63718d700 1 --2- 172.21.15.23:0/425556807 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe63013aeb0 0x7fe6300a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:27.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.256+0000 7fe63718d700 1 --2- 172.21.15.23:0/425556807 >> v2:172.21.15.155:3300/0 conn(0x7fe6300aaab0 0x7fe63013ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:27.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.256+0000 7fe63718d700 1 --2- 172.21.15.23:0/425556807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6300aae90 0x7fe6301381a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:27.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.256+0000 7fe63718d700 1 -- 172.21.15.23:0/425556807 >> 172.21.15.23:0/425556807 conn(0x7fe6300678e0 msgr2=0x7fe630067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:27.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.256+0000 7fe63718d700 1 -- 172.21.15.23:0/425556807 shutdown_connections 2020-11-20T11:32:27.266 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.256+0000 7fe63718d700 1 -- 172.21.15.23:0/425556807 wait complete. 2020-11-20T11:32:27.266 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.257+0000 7fe63718d700 1 Processor -- start 2020-11-20T11:32:27.266 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.257+0000 7fe63718d700 1 -- start start 2020-11-20T11:32:27.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.258+0000 7fe63718d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6300aaab0 0x7fe6300a73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:27.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.258+0000 7fe63718d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6300aae90 0x7fe6300a7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:27.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.258+0000 7fe63718d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6300a58c0 0x7fe63014a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:27.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.258+0000 7fe63718d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6300a80d0 con 0x7fe6300a58c0 2020-11-20T11:32:27.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.258+0000 7fe63718d700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6300a5dd0 con 0x7fe6300aae90 2020-11-20T11:32:27.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.258+0000 7fe63718d700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6300b0250 con 0x7fe6300aaab0 2020-11-20T11:32:27.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.258+0000 7fe63572a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6300a58c0 0x7fe63014a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:27.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.258+0000 7fe63572a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6300a58c0 0x7fe63014a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40658/0 (socket says 172.21.15.23:40658) 2020-11-20T11:32:27.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.258+0000 7fe63572a700 1 -- 172.21.15.23:0/3194377556 learned_addr learned my addr 172.21.15.23:0/3194377556 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:27.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.258+0000 7fe634f29700 1 --2- 172.21.15.23:0/3194377556 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6300aaab0 0x7fe6300a73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:27.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.258+0000 7fe62ffff700 1 --2- 172.21.15.23:0/3194377556 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6300aae90 0x7fe6300a7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:27.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.258+0000 7fe63572a700 1 -- 172.21.15.23:0/3194377556 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6300aaab0 msgr2=0x7fe6300a73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:27.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.258+0000 7fe63572a700 1 --2- 172.21.15.23:0/3194377556 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6300aaab0 0x7fe6300a73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:27.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.258+0000 7fe63572a700 1 -- 172.21.15.23:0/3194377556 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6300aae90 msgr2=0x7fe6300a7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:27.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.258+0000 7fe63572a700 1 --2- 172.21.15.23:0/3194377556 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6300aae90 0x7fe6300a7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:27.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.258+0000 7fe63572a700 1 -- 172.21.15.23:0/3194377556 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6300b04f0 con 0x7fe6300a58c0 2020-11-20T11:32:27.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.258+0000 7fe634f29700 1 --2- 172.21.15.23:0/3194377556 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6300aaab0 0x7fe6300a73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:27.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.258+0000 7fe63572a700 1 --2- 172.21.15.23:0/3194377556 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6300a58c0 0x7fe63014a0b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fe62800dbb0 tx=0x7fe628008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:27.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.258+0000 7fe62cff9700 1 -- 172.21.15.23:0/3194377556 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe628009940 con 0x7fe6300a58c0 2020-11-20T11:32:27.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.258+0000 7fe62cff9700 1 -- 172.21.15.23:0/3194377556 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe628009aa0 con 0x7fe6300a58c0 2020-11-20T11:32:27.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.258+0000 7fe63718d700 1 -- 172.21.15.23:0/3194377556 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6300b0990 con 0x7fe6300a58c0 2020-11-20T11:32:27.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.258+0000 7fe62cff9700 1 -- 172.21.15.23:0/3194377556 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe628009e40 con 0x7fe6300a58c0 2020-11-20T11:32:27.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.258+0000 7fe63718d700 1 -- 172.21.15.23:0/3194377556 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe63014a7c0 con 0x7fe6300a58c0 2020-11-20T11:32:27.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.259+0000 7fe63718d700 1 -- 172.21.15.23:0/3194377556 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe63005c7d0 con 0x7fe6300a58c0 2020-11-20T11:32:27.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.260+0000 7fe62cff9700 1 -- 172.21.15.23:0/3194377556 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe628013690 con 0x7fe6300a58c0 2020-11-20T11:32:27.274 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.260+0000 7fe62cff9700 1 --2- 172.21.15.23:0/3194377556 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe608056d90 0x7fe608059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:27.274 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.260+0000 7fe62cff9700 1 -- 172.21.15.23:0/3194377556 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fe6280706f0 con 0x7fe6300a58c0 2020-11-20T11:32:27.274 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.260+0000 7fe634f29700 1 --2- 172.21.15.23:0/3194377556 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe608056d90 0x7fe608059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:27.274 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.261+0000 7fe634f29700 1 --2- 172.21.15.23:0/3194377556 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe608056d90 0x7fe608059250 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fe624006fd0 tx=0x7fe624008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:27.274 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.263+0000 7fe62cff9700 1 -- 172.21.15.23:0/3194377556 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe6280459e0 con 0x7fe6300a58c0 2020-11-20T11:32:27.470 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:27.467+0000 7fe63718d700 1 -- 172.21.15.23:0/3194377556 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7fe630044710 con 0x7fe6300a58c0 2020-11-20T11:32:28.129 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.126+0000 7fe62cff9700 1 -- 172.21.15.23:0/3194377556 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v7) v1 ==== 115+0+0 (secure 0 0 0) 0x7fe628049000 con 0x7fe6300a58c0 2020-11-20T11:32:28.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.128+0000 7fe63718d700 1 -- 172.21.15.23:0/3194377556 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe608056d90 msgr2=0x7fe608059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:28.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.128+0000 7fe63718d700 1 --2- 172.21.15.23:0/3194377556 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe608056d90 0x7fe608059250 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fe624006fd0 tx=0x7fe624008040).stop 2020-11-20T11:32:28.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.128+0000 7fe63718d700 1 -- 172.21.15.23:0/3194377556 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6300a58c0 msgr2=0x7fe63014a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:28.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.128+0000 7fe63718d700 1 --2- 172.21.15.23:0/3194377556 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6300a58c0 0x7fe63014a0b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fe62800dbb0 tx=0x7fe628008040).stop 2020-11-20T11:32:28.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.128+0000 7fe63718d700 1 -- 172.21.15.23:0/3194377556 shutdown_connections 2020-11-20T11:32:28.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.128+0000 7fe63718d700 1 --2- 172.21.15.23:0/3194377556 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6300a58c0 0x7fe63014a0b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:28.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.128+0000 7fe63718d700 1 --2- 172.21.15.23:0/3194377556 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe608056d90 0x7fe608059250 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:28.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.128+0000 7fe63718d700 1 --2- 172.21.15.23:0/3194377556 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6300aaab0 0x7fe6300a73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:28.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.128+0000 7fe63718d700 1 --2- 172.21.15.23:0/3194377556 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6300aae90 0x7fe6300a7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:28.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.128+0000 7fe63718d700 1 -- 172.21.15.23:0/3194377556 >> 172.21.15.23:0/3194377556 conn(0x7fe6300678e0 msgr2=0x7fe630136690 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:28.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.129+0000 7fe63718d700 1 -- 172.21.15.23:0/3194377556 shutdown_connections 2020-11-20T11:32:28.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.129+0000 7fe63718d700 1 -- 172.21.15.23:0/3194377556 wait complete. 2020-11-20T11:32:28.145 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T11:32:28.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.263+0000 7f2daed7c700 1 Processor -- start 2020-11-20T11:32:28.266 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.263+0000 7f2daed7c700 1 -- start start 2020-11-20T11:32:28.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.263+0000 7f2daed7c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f2da8060a80 0x7f2da8061950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:28.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.263+0000 7f2daed7c700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2da80a9d40 0x7f2da80aa1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:28.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.263+0000 7f2daed7c700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f2da8137920 0x7f2da8137db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:28.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.263+0000 7f2daed7c700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2da8067bc0 con 0x7f2da8060a80 2020-11-20T11:32:28.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.263+0000 7f2daed7c700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2da8067e60 con 0x7f2da8137920 2020-11-20T11:32:28.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.263+0000 7f2daed7c700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2da8068100 con 0x7f2da80a9d40 2020-11-20T11:32:28.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.264+0000 7f2dacb18700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f2da8060a80 0x7f2da8061950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:28.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.264+0000 7f2dacb18700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f2da8060a80 0x7f2da8061950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40662/0 (socket says 172.21.15.23:40662) 2020-11-20T11:32:28.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.264+0000 7f2dacb18700 1 -- 172.21.15.23:0/2112139589 learned_addr learned my addr 172.21.15.23:0/2112139589 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:28.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.264+0000 7f2dad319700 1 --2- 172.21.15.23:0/2112139589 >> v2:172.21.15.155:3300/0 conn(0x7f2da8137920 0x7f2da8137db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:28.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.264+0000 7f2da7fff700 1 --2- 172.21.15.23:0/2112139589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2da80a9d40 0x7f2da80aa1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:28.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.264+0000 7f2dacb18700 1 -- 172.21.15.23:0/2112139589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2da80a9d40 msgr2=0x7f2da80aa1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:28.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.264+0000 7f2dacb18700 1 --2- 172.21.15.23:0/2112139589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2da80a9d40 0x7f2da80aa1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:28.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.264+0000 7f2dacb18700 1 -- 172.21.15.23:0/2112139589 >> v2:172.21.15.155:3300/0 conn(0x7f2da8137920 msgr2=0x7f2da8137db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:28.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.264+0000 7f2dacb18700 1 --2- 172.21.15.23:0/2112139589 >> v2:172.21.15.155:3300/0 conn(0x7f2da8137920 0x7f2da8137db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:28.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.264+0000 7f2dacb18700 1 -- 172.21.15.23:0/2112139589 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da8060e60 con 0x7f2da8060a80 2020-11-20T11:32:28.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.264+0000 7f2dad319700 1 --2- 172.21.15.23:0/2112139589 >> v2:172.21.15.155:3300/0 conn(0x7f2da8137920 0x7f2da8137db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:28.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.264+0000 7f2da7fff700 1 --2- 172.21.15.23:0/2112139589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2da80a9d40 0x7f2da80aa1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:28.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.264+0000 7f2dacb18700 1 --2- 172.21.15.23:0/2112139589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2da8060a80 0x7f2da8061950 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f2da80616e0 tx=0x7f2d9c006ef0).ready entity=mon.0 client_cookie=3045cd5aac436079 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:28.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.264+0000 7f2da77fe700 1 -- 172.21.15.23:0/2112139589 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c00b860 con 0x7f2da8060a80 2020-11-20T11:32:28.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.264+0000 7f2da77fe700 1 -- 172.21.15.23:0/2112139589 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d9c000e90 con 0x7f2da8060a80 2020-11-20T11:32:28.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.264+0000 7f2da77fe700 1 -- 172.21.15.23:0/2112139589 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c00bbd0 con 0x7f2da8060a80 2020-11-20T11:32:28.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.265+0000 7f2daed7c700 1 -- 172.21.15.23:0/2112139589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2da8060a80 msgr2=0x7f2da8061950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:28.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.265+0000 7f2daed7c700 1 --2- 172.21.15.23:0/2112139589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2da8060a80 0x7f2da8061950 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f2da80616e0 tx=0x7f2d9c006ef0).stop 2020-11-20T11:32:28.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.265+0000 7f2daed7c700 1 -- 172.21.15.23:0/2112139589 shutdown_connections 2020-11-20T11:32:28.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.265+0000 7f2daed7c700 1 --2- 172.21.15.23:0/2112139589 >> v2:172.21.15.155:3300/0 conn(0x7f2da8137920 0x7f2da8137db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:28.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.265+0000 7f2daed7c700 1 --2- 172.21.15.23:0/2112139589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2da8060a80 0x7f2da8061950 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:28.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.265+0000 7f2daed7c700 1 --2- 172.21.15.23:0/2112139589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2da80a9d40 0x7f2da80aa1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:28.274 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.265+0000 7f2daed7c700 1 -- 172.21.15.23:0/2112139589 >> 172.21.15.23:0/2112139589 conn(0x7f2da8066730 msgr2=0x7f2da80601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:28.274 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.265+0000 7f2daed7c700 1 -- 172.21.15.23:0/2112139589 shutdown_connections 2020-11-20T11:32:28.274 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.265+0000 7f2daed7c700 1 -- 172.21.15.23:0/2112139589 wait complete. 2020-11-20T11:32:28.274 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.266+0000 7f2daed7c700 1 Processor -- start 2020-11-20T11:32:28.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.266+0000 7f2daed7c700 1 -- start start 2020-11-20T11:32:28.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.266+0000 7f2daed7c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2da8060a80 0x7f2da80a6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:28.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.266+0000 7f2daed7c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2da80a9d40 0x7f2da80a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:28.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.266+0000 7f2daed7c700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2da8137920 0x7f2da814d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:28.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.267+0000 7f2daed7c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da813e650 con 0x7f2da8060a80 2020-11-20T11:32:28.278 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.267+0000 7f2daed7c700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da813d860 con 0x7f2da80a9d40 2020-11-20T11:32:28.278 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.267+0000 7f2daed7c700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2da813e880 con 0x7f2da8137920 2020-11-20T11:32:28.278 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.267+0000 7f2dacb18700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2da8060a80 0x7f2da80a6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:28.278 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.267+0000 7f2dacb18700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2da8060a80 0x7f2da80a6520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40670/0 (socket says 172.21.15.23:40670) 2020-11-20T11:32:28.278 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.267+0000 7f2dacb18700 1 -- 172.21.15.23:0/811733368 learned_addr learned my addr 172.21.15.23:0/811733368 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:28.279 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.267+0000 7f2da7fff700 1 --2- 172.21.15.23:0/811733368 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2da80a9d40 0x7f2da80a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:28.279 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.267+0000 7f2dad319700 1 --2- 172.21.15.23:0/811733368 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2da8137920 0x7f2da814d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:28.279 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.267+0000 7f2dacb18700 1 -- 172.21.15.23:0/811733368 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2da8137920 msgr2=0x7f2da814d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:28.279 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.267+0000 7f2dacb18700 1 --2- 172.21.15.23:0/811733368 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2da8137920 0x7f2da814d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:28.279 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.267+0000 7f2dacb18700 1 -- 172.21.15.23:0/811733368 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2da80a9d40 msgr2=0x7f2da80a87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:28.280 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.267+0000 7f2dacb18700 1 --2- 172.21.15.23:0/811733368 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2da80a9d40 0x7f2da80a87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:28.280 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.267+0000 7f2dacb18700 1 -- 172.21.15.23:0/811733368 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da813eb20 con 0x7f2da8060a80 2020-11-20T11:32:28.280 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.267+0000 7f2dacb18700 1 --2- 172.21.15.23:0/811733368 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2da8060a80 0x7f2da80a6520 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f2d9c00b3c0 tx=0x7f2d9c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:28.280 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.267+0000 7f2da4ff9700 1 -- 172.21.15.23:0/811733368 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c011cd0 con 0x7f2da8060a80 2020-11-20T11:32:28.280 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.267+0000 7f2da4ff9700 1 -- 172.21.15.23:0/811733368 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d9c011e30 con 0x7f2da8060a80 2020-11-20T11:32:28.281 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.267+0000 7f2daed7c700 1 -- 172.21.15.23:0/811733368 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2da80a6a10 con 0x7f2da8060a80 2020-11-20T11:32:28.281 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.267+0000 7f2da4ff9700 1 -- 172.21.15.23:0/811733368 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9c01c6a0 con 0x7f2da8060a80 2020-11-20T11:32:28.281 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.267+0000 7f2daed7c700 1 -- 172.21.15.23:0/811733368 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2da813efc0 con 0x7f2da8060a80 2020-11-20T11:32:28.281 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.268+0000 7f2daed7c700 1 -- 172.21.15.23:0/811733368 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2da813f1b0 con 0x7f2da8060a80 2020-11-20T11:32:28.281 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.268+0000 7f2da4ff9700 1 -- 172.21.15.23:0/811733368 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2d9c017070 con 0x7f2da8060a80 2020-11-20T11:32:28.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.269+0000 7f2da4ff9700 1 --2- 172.21.15.23:0/811733368 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2d80056d60 0x7f2d80059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:28.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.269+0000 7f2da4ff9700 1 -- 172.21.15.23:0/811733368 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f2d9c071b40 con 0x7f2da8060a80 2020-11-20T11:32:28.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.269+0000 7f2da7fff700 1 --2- 172.21.15.23:0/811733368 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2d80056d60 0x7f2d80059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:28.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.270+0000 7f2da7fff700 1 --2- 172.21.15.23:0/811733368 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2d80056d60 0x7f2d80059220 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f2d94006c10 tx=0x7f2d94008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:28.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.273+0000 7f2da4ff9700 1 -- 172.21.15.23:0/811733368 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2d9c046e30 con 0x7f2da8060a80 2020-11-20T11:32:28.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.473+0000 7f2daed7c700 1 -- 172.21.15.23:0/811733368 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2da8044710 con 0x7f2da8060a80 2020-11-20T11:32:28.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.474+0000 7f2da4ff9700 1 -- 172.21.15.23:0/811733368 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+2128 (secure 0 0 0) 0x7f2d9c015070 con 0x7f2da8060a80 2020-11-20T11:32:28.477 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:32:28.477 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4223,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/564012815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":564012815},{"type":"v1","addr":"172.21.15.155:6835","nonce":564012815}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T11:32:25.469149+0000","modified":"2020-11-20T11:32:28.118344+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.23:6833/919963501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":919963501},{"type":"v1","addr":"172.21.15.23:6833","nonce":919963501}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T11:32:28.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.475+0000 7f2daed7c700 1 -- 172.21.15.23:0/811733368 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2d80056d60 msgr2=0x7f2d80059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:28.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.475+0000 7f2daed7c700 1 --2- 172.21.15.23:0/811733368 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2d80056d60 0x7f2d80059220 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f2d94006c10 tx=0x7f2d94008040).stop 2020-11-20T11:32:28.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.475+0000 7f2daed7c700 1 -- 172.21.15.23:0/811733368 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2da8060a80 msgr2=0x7f2da80a6520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:28.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.475+0000 7f2daed7c700 1 --2- 172.21.15.23:0/811733368 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2da8060a80 0x7f2da80a6520 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f2d9c00b3c0 tx=0x7f2d9c016000).stop 2020-11-20T11:32:28.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.475+0000 7f2daed7c700 1 -- 172.21.15.23:0/811733368 shutdown_connections 2020-11-20T11:32:28.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.475+0000 7f2daed7c700 1 --2- 172.21.15.23:0/811733368 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2d80056d60 0x7f2d80059220 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:28.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.475+0000 7f2daed7c700 1 --2- 172.21.15.23:0/811733368 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2da8137920 0x7f2da814d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:28.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.475+0000 7f2daed7c700 1 --2- 172.21.15.23:0/811733368 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2da8060a80 0x7f2da80a6520 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:28.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.475+0000 7f2daed7c700 1 --2- 172.21.15.23:0/811733368 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2da80a9d40 0x7f2da80a87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:28.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.475+0000 7f2daed7c700 1 -- 172.21.15.23:0/811733368 >> 172.21.15.23:0/811733368 conn(0x7f2da8066730 msgr2=0x7f2da81369a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:28.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.476+0000 7f2daed7c700 1 -- 172.21.15.23:0/811733368 shutdown_connections 2020-11-20T11:32:28.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.476+0000 7f2daed7c700 1 -- 172.21.15.23:0/811733368 wait complete. 2020-11-20T11:32:28.481 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 7 2020-11-20T11:32:28.490 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T11:32:28.603 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.600+0000 7f2d0d504700 1 Processor -- start 2020-11-20T11:32:28.603 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.600+0000 7f2d0d504700 1 -- start start 2020-11-20T11:32:28.606 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.601+0000 7f2d0d504700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f2d080ab2a0 0x7f2d0813e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:28.606 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.601+0000 7f2d0d504700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2d080ad8d0 0x7f2d080a8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:28.606 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.601+0000 7f2d0d504700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d080b1a20 con 0x7f2d080a4510 2020-11-20T11:32:28.607 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.601+0000 7f2d0d504700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d0813f4a0 con 0x7f2d080ab2a0 2020-11-20T11:32:28.607 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.601+0000 7f2d0d504700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d080aa0e0 con 0x7f2d080ad8d0 2020-11-20T11:32:28.607 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.601+0000 7f2d06ffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f2d080ab2a0 0x7f2d0813e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:28.607 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.601+0000 7f2d077fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f2d080a4510 0x7f2d080a49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:54930/0 (socket says 172.21.15.23:54930) 2020-11-20T11:32:28.607 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.601+0000 7f2d077fe700 1 -- 172.21.15.23:0/1341151631 learned_addr learned my addr 172.21.15.23:0/1341151631 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:28.608 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.601+0000 7f2d067fc700 1 --2- 172.21.15.23:0/1341151631 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2d080ad8d0 0x7f2d080a8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:28.608 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.601+0000 7f2d06ffd700 1 -- 172.21.15.23:0/1341151631 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2d080ad8d0 msgr2=0x7f2d080a8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:28.608 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.601+0000 7f2d06ffd700 1 --2- 172.21.15.23:0/1341151631 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2d080ad8d0 0x7f2d080a8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:28.608 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.601+0000 7f2d06ffd700 1 -- 172.21.15.23:0/1341151631 >> v1:172.21.15.23:6789/0 conn(0x7f2d080a4510 legacy=0x7f2d080a49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:28.609 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.601+0000 7f2d06ffd700 1 -- 172.21.15.23:0/1341151631 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d080a8b90 con 0x7f2d080ab2a0 2020-11-20T11:32:28.609 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.601+0000 7f2d05ffb700 1 -- 172.21.15.23:0/1341151631 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 87160199 0 0) 0x7f2d080b1a20 con 0x7f2d080a4510 2020-11-20T11:32:28.609 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.602+0000 7f2d06ffd700 1 --2- 172.21.15.23:0/1341151631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d080ab2a0 0x7f2d0813e570 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f2cfc006fd0 tx=0x7f2cfc006e30).ready entity=mon.0 client_cookie=fce04765dd61ab02 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:28.609 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.602+0000 7f2d05ffb700 1 -- 172.21.15.23:0/1341151631 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc0097e0 con 0x7f2d080ab2a0 2020-11-20T11:32:28.609 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.602+0000 7f2d05ffb700 1 -- 172.21.15.23:0/1341151631 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cfc000e90 con 0x7f2d080ab2a0 2020-11-20T11:32:28.610 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.602+0000 7f2d05ffb700 1 -- 172.21.15.23:0/1341151631 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc009b50 con 0x7f2d080ab2a0 2020-11-20T11:32:28.610 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.602+0000 7f2d0d504700 1 -- 172.21.15.23:0/1341151631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d080ab2a0 msgr2=0x7f2d0813e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:28.610 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.602+0000 7f2d0d504700 1 --2- 172.21.15.23:0/1341151631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d080ab2a0 0x7f2d0813e570 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f2cfc006fd0 tx=0x7f2cfc006e30).stop 2020-11-20T11:32:28.610 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.602+0000 7f2d0d504700 1 -- 172.21.15.23:0/1341151631 shutdown_connections 2020-11-20T11:32:28.610 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.602+0000 7f2d0d504700 1 --2- 172.21.15.23:0/1341151631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d080ab2a0 0x7f2d0813e570 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:28.611 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.602+0000 7f2d0d504700 1 --2- 172.21.15.23:0/1341151631 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2d080ad8d0 0x7f2d080a8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:28.611 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.602+0000 7f2d0d504700 1 -- 172.21.15.23:0/1341151631 >> 172.21.15.23:0/1341151631 conn(0x7f2d080683b0 msgr2=0x7f2d080b1310 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:28.611 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.602+0000 7f2d0d504700 1 -- 172.21.15.23:0/1341151631 shutdown_connections 2020-11-20T11:32:28.611 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.602+0000 7f2d0d504700 1 -- 172.21.15.23:0/1341151631 wait complete. 2020-11-20T11:32:28.614 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.603+0000 7f2d0d504700 1 Processor -- start 2020-11-20T11:32:28.614 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.603+0000 7f2d0d504700 1 -- start start 2020-11-20T11:32:28.614 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.604+0000 7f2d0d504700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d080a4510 0x7f2d081419c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:28.614 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.604+0000 7f2d0d504700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d080ab2a0 0x7f2d0809f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:28.615 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.604+0000 7f2d0d504700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2d080ad8d0 0x7f2d0814a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:28.615 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.604+0000 7f2d0d504700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d080a0c60 con 0x7f2d080a4510 2020-11-20T11:32:28.615 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.604+0000 7f2d0d504700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0809f0d0 con 0x7f2d080ab2a0 2020-11-20T11:32:28.615 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.604+0000 7f2d0d504700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0809f370 con 0x7f2d080ad8d0 2020-11-20T11:32:28.615 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.604+0000 7f2d06ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d080a4510 0x7f2d081419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:28.616 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.604+0000 7f2d06ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d080a4510 0x7f2d081419c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40682/0 (socket says 172.21.15.23:40682) 2020-11-20T11:32:28.616 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.604+0000 7f2d06ffd700 1 -- 172.21.15.23:0/634325773 learned_addr learned my addr 172.21.15.23:0/634325773 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:28.616 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.604+0000 7f2d06ffd700 1 -- 172.21.15.23:0/634325773 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2d080ad8d0 msgr2=0x7f2d0814a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:28.616 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.604+0000 7f2d06ffd700 1 --2- 172.21.15.23:0/634325773 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2d080ad8d0 0x7f2d0814a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:32:28.617 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.604+0000 7f2d06ffd700 1 -- 172.21.15.23:0/634325773 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d080ab2a0 msgr2=0x7f2d0809f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:28.617 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.604+0000 7f2d06ffd700 1 --2- 172.21.15.23:0/634325773 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d080ab2a0 0x7f2d0809f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:32:28.617 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.604+0000 7f2d06ffd700 1 -- 172.21.15.23:0/634325773 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d0809fd00 con 0x7f2d080a4510 2020-11-20T11:32:28.617 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.604+0000 7f2d06ffd700 1 --2- 172.21.15.23:0/634325773 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d080a4510 0x7f2d081419c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f2cfc006e70 tx=0x7f2cfc015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:28.618 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.605+0000 7f2ceeffd700 1 -- 172.21.15.23:0/634325773 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc011d70 con 0x7f2d080a4510 2020-11-20T11:32:28.618 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.605+0000 7f2ceeffd700 1 -- 172.21.15.23:0/634325773 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cfc01e460 con 0x7f2d080a4510 2020-11-20T11:32:28.618 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.605+0000 7f2ceeffd700 1 -- 172.21.15.23:0/634325773 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc01e7e0 con 0x7f2d080a4510 2020-11-20T11:32:28.618 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.605+0000 7f2d0d504700 1 -- 172.21.15.23:0/634325773 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d0814a0e0 con 0x7f2d080a4510 2020-11-20T11:32:28.618 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.605+0000 7f2d0d504700 1 -- 172.21.15.23:0/634325773 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d0814aca0 con 0x7f2d080a4510 2020-11-20T11:32:28.619 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.606+0000 7f2d0d504700 1 -- 172.21.15.23:0/634325773 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d08060640 con 0x7f2d080a4510 2020-11-20T11:32:28.619 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.606+0000 7f2ceeffd700 1 -- 172.21.15.23:0/634325773 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2cfc025030 con 0x7f2d080a4510 2020-11-20T11:32:28.619 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.606+0000 7f2ceeffd700 1 --2- 172.21.15.23:0/634325773 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2ce4056d60 0x7f2ce4059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:28.619 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.606+0000 7f2ceeffd700 1 -- 172.21.15.23:0/634325773 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f2cfc071720 con 0x7f2d080a4510 2020-11-20T11:32:28.619 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.607+0000 7f2d067fc700 1 --2- 172.21.15.23:0/634325773 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2ce4056d60 0x7f2ce4059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:28.620 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.607+0000 7f2d067fc700 1 --2- 172.21.15.23:0/634325773 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2ce4056d60 0x7f2ce4059220 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f2d08142130 tx=0x7f2cf8008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:28.620 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.610+0000 7f2ceeffd700 1 -- 172.21.15.23:0/634325773 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2cfc046ac0 con 0x7f2d080a4510 2020-11-20T11:32:28.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.762+0000 7f2d0d504700 1 -- 172.21.15.23:0/634325773 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d0809f530 con 0x7f2d080a4510 2020-11-20T11:32:28.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.763+0000 7f2ceeffd700 1 -- 172.21.15.23:0/634325773 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17087 (secure 0 0 0) 0x7f2cfc0312d0 con 0x7f2d080a4510 2020-11-20T11:32:28.766 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:32:28.766 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":25,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","created":"2020-11-20T11:32:06.136679+0000","modified":"2020-11-20T11:32:26.470927+0000","last_up_change":"2020-11-20T11:32:13.536194+0000","last_in_change":"2020-11-20T11:32:13.536194+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T11:32:09.559868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T11:32:17.000782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T11:32:20.423341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T11:32:23.126273+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e9d219f5-e40e-4b2b-9468-deb9c4074a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":23083},{"type":"v1","addr":"172.21.15.23:6801","nonce":23083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":23083},{"type":"v1","addr":"172.21.15.23:6803","nonce":23083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":23083},{"type":"v1","addr":"172.21.15.23:6807","nonce":23083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":23083},{"type":"v1","addr":"172.21.15.23:6805","nonce":23083}]},"public_addr":"172.21.15.23:6801/23083","cluster_addr":"172.21.15.23:6803/23083","heartbeat_back_addr":"172.21.15.23:6807/23083","heartbeat_front_addr":"172.21.15.23:6805/23083","state":["exists","up"]},{"osd":1,"uuid":"2328adde-2108-4fe0-8138-f0d096a496a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":23099},{"type":"v1","addr":"172.21.15.23:6817","nonce":23099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":23099},{"type":"v1","addr":"172.21.15.23:6819","nonce":23099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":23099},{"type":"v1","addr":"172.21.15.23:6824","nonce":23099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":23099},{"type":"v1","addr":"172.21.15.23:6821","nonce":23099}]},"public_addr":"172.21.15.23:6817/23099","cluster_addr":"172.21.15.23:6819/23099","heartbeat_back_addr":"172.21.15.23:6824/23099","heartbeat_front_addr":"172.21.15.23:6821/23099","state":["exists","up"]},{"osd":2,"uuid":"13367708-59a2-4363-8a3a-ab0cb97145ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":23084},{"type":"v1","addr":"172.21.15.23:6809","nonce":23084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":23084},{"type":"v1","addr":"172.21.15.23:6811","nonce":23084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":23084},{"type":"v1","addr":"172.21.15.23:6815","nonce":23084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":23084},{"type":"v1","addr":"172.21.15.23:6813","nonce":23084}]},"public_addr":"172.21.15.23:6809/23084","cluster_addr":"172.21.15.23:6811/23084","heartbeat_back_addr":"172.21.15.23:6815/23084","heartbeat_front_addr":"172.21.15.23:6813/23084","state":["exists","up"]},{"osd":3,"uuid":"1f147dd7-9289-47b0-ad1c-4088618d6faa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6823","nonce":23114},{"type":"v1","addr":"172.21.15.23:6825","nonce":23114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":23114},{"type":"v1","addr":"172.21.15.23:6827","nonce":23114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":23114},{"type":"v1","addr":"172.21.15.23:6831","nonce":23114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":23114},{"type":"v1","addr":"172.21.15.23:6829","nonce":23114}]},"public_addr":"172.21.15.23:6825/23114","cluster_addr":"172.21.15.23:6827/23114","heartbeat_back_addr":"172.21.15.23:6831/23114","heartbeat_front_addr":"172.21.15.23:6829/23114","state":["exists","up"]},{"osd":4,"uuid":"d008b039-4bc5-4135-9678-1511e58efc29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":23280},{"type":"v1","addr":"172.21.15.155:6801","nonce":23280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":23280},{"type":"v1","addr":"172.21.15.155:6803","nonce":23280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":23280},{"type":"v1","addr":"172.21.15.155:6807","nonce":23280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":23280},{"type":"v1","addr":"172.21.15.155:6805","nonce":23280}]},"public_addr":"172.21.15.155:6801/23280","cluster_addr":"172.21.15.155:6803/23280","heartbeat_back_addr":"172.21.15.155:6807/23280","heartbeat_front_addr":"172.21.15.155:6805/23280","state":["exists","up"]},{"osd":5,"uuid":"56715e6f-cf6e-44a6-9e39-93ab1224095f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":23298},{"type":"v1","addr":"172.21.15.155:6817","nonce":23298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":23298},{"type":"v1","addr":"172.21.15.155:6819","nonce":23298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":23298},{"type":"v1","addr":"172.21.15.155:6823","nonce":23298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":23298},{"type":"v1","addr":"172.21.15.155:6821","nonce":23298}]},"public_addr":"172.21.15.155:6817/23298","cluster_addr":"172.21.15.155:6819/23298","heartbeat_back_addr":"172.21.15.155:6823/23298","heartbeat_front_addr":"172.21.15.155:6821/23298","state":["exists","up"]},{"osd":6,"uuid":"db1e61ca-8e14-4a1f-8607-6e01b148d61c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":23307},{"type":"v1","addr":"172.21.15.155:6825","nonce":23307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":23307},{"type":"v1","addr":"172.21.15.155:6827","nonce":23307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":23307},{"type":"v1","addr":"172.21.15.155:6831","nonce":23307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":23307},{"type":"v1","addr":"172.21.15.155:6829","nonce":23307}]},"public_addr":"172.21.15.155:6825/23307","cluster_addr":"172.21.15.155:6827/23307","heartbeat_back_addr":"172.21.15.155:6831/23307","heartbeat_front_addr":"172.21.15.155:6829/23307","state":["exists","up"]},{"osd":7,"uuid":"d44419be-ef89-462c-8cf1-3c6db2a6944b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":23281},{"type":"v1","addr":"172.21.15.155:6809","nonce":23281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":23281},{"type":"v1","addr":"172.21.15.155:6811","nonce":23281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":23281},{"type":"v1","addr":"172.21.15.155:6815","nonce":23281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":23281},{"type":"v1","addr":"172.21.15.155:6813","nonce":23281}]},"public_addr":"172.21.15.155:6809/23281","cluster_addr":"172.21.15.155:6811/23281","heartbeat_back_addr":"172.21.15.155:6815/23281","heartbeat_front_addr":"172.21.15.155:6813/23281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.486702+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.509396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.533414+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.509489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.467600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.513068+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.536636+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.531772+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T11:32:28.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.764+0000 7f2d0d504700 1 -- 172.21.15.23:0/634325773 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2ce4056d60 msgr2=0x7f2ce4059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:28.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.764+0000 7f2d0d504700 1 --2- 172.21.15.23:0/634325773 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2ce4056d60 0x7f2ce4059220 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f2d08142130 tx=0x7f2cf8008040).stop 2020-11-20T11:32:28.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.764+0000 7f2d0d504700 1 -- 172.21.15.23:0/634325773 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d080a4510 msgr2=0x7f2d081419c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:28.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.764+0000 7f2d0d504700 1 --2- 172.21.15.23:0/634325773 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d080a4510 0x7f2d081419c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f2cfc006e70 tx=0x7f2cfc015000).stop 2020-11-20T11:32:28.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.764+0000 7f2d0d504700 1 -- 172.21.15.23:0/634325773 shutdown_connections 2020-11-20T11:32:28.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.764+0000 7f2d0d504700 1 --2- 172.21.15.23:0/634325773 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2ce4056d60 0x7f2ce4059220 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:28.769 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.764+0000 7f2d0d504700 1 --2- 172.21.15.23:0/634325773 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2d080ad8d0 0x7f2d0814a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:32:28.769 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.764+0000 7f2d0d504700 1 --2- 172.21.15.23:0/634325773 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d080a4510 0x7f2d081419c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:28.769 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.764+0000 7f2d0d504700 1 --2- 172.21.15.23:0/634325773 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2d080ab2a0 0x7f2d0809f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:32:28.769 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.764+0000 7f2d0d504700 1 -- 172.21.15.23:0/634325773 >> 172.21.15.23:0/634325773 conn(0x7f2d080683b0 msgr2=0x7f2d080a9760 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:28.770 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.765+0000 7f2d0d504700 1 -- 172.21.15.23:0/634325773 shutdown_connections 2020-11-20T11:32:28.770 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.765+0000 7f2d0d504700 1 -- 172.21.15.23:0/634325773 wait complete. 2020-11-20T11:32:28.779 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T11:32:28.780 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-11-20T11:32:28.893 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.891+0000 7fa36169c700 1 Processor -- start 2020-11-20T11:32:28.894 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.891+0000 7fa36169c700 1 -- start start 2020-11-20T11:32:28.896 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.891+0000 7fa36169c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa35c060a80 0x7fa35c061950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:28.896 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.891+0000 7fa36169c700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa35c0a9d40 0x7fa35c0aa1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:28.897 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.891+0000 7fa36169c700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa35c067bc0 con 0x7fa35c137920 2020-11-20T11:32:28.897 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.891+0000 7fa36169c700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa35c067e60 con 0x7fa35c060a80 2020-11-20T11:32:28.897 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.891+0000 7fa36169c700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa35c068100 con 0x7fa35c0a9d40 2020-11-20T11:32:28.897 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.891+0000 7fa35affd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa35c060a80 0x7fa35c061950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:28.897 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.891+0000 7fa35b7fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fa35c137920 0x7fa35c137db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:54942/0 (socket says 172.21.15.23:54942) 2020-11-20T11:32:28.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.891+0000 7fa35b7fe700 1 -- 172.21.15.23:0/4271178287 learned_addr learned my addr 172.21.15.23:0/4271178287 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:28.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.892+0000 7fa35a7fc700 1 --2- 172.21.15.23:0/4271178287 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa35c0a9d40 0x7fa35c0aa1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:28.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.892+0000 7fa35affd700 1 -- 172.21.15.23:0/4271178287 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa35c0a9d40 msgr2=0x7fa35c0aa1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:28.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.892+0000 7fa35affd700 1 --2- 172.21.15.23:0/4271178287 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa35c0a9d40 0x7fa35c0aa1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:28.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.892+0000 7fa35affd700 1 -- 172.21.15.23:0/4271178287 >> v1:172.21.15.23:6789/0 conn(0x7fa35c137920 legacy=0x7fa35c137db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:28.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.892+0000 7fa359ffb700 1 -- 172.21.15.23:0/4271178287 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2206305744 0 0) 0x7fa35c067bc0 con 0x7fa35c137920 2020-11-20T11:32:28.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.892+0000 7fa35affd700 1 -- 172.21.15.23:0/4271178287 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa35c060e60 con 0x7fa35c060a80 2020-11-20T11:32:28.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.892+0000 7fa35affd700 1 --2- 172.21.15.23:0/4271178287 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa35c060a80 0x7fa35c061950 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fa34c006fd0 tx=0x7fa34c006e30).ready entity=mon.0 client_cookie=b77ac06659f2da86 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:28.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.892+0000 7fa359ffb700 1 -- 172.21.15.23:0/4271178287 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa34c0097e0 con 0x7fa35c060a80 2020-11-20T11:32:28.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.892+0000 7fa359ffb700 1 -- 172.21.15.23:0/4271178287 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa34c000e90 con 0x7fa35c060a80 2020-11-20T11:32:28.900 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.892+0000 7fa359ffb700 1 -- 172.21.15.23:0/4271178287 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa34c009b50 con 0x7fa35c060a80 2020-11-20T11:32:28.900 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.892+0000 7fa36169c700 1 -- 172.21.15.23:0/4271178287 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa35c060a80 msgr2=0x7fa35c061950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:28.900 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.892+0000 7fa36169c700 1 --2- 172.21.15.23:0/4271178287 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa35c060a80 0x7fa35c061950 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fa34c006fd0 tx=0x7fa34c006e30).stop 2020-11-20T11:32:28.900 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.892+0000 7fa36169c700 1 -- 172.21.15.23:0/4271178287 shutdown_connections 2020-11-20T11:32:28.900 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.892+0000 7fa36169c700 1 --2- 172.21.15.23:0/4271178287 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa35c060a80 0x7fa35c061950 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:28.901 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.892+0000 7fa36169c700 1 --2- 172.21.15.23:0/4271178287 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa35c0a9d40 0x7fa35c0aa1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:28.901 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.892+0000 7fa36169c700 1 -- 172.21.15.23:0/4271178287 >> 172.21.15.23:0/4271178287 conn(0x7fa35c066730 msgr2=0x7fa35c0601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:28.901 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.893+0000 7fa36169c700 1 -- 172.21.15.23:0/4271178287 shutdown_connections 2020-11-20T11:32:28.901 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.893+0000 7fa36169c700 1 -- 172.21.15.23:0/4271178287 wait complete. 2020-11-20T11:32:28.901 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.894+0000 7fa36169c700 1 Processor -- start 2020-11-20T11:32:28.902 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.894+0000 7fa36169c700 1 -- start start 2020-11-20T11:32:28.904 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.894+0000 7fa36169c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa35c060a80 0x7fa35c0a89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:28.904 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.894+0000 7fa36169c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa35c0a9d40 0x7fa35c0a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:28.904 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.894+0000 7fa36169c700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa35c137920 0x7fa35c134340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:28.905 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.894+0000 7fa36169c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa35c0a7470 con 0x7fa35c060a80 2020-11-20T11:32:28.905 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.894+0000 7fa36169c700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa35c0a71d0 con 0x7fa35c0a9d40 2020-11-20T11:32:28.905 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.894+0000 7fa36169c700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa35c0a5580 con 0x7fa35c137920 2020-11-20T11:32:28.905 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.894+0000 7fa35affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa35c060a80 0x7fa35c0a89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:28.905 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.894+0000 7fa35affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa35c060a80 0x7fa35c0a89c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40698/0 (socket says 172.21.15.23:40698) 2020-11-20T11:32:28.906 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.894+0000 7fa35affd700 1 -- 172.21.15.23:0/2009090027 learned_addr learned my addr 172.21.15.23:0/2009090027 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:28.906 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.894+0000 7fa35a7fc700 1 --2- 172.21.15.23:0/2009090027 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa35c0a9d40 0x7fa35c0a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:28.906 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.894+0000 7fa35b7fe700 1 --2- 172.21.15.23:0/2009090027 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa35c137920 0x7fa35c134340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:28.906 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.895+0000 7fa35affd700 1 -- 172.21.15.23:0/2009090027 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa35c137920 msgr2=0x7fa35c134340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:28.906 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.895+0000 7fa35affd700 1 --2- 172.21.15.23:0/2009090027 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa35c137920 0x7fa35c134340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:28.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.895+0000 7fa35affd700 1 -- 172.21.15.23:0/2009090027 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa35c0a9d40 msgr2=0x7fa35c0a5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:28.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.895+0000 7fa35affd700 1 --2- 172.21.15.23:0/2009090027 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa35c0a9d40 0x7fa35c0a5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:28.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.895+0000 7fa35affd700 1 -- 172.21.15.23:0/2009090027 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa35c138a10 con 0x7fa35c060a80 2020-11-20T11:32:28.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.895+0000 7fa35b7fe700 1 --2- 172.21.15.23:0/2009090027 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa35c137920 0x7fa35c134340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:28.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.895+0000 7fa35a7fc700 1 --2- 172.21.15.23:0/2009090027 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa35c0a9d40 0x7fa35c0a5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:28.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.895+0000 7fa35affd700 1 --2- 172.21.15.23:0/2009090027 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa35c060a80 0x7fa35c0a89c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fa34c00b190 tx=0x7fa34c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:28.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.895+0000 7fa342ffd700 1 -- 172.21.15.23:0/2009090027 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa34c011d70 con 0x7fa35c060a80 2020-11-20T11:32:28.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.895+0000 7fa342ffd700 1 -- 172.21.15.23:0/2009090027 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa34c01e460 con 0x7fa35c060a80 2020-11-20T11:32:28.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.895+0000 7fa36169c700 1 -- 172.21.15.23:0/2009090027 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa35c0a2fe0 con 0x7fa35c060a80 2020-11-20T11:32:28.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.895+0000 7fa342ffd700 1 -- 172.21.15.23:0/2009090027 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa34c01e800 con 0x7fa35c060a80 2020-11-20T11:32:28.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.895+0000 7fa36169c700 1 -- 172.21.15.23:0/2009090027 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa35c0a5f10 con 0x7fa35c060a80 2020-11-20T11:32:28.909 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.896+0000 7fa36169c700 1 -- 172.21.15.23:0/2009090027 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa35c0602c0 con 0x7fa35c060a80 2020-11-20T11:32:28.909 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.896+0000 7fa342ffd700 1 -- 172.21.15.23:0/2009090027 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa34c025030 con 0x7fa35c060a80 2020-11-20T11:32:28.909 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.896+0000 7fa342ffd700 1 --2- 172.21.15.23:0/2009090027 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa338056d60 0x7fa338059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:28.909 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.896+0000 7fa342ffd700 1 -- 172.21.15.23:0/2009090027 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fa34c071760 con 0x7fa35c060a80 2020-11-20T11:32:28.909 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.897+0000 7fa35a7fc700 1 --2- 172.21.15.23:0/2009090027 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa338056d60 0x7fa338059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:28.910 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.897+0000 7fa35a7fc700 1 --2- 172.21.15.23:0/2009090027 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa338056d60 0x7fa338059220 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fa35c0a7ff0 tx=0x7fa350008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:28.910 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:28.900+0000 7fa342ffd700 1 -- 172.21.15.23:0/2009090027 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa34c046b00 con 0x7fa35c060a80 2020-11-20T11:32:29.053 INFO:tasks.ceph.mds.a.smithi023.stderr:2020-11-20T11:32:29.050+0000 7f3849675700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2020-11-20T11:32:29.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.051+0000 7fa36169c700 1 -- 172.21.15.23:0/2009090027 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa35c0a95b0 con 0x7fa35c060a80 2020-11-20T11:32:29.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.052+0000 7fa342ffd700 1 -- 172.21.15.23:0/2009090027 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17087 (secure 0 0 0) 0x7fa34c0312d0 con 0x7fa35c060a80 2020-11-20T11:32:29.055 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:32:29.055 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":25,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","created":"2020-11-20T11:32:06.136679+0000","modified":"2020-11-20T11:32:26.470927+0000","last_up_change":"2020-11-20T11:32:13.536194+0000","last_in_change":"2020-11-20T11:32:13.536194+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T11:32:09.559868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T11:32:17.000782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T11:32:20.423341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T11:32:23.126273+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e9d219f5-e40e-4b2b-9468-deb9c4074a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":23083},{"type":"v1","addr":"172.21.15.23:6801","nonce":23083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":23083},{"type":"v1","addr":"172.21.15.23:6803","nonce":23083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":23083},{"type":"v1","addr":"172.21.15.23:6807","nonce":23083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":23083},{"type":"v1","addr":"172.21.15.23:6805","nonce":23083}]},"public_addr":"172.21.15.23:6801/23083","cluster_addr":"172.21.15.23:6803/23083","heartbeat_back_addr":"172.21.15.23:6807/23083","heartbeat_front_addr":"172.21.15.23:6805/23083","state":["exists","up"]},{"osd":1,"uuid":"2328adde-2108-4fe0-8138-f0d096a496a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":23099},{"type":"v1","addr":"172.21.15.23:6817","nonce":23099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":23099},{"type":"v1","addr":"172.21.15.23:6819","nonce":23099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":23099},{"type":"v1","addr":"172.21.15.23:6824","nonce":23099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":23099},{"type":"v1","addr":"172.21.15.23:6821","nonce":23099}]},"public_addr":"172.21.15.23:6817/23099","cluster_addr":"172.21.15.23:6819/23099","heartbeat_back_addr":"172.21.15.23:6824/23099","heartbeat_front_addr":"172.21.15.23:6821/23099","state":["exists","up"]},{"osd":2,"uuid":"13367708-59a2-4363-8a3a-ab0cb97145ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":23084},{"type":"v1","addr":"172.21.15.23:6809","nonce":23084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":23084},{"type":"v1","addr":"172.21.15.23:6811","nonce":23084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":23084},{"type":"v1","addr":"172.21.15.23:6815","nonce":23084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":23084},{"type":"v1","addr":"172.21.15.23:6813","nonce":23084}]},"public_addr":"172.21.15.23:6809/23084","cluster_addr":"172.21.15.23:6811/23084","heartbeat_back_addr":"172.21.15.23:6815/23084","heartbeat_front_addr":"172.21.15.23:6813/23084","state":["exists","up"]},{"osd":3,"uuid":"1f147dd7-9289-47b0-ad1c-4088618d6faa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6823","nonce":23114},{"type":"v1","addr":"172.21.15.23:6825","nonce":23114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":23114},{"type":"v1","addr":"172.21.15.23:6827","nonce":23114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":23114},{"type":"v1","addr":"172.21.15.23:6831","nonce":23114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":23114},{"type":"v1","addr":"172.21.15.23:6829","nonce":23114}]},"public_addr":"172.21.15.23:6825/23114","cluster_addr":"172.21.15.23:6827/23114","heartbeat_back_addr":"172.21.15.23:6831/23114","heartbeat_front_addr":"172.21.15.23:6829/23114","state":["exists","up"]},{"osd":4,"uuid":"d008b039-4bc5-4135-9678-1511e58efc29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":23280},{"type":"v1","addr":"172.21.15.155:6801","nonce":23280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":23280},{"type":"v1","addr":"172.21.15.155:6803","nonce":23280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":23280},{"type":"v1","addr":"172.21.15.155:6807","nonce":23280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":23280},{"type":"v1","addr":"172.21.15.155:6805","nonce":23280}]},"public_addr":"172.21.15.155:6801/23280","cluster_addr":"172.21.15.155:6803/23280","heartbeat_back_addr":"172.21.15.155:6807/23280","heartbeat_front_addr":"172.21.15.155:6805/23280","state":["exists","up"]},{"osd":5,"uuid":"56715e6f-cf6e-44a6-9e39-93ab1224095f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":23298},{"type":"v1","addr":"172.21.15.155:6817","nonce":23298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":23298},{"type":"v1","addr":"172.21.15.155:6819","nonce":23298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":23298},{"type":"v1","addr":"172.21.15.155:6823","nonce":23298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":23298},{"type":"v1","addr":"172.21.15.155:6821","nonce":23298}]},"public_addr":"172.21.15.155:6817/23298","cluster_addr":"172.21.15.155:6819/23298","heartbeat_back_addr":"172.21.15.155:6823/23298","heartbeat_front_addr":"172.21.15.155:6821/23298","state":["exists","up"]},{"osd":6,"uuid":"db1e61ca-8e14-4a1f-8607-6e01b148d61c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":23307},{"type":"v1","addr":"172.21.15.155:6825","nonce":23307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":23307},{"type":"v1","addr":"172.21.15.155:6827","nonce":23307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":23307},{"type":"v1","addr":"172.21.15.155:6831","nonce":23307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":23307},{"type":"v1","addr":"172.21.15.155:6829","nonce":23307}]},"public_addr":"172.21.15.155:6825/23307","cluster_addr":"172.21.15.155:6827/23307","heartbeat_back_addr":"172.21.15.155:6831/23307","heartbeat_front_addr":"172.21.15.155:6829/23307","state":["exists","up"]},{"osd":7,"uuid":"d44419be-ef89-462c-8cf1-3c6db2a6944b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":23281},{"type":"v1","addr":"172.21.15.155:6809","nonce":23281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":23281},{"type":"v1","addr":"172.21.15.155:6811","nonce":23281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":23281},{"type":"v1","addr":"172.21.15.155:6815","nonce":23281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":23281},{"type":"v1","addr":"172.21.15.155:6813","nonce":23281}]},"public_addr":"172.21.15.155:6809/23281","cluster_addr":"172.21.15.155:6811/23281","heartbeat_back_addr":"172.21.15.155:6815/23281","heartbeat_front_addr":"172.21.15.155:6813/23281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.486702+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.509396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.533414+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.509489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.467600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.513068+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.536636+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.531772+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T11:32:29.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.053+0000 7fa36169c700 1 -- 172.21.15.23:0/2009090027 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa338056d60 msgr2=0x7fa338059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:29.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.053+0000 7fa36169c700 1 --2- 172.21.15.23:0/2009090027 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa338056d60 0x7fa338059220 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fa35c0a7ff0 tx=0x7fa350008040).stop 2020-11-20T11:32:29.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.053+0000 7fa36169c700 1 -- 172.21.15.23:0/2009090027 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa35c060a80 msgr2=0x7fa35c0a89c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:29.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.053+0000 7fa36169c700 1 --2- 172.21.15.23:0/2009090027 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa35c060a80 0x7fa35c0a89c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fa34c00b190 tx=0x7fa34c015000).stop 2020-11-20T11:32:29.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.053+0000 7fa36169c700 1 -- 172.21.15.23:0/2009090027 shutdown_connections 2020-11-20T11:32:29.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.053+0000 7fa36169c700 1 --2- 172.21.15.23:0/2009090027 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa338056d60 0x7fa338059220 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:29.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.053+0000 7fa36169c700 1 --2- 172.21.15.23:0/2009090027 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa35c137920 0x7fa35c134340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:29.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.053+0000 7fa36169c700 1 --2- 172.21.15.23:0/2009090027 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa35c060a80 0x7fa35c0a89c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:29.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.053+0000 7fa36169c700 1 --2- 172.21.15.23:0/2009090027 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa35c0a9d40 0x7fa35c0a5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:29.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.053+0000 7fa36169c700 1 -- 172.21.15.23:0/2009090027 >> 172.21.15.23:0/2009090027 conn(0x7fa35c066730 msgr2=0x7fa35c137db0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:29.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.053+0000 7fa36169c700 1 -- 172.21.15.23:0/2009090027 shutdown_connections 2020-11-20T11:32:29.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.053+0000 7fa36169c700 1 -- 172.21.15.23:0/2009090027 wait complete. 2020-11-20T11:32:29.068 INFO:tasks.ceph.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T11:32:09.559868+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T11:32:17.000782+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T11:32:20.423341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T11:32:23.126273+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T11:32:29.068 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T11:32:29.185 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.182+0000 7f09e8c31700 1 Processor -- start 2020-11-20T11:32:29.185 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.182+0000 7f09e8c31700 1 -- start start 2020-11-20T11:32:29.189 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.183+0000 7f09e8c31700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f09e40aaab0 0x7f09e413ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:29.189 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.183+0000 7f09e8c31700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e41388b0 con 0x7f09e413aeb0 2020-11-20T11:32:29.189 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.183+0000 7f09e8c31700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e413e600 con 0x7f09e40aae90 2020-11-20T11:32:29.189 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.183+0000 7f09e8c31700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09e413e8a0 con 0x7f09e40aaab0 2020-11-20T11:32:29.190 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.183+0000 7f09e2d9d700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f09e413aeb0 0x7f09e40a8c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:54956/0 (socket says 172.21.15.23:54956) 2020-11-20T11:32:29.190 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.183+0000 7f09e2d9d700 1 -- 172.21.15.23:0/3071318111 learned_addr learned my addr 172.21.15.23:0/3071318111 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:29.190 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.183+0000 7f09e259c700 1 --2- 172.21.15.23:0/3071318111 >> v2:172.21.15.155:3300/0 conn(0x7f09e40aaab0 0x7f09e413ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:29.190 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.183+0000 7f09e159a700 1 -- 172.21.15.23:0/3071318111 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1150003338 0 0) 0x7f09e41388b0 con 0x7f09e413aeb0 2020-11-20T11:32:29.190 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.183+0000 7f09e159a700 1 -- 172.21.15.23:0/3071318111 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09c8003280 con 0x7f09e413aeb0 2020-11-20T11:32:29.191 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.184+0000 7f09e159a700 1 -- 172.21.15.23:0/3071318111 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4032563804 0 0) 0x7f09e413e600 con 0x7f09e40aae90 2020-11-20T11:32:29.191 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.184+0000 7f09e159a700 1 -- 172.21.15.23:0/3071318111 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09e41388b0 con 0x7f09e40aae90 2020-11-20T11:32:29.191 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.184+0000 7f09e159a700 1 -- 172.21.15.23:0/3071318111 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1401315935 0 0) 0x7f09c8003280 con 0x7f09e413aeb0 2020-11-20T11:32:29.191 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.184+0000 7f09e159a700 1 -- 172.21.15.23:0/3071318111 >> v2:172.21.15.155:3300/0 conn(0x7f09e40aaab0 msgr2=0x7f09e413ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:29.191 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.184+0000 7f09e159a700 1 --2- 172.21.15.23:0/3071318111 >> v2:172.21.15.155:3300/0 conn(0x7f09e40aaab0 0x7f09e413ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:29.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.184+0000 7f09e159a700 1 -- 172.21.15.23:0/3071318111 >> v1:172.21.15.155:6789/0 conn(0x7f09e40aae90 legacy=0x7f09e41381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:29.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.184+0000 7f09e159a700 1 -- 172.21.15.23:0/3071318111 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e40ab6d0 con 0x7f09e413aeb0 2020-11-20T11:32:29.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.184+0000 7f09e159a700 1 -- 172.21.15.23:0/3071318111 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f09dc0018a0 con 0x7f09e413aeb0 2020-11-20T11:32:29.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.184+0000 7f09e159a700 1 -- 172.21.15.23:0/3071318111 >> v1:172.21.15.23:6789/0 conn(0x7f09e413aeb0 legacy=0x7f09e40a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:29.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.184+0000 7f09e159a700 1 --2- 172.21.15.23:0/3071318111 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f09c8003950 0x7f09c8003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:29.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.184+0000 7f09e159a700 1 --2- 172.21.15.23:0/3071318111 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09c8004520 0x7f09c80049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:29.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.184+0000 7f09e159a700 1 --2- 172.21.15.23:0/3071318111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09c8004e90 0x7f09c800b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:29.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.184+0000 7f09e159a700 1 -- 172.21.15.23:0/3071318111 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e413e8a0 con 0x7f09c8004e90 2020-11-20T11:32:29.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.184+0000 7f09e159a700 1 -- 172.21.15.23:0/3071318111 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c800bc60 con 0x7f09c8004520 2020-11-20T11:32:29.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.184+0000 7f09e159a700 1 -- 172.21.15.23:0/3071318111 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c800bdd0 con 0x7f09c8003950 2020-11-20T11:32:29.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.184+0000 7f09e2d9d700 1 --2- 172.21.15.23:0/3071318111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09c8004e90 0x7f09c800b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:29.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.184+0000 7f09e1d9b700 1 --2- 172.21.15.23:0/3071318111 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09c8004520 0x7f09c80049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:29.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.184+0000 7f09e259c700 1 --2- 172.21.15.23:0/3071318111 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f09c8003950 0x7f09c8003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:29.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.184+0000 7f09e2d9d700 1 -- 172.21.15.23:0/3071318111 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f09c8003950 msgr2=0x7f09c8003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:29.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.184+0000 7f09e2d9d700 1 --2- 172.21.15.23:0/3071318111 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f09c8003950 0x7f09c8003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:29.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.184+0000 7f09e2d9d700 1 -- 172.21.15.23:0/3071318111 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09c8004520 msgr2=0x7f09c80049a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:29.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.184+0000 7f09e2d9d700 1 --2- 172.21.15.23:0/3071318111 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09c8004520 0x7f09c80049a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:29.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.184+0000 7f09e2d9d700 1 -- 172.21.15.23:0/3071318111 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c8003280 con 0x7f09c8004e90 2020-11-20T11:32:29.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.184+0000 7f09e2d9d700 1 -- 172.21.15.23:0/3071318111 reap_dead start 2020-11-20T11:32:29.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.184+0000 7f09e2d9d700 1 --2- 172.21.15.23:0/3071318111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09c8004e90 0x7f09c800b770 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f09dc00c550 tx=0x7f09dc007a60).ready entity=mon.0 client_cookie=c7996df2b5a03cd3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:29.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.185+0000 7f09e159a700 1 -- 172.21.15.23:0/3071318111 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09dc005bc0 con 0x7f09c8004e90 2020-11-20T11:32:29.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.185+0000 7f09e159a700 1 -- 172.21.15.23:0/3071318111 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09dc005d20 con 0x7f09c8004e90 2020-11-20T11:32:29.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.185+0000 7f09e159a700 1 -- 172.21.15.23:0/3071318111 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09dc007320 con 0x7f09c8004e90 2020-11-20T11:32:29.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.185+0000 7f09e8c31700 1 -- 172.21.15.23:0/3071318111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09c8004e90 msgr2=0x7f09c800b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:29.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.185+0000 7f09e8c31700 1 --2- 172.21.15.23:0/3071318111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09c8004e90 0x7f09c800b770 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f09dc00c550 tx=0x7f09dc007a60).stop 2020-11-20T11:32:29.197 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.185+0000 7f09e8c31700 1 -- 172.21.15.23:0/3071318111 shutdown_connections 2020-11-20T11:32:29.197 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.185+0000 7f09e8c31700 1 --2- 172.21.15.23:0/3071318111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09c8004e90 0x7f09c800b770 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:29.197 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.185+0000 7f09e8c31700 1 -- 172.21.15.23:0/3071318111 >> 172.21.15.23:0/3071318111 conn(0x7f09e40678e0 msgr2=0x7f09e4067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:29.197 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.185+0000 7f09e8c31700 1 -- 172.21.15.23:0/3071318111 shutdown_connections 2020-11-20T11:32:29.197 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.185+0000 7f09e8c31700 1 -- 172.21.15.23:0/3071318111 wait complete. 2020-11-20T11:32:29.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.186+0000 7f09e8c31700 1 Processor -- start 2020-11-20T11:32:29.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.186+0000 7f09e8c31700 1 -- start start 2020-11-20T11:32:29.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.187+0000 7f09e8c31700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09c8004e90 0x7f09e40b05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:29.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.187+0000 7f09e8c31700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f09e40a60c0 0x7f09e40a6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:29.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.187+0000 7f09e8c31700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09e40a6f80 0x7f09e413ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:29.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.187+0000 7f09e8c31700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e40a7400 con 0x7f09e40a6f80 2020-11-20T11:32:29.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.187+0000 7f09e8c31700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e409e390 con 0x7f09c8004e90 2020-11-20T11:32:29.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.187+0000 7f09e8c31700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09e409e630 con 0x7f09e40a60c0 2020-11-20T11:32:29.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.187+0000 7f09e2d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09e40a6f80 0x7f09e413ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:29.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.187+0000 7f09e2d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09e40a6f80 0x7f09e413ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40720/0 (socket says 172.21.15.23:40720) 2020-11-20T11:32:29.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.187+0000 7f09e2d9d700 1 -- 172.21.15.23:0/172578551 learned_addr learned my addr 172.21.15.23:0/172578551 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:29.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.187+0000 7f09e259c700 1 --2- 172.21.15.23:0/172578551 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09c8004e90 0x7f09e40b05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:29.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.187+0000 7f09e1d9b700 1 --2- 172.21.15.23:0/172578551 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f09e40a60c0 0x7f09e40a6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:29.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.187+0000 7f09e2d9d700 1 -- 172.21.15.23:0/172578551 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f09e40a60c0 msgr2=0x7f09e40a6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:29.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.187+0000 7f09e2d9d700 1 --2- 172.21.15.23:0/172578551 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f09e40a60c0 0x7f09e40a6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:29.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.187+0000 7f09e2d9d700 1 -- 172.21.15.23:0/172578551 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09c8004e90 msgr2=0x7f09e40b05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:29.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.187+0000 7f09e2d9d700 1 --2- 172.21.15.23:0/172578551 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09c8004e90 0x7f09e40b05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:29.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.187+0000 7f09e2d9d700 1 -- 172.21.15.23:0/172578551 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e409e8d0 con 0x7f09e40a6f80 2020-11-20T11:32:29.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.187+0000 7f09e2d9d700 1 --2- 172.21.15.23:0/172578551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09e40a6f80 0x7f09e413ec60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f09dc0014a0 tx=0x7f09dc015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:29.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.187+0000 7f09d27fc700 1 -- 172.21.15.23:0/172578551 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09dc007690 con 0x7f09e40a6f80 2020-11-20T11:32:29.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.187+0000 7f09d27fc700 1 -- 172.21.15.23:0/172578551 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09dc0077f0 con 0x7f09e40a6f80 2020-11-20T11:32:29.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.187+0000 7f09e8c31700 1 -- 172.21.15.23:0/172578551 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09e40a6540 con 0x7f09e40a6f80 2020-11-20T11:32:29.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.187+0000 7f09d27fc700 1 -- 172.21.15.23:0/172578551 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09dc004b10 con 0x7f09e40a6f80 2020-11-20T11:32:29.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.187+0000 7f09e8c31700 1 -- 172.21.15.23:0/172578551 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e40b1050 con 0x7f09e40a6f80 2020-11-20T11:32:29.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.188+0000 7f09e8c31700 1 -- 172.21.15.23:0/172578551 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09e405c7d0 con 0x7f09e40a6f80 2020-11-20T11:32:29.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.188+0000 7f09d27fc700 1 -- 172.21.15.23:0/172578551 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f09dc026070 con 0x7f09e40a6f80 2020-11-20T11:32:29.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.189+0000 7f09d27fc700 1 --2- 172.21.15.23:0/172578551 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f09b8056d60 0x7f09b8059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:29.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.189+0000 7f09d27fc700 1 -- 172.21.15.23:0/172578551 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f09dc070b70 con 0x7f09e40a6f80 2020-11-20T11:32:29.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.189+0000 7f09e259c700 1 --2- 172.21.15.23:0/172578551 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f09b8056d60 0x7f09b8059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:29.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.189+0000 7f09e259c700 1 --2- 172.21.15.23:0/172578551 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f09b8056d60 0x7f09b8059220 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f09d40068a0 tx=0x7f09d4006e30).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:29.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.192+0000 7f09d27fc700 1 -- 172.21.15.23:0/172578551 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f09dc045e60 con 0x7f09e40a6f80 2020-11-20T11:32:29.226 INFO:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:32:29.228 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:32:29.346 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.343+0000 7f09e8c31700 1 -- 172.21.15.23:0/172578551 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f09e409ea60 con 0x7f09e40a6f80 2020-11-20T11:32:29.346 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.343+0000 7f09d27fc700 1 -- 172.21.15.23:0/172578551 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v25) v1 ==== 110+0+10 (secure 0 0 0) 0x7f09dc049480 con 0x7f09e40a6f80 2020-11-20T11:32:29.347 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2020-11-20T11:32:29.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.345+0000 7f09e8c31700 1 -- 172.21.15.23:0/172578551 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f09b8056d60 msgr2=0x7f09b8059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:29.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.345+0000 7f09e8c31700 1 --2- 172.21.15.23:0/172578551 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f09b8056d60 0x7f09b8059220 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f09d40068a0 tx=0x7f09d4006e30).stop 2020-11-20T11:32:29.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.345+0000 7f09e8c31700 1 -- 172.21.15.23:0/172578551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09e40a6f80 msgr2=0x7f09e413ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:29.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.345+0000 7f09e8c31700 1 --2- 172.21.15.23:0/172578551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09e40a6f80 0x7f09e413ec60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f09dc0014a0 tx=0x7f09dc015000).stop 2020-11-20T11:32:29.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.345+0000 7f09e8c31700 1 -- 172.21.15.23:0/172578551 shutdown_connections 2020-11-20T11:32:29.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.345+0000 7f09e8c31700 1 --2- 172.21.15.23:0/172578551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09e40a6f80 0x7f09e413ec60 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:29.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.345+0000 7f09e8c31700 1 --2- 172.21.15.23:0/172578551 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09c8004e90 0x7f09e40b05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:29.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.345+0000 7f09e8c31700 1 --2- 172.21.15.23:0/172578551 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f09b8056d60 0x7f09b8059220 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:29.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.345+0000 7f09e8c31700 1 --2- 172.21.15.23:0/172578551 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f09e40a60c0 0x7f09e40a6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:29.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.345+0000 7f09e8c31700 1 -- 172.21.15.23:0/172578551 >> 172.21.15.23:0/172578551 conn(0x7f09e40678e0 msgr2=0x7f09e41378b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:29.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.345+0000 7f09e8c31700 1 -- 172.21.15.23:0/172578551 shutdown_connections 2020-11-20T11:32:29.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.345+0000 7f09e8c31700 1 -- 172.21.15.23:0/172578551 wait complete. 2020-11-20T11:32:29.359 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T11:32:29.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.472+0000 7fa644913700 1 Processor -- start 2020-11-20T11:32:29.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.472+0000 7fa644913700 1 -- start start 2020-11-20T11:32:29.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.473+0000 7fa644913700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa6400608c0 0x7fa6400aa0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:29.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.473+0000 7fa644913700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6400aa590 con 0x7fa640060ca0 2020-11-20T11:32:29.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.473+0000 7fa644913700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6400aabe0 con 0x7fa640138ff0 2020-11-20T11:32:29.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.473+0000 7fa644913700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6400ab300 con 0x7fa6400608c0 2020-11-20T11:32:29.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.473+0000 7fa63dd9b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fa640060ca0 0x7fa6400a6d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:54974/0 (socket says 172.21.15.23:54974) 2020-11-20T11:32:29.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.473+0000 7fa63dd9b700 1 -- 172.21.15.23:0/1449504025 learned_addr learned my addr 172.21.15.23:0/1449504025 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:29.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.473+0000 7fa63d59a700 1 -- 172.21.15.23:0/1449504025 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3228718286 0 0) 0x7fa6400aa590 con 0x7fa640060ca0 2020-11-20T11:32:29.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.473+0000 7fa63d59a700 1 -- 172.21.15.23:0/1449504025 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa624003280 con 0x7fa640060ca0 2020-11-20T11:32:29.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.473+0000 7fa63e59c700 1 --2- 172.21.15.23:0/1449504025 >> v2:172.21.15.155:3300/0 conn(0x7fa6400608c0 0x7fa6400aa0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:29.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.474+0000 7fa63d59a700 1 -- 172.21.15.23:0/1449504025 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3835511586 0 0) 0x7fa624003280 con 0x7fa640060ca0 2020-11-20T11:32:29.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.474+0000 7fa63d59a700 1 -- 172.21.15.23:0/1449504025 >> v2:172.21.15.155:3300/0 conn(0x7fa6400608c0 msgr2=0x7fa6400aa0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:29.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.474+0000 7fa63d59a700 1 --2- 172.21.15.23:0/1449504025 >> v2:172.21.15.155:3300/0 conn(0x7fa6400608c0 0x7fa6400aa0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:29.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.474+0000 7fa63d59a700 1 -- 172.21.15.23:0/1449504025 >> v1:172.21.15.155:6789/0 conn(0x7fa640138ff0 legacy=0x7fa640139480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:29.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.474+0000 7fa63d59a700 1 -- 172.21.15.23:0/1449504025 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6400aae80 con 0x7fa640060ca0 2020-11-20T11:32:29.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.474+0000 7fa63d59a700 1 -- 172.21.15.23:0/1449504025 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7fa634001520 con 0x7fa640060ca0 2020-11-20T11:32:29.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.474+0000 7fa63d59a700 1 -- 172.21.15.23:0/1449504025 >> v1:172.21.15.23:6789/0 conn(0x7fa640060ca0 legacy=0x7fa6400a6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:29.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.474+0000 7fa63d59a700 1 --2- 172.21.15.23:0/1449504025 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa624003770 0x7fa624003b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:29.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.474+0000 7fa63d59a700 1 --2- 172.21.15.23:0/1449504025 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa624004520 0x7fa6240049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:29.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.474+0000 7fa63d59a700 1 --2- 172.21.15.23:0/1449504025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa624004e90 0x7fa62400b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:29.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.474+0000 7fa63d59a700 1 -- 172.21.15.23:0/1449504025 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6400ab300 con 0x7fa624004e90 2020-11-20T11:32:29.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.474+0000 7fa63d59a700 1 -- 172.21.15.23:0/1449504025 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6240042a0 con 0x7fa624004520 2020-11-20T11:32:29.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.474+0000 7fa63d59a700 1 -- 172.21.15.23:0/1449504025 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa62400bc60 con 0x7fa624003770 2020-11-20T11:32:29.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.474+0000 7fa63ed9d700 1 --2- 172.21.15.23:0/1449504025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa624004e90 0x7fa62400b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:29.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.474+0000 7fa63e59c700 1 --2- 172.21.15.23:0/1449504025 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa624003770 0x7fa624003b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:29.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.474+0000 7fa63dd9b700 1 --2- 172.21.15.23:0/1449504025 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa624004520 0x7fa6240049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:29.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.474+0000 7fa63ed9d700 1 -- 172.21.15.23:0/1449504025 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa624003770 msgr2=0x7fa624003b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:29.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.474+0000 7fa63ed9d700 1 --2- 172.21.15.23:0/1449504025 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa624003770 0x7fa624003b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:29.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.474+0000 7fa63ed9d700 1 -- 172.21.15.23:0/1449504025 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa624004520 msgr2=0x7fa6240049a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:29.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.474+0000 7fa63ed9d700 1 --2- 172.21.15.23:0/1449504025 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa624004520 0x7fa6240049a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:29.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.474+0000 7fa63ed9d700 1 -- 172.21.15.23:0/1449504025 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa624003280 con 0x7fa624004e90 2020-11-20T11:32:29.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.474+0000 7fa63ed9d700 1 -- 172.21.15.23:0/1449504025 reap_dead start 2020-11-20T11:32:29.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.474+0000 7fa63ed9d700 1 --2- 172.21.15.23:0/1449504025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa624004e90 0x7fa62400b770 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fa638001ef0 tx=0x7fa638007f50).ready entity=mon.0 client_cookie=a2faa02fdc6d5098 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:29.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.475+0000 7fa63d59a700 1 -- 172.21.15.23:0/1449504025 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63800c930 con 0x7fa624004e90 2020-11-20T11:32:29.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.475+0000 7fa63d59a700 1 -- 172.21.15.23:0/1449504025 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa63800ca90 con 0x7fa624004e90 2020-11-20T11:32:29.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.475+0000 7fa63d59a700 1 -- 172.21.15.23:0/1449504025 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63800ce10 con 0x7fa624004e90 2020-11-20T11:32:29.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.475+0000 7fa644913700 1 -- 172.21.15.23:0/1449504025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa624004e90 msgr2=0x7fa62400b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:29.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.475+0000 7fa644913700 1 --2- 172.21.15.23:0/1449504025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa624004e90 0x7fa62400b770 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fa638001ef0 tx=0x7fa638007f50).stop 2020-11-20T11:32:29.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.475+0000 7fa644913700 1 -- 172.21.15.23:0/1449504025 shutdown_connections 2020-11-20T11:32:29.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.475+0000 7fa644913700 1 --2- 172.21.15.23:0/1449504025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa624004e90 0x7fa62400b770 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:29.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.475+0000 7fa644913700 1 -- 172.21.15.23:0/1449504025 >> 172.21.15.23:0/1449504025 conn(0x7fa640066750 msgr2=0x7fa6400601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:29.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.475+0000 7fa644913700 1 -- 172.21.15.23:0/1449504025 shutdown_connections 2020-11-20T11:32:29.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.475+0000 7fa644913700 1 -- 172.21.15.23:0/1449504025 wait complete. 2020-11-20T11:32:29.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.476+0000 7fa644913700 1 Processor -- start 2020-11-20T11:32:29.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.476+0000 7fa644913700 1 -- start start 2020-11-20T11:32:29.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.477+0000 7fa644913700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa624004e90 0x7fa6400a32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:29.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.477+0000 7fa644913700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6400ae5f0 0x7fa64014b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:29.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.477+0000 7fa644913700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6400adf50 0x7fa64014b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:29.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.477+0000 7fa644913700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6400a94e0 con 0x7fa6400adf50 2020-11-20T11:32:29.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.477+0000 7fa644913700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6400ad570 con 0x7fa6400ae5f0 2020-11-20T11:32:29.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.477+0000 7fa644913700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6400ad810 con 0x7fa624004e90 2020-11-20T11:32:29.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.477+0000 7fa63ed9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6400adf50 0x7fa64014b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:29.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.477+0000 7fa63ed9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6400adf50 0x7fa64014b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40736/0 (socket says 172.21.15.23:40736) 2020-11-20T11:32:29.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.477+0000 7fa63ed9d700 1 -- 172.21.15.23:0/4118987804 learned_addr learned my addr 172.21.15.23:0/4118987804 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:29.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.477+0000 7fa63dd9b700 1 --2- 172.21.15.23:0/4118987804 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6400ae5f0 0x7fa64014b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:29.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.477+0000 7fa63e59c700 1 --2- 172.21.15.23:0/4118987804 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa624004e90 0x7fa6400a32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:29.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.477+0000 7fa63ed9d700 1 -- 172.21.15.23:0/4118987804 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa624004e90 msgr2=0x7fa6400a32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:29.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.477+0000 7fa63ed9d700 1 --2- 172.21.15.23:0/4118987804 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa624004e90 0x7fa6400a32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:29.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.477+0000 7fa63ed9d700 1 -- 172.21.15.23:0/4118987804 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6400ae5f0 msgr2=0x7fa64014b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:29.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.477+0000 7fa63ed9d700 1 --2- 172.21.15.23:0/4118987804 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6400ae5f0 0x7fa64014b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:29.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.477+0000 7fa63ed9d700 1 -- 172.21.15.23:0/4118987804 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6400adab0 con 0x7fa6400adf50 2020-11-20T11:32:29.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.477+0000 7fa63ed9d700 1 --2- 172.21.15.23:0/4118987804 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6400adf50 0x7fa64014b900 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fa640064bc0 tx=0x7fa638016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:29.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.477+0000 7fa62e7fc700 1 -- 172.21.15.23:0/4118987804 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa638013a80 con 0x7fa6400adf50 2020-11-20T11:32:29.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.477+0000 7fa62e7fc700 1 -- 172.21.15.23:0/4118987804 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa638013be0 con 0x7fa6400adf50 2020-11-20T11:32:29.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.477+0000 7fa644913700 1 -- 172.21.15.23:0/4118987804 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6400a6340 con 0x7fa6400adf50 2020-11-20T11:32:29.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.477+0000 7fa62e7fc700 1 -- 172.21.15.23:0/4118987804 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63801e480 con 0x7fa6400adf50 2020-11-20T11:32:29.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.477+0000 7fa644913700 1 -- 172.21.15.23:0/4118987804 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6400a3790 con 0x7fa6400adf50 2020-11-20T11:32:29.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.478+0000 7fa644913700 1 -- 172.21.15.23:0/4118987804 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6400a3a40 con 0x7fa6400adf50 2020-11-20T11:32:29.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.479+0000 7fa62e7fc700 1 -- 172.21.15.23:0/4118987804 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa638025030 con 0x7fa6400adf50 2020-11-20T11:32:29.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.479+0000 7fa62e7fc700 1 --2- 172.21.15.23:0/4118987804 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa614056d90 0x7fa614059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:29.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.479+0000 7fa62e7fc700 1 -- 172.21.15.23:0/4118987804 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fa6380712e0 con 0x7fa6400adf50 2020-11-20T11:32:29.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.479+0000 7fa63e59c700 1 --2- 172.21.15.23:0/4118987804 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa614056d90 0x7fa614059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:29.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.480+0000 7fa63e59c700 1 --2- 172.21.15.23:0/4118987804 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa614056d90 0x7fa614059250 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fa630006fb0 tx=0x7fa63000b040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:29.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.483+0000 7fa62e7fc700 1 -- 172.21.15.23:0/4118987804 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa638046680 con 0x7fa6400adf50 2020-11-20T11:32:29.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.631+0000 7fa644913700 1 -- 172.21.15.23:0/4118987804 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fa640044710 con 0x7fa6400adf50 2020-11-20T11:32:30.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.631+0000 7fa62e7fc700 1 -- 172.21.15.23:0/4118987804 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v25) v1 ==== 92+0+10 (secure 0 0 0) 0x7fa638023020 con 0x7fa6400adf50 2020-11-20T11:32:30.074 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 8 2020-11-20T11:32:30.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.633+0000 7fa644913700 1 -- 172.21.15.23:0/4118987804 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa614056d90 msgr2=0x7fa614059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:30.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.633+0000 7fa644913700 1 --2- 172.21.15.23:0/4118987804 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa614056d90 0x7fa614059250 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fa630006fb0 tx=0x7fa63000b040).stop 2020-11-20T11:32:30.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.633+0000 7fa644913700 1 -- 172.21.15.23:0/4118987804 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6400adf50 msgr2=0x7fa64014b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:30.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.633+0000 7fa644913700 1 --2- 172.21.15.23:0/4118987804 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6400adf50 0x7fa64014b900 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fa640064bc0 tx=0x7fa638016000).stop 2020-11-20T11:32:30.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.633+0000 7fa644913700 1 -- 172.21.15.23:0/4118987804 shutdown_connections 2020-11-20T11:32:30.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.633+0000 7fa644913700 1 --2- 172.21.15.23:0/4118987804 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6400adf50 0x7fa64014b900 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:30.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.633+0000 7fa644913700 1 --2- 172.21.15.23:0/4118987804 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa614056d90 0x7fa614059250 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:30.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.633+0000 7fa644913700 1 --2- 172.21.15.23:0/4118987804 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa624004e90 0x7fa6400a32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:30.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.633+0000 7fa644913700 1 --2- 172.21.15.23:0/4118987804 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6400ae5f0 0x7fa64014b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:30.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.633+0000 7fa644913700 1 -- 172.21.15.23:0/4118987804 >> 172.21.15.23:0/4118987804 conn(0x7fa640066750 msgr2=0x7fa640136c70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:30.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.633+0000 7fa644913700 1 -- 172.21.15.23:0/4118987804 shutdown_connections 2020-11-20T11:32:30.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:29.633+0000 7fa644913700 1 -- 172.21.15.23:0/4118987804 wait complete. 2020-11-20T11:32:30.079 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T11:32:30.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.199+0000 7fc072c6b700 1 Processor -- start 2020-11-20T11:32:30.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.199+0000 7fc072c6b700 1 -- start start 2020-11-20T11:32:30.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.200+0000 7fc072c6b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc06c0ad9e0 0x7fc06c0658b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:30.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.200+0000 7fc072c6b700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc06c0a8b90 con 0x7fc06c13c070 2020-11-20T11:32:30.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.200+0000 7fc072c6b700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc06c065da0 con 0x7fc06c0ad600 2020-11-20T11:32:30.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.200+0000 7fc072c6b700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc06c0a83b0 con 0x7fc06c0ad9e0 2020-11-20T11:32:30.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.200+0000 7fc06bfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc06c0ad9e0 0x7fc06c0658b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:30.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.200+0000 7fc071208700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fc06c13c070 0x7fc06c0a6b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:54994/0 (socket says 172.21.15.23:54994) 2020-11-20T11:32:30.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.200+0000 7fc071208700 1 -- 172.21.15.23:0/3298257292 learned_addr learned my addr 172.21.15.23:0/3298257292 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:30.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.200+0000 7fc06bfff700 1 -- 172.21.15.23:0/3298257292 >> v1:172.21.15.155:6789/0 conn(0x7fc06c0ad600 legacy=0x7fc06c13c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:30.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.200+0000 7fc06bfff700 1 -- 172.21.15.23:0/3298257292 >> v1:172.21.15.23:6789/0 conn(0x7fc06c13c070 legacy=0x7fc06c0a6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:30.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.200+0000 7fc06b7fe700 1 -- 172.21.15.23:0/3298257292 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318810076 0 0) 0x7fc06c0a8b90 con 0x7fc06c13c070 2020-11-20T11:32:30.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.200+0000 7fc06bfff700 1 -- 172.21.15.23:0/3298257292 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc06c0a8650 con 0x7fc06c0ad9e0 2020-11-20T11:32:30.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.200+0000 7fc06bfff700 1 --2- 172.21.15.23:0/3298257292 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc06c0ad9e0 0x7fc06c0658b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fc058006fd0 tx=0x7fc058006e30).ready entity=mon.0 client_cookie=38d0910aa6834bc2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:30.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.200+0000 7fc06b7fe700 1 -- 172.21.15.23:0/3298257292 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0580097e0 con 0x7fc06c0ad9e0 2020-11-20T11:32:30.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.200+0000 7fc06b7fe700 1 -- 172.21.15.23:0/3298257292 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc058000e90 con 0x7fc06c0ad9e0 2020-11-20T11:32:30.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.200+0000 7fc06b7fe700 1 -- 172.21.15.23:0/3298257292 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc058009b50 con 0x7fc06c0ad9e0 2020-11-20T11:32:30.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.201+0000 7fc072c6b700 1 -- 172.21.15.23:0/3298257292 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc06c0ad9e0 msgr2=0x7fc06c0658b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:30.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.201+0000 7fc072c6b700 1 --2- 172.21.15.23:0/3298257292 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc06c0ad9e0 0x7fc06c0658b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fc058006fd0 tx=0x7fc058006e30).stop 2020-11-20T11:32:30.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.201+0000 7fc072c6b700 1 -- 172.21.15.23:0/3298257292 shutdown_connections 2020-11-20T11:32:30.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.201+0000 7fc072c6b700 1 --2- 172.21.15.23:0/3298257292 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc06c0ad9e0 0x7fc06c0658b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:30.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.201+0000 7fc072c6b700 1 -- 172.21.15.23:0/3298257292 >> 172.21.15.23:0/3298257292 conn(0x7fc06c060c80 msgr2=0x7fc06c0acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:30.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.201+0000 7fc072c6b700 1 -- 172.21.15.23:0/3298257292 shutdown_connections 2020-11-20T11:32:30.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.201+0000 7fc072c6b700 1 -- 172.21.15.23:0/3298257292 wait complete. 2020-11-20T11:32:30.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.202+0000 7fc072c6b700 1 Processor -- start 2020-11-20T11:32:30.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.202+0000 7fc072c6b700 1 -- start start 2020-11-20T11:32:30.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.203+0000 7fc072c6b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc06c0ad600 0x7fc06c1371f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:30.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.203+0000 7fc072c6b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc06c0ad9e0 0x7fc06c0a2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:30.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.203+0000 7fc072c6b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc06c13c070 0x7fc06c0a4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:30.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.203+0000 7fc072c6b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06c0a5dc0 con 0x7fc06c0ad9e0 2020-11-20T11:32:30.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.203+0000 7fc072c6b700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06c0a3320 con 0x7fc06c13c070 2020-11-20T11:32:30.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.203+0000 7fc072c6b700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc06c0ac300 con 0x7fc06c0ad600 2020-11-20T11:32:30.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.203+0000 7fc06bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc06c0ad9e0 0x7fc06c0a2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:30.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.203+0000 7fc06bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc06c0ad9e0 0x7fc06c0a2e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40748/0 (socket says 172.21.15.23:40748) 2020-11-20T11:32:30.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.203+0000 7fc06bfff700 1 -- 172.21.15.23:0/531984464 learned_addr learned my addr 172.21.15.23:0/531984464 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:30.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.203+0000 7fc070a07700 1 --2- 172.21.15.23:0/531984464 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc06c0ad600 0x7fc06c1371f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:30.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.203+0000 7fc071208700 1 --2- 172.21.15.23:0/531984464 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc06c13c070 0x7fc06c0a4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:30.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.203+0000 7fc06bfff700 1 -- 172.21.15.23:0/531984464 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc06c0ad600 msgr2=0x7fc06c1371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:30.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.203+0000 7fc06bfff700 1 --2- 172.21.15.23:0/531984464 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc06c0ad600 0x7fc06c1371f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:30.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.203+0000 7fc06bfff700 1 -- 172.21.15.23:0/531984464 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc06c13c070 msgr2=0x7fc06c0a4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:30.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.203+0000 7fc06bfff700 1 --2- 172.21.15.23:0/531984464 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc06c13c070 0x7fc06c0a4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:30.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.203+0000 7fc06bfff700 1 -- 172.21.15.23:0/531984464 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc06c0ac5a0 con 0x7fc06c0ad9e0 2020-11-20T11:32:30.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.203+0000 7fc070a07700 1 --2- 172.21.15.23:0/531984464 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc06c0ad600 0x7fc06c1371f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:32:30.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.203+0000 7fc071208700 1 --2- 172.21.15.23:0/531984464 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc06c13c070 0x7fc06c0a4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:32:30.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.203+0000 7fc06bfff700 1 --2- 172.21.15.23:0/531984464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc06c0ad9e0 0x7fc06c0a2e30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fc058010000 tx=0x7fc058015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:30.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.203+0000 7fc068ff9700 1 -- 172.21.15.23:0/531984464 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc058011d50 con 0x7fc06c0ad9e0 2020-11-20T11:32:30.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.203+0000 7fc068ff9700 1 -- 172.21.15.23:0/531984464 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc05801e460 con 0x7fc06c0ad9e0 2020-11-20T11:32:30.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.203+0000 7fc072c6b700 1 -- 172.21.15.23:0/531984464 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc06c0abea0 con 0x7fc06c0ad9e0 2020-11-20T11:32:30.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.203+0000 7fc068ff9700 1 -- 172.21.15.23:0/531984464 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc05801e7e0 con 0x7fc06c0ad9e0 2020-11-20T11:32:30.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.203+0000 7fc072c6b700 1 -- 172.21.15.23:0/531984464 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc06c0a55d0 con 0x7fc06c0ad9e0 2020-11-20T11:32:30.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.204+0000 7fc072c6b700 1 -- 172.21.15.23:0/531984464 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc06c0a6590 con 0x7fc06c0ad9e0 2020-11-20T11:32:30.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.204+0000 7fc068ff9700 1 -- 172.21.15.23:0/531984464 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc058025030 con 0x7fc06c0ad9e0 2020-11-20T11:32:30.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.205+0000 7fc068ff9700 1 --2- 172.21.15.23:0/531984464 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc044056d60 0x7fc044059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:30.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.205+0000 7fc068ff9700 1 -- 172.21.15.23:0/531984464 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fc0580716b0 con 0x7fc06c0ad9e0 2020-11-20T11:32:30.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.205+0000 7fc070a07700 1 --2- 172.21.15.23:0/531984464 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc044056d60 0x7fc044059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:30.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.205+0000 7fc070a07700 1 --2- 172.21.15.23:0/531984464 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc044056d60 0x7fc044059220 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fc060001550 tx=0x7fc060009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:30.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.209+0000 7fc068ff9700 1 -- 172.21.15.23:0/531984464 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc058046a50 con 0x7fc06c0ad9e0 2020-11-20T11:32:30.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.357+0000 7fc072c6b700 1 -- 172.21.15.23:0/531984464 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc06c0ac730 con 0x7fc06c0ad9e0 2020-11-20T11:32:30.360 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.357+0000 7fc068ff9700 1 -- 172.21.15.23:0/531984464 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+10 (secure 0 0 0) 0x7fc058023020 con 0x7fc06c0ad9e0 2020-11-20T11:32:30.360 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 8 2020-11-20T11:32:30.361 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.358+0000 7fc072c6b700 1 -- 172.21.15.23:0/531984464 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc044056d60 msgr2=0x7fc044059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:30.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.358+0000 7fc072c6b700 1 --2- 172.21.15.23:0/531984464 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc044056d60 0x7fc044059220 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fc060001550 tx=0x7fc060009040).stop 2020-11-20T11:32:30.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.358+0000 7fc072c6b700 1 -- 172.21.15.23:0/531984464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc06c0ad9e0 msgr2=0x7fc06c0a2e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:30.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.358+0000 7fc072c6b700 1 --2- 172.21.15.23:0/531984464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc06c0ad9e0 0x7fc06c0a2e30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fc058010000 tx=0x7fc058015000).stop 2020-11-20T11:32:30.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.358+0000 7fc072c6b700 1 -- 172.21.15.23:0/531984464 shutdown_connections 2020-11-20T11:32:30.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.358+0000 7fc072c6b700 1 --2- 172.21.15.23:0/531984464 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc06c13c070 0x7fc06c0a4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:30.363 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.358+0000 7fc072c6b700 1 --2- 172.21.15.23:0/531984464 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc044056d60 0x7fc044059220 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:30.363 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.358+0000 7fc072c6b700 1 --2- 172.21.15.23:0/531984464 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc06c0ad600 0x7fc06c1371f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:30.363 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.358+0000 7fc072c6b700 1 --2- 172.21.15.23:0/531984464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc06c0ad9e0 0x7fc06c0a2e30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:30.363 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.358+0000 7fc072c6b700 1 -- 172.21.15.23:0/531984464 >> 172.21.15.23:0/531984464 conn(0x7fc06c060c80 msgr2=0x7fc06c13d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:30.363 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.359+0000 7fc072c6b700 1 -- 172.21.15.23:0/531984464 shutdown_connections 2020-11-20T11:32:30.364 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.359+0000 7fc072c6b700 1 -- 172.21.15.23:0/531984464 wait complete. 2020-11-20T11:32:30.373 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T11:32:30.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.487+0000 7f5a67fff700 1 Processor -- start 2020-11-20T11:32:30.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.487+0000 7f5a67fff700 1 -- start start 2020-11-20T11:32:30.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.487+0000 7f5a67fff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5a68061380 0x7f5a680a8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:30.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.487+0000 7f5a67fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5a680a7410 0x7f5a6813efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:30.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.487+0000 7f5a67fff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a680a5000 con 0x7f5a680a8d70 2020-11-20T11:32:30.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.487+0000 7f5a67fff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a680a78a0 con 0x7f5a680a7410 2020-11-20T11:32:30.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.487+0000 7f5a67fff700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a680a5d30 con 0x7f5a68061380 2020-11-20T11:32:30.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.487+0000 7f5a677fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5a680a7410 0x7f5a6813efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:30.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.487+0000 7f5a667fc700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f5a680a8d70 0x7f5a6813e8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55004/0 (socket says 172.21.15.23:55004) 2020-11-20T11:32:30.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.487+0000 7f5a667fc700 1 -- 172.21.15.23:0/2705683353 learned_addr learned my addr 172.21.15.23:0/2705683353 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:30.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.488+0000 7f5a66ffd700 1 --2- 172.21.15.23:0/2705683353 >> v2:172.21.15.155:3300/0 conn(0x7f5a68061380 0x7f5a680a8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:30.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.488+0000 7f5a677fe700 1 -- 172.21.15.23:0/2705683353 >> v2:172.21.15.155:3300/0 conn(0x7f5a68061380 msgr2=0x7f5a680a8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:30.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.488+0000 7f5a677fe700 1 --2- 172.21.15.23:0/2705683353 >> v2:172.21.15.155:3300/0 conn(0x7f5a68061380 0x7f5a680a8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:30.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.488+0000 7f5a677fe700 1 -- 172.21.15.23:0/2705683353 >> v1:172.21.15.23:6789/0 conn(0x7f5a680a8d70 legacy=0x7f5a6813e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:30.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.488+0000 7f5a677fe700 1 -- 172.21.15.23:0/2705683353 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a680a5fd0 con 0x7f5a680a7410 2020-11-20T11:32:30.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.488+0000 7f5a677fe700 1 --2- 172.21.15.23:0/2705683353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a680a7410 0x7f5a6813efe0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f5a68062e10 tx=0x7f5a60009220).ready entity=mon.0 client_cookie=3cf926b41ed44f85 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:30.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.488+0000 7f5a65ffb700 1 -- 172.21.15.23:0/2705683353 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6000bbf0 con 0x7f5a680a7410 2020-11-20T11:32:30.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.488+0000 7f5a65ffb700 1 -- 172.21.15.23:0/2705683353 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a6000bd50 con 0x7f5a680a7410 2020-11-20T11:32:30.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.488+0000 7f5a65ffb700 1 -- 172.21.15.23:0/2705683353 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a60011610 con 0x7f5a680a7410 2020-11-20T11:32:30.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.488+0000 7f5a67fff700 1 -- 172.21.15.23:0/2705683353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a680a7410 msgr2=0x7f5a6813efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:30.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.488+0000 7f5a67fff700 1 --2- 172.21.15.23:0/2705683353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a680a7410 0x7f5a6813efe0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f5a68062e10 tx=0x7f5a60009220).stop 2020-11-20T11:32:30.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.488+0000 7f5a67fff700 1 -- 172.21.15.23:0/2705683353 shutdown_connections 2020-11-20T11:32:30.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.488+0000 7f5a67fff700 1 --2- 172.21.15.23:0/2705683353 >> v2:172.21.15.155:3300/0 conn(0x7f5a68061380 0x7f5a680a8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:30.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.488+0000 7f5a67fff700 1 --2- 172.21.15.23:0/2705683353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a680a7410 0x7f5a6813efe0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:30.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.488+0000 7f5a67fff700 1 -- 172.21.15.23:0/2705683353 >> 172.21.15.23:0/2705683353 conn(0x7f5a680ae0a0 msgr2=0x7f5a680aa0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:30.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.488+0000 7f5a67fff700 1 -- 172.21.15.23:0/2705683353 shutdown_connections 2020-11-20T11:32:30.497 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.488+0000 7f5a67fff700 1 -- 172.21.15.23:0/2705683353 wait complete. 2020-11-20T11:32:30.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.490+0000 7f5a67fff700 1 Processor -- start 2020-11-20T11:32:30.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.490+0000 7f5a67fff700 1 -- start start 2020-11-20T11:32:30.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.490+0000 7f5a67fff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5a68061380 0x7f5a680a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:30.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.490+0000 7f5a67fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a680a7410 0x7f5a6813f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:30.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.490+0000 7f5a67fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a680a8d70 0x7f5a6809d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:30.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.490+0000 7f5a67fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6809ebb0 con 0x7f5a680a8d70 2020-11-20T11:32:30.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.490+0000 7f5a67fff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a680a2df0 con 0x7f5a680a7410 2020-11-20T11:32:30.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.490+0000 7f5a67fff700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a680a3090 con 0x7f5a68061380 2020-11-20T11:32:30.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.490+0000 7f5a677fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a680a8d70 0x7f5a6809d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:30.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.490+0000 7f5a677fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a680a8d70 0x7f5a6809d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40762/0 (socket says 172.21.15.23:40762) 2020-11-20T11:32:30.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.490+0000 7f5a677fe700 1 -- 172.21.15.23:0/933053424 learned_addr learned my addr 172.21.15.23:0/933053424 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:30.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.490+0000 7f5a66ffd700 1 --2- 172.21.15.23:0/933053424 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5a68061380 0x7f5a680a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:30.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.490+0000 7f5a667fc700 1 --2- 172.21.15.23:0/933053424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a680a7410 0x7f5a6813f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:30.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.491+0000 7f5a677fe700 1 -- 172.21.15.23:0/933053424 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5a68061380 msgr2=0x7f5a680a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:30.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.491+0000 7f5a677fe700 1 --2- 172.21.15.23:0/933053424 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5a68061380 0x7f5a680a37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:30.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.491+0000 7f5a677fe700 1 -- 172.21.15.23:0/933053424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a680a7410 msgr2=0x7f5a6813f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:30.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.491+0000 7f5a677fe700 1 --2- 172.21.15.23:0/933053424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a680a7410 0x7f5a6813f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:30.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.491+0000 7f5a677fe700 1 -- 172.21.15.23:0/933053424 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a680a3330 con 0x7f5a680a8d70 2020-11-20T11:32:30.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.491+0000 7f5a667fc700 1 --2- 172.21.15.23:0/933053424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a680a7410 0x7f5a6813f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:30.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.491+0000 7f5a66ffd700 1 --2- 172.21.15.23:0/933053424 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5a68061380 0x7f5a680a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:30.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.491+0000 7f5a677fe700 1 --2- 172.21.15.23:0/933053424 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a680a8d70 0x7f5a6809d4a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f5a600090b0 tx=0x7f5a60016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:30.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.491+0000 7f5a4effd700 1 -- 172.21.15.23:0/933053424 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a60010700 con 0x7f5a680a8d70 2020-11-20T11:32:30.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.491+0000 7f5a4effd700 1 -- 172.21.15.23:0/933053424 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a60010860 con 0x7f5a680a8d70 2020-11-20T11:32:30.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.491+0000 7f5a4effd700 1 -- 172.21.15.23:0/933053424 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a60010c00 con 0x7f5a680a8d70 2020-11-20T11:32:30.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.491+0000 7f5a67fff700 1 -- 172.21.15.23:0/933053424 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a6813fab0 con 0x7f5a680a8d70 2020-11-20T11:32:30.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.491+0000 7f5a67fff700 1 -- 172.21.15.23:0/933053424 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a6809f860 con 0x7f5a680a8d70 2020-11-20T11:32:30.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.492+0000 7f5a67fff700 1 -- 172.21.15.23:0/933053424 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a6805cf90 con 0x7f5a680a8d70 2020-11-20T11:32:30.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.492+0000 7f5a4effd700 1 -- 172.21.15.23:0/933053424 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5a60017070 con 0x7f5a680a8d70 2020-11-20T11:32:30.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.492+0000 7f5a4effd700 1 --2- 172.21.15.23:0/933053424 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5a44056d90 0x7f5a44059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:30.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.492+0000 7f5a4effd700 1 -- 172.21.15.23:0/933053424 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f5a60072450 con 0x7f5a680a8d70 2020-11-20T11:32:30.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.493+0000 7f5a66ffd700 1 --2- 172.21.15.23:0/933053424 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5a44056d90 0x7f5a44059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:30.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.493+0000 7f5a66ffd700 1 --2- 172.21.15.23:0/933053424 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5a44056d90 0x7f5a44059250 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f5a58006fd0 tx=0x7f5a58008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:30.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.496+0000 7f5a4effd700 1 -- 172.21.15.23:0/933053424 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5a60047740 con 0x7f5a680a8d70 2020-11-20T11:32:30.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.645+0000 7f5a67fff700 1 -- 172.21.15.23:0/933053424 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5a68057dc0 con 0x7f5a680a8d70 2020-11-20T11:32:30.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.645+0000 7f5a4effd700 1 -- 172.21.15.23:0/933053424 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+10 (secure 0 0 0) 0x7f5a60015070 con 0x7f5a680a8d70 2020-11-20T11:32:30.648 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 8 2020-11-20T11:32:30.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.646+0000 7f5a67fff700 1 -- 172.21.15.23:0/933053424 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5a44056d90 msgr2=0x7f5a44059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:30.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.646+0000 7f5a67fff700 1 --2- 172.21.15.23:0/933053424 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5a44056d90 0x7f5a44059250 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f5a58006fd0 tx=0x7f5a58008040).stop 2020-11-20T11:32:30.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.646+0000 7f5a67fff700 1 -- 172.21.15.23:0/933053424 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a680a8d70 msgr2=0x7f5a6809d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:30.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.646+0000 7f5a67fff700 1 --2- 172.21.15.23:0/933053424 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a680a8d70 0x7f5a6809d4a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f5a600090b0 tx=0x7f5a60016000).stop 2020-11-20T11:32:30.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.647+0000 7f5a67fff700 1 -- 172.21.15.23:0/933053424 shutdown_connections 2020-11-20T11:32:30.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.647+0000 7f5a67fff700 1 --2- 172.21.15.23:0/933053424 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a680a8d70 0x7f5a6809d4a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:30.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.647+0000 7f5a67fff700 1 --2- 172.21.15.23:0/933053424 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5a44056d90 0x7f5a44059250 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:30.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.647+0000 7f5a67fff700 1 --2- 172.21.15.23:0/933053424 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5a68061380 0x7f5a680a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:30.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.647+0000 7f5a67fff700 1 --2- 172.21.15.23:0/933053424 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5a680a7410 0x7f5a6813f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:30.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.647+0000 7f5a67fff700 1 -- 172.21.15.23:0/933053424 >> 172.21.15.23:0/933053424 conn(0x7f5a680ae0a0 msgr2=0x7f5a68140970 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:30.652 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.647+0000 7f5a67fff700 1 -- 172.21.15.23:0/933053424 shutdown_connections 2020-11-20T11:32:30.652 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.647+0000 7f5a67fff700 1 -- 172.21.15.23:0/933053424 wait complete. 2020-11-20T11:32:30.660 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2020-11-20T11:32:30.661 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2020-11-20T11:32:30.661 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2020-11-20T11:32:30.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.771+0000 7fce18d1c700 1 Processor -- start 2020-11-20T11:32:30.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.771+0000 7fce18d1c700 1 -- start start 2020-11-20T11:32:30.782 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.772+0000 7fce18d1c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fce14060a80 0x7fce14061950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:30.782 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.772+0000 7fce18d1c700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce14067bc0 con 0x7fce140a9d40 2020-11-20T11:32:30.782 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.772+0000 7fce18d1c700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce14067e60 con 0x7fce140aa1c0 2020-11-20T11:32:30.783 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.772+0000 7fce18d1c700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce140aa630 con 0x7fce14060a80 2020-11-20T11:32:30.783 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.772+0000 7fce1259c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fce14060a80 0x7fce14061950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:30.783 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.772+0000 7fce11d9b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fce140a9d40 0x7fce14137920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55016/0 (socket says 172.21.15.23:55016) 2020-11-20T11:32:30.783 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.772+0000 7fce11d9b700 1 -- 172.21.15.23:0/3193099278 learned_addr learned my addr 172.21.15.23:0/3193099278 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:30.783 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.773+0000 7fce1259c700 1 -- 172.21.15.23:0/3193099278 >> v1:172.21.15.155:6789/0 conn(0x7fce140aa1c0 legacy=0x7fce14138030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:30.784 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.773+0000 7fce1259c700 1 -- 172.21.15.23:0/3193099278 >> v1:172.21.15.23:6789/0 conn(0x7fce140a9d40 legacy=0x7fce14137920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:30.784 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.773+0000 7fce1259c700 1 -- 172.21.15.23:0/3193099278 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce14068100 con 0x7fce14060a80 2020-11-20T11:32:30.784 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.773+0000 7fce1159a700 1 -- 172.21.15.23:0/3193099278 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2977831461 0 0) 0x7fce14067bc0 con 0x7fce140a9d40 2020-11-20T11:32:30.785 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.773+0000 7fce1259c700 1 --2- 172.21.15.23:0/3193099278 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce14060a80 0x7fce14061950 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fce04006fd0 tx=0x7fce04006e30).ready entity=mon.0 client_cookie=5ecbd2d0b3cd4b40 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:30.785 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.773+0000 7fce1159a700 1 -- 172.21.15.23:0/3193099278 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce040097e0 con 0x7fce14060a80 2020-11-20T11:32:30.786 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.773+0000 7fce1159a700 1 -- 172.21.15.23:0/3193099278 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce04000e90 con 0x7fce14060a80 2020-11-20T11:32:30.786 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.773+0000 7fce1159a700 1 -- 172.21.15.23:0/3193099278 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce04009b50 con 0x7fce14060a80 2020-11-20T11:32:30.787 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.773+0000 7fce18d1c700 1 -- 172.21.15.23:0/3193099278 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce14060a80 msgr2=0x7fce14061950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:30.787 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.773+0000 7fce18d1c700 1 --2- 172.21.15.23:0/3193099278 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce14060a80 0x7fce14061950 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fce04006fd0 tx=0x7fce04006e30).stop 2020-11-20T11:32:30.788 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.774+0000 7fce18d1c700 1 -- 172.21.15.23:0/3193099278 shutdown_connections 2020-11-20T11:32:30.788 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.774+0000 7fce18d1c700 1 --2- 172.21.15.23:0/3193099278 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce14060a80 0x7fce14061950 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:30.788 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.774+0000 7fce18d1c700 1 -- 172.21.15.23:0/3193099278 >> 172.21.15.23:0/3193099278 conn(0x7fce14066730 msgr2=0x7fce140601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:30.789 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.774+0000 7fce18d1c700 1 -- 172.21.15.23:0/3193099278 shutdown_connections 2020-11-20T11:32:30.789 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.774+0000 7fce18d1c700 1 -- 172.21.15.23:0/3193099278 wait complete. 2020-11-20T11:32:30.789 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.775+0000 7fce18d1c700 1 Processor -- start 2020-11-20T11:32:30.789 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.775+0000 7fce18d1c700 1 -- start start 2020-11-20T11:32:30.790 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.775+0000 7fce18d1c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fce14060a80 0x7fce140a8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:30.790 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.775+0000 7fce18d1c700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fce140a9d40 0x7fce140a35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:30.790 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.775+0000 7fce18d1c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce140aa1c0 0x7fce140a7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:30.790 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.775+0000 7fce18d1c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce140a9110 con 0x7fce140aa1c0 2020-11-20T11:32:30.791 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.775+0000 7fce18d1c700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce140a7a00 con 0x7fce14060a80 2020-11-20T11:32:30.791 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.775+0000 7fce18d1c700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce140a7c30 con 0x7fce140a9d40 2020-11-20T11:32:30.791 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.775+0000 7fce12d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce140aa1c0 0x7fce140a7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:30.791 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.775+0000 7fce12d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce140aa1c0 0x7fce140a7020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40776/0 (socket says 172.21.15.23:40776) 2020-11-20T11:32:30.792 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.775+0000 7fce12d9d700 1 -- 172.21.15.23:0/2620544321 learned_addr learned my addr 172.21.15.23:0/2620544321 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:30.792 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.775+0000 7fce1259c700 1 --2- 172.21.15.23:0/2620544321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fce14060a80 0x7fce140a8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:30.792 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.775+0000 7fce11d9b700 1 --2- 172.21.15.23:0/2620544321 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fce140a9d40 0x7fce140a35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:30.792 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.776+0000 7fce12d9d700 1 -- 172.21.15.23:0/2620544321 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fce140a9d40 msgr2=0x7fce140a35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:30.792 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.776+0000 7fce12d9d700 1 --2- 172.21.15.23:0/2620544321 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fce140a9d40 0x7fce140a35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:30.793 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.776+0000 7fce12d9d700 1 -- 172.21.15.23:0/2620544321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fce14060a80 msgr2=0x7fce140a8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:30.793 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.776+0000 7fce12d9d700 1 --2- 172.21.15.23:0/2620544321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fce14060a80 0x7fce140a8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:30.793 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.776+0000 7fce12d9d700 1 -- 172.21.15.23:0/2620544321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce140a7ed0 con 0x7fce140aa1c0 2020-11-20T11:32:30.793 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.776+0000 7fce12d9d700 1 --2- 172.21.15.23:0/2620544321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce140aa1c0 0x7fce140a7020 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fce0c00ca90 tx=0x7fce0c001400).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:30.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.776+0000 7fce027fc700 1 -- 172.21.15.23:0/2620544321 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0c00ce70 con 0x7fce140aa1c0 2020-11-20T11:32:30.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.776+0000 7fce027fc700 1 -- 172.21.15.23:0/2620544321 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce0c00a760 con 0x7fce140aa1c0 2020-11-20T11:32:30.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.776+0000 7fce18d1c700 1 -- 172.21.15.23:0/2620544321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce140afa70 con 0x7fce140aa1c0 2020-11-20T11:32:30.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.776+0000 7fce027fc700 1 -- 172.21.15.23:0/2620544321 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0c00aae0 con 0x7fce140aa1c0 2020-11-20T11:32:30.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.776+0000 7fce18d1c700 1 -- 172.21.15.23:0/2620544321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce1414d960 con 0x7fce140aa1c0 2020-11-20T11:32:30.795 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.777+0000 7fce18d1c700 1 -- 172.21.15.23:0/2620544321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce140a7750 con 0x7fce140aa1c0 2020-11-20T11:32:30.795 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.777+0000 7fce027fc700 1 -- 172.21.15.23:0/2620544321 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fce0c019030 con 0x7fce140aa1c0 2020-11-20T11:32:30.795 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.777+0000 7fce027fc700 1 --2- 172.21.15.23:0/2620544321 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcde8056d60 0x7fcde8059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:30.795 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.778+0000 7fce027fc700 1 -- 172.21.15.23:0/2620544321 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fce0c06f7c0 con 0x7fce140aa1c0 2020-11-20T11:32:30.796 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.778+0000 7fce1259c700 1 --2- 172.21.15.23:0/2620544321 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcde8056d60 0x7fcde8059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:30.796 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.778+0000 7fce1259c700 1 --2- 172.21.15.23:0/2620544321 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcde8056d60 0x7fcde8059220 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fce0400b150 tx=0x7fce04015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:30.797 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.781+0000 7fce027fc700 1 -- 172.21.15.23:0/2620544321 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fce0c043fd0 con 0x7fce140aa1c0 2020-11-20T11:32:30.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.964+0000 7fce18d1c700 1 -- 172.21.15.23:0/2620544321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fce140afc00 con 0x7fce140aa1c0 2020-11-20T11:32:30.969 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.967+0000 7fce027fc700 1 -- 172.21.15.23:0/2620544321 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+135028 (secure 0 0 0) 0x7fce0c041f90 con 0x7fce140aa1c0 2020-11-20T11:32:30.972 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:32:30.972 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.968+0000 7fce18d1c700 1 -- 172.21.15.23:0/2620544321 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcde8056d60 msgr2=0x7fcde8059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:30.973 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.968+0000 7fce18d1c700 1 --2- 172.21.15.23:0/2620544321 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcde8056d60 0x7fcde8059220 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fce0400b150 tx=0x7fce04015000).stop 2020-11-20T11:32:30.973 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.968+0000 7fce18d1c700 1 -- 172.21.15.23:0/2620544321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce140aa1c0 msgr2=0x7fce140a7020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:30.973 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.968+0000 7fce18d1c700 1 --2- 172.21.15.23:0/2620544321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce140aa1c0 0x7fce140a7020 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fce0c00ca90 tx=0x7fce0c001400).stop 2020-11-20T11:32:30.973 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.969+0000 7fce18d1c700 1 -- 172.21.15.23:0/2620544321 shutdown_connections 2020-11-20T11:32:30.973 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.969+0000 7fce18d1c700 1 --2- 172.21.15.23:0/2620544321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce140aa1c0 0x7fce140a7020 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:30.974 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.969+0000 7fce18d1c700 1 --2- 172.21.15.23:0/2620544321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fce14060a80 0x7fce140a8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:30.974 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.969+0000 7fce18d1c700 1 --2- 172.21.15.23:0/2620544321 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcde8056d60 0x7fcde8059220 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:30.974 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.969+0000 7fce18d1c700 1 --2- 172.21.15.23:0/2620544321 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fce140a9d40 0x7fce140a35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:30.974 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.969+0000 7fce18d1c700 1 -- 172.21.15.23:0/2620544321 >> 172.21.15.23:0/2620544321 conn(0x7fce14066730 msgr2=0x7fce140ac3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:30.974 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.969+0000 7fce18d1c700 1 -- 172.21.15.23:0/2620544321 shutdown_connections 2020-11-20T11:32:30.975 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:30.969+0000 7fce18d1c700 1 -- 172.21.15.23:0/2620544321 wait complete. 2020-11-20T11:32:30.983 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":5,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":22693},{"type":"v1","addr":"172.21.15.155:6833","nonce":22693}]},"active_addr":"172.21.15.155:6833/22693","active_change":"2020-11-20T11:32:09.513324+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[{"gid":4104,"name":"y","mgr_features":4540138297136906239,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.155:0","nonce":619116038}]},{"addrvec":[{"type":"v2","addr":"172.21.15.155:0","nonce":2113846144}]},{"addrvec":[{"type":"v2","addr":"172.21.15.155:0","nonce":1346665239}]}]}} 2020-11-20T11:32:30.987 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2020-11-20T11:32:30.987 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2020-11-20T11:32:30.987 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T11:32:31.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.098+0000 7ff342ccc700 1 Processor -- start 2020-11-20T11:32:31.102 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.098+0000 7ff342ccc700 1 -- start start 2020-11-20T11:32:31.104 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.099+0000 7ff342ccc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff33c0aae90 0x7ff33c1381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.104 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.099+0000 7ff342ccc700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff33c13aeb0 0x7ff33c0a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.104 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.099+0000 7ff342ccc700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff33c13e600 con 0x7ff33c0aaab0 2020-11-20T11:32:31.104 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.099+0000 7ff342ccc700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff33c13e8a0 con 0x7ff33c0aae90 2020-11-20T11:32:31.104 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.099+0000 7ff342ccc700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff33c0ab6d0 con 0x7ff33c13aeb0 2020-11-20T11:32:31.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.099+0000 7ff33bfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff33c0aae90 0x7ff33c1381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:31.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.099+0000 7ff340a68700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7ff33c0aaab0 0x7ff33c13ed40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55030/0 (socket says 172.21.15.23:55030) 2020-11-20T11:32:31.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.099+0000 7ff33bfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff33c0aae90 0x7ff33c1381a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40782/0 (socket says 172.21.15.23:40782) 2020-11-20T11:32:31.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.099+0000 7ff340a68700 1 -- 172.21.15.23:0/1812204830 learned_addr learned my addr 172.21.15.23:0/1812204830 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:31.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.099+0000 7ff341269700 1 --2- 172.21.15.23:0/1812204830 >> v2:172.21.15.155:3300/0 conn(0x7ff33c13aeb0 0x7ff33c0a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:31.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.100+0000 7ff33bfff700 1 -- 172.21.15.23:0/1812204830 >> v2:172.21.15.155:3300/0 conn(0x7ff33c13aeb0 msgr2=0x7ff33c0a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:31.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.100+0000 7ff33bfff700 1 --2- 172.21.15.23:0/1812204830 >> v2:172.21.15.155:3300/0 conn(0x7ff33c13aeb0 0x7ff33c0a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:31.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.100+0000 7ff33bfff700 1 -- 172.21.15.23:0/1812204830 >> v1:172.21.15.23:6789/0 conn(0x7ff33c0aaab0 legacy=0x7ff33c13ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.100+0000 7ff33bfff700 1 -- 172.21.15.23:0/1812204830 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff33c138690 con 0x7ff33c0aae90 2020-11-20T11:32:31.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.100+0000 7ff33bfff700 1 --2- 172.21.15.23:0/1812204830 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff33c0aae90 0x7ff33c1381a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7ff33c062f40 tx=0x7ff328006e30).ready entity=mon.0 client_cookie=d57a4026c42debbd server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:31.107 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.100+0000 7ff33b7fe700 1 -- 172.21.15.23:0/1812204830 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3280097b0 con 0x7ff33c0aae90 2020-11-20T11:32:31.107 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.100+0000 7ff33b7fe700 1 -- 172.21.15.23:0/1812204830 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff328000e90 con 0x7ff33c0aae90 2020-11-20T11:32:31.107 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.100+0000 7ff33b7fe700 1 -- 172.21.15.23:0/1812204830 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff328009b20 con 0x7ff33c0aae90 2020-11-20T11:32:31.107 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.100+0000 7ff342ccc700 1 -- 172.21.15.23:0/1812204830 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff33c0aae90 msgr2=0x7ff33c1381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.107 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.100+0000 7ff342ccc700 1 --2- 172.21.15.23:0/1812204830 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff33c0aae90 0x7ff33c1381a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7ff33c062f40 tx=0x7ff328006e30).stop 2020-11-20T11:32:31.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.100+0000 7ff342ccc700 1 -- 172.21.15.23:0/1812204830 shutdown_connections 2020-11-20T11:32:31.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.100+0000 7ff342ccc700 1 --2- 172.21.15.23:0/1812204830 >> v2:172.21.15.155:3300/0 conn(0x7ff33c13aeb0 0x7ff33c0a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:31.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.100+0000 7ff342ccc700 1 --2- 172.21.15.23:0/1812204830 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff33c0aae90 0x7ff33c1381a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:31.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.100+0000 7ff342ccc700 1 -- 172.21.15.23:0/1812204830 >> 172.21.15.23:0/1812204830 conn(0x7ff33c0678e0 msgr2=0x7ff33c067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:31.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.100+0000 7ff342ccc700 1 -- 172.21.15.23:0/1812204830 shutdown_connections 2020-11-20T11:32:31.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.100+0000 7ff342ccc700 1 -- 172.21.15.23:0/1812204830 wait complete. 2020-11-20T11:32:31.111 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.101+0000 7ff342ccc700 1 Processor -- start 2020-11-20T11:32:31.111 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.102+0000 7ff342ccc700 1 -- start start 2020-11-20T11:32:31.111 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.102+0000 7ff342ccc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff33c0aaab0 0x7ff33c0b0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.111 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.102+0000 7ff342ccc700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff33c0aae90 0x7ff33c09e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.111 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.102+0000 7ff342ccc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff33c0a4c80 0x7ff33c14d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.112 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.102+0000 7ff342ccc700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33c09f140 con 0x7ff33c0aaab0 2020-11-20T11:32:31.112 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.102+0000 7ff342ccc700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33c0a7370 con 0x7ff33c0a4c80 2020-11-20T11:32:31.112 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.102+0000 7ff342ccc700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff33c0a7610 con 0x7ff33c0aae90 2020-11-20T11:32:31.112 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.102+0000 7ff340a68700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff33c0aaab0 0x7ff33c0b0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:31.112 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.102+0000 7ff340a68700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff33c0aaab0 0x7ff33c0b0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40786/0 (socket says 172.21.15.23:40786) 2020-11-20T11:32:31.113 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.102+0000 7ff340a68700 1 -- 172.21.15.23:0/2381955860 learned_addr learned my addr 172.21.15.23:0/2381955860 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:31.113 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.102+0000 7ff340a68700 1 -- 172.21.15.23:0/2381955860 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff33c0aae90 msgr2=0x7ff33c09e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.113 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.102+0000 7ff340a68700 1 --2- 172.21.15.23:0/2381955860 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff33c0aae90 0x7ff33c09e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:32:31.113 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.102+0000 7ff340a68700 1 -- 172.21.15.23:0/2381955860 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff33c0a4c80 msgr2=0x7ff33c14d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.113 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.102+0000 7ff340a68700 1 --2- 172.21.15.23:0/2381955860 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff33c0a4c80 0x7ff33c14d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:32:31.114 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.102+0000 7ff340a68700 1 -- 172.21.15.23:0/2381955860 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff33c09ee40 con 0x7ff33c0aaab0 2020-11-20T11:32:31.114 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.103+0000 7ff340a68700 1 --2- 172.21.15.23:0/2381955860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff33c0aaab0 0x7ff33c0b0b30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7ff33000c710 tx=0x7ff330007c80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:31.114 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.103+0000 7ff338ff9700 1 -- 172.21.15.23:0/2381955860 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff330007320 con 0x7ff33c0aaab0 2020-11-20T11:32:31.114 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.103+0000 7ff338ff9700 1 -- 172.21.15.23:0/2381955860 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff330007480 con 0x7ff33c0aaab0 2020-11-20T11:32:31.114 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.103+0000 7ff342ccc700 1 -- 172.21.15.23:0/2381955860 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff33c0a7ab0 con 0x7ff33c0aaab0 2020-11-20T11:32:31.115 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.103+0000 7ff338ff9700 1 -- 172.21.15.23:0/2381955860 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff330005840 con 0x7ff33c0aaab0 2020-11-20T11:32:31.115 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.103+0000 7ff342ccc700 1 -- 172.21.15.23:0/2381955860 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff33c133680 con 0x7ff33c0aaab0 2020-11-20T11:32:31.115 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.104+0000 7ff342ccc700 1 -- 172.21.15.23:0/2381955860 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff33c05c7d0 con 0x7ff33c0aaab0 2020-11-20T11:32:31.115 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.104+0000 7ff338ff9700 1 -- 172.21.15.23:0/2381955860 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff330010030 con 0x7ff33c0aaab0 2020-11-20T11:32:31.115 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.104+0000 7ff338ff9700 1 --2- 172.21.15.23:0/2381955860 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff314056d60 0x7ff314059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.116 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.104+0000 7ff338ff9700 1 -- 172.21.15.23:0/2381955860 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7ff33006fe30 con 0x7ff33c0aaab0 2020-11-20T11:32:31.116 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.105+0000 7ff33bfff700 1 --2- 172.21.15.23:0/2381955860 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff314056d60 0x7ff314059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:31.116 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.105+0000 7ff33bfff700 1 --2- 172.21.15.23:0/2381955860 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff314056d60 0x7ff314059220 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7ff33c0ad9b0 tx=0x7ff328015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:31.116 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.108+0000 7ff338ff9700 1 -- 172.21.15.23:0/2381955860 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff330045120 con 0x7ff33c0aaab0 2020-11-20T11:32:31.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.256+0000 7ff342ccc700 1 -- 172.21.15.23:0/2381955860 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff33c0a77d0 con 0x7ff33c0aaab0 2020-11-20T11:32:31.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.257+0000 7ff338ff9700 1 -- 172.21.15.23:0/2381955860 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17087 (secure 0 0 0) 0x7ff330048740 con 0x7ff33c0aaab0 2020-11-20T11:32:31.260 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:32:31.260 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":25,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","created":"2020-11-20T11:32:06.136679+0000","modified":"2020-11-20T11:32:26.470927+0000","last_up_change":"2020-11-20T11:32:13.536194+0000","last_in_change":"2020-11-20T11:32:13.536194+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T11:32:09.559868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T11:32:17.000782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T11:32:20.423341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T11:32:23.126273+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e9d219f5-e40e-4b2b-9468-deb9c4074a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":23083},{"type":"v1","addr":"172.21.15.23:6801","nonce":23083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":23083},{"type":"v1","addr":"172.21.15.23:6803","nonce":23083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":23083},{"type":"v1","addr":"172.21.15.23:6807","nonce":23083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":23083},{"type":"v1","addr":"172.21.15.23:6805","nonce":23083}]},"public_addr":"172.21.15.23:6801/23083","cluster_addr":"172.21.15.23:6803/23083","heartbeat_back_addr":"172.21.15.23:6807/23083","heartbeat_front_addr":"172.21.15.23:6805/23083","state":["exists","up"]},{"osd":1,"uuid":"2328adde-2108-4fe0-8138-f0d096a496a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":23099},{"type":"v1","addr":"172.21.15.23:6817","nonce":23099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":23099},{"type":"v1","addr":"172.21.15.23:6819","nonce":23099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":23099},{"type":"v1","addr":"172.21.15.23:6824","nonce":23099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":23099},{"type":"v1","addr":"172.21.15.23:6821","nonce":23099}]},"public_addr":"172.21.15.23:6817/23099","cluster_addr":"172.21.15.23:6819/23099","heartbeat_back_addr":"172.21.15.23:6824/23099","heartbeat_front_addr":"172.21.15.23:6821/23099","state":["exists","up"]},{"osd":2,"uuid":"13367708-59a2-4363-8a3a-ab0cb97145ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":23084},{"type":"v1","addr":"172.21.15.23:6809","nonce":23084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":23084},{"type":"v1","addr":"172.21.15.23:6811","nonce":23084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":23084},{"type":"v1","addr":"172.21.15.23:6815","nonce":23084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":23084},{"type":"v1","addr":"172.21.15.23:6813","nonce":23084}]},"public_addr":"172.21.15.23:6809/23084","cluster_addr":"172.21.15.23:6811/23084","heartbeat_back_addr":"172.21.15.23:6815/23084","heartbeat_front_addr":"172.21.15.23:6813/23084","state":["exists","up"]},{"osd":3,"uuid":"1f147dd7-9289-47b0-ad1c-4088618d6faa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6823","nonce":23114},{"type":"v1","addr":"172.21.15.23:6825","nonce":23114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":23114},{"type":"v1","addr":"172.21.15.23:6827","nonce":23114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":23114},{"type":"v1","addr":"172.21.15.23:6831","nonce":23114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":23114},{"type":"v1","addr":"172.21.15.23:6829","nonce":23114}]},"public_addr":"172.21.15.23:6825/23114","cluster_addr":"172.21.15.23:6827/23114","heartbeat_back_addr":"172.21.15.23:6831/23114","heartbeat_front_addr":"172.21.15.23:6829/23114","state":["exists","up"]},{"osd":4,"uuid":"d008b039-4bc5-4135-9678-1511e58efc29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":23280},{"type":"v1","addr":"172.21.15.155:6801","nonce":23280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":23280},{"type":"v1","addr":"172.21.15.155:6803","nonce":23280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":23280},{"type":"v1","addr":"172.21.15.155:6807","nonce":23280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":23280},{"type":"v1","addr":"172.21.15.155:6805","nonce":23280}]},"public_addr":"172.21.15.155:6801/23280","cluster_addr":"172.21.15.155:6803/23280","heartbeat_back_addr":"172.21.15.155:6807/23280","heartbeat_front_addr":"172.21.15.155:6805/23280","state":["exists","up"]},{"osd":5,"uuid":"56715e6f-cf6e-44a6-9e39-93ab1224095f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":23298},{"type":"v1","addr":"172.21.15.155:6817","nonce":23298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":23298},{"type":"v1","addr":"172.21.15.155:6819","nonce":23298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":23298},{"type":"v1","addr":"172.21.15.155:6823","nonce":23298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":23298},{"type":"v1","addr":"172.21.15.155:6821","nonce":23298}]},"public_addr":"172.21.15.155:6817/23298","cluster_addr":"172.21.15.155:6819/23298","heartbeat_back_addr":"172.21.15.155:6823/23298","heartbeat_front_addr":"172.21.15.155:6821/23298","state":["exists","up"]},{"osd":6,"uuid":"db1e61ca-8e14-4a1f-8607-6e01b148d61c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":23307},{"type":"v1","addr":"172.21.15.155:6825","nonce":23307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":23307},{"type":"v1","addr":"172.21.15.155:6827","nonce":23307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":23307},{"type":"v1","addr":"172.21.15.155:6831","nonce":23307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":23307},{"type":"v1","addr":"172.21.15.155:6829","nonce":23307}]},"public_addr":"172.21.15.155:6825/23307","cluster_addr":"172.21.15.155:6827/23307","heartbeat_back_addr":"172.21.15.155:6831/23307","heartbeat_front_addr":"172.21.15.155:6829/23307","state":["exists","up"]},{"osd":7,"uuid":"d44419be-ef89-462c-8cf1-3c6db2a6944b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":23281},{"type":"v1","addr":"172.21.15.155:6809","nonce":23281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":23281},{"type":"v1","addr":"172.21.15.155:6811","nonce":23281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":23281},{"type":"v1","addr":"172.21.15.155:6815","nonce":23281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":23281},{"type":"v1","addr":"172.21.15.155:6813","nonce":23281}]},"public_addr":"172.21.15.155:6809/23281","cluster_addr":"172.21.15.155:6811/23281","heartbeat_back_addr":"172.21.15.155:6815/23281","heartbeat_front_addr":"172.21.15.155:6813/23281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.486702+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.509396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.533414+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.509489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.467600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.513068+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.536636+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.531772+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T11:32:31.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.258+0000 7ff342ccc700 1 -- 172.21.15.23:0/2381955860 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff314056d60 msgr2=0x7ff314059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.258+0000 7ff342ccc700 1 --2- 172.21.15.23:0/2381955860 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff314056d60 0x7ff314059220 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7ff33c0ad9b0 tx=0x7ff328015000).stop 2020-11-20T11:32:31.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.258+0000 7ff342ccc700 1 -- 172.21.15.23:0/2381955860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff33c0aaab0 msgr2=0x7ff33c0b0b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.258+0000 7ff342ccc700 1 --2- 172.21.15.23:0/2381955860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff33c0aaab0 0x7ff33c0b0b30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7ff33000c710 tx=0x7ff330007c80).stop 2020-11-20T11:32:31.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.258+0000 7ff342ccc700 1 -- 172.21.15.23:0/2381955860 shutdown_connections 2020-11-20T11:32:31.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.258+0000 7ff342ccc700 1 --2- 172.21.15.23:0/2381955860 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff33c0a4c80 0x7ff33c14d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:32:31.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.258+0000 7ff342ccc700 1 --2- 172.21.15.23:0/2381955860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff33c0aaab0 0x7ff33c0b0b30 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:31.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.258+0000 7ff342ccc700 1 --2- 172.21.15.23:0/2381955860 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff314056d60 0x7ff314059220 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:31.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.258+0000 7ff342ccc700 1 --2- 172.21.15.23:0/2381955860 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff33c0aae90 0x7ff33c09e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:32:31.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.258+0000 7ff342ccc700 1 -- 172.21.15.23:0/2381955860 >> 172.21.15.23:0/2381955860 conn(0x7ff33c0678e0 msgr2=0x7ff33c137f40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:31.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.258+0000 7ff342ccc700 1 -- 172.21.15.23:0/2381955860 shutdown_connections 2020-11-20T11:32:31.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.258+0000 7ff342ccc700 1 -- 172.21.15.23:0/2381955860 wait complete. 2020-11-20T11:32:31.273 INFO:tasks.ceph.ceph_manager.ceph:all up! 2020-11-20T11:32:31.274 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T11:32:31.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.385+0000 7fb72bfff700 1 Processor -- start 2020-11-20T11:32:31.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.385+0000 7fb72bfff700 1 -- start start 2020-11-20T11:32:31.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.385+0000 7fb72bfff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb72c0665f0 0x7fb72c13adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.385+0000 7fb72bfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb72c0ab2a0 0x7fb72c137f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.386+0000 7fb72bfff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb72c13bd10 con 0x7fb72c066210 2020-11-20T11:32:31.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.386+0000 7fb72bfff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb72c0ab700 con 0x7fb72c0ab2a0 2020-11-20T11:32:31.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.386+0000 7fb72bfff700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb72c0a9700 con 0x7fb72c0665f0 2020-11-20T11:32:31.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.386+0000 7fb72b7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb72c0ab2a0 0x7fb72c137f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:31.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.386+0000 7fb72affd700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fb72c066210 0x7fb72c0a8ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55044/0 (socket says 172.21.15.23:55044) 2020-11-20T11:32:31.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.386+0000 7fb72affd700 1 -- 172.21.15.23:0/2912182088 learned_addr learned my addr 172.21.15.23:0/2912182088 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:31.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.386+0000 7fb72a7fc700 1 --2- 172.21.15.23:0/2912182088 >> v2:172.21.15.155:3300/0 conn(0x7fb72c0665f0 0x7fb72c13adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:31.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.386+0000 7fb72b7fe700 1 -- 172.21.15.23:0/2912182088 >> v2:172.21.15.155:3300/0 conn(0x7fb72c0665f0 msgr2=0x7fb72c13adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:31.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.386+0000 7fb72b7fe700 1 --2- 172.21.15.23:0/2912182088 >> v2:172.21.15.155:3300/0 conn(0x7fb72c0665f0 0x7fb72c13adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:31.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.386+0000 7fb72b7fe700 1 -- 172.21.15.23:0/2912182088 >> v1:172.21.15.23:6789/0 conn(0x7fb72c066210 legacy=0x7fb72c0a8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.394 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.386+0000 7fb72b7fe700 1 -- 172.21.15.23:0/2912182088 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72c0aa930 con 0x7fb72c0ab2a0 2020-11-20T11:32:31.394 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.386+0000 7fb729ffb700 1 -- 172.21.15.23:0/2912182088 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928068894 0 0) 0x7fb72c13bd10 con 0x7fb72c066210 2020-11-20T11:32:31.394 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.386+0000 7fb72a7fc700 1 --2- 172.21.15.23:0/2912182088 >> v2:172.21.15.155:3300/0 conn(0x7fb72c0665f0 0x7fb72c13adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:31.394 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.386+0000 7fb72b7fe700 1 --2- 172.21.15.23:0/2912182088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb72c0ab2a0 0x7fb72c137f30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fb72c0a98a0 tx=0x7fb724009220).ready entity=mon.0 client_cookie=9d5305d1f5fc919f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:31.394 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.386+0000 7fb729ffb700 1 -- 172.21.15.23:0/2912182088 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72400bbc0 con 0x7fb72c0ab2a0 2020-11-20T11:32:31.395 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.386+0000 7fb729ffb700 1 -- 172.21.15.23:0/2912182088 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb72400bd20 con 0x7fb72c0ab2a0 2020-11-20T11:32:31.395 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.386+0000 7fb729ffb700 1 -- 172.21.15.23:0/2912182088 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb724011610 con 0x7fb72c0ab2a0 2020-11-20T11:32:31.395 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.387+0000 7fb72bfff700 1 -- 172.21.15.23:0/2912182088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb72c0ab2a0 msgr2=0x7fb72c137f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.395 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.387+0000 7fb72bfff700 1 --2- 172.21.15.23:0/2912182088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb72c0ab2a0 0x7fb72c137f30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fb72c0a98a0 tx=0x7fb724009220).stop 2020-11-20T11:32:31.396 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.387+0000 7fb72bfff700 1 -- 172.21.15.23:0/2912182088 shutdown_connections 2020-11-20T11:32:31.396 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.387+0000 7fb72bfff700 1 --2- 172.21.15.23:0/2912182088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb72c0ab2a0 0x7fb72c137f30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:31.396 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.387+0000 7fb72bfff700 1 --2- 172.21.15.23:0/2912182088 >> v2:172.21.15.155:3300/0 conn(0x7fb72c0665f0 0x7fb72c13adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:31.396 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.387+0000 7fb72bfff700 1 -- 172.21.15.23:0/2912182088 >> 172.21.15.23:0/2912182088 conn(0x7fb72c0612c0 msgr2=0x7fb72c0616d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:31.397 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.387+0000 7fb72bfff700 1 -- 172.21.15.23:0/2912182088 shutdown_connections 2020-11-20T11:32:31.397 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.387+0000 7fb72bfff700 1 -- 172.21.15.23:0/2912182088 wait complete. 2020-11-20T11:32:31.397 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.388+0000 7fb72bfff700 1 Processor -- start 2020-11-20T11:32:31.397 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.388+0000 7fb72bfff700 1 -- start start 2020-11-20T11:32:31.400 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.389+0000 7fb72bfff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb72c066210 0x7fb72c13dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.400 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.389+0000 7fb72bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb72c0665f0 0x7fb72c0a7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.400 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.389+0000 7fb72bfff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb72c0ab2a0 0x7fb72c14d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.400 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.389+0000 7fb72bfff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c13e530 con 0x7fb72c0665f0 2020-11-20T11:32:31.401 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.389+0000 7fb72bfff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c13e290 con 0x7fb72c066210 2020-11-20T11:32:31.401 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.389+0000 7fb72bfff700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c14d480 con 0x7fb72c0ab2a0 2020-11-20T11:32:31.401 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.389+0000 7fb72a7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb72c0665f0 0x7fb72c0a7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:31.401 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.389+0000 7fb72a7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb72c0665f0 0x7fb72c0a7990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40802/0 (socket says 172.21.15.23:40802) 2020-11-20T11:32:31.402 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.389+0000 7fb72a7fc700 1 -- 172.21.15.23:0/1298384405 learned_addr learned my addr 172.21.15.23:0/1298384405 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:31.402 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.389+0000 7fb72affd700 1 --2- 172.21.15.23:0/1298384405 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb72c066210 0x7fb72c13dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:31.402 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.389+0000 7fb72b7fe700 1 --2- 172.21.15.23:0/1298384405 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb72c0ab2a0 0x7fb72c14d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:31.403 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.389+0000 7fb72a7fc700 1 -- 172.21.15.23:0/1298384405 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb72c0ab2a0 msgr2=0x7fb72c14d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.403 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.389+0000 7fb72a7fc700 1 --2- 172.21.15.23:0/1298384405 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb72c0ab2a0 0x7fb72c14d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:31.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.389+0000 7fb72a7fc700 1 -- 172.21.15.23:0/1298384405 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb72c066210 msgr2=0x7fb72c13dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.389+0000 7fb72a7fc700 1 --2- 172.21.15.23:0/1298384405 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb72c066210 0x7fb72c13dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:31.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.389+0000 7fb72a7fc700 1 -- 172.21.15.23:0/1298384405 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72c0a2e40 con 0x7fb72c0665f0 2020-11-20T11:32:31.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.389+0000 7fb72affd700 1 --2- 172.21.15.23:0/1298384405 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb72c066210 0x7fb72c13dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:32:31.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.389+0000 7fb72a7fc700 1 --2- 172.21.15.23:0/1298384405 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb72c0665f0 0x7fb72c0a7990 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fb72000db60 tx=0x7fb720008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:31.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.389+0000 7fb712ffd700 1 -- 172.21.15.23:0/1298384405 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb720009820 con 0x7fb72c0665f0 2020-11-20T11:32:31.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.389+0000 7fb712ffd700 1 -- 172.21.15.23:0/1298384405 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb720009980 con 0x7fb72c0665f0 2020-11-20T11:32:31.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.389+0000 7fb72bfff700 1 -- 172.21.15.23:0/1298384405 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb72c13c660 con 0x7fb72c0665f0 2020-11-20T11:32:31.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.389+0000 7fb712ffd700 1 -- 172.21.15.23:0/1298384405 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb720009d40 con 0x7fb72c0665f0 2020-11-20T11:32:31.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.389+0000 7fb72bfff700 1 -- 172.21.15.23:0/1298384405 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb72c14e050 con 0x7fb72c0665f0 2020-11-20T11:32:31.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.390+0000 7fb72bfff700 1 -- 172.21.15.23:0/1298384405 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb72c14e270 con 0x7fb72c0665f0 2020-11-20T11:32:31.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.391+0000 7fb712ffd700 1 -- 172.21.15.23:0/1298384405 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb720013690 con 0x7fb72c0665f0 2020-11-20T11:32:31.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.391+0000 7fb712ffd700 1 --2- 172.21.15.23:0/1298384405 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb708056d60 0x7fb708059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.391+0000 7fb712ffd700 1 -- 172.21.15.23:0/1298384405 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fb720070190 con 0x7fb72c0665f0 2020-11-20T11:32:31.407 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.391+0000 7fb72affd700 1 --2- 172.21.15.23:0/1298384405 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb708056d60 0x7fb708059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:31.407 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.391+0000 7fb72affd700 1 --2- 172.21.15.23:0/1298384405 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb708056d60 0x7fb708059220 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fb71c0015d0 tx=0x7fb71c009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:31.407 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.395+0000 7fb712ffd700 1 -- 172.21.15.23:0/1298384405 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb72c14e270 con 0x7fb72c0665f0 2020-11-20T11:32:31.547 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.545+0000 7fb72bfff700 1 -- 172.21.15.23:0/1298384405 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb72c14e270 con 0x7fb72c0665f0 2020-11-20T11:32:31.548 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.545+0000 7fb712ffd700 1 -- 172.21.15.23:0/1298384405 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17087 (secure 0 0 0) 0x7fb72c14e270 con 0x7fb72c0665f0 2020-11-20T11:32:31.549 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:32:31.549 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":25,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","created":"2020-11-20T11:32:06.136679+0000","modified":"2020-11-20T11:32:26.470927+0000","last_up_change":"2020-11-20T11:32:13.536194+0000","last_in_change":"2020-11-20T11:32:13.536194+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T11:32:09.559868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T11:32:17.000782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T11:32:20.423341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T11:32:23.126273+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e9d219f5-e40e-4b2b-9468-deb9c4074a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":23083},{"type":"v1","addr":"172.21.15.23:6801","nonce":23083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":23083},{"type":"v1","addr":"172.21.15.23:6803","nonce":23083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":23083},{"type":"v1","addr":"172.21.15.23:6807","nonce":23083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":23083},{"type":"v1","addr":"172.21.15.23:6805","nonce":23083}]},"public_addr":"172.21.15.23:6801/23083","cluster_addr":"172.21.15.23:6803/23083","heartbeat_back_addr":"172.21.15.23:6807/23083","heartbeat_front_addr":"172.21.15.23:6805/23083","state":["exists","up"]},{"osd":1,"uuid":"2328adde-2108-4fe0-8138-f0d096a496a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":23099},{"type":"v1","addr":"172.21.15.23:6817","nonce":23099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":23099},{"type":"v1","addr":"172.21.15.23:6819","nonce":23099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":23099},{"type":"v1","addr":"172.21.15.23:6824","nonce":23099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":23099},{"type":"v1","addr":"172.21.15.23:6821","nonce":23099}]},"public_addr":"172.21.15.23:6817/23099","cluster_addr":"172.21.15.23:6819/23099","heartbeat_back_addr":"172.21.15.23:6824/23099","heartbeat_front_addr":"172.21.15.23:6821/23099","state":["exists","up"]},{"osd":2,"uuid":"13367708-59a2-4363-8a3a-ab0cb97145ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":23084},{"type":"v1","addr":"172.21.15.23:6809","nonce":23084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":23084},{"type":"v1","addr":"172.21.15.23:6811","nonce":23084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":23084},{"type":"v1","addr":"172.21.15.23:6815","nonce":23084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":23084},{"type":"v1","addr":"172.21.15.23:6813","nonce":23084}]},"public_addr":"172.21.15.23:6809/23084","cluster_addr":"172.21.15.23:6811/23084","heartbeat_back_addr":"172.21.15.23:6815/23084","heartbeat_front_addr":"172.21.15.23:6813/23084","state":["exists","up"]},{"osd":3,"uuid":"1f147dd7-9289-47b0-ad1c-4088618d6faa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6823","nonce":23114},{"type":"v1","addr":"172.21.15.23:6825","nonce":23114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":23114},{"type":"v1","addr":"172.21.15.23:6827","nonce":23114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":23114},{"type":"v1","addr":"172.21.15.23:6831","nonce":23114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":23114},{"type":"v1","addr":"172.21.15.23:6829","nonce":23114}]},"public_addr":"172.21.15.23:6825/23114","cluster_addr":"172.21.15.23:6827/23114","heartbeat_back_addr":"172.21.15.23:6831/23114","heartbeat_front_addr":"172.21.15.23:6829/23114","state":["exists","up"]},{"osd":4,"uuid":"d008b039-4bc5-4135-9678-1511e58efc29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":23280},{"type":"v1","addr":"172.21.15.155:6801","nonce":23280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":23280},{"type":"v1","addr":"172.21.15.155:6803","nonce":23280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":23280},{"type":"v1","addr":"172.21.15.155:6807","nonce":23280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":23280},{"type":"v1","addr":"172.21.15.155:6805","nonce":23280}]},"public_addr":"172.21.15.155:6801/23280","cluster_addr":"172.21.15.155:6803/23280","heartbeat_back_addr":"172.21.15.155:6807/23280","heartbeat_front_addr":"172.21.15.155:6805/23280","state":["exists","up"]},{"osd":5,"uuid":"56715e6f-cf6e-44a6-9e39-93ab1224095f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":23298},{"type":"v1","addr":"172.21.15.155:6817","nonce":23298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":23298},{"type":"v1","addr":"172.21.15.155:6819","nonce":23298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":23298},{"type":"v1","addr":"172.21.15.155:6823","nonce":23298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":23298},{"type":"v1","addr":"172.21.15.155:6821","nonce":23298}]},"public_addr":"172.21.15.155:6817/23298","cluster_addr":"172.21.15.155:6819/23298","heartbeat_back_addr":"172.21.15.155:6823/23298","heartbeat_front_addr":"172.21.15.155:6821/23298","state":["exists","up"]},{"osd":6,"uuid":"db1e61ca-8e14-4a1f-8607-6e01b148d61c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":23307},{"type":"v1","addr":"172.21.15.155:6825","nonce":23307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":23307},{"type":"v1","addr":"172.21.15.155:6827","nonce":23307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":23307},{"type":"v1","addr":"172.21.15.155:6831","nonce":23307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":23307},{"type":"v1","addr":"172.21.15.155:6829","nonce":23307}]},"public_addr":"172.21.15.155:6825/23307","cluster_addr":"172.21.15.155:6827/23307","heartbeat_back_addr":"172.21.15.155:6831/23307","heartbeat_front_addr":"172.21.15.155:6829/23307","state":["exists","up"]},{"osd":7,"uuid":"d44419be-ef89-462c-8cf1-3c6db2a6944b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":23281},{"type":"v1","addr":"172.21.15.155:6809","nonce":23281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":23281},{"type":"v1","addr":"172.21.15.155:6811","nonce":23281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":23281},{"type":"v1","addr":"172.21.15.155:6815","nonce":23281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":23281},{"type":"v1","addr":"172.21.15.155:6813","nonce":23281}]},"public_addr":"172.21.15.155:6809/23281","cluster_addr":"172.21.15.155:6811/23281","heartbeat_back_addr":"172.21.15.155:6815/23281","heartbeat_front_addr":"172.21.15.155:6813/23281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.486702+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.509396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.533414+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.509489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.467600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.513068+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.536636+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.531772+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T11:32:31.550 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.547+0000 7fb72bfff700 1 -- 172.21.15.23:0/1298384405 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb708056d60 msgr2=0x7fb708059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.550 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.547+0000 7fb72bfff700 1 --2- 172.21.15.23:0/1298384405 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb708056d60 0x7fb708059220 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fb71c0015d0 tx=0x7fb71c009040).stop 2020-11-20T11:32:31.551 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.547+0000 7fb72bfff700 1 -- 172.21.15.23:0/1298384405 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb72c0665f0 msgr2=0x7fb72c0a7990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.551 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.547+0000 7fb72bfff700 1 --2- 172.21.15.23:0/1298384405 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb72c0665f0 0x7fb72c0a7990 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fb72000db60 tx=0x7fb720008040).stop 2020-11-20T11:32:31.551 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.547+0000 7fb72bfff700 1 -- 172.21.15.23:0/1298384405 shutdown_connections 2020-11-20T11:32:31.551 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.547+0000 7fb72bfff700 1 --2- 172.21.15.23:0/1298384405 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb708056d60 0x7fb708059220 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:31.552 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.547+0000 7fb72bfff700 1 --2- 172.21.15.23:0/1298384405 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb72c0ab2a0 0x7fb72c14d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:31.552 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.547+0000 7fb72bfff700 1 --2- 172.21.15.23:0/1298384405 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb72c066210 0x7fb72c13dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:31.552 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.547+0000 7fb72bfff700 1 --2- 172.21.15.23:0/1298384405 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb72c0665f0 0x7fb72c0a7990 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:31.552 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.547+0000 7fb72bfff700 1 -- 172.21.15.23:0/1298384405 >> 172.21.15.23:0/1298384405 conn(0x7fb72c0612c0 msgr2=0x7fb72c0a8e90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:31.553 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.547+0000 7fb72bfff700 1 -- 172.21.15.23:0/1298384405 shutdown_connections 2020-11-20T11:32:31.553 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.547+0000 7fb72bfff700 1 -- 172.21.15.23:0/1298384405 wait complete. 2020-11-20T11:32:31.562 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2020-11-20T11:32:31.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.673+0000 7f0e89afa700 1 Processor -- start 2020-11-20T11:32:31.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.673+0000 7f0e89afa700 1 -- start start 2020-11-20T11:32:31.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.674+0000 7f0e89afa700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0e840aa1c0 0x7f0e840ac290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.674+0000 7f0e89afa700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e84067bc0 con 0x7f0e84060a80 2020-11-20T11:32:31.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.674+0000 7f0e89afa700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e84067e60 con 0x7f0e840a9d40 2020-11-20T11:32:31.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.674+0000 7f0e89afa700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e840aa630 con 0x7f0e840aa1c0 2020-11-20T11:32:31.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.674+0000 7f0e837fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f0e84060a80 0x7f0e84061950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55058/0 (socket says 172.21.15.23:55058) 2020-11-20T11:32:31.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.674+0000 7f0e837fe700 1 -- 172.21.15.23:0/271024480 learned_addr learned my addr 172.21.15.23:0/271024480 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:31.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.674+0000 7f0e83fff700 1 --2- 172.21.15.23:0/271024480 >> v2:172.21.15.155:3300/0 conn(0x7f0e840aa1c0 0x7f0e840ac290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:31.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.674+0000 7f0e827fc700 1 -- 172.21.15.23:0/271024480 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3741609889 0 0) 0x7f0e84067bc0 con 0x7f0e84060a80 2020-11-20T11:32:31.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.674+0000 7f0e827fc700 1 -- 172.21.15.23:0/271024480 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e70003280 con 0x7f0e84060a80 2020-11-20T11:32:31.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.674+0000 7f0e827fc700 1 -- 172.21.15.23:0/271024480 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2404088804 0 0) 0x7f0e70003280 con 0x7f0e84060a80 2020-11-20T11:32:31.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.674+0000 7f0e827fc700 1 -- 172.21.15.23:0/271024480 >> v2:172.21.15.155:3300/0 conn(0x7f0e840aa1c0 msgr2=0x7f0e840ac290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:31.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.674+0000 7f0e827fc700 1 --2- 172.21.15.23:0/271024480 >> v2:172.21.15.155:3300/0 conn(0x7f0e840aa1c0 0x7f0e840ac290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:31.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.674+0000 7f0e827fc700 1 -- 172.21.15.23:0/271024480 >> v1:172.21.15.155:6789/0 conn(0x7f0e840a9d40 legacy=0x7f0e840abb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.674+0000 7f0e827fc700 1 -- 172.21.15.23:0/271024480 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e84068100 con 0x7f0e84060a80 2020-11-20T11:32:31.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.674+0000 7f0e827fc700 1 -- 172.21.15.23:0/271024480 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f0e74001520 con 0x7f0e84060a80 2020-11-20T11:32:31.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.674+0000 7f0e827fc700 1 -- 172.21.15.23:0/271024480 >> v1:172.21.15.23:6789/0 conn(0x7f0e84060a80 legacy=0x7f0e84061950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.674+0000 7f0e827fc700 1 --2- 172.21.15.23:0/271024480 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0e70003770 0x7f0e70003b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.675+0000 7f0e827fc700 1 --2- 172.21.15.23:0/271024480 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e70004520 0x7f0e700049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.675+0000 7f0e827fc700 1 --2- 172.21.15.23:0/271024480 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e70004e90 0x7f0e7000b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.675+0000 7f0e827fc700 1 -- 172.21.15.23:0/271024480 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e700042a0 con 0x7f0e70004520 2020-11-20T11:32:31.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.675+0000 7f0e827fc700 1 -- 172.21.15.23:0/271024480 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7000bc60 con 0x7f0e70004e90 2020-11-20T11:32:31.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.675+0000 7f0e827fc700 1 -- 172.21.15.23:0/271024480 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7000bda0 con 0x7f0e70003770 2020-11-20T11:32:31.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.675+0000 7f0e83fff700 1 --2- 172.21.15.23:0/271024480 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e70004e90 0x7f0e7000b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:31.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.675+0000 7f0e82ffd700 1 --2- 172.21.15.23:0/271024480 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e70004520 0x7f0e700049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:31.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.675+0000 7f0e837fe700 1 --2- 172.21.15.23:0/271024480 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0e70003770 0x7f0e70003b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:31.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.675+0000 7f0e82ffd700 1 -- 172.21.15.23:0/271024480 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0e70003770 msgr2=0x7f0e70003b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:31.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.675+0000 7f0e82ffd700 1 --2- 172.21.15.23:0/271024480 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0e70003770 0x7f0e70003b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:31.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.675+0000 7f0e82ffd700 1 -- 172.21.15.23:0/271024480 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e70004e90 msgr2=0x7f0e7000b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:31.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.675+0000 7f0e82ffd700 1 --2- 172.21.15.23:0/271024480 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e70004e90 0x7f0e7000b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:31.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.675+0000 7f0e82ffd700 1 -- 172.21.15.23:0/271024480 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e70003280 con 0x7f0e70004520 2020-11-20T11:32:31.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.675+0000 7f0e83fff700 1 -- 172.21.15.23:0/271024480 reap_dead start 2020-11-20T11:32:31.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.675+0000 7f0e82ffd700 1 --2- 172.21.15.23:0/271024480 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e70004520 0x7f0e700049a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f0e78001960 tx=0x7f0e78007bd0).ready entity=mon.0 client_cookie=4330a87252445278 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:31.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.675+0000 7f0e827fc700 1 -- 172.21.15.23:0/271024480 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e78007020 con 0x7f0e70004520 2020-11-20T11:32:31.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.675+0000 7f0e827fc700 1 -- 172.21.15.23:0/271024480 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e78007180 con 0x7f0e70004520 2020-11-20T11:32:31.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.675+0000 7f0e827fc700 1 -- 172.21.15.23:0/271024480 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7800c860 con 0x7f0e70004520 2020-11-20T11:32:31.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.676+0000 7f0e89afa700 1 -- 172.21.15.23:0/271024480 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e70004520 msgr2=0x7f0e700049a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.676+0000 7f0e89afa700 1 --2- 172.21.15.23:0/271024480 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e70004520 0x7f0e700049a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f0e78001960 tx=0x7f0e78007bd0).stop 2020-11-20T11:32:31.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.676+0000 7f0e89afa700 1 -- 172.21.15.23:0/271024480 shutdown_connections 2020-11-20T11:32:31.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.676+0000 7f0e89afa700 1 --2- 172.21.15.23:0/271024480 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e70004520 0x7f0e700049a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:31.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.676+0000 7f0e89afa700 1 -- 172.21.15.23:0/271024480 >> 172.21.15.23:0/271024480 conn(0x7f0e84066730 msgr2=0x7f0e840601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:31.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.676+0000 7f0e89afa700 1 -- 172.21.15.23:0/271024480 shutdown_connections 2020-11-20T11:32:31.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.676+0000 7f0e89afa700 1 -- 172.21.15.23:0/271024480 wait complete. 2020-11-20T11:32:31.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.677+0000 7f0e89afa700 1 Processor -- start 2020-11-20T11:32:31.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.677+0000 7f0e89afa700 1 -- start start 2020-11-20T11:32:31.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.677+0000 7f0e89afa700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e70004520 0x7f0e8413c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.677+0000 7f0e89afa700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0e84134720 0x7f0e840a6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.677+0000 7f0e89afa700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e840a7070 0x7f0e8414d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.677+0000 7f0e89afa700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e840a8030 con 0x7f0e70004520 2020-11-20T11:32:31.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.677+0000 7f0e89afa700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e840a74f0 con 0x7f0e840a7070 2020-11-20T11:32:31.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.677+0000 7f0e89afa700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8413c160 con 0x7f0e84134720 2020-11-20T11:32:31.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.678+0000 7f0e837fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e70004520 0x7f0e8413c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:31.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.678+0000 7f0e837fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e70004520 0x7f0e8413c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40820/0 (socket says 172.21.15.23:40820) 2020-11-20T11:32:31.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.678+0000 7f0e837fe700 1 -- 172.21.15.23:0/3720930837 learned_addr learned my addr 172.21.15.23:0/3720930837 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:31.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.678+0000 7f0e83fff700 1 --2- 172.21.15.23:0/3720930837 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e840a7070 0x7f0e8414d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:31.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.678+0000 7f0e82ffd700 1 --2- 172.21.15.23:0/3720930837 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0e84134720 0x7f0e840a6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:31.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.678+0000 7f0e837fe700 1 -- 172.21.15.23:0/3720930837 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0e84134720 msgr2=0x7f0e840a6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.678+0000 7f0e837fe700 1 --2- 172.21.15.23:0/3720930837 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0e84134720 0x7f0e840a6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:31.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.678+0000 7f0e837fe700 1 -- 172.21.15.23:0/3720930837 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e840a7070 msgr2=0x7f0e8414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.678+0000 7f0e837fe700 1 --2- 172.21.15.23:0/3720930837 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e840a7070 0x7f0e8414d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:31.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.678+0000 7f0e837fe700 1 -- 172.21.15.23:0/3720930837 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e840a8290 con 0x7f0e70004520 2020-11-20T11:32:31.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.678+0000 7f0e82ffd700 1 --2- 172.21.15.23:0/3720930837 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0e84134720 0x7f0e840a6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:32:31.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.678+0000 7f0e837fe700 1 --2- 172.21.15.23:0/3720930837 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e70004520 0x7f0e8413c820 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f0e74004b70 tx=0x7f0e84068290).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:31.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.678+0000 7f0e637fe700 1 -- 172.21.15.23:0/3720930837 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7400f9a0 con 0x7f0e70004520 2020-11-20T11:32:31.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.678+0000 7f0e637fe700 1 -- 172.21.15.23:0/3720930837 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e7400fb00 con 0x7f0e70004520 2020-11-20T11:32:31.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.678+0000 7f0e637fe700 1 -- 172.21.15.23:0/3720930837 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7400fea0 con 0x7f0e70004520 2020-11-20T11:32:31.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.678+0000 7f0e89afa700 1 -- 172.21.15.23:0/3720930837 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e840a8530 con 0x7f0e70004520 2020-11-20T11:32:31.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.678+0000 7f0e89afa700 1 -- 172.21.15.23:0/3720930837 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e8414ded0 con 0x7f0e70004520 2020-11-20T11:32:31.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.679+0000 7f0e89afa700 1 -- 172.21.15.23:0/3720930837 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0e840a6310 con 0x7f0e70004520 2020-11-20T11:32:31.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.679+0000 7f0e637fe700 1 -- 172.21.15.23:0/3720930837 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0e74012560 con 0x7f0e70004520 2020-11-20T11:32:31.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.680+0000 7f0e637fe700 1 --2- 172.21.15.23:0/3720930837 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0e5c056d90 0x7f0e5c059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.680+0000 7f0e637fe700 1 -- 172.21.15.23:0/3720930837 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f0e7406f860 con 0x7f0e70004520 2020-11-20T11:32:31.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.680+0000 7f0e637fe700 1 --2- 172.21.15.23:0/3720930837 >> [v2:172.21.15.23:6800/23083,v1:172.21.15.23:6801/23083] conn(0x7f0e5c05e970 0x7f0e5c060d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.680+0000 7f0e637fe700 1 -- 172.21.15.23:0/3720930837 --> [v2:172.21.15.23:6800/23083,v1:172.21.15.23:6801/23083] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0e5c0613f0 con 0x7f0e5c05e970 2020-11-20T11:32:31.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.680+0000 7f0e637fe700 1 -- 172.21.15.23:0/3720930837 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e74074220 con 0x7f0e70004520 2020-11-20T11:32:31.699 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.680+0000 7f0e82ffd700 1 --2- 172.21.15.23:0/3720930837 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0e5c056d90 0x7f0e5c059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:31.699 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.680+0000 7f0e83fff700 1 --2- 172.21.15.23:0/3720930837 >> [v2:172.21.15.23:6800/23083,v1:172.21.15.23:6801/23083] conn(0x7f0e5c05e970 0x7f0e5c060d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:31.699 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.680+0000 7f0e82ffd700 1 --2- 172.21.15.23:0/3720930837 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0e5c056d90 0x7f0e5c059250 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f0e84134ba0 tx=0x7f0e78015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:31.699 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.680+0000 7f0e83fff700 1 --2- 172.21.15.23:0/3720930837 >> [v2:172.21.15.23:6800/23083,v1:172.21.15.23:6801/23083] conn(0x7f0e5c05e970 0x7f0e5c060d90 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:31.700 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.681+0000 7f0e637fe700 1 -- 172.21.15.23:0/3720930837 <== osd.0 v2:172.21.15.23:6800/23083 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7f0e5c0613f0 con 0x7f0e5c05e970 2020-11-20T11:32:31.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.695+0000 7f0e89afa700 1 -- 172.21.15.23:0/3720930837 --> [v2:172.21.15.23:6800/23083,v1:172.21.15.23:6801/23083] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0e84044710 con 0x7f0e5c05e970 2020-11-20T11:32:31.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.695+0000 7f0e637fe700 1 -- 172.21.15.23:0/3720930837 <== osd.0 v2:172.21.15.23:6800/23083 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0e84044710 con 0x7f0e5c05e970 2020-11-20T11:32:31.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.696+0000 7f0e89afa700 1 -- 172.21.15.23:0/3720930837 >> [v2:172.21.15.23:6800/23083,v1:172.21.15.23:6801/23083] conn(0x7f0e5c05e970 msgr2=0x7f0e5c060d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.696+0000 7f0e89afa700 1 --2- 172.21.15.23:0/3720930837 >> [v2:172.21.15.23:6800/23083,v1:172.21.15.23:6801/23083] conn(0x7f0e5c05e970 0x7f0e5c060d90 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:31.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.696+0000 7f0e89afa700 1 -- 172.21.15.23:0/3720930837 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0e5c056d90 msgr2=0x7f0e5c059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.696+0000 7f0e89afa700 1 --2- 172.21.15.23:0/3720930837 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0e5c056d90 0x7f0e5c059250 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f0e84134ba0 tx=0x7f0e78015000).stop 2020-11-20T11:32:31.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.696+0000 7f0e89afa700 1 -- 172.21.15.23:0/3720930837 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e70004520 msgr2=0x7f0e8413c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.696+0000 7f0e89afa700 1 --2- 172.21.15.23:0/3720930837 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e70004520 0x7f0e8413c820 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f0e74004b70 tx=0x7f0e84068290).stop 2020-11-20T11:32:31.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.696+0000 7f0e83fff700 1 -- 172.21.15.23:0/3720930837 reap_dead start 2020-11-20T11:32:31.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.696+0000 7f0e89afa700 1 -- 172.21.15.23:0/3720930837 shutdown_connections 2020-11-20T11:32:31.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.696+0000 7f0e89afa700 1 -- 172.21.15.23:0/3720930837 >> 172.21.15.23:0/3720930837 conn(0x7f0e84066730 msgr2=0x7f0e840aba20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:31.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.696+0000 7f0e89afa700 1 -- 172.21.15.23:0/3720930837 shutdown_connections 2020-11-20T11:32:31.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.696+0000 7f0e89afa700 1 -- 172.21.15.23:0/3720930837 wait complete. 2020-11-20T11:32:31.709 INFO:teuthology.orchestra.run.smithi023.stdout:55834574853 2020-11-20T11:32:31.710 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2020-11-20T11:32:31.824 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.822+0000 7f583359e700 1 Processor -- start 2020-11-20T11:32:31.825 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.822+0000 7f583359e700 1 -- start start 2020-11-20T11:32:31.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.822+0000 7f583359e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f58340a27e0 0x7f58340a7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.822+0000 7f583359e700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58340a1ae0 con 0x7f5834061380 2020-11-20T11:32:31.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.823+0000 7f583359e700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58340a9270 con 0x7f58340a84b0 2020-11-20T11:32:31.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.823+0000 7f583359e700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58340a9050 con 0x7f58340a27e0 2020-11-20T11:32:31.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.823+0000 7f583259c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f5834061380 0x7f58340a20d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55080/0 (socket says 172.21.15.23:55080) 2020-11-20T11:32:31.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.823+0000 7f583259c700 1 -- 172.21.15.23:0/678028299 learned_addr learned my addr 172.21.15.23:0/678028299 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:31.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.823+0000 7f5831d9b700 1 --2- 172.21.15.23:0/678028299 >> v2:172.21.15.155:3300/0 conn(0x7f58340a27e0 0x7f58340a7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:31.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.823+0000 7f583159a700 1 -- 172.21.15.23:0/678028299 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 324006326 0 0) 0x7f58340a1ae0 con 0x7f5834061380 2020-11-20T11:32:31.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.823+0000 7f583159a700 1 -- 172.21.15.23:0/678028299 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5814003280 con 0x7f5834061380 2020-11-20T11:32:31.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.823+0000 7f5831d9b700 1 -- 172.21.15.23:0/678028299 >> v1:172.21.15.155:6789/0 conn(0x7f58340a84b0 legacy=0x7f58340a8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.823+0000 7f5831d9b700 1 -- 172.21.15.23:0/678028299 >> v1:172.21.15.23:6789/0 conn(0x7f5834061380 legacy=0x7f58340a20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.823+0000 7f5831d9b700 1 -- 172.21.15.23:0/678028299 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58340a9510 con 0x7f58340a27e0 2020-11-20T11:32:31.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.823+0000 7f5831d9b700 1 --2- 172.21.15.23:0/678028299 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f58340a27e0 0x7f58340a7fc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f582400ba00 tx=0x7f5824006e30).ready entity=mon.1 client_cookie=be97ad77b59f5d61 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:31.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.823+0000 7f583159a700 1 -- 172.21.15.23:0/678028299 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58240097a0 con 0x7f58340a27e0 2020-11-20T11:32:31.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.823+0000 7f583159a700 1 -- 172.21.15.23:0/678028299 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5824000e90 con 0x7f58340a27e0 2020-11-20T11:32:31.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.823+0000 7f583159a700 1 -- 172.21.15.23:0/678028299 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5824009b10 con 0x7f58340a27e0 2020-11-20T11:32:31.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.824+0000 7f583359e700 1 -- 172.21.15.23:0/678028299 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f58340a27e0 msgr2=0x7f58340a7fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.824+0000 7f583359e700 1 --2- 172.21.15.23:0/678028299 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f58340a27e0 0x7f58340a7fc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f582400ba00 tx=0x7f5824006e30).stop 2020-11-20T11:32:31.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.824+0000 7f583359e700 1 -- 172.21.15.23:0/678028299 shutdown_connections 2020-11-20T11:32:31.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.824+0000 7f583359e700 1 --2- 172.21.15.23:0/678028299 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f58340a27e0 0x7f58340a7fc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:31.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.824+0000 7f583359e700 1 -- 172.21.15.23:0/678028299 >> 172.21.15.23:0/678028299 conn(0x7f58340ae0a0 msgr2=0x7f58340aa0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:31.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.824+0000 7f583359e700 1 -- 172.21.15.23:0/678028299 shutdown_connections 2020-11-20T11:32:31.832 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.824+0000 7f583359e700 1 -- 172.21.15.23:0/678028299 wait complete. 2020-11-20T11:32:31.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.825+0000 7f583359e700 1 Processor -- start 2020-11-20T11:32:31.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.825+0000 7f583359e700 1 -- start start 2020-11-20T11:32:31.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.826+0000 7f583359e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5834061380 0x7f58340a3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.826+0000 7f583359e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f58340a27e0 0x7f583409d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.826+0000 7f583359e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f58340a84b0 0x7f583413e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.826+0000 7f583359e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58340a4f90 con 0x7f5834061380 2020-11-20T11:32:31.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.826+0000 7f583359e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58340a51c0 con 0x7f58340a84b0 2020-11-20T11:32:31.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.826+0000 7f583359e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58340a5460 con 0x7f58340a27e0 2020-11-20T11:32:31.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.826+0000 7f583259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5834061380 0x7f58340a3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:31.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.826+0000 7f583259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5834061380 0x7f58340a3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40836/0 (socket says 172.21.15.23:40836) 2020-11-20T11:32:31.837 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.826+0000 7f583259c700 1 -- 172.21.15.23:0/3843165501 learned_addr learned my addr 172.21.15.23:0/3843165501 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:31.837 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.826+0000 7f5832d9d700 1 --2- 172.21.15.23:0/3843165501 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f58340a84b0 0x7f583413e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:31.837 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.826+0000 7f5831d9b700 1 --2- 172.21.15.23:0/3843165501 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f58340a27e0 0x7f583409d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:31.837 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.826+0000 7f583259c700 1 -- 172.21.15.23:0/3843165501 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f58340a27e0 msgr2=0x7f583409d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.837 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.826+0000 7f583259c700 1 --2- 172.21.15.23:0/3843165501 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f58340a27e0 0x7f583409d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:31.838 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.826+0000 7f583259c700 1 -- 172.21.15.23:0/3843165501 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f58340a84b0 msgr2=0x7f583413e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.838 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.826+0000 7f583259c700 1 --2- 172.21.15.23:0/3843165501 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f58340a84b0 0x7f583413e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:31.838 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.826+0000 7f583259c700 1 -- 172.21.15.23:0/3843165501 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58340a5700 con 0x7f5834061380 2020-11-20T11:32:31.838 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.826+0000 7f5831d9b700 1 --2- 172.21.15.23:0/3843165501 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f58340a27e0 0x7f583409d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:31.839 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.826+0000 7f583259c700 1 --2- 172.21.15.23:0/3843165501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5834061380 0x7f58340a3090 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f581c00c770 tx=0x7f581c007b10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:31.839 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.826+0000 7f582a7fc700 1 -- 172.21.15.23:0/3843165501 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c007320 con 0x7f5834061380 2020-11-20T11:32:31.839 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.826+0000 7f582a7fc700 1 -- 172.21.15.23:0/3843165501 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f581c007480 con 0x7f5834061380 2020-11-20T11:32:31.839 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.826+0000 7f582a7fc700 1 -- 172.21.15.23:0/3843165501 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c005840 con 0x7f5834061380 2020-11-20T11:32:31.839 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.826+0000 7f583359e700 1 -- 172.21.15.23:0/3843165501 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58340a4a60 con 0x7f5834061380 2020-11-20T11:32:31.840 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.826+0000 7f583359e700 1 -- 172.21.15.23:0/3843165501 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f583413ef70 con 0x7f5834061380 2020-11-20T11:32:31.840 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.827+0000 7f583359e700 1 -- 172.21.15.23:0/3843165501 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f583405cf90 con 0x7f5834061380 2020-11-20T11:32:31.840 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.828+0000 7f582a7fc700 1 -- 172.21.15.23:0/3843165501 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f581c010030 con 0x7f5834061380 2020-11-20T11:32:31.840 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.828+0000 7f582a7fc700 1 --2- 172.21.15.23:0/3843165501 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5810056d60 0x7f5810059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.840 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.828+0000 7f582a7fc700 1 -- 172.21.15.23:0/3843165501 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f581c06fdc0 con 0x7f5834061380 2020-11-20T11:32:31.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.828+0000 7f582a7fc700 1 --2- 172.21.15.23:0/3843165501 >> [v2:172.21.15.23:6816/23099,v1:172.21.15.23:6817/23099] conn(0x7f581005e8c0 0x7f5810060ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.828+0000 7f582a7fc700 1 -- 172.21.15.23:0/3843165501 --> [v2:172.21.15.23:6816/23099,v1:172.21.15.23:6817/23099] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5810061340 con 0x7f581005e8c0 2020-11-20T11:32:31.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.828+0000 7f582a7fc700 1 -- 172.21.15.23:0/3843165501 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f581c0723b0 con 0x7f5834061380 2020-11-20T11:32:31.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.828+0000 7f5831d9b700 1 --2- 172.21.15.23:0/3843165501 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5810056d60 0x7f5810059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:31.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.828+0000 7f5832d9d700 1 --2- 172.21.15.23:0/3843165501 >> [v2:172.21.15.23:6816/23099,v1:172.21.15.23:6817/23099] conn(0x7f581005e8c0 0x7f5810060ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:31.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.828+0000 7f5831d9b700 1 --2- 172.21.15.23:0/3843165501 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5810056d60 0x7f5810059220 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f583413f5d0 tx=0x7f5824015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:31.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.829+0000 7f5832d9d700 1 --2- 172.21.15.23:0/3843165501 >> [v2:172.21.15.23:6816/23099,v1:172.21.15.23:6817/23099] conn(0x7f581005e8c0 0x7f5810060ce0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:31.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.829+0000 7f582a7fc700 1 -- 172.21.15.23:0/3843165501 <== osd.1 v2:172.21.15.23:6816/23099 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7f5810061340 con 0x7f581005e8c0 2020-11-20T11:32:31.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.843+0000 7f583359e700 1 -- 172.21.15.23:0/3843165501 --> [v2:172.21.15.23:6816/23099,v1:172.21.15.23:6817/23099] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f58340a5890 con 0x7f581005e8c0 2020-11-20T11:32:31.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.844+0000 7f582a7fc700 1 -- 172.21.15.23:0/3843165501 <== osd.1 v2:172.21.15.23:6816/23099 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f582c002aa0 con 0x7f581005e8c0 2020-11-20T11:32:31.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.844+0000 7f583359e700 1 -- 172.21.15.23:0/3843165501 >> [v2:172.21.15.23:6816/23099,v1:172.21.15.23:6817/23099] conn(0x7f581005e8c0 msgr2=0x7f5810060ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.844+0000 7f583359e700 1 --2- 172.21.15.23:0/3843165501 >> [v2:172.21.15.23:6816/23099,v1:172.21.15.23:6817/23099] conn(0x7f581005e8c0 0x7f5810060ce0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:31.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.844+0000 7f583359e700 1 -- 172.21.15.23:0/3843165501 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5810056d60 msgr2=0x7f5810059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.844+0000 7f583359e700 1 --2- 172.21.15.23:0/3843165501 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5810056d60 0x7f5810059220 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f583413f5d0 tx=0x7f5824015000).stop 2020-11-20T11:32:31.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.844+0000 7f583359e700 1 -- 172.21.15.23:0/3843165501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5834061380 msgr2=0x7f58340a3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.844+0000 7f583359e700 1 --2- 172.21.15.23:0/3843165501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5834061380 0x7f58340a3090 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f581c00c770 tx=0x7f581c007b10).stop 2020-11-20T11:32:31.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.844+0000 7f5832d9d700 1 -- 172.21.15.23:0/3843165501 reap_dead start 2020-11-20T11:32:31.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.844+0000 7f583359e700 1 -- 172.21.15.23:0/3843165501 shutdown_connections 2020-11-20T11:32:31.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.844+0000 7f583359e700 1 -- 172.21.15.23:0/3843165501 >> 172.21.15.23:0/3843165501 conn(0x7f58340ae0a0 msgr2=0x7f58340ab8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:31.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.844+0000 7f583359e700 1 -- 172.21.15.23:0/3843165501 shutdown_connections 2020-11-20T11:32:31.850 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.844+0000 7f583359e700 1 -- 172.21.15.23:0/3843165501 wait complete. 2020-11-20T11:32:31.858 INFO:teuthology.orchestra.run.smithi023.stdout:60129542149 2020-11-20T11:32:31.859 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2020-11-20T11:32:31.974 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.971+0000 7fd906aff700 1 Processor -- start 2020-11-20T11:32:31.974 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.971+0000 7fd906aff700 1 -- start start 2020-11-20T11:32:31.976 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.972+0000 7fd906aff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9000aae90 0x7fd9001381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.976 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.972+0000 7fd906aff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90013e600 con 0x7fd90013aeb0 2020-11-20T11:32:31.977 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.972+0000 7fd906aff700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90013e8a0 con 0x7fd9000aaab0 2020-11-20T11:32:31.977 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.972+0000 7fd906aff700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9000ab6d0 con 0x7fd9000aae90 2020-11-20T11:32:31.977 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.972+0000 7fd90509c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fd90013aeb0 0x7fd9000a8c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55100/0 (socket says 172.21.15.23:55100) 2020-11-20T11:32:31.977 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.972+0000 7fd90509c700 1 -- 172.21.15.23:0/4274253197 learned_addr learned my addr 172.21.15.23:0/4274253197 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:31.977 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.972+0000 7fd8fffff700 1 --2- 172.21.15.23:0/4274253197 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9000aae90 0x7fd9001381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:31.978 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.972+0000 7fd8ff7fe700 1 -- 172.21.15.23:0/4274253197 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 636630772 0 0) 0x7fd90013e600 con 0x7fd90013aeb0 2020-11-20T11:32:31.978 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.972+0000 7fd8ff7fe700 1 -- 172.21.15.23:0/4274253197 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8e4003280 con 0x7fd90013aeb0 2020-11-20T11:32:31.978 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.972+0000 7fd8ff7fe700 1 -- 172.21.15.23:0/4274253197 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 128712152 0 0) 0x7fd90013e8a0 con 0x7fd9000aaab0 2020-11-20T11:32:31.978 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.972+0000 7fd8ff7fe700 1 -- 172.21.15.23:0/4274253197 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd90013e600 con 0x7fd9000aaab0 2020-11-20T11:32:31.978 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.972+0000 7fd8fffff700 1 -- 172.21.15.23:0/4274253197 >> v1:172.21.15.155:6789/0 conn(0x7fd9000aaab0 legacy=0x7fd90013ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.979 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.972+0000 7fd8fffff700 1 -- 172.21.15.23:0/4274253197 >> v1:172.21.15.23:6789/0 conn(0x7fd90013aeb0 legacy=0x7fd9000a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.979 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.972+0000 7fd8fffff700 1 -- 172.21.15.23:0/4274253197 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9000a9370 con 0x7fd9000aae90 2020-11-20T11:32:31.979 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.972+0000 7fd8fffff700 1 --2- 172.21.15.23:0/4274253197 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9000aae90 0x7fd9001381a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fd8ec00ba00 tx=0x7fd8ec006e30).ready entity=mon.2 client_cookie=fc536fc82255be6b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:31.979 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.973+0000 7fd8ff7fe700 1 -- 172.21.15.23:0/4274253197 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec009630 con 0x7fd9000aae90 2020-11-20T11:32:31.979 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.973+0000 7fd8ff7fe700 1 -- 172.21.15.23:0/4274253197 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ec000e90 con 0x7fd9000aae90 2020-11-20T11:32:31.980 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.973+0000 7fd8ff7fe700 1 -- 172.21.15.23:0/4274253197 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec0099a0 con 0x7fd9000aae90 2020-11-20T11:32:31.980 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.973+0000 7fd906aff700 1 -- 172.21.15.23:0/4274253197 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9000aae90 msgr2=0x7fd9001381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.980 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.973+0000 7fd906aff700 1 --2- 172.21.15.23:0/4274253197 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9000aae90 0x7fd9001381a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fd8ec00ba00 tx=0x7fd8ec006e30).stop 2020-11-20T11:32:31.980 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.973+0000 7fd906aff700 1 -- 172.21.15.23:0/4274253197 shutdown_connections 2020-11-20T11:32:31.980 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.973+0000 7fd906aff700 1 --2- 172.21.15.23:0/4274253197 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9000aae90 0x7fd9001381a0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:31.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.973+0000 7fd906aff700 1 -- 172.21.15.23:0/4274253197 >> 172.21.15.23:0/4274253197 conn(0x7fd9000678e0 msgr2=0x7fd900067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:31.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.973+0000 7fd906aff700 1 -- 172.21.15.23:0/4274253197 shutdown_connections 2020-11-20T11:32:31.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.973+0000 7fd906aff700 1 -- 172.21.15.23:0/4274253197 wait complete. 2020-11-20T11:32:31.984 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.974+0000 7fd906aff700 1 Processor -- start 2020-11-20T11:32:31.984 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.974+0000 7fd906aff700 1 -- start start 2020-11-20T11:32:31.985 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.975+0000 7fd906aff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9000aaab0 0x7fd900135cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.985 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.975+0000 7fd906aff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9000aae90 0x7fd900142510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.985 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.975+0000 7fd906aff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9000a7210 0x7fd9001335b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.985 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.975+0000 7fd906aff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9000a7720 con 0x7fd9000a7210 2020-11-20T11:32:31.986 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.975+0000 7fd906aff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd900142070 con 0x7fd9000aae90 2020-11-20T11:32:31.986 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.975+0000 7fd906aff700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd900142a00 con 0x7fd9000aaab0 2020-11-20T11:32:31.986 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.975+0000 7fd90509c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9000a7210 0x7fd9001335b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:31.986 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.975+0000 7fd90509c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9000a7210 0x7fd9001335b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40856/0 (socket says 172.21.15.23:40856) 2020-11-20T11:32:31.986 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.975+0000 7fd90509c700 1 -- 172.21.15.23:0/3807700202 learned_addr learned my addr 172.21.15.23:0/3807700202 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:31.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.975+0000 7fd8fffff700 1 --2- 172.21.15.23:0/3807700202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9000aae90 0x7fd900142510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:31.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.975+0000 7fd90489b700 1 --2- 172.21.15.23:0/3807700202 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9000aaab0 0x7fd900135cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:31.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.975+0000 7fd90509c700 1 -- 172.21.15.23:0/3807700202 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9000aaab0 msgr2=0x7fd900135cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.975+0000 7fd90509c700 1 --2- 172.21.15.23:0/3807700202 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9000aaab0 0x7fd900135cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:31.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.975+0000 7fd90509c700 1 -- 172.21.15.23:0/3807700202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9000aae90 msgr2=0x7fd900142510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.975+0000 7fd90509c700 1 --2- 172.21.15.23:0/3807700202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9000aae90 0x7fd900142510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:31.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.975+0000 7fd90509c700 1 -- 172.21.15.23:0/3807700202 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9000a79c0 con 0x7fd9000a7210 2020-11-20T11:32:31.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.975+0000 7fd90509c700 1 --2- 172.21.15.23:0/3807700202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9000a7210 0x7fd9001335b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fd8f800ca60 tx=0x7fd8f8007e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:31.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.976+0000 7fd8fcff9700 1 -- 172.21.15.23:0/3807700202 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f80059b0 con 0x7fd9000a7210 2020-11-20T11:32:31.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.976+0000 7fd8fcff9700 1 -- 172.21.15.23:0/3807700202 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f8005b10 con 0x7fd9000a7210 2020-11-20T11:32:31.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.976+0000 7fd906aff700 1 -- 172.21.15.23:0/3807700202 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9000a52b0 con 0x7fd9000a7210 2020-11-20T11:32:31.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.976+0000 7fd8fcff9700 1 -- 172.21.15.23:0/3807700202 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f8005e90 con 0x7fd9000a7210 2020-11-20T11:32:31.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.976+0000 7fd906aff700 1 -- 172.21.15.23:0/3807700202 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd90013f860 con 0x7fd9000a7210 2020-11-20T11:32:31.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.976+0000 7fd906aff700 1 -- 172.21.15.23:0/3807700202 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd90005c7d0 con 0x7fd9000a7210 2020-11-20T11:32:31.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.977+0000 7fd8fcff9700 1 -- 172.21.15.23:0/3807700202 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd8f8010030 con 0x7fd9000a7210 2020-11-20T11:32:31.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.977+0000 7fd8fcff9700 1 --2- 172.21.15.23:0/3807700202 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd8d8056d90 0x7fd8d8059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.977+0000 7fd8fcff9700 1 -- 172.21.15.23:0/3807700202 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fd8f806ff80 con 0x7fd9000a7210 2020-11-20T11:32:31.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.977+0000 7fd8fcff9700 1 --2- 172.21.15.23:0/3807700202 >> [v2:172.21.15.23:6808/23084,v1:172.21.15.23:6809/23084] conn(0x7fd8d805e800 0x7fd8d8060c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:31.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.977+0000 7fd8fcff9700 1 -- 172.21.15.23:0/3807700202 --> [v2:172.21.15.23:6808/23084,v1:172.21.15.23:6809/23084] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd8d8061280 con 0x7fd8d805e800 2020-11-20T11:32:31.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.977+0000 7fd8fcff9700 1 -- 172.21.15.23:0/3807700202 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8f80725f0 con 0x7fd9000a7210 2020-11-20T11:32:31.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.977+0000 7fd90489b700 1 --2- 172.21.15.23:0/3807700202 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd8d8056d90 0x7fd8d8059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:31.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.978+0000 7fd8fffff700 1 --2- 172.21.15.23:0/3807700202 >> [v2:172.21.15.23:6808/23084,v1:172.21.15.23:6809/23084] conn(0x7fd8d805e800 0x7fd8d8060c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:31.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.978+0000 7fd90489b700 1 --2- 172.21.15.23:0/3807700202 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd8d8056d90 0x7fd8d8059250 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fd8f4000c00 tx=0x7fd8f4007a40).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:31.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.978+0000 7fd8fffff700 1 --2- 172.21.15.23:0/3807700202 >> [v2:172.21.15.23:6808/23084,v1:172.21.15.23:6809/23084] conn(0x7fd8d805e800 0x7fd8d8060c20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:31.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.979+0000 7fd8fcff9700 1 -- 172.21.15.23:0/3807700202 <== osd.2 v2:172.21.15.23:6808/23084 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7fd8d8061280 con 0x7fd8d805e800 2020-11-20T11:32:31.995 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.993+0000 7fd906aff700 1 -- 172.21.15.23:0/3807700202 --> [v2:172.21.15.23:6808/23084,v1:172.21.15.23:6809/23084] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd90013b0f0 con 0x7fd8d805e800 2020-11-20T11:32:31.996 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.993+0000 7fd8fcff9700 1 -- 172.21.15.23:0/3807700202 <== osd.2 v2:172.21.15.23:6808/23084 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd90013b0f0 con 0x7fd8d805e800 2020-11-20T11:32:31.997 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.993+0000 7fd906aff700 1 -- 172.21.15.23:0/3807700202 >> [v2:172.21.15.23:6808/23084,v1:172.21.15.23:6809/23084] conn(0x7fd8d805e800 msgr2=0x7fd8d8060c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.997 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.993+0000 7fd906aff700 1 --2- 172.21.15.23:0/3807700202 >> [v2:172.21.15.23:6808/23084,v1:172.21.15.23:6809/23084] conn(0x7fd8d805e800 0x7fd8d8060c20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:31.997 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.993+0000 7fd906aff700 1 -- 172.21.15.23:0/3807700202 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd8d8056d90 msgr2=0x7fd8d8059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.997 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.993+0000 7fd906aff700 1 --2- 172.21.15.23:0/3807700202 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd8d8056d90 0x7fd8d8059250 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fd8f4000c00 tx=0x7fd8f4007a40).stop 2020-11-20T11:32:31.998 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.993+0000 7fd906aff700 1 -- 172.21.15.23:0/3807700202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9000a7210 msgr2=0x7fd9001335b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:31.998 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.993+0000 7fd906aff700 1 --2- 172.21.15.23:0/3807700202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9000a7210 0x7fd9001335b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fd8f800ca60 tx=0x7fd8f8007e00).stop 2020-11-20T11:32:31.998 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.993+0000 7fd90509c700 1 -- 172.21.15.23:0/3807700202 reap_dead start 2020-11-20T11:32:31.998 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.993+0000 7fd906aff700 1 -- 172.21.15.23:0/3807700202 shutdown_connections 2020-11-20T11:32:31.998 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.993+0000 7fd906aff700 1 -- 172.21.15.23:0/3807700202 >> 172.21.15.23:0/3807700202 conn(0x7fd9000678e0 msgr2=0x7fd9000acf80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:31.999 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.993+0000 7fd906aff700 1 -- 172.21.15.23:0/3807700202 shutdown_connections 2020-11-20T11:32:31.999 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:31.993+0000 7fd906aff700 1 -- 172.21.15.23:0/3807700202 wait complete. 2020-11-20T11:32:32.007 INFO:teuthology.orchestra.run.smithi023.stdout:60129542150 2020-11-20T11:32:32.007 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2020-11-20T11:32:32.123 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.120+0000 7fcfc4a32700 1 Processor -- start 2020-11-20T11:32:32.123 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.120+0000 7fcfc4a32700 1 -- start start 2020-11-20T11:32:32.128 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.121+0000 7fcfc4a32700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fcfc013aeb0 0x7fcfc00a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.128 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.121+0000 7fcfc4a32700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc01388b0 con 0x7fcfc00aaab0 2020-11-20T11:32:32.129 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.121+0000 7fcfc4a32700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfc013e600 con 0x7fcfc00aae90 2020-11-20T11:32:32.129 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.121+0000 7fcfc4a32700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfc013e8a0 con 0x7fcfc013aeb0 2020-11-20T11:32:32.129 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.121+0000 7fcfbe59c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fcfc00aaab0 0x7fcfc013ed40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55112/0 (socket says 172.21.15.23:55112) 2020-11-20T11:32:32.129 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.121+0000 7fcfbe59c700 1 -- 172.21.15.23:0/500375316 learned_addr learned my addr 172.21.15.23:0/500375316 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:32.129 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.121+0000 7fcfbd59a700 1 -- 172.21.15.23:0/500375316 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1608768918 0 0) 0x7fcfc01388b0 con 0x7fcfc00aaab0 2020-11-20T11:32:32.130 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.121+0000 7fcfbed9d700 1 --2- 172.21.15.23:0/500375316 >> v2:172.21.15.155:3300/0 conn(0x7fcfc013aeb0 0x7fcfc00a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.130 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.121+0000 7fcfbd59a700 1 -- 172.21.15.23:0/500375316 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfa4003280 con 0x7fcfc00aaab0 2020-11-20T11:32:32.130 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.121+0000 7fcfbd59a700 1 -- 172.21.15.23:0/500375316 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1539809820 0 0) 0x7fcfa4003280 con 0x7fcfc00aaab0 2020-11-20T11:32:32.130 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.121+0000 7fcfbd59a700 1 -- 172.21.15.23:0/500375316 >> v2:172.21.15.155:3300/0 conn(0x7fcfc013aeb0 msgr2=0x7fcfc00a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:32.130 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.121+0000 7fcfbd59a700 1 --2- 172.21.15.23:0/500375316 >> v2:172.21.15.155:3300/0 conn(0x7fcfc013aeb0 0x7fcfc00a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.121+0000 7fcfbd59a700 1 -- 172.21.15.23:0/500375316 >> v1:172.21.15.155:6789/0 conn(0x7fcfc00aae90 legacy=0x7fcfc01381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.121+0000 7fcfbd59a700 1 -- 172.21.15.23:0/500375316 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc00ab6d0 con 0x7fcfc00aaab0 2020-11-20T11:32:32.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.121+0000 7fcfbd59a700 1 -- 172.21.15.23:0/500375316 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7fcfb0001520 con 0x7fcfc00aaab0 2020-11-20T11:32:32.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.121+0000 7fcfbd59a700 1 -- 172.21.15.23:0/500375316 >> v1:172.21.15.23:6789/0 conn(0x7fcfc00aaab0 legacy=0x7fcfc013ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.121+0000 7fcfbd59a700 1 --2- 172.21.15.23:0/500375316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfa4003770 0x7fcfa4003b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.121+0000 7fcfbd59a700 1 --2- 172.21.15.23:0/500375316 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfa4004520 0x7fcfa40049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.121+0000 7fcfbd59a700 1 --2- 172.21.15.23:0/500375316 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfa4004e90 0x7fcfa400b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.121+0000 7fcfbd59a700 1 -- 172.21.15.23:0/500375316 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc013e8a0 con 0x7fcfa4003770 2020-11-20T11:32:32.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.121+0000 7fcfbd59a700 1 -- 172.21.15.23:0/500375316 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa40042a0 con 0x7fcfa4004520 2020-11-20T11:32:32.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.122+0000 7fcfbd59a700 1 -- 172.21.15.23:0/500375316 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa400bc60 con 0x7fcfa4004e90 2020-11-20T11:32:32.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.122+0000 7fcfbe59c700 1 --2- 172.21.15.23:0/500375316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfa4003770 0x7fcfa4003b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.122+0000 7fcfbdd9b700 1 --2- 172.21.15.23:0/500375316 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfa4004520 0x7fcfa40049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.122+0000 7fcfbed9d700 1 --2- 172.21.15.23:0/500375316 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfa4004e90 0x7fcfa400b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.122+0000 7fcfbe59c700 1 -- 172.21.15.23:0/500375316 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfa4004e90 msgr2=0x7fcfa400b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:32.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.122+0000 7fcfbe59c700 1 --2- 172.21.15.23:0/500375316 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfa4004e90 0x7fcfa400b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.122+0000 7fcfbe59c700 1 -- 172.21.15.23:0/500375316 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfa4004520 msgr2=0x7fcfa40049a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:32.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.122+0000 7fcfbe59c700 1 --2- 172.21.15.23:0/500375316 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfa4004520 0x7fcfa40049a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.122+0000 7fcfbe59c700 1 -- 172.21.15.23:0/500375316 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa4003280 con 0x7fcfa4003770 2020-11-20T11:32:32.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.122+0000 7fcfbdd9b700 1 --2- 172.21.15.23:0/500375316 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfa4004520 0x7fcfa40049a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:32.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.122+0000 7fcfbed9d700 1 --2- 172.21.15.23:0/500375316 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfa4004e90 0x7fcfa400b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:32:32.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.122+0000 7fcfbed9d700 1 -- 172.21.15.23:0/500375316 reap_dead start 2020-11-20T11:32:32.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.122+0000 7fcfbe59c700 1 --2- 172.21.15.23:0/500375316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfa4003770 0x7fcfa4003b90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fcfb000c410 tx=0x7fcfb0007810).ready entity=mon.0 client_cookie=c63e98c1564646a0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:32.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.122+0000 7fcfbd59a700 1 -- 172.21.15.23:0/500375316 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb00070d0 con 0x7fcfa4003770 2020-11-20T11:32:32.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.122+0000 7fcfbd59a700 1 -- 172.21.15.23:0/500375316 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb0007230 con 0x7fcfa4003770 2020-11-20T11:32:32.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.122+0000 7fcfbd59a700 1 -- 172.21.15.23:0/500375316 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0005630 con 0x7fcfa4003770 2020-11-20T11:32:32.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.122+0000 7fcfc4a32700 1 -- 172.21.15.23:0/500375316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfa4003770 msgr2=0x7fcfa4003b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.122+0000 7fcfc4a32700 1 --2- 172.21.15.23:0/500375316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfa4003770 0x7fcfa4003b90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fcfb000c410 tx=0x7fcfb0007810).stop 2020-11-20T11:32:32.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.123+0000 7fcfc4a32700 1 -- 172.21.15.23:0/500375316 shutdown_connections 2020-11-20T11:32:32.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.123+0000 7fcfc4a32700 1 --2- 172.21.15.23:0/500375316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfa4003770 0x7fcfa4003b90 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.123+0000 7fcfc4a32700 1 -- 172.21.15.23:0/500375316 >> 172.21.15.23:0/500375316 conn(0x7fcfc00678e0 msgr2=0x7fcfc0067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:32.137 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.123+0000 7fcfc4a32700 1 -- 172.21.15.23:0/500375316 shutdown_connections 2020-11-20T11:32:32.137 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.123+0000 7fcfc4a32700 1 -- 172.21.15.23:0/500375316 wait complete. 2020-11-20T11:32:32.137 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.124+0000 7fcfc4a32700 1 Processor -- start 2020-11-20T11:32:32.137 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.124+0000 7fcfc4a32700 1 -- start start 2020-11-20T11:32:32.137 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.124+0000 7fcfc4a32700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfa4003770 0x7fcfc009ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.124+0000 7fcfc4a32700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfc013d510 0x7fcfc014aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.124+0000 7fcfc4a32700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfc014af00 0x7fcfc014d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.124+0000 7fcfc4a32700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc013d990 con 0x7fcfc013d510 2020-11-20T11:32:32.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.124+0000 7fcfc4a32700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc014b380 con 0x7fcfa4003770 2020-11-20T11:32:32.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.124+0000 7fcfc4a32700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc009f590 con 0x7fcfc014af00 2020-11-20T11:32:32.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.124+0000 7fcfbdd9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfc013d510 0x7fcfc014aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.124+0000 7fcfbdd9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfc013d510 0x7fcfc014aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40876/0 (socket says 172.21.15.23:40876) 2020-11-20T11:32:32.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.124+0000 7fcfbdd9b700 1 -- 172.21.15.23:0/3107170095 learned_addr learned my addr 172.21.15.23:0/3107170095 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:32.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.124+0000 7fcfbe59c700 1 --2- 172.21.15.23:0/3107170095 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfa4003770 0x7fcfc009ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.125+0000 7fcfbed9d700 1 --2- 172.21.15.23:0/3107170095 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfc014af00 0x7fcfc014d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.125+0000 7fcfbdd9b700 1 -- 172.21.15.23:0/3107170095 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfc014af00 msgr2=0x7fcfc014d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.125+0000 7fcfbdd9b700 1 --2- 172.21.15.23:0/3107170095 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfc014af00 0x7fcfc014d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.125+0000 7fcfbdd9b700 1 -- 172.21.15.23:0/3107170095 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfa4003770 msgr2=0x7fcfc009ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.125+0000 7fcfbdd9b700 1 --2- 172.21.15.23:0/3107170095 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfa4003770 0x7fcfc009ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.125+0000 7fcfbdd9b700 1 -- 172.21.15.23:0/3107170095 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc009f2f0 con 0x7fcfc013d510 2020-11-20T11:32:32.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.125+0000 7fcfbed9d700 1 --2- 172.21.15.23:0/3107170095 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfc014af00 0x7fcfc014d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:32.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.125+0000 7fcfbdd9b700 1 --2- 172.21.15.23:0/3107170095 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfc013d510 0x7fcfc014aa10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fcfb400eb10 tx=0x7fcfb4009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:32.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.125+0000 7fcfae7fc700 1 -- 172.21.15.23:0/3107170095 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4001cf0 con 0x7fcfc013d510 2020-11-20T11:32:32.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.125+0000 7fcfae7fc700 1 -- 172.21.15.23:0/3107170095 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb4007020 con 0x7fcfc013d510 2020-11-20T11:32:32.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.125+0000 7fcfae7fc700 1 -- 172.21.15.23:0/3107170095 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb40073a0 con 0x7fcfc013d510 2020-11-20T11:32:32.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.125+0000 7fcfc4a32700 1 -- 172.21.15.23:0/3107170095 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfc014a570 con 0x7fcfc013d510 2020-11-20T11:32:32.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.125+0000 7fcfc4a32700 1 -- 172.21.15.23:0/3107170095 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfc013db50 con 0x7fcfc013d510 2020-11-20T11:32:32.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.126+0000 7fcfc4a32700 1 -- 172.21.15.23:0/3107170095 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcfc005c7d0 con 0x7fcfc013d510 2020-11-20T11:32:32.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.126+0000 7fcfae7fc700 1 -- 172.21.15.23:0/3107170095 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fcfb400ca80 con 0x7fcfc013d510 2020-11-20T11:32:32.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.126+0000 7fcfae7fc700 1 --2- 172.21.15.23:0/3107170095 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcf94056d60 0x7fcf94059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.126+0000 7fcfae7fc700 1 -- 172.21.15.23:0/3107170095 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fcfb40703d0 con 0x7fcfc013d510 2020-11-20T11:32:32.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.127+0000 7fcfae7fc700 1 --2- 172.21.15.23:0/3107170095 >> [v2:172.21.15.23:6823/23114,v1:172.21.15.23:6825/23114] conn(0x7fcf9405e9c0 0x7fcf94060de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.127+0000 7fcfae7fc700 1 -- 172.21.15.23:0/3107170095 --> [v2:172.21.15.23:6823/23114,v1:172.21.15.23:6825/23114] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf94061440 con 0x7fcf9405e9c0 2020-11-20T11:32:32.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.127+0000 7fcfae7fc700 1 -- 172.21.15.23:0/3107170095 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcfb40728b0 con 0x7fcfc013d510 2020-11-20T11:32:32.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.127+0000 7fcfbed9d700 1 --2- 172.21.15.23:0/3107170095 >> [v2:172.21.15.23:6823/23114,v1:172.21.15.23:6825/23114] conn(0x7fcf9405e9c0 0x7fcf94060de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.127+0000 7fcfbe59c700 1 --2- 172.21.15.23:0/3107170095 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcf94056d60 0x7fcf94059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.127+0000 7fcfbe59c700 1 --2- 172.21.15.23:0/3107170095 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcf94056d60 0x7fcf94059220 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fcfb0000f60 tx=0x7fcfb0015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:32.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.127+0000 7fcfbed9d700 1 --2- 172.21.15.23:0/3107170095 >> [v2:172.21.15.23:6823/23114,v1:172.21.15.23:6825/23114] conn(0x7fcf9405e9c0 0x7fcf94060de0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:32.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.128+0000 7fcfae7fc700 1 -- 172.21.15.23:0/3107170095 <== osd.3 v2:172.21.15.23:6823/23114 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7fcf94061440 con 0x7fcf9405e9c0 2020-11-20T11:32:32.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.142+0000 7fcfc4a32700 1 -- 172.21.15.23:0/3107170095 --> [v2:172.21.15.23:6823/23114,v1:172.21.15.23:6825/23114] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcfc0060430 con 0x7fcf9405e9c0 2020-11-20T11:32:32.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.142+0000 7fcfae7fc700 1 -- 172.21.15.23:0/3107170095 <== osd.3 v2:172.21.15.23:6823/23114 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fcfc0060430 con 0x7fcf9405e9c0 2020-11-20T11:32:32.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.143+0000 7fcfc4a32700 1 -- 172.21.15.23:0/3107170095 >> [v2:172.21.15.23:6823/23114,v1:172.21.15.23:6825/23114] conn(0x7fcf9405e9c0 msgr2=0x7fcf94060de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.143+0000 7fcfc4a32700 1 --2- 172.21.15.23:0/3107170095 >> [v2:172.21.15.23:6823/23114,v1:172.21.15.23:6825/23114] conn(0x7fcf9405e9c0 0x7fcf94060de0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.143+0000 7fcfc4a32700 1 -- 172.21.15.23:0/3107170095 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcf94056d60 msgr2=0x7fcf94059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.143+0000 7fcfc4a32700 1 --2- 172.21.15.23:0/3107170095 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcf94056d60 0x7fcf94059220 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fcfb0000f60 tx=0x7fcfb0015000).stop 2020-11-20T11:32:32.148 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.143+0000 7fcfc4a32700 1 -- 172.21.15.23:0/3107170095 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfc013d510 msgr2=0x7fcfc014aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.148 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.143+0000 7fcfc4a32700 1 --2- 172.21.15.23:0/3107170095 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfc013d510 0x7fcfc014aa10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fcfb400eb10 tx=0x7fcfb4009040).stop 2020-11-20T11:32:32.148 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.143+0000 7fcfbed9d700 1 -- 172.21.15.23:0/3107170095 reap_dead start 2020-11-20T11:32:32.148 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.143+0000 7fcfc4a32700 1 -- 172.21.15.23:0/3107170095 shutdown_connections 2020-11-20T11:32:32.148 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.143+0000 7fcfc4a32700 1 -- 172.21.15.23:0/3107170095 >> 172.21.15.23:0/3107170095 conn(0x7fcfc00678e0 msgr2=0x7fcfc0137700 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:32.149 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.143+0000 7fcfc4a32700 1 -- 172.21.15.23:0/3107170095 shutdown_connections 2020-11-20T11:32:32.149 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.143+0000 7fcfc4a32700 1 -- 172.21.15.23:0/3107170095 wait complete. 2020-11-20T11:32:32.157 INFO:teuthology.orchestra.run.smithi023.stdout:60129542151 2020-11-20T11:32:32.157 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2020-11-20T11:32:32.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.269+0000 7f8bbf218700 1 Processor -- start 2020-11-20T11:32:32.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.269+0000 7f8bbf218700 1 -- start start 2020-11-20T11:32:32.274 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.270+0000 7f8bbf218700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8bb80aa1c0 0x7f8bb80ac290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.274 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.270+0000 7f8bbf218700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bb8067bc0 con 0x7f8bb8060a80 2020-11-20T11:32:32.275 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.270+0000 7f8bbf218700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bb8067e60 con 0x7f8bb80a9d40 2020-11-20T11:32:32.275 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.270+0000 7f8bbf218700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bb80aa630 con 0x7f8bb80aa1c0 2020-11-20T11:32:32.275 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.270+0000 7f8bbd7b5700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8bb80aa1c0 0x7f8bb80ac290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.275 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.270+0000 7f8bbcfb4700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f8bb8060a80 0x7f8bb8061950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55134/0 (socket says 172.21.15.23:55134) 2020-11-20T11:32:32.275 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.270+0000 7f8bbcfb4700 1 -- 172.21.15.23:0/3960537970 learned_addr learned my addr 172.21.15.23:0/3960537970 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:32.276 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.270+0000 7f8bbd7b5700 1 -- 172.21.15.23:0/3960537970 >> v1:172.21.15.155:6789/0 conn(0x7f8bb80a9d40 legacy=0x7f8bb80abb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.276 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.270+0000 7f8bbd7b5700 1 -- 172.21.15.23:0/3960537970 >> v1:172.21.15.23:6789/0 conn(0x7f8bb8060a80 legacy=0x7f8bb8061950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.276 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.270+0000 7f8bbd7b5700 1 -- 172.21.15.23:0/3960537970 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb8068100 con 0x7f8bb80aa1c0 2020-11-20T11:32:32.276 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.270+0000 7f8bbd7b5700 1 --2- 172.21.15.23:0/3960537970 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bb80aa1c0 0x7f8bb80ac290 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f8bb0009ce0 tx=0x7f8bb0009220).ready entity=mon.0 client_cookie=4173b477a0cc9ac7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:32.276 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.271+0000 7f8bb77fe700 1 -- 172.21.15.23:0/3960537970 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb000baf0 con 0x7f8bb80aa1c0 2020-11-20T11:32:32.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.271+0000 7f8bb77fe700 1 -- 172.21.15.23:0/3960537970 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bb000bc50 con 0x7f8bb80aa1c0 2020-11-20T11:32:32.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.271+0000 7f8bb77fe700 1 -- 172.21.15.23:0/3960537970 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb0011400 con 0x7f8bb80aa1c0 2020-11-20T11:32:32.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.271+0000 7f8bbf218700 1 -- 172.21.15.23:0/3960537970 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bb80aa1c0 msgr2=0x7f8bb80ac290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.271+0000 7f8bbf218700 1 --2- 172.21.15.23:0/3960537970 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bb80aa1c0 0x7f8bb80ac290 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f8bb0009ce0 tx=0x7f8bb0009220).stop 2020-11-20T11:32:32.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.271+0000 7f8bbf218700 1 -- 172.21.15.23:0/3960537970 shutdown_connections 2020-11-20T11:32:32.278 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.271+0000 7f8bbf218700 1 --2- 172.21.15.23:0/3960537970 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bb80aa1c0 0x7f8bb80ac290 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.278 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.271+0000 7f8bbf218700 1 -- 172.21.15.23:0/3960537970 >> 172.21.15.23:0/3960537970 conn(0x7f8bb8066730 msgr2=0x7f8bb80601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:32.278 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.271+0000 7f8bbf218700 1 -- 172.21.15.23:0/3960537970 shutdown_connections 2020-11-20T11:32:32.278 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.271+0000 7f8bbf218700 1 -- 172.21.15.23:0/3960537970 wait complete. 2020-11-20T11:32:32.281 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.272+0000 7f8bbf218700 1 Processor -- start 2020-11-20T11:32:32.281 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.272+0000 7f8bbf218700 1 -- start start 2020-11-20T11:32:32.281 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.273+0000 7f8bbf218700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bb8060a80 0x7f8bb809e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.273+0000 7f8bbf218700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bb80a9d40 0x7f8bb813d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.273+0000 7f8bbf218700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8bb80aa1c0 0x7f8bb813d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.273+0000 7f8bbf218700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb813e5c0 con 0x7f8bb80a9d40 2020-11-20T11:32:32.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.273+0000 7f8bbf218700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb813e860 con 0x7f8bb8060a80 2020-11-20T11:32:32.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.273+0000 7f8bbf218700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb809ee10 con 0x7f8bb80aa1c0 2020-11-20T11:32:32.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.273+0000 7f8bb7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bb80a9d40 0x7f8bb813d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.273+0000 7f8bb7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bb80a9d40 0x7f8bb813d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40892/0 (socket says 172.21.15.23:40892) 2020-11-20T11:32:32.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.273+0000 7f8bb7fff700 1 -- 172.21.15.23:0/4005253625 learned_addr learned my addr 172.21.15.23:0/4005253625 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:32.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.273+0000 7f8bbcfb4700 1 --2- 172.21.15.23:0/4005253625 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bb8060a80 0x7f8bb809e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.273+0000 7f8bbd7b5700 1 --2- 172.21.15.23:0/4005253625 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8bb80aa1c0 0x7f8bb813d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.273+0000 7f8bb7fff700 1 -- 172.21.15.23:0/4005253625 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8bb80aa1c0 msgr2=0x7f8bb813d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.273+0000 7f8bb7fff700 1 --2- 172.21.15.23:0/4005253625 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8bb80aa1c0 0x7f8bb813d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.273+0000 7f8bb7fff700 1 -- 172.21.15.23:0/4005253625 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bb8060a80 msgr2=0x7f8bb809e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.273+0000 7f8bb7fff700 1 --2- 172.21.15.23:0/4005253625 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bb8060a80 0x7f8bb809e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.273+0000 7f8bb7fff700 1 -- 172.21.15.23:0/4005253625 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb813e000 con 0x7f8bb80a9d40 2020-11-20T11:32:32.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.273+0000 7f8bb7fff700 1 --2- 172.21.15.23:0/4005253625 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bb80a9d40 0x7f8bb813d3a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f8ba400c710 tx=0x7f8ba4007c90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:32.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.274+0000 7f8bb4ff9700 1 -- 172.21.15.23:0/4005253625 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba4007020 con 0x7f8bb80a9d40 2020-11-20T11:32:32.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.274+0000 7f8bb4ff9700 1 -- 172.21.15.23:0/4005253625 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ba4001e90 con 0x7f8bb80a9d40 2020-11-20T11:32:32.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.274+0000 7f8bbf218700 1 -- 172.21.15.23:0/4005253625 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bb813e250 con 0x7f8bb80a9d40 2020-11-20T11:32:32.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.274+0000 7f8bb4ff9700 1 -- 172.21.15.23:0/4005253625 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba40073b0 con 0x7f8bb80a9d40 2020-11-20T11:32:32.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.274+0000 7f8bbf218700 1 -- 172.21.15.23:0/4005253625 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bb80a79a0 con 0x7f8bb80a9d40 2020-11-20T11:32:32.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.274+0000 7f8bbf218700 1 -- 172.21.15.23:0/4005253625 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8bb813ebe0 con 0x7f8bb80a9d40 2020-11-20T11:32:32.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.275+0000 7f8bb4ff9700 1 -- 172.21.15.23:0/4005253625 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8ba401a060 con 0x7f8bb80a9d40 2020-11-20T11:32:32.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.275+0000 7f8bb4ff9700 1 --2- 172.21.15.23:0/4005253625 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8b90056d60 0x7f8b90059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.275+0000 7f8bb4ff9700 1 -- 172.21.15.23:0/4005253625 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f8ba406f6f0 con 0x7f8bb80a9d40 2020-11-20T11:32:32.287 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.275+0000 7f8bb4ff9700 1 --2- 172.21.15.23:0/4005253625 >> [v2:172.21.15.155:6800/23280,v1:172.21.15.155:6801/23280] conn(0x7f8b9005e940 0x7f8b90060d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.287 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.275+0000 7f8bb4ff9700 1 -- 172.21.15.23:0/4005253625 --> [v2:172.21.15.155:6800/23280,v1:172.21.15.155:6801/23280] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8b900613c0 con 0x7f8b9005e940 2020-11-20T11:32:32.287 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.275+0000 7f8bb4ff9700 1 -- 172.21.15.23:0/4005253625 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ba4071d60 con 0x7f8bb80a9d40 2020-11-20T11:32:32.287 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.276+0000 7f8bbcfb4700 1 --2- 172.21.15.23:0/4005253625 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8b90056d60 0x7f8b90059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.287 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.276+0000 7f8bbd7b5700 1 --2- 172.21.15.23:0/4005253625 >> [v2:172.21.15.155:6800/23280,v1:172.21.15.155:6801/23280] conn(0x7f8b9005e940 0x7f8b90060d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.288 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.276+0000 7f8bbcfb4700 1 --2- 172.21.15.23:0/4005253625 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8b90056d60 0x7f8b90059220 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f8bac0055e0 tx=0x7f8bac009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:32.288 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.276+0000 7f8bbd7b5700 1 --2- 172.21.15.23:0/4005253625 >> [v2:172.21.15.155:6800/23280,v1:172.21.15.155:6801/23280] conn(0x7f8b9005e940 0x7f8b90060d60 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:32.288 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.277+0000 7f8bb4ff9700 1 -- 172.21.15.23:0/4005253625 <== osd.4 v2:172.21.15.155:6800/23280 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7f8b900613c0 con 0x7f8b9005e940 2020-11-20T11:32:32.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.291+0000 7f8bbf218700 1 -- 172.21.15.23:0/4005253625 --> [v2:172.21.15.155:6800/23280,v1:172.21.15.155:6801/23280] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8bb813eda0 con 0x7f8b9005e940 2020-11-20T11:32:32.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.292+0000 7f8bb4ff9700 1 -- 172.21.15.23:0/4005253625 <== osd.4 v2:172.21.15.155:6800/23280 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8bb813eda0 con 0x7f8b9005e940 2020-11-20T11:32:32.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.292+0000 7f8bbf218700 1 -- 172.21.15.23:0/4005253625 >> [v2:172.21.15.155:6800/23280,v1:172.21.15.155:6801/23280] conn(0x7f8b9005e940 msgr2=0x7f8b90060d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.292+0000 7f8bbf218700 1 --2- 172.21.15.23:0/4005253625 >> [v2:172.21.15.155:6800/23280,v1:172.21.15.155:6801/23280] conn(0x7f8b9005e940 0x7f8b90060d60 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.292+0000 7f8bbf218700 1 -- 172.21.15.23:0/4005253625 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8b90056d60 msgr2=0x7f8b90059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.292+0000 7f8bbf218700 1 --2- 172.21.15.23:0/4005253625 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8b90056d60 0x7f8b90059220 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f8bac0055e0 tx=0x7f8bac009040).stop 2020-11-20T11:32:32.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.292+0000 7f8bbf218700 1 -- 172.21.15.23:0/4005253625 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bb80a9d40 msgr2=0x7f8bb813d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.292+0000 7f8bbf218700 1 --2- 172.21.15.23:0/4005253625 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bb80a9d40 0x7f8bb813d3a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f8ba400c710 tx=0x7f8ba4007c90).stop 2020-11-20T11:32:32.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.292+0000 7f8bbd7b5700 1 -- 172.21.15.23:0/4005253625 reap_dead start 2020-11-20T11:32:32.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.292+0000 7f8bbf218700 1 -- 172.21.15.23:0/4005253625 shutdown_connections 2020-11-20T11:32:32.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.292+0000 7f8bbf218700 1 -- 172.21.15.23:0/4005253625 >> 172.21.15.23:0/4005253625 conn(0x7f8bb8066730 msgr2=0x7f8bb8134c30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:32.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.292+0000 7f8bbf218700 1 -- 172.21.15.23:0/4005253625 shutdown_connections 2020-11-20T11:32:32.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.292+0000 7f8bbf218700 1 -- 172.21.15.23:0/4005253625 wait complete. 2020-11-20T11:32:32.306 INFO:teuthology.orchestra.run.smithi023.stdout:55834574853 2020-11-20T11:32:32.306 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2020-11-20T11:32:32.421 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.418+0000 7fbe04e48700 1 Processor -- start 2020-11-20T11:32:32.421 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.418+0000 7fbe04e48700 1 -- start start 2020-11-20T11:32:32.423 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.418+0000 7fbe04e48700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fbe000ae760 0x7fbe0013e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.423 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.418+0000 7fbe04e48700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbe00065100 0x7fbe00145610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.423 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.418+0000 7fbe04e48700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe000661b0 con 0x7fbe000aeb40 2020-11-20T11:32:32.424 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.419+0000 7fbe04e48700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe0013ad60 con 0x7fbe000ae760 2020-11-20T11:32:32.424 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.419+0000 7fbe04e48700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe00066c70 con 0x7fbe00065100 2020-11-20T11:32:32.424 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.419+0000 7fbdfe59c700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fbe000ae760 0x7fbe0013e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.424 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.419+0000 7fbdfed9d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbe00065100 0x7fbe00145610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.424 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.419+0000 7fbdfed9d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbe00065100 0x7fbe00145610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:51144/0 (socket says 172.21.15.23:51144) 2020-11-20T11:32:32.425 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.419+0000 7fbdf7fff700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fbe000aeb40 0x7fbe00137ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:33124/0 (socket says 172.21.15.23:33124) 2020-11-20T11:32:32.425 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.419+0000 7fbdfed9d700 1 -- 172.21.15.23:0/1438344400 learned_addr learned my addr 172.21.15.23:0/1438344400 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:32.425 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.419+0000 7fbdfe59c700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fbe000ae760 0x7fbe0013e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:33796/0 (socket says 172.21.15.23:33796) 2020-11-20T11:32:32.425 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.419+0000 7fbdfed9d700 1 -- 172.21.15.23:0/1438344400 >> v2:172.21.15.155:3300/0 conn(0x7fbe000ae760 msgr2=0x7fbe0013e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:32.426 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.419+0000 7fbdfed9d700 1 --2- 172.21.15.23:0/1438344400 >> v2:172.21.15.155:3300/0 conn(0x7fbe000ae760 0x7fbe0013e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.426 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.419+0000 7fbdfed9d700 1 -- 172.21.15.23:0/1438344400 >> v1:172.21.15.155:6789/0 conn(0x7fbe000aeb40 legacy=0x7fbe00137ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.426 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.419+0000 7fbdfed9d700 1 -- 172.21.15.23:0/1438344400 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe0013e470 con 0x7fbe00065100 2020-11-20T11:32:32.426 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.419+0000 7fbdfed9d700 1 --2- 172.21.15.23:0/1438344400 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbe00065100 0x7fbe00145610 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fbdf8009d90 tx=0x7fbdf80092d0).ready entity=mon.2 client_cookie=2d4d9c824f86b6fe server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:32.426 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.419+0000 7fbdfdd9b700 1 -- 172.21.15.23:0/1438344400 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf800bbf0 con 0x7fbe00065100 2020-11-20T11:32:32.427 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.419+0000 7fbdfdd9b700 1 -- 172.21.15.23:0/1438344400 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdf800bd50 con 0x7fbe00065100 2020-11-20T11:32:32.427 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.419+0000 7fbdfdd9b700 1 -- 172.21.15.23:0/1438344400 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf8011610 con 0x7fbe00065100 2020-11-20T11:32:32.427 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.420+0000 7fbe04e48700 1 -- 172.21.15.23:0/1438344400 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbe00065100 msgr2=0x7fbe00145610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.427 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.420+0000 7fbe04e48700 1 --2- 172.21.15.23:0/1438344400 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbe00065100 0x7fbe00145610 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fbdf8009d90 tx=0x7fbdf80092d0).stop 2020-11-20T11:32:32.428 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.420+0000 7fbe04e48700 1 -- 172.21.15.23:0/1438344400 shutdown_connections 2020-11-20T11:32:32.428 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.420+0000 7fbe04e48700 1 --2- 172.21.15.23:0/1438344400 >> v2:172.21.15.155:3300/0 conn(0x7fbe000ae760 0x7fbe0013e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.428 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.420+0000 7fbe04e48700 1 --2- 172.21.15.23:0/1438344400 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbe00065100 0x7fbe00145610 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.428 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.420+0000 7fbe04e48700 1 -- 172.21.15.23:0/1438344400 >> 172.21.15.23:0/1438344400 conn(0x7fbe00061d80 msgr2=0x7fbe000ae050 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:32.428 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.420+0000 7fbe04e48700 1 -- 172.21.15.23:0/1438344400 shutdown_connections 2020-11-20T11:32:32.429 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.420+0000 7fbe04e48700 1 -- 172.21.15.23:0/1438344400 wait complete. 2020-11-20T11:32:32.431 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.421+0000 7fbe04e48700 1 Processor -- start 2020-11-20T11:32:32.431 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.421+0000 7fbe04e48700 1 -- start start 2020-11-20T11:32:32.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.421+0000 7fbe04e48700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe00065100 0x7fbe000acff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.421+0000 7fbe04e48700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe000ae760 0x7fbe000a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.421+0000 7fbe04e48700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbe000aeb40 0x7fbe00134710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.421+0000 7fbe04e48700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0014a250 con 0x7fbe000ae760 2020-11-20T11:32:32.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.421+0000 7fbe04e48700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0014a4f0 con 0x7fbe00065100 2020-11-20T11:32:32.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.421+0000 7fbe04e48700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0014a790 con 0x7fbe000aeb40 2020-11-20T11:32:32.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.422+0000 7fbdf7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe000ae760 0x7fbe000a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.422+0000 7fbdf7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe000ae760 0x7fbe000a53c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40908/0 (socket says 172.21.15.23:40908) 2020-11-20T11:32:32.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.422+0000 7fbdf7fff700 1 -- 172.21.15.23:0/3385692829 learned_addr learned my addr 172.21.15.23:0/3385692829 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:32.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.422+0000 7fbdfe59c700 1 --2- 172.21.15.23:0/3385692829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe00065100 0x7fbe000acff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.422+0000 7fbdfed9d700 1 --2- 172.21.15.23:0/3385692829 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbe000aeb40 0x7fbe00134710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.422+0000 7fbdf7fff700 1 -- 172.21.15.23:0/3385692829 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbe000aeb40 msgr2=0x7fbe00134710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.422+0000 7fbdf7fff700 1 --2- 172.21.15.23:0/3385692829 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbe000aeb40 0x7fbe00134710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.422+0000 7fbdf7fff700 1 -- 172.21.15.23:0/3385692829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe00065100 msgr2=0x7fbe000acff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.422+0000 7fbdf7fff700 1 --2- 172.21.15.23:0/3385692829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe00065100 0x7fbe000acff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.422+0000 7fbdf7fff700 1 -- 172.21.15.23:0/3385692829 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe0014ac30 con 0x7fbe000ae760 2020-11-20T11:32:32.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.422+0000 7fbdf7fff700 1 --2- 172.21.15.23:0/3385692829 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe000ae760 0x7fbe000a53c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fbde800c6e0 tx=0x7fbde8007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:32.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.422+0000 7fbdf67fc700 1 -- 172.21.15.23:0/3385692829 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde80072e0 con 0x7fbe000ae760 2020-11-20T11:32:32.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.422+0000 7fbdf67fc700 1 -- 172.21.15.23:0/3385692829 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde8007440 con 0x7fbe000ae760 2020-11-20T11:32:32.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.422+0000 7fbe04e48700 1 -- 172.21.15.23:0/3385692829 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe00138970 con 0x7fbe000ae760 2020-11-20T11:32:32.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.422+0000 7fbdf67fc700 1 -- 172.21.15.23:0/3385692829 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde8005780 con 0x7fbe000ae760 2020-11-20T11:32:32.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.422+0000 7fbe04e48700 1 -- 172.21.15.23:0/3385692829 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe00149db0 con 0x7fbe000ae760 2020-11-20T11:32:32.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.423+0000 7fbe04e48700 1 -- 172.21.15.23:0/3385692829 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbe00062760 con 0x7fbe000ae760 2020-11-20T11:32:32.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.423+0000 7fbdf67fc700 1 -- 172.21.15.23:0/3385692829 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbde8010070 con 0x7fbe000ae760 2020-11-20T11:32:32.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.424+0000 7fbdf67fc700 1 --2- 172.21.15.23:0/3385692829 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbdd4056d60 0x7fbdd4059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.437 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.424+0000 7fbdf67fc700 1 -- 172.21.15.23:0/3385692829 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fbde806f5c0 con 0x7fbe000ae760 2020-11-20T11:32:32.437 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.424+0000 7fbdf67fc700 1 --2- 172.21.15.23:0/3385692829 >> [v2:172.21.15.155:6816/23298,v1:172.21.15.155:6817/23298] conn(0x7fbdd405e940 0x7fbdd4060d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.437 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.424+0000 7fbdf67fc700 1 -- 172.21.15.23:0/3385692829 --> [v2:172.21.15.155:6816/23298,v1:172.21.15.155:6817/23298] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbdd40613c0 con 0x7fbdd405e940 2020-11-20T11:32:32.437 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.424+0000 7fbdf67fc700 1 -- 172.21.15.23:0/3385692829 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbde8071c30 con 0x7fbe000ae760 2020-11-20T11:32:32.437 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.424+0000 7fbdfe59c700 1 --2- 172.21.15.23:0/3385692829 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbdd4056d60 0x7fbdd4059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.438 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.424+0000 7fbdfe59c700 1 --2- 172.21.15.23:0/3385692829 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbdd4056d60 0x7fbdd4059220 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fbe000aa0e0 tx=0x7fbdf0006e30).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:32.438 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.425+0000 7fbdfed9d700 1 --2- 172.21.15.23:0/3385692829 >> [v2:172.21.15.155:6816/23298,v1:172.21.15.155:6817/23298] conn(0x7fbdd405e940 0x7fbdd4060d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.438 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.426+0000 7fbdfed9d700 1 --2- 172.21.15.23:0/3385692829 >> [v2:172.21.15.155:6816/23298,v1:172.21.15.155:6817/23298] conn(0x7fbdd405e940 0x7fbdd4060d60 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:32.438 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.427+0000 7fbdf67fc700 1 -- 172.21.15.23:0/3385692829 <== osd.5 v2:172.21.15.155:6816/23298 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7fbdd40613c0 con 0x7fbdd405e940 2020-11-20T11:32:32.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.441+0000 7fbe04e48700 1 -- 172.21.15.23:0/3385692829 --> [v2:172.21.15.155:6816/23298,v1:172.21.15.155:6817/23298] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbe0005c7d0 con 0x7fbdd405e940 2020-11-20T11:32:32.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.442+0000 7fbdf67fc700 1 -- 172.21.15.23:0/3385692829 <== osd.5 v2:172.21.15.155:6816/23298 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbe0005c7d0 con 0x7fbdd405e940 2020-11-20T11:32:32.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.442+0000 7fbe04e48700 1 -- 172.21.15.23:0/3385692829 >> [v2:172.21.15.155:6816/23298,v1:172.21.15.155:6817/23298] conn(0x7fbdd405e940 msgr2=0x7fbdd4060d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.442+0000 7fbe04e48700 1 --2- 172.21.15.23:0/3385692829 >> [v2:172.21.15.155:6816/23298,v1:172.21.15.155:6817/23298] conn(0x7fbdd405e940 0x7fbdd4060d60 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.442+0000 7fbe04e48700 1 -- 172.21.15.23:0/3385692829 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbdd4056d60 msgr2=0x7fbdd4059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.442+0000 7fbe04e48700 1 --2- 172.21.15.23:0/3385692829 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbdd4056d60 0x7fbdd4059220 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fbe000aa0e0 tx=0x7fbdf0006e30).stop 2020-11-20T11:32:32.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.442+0000 7fbe04e48700 1 -- 172.21.15.23:0/3385692829 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe000ae760 msgr2=0x7fbe000a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.442+0000 7fbe04e48700 1 --2- 172.21.15.23:0/3385692829 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe000ae760 0x7fbe000a53c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fbde800c6e0 tx=0x7fbde8007c50).stop 2020-11-20T11:32:32.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.442+0000 7fbdfed9d700 1 -- 172.21.15.23:0/3385692829 reap_dead start 2020-11-20T11:32:32.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.442+0000 7fbe04e48700 1 -- 172.21.15.23:0/3385692829 shutdown_connections 2020-11-20T11:32:32.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.442+0000 7fbe04e48700 1 -- 172.21.15.23:0/3385692829 >> 172.21.15.23:0/3385692829 conn(0x7fbe00061d80 msgr2=0x7fbe001380f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:32.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.442+0000 7fbe04e48700 1 -- 172.21.15.23:0/3385692829 shutdown_connections 2020-11-20T11:32:32.448 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.443+0000 7fbe04e48700 1 -- 172.21.15.23:0/3385692829 wait complete. 2020-11-20T11:32:32.456 INFO:teuthology.orchestra.run.smithi023.stdout:55834574854 2020-11-20T11:32:32.456 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2020-11-20T11:32:32.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.568+0000 7f9fe79be700 1 Processor -- start 2020-11-20T11:32:32.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.568+0000 7f9fe79be700 1 -- start start 2020-11-20T11:32:32.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.569+0000 7f9fe79be700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9fe00aaab0 0x7f9fe013ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.569+0000 7f9fe79be700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe01388b0 con 0x7f9fe00aae90 2020-11-20T11:32:32.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.569+0000 7f9fe79be700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe013e600 con 0x7f9fe013aeb0 2020-11-20T11:32:32.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.569+0000 7f9fe79be700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe013e8a0 con 0x7f9fe00aaab0 2020-11-20T11:32:32.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.569+0000 7f9fe4f59700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f9fe00aae90 0x7f9fe01381a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55168/0 (socket says 172.21.15.23:55168) 2020-11-20T11:32:32.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.569+0000 7f9fe4f59700 1 -- 172.21.15.23:0/3047278258 learned_addr learned my addr 172.21.15.23:0/3047278258 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:32.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.569+0000 7f9fe575a700 1 --2- 172.21.15.23:0/3047278258 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9fe00aaab0 0x7f9fe013ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.569+0000 7f9fcffff700 1 -- 172.21.15.23:0/3047278258 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2605616857 0 0) 0x7f9fe01388b0 con 0x7f9fe00aae90 2020-11-20T11:32:32.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.570+0000 7f9fcffff700 1 -- 172.21.15.23:0/3047278258 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fc8003280 con 0x7f9fe00aae90 2020-11-20T11:32:32.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.570+0000 7f9fcffff700 1 -- 172.21.15.23:0/3047278258 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1026290412 0 0) 0x7f9fc8003280 con 0x7f9fe00aae90 2020-11-20T11:32:32.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.570+0000 7f9fcffff700 1 -- 172.21.15.23:0/3047278258 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9fe00aaab0 msgr2=0x7f9fe013ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:32.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.570+0000 7f9fcffff700 1 --2- 172.21.15.23:0/3047278258 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9fe00aaab0 0x7f9fe013ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.570+0000 7f9fcffff700 1 -- 172.21.15.23:0/3047278258 >> v1:172.21.15.155:6789/0 conn(0x7f9fe013aeb0 legacy=0x7f9fe00a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.570+0000 7f9fcffff700 1 -- 172.21.15.23:0/3047278258 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe00ab6d0 con 0x7f9fe00aae90 2020-11-20T11:32:32.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.570+0000 7f9fcffff700 1 -- 172.21.15.23:0/3047278258 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f9fdc001520 con 0x7f9fe00aae90 2020-11-20T11:32:32.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.570+0000 7f9fcffff700 1 -- 172.21.15.23:0/3047278258 >> v1:172.21.15.23:6789/0 conn(0x7f9fe00aae90 legacy=0x7f9fe01381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.570+0000 7f9fcffff700 1 --2- 172.21.15.23:0/3047278258 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9fe00aaab0 0x7f9fc80037a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.570+0000 7f9fcffff700 1 --2- 172.21.15.23:0/3047278258 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fc8003cb0 0x7f9fc8004520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.570+0000 7f9fcffff700 1 --2- 172.21.15.23:0/3047278258 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9fc8004a10 0x7f9fc8004e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.570+0000 7f9fcffff700 1 -- 172.21.15.23:0/3047278258 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe013e8a0 con 0x7f9fc8003cb0 2020-11-20T11:32:32.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.570+0000 7f9fcffff700 1 -- 172.21.15.23:0/3047278258 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc8004240 con 0x7f9fc8004a10 2020-11-20T11:32:32.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.570+0000 7f9fcffff700 1 -- 172.21.15.23:0/3047278258 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc800b770 con 0x7f9fe00aaab0 2020-11-20T11:32:32.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.570+0000 7f9fe4f59700 1 --2- 172.21.15.23:0/3047278258 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fc8003cb0 0x7f9fc8004520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.570+0000 7f9fe575a700 1 --2- 172.21.15.23:0/3047278258 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9fe00aaab0 0x7f9fc80037a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.570+0000 7f9fe4f59700 1 -- 172.21.15.23:0/3047278258 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9fe00aaab0 msgr2=0x7f9fc80037a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:32.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.570+0000 7f9fe4f59700 1 --2- 172.21.15.23:0/3047278258 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9fe00aaab0 0x7f9fc80037a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.570+0000 7f9fe4f59700 1 -- 172.21.15.23:0/3047278258 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9fc8004a10 msgr2=0x7f9fc8004e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:32.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.570+0000 7f9fe4f59700 1 --2- 172.21.15.23:0/3047278258 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9fc8004a10 0x7f9fc8004e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:32:32.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.570+0000 7f9fe4f59700 1 -- 172.21.15.23:0/3047278258 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc8003280 con 0x7f9fc8003cb0 2020-11-20T11:32:32.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.570+0000 7f9fe4f59700 1 --2- 172.21.15.23:0/3047278258 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fc8003cb0 0x7f9fc8004520 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f9fe0068560 tx=0x7f9fdc007810).ready entity=mon.0 client_cookie=1e70dd308dd0640b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:32.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.570+0000 7f9fcffff700 1 -- 172.21.15.23:0/3047278258 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc00ce90 con 0x7f9fc8003cb0 2020-11-20T11:32:32.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.571+0000 7f9fcffff700 1 -- 172.21.15.23:0/3047278258 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fdc0070d0 con 0x7f9fc8003cb0 2020-11-20T11:32:32.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.571+0000 7f9fcffff700 1 -- 172.21.15.23:0/3047278258 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc007470 con 0x7f9fc8003cb0 2020-11-20T11:32:32.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.571+0000 7f9fe79be700 1 -- 172.21.15.23:0/3047278258 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fc8003cb0 msgr2=0x7f9fc8004520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.571+0000 7f9fe79be700 1 --2- 172.21.15.23:0/3047278258 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fc8003cb0 0x7f9fc8004520 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f9fe0068560 tx=0x7f9fdc007810).stop 2020-11-20T11:32:32.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.571+0000 7f9fe5f5b700 1 -- 172.21.15.23:0/3047278258 reap_dead start 2020-11-20T11:32:32.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.571+0000 7f9fe79be700 1 -- 172.21.15.23:0/3047278258 shutdown_connections 2020-11-20T11:32:32.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.571+0000 7f9fe79be700 1 -- 172.21.15.23:0/3047278258 >> 172.21.15.23:0/3047278258 conn(0x7f9fe00678e0 msgr2=0x7f9fe0067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:32.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.571+0000 7f9fe79be700 1 -- 172.21.15.23:0/3047278258 shutdown_connections 2020-11-20T11:32:32.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.571+0000 7f9fe79be700 1 -- 172.21.15.23:0/3047278258 wait complete. 2020-11-20T11:32:32.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.572+0000 7f9fe79be700 1 Processor -- start 2020-11-20T11:32:32.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.572+0000 7f9fe79be700 1 -- start start 2020-11-20T11:32:32.585 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.573+0000 7f9fe79be700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fe00a7770 0x7f9fe01418e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.573+0000 7f9fe79be700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9fe009eeb0 0x7f9fe0140d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.573+0000 7f9fe79be700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9fe01411f0 0x7f9fe014a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.573+0000 7f9fe79be700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe014a500 con 0x7f9fe00a7770 2020-11-20T11:32:32.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.573+0000 7f9fe79be700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe0142110 con 0x7f9fe01411f0 2020-11-20T11:32:32.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.573+0000 7f9fe79be700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe01423b0 con 0x7f9fe009eeb0 2020-11-20T11:32:32.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.573+0000 7f9fe575a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fe00a7770 0x7f9fe01418e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.573+0000 7f9fe575a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fe00a7770 0x7f9fe01418e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40928/0 (socket says 172.21.15.23:40928) 2020-11-20T11:32:32.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.573+0000 7f9fe575a700 1 -- 172.21.15.23:0/1475737081 learned_addr learned my addr 172.21.15.23:0/1475737081 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:32.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.573+0000 7f9fe5f5b700 1 --2- 172.21.15.23:0/1475737081 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9fe01411f0 0x7f9fe014a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.573+0000 7f9fe575a700 1 -- 172.21.15.23:0/1475737081 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9fe009eeb0 msgr2=0x7f9fe0140d00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2020-11-20T11:32:32.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.573+0000 7f9fe575a700 1 --2- 172.21.15.23:0/1475737081 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9fe009eeb0 0x7f9fe0140d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:32:32.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.573+0000 7f9fe575a700 1 -- 172.21.15.23:0/1475737081 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9fe01411f0 msgr2=0x7f9fe014a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.573+0000 7f9fe575a700 1 --2- 172.21.15.23:0/1475737081 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9fe01411f0 0x7f9fe014a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.573+0000 7f9fe575a700 1 -- 172.21.15.23:0/1475737081 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe0140860 con 0x7f9fe00a7770 2020-11-20T11:32:32.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.573+0000 7f9fe575a700 1 --2- 172.21.15.23:0/1475737081 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fe00a7770 0x7f9fe01418e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f9fd0011cc0 tx=0x7f9fd0006e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:32.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.573+0000 7f9fcd7fa700 1 -- 172.21.15.23:0/1475737081 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd0011e70 con 0x7f9fe00a7770 2020-11-20T11:32:32.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.573+0000 7f9fcd7fa700 1 -- 172.21.15.23:0/1475737081 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fd0000e90 con 0x7f9fe00a7770 2020-11-20T11:32:32.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.573+0000 7f9fe79be700 1 -- 172.21.15.23:0/1475737081 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fe00a7b50 con 0x7f9fe00a7770 2020-11-20T11:32:32.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.573+0000 7f9fcd7fa700 1 -- 172.21.15.23:0/1475737081 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd000fee0 con 0x7f9fe00a7770 2020-11-20T11:32:32.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.573+0000 7f9fe79be700 1 -- 172.21.15.23:0/1475737081 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fe014ae20 con 0x7f9fe00a7770 2020-11-20T11:32:32.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.574+0000 7f9fe79be700 1 -- 172.21.15.23:0/1475737081 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9fe005c7d0 con 0x7f9fe00a7770 2020-11-20T11:32:32.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.575+0000 7f9fcd7fa700 1 -- 172.21.15.23:0/1475737081 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9fd0014650 con 0x7f9fe00a7770 2020-11-20T11:32:32.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.575+0000 7f9fcd7fa700 1 --2- 172.21.15.23:0/1475737081 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9fb8056d90 0x7f9fb8059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.575+0000 7f9fcd7fa700 1 -- 172.21.15.23:0/1475737081 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f9fd00709b0 con 0x7f9fe00a7770 2020-11-20T11:32:32.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.575+0000 7f9fcd7fa700 1 --2- 172.21.15.23:0/1475737081 >> [v2:172.21.15.155:6824/23307,v1:172.21.15.155:6825/23307] conn(0x7f9fb805e970 0x7f9fb8060d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.575+0000 7f9fcd7fa700 1 -- 172.21.15.23:0/1475737081 --> [v2:172.21.15.155:6824/23307,v1:172.21.15.155:6825/23307] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9fb80613f0 con 0x7f9fb805e970 2020-11-20T11:32:32.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.575+0000 7f9fcd7fa700 1 -- 172.21.15.23:0/1475737081 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9fd0075220 con 0x7f9fe00a7770 2020-11-20T11:32:32.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.575+0000 7f9fe4f59700 1 --2- 172.21.15.23:0/1475737081 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9fb8056d90 0x7f9fb8059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.575+0000 7f9fe5f5b700 1 --2- 172.21.15.23:0/1475737081 >> [v2:172.21.15.155:6824/23307,v1:172.21.15.155:6825/23307] conn(0x7f9fb805e970 0x7f9fb8060d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.575+0000 7f9fe4f59700 1 --2- 172.21.15.23:0/1475737081 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9fb8056d90 0x7f9fb8059250 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f9fe0068560 tx=0x7f9fdc015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:32.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.576+0000 7f9fe5f5b700 1 --2- 172.21.15.23:0/1475737081 >> [v2:172.21.15.155:6824/23307,v1:172.21.15.155:6825/23307] conn(0x7f9fb805e970 0x7f9fb8060d90 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:32.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.577+0000 7f9fcd7fa700 1 -- 172.21.15.23:0/1475737081 <== osd.6 v2:172.21.15.155:6824/23307 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7f9fb80613f0 con 0x7f9fb805e970 2020-11-20T11:32:32.593 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.591+0000 7f9fe79be700 1 -- 172.21.15.23:0/1475737081 --> [v2:172.21.15.155:6824/23307,v1:172.21.15.155:6825/23307] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9fe0044710 con 0x7f9fb805e970 2020-11-20T11:32:32.595 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.591+0000 7f9fcd7fa700 1 -- 172.21.15.23:0/1475737081 <== osd.6 v2:172.21.15.155:6824/23307 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9fe0044710 con 0x7f9fb805e970 2020-11-20T11:32:32.595 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.592+0000 7f9fe79be700 1 -- 172.21.15.23:0/1475737081 >> [v2:172.21.15.155:6824/23307,v1:172.21.15.155:6825/23307] conn(0x7f9fb805e970 msgr2=0x7f9fb8060d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.595 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.592+0000 7f9fe79be700 1 --2- 172.21.15.23:0/1475737081 >> [v2:172.21.15.155:6824/23307,v1:172.21.15.155:6825/23307] conn(0x7f9fb805e970 0x7f9fb8060d90 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.595 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.592+0000 7f9fe79be700 1 -- 172.21.15.23:0/1475737081 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9fb8056d90 msgr2=0x7f9fb8059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.596 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.592+0000 7f9fe79be700 1 --2- 172.21.15.23:0/1475737081 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9fb8056d90 0x7f9fb8059250 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f9fe0068560 tx=0x7f9fdc015000).stop 2020-11-20T11:32:32.596 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.592+0000 7f9fe79be700 1 -- 172.21.15.23:0/1475737081 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fe00a7770 msgr2=0x7f9fe01418e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.596 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.592+0000 7f9fe79be700 1 --2- 172.21.15.23:0/1475737081 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fe00a7770 0x7f9fe01418e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f9fd0011cc0 tx=0x7f9fd0006e30).stop 2020-11-20T11:32:32.596 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.592+0000 7f9fe5f5b700 1 -- 172.21.15.23:0/1475737081 reap_dead start 2020-11-20T11:32:32.596 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.592+0000 7f9fe79be700 1 -- 172.21.15.23:0/1475737081 shutdown_connections 2020-11-20T11:32:32.597 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.592+0000 7f9fe79be700 1 -- 172.21.15.23:0/1475737081 >> 172.21.15.23:0/1475737081 conn(0x7f9fe00678e0 msgr2=0x7f9fe01383b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:32.597 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.592+0000 7f9fe79be700 1 -- 172.21.15.23:0/1475737081 shutdown_connections 2020-11-20T11:32:32.597 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.592+0000 7f9fe79be700 1 -- 172.21.15.23:0/1475737081 wait complete. 2020-11-20T11:32:32.605 INFO:teuthology.orchestra.run.smithi023.stdout:60129542150 2020-11-20T11:32:32.606 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2020-11-20T11:32:32.740 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.738+0000 7fd5d62eb700 1 Processor -- start 2020-11-20T11:32:32.741 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.738+0000 7fd5d62eb700 1 -- start start 2020-11-20T11:32:32.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.738+0000 7fd5d62eb700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5d00a9d40 0x7fd5d00aa1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.738+0000 7fd5d62eb700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd5d00abb80 0x7fd5d00ac010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.738+0000 7fd5d62eb700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5d0067bc0 con 0x7fd5d0060a80 2020-11-20T11:32:32.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.738+0000 7fd5d62eb700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5d0062060 con 0x7fd5d00abb80 2020-11-20T11:32:32.744 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.738+0000 7fd5d62eb700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d0067e60 con 0x7fd5d00a9d40 2020-11-20T11:32:32.744 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.738+0000 7fd5d4888700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd5d00abb80 0x7fd5d00ac010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.738+0000 7fd5d4888700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd5d00abb80 0x7fd5d00ac010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40942/0 (socket says 172.21.15.23:40942) 2020-11-20T11:32:32.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.738+0000 7fd5d4888700 1 -- 172.21.15.23:0/2876617103 learned_addr learned my addr 172.21.15.23:0/2876617103 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:32.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.739+0000 7fd5cf7fe700 1 --2- 172.21.15.23:0/2876617103 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5d00a9d40 0x7fd5d00aa1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.739+0000 7fd5d4888700 1 -- 172.21.15.23:0/2876617103 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5d00a9d40 msgr2=0x7fd5d00aa1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:32.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.739+0000 7fd5d4888700 1 --2- 172.21.15.23:0/2876617103 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5d00a9d40 0x7fd5d00aa1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.739+0000 7fd5d4888700 1 -- 172.21.15.23:0/2876617103 >> v1:172.21.15.155:6789/0 conn(0x7fd5d0060a80 legacy=0x7fd5d0061950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.739+0000 7fd5d4888700 1 -- 172.21.15.23:0/2876617103 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d0068100 con 0x7fd5d00abb80 2020-11-20T11:32:32.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.739+0000 7fd5d4888700 1 --2- 172.21.15.23:0/2876617103 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5d00abb80 0x7fd5d00ac010 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fd5c8009d90 tx=0x7fd5c80092d0).ready entity=mon.0 client_cookie=c5b911b65b929be9 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:32.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.739+0000 7fd5ceffd700 1 -- 172.21.15.23:0/2876617103 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c800bbf0 con 0x7fd5d00abb80 2020-11-20T11:32:32.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.739+0000 7fd5ceffd700 1 -- 172.21.15.23:0/2876617103 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5c800bd50 con 0x7fd5d00abb80 2020-11-20T11:32:32.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.739+0000 7fd5ceffd700 1 -- 172.21.15.23:0/2876617103 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c8011610 con 0x7fd5d00abb80 2020-11-20T11:32:32.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.739+0000 7fd5d62eb700 1 -- 172.21.15.23:0/2876617103 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5d00abb80 msgr2=0x7fd5d00ac010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.739+0000 7fd5d62eb700 1 --2- 172.21.15.23:0/2876617103 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5d00abb80 0x7fd5d00ac010 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fd5c8009d90 tx=0x7fd5c80092d0).stop 2020-11-20T11:32:32.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.740+0000 7fd5d62eb700 1 -- 172.21.15.23:0/2876617103 shutdown_connections 2020-11-20T11:32:32.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.740+0000 7fd5d62eb700 1 --2- 172.21.15.23:0/2876617103 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5d00abb80 0x7fd5d00ac010 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.740+0000 7fd5d62eb700 1 --2- 172.21.15.23:0/2876617103 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5d00a9d40 0x7fd5d00aa1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.740+0000 7fd5d62eb700 1 -- 172.21.15.23:0/2876617103 >> 172.21.15.23:0/2876617103 conn(0x7fd5d0066730 msgr2=0x7fd5d00601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:32.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.740+0000 7fd5d62eb700 1 -- 172.21.15.23:0/2876617103 shutdown_connections 2020-11-20T11:32:32.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.740+0000 7fd5d62eb700 1 -- 172.21.15.23:0/2876617103 wait complete. 2020-11-20T11:32:32.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.741+0000 7fd5d62eb700 1 Processor -- start 2020-11-20T11:32:32.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.741+0000 7fd5d62eb700 1 -- start start 2020-11-20T11:32:32.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.741+0000 7fd5d62eb700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5d0060a80 0x7fd5d0137180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.741+0000 7fd5d62eb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5d00a9d40 0x7fd5d00a7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.741+0000 7fd5d62eb700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5d00abb80 0x7fd5d00a66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.741+0000 7fd5d62eb700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d0138b90 con 0x7fd5d00a9d40 2020-11-20T11:32:32.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.741+0000 7fd5d62eb700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d013c3b0 con 0x7fd5d0060a80 2020-11-20T11:32:32.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.741+0000 7fd5d62eb700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d013c5e0 con 0x7fd5d00abb80 2020-11-20T11:32:32.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.741+0000 7fd5cf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5d00a9d40 0x7fd5d00a7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.741+0000 7fd5cf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5d00a9d40 0x7fd5d00a7440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40946/0 (socket says 172.21.15.23:40946) 2020-11-20T11:32:32.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.741+0000 7fd5cf7fe700 1 -- 172.21.15.23:0/3077016449 learned_addr learned my addr 172.21.15.23:0/3077016449 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:32.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.742+0000 7fd5cffff700 1 --2- 172.21.15.23:0/3077016449 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5d0060a80 0x7fd5d0137180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.742+0000 7fd5d4888700 1 --2- 172.21.15.23:0/3077016449 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5d00abb80 0x7fd5d00a66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.742+0000 7fd5cf7fe700 1 -- 172.21.15.23:0/3077016449 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5d00abb80 msgr2=0x7fd5d00a66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.742+0000 7fd5cf7fe700 1 --2- 172.21.15.23:0/3077016449 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5d00abb80 0x7fd5d00a66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.742+0000 7fd5cf7fe700 1 -- 172.21.15.23:0/3077016449 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5d0060a80 msgr2=0x7fd5d0137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.742+0000 7fd5cf7fe700 1 --2- 172.21.15.23:0/3077016449 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5d0060a80 0x7fd5d0137180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.742+0000 7fd5cf7fe700 1 -- 172.21.15.23:0/3077016449 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d00a7930 con 0x7fd5d00a9d40 2020-11-20T11:32:32.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.742+0000 7fd5cf7fe700 1 --2- 172.21.15.23:0/3077016449 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5d00a9d40 0x7fd5d00a7440 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fd5c400db60 tx=0x7fd5c4008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:32.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.742+0000 7fd5affff700 1 -- 172.21.15.23:0/3077016449 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c400be50 con 0x7fd5d00a9d40 2020-11-20T11:32:32.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.742+0000 7fd5affff700 1 -- 172.21.15.23:0/3077016449 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5c40097c0 con 0x7fd5d00a9d40 2020-11-20T11:32:32.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.742+0000 7fd5affff700 1 -- 172.21.15.23:0/3077016449 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4009b40 con 0x7fd5d00a9d40 2020-11-20T11:32:32.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.742+0000 7fd5d62eb700 1 -- 172.21.15.23:0/3077016449 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5d0141920 con 0x7fd5d00a9d40 2020-11-20T11:32:32.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.742+0000 7fd5d62eb700 1 -- 172.21.15.23:0/3077016449 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5d00a6d60 con 0x7fd5d00a9d40 2020-11-20T11:32:32.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.743+0000 7fd5d62eb700 1 -- 172.21.15.23:0/3077016449 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd5d013c960 con 0x7fd5d00a9d40 2020-11-20T11:32:32.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.743+0000 7fd5affff700 1 -- 172.21.15.23:0/3077016449 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd5c4013480 con 0x7fd5d00a9d40 2020-11-20T11:32:32.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.743+0000 7fd5affff700 1 --2- 172.21.15.23:0/3077016449 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd5a8056d60 0x7fd5a8059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.744+0000 7fd5affff700 1 -- 172.21.15.23:0/3077016449 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fd5c40702a0 con 0x7fd5d00a9d40 2020-11-20T11:32:32.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.744+0000 7fd5affff700 1 --2- 172.21.15.23:0/3077016449 >> [v2:172.21.15.155:6808/23281,v1:172.21.15.155:6809/23281] conn(0x7fd5a805e940 0x7fd5a8060d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.744+0000 7fd5affff700 1 -- 172.21.15.23:0/3077016449 --> [v2:172.21.15.155:6808/23281,v1:172.21.15.155:6809/23281] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd5a80613c0 con 0x7fd5a805e940 2020-11-20T11:32:32.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.744+0000 7fd5cffff700 1 --2- 172.21.15.23:0/3077016449 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd5a8056d60 0x7fd5a8059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.744+0000 7fd5affff700 1 -- 172.21.15.23:0/3077016449 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd5c4072910 con 0x7fd5d00a9d40 2020-11-20T11:32:32.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.744+0000 7fd5d4888700 1 --2- 172.21.15.23:0/3077016449 >> [v2:172.21.15.155:6808/23281,v1:172.21.15.155:6809/23281] conn(0x7fd5a805e940 0x7fd5a8060d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.744+0000 7fd5cffff700 1 --2- 172.21.15.23:0/3077016449 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd5a8056d60 0x7fd5a8059220 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fd5c00037b0 tx=0x7fd5c0000bc0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:32.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.744+0000 7fd5d4888700 1 --2- 172.21.15.23:0/3077016449 >> [v2:172.21.15.155:6808/23281,v1:172.21.15.155:6809/23281] conn(0x7fd5a805e940 0x7fd5a8060d60 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:32.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.745+0000 7fd5affff700 1 -- 172.21.15.23:0/3077016449 <== osd.7 v2:172.21.15.155:6808/23281 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7fd5a80613c0 con 0x7fd5a805e940 2020-11-20T11:32:32.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.759+0000 7fd5d62eb700 1 -- 172.21.15.23:0/3077016449 --> [v2:172.21.15.155:6808/23281,v1:172.21.15.155:6809/23281] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd5d00ae290 con 0x7fd5a805e940 2020-11-20T11:32:32.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.760+0000 7fd5affff700 1 -- 172.21.15.23:0/3077016449 <== osd.7 v2:172.21.15.155:6808/23281 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd5d00ae290 con 0x7fd5a805e940 2020-11-20T11:32:32.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.761+0000 7fd5d62eb700 1 -- 172.21.15.23:0/3077016449 >> [v2:172.21.15.155:6808/23281,v1:172.21.15.155:6809/23281] conn(0x7fd5a805e940 msgr2=0x7fd5a8060d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.761+0000 7fd5d62eb700 1 --2- 172.21.15.23:0/3077016449 >> [v2:172.21.15.155:6808/23281,v1:172.21.15.155:6809/23281] conn(0x7fd5a805e940 0x7fd5a8060d60 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.761+0000 7fd5d62eb700 1 -- 172.21.15.23:0/3077016449 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd5a8056d60 msgr2=0x7fd5a8059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.761+0000 7fd5d62eb700 1 --2- 172.21.15.23:0/3077016449 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd5a8056d60 0x7fd5a8059220 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fd5c00037b0 tx=0x7fd5c0000bc0).stop 2020-11-20T11:32:32.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.761+0000 7fd5d62eb700 1 -- 172.21.15.23:0/3077016449 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5d00a9d40 msgr2=0x7fd5d00a7440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.761+0000 7fd5d62eb700 1 --2- 172.21.15.23:0/3077016449 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5d00a9d40 0x7fd5d00a7440 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fd5c400db60 tx=0x7fd5c4008040).stop 2020-11-20T11:32:32.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.761+0000 7fd5d4888700 1 -- 172.21.15.23:0/3077016449 reap_dead start 2020-11-20T11:32:32.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.761+0000 7fd5d62eb700 1 -- 172.21.15.23:0/3077016449 shutdown_connections 2020-11-20T11:32:32.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.761+0000 7fd5d62eb700 1 -- 172.21.15.23:0/3077016449 >> 172.21.15.23:0/3077016449 conn(0x7fd5d0066730 msgr2=0x7fd5d013b630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:32.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.761+0000 7fd5d62eb700 1 -- 172.21.15.23:0/3077016449 shutdown_connections 2020-11-20T11:32:32.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.761+0000 7fd5d62eb700 1 -- 172.21.15.23:0/3077016449 wait complete. 2020-11-20T11:32:32.776 INFO:teuthology.orchestra.run.smithi023.stdout:55834574854 2020-11-20T11:32:32.781 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2020-11-20T11:32:32.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.904+0000 7fe4f7333700 1 Processor -- start 2020-11-20T11:32:32.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.905+0000 7fe4f7333700 1 -- start start 2020-11-20T11:32:32.910 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.905+0000 7fe4f7333700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe4f00aaab0 0x7fe4f013ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.910 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.905+0000 7fe4f7333700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe4f013aeb0 0x7fe4f00a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.910 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.905+0000 7fe4f7333700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f01388b0 con 0x7fe4f00aae90 2020-11-20T11:32:32.911 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.905+0000 7fe4f7333700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4f013e600 con 0x7fe4f013aeb0 2020-11-20T11:32:32.911 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.905+0000 7fe4f7333700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4f013e8a0 con 0x7fe4f00aaab0 2020-11-20T11:32:32.911 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.905+0000 7fe4f58d0700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe4f013aeb0 0x7fe4f00a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.911 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.905+0000 7fe4f48ce700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fe4f00aae90 0x7fe4f01381a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55204/0 (socket says 172.21.15.23:55204) 2020-11-20T11:32:32.911 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.905+0000 7fe4f58d0700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe4f013aeb0 0x7fe4f00a8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40958/0 (socket says 172.21.15.23:40958) 2020-11-20T11:32:32.912 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.905+0000 7fe4f48ce700 1 -- 172.21.15.23:0/2941017201 learned_addr learned my addr 172.21.15.23:0/2941017201 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:32.912 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.906+0000 7fe4f50cf700 1 --2- 172.21.15.23:0/2941017201 >> v2:172.21.15.155:3300/0 conn(0x7fe4f00aaab0 0x7fe4f013ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.912 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.906+0000 7fe4f58d0700 1 -- 172.21.15.23:0/2941017201 >> v2:172.21.15.155:3300/0 conn(0x7fe4f00aaab0 msgr2=0x7fe4f013ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:32.912 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.906+0000 7fe4f58d0700 1 --2- 172.21.15.23:0/2941017201 >> v2:172.21.15.155:3300/0 conn(0x7fe4f00aaab0 0x7fe4f013ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.912 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.906+0000 7fe4f58d0700 1 -- 172.21.15.23:0/2941017201 >> v1:172.21.15.23:6789/0 conn(0x7fe4f00aae90 legacy=0x7fe4f01381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.913 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.906+0000 7fe4e3fff700 1 -- 172.21.15.23:0/2941017201 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3529294596 0 0) 0x7fe4f01388b0 con 0x7fe4f00aae90 2020-11-20T11:32:32.913 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.906+0000 7fe4f58d0700 1 -- 172.21.15.23:0/2941017201 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f00ab6d0 con 0x7fe4f013aeb0 2020-11-20T11:32:32.913 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.906+0000 7fe4f58d0700 1 --2- 172.21.15.23:0/2941017201 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4f013aeb0 0x7fe4f00a8c60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fe4f00b23d0 tx=0x7fe4e8009220).ready entity=mon.0 client_cookie=e1648bbb9a951008 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:32.913 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.906+0000 7fe4e3fff700 1 -- 172.21.15.23:0/2941017201 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e800bbf0 con 0x7fe4f013aeb0 2020-11-20T11:32:32.913 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.906+0000 7fe4e3fff700 1 -- 172.21.15.23:0/2941017201 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e800bd50 con 0x7fe4f013aeb0 2020-11-20T11:32:32.914 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.906+0000 7fe4e3fff700 1 -- 172.21.15.23:0/2941017201 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e8011610 con 0x7fe4f013aeb0 2020-11-20T11:32:32.914 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.906+0000 7fe4f7333700 1 -- 172.21.15.23:0/2941017201 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4f013aeb0 msgr2=0x7fe4f00a8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.914 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.906+0000 7fe4f7333700 1 --2- 172.21.15.23:0/2941017201 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4f013aeb0 0x7fe4f00a8c60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fe4f00b23d0 tx=0x7fe4e8009220).stop 2020-11-20T11:32:32.914 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.906+0000 7fe4f7333700 1 -- 172.21.15.23:0/2941017201 shutdown_connections 2020-11-20T11:32:32.914 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.906+0000 7fe4f7333700 1 --2- 172.21.15.23:0/2941017201 >> v2:172.21.15.155:3300/0 conn(0x7fe4f00aaab0 0x7fe4f013ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.915 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.906+0000 7fe4f7333700 1 --2- 172.21.15.23:0/2941017201 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4f013aeb0 0x7fe4f00a8c60 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.915 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.906+0000 7fe4f7333700 1 -- 172.21.15.23:0/2941017201 >> 172.21.15.23:0/2941017201 conn(0x7fe4f00678e0 msgr2=0x7fe4f0067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:32.915 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.907+0000 7fe4f7333700 1 -- 172.21.15.23:0/2941017201 shutdown_connections 2020-11-20T11:32:32.915 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.907+0000 7fe4f7333700 1 -- 172.21.15.23:0/2941017201 wait complete. 2020-11-20T11:32:32.915 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.907+0000 7fe4f7333700 1 Processor -- start 2020-11-20T11:32:32.916 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.908+0000 7fe4f7333700 1 -- start start 2020-11-20T11:32:32.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.908+0000 7fe4f7333700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4f00aaab0 0x7fe4f0140d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.908+0000 7fe4f7333700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4f00aae90 0x7fe4f00a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.908+0000 7fe4f7333700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe4f00a6320 0x7fe4f014a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.908+0000 7fe4f7333700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f00a7430 con 0x7fe4f00aae90 2020-11-20T11:32:32.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.908+0000 7fe4f7333700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f014a2f0 con 0x7fe4f00aaab0 2020-11-20T11:32:32.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.908+0000 7fe4f7333700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f009dec0 con 0x7fe4f00a6320 2020-11-20T11:32:32.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.908+0000 7fe4f48ce700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4f00aae90 0x7fe4f00a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.908+0000 7fe4f48ce700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4f00aae90 0x7fe4f00a54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40962/0 (socket says 172.21.15.23:40962) 2020-11-20T11:32:32.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.908+0000 7fe4f48ce700 1 -- 172.21.15.23:0/3073223592 learned_addr learned my addr 172.21.15.23:0/3073223592 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:32.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.908+0000 7fe4f50cf700 1 --2- 172.21.15.23:0/3073223592 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4f00aaab0 0x7fe4f0140d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.909+0000 7fe4f58d0700 1 --2- 172.21.15.23:0/3073223592 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe4f00a6320 0x7fe4f014a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.909+0000 7fe4f48ce700 1 -- 172.21.15.23:0/3073223592 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe4f00a6320 msgr2=0x7fe4f014a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.909+0000 7fe4f48ce700 1 --2- 172.21.15.23:0/3073223592 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe4f00a6320 0x7fe4f014a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.909+0000 7fe4f48ce700 1 -- 172.21.15.23:0/3073223592 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4f00aaab0 msgr2=0x7fe4f0140d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:32.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.909+0000 7fe4f48ce700 1 --2- 172.21.15.23:0/3073223592 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4f00aaab0 0x7fe4f0140d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:32.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.909+0000 7fe4f48ce700 1 -- 172.21.15.23:0/3073223592 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f009e160 con 0x7fe4f00aae90 2020-11-20T11:32:32.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.909+0000 7fe4f48ce700 1 --2- 172.21.15.23:0/3073223592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4f00aae90 0x7fe4f00a54f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fe4ec00c7c0 tx=0x7fe4ec007b00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:32.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.909+0000 7fe4e17fa700 1 -- 172.21.15.23:0/3073223592 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec0072e0 con 0x7fe4f00aae90 2020-11-20T11:32:32.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.909+0000 7fe4e17fa700 1 -- 172.21.15.23:0/3073223592 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4ec007440 con 0x7fe4f00aae90 2020-11-20T11:32:32.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.909+0000 7fe4f7333700 1 -- 172.21.15.23:0/3073223592 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4f00a8120 con 0x7fe4f00aae90 2020-11-20T11:32:32.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.909+0000 7fe4e17fa700 1 -- 172.21.15.23:0/3073223592 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec005840 con 0x7fe4f00aae90 2020-11-20T11:32:32.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.909+0000 7fe4f7333700 1 -- 172.21.15.23:0/3073223592 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4f00a59e0 con 0x7fe4f00aae90 2020-11-20T11:32:32.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.910+0000 7fe4f7333700 1 -- 172.21.15.23:0/3073223592 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4f005c7d0 con 0x7fe4f00aae90 2020-11-20T11:32:32.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.910+0000 7fe4e17fa700 1 -- 172.21.15.23:0/3073223592 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe4ec01a060 con 0x7fe4f00aae90 2020-11-20T11:32:32.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.911+0000 7fe4e17fa700 1 --2- 172.21.15.23:0/3073223592 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe4c8056d60 0x7fe4c8059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:32.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.911+0000 7fe4e17fa700 1 -- 172.21.15.23:0/3073223592 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fe4ec06f7c0 con 0x7fe4f00aae90 2020-11-20T11:32:32.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.912+0000 7fe4f50cf700 1 --2- 172.21.15.23:0/3073223592 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe4c8056d60 0x7fe4c8059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:32.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.912+0000 7fe4f50cf700 1 --2- 172.21.15.23:0/3073223592 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe4c8056d60 0x7fe4c8059220 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fe4e4006fd0 tx=0x7fe4e4008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:32.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:32.914+0000 7fe4e17fa700 1 -- 172.21.15.23:0/3073223592 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe4ec044ab0 con 0x7fe4f00aae90 2020-11-20T11:32:33.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:33.063+0000 7fe4f7333700 1 -- 172.21.15.23:0/3073223592 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fe4f0044710 con 0x7fe4f00aae90 2020-11-20T11:32:33.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:33.063+0000 7fe4e17fa700 1 -- 172.21.15.23:0/3073223592 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe4ec018070 con 0x7fe4f00aae90 2020-11-20T11:32:33.067 INFO:teuthology.orchestra.run.smithi023.stdout:55834574852 2020-11-20T11:32:33.068 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:33.065+0000 7fe4f7333700 1 -- 172.21.15.23:0/3073223592 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe4c8056d60 msgr2=0x7fe4c8059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:33.068 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:33.065+0000 7fe4f7333700 1 --2- 172.21.15.23:0/3073223592 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe4c8056d60 0x7fe4c8059220 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fe4e4006fd0 tx=0x7fe4e4008040).stop 2020-11-20T11:32:33.068 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:33.065+0000 7fe4f7333700 1 -- 172.21.15.23:0/3073223592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4f00aae90 msgr2=0x7fe4f00a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:33.069 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:33.065+0000 7fe4f7333700 1 --2- 172.21.15.23:0/3073223592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4f00aae90 0x7fe4f00a54f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fe4ec00c7c0 tx=0x7fe4ec007b00).stop 2020-11-20T11:32:33.069 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:33.065+0000 7fe4f7333700 1 -- 172.21.15.23:0/3073223592 shutdown_connections 2020-11-20T11:32:33.069 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:33.065+0000 7fe4f7333700 1 --2- 172.21.15.23:0/3073223592 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe4c8056d60 0x7fe4c8059220 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:33.069 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:33.065+0000 7fe4f7333700 1 --2- 172.21.15.23:0/3073223592 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe4f00a6320 0x7fe4f014a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:33.070 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:33.065+0000 7fe4f7333700 1 --2- 172.21.15.23:0/3073223592 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe4f00aaab0 0x7fe4f0140d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:33.070 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:33.065+0000 7fe4f7333700 1 --2- 172.21.15.23:0/3073223592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe4f00aae90 0x7fe4f00a54f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:33.070 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:33.065+0000 7fe4f7333700 1 -- 172.21.15.23:0/3073223592 >> 172.21.15.23:0/3073223592 conn(0x7fe4f00678e0 msgr2=0x7fe4f00adce0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:33.070 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:33.065+0000 7fe4f7333700 1 -- 172.21.15.23:0/3073223592 shutdown_connections 2020-11-20T11:32:33.070 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:33.065+0000 7fe4f7333700 1 -- 172.21.15.23:0/3073223592 wait complete. 2020-11-20T11:32:33.079 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.0 2020-11-20T11:32:34.080 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2020-11-20T11:32:34.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.206+0000 7f1dc9d42700 1 Processor -- start 2020-11-20T11:32:34.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.206+0000 7f1dc9d42700 1 -- start start 2020-11-20T11:32:34.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.207+0000 7f1dc9d42700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1dc4060a80 0x7f1dc4061950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:34.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.207+0000 7f1dc9d42700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f1dc40a9d40 0x7f1dc40aa1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:34.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.207+0000 7f1dc9d42700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc4067bc0 con 0x7f1dc4137920 2020-11-20T11:32:34.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.207+0000 7f1dc9d42700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dc4067e60 con 0x7f1dc4060a80 2020-11-20T11:32:34.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.207+0000 7f1dc9d42700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dc4068100 con 0x7f1dc40a9d40 2020-11-20T11:32:34.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.207+0000 7f1dc37fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1dc4060a80 0x7f1dc4061950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:34.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.207+0000 7f1dc3fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f1dc4137920 0x7f1dc4137d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55220/0 (socket says 172.21.15.23:55220) 2020-11-20T11:32:34.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.207+0000 7f1dc3fff700 1 -- 172.21.15.23:0/3360442548 learned_addr learned my addr 172.21.15.23:0/3360442548 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:34.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.207+0000 7f1dc2ffd700 1 --2- 172.21.15.23:0/3360442548 >> v2:172.21.15.155:3300/0 conn(0x7f1dc40a9d40 0x7f1dc40aa1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:34.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.207+0000 7f1dc27fc700 1 -- 172.21.15.23:0/3360442548 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3604012536 0 0) 0x7f1dc4067bc0 con 0x7f1dc4137920 2020-11-20T11:32:34.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.207+0000 7f1dc27fc700 1 -- 172.21.15.23:0/3360442548 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1db0003280 con 0x7f1dc4137920 2020-11-20T11:32:34.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.207+0000 7f1dc37fe700 1 -- 172.21.15.23:0/3360442548 >> v2:172.21.15.155:3300/0 conn(0x7f1dc40a9d40 msgr2=0x7f1dc40aa1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:34.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.207+0000 7f1dc37fe700 1 --2- 172.21.15.23:0/3360442548 >> v2:172.21.15.155:3300/0 conn(0x7f1dc40a9d40 0x7f1dc40aa1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:34.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.207+0000 7f1dc37fe700 1 -- 172.21.15.23:0/3360442548 >> v1:172.21.15.23:6789/0 conn(0x7f1dc4137920 legacy=0x7f1dc4137d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:34.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.207+0000 7f1dc37fe700 1 -- 172.21.15.23:0/3360442548 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc4060e60 con 0x7f1dc4060a80 2020-11-20T11:32:34.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.208+0000 7f1dc37fe700 1 --2- 172.21.15.23:0/3360442548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1dc4060a80 0x7f1dc4061950 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f1dc40672f0 tx=0x7f1db4006e30).ready entity=mon.0 client_cookie=8b92be3defc712a2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:34.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.208+0000 7f1dc27fc700 1 -- 172.21.15.23:0/3360442548 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db40097b0 con 0x7f1dc4060a80 2020-11-20T11:32:34.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.208+0000 7f1dc27fc700 1 -- 172.21.15.23:0/3360442548 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db4000e90 con 0x7f1dc4060a80 2020-11-20T11:32:34.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.208+0000 7f1dc27fc700 1 -- 172.21.15.23:0/3360442548 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db4009b20 con 0x7f1dc4060a80 2020-11-20T11:32:34.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.208+0000 7f1dc9d42700 1 -- 172.21.15.23:0/3360442548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1dc4060a80 msgr2=0x7f1dc4061950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:34.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.208+0000 7f1dc9d42700 1 --2- 172.21.15.23:0/3360442548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1dc4060a80 0x7f1dc4061950 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f1dc40672f0 tx=0x7f1db4006e30).stop 2020-11-20T11:32:34.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.208+0000 7f1dc9d42700 1 -- 172.21.15.23:0/3360442548 shutdown_connections 2020-11-20T11:32:34.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.208+0000 7f1dc9d42700 1 --2- 172.21.15.23:0/3360442548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1dc4060a80 0x7f1dc4061950 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:34.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.208+0000 7f1dc9d42700 1 --2- 172.21.15.23:0/3360442548 >> v2:172.21.15.155:3300/0 conn(0x7f1dc40a9d40 0x7f1dc40aa1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:34.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.208+0000 7f1dc9d42700 1 -- 172.21.15.23:0/3360442548 >> 172.21.15.23:0/3360442548 conn(0x7f1dc4066730 msgr2=0x7f1dc40601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:34.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.208+0000 7f1dc9d42700 1 -- 172.21.15.23:0/3360442548 shutdown_connections 2020-11-20T11:32:34.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.208+0000 7f1dc9d42700 1 -- 172.21.15.23:0/3360442548 wait complete. 2020-11-20T11:32:34.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.209+0000 7f1dc9d42700 1 Processor -- start 2020-11-20T11:32:34.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.209+0000 7f1dc9d42700 1 -- start start 2020-11-20T11:32:34.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.210+0000 7f1dc9d42700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1dc4060a80 0x7f1dc40a34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:34.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.210+0000 7f1dc9d42700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1dc40a9d40 0x7f1dc40a8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:34.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.210+0000 7f1dc9d42700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1dc4137920 0x7f1dc40a6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:34.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.210+0000 7f1dc9d42700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc40a78a0 con 0x7f1dc4137920 2020-11-20T11:32:34.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.210+0000 7f1dc9d42700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc40a7ad0 con 0x7f1dc4060a80 2020-11-20T11:32:34.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.210+0000 7f1dc9d42700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc40a7d70 con 0x7f1dc40a9d40 2020-11-20T11:32:34.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.210+0000 7f1dc3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1dc4137920 0x7f1dc40a6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:34.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.210+0000 7f1dc3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1dc4137920 0x7f1dc40a6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40978/0 (socket says 172.21.15.23:40978) 2020-11-20T11:32:34.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.210+0000 7f1dc3fff700 1 -- 172.21.15.23:0/4102125756 learned_addr learned my addr 172.21.15.23:0/4102125756 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:34.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.210+0000 7f1dc37fe700 1 --2- 172.21.15.23:0/4102125756 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1dc4060a80 0x7f1dc40a34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:34.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.210+0000 7f1dc2ffd700 1 --2- 172.21.15.23:0/4102125756 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1dc40a9d40 0x7f1dc40a8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:34.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.210+0000 7f1dc3fff700 1 -- 172.21.15.23:0/4102125756 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1dc40a9d40 msgr2=0x7f1dc40a8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:34.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.210+0000 7f1dc3fff700 1 --2- 172.21.15.23:0/4102125756 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1dc40a9d40 0x7f1dc40a8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:34.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.211+0000 7f1dc3fff700 1 -- 172.21.15.23:0/4102125756 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1dc4060a80 msgr2=0x7f1dc40a34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:34.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.211+0000 7f1dc3fff700 1 --2- 172.21.15.23:0/4102125756 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1dc4060a80 0x7f1dc40a34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:34.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.211+0000 7f1dc3fff700 1 -- 172.21.15.23:0/4102125756 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc40a7400 con 0x7f1dc4137920 2020-11-20T11:32:34.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.211+0000 7f1dc3fff700 1 --2- 172.21.15.23:0/4102125756 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1dc4137920 0x7f1dc40a6f10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f1dbc001460 tx=0x7f1dbc007e60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:34.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.211+0000 7f1da37fe700 1 -- 172.21.15.23:0/4102125756 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc005bc0 con 0x7f1dc4137920 2020-11-20T11:32:34.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.211+0000 7f1da37fe700 1 -- 172.21.15.23:0/4102125756 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dbc005d20 con 0x7f1dc4137920 2020-11-20T11:32:34.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.211+0000 7f1da37fe700 1 -- 172.21.15.23:0/4102125756 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc007320 con 0x7f1dc4137920 2020-11-20T11:32:34.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.211+0000 7f1dc9d42700 1 -- 172.21.15.23:0/4102125756 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dc413b740 con 0x7f1dc4137920 2020-11-20T11:32:34.225 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.211+0000 7f1dc9d42700 1 -- 172.21.15.23:0/4102125756 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dc414d830 con 0x7f1dc4137920 2020-11-20T11:32:34.225 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.212+0000 7f1dc9d42700 1 -- 172.21.15.23:0/4102125756 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dc40a7f30 con 0x7f1dc4137920 2020-11-20T11:32:34.225 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.212+0000 7f1da37fe700 1 -- 172.21.15.23:0/4102125756 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1dbc010030 con 0x7f1dc4137920 2020-11-20T11:32:34.225 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.212+0000 7f1da37fe700 1 --2- 172.21.15.23:0/4102125756 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1d9c056d90 0x7f1d9c059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:34.225 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.213+0000 7f1da37fe700 1 -- 172.21.15.23:0/4102125756 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f1dbc070030 con 0x7f1dc4137920 2020-11-20T11:32:34.226 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.213+0000 7f1dc37fe700 1 --2- 172.21.15.23:0/4102125756 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1d9c056d90 0x7f1d9c059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:34.226 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.213+0000 7f1dc37fe700 1 --2- 172.21.15.23:0/4102125756 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1d9c056d90 0x7f1d9c059250 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f1db4006e70 tx=0x7f1db4015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:34.226 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.216+0000 7f1da37fe700 1 -- 172.21.15.23:0/4102125756 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1dbc045320 con 0x7f1dc4137920 2020-11-20T11:32:34.367 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.364+0000 7f1dc9d42700 1 -- 172.21.15.23:0/4102125756 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f1dc4044710 con 0x7f1dc4137920 2020-11-20T11:32:34.368 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.365+0000 7f1da37fe700 1 -- 172.21.15.23:0/4102125756 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1dbc048940 con 0x7f1dc4137920 2020-11-20T11:32:34.368 INFO:teuthology.orchestra.run.smithi023.stdout:55834574854 2020-11-20T11:32:34.369 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.366+0000 7f1dc9d42700 1 -- 172.21.15.23:0/4102125756 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1d9c056d90 msgr2=0x7f1d9c059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:34.369 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.366+0000 7f1dc9d42700 1 --2- 172.21.15.23:0/4102125756 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1d9c056d90 0x7f1d9c059250 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f1db4006e70 tx=0x7f1db4015000).stop 2020-11-20T11:32:34.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.366+0000 7f1dc9d42700 1 -- 172.21.15.23:0/4102125756 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1dc4137920 msgr2=0x7f1dc40a6f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:34.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.366+0000 7f1dc9d42700 1 --2- 172.21.15.23:0/4102125756 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1dc4137920 0x7f1dc40a6f10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f1dbc001460 tx=0x7f1dbc007e60).stop 2020-11-20T11:32:34.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.366+0000 7f1dc9d42700 1 -- 172.21.15.23:0/4102125756 shutdown_connections 2020-11-20T11:32:34.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.366+0000 7f1dc9d42700 1 --2- 172.21.15.23:0/4102125756 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1dc4137920 0x7f1dc40a6f10 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:34.371 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.366+0000 7f1dc9d42700 1 --2- 172.21.15.23:0/4102125756 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1dc4060a80 0x7f1dc40a34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:34.371 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.366+0000 7f1dc9d42700 1 --2- 172.21.15.23:0/4102125756 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1d9c056d90 0x7f1d9c059250 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:34.371 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.366+0000 7f1dc9d42700 1 --2- 172.21.15.23:0/4102125756 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1dc40a9d40 0x7f1dc40a8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:34.371 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.366+0000 7f1dc9d42700 1 -- 172.21.15.23:0/4102125756 >> 172.21.15.23:0/4102125756 conn(0x7f1dc4066730 msgr2=0x7f1dc41369a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:34.371 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.366+0000 7f1dc9d42700 1 -- 172.21.15.23:0/4102125756 shutdown_connections 2020-11-20T11:32:34.372 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.366+0000 7f1dc9d42700 1 -- 172.21.15.23:0/4102125756 wait complete. 2020-11-20T11:32:34.380 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.0 2020-11-20T11:32:34.381 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2020-11-20T11:32:34.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.493+0000 7f2554883700 1 Processor -- start 2020-11-20T11:32:34.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.493+0000 7f2554883700 1 -- start start 2020-11-20T11:32:34.498 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.493+0000 7f2554883700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25500a9040 0x7f25500aaf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:34.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.493+0000 7f2554883700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f25500ace70 0x7f255013de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:34.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.493+0000 7f2554883700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f25500ad2f0 0x7f255013e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:34.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.493+0000 7f2554883700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25500aa8a0 con 0x7f25500ad2f0 2020-11-20T11:32:34.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.493+0000 7f2554883700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25500ad860 con 0x7f25500ace70 2020-11-20T11:32:34.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.493+0000 7f2554883700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25500aaa90 con 0x7f25500a9040 2020-11-20T11:32:34.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.493+0000 7f254ffff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f25500ad2f0 0x7f255013e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:34.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.494+0000 7f254ffff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f25500ad2f0 0x7f255013e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40986/0 (socket says 172.21.15.23:40986) 2020-11-20T11:32:34.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.494+0000 7f254ffff700 1 -- 172.21.15.23:0/3588409434 learned_addr learned my addr 172.21.15.23:0/3588409434 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:34.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.494+0000 7f254f7fe700 1 --2- 172.21.15.23:0/3588409434 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25500a9040 0x7f25500aaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:34.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.494+0000 7f254effd700 1 --2- 172.21.15.23:0/3588409434 >> v2:172.21.15.155:3300/0 conn(0x7f25500ace70 0x7f255013de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:34.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.494+0000 7f254ffff700 1 -- 172.21.15.23:0/3588409434 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25500a9040 msgr2=0x7f25500aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:34.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.494+0000 7f254ffff700 1 --2- 172.21.15.23:0/3588409434 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25500a9040 0x7f25500aaf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:34.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.494+0000 7f254ffff700 1 -- 172.21.15.23:0/3588409434 >> v2:172.21.15.155:3300/0 conn(0x7f25500ace70 msgr2=0x7f255013de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:34.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.494+0000 7f254ffff700 1 --2- 172.21.15.23:0/3588409434 >> v2:172.21.15.155:3300/0 conn(0x7f25500ace70 0x7f255013de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:34.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.494+0000 7f254ffff700 1 -- 172.21.15.23:0/3588409434 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25500a9420 con 0x7f25500ad2f0 2020-11-20T11:32:34.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.494+0000 7f254ffff700 1 --2- 172.21.15.23:0/3588409434 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25500ad2f0 0x7f255013e2f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f255005ca40 tx=0x7f25480092d0).ready entity=mon.0 client_cookie=f5ed13bbe7a4b6bd server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:34.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.494+0000 7f254e7fc700 1 -- 172.21.15.23:0/3588409434 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f254800bbf0 con 0x7f25500ad2f0 2020-11-20T11:32:34.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.494+0000 7f254e7fc700 1 -- 172.21.15.23:0/3588409434 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f254800bd50 con 0x7f25500ad2f0 2020-11-20T11:32:34.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.494+0000 7f254e7fc700 1 -- 172.21.15.23:0/3588409434 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2548011610 con 0x7f25500ad2f0 2020-11-20T11:32:34.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.495+0000 7f2554883700 1 -- 172.21.15.23:0/3588409434 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25500ad2f0 msgr2=0x7f255013e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:34.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.495+0000 7f2554883700 1 --2- 172.21.15.23:0/3588409434 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25500ad2f0 0x7f255013e2f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f255005ca40 tx=0x7f25480092d0).stop 2020-11-20T11:32:34.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.495+0000 7f2554883700 1 -- 172.21.15.23:0/3588409434 shutdown_connections 2020-11-20T11:32:34.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.495+0000 7f2554883700 1 --2- 172.21.15.23:0/3588409434 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25500ad2f0 0x7f255013e2f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:34.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.495+0000 7f2554883700 1 --2- 172.21.15.23:0/3588409434 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25500a9040 0x7f25500aaf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:34.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.495+0000 7f2554883700 1 --2- 172.21.15.23:0/3588409434 >> v2:172.21.15.155:3300/0 conn(0x7f25500ace70 0x7f255013de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:34.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.495+0000 7f2554883700 1 -- 172.21.15.23:0/3588409434 >> 172.21.15.23:0/3588409434 conn(0x7f25500683b0 msgr2=0x7f25500b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:34.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.495+0000 7f2554883700 1 -- 172.21.15.23:0/3588409434 shutdown_connections 2020-11-20T11:32:34.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.495+0000 7f2554883700 1 -- 172.21.15.23:0/3588409434 wait complete. 2020-11-20T11:32:34.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.496+0000 7f2554883700 1 Processor -- start 2020-11-20T11:32:34.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.496+0000 7f2554883700 1 -- start start 2020-11-20T11:32:34.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.496+0000 7f2554883700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25500a9040 0x7f25500a1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:34.508 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.496+0000 7f2554883700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25500ace70 0x7f2550140f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:34.508 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.496+0000 7f2554883700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25500ad2f0 0x7f255009dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:34.508 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.496+0000 7f2554883700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255005ccf0 con 0x7f25500ad2f0 2020-11-20T11:32:34.508 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.496+0000 7f2554883700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25500a1c20 con 0x7f25500a9040 2020-11-20T11:32:34.509 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.496+0000 7f2554883700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f255009d620 con 0x7f25500ace70 2020-11-20T11:32:34.509 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.497+0000 7f254ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25500ad2f0 0x7f255009dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:34.509 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.497+0000 7f254ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25500ad2f0 0x7f255009dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40992/0 (socket says 172.21.15.23:40992) 2020-11-20T11:32:34.509 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.497+0000 7f254ffff700 1 -- 172.21.15.23:0/3563410571 learned_addr learned my addr 172.21.15.23:0/3563410571 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:34.509 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.497+0000 7f254effd700 1 --2- 172.21.15.23:0/3563410571 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25500ace70 0x7f2550140f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:34.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.497+0000 7f254f7fe700 1 --2- 172.21.15.23:0/3563410571 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25500a9040 0x7f25500a1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:34.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.497+0000 7f254ffff700 1 -- 172.21.15.23:0/3563410571 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25500ace70 msgr2=0x7f2550140f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:34.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.497+0000 7f254ffff700 1 --2- 172.21.15.23:0/3563410571 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25500ace70 0x7f2550140f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:34.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.497+0000 7f254ffff700 1 -- 172.21.15.23:0/3563410571 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25500a9040 msgr2=0x7f25500a1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:34.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.497+0000 7f254ffff700 1 --2- 172.21.15.23:0/3563410571 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25500a9040 0x7f25500a1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:34.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.497+0000 7f254ffff700 1 -- 172.21.15.23:0/3563410571 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f255009f7c0 con 0x7f25500ad2f0 2020-11-20T11:32:34.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.497+0000 7f254ffff700 1 --2- 172.21.15.23:0/3563410571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25500ad2f0 0x7f255009dac0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f2548007fd0 tx=0x7f2548016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:34.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.497+0000 7f252f7fe700 1 -- 172.21.15.23:0/3563410571 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2548009ea0 con 0x7f25500ad2f0 2020-11-20T11:32:34.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.497+0000 7f252f7fe700 1 -- 172.21.15.23:0/3563410571 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f254800bec0 con 0x7f25500ad2f0 2020-11-20T11:32:34.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.497+0000 7f2554883700 1 -- 172.21.15.23:0/3563410571 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f255009f9b0 con 0x7f25500ad2f0 2020-11-20T11:32:34.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.497+0000 7f252f7fe700 1 -- 172.21.15.23:0/3563410571 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2548010cf0 con 0x7f25500ad2f0 2020-11-20T11:32:34.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.497+0000 7f2554883700 1 -- 172.21.15.23:0/3563410571 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f255009e170 con 0x7f25500ad2f0 2020-11-20T11:32:34.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.498+0000 7f2554883700 1 -- 172.21.15.23:0/3563410571 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f255009e360 con 0x7f25500ad2f0 2020-11-20T11:32:34.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.498+0000 7f252f7fe700 1 -- 172.21.15.23:0/3563410571 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2548017070 con 0x7f25500ad2f0 2020-11-20T11:32:34.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.499+0000 7f252f7fe700 1 --2- 172.21.15.23:0/3563410571 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2528056d60 0x7f2528059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:34.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.499+0000 7f252f7fe700 1 -- 172.21.15.23:0/3563410571 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f2548071cf0 con 0x7f25500ad2f0 2020-11-20T11:32:34.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.499+0000 7f254f7fe700 1 --2- 172.21.15.23:0/3563410571 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2528056d60 0x7f2528059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:34.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.499+0000 7f254f7fe700 1 --2- 172.21.15.23:0/3563410571 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2528056d60 0x7f2528059220 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f2544006fd0 tx=0x7f2544008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:34.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.503+0000 7f252f7fe700 1 -- 172.21.15.23:0/3563410571 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f255009e360 con 0x7f25500ad2f0 2020-11-20T11:32:34.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.652+0000 7f2554883700 1 -- 172.21.15.23:0/3563410571 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f255009e360 con 0x7f25500ad2f0 2020-11-20T11:32:34.656 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.653+0000 7f252f7fe700 1 -- 172.21.15.23:0/3563410571 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f255009e360 con 0x7f25500ad2f0 2020-11-20T11:32:34.656 INFO:teuthology.orchestra.run.smithi023.stdout:60129542149 2020-11-20T11:32:34.657 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.654+0000 7f2554883700 1 -- 172.21.15.23:0/3563410571 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2528056d60 msgr2=0x7f2528059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:34.657 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.654+0000 7f2554883700 1 --2- 172.21.15.23:0/3563410571 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2528056d60 0x7f2528059220 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f2544006fd0 tx=0x7f2544008040).stop 2020-11-20T11:32:34.657 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.654+0000 7f2554883700 1 -- 172.21.15.23:0/3563410571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25500ad2f0 msgr2=0x7f255009dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:34.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.654+0000 7f2554883700 1 --2- 172.21.15.23:0/3563410571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25500ad2f0 0x7f255009dac0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f2548007fd0 tx=0x7f2548016000).stop 2020-11-20T11:32:34.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.654+0000 7f2554883700 1 -- 172.21.15.23:0/3563410571 shutdown_connections 2020-11-20T11:32:34.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.654+0000 7f2554883700 1 --2- 172.21.15.23:0/3563410571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25500ad2f0 0x7f255009dac0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:34.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.654+0000 7f2554883700 1 --2- 172.21.15.23:0/3563410571 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f25500a9040 0x7f25500a1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:34.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.654+0000 7f2554883700 1 --2- 172.21.15.23:0/3563410571 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2528056d60 0x7f2528059220 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:34.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.654+0000 7f2554883700 1 --2- 172.21.15.23:0/3563410571 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25500ace70 0x7f2550140f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:34.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.654+0000 7f2554883700 1 -- 172.21.15.23:0/3563410571 >> 172.21.15.23:0/3563410571 conn(0x7f25500683b0 msgr2=0x7f25501392d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:34.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.654+0000 7f2554883700 1 -- 172.21.15.23:0/3563410571 shutdown_connections 2020-11-20T11:32:34.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.654+0000 7f2554883700 1 -- 172.21.15.23:0/3563410571 wait complete. 2020-11-20T11:32:34.668 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.1 2020-11-20T11:32:34.669 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2020-11-20T11:32:34.784 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.781+0000 7f0211546700 1 Processor -- start 2020-11-20T11:32:34.784 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.781+0000 7f0211546700 1 -- start start 2020-11-20T11:32:34.789 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.782+0000 7f0211546700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f020c0665f0 0x7f020c13adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:34.789 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.782+0000 7f0211546700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f020c13bd10 con 0x7f020c066210 2020-11-20T11:32:34.790 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.782+0000 7f0211546700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f020c0aa930 con 0x7f020c0ab2a0 2020-11-20T11:32:34.790 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.782+0000 7f0211546700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f020c0ab700 con 0x7f020c0665f0 2020-11-20T11:32:34.790 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.782+0000 7f020bfff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f020c066210 0x7f020c0a8ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55246/0 (socket says 172.21.15.23:55246) 2020-11-20T11:32:34.790 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.782+0000 7f020bfff700 1 -- 172.21.15.23:0/1086474604 learned_addr learned my addr 172.21.15.23:0/1086474604 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:34.791 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.782+0000 7f020b7fe700 1 --2- 172.21.15.23:0/1086474604 >> v2:172.21.15.155:3300/0 conn(0x7f020c0665f0 0x7f020c13adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:34.791 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.782+0000 7f020affd700 1 -- 172.21.15.23:0/1086474604 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3485882714 0 0) 0x7f020c13bd10 con 0x7f020c066210 2020-11-20T11:32:34.791 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.782+0000 7f020affd700 1 -- 172.21.15.23:0/1086474604 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01f8003280 con 0x7f020c066210 2020-11-20T11:32:34.791 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.782+0000 7f020affd700 1 -- 172.21.15.23:0/1086474604 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 660678058 0 0) 0x7f01f8003280 con 0x7f020c066210 2020-11-20T11:32:34.791 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.782+0000 7f020affd700 1 -- 172.21.15.23:0/1086474604 >> v2:172.21.15.155:3300/0 conn(0x7f020c0665f0 msgr2=0x7f020c13adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:34.792 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.782+0000 7f020affd700 1 --2- 172.21.15.23:0/1086474604 >> v2:172.21.15.155:3300/0 conn(0x7f020c0665f0 0x7f020c13adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:34.792 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.782+0000 7f020affd700 1 -- 172.21.15.23:0/1086474604 >> v1:172.21.15.155:6789/0 conn(0x7f020c0ab2a0 legacy=0x7f020c137f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:34.792 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.782+0000 7f020affd700 1 -- 172.21.15.23:0/1086474604 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f020c0a9700 con 0x7f020c066210 2020-11-20T11:32:34.792 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.782+0000 7f020affd700 1 -- 172.21.15.23:0/1086474604 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f0200001520 con 0x7f020c066210 2020-11-20T11:32:34.793 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.782+0000 7f020affd700 1 -- 172.21.15.23:0/1086474604 >> v1:172.21.15.23:6789/0 conn(0x7f020c066210 legacy=0x7f020c0a8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:34.793 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.782+0000 7f020affd700 1 --2- 172.21.15.23:0/1086474604 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f01f8003770 0x7f01f8003b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:34.793 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.782+0000 7f020affd700 1 --2- 172.21.15.23:0/1086474604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01f8004520 0x7f01f80049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:34.793 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.782+0000 7f020affd700 1 --2- 172.21.15.23:0/1086474604 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f01f8004e90 0x7f01f800b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:34.793 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.782+0000 7f020affd700 1 -- 172.21.15.23:0/1086474604 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020c0ab700 con 0x7f01f8004520 2020-11-20T11:32:34.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.782+0000 7f020affd700 1 -- 172.21.15.23:0/1086474604 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f80042a0 con 0x7f01f8004e90 2020-11-20T11:32:34.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.782+0000 7f020affd700 1 -- 172.21.15.23:0/1086474604 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01f800bc60 con 0x7f01f8003770 2020-11-20T11:32:34.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.782+0000 7f020b7fe700 1 --2- 172.21.15.23:0/1086474604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01f8004520 0x7f01f80049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:34.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.783+0000 7f0210d45700 1 --2- 172.21.15.23:0/1086474604 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f01f8004e90 0x7f01f800b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:34.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.783+0000 7f020bfff700 1 --2- 172.21.15.23:0/1086474604 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f01f8003770 0x7f01f8003b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:34.795 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.783+0000 7f020b7fe700 1 -- 172.21.15.23:0/1086474604 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f01f8003770 msgr2=0x7f01f8003b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:34.795 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.783+0000 7f020b7fe700 1 --2- 172.21.15.23:0/1086474604 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f01f8003770 0x7f01f8003b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:34.795 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.783+0000 7f020b7fe700 1 -- 172.21.15.23:0/1086474604 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f01f8004e90 msgr2=0x7f01f800b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:34.796 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.783+0000 7f020b7fe700 1 --2- 172.21.15.23:0/1086474604 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f01f8004e90 0x7f01f800b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:34.796 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.783+0000 7f0210d45700 1 -- 172.21.15.23:0/1086474604 reap_dead start 2020-11-20T11:32:34.796 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.783+0000 7f020b7fe700 1 -- 172.21.15.23:0/1086474604 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f8003280 con 0x7f01f8004520 2020-11-20T11:32:34.796 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.783+0000 7f020bfff700 1 --2- 172.21.15.23:0/1086474604 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f01f8003770 0x7f01f8003b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:34.797 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.783+0000 7f020b7fe700 1 --2- 172.21.15.23:0/1086474604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01f8004520 0x7f01f80049a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f01fc00bc40 tx=0x7f01fc006e30).ready entity=mon.0 client_cookie=2841cfb4d0501444 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:34.797 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.783+0000 7f020affd700 1 -- 172.21.15.23:0/1086474604 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01fc00b350 con 0x7f01f8004520 2020-11-20T11:32:34.797 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.783+0000 7f020affd700 1 -- 172.21.15.23:0/1086474604 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01fc00ea00 con 0x7f01f8004520 2020-11-20T11:32:34.797 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.783+0000 7f020affd700 1 -- 172.21.15.23:0/1086474604 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01fc00ed80 con 0x7f01f8004520 2020-11-20T11:32:34.797 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.783+0000 7f0211546700 1 -- 172.21.15.23:0/1086474604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01f8004520 msgr2=0x7f01f80049a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:34.798 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.783+0000 7f0211546700 1 --2- 172.21.15.23:0/1086474604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01f8004520 0x7f01f80049a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f01fc00bc40 tx=0x7f01fc006e30).stop 2020-11-20T11:32:34.798 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.783+0000 7f0211546700 1 -- 172.21.15.23:0/1086474604 shutdown_connections 2020-11-20T11:32:34.798 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.783+0000 7f0211546700 1 --2- 172.21.15.23:0/1086474604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01f8004520 0x7f01f80049a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:34.798 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.783+0000 7f0211546700 1 -- 172.21.15.23:0/1086474604 >> 172.21.15.23:0/1086474604 conn(0x7f020c0612c0 msgr2=0x7f020c0616d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:34.798 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.783+0000 7f0211546700 1 -- 172.21.15.23:0/1086474604 shutdown_connections 2020-11-20T11:32:34.799 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.783+0000 7f0211546700 1 -- 172.21.15.23:0/1086474604 wait complete. 2020-11-20T11:32:34.799 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.784+0000 7f0211546700 1 Processor -- start 2020-11-20T11:32:34.799 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.785+0000 7f0211546700 1 -- start start 2020-11-20T11:32:34.799 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.785+0000 7f0211546700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f01f8004520 0x7f020c0a38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:34.799 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.785+0000 7f0211546700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f020c0a2db0 0x7f020c14dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:34.800 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.785+0000 7f0211546700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f020c13c6f0 0x7f020c133cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:34.800 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.785+0000 7f0211546700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020c0a3dc0 con 0x7f020c0a2db0 2020-11-20T11:32:34.800 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.785+0000 7f0211546700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020c0a23d0 con 0x7f020c13c6f0 2020-11-20T11:32:34.800 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.785+0000 7f0211546700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f020c0a2670 con 0x7f01f8004520 2020-11-20T11:32:34.800 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.785+0000 7f020b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f020c0a2db0 0x7f020c14dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:34.801 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.785+0000 7f020b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f020c0a2db0 0x7f020c14dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41010/0 (socket says 172.21.15.23:41010) 2020-11-20T11:32:34.801 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.785+0000 7f020b7fe700 1 -- 172.21.15.23:0/3508272378 learned_addr learned my addr 172.21.15.23:0/3508272378 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:34.801 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.786+0000 7f020bfff700 1 --2- 172.21.15.23:0/3508272378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f01f8004520 0x7f020c0a38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:34.801 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.786+0000 7f0210d45700 1 --2- 172.21.15.23:0/3508272378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f020c13c6f0 0x7f020c133cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:34.802 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.786+0000 7f020b7fe700 1 -- 172.21.15.23:0/3508272378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f01f8004520 msgr2=0x7f020c0a38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:34.802 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.786+0000 7f020b7fe700 1 --2- 172.21.15.23:0/3508272378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f01f8004520 0x7f020c0a38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:34.802 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.786+0000 7f020b7fe700 1 -- 172.21.15.23:0/3508272378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f020c13c6f0 msgr2=0x7f020c133cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:34.802 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.786+0000 7f020b7fe700 1 --2- 172.21.15.23:0/3508272378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f020c13c6f0 0x7f020c133cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:34.802 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.786+0000 7f020b7fe700 1 -- 172.21.15.23:0/3508272378 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f020c0a2910 con 0x7f020c0a2db0 2020-11-20T11:32:34.803 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.786+0000 7f0210d45700 1 --2- 172.21.15.23:0/3508272378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f020c13c6f0 0x7f020c133cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:32:34.803 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.786+0000 7f020b7fe700 1 --2- 172.21.15.23:0/3508272378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f020c0a2db0 0x7f020c14dad0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f01fc00f010 tx=0x7f01fc019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:34.803 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.786+0000 7f01ebfff700 1 -- 172.21.15.23:0/3508272378 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01fc012e20 con 0x7f020c0a2db0 2020-11-20T11:32:34.803 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.786+0000 7f01ebfff700 1 -- 172.21.15.23:0/3508272378 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01fc000e90 con 0x7f020c0a2db0 2020-11-20T11:32:34.804 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.786+0000 7f01ebfff700 1 -- 172.21.15.23:0/3508272378 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01fc024640 con 0x7f020c0a2db0 2020-11-20T11:32:34.804 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.786+0000 7f0211546700 1 -- 172.21.15.23:0/3508272378 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f020c14dfc0 con 0x7f020c0a2db0 2020-11-20T11:32:34.804 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.786+0000 7f0211546700 1 -- 172.21.15.23:0/3508272378 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f020c13cb50 con 0x7f020c0a2db0 2020-11-20T11:32:34.804 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.787+0000 7f0211546700 1 -- 172.21.15.23:0/3508272378 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f020c061a30 con 0x7f020c0a2db0 2020-11-20T11:32:34.805 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.787+0000 7f01ebfff700 1 -- 172.21.15.23:0/3508272378 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f01fc02c3f0 con 0x7f020c0a2db0 2020-11-20T11:32:34.805 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.787+0000 7f01ebfff700 1 --2- 172.21.15.23:0/3508272378 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f01e4056d90 0x7f01e4059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:34.806 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.787+0000 7f01ebfff700 1 -- 172.21.15.23:0/3508272378 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f01fc0715e0 con 0x7f020c0a2db0 2020-11-20T11:32:34.806 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.788+0000 7f020bfff700 1 --2- 172.21.15.23:0/3508272378 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f01e4056d90 0x7f01e4059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:34.806 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.788+0000 7f020bfff700 1 --2- 172.21.15.23:0/3508272378 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f01e4056d90 0x7f01e4059250 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f020c045440 tx=0x7f0200009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:34.806 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.791+0000 7f01ebfff700 1 -- 172.21.15.23:0/3508272378 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f01fc046960 con 0x7f020c0a2db0 2020-11-20T11:32:34.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.941+0000 7f0211546700 1 -- 172.21.15.23:0/3508272378 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f020c062760 con 0x7f020c0a2db0 2020-11-20T11:32:34.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.941+0000 7f01ebfff700 1 -- 172.21.15.23:0/3508272378 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f01fc028090 con 0x7f020c0a2db0 2020-11-20T11:32:34.944 INFO:teuthology.orchestra.run.smithi023.stdout:60129542150 2020-11-20T11:32:34.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.943+0000 7f0211546700 1 -- 172.21.15.23:0/3508272378 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f01e4056d90 msgr2=0x7f01e4059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:34.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.943+0000 7f0211546700 1 --2- 172.21.15.23:0/3508272378 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f01e4056d90 0x7f01e4059250 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f020c045440 tx=0x7f0200009040).stop 2020-11-20T11:32:34.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.943+0000 7f0211546700 1 -- 172.21.15.23:0/3508272378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f020c0a2db0 msgr2=0x7f020c14dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:34.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.943+0000 7f0211546700 1 --2- 172.21.15.23:0/3508272378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f020c0a2db0 0x7f020c14dad0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f01fc00f010 tx=0x7f01fc019000).stop 2020-11-20T11:32:34.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.943+0000 7f0211546700 1 -- 172.21.15.23:0/3508272378 shutdown_connections 2020-11-20T11:32:34.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.943+0000 7f0211546700 1 --2- 172.21.15.23:0/3508272378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f020c13c6f0 0x7f020c133cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:34.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.943+0000 7f0211546700 1 --2- 172.21.15.23:0/3508272378 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f01e4056d90 0x7f01e4059250 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:34.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.943+0000 7f0211546700 1 --2- 172.21.15.23:0/3508272378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f01f8004520 0x7f020c0a38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:34.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.943+0000 7f0211546700 1 --2- 172.21.15.23:0/3508272378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f020c0a2db0 0x7f020c14dad0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:34.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.943+0000 7f0211546700 1 -- 172.21.15.23:0/3508272378 >> 172.21.15.23:0/3508272378 conn(0x7f020c0612c0 msgr2=0x7f020c0abca0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:34.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.943+0000 7f0211546700 1 -- 172.21.15.23:0/3508272378 shutdown_connections 2020-11-20T11:32:34.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:34.943+0000 7f0211546700 1 -- 172.21.15.23:0/3508272378 wait complete. 2020-11-20T11:32:34.957 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.2 2020-11-20T11:32:34.957 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2020-11-20T11:32:35.072 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.069+0000 7f821f7a9700 1 Processor -- start 2020-11-20T11:32:35.072 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.069+0000 7f821f7a9700 1 -- start start 2020-11-20T11:32:35.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.070+0000 7f821f7a9700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8218061380 0x7f82180aaac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:35.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.070+0000 7f821f7a9700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82180a9dd0 0x7f82180a3a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:35.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.070+0000 7f821f7a9700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82180aafb0 con 0x7f82180a3f80 2020-11-20T11:32:35.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.070+0000 7f821f7a9700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82180a3050 con 0x7f8218061380 2020-11-20T11:32:35.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.070+0000 7f821f7a9700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82180a32f0 con 0x7f82180a9dd0 2020-11-20T11:32:35.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.070+0000 7f821d545700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8218061380 0x7f82180aaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:35.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.070+0000 7f821dd46700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f82180a3f80 0x7f821813e3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55270/0 (socket says 172.21.15.23:55270) 2020-11-20T11:32:35.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.070+0000 7f821dd46700 1 -- 172.21.15.23:0/4232064500 learned_addr learned my addr 172.21.15.23:0/4232064500 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:35.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.070+0000 7f821d545700 1 --2- 172.21.15.23:0/4232064500 >> v2:172.21.15.23:3300/0 conn(0x7f8218061380 0x7f82180aaac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41016/0 (socket says 172.21.15.23:41016) 2020-11-20T11:32:35.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.070+0000 7f821cd44700 1 --2- 172.21.15.23:0/4232064500 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82180a9dd0 0x7f82180a3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:35.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.070+0000 7f821d545700 1 -- 172.21.15.23:0/4232064500 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82180a9dd0 msgr2=0x7f82180a3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:35.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.070+0000 7f821d545700 1 --2- 172.21.15.23:0/4232064500 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82180a9dd0 0x7f82180a3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:35.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.070+0000 7f821d545700 1 -- 172.21.15.23:0/4232064500 >> v1:172.21.15.23:6789/0 conn(0x7f82180a3f80 legacy=0x7f821813e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:35.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.070+0000 7f821d545700 1 -- 172.21.15.23:0/4232064500 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82180a3590 con 0x7f8218061380 2020-11-20T11:32:35.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.070+0000 7f8207fff700 1 -- 172.21.15.23:0/4232064500 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1087166309 0 0) 0x7f82180aafb0 con 0x7f82180a3f80 2020-11-20T11:32:35.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.070+0000 7f821d545700 1 --2- 172.21.15.23:0/4232064500 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8218061380 0x7f82180aaac0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f8214006fd0 tx=0x7f8214006e30).ready entity=mon.0 client_cookie=46fbfa4ff5ee9a89 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:35.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.070+0000 7f8207fff700 1 -- 172.21.15.23:0/4232064500 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82140097e0 con 0x7f8218061380 2020-11-20T11:32:35.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.070+0000 7f8207fff700 1 -- 172.21.15.23:0/4232064500 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8214000e90 con 0x7f8218061380 2020-11-20T11:32:35.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.070+0000 7f8207fff700 1 -- 172.21.15.23:0/4232064500 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8214009b50 con 0x7f8218061380 2020-11-20T11:32:35.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.071+0000 7f821f7a9700 1 -- 172.21.15.23:0/4232064500 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8218061380 msgr2=0x7f82180aaac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:35.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.071+0000 7f821f7a9700 1 --2- 172.21.15.23:0/4232064500 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8218061380 0x7f82180aaac0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f8214006fd0 tx=0x7f8214006e30).stop 2020-11-20T11:32:35.079 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.071+0000 7f821f7a9700 1 -- 172.21.15.23:0/4232064500 shutdown_connections 2020-11-20T11:32:35.079 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.071+0000 7f821f7a9700 1 --2- 172.21.15.23:0/4232064500 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8218061380 0x7f82180aaac0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:35.079 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.071+0000 7f821f7a9700 1 --2- 172.21.15.23:0/4232064500 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82180a9dd0 0x7f82180a3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:35.079 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.071+0000 7f821f7a9700 1 -- 172.21.15.23:0/4232064500 >> 172.21.15.23:0/4232064500 conn(0x7f82180ae280 msgr2=0x7f82180aa290 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:35.079 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.071+0000 7f821f7a9700 1 -- 172.21.15.23:0/4232064500 shutdown_connections 2020-11-20T11:32:35.080 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.071+0000 7f821f7a9700 1 -- 172.21.15.23:0/4232064500 wait complete. 2020-11-20T11:32:35.082 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.072+0000 7f821f7a9700 1 Processor -- start 2020-11-20T11:32:35.082 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.072+0000 7f821f7a9700 1 -- start start 2020-11-20T11:32:35.082 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.073+0000 7f821f7a9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8218061380 0x7f8218067320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:35.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.073+0000 7f821f7a9700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82180a3f80 0x7f821814a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:35.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.073+0000 7f821f7a9700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82180a9dd0 0x7f82180a67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:35.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.073+0000 7f821f7a9700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821814abf0 con 0x7f8218061380 2020-11-20T11:32:35.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.073+0000 7f821f7a9700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821814ae90 con 0x7f82180a9dd0 2020-11-20T11:32:35.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.073+0000 7f821f7a9700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f821814b130 con 0x7f82180a3f80 2020-11-20T11:32:35.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.073+0000 7f821d545700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8218061380 0x7f8218067320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:35.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.073+0000 7f821d545700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8218061380 0x7f8218067320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41022/0 (socket says 172.21.15.23:41022) 2020-11-20T11:32:35.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.073+0000 7f821d545700 1 -- 172.21.15.23:0/59742183 learned_addr learned my addr 172.21.15.23:0/59742183 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:35.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.073+0000 7f821cd44700 1 --2- 172.21.15.23:0/59742183 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82180a3f80 0x7f821814a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:35.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.073+0000 7f821dd46700 1 --2- 172.21.15.23:0/59742183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82180a9dd0 0x7f82180a67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:35.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.073+0000 7f821d545700 1 -- 172.21.15.23:0/59742183 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82180a3f80 msgr2=0x7f821814a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:35.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.073+0000 7f821d545700 1 --2- 172.21.15.23:0/59742183 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82180a3f80 0x7f821814a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:35.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.073+0000 7f821d545700 1 -- 172.21.15.23:0/59742183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82180a9dd0 msgr2=0x7f82180a67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:35.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.073+0000 7f821d545700 1 --2- 172.21.15.23:0/59742183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82180a9dd0 0x7f82180a67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:35.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.073+0000 7f821d545700 1 -- 172.21.15.23:0/59742183 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f821814a7d0 con 0x7f8218061380 2020-11-20T11:32:35.086 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.073+0000 7f821d545700 1 --2- 172.21.15.23:0/59742183 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8218061380 0x7f8218067320 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f821400b190 tx=0x7f8214015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:35.086 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.073+0000 7f82057fa700 1 -- 172.21.15.23:0/59742183 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8214011b00 con 0x7f8218061380 2020-11-20T11:32:35.086 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.073+0000 7f82057fa700 1 -- 172.21.15.23:0/59742183 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8214011c60 con 0x7f8218061380 2020-11-20T11:32:35.086 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.073+0000 7f82057fa700 1 -- 172.21.15.23:0/59742183 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f821401e460 con 0x7f8218061380 2020-11-20T11:32:35.086 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.073+0000 7f821f7a9700 1 -- 172.21.15.23:0/59742183 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f821814a9c0 con 0x7f8218061380 2020-11-20T11:32:35.087 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.074+0000 7f821f7a9700 1 -- 172.21.15.23:0/59742183 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82180a63a0 con 0x7f8218061380 2020-11-20T11:32:35.087 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.074+0000 7f821f7a9700 1 -- 172.21.15.23:0/59742183 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82180a1ab0 con 0x7f8218061380 2020-11-20T11:32:35.087 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.075+0000 7f82057fa700 1 -- 172.21.15.23:0/59742183 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8214025030 con 0x7f8218061380 2020-11-20T11:32:35.087 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.075+0000 7f82057fa700 1 --2- 172.21.15.23:0/59742183 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f81f0056d60 0x7f81f0059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:35.087 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.075+0000 7f82057fa700 1 -- 172.21.15.23:0/59742183 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f82140714c0 con 0x7f8218061380 2020-11-20T11:32:35.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.075+0000 7f821cd44700 1 --2- 172.21.15.23:0/59742183 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f81f0056d60 0x7f81f0059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:35.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.076+0000 7f821cd44700 1 --2- 172.21.15.23:0/59742183 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f81f0056d60 0x7f81f0059220 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f82180a6570 tx=0x7f8208006e30).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:35.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.079+0000 7f82057fa700 1 -- 172.21.15.23:0/59742183 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f82140468a0 con 0x7f8218061380 2020-11-20T11:32:35.232 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.229+0000 7f821f7a9700 1 -- 172.21.15.23:0/59742183 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f8218061e10 con 0x7f8218061380 2020-11-20T11:32:35.232 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.229+0000 7f82057fa700 1 -- 172.21.15.23:0/59742183 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8214023020 con 0x7f8218061380 2020-11-20T11:32:35.233 INFO:teuthology.orchestra.run.smithi023.stdout:60129542151 2020-11-20T11:32:35.234 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.230+0000 7f821f7a9700 1 -- 172.21.15.23:0/59742183 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f81f0056d60 msgr2=0x7f81f0059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:35.234 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.230+0000 7f821f7a9700 1 --2- 172.21.15.23:0/59742183 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f81f0056d60 0x7f81f0059220 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f82180a6570 tx=0x7f8208006e30).stop 2020-11-20T11:32:35.234 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.230+0000 7f821f7a9700 1 -- 172.21.15.23:0/59742183 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8218061380 msgr2=0x7f8218067320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:35.234 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.230+0000 7f821f7a9700 1 --2- 172.21.15.23:0/59742183 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8218061380 0x7f8218067320 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f821400b190 tx=0x7f8214015000).stop 2020-11-20T11:32:35.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.231+0000 7f821f7a9700 1 -- 172.21.15.23:0/59742183 shutdown_connections 2020-11-20T11:32:35.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.231+0000 7f821f7a9700 1 --2- 172.21.15.23:0/59742183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82180a9dd0 0x7f82180a67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:35.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.231+0000 7f821f7a9700 1 --2- 172.21.15.23:0/59742183 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8218061380 0x7f8218067320 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:35.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.231+0000 7f821f7a9700 1 --2- 172.21.15.23:0/59742183 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f81f0056d60 0x7f81f0059220 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:35.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.231+0000 7f821f7a9700 1 --2- 172.21.15.23:0/59742183 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82180a3f80 0x7f821814a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:35.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.231+0000 7f821f7a9700 1 -- 172.21.15.23:0/59742183 >> 172.21.15.23:0/59742183 conn(0x7f82180ae280 msgr2=0x7f8218135db0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:35.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.231+0000 7f821f7a9700 1 -- 172.21.15.23:0/59742183 shutdown_connections 2020-11-20T11:32:35.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.231+0000 7f821f7a9700 1 -- 172.21.15.23:0/59742183 wait complete. 2020-11-20T11:32:35.245 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.3 2020-11-20T11:32:35.246 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2020-11-20T11:32:35.378 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.374+0000 7f8d29ba1700 1 Processor -- start 2020-11-20T11:32:35.379 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.374+0000 7f8d29ba1700 1 -- start start 2020-11-20T11:32:35.379 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.374+0000 7f8d29ba1700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8d240a2460 0x7f8d24137df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:35.379 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.374+0000 7f8d29ba1700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d240aaab0 con 0x7f8d24067a10 2020-11-20T11:32:35.379 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.374+0000 7f8d29ba1700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d240aad50 con 0x7f8d240a2080 2020-11-20T11:32:35.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.374+0000 7f8d29ba1700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d2413aa50 con 0x7f8d240a2460 2020-11-20T11:32:35.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.375+0000 7f8d23fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8d240a2460 0x7f8d24137df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:35.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.375+0000 7f8d293a0700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f8d24067a10 0x7f8d241453c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55284/0 (socket says 172.21.15.23:55284) 2020-11-20T11:32:35.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.375+0000 7f8d293a0700 1 -- 172.21.15.23:0/2165442917 learned_addr learned my addr 172.21.15.23:0/2165442917 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:35.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.375+0000 7f8d23fff700 1 -- 172.21.15.23:0/2165442917 >> v1:172.21.15.155:6789/0 conn(0x7f8d240a2080 legacy=0x7f8d240aef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:35.382 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.375+0000 7f8d23fff700 1 -- 172.21.15.23:0/2165442917 >> v1:172.21.15.23:6789/0 conn(0x7f8d24067a10 legacy=0x7f8d241453c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:35.382 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.375+0000 7f8d23fff700 1 -- 172.21.15.23:0/2165442917 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2413b920 con 0x7f8d240a2460 2020-11-20T11:32:35.383 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.375+0000 7f8d237fe700 1 -- 172.21.15.23:0/2165442917 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1725724232 0 0) 0x7f8d240aaab0 con 0x7f8d24067a10 2020-11-20T11:32:35.384 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.375+0000 7f8d23fff700 1 --2- 172.21.15.23:0/2165442917 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d240a2460 0x7f8d24137df0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f8d10006fd0 tx=0x7f8d10006e30).ready entity=mon.0 client_cookie=d01bee75731c746c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:35.384 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.375+0000 7f8d237fe700 1 -- 172.21.15.23:0/2165442917 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d100097e0 con 0x7f8d240a2460 2020-11-20T11:32:35.385 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.375+0000 7f8d237fe700 1 -- 172.21.15.23:0/2165442917 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d10000e90 con 0x7f8d240a2460 2020-11-20T11:32:35.385 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.375+0000 7f8d237fe700 1 -- 172.21.15.23:0/2165442917 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d10009b50 con 0x7f8d240a2460 2020-11-20T11:32:35.386 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.376+0000 7f8d29ba1700 1 -- 172.21.15.23:0/2165442917 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d240a2460 msgr2=0x7f8d24137df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:35.387 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.376+0000 7f8d29ba1700 1 --2- 172.21.15.23:0/2165442917 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d240a2460 0x7f8d24137df0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f8d10006fd0 tx=0x7f8d10006e30).stop 2020-11-20T11:32:35.387 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.376+0000 7f8d29ba1700 1 -- 172.21.15.23:0/2165442917 shutdown_connections 2020-11-20T11:32:35.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.376+0000 7f8d29ba1700 1 --2- 172.21.15.23:0/2165442917 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d240a2460 0x7f8d24137df0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:35.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.376+0000 7f8d29ba1700 1 -- 172.21.15.23:0/2165442917 >> 172.21.15.23:0/2165442917 conn(0x7f8d24063980 msgr2=0x7f8d24063d90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:35.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.376+0000 7f8d29ba1700 1 -- 172.21.15.23:0/2165442917 shutdown_connections 2020-11-20T11:32:35.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.376+0000 7f8d29ba1700 1 -- 172.21.15.23:0/2165442917 wait complete. 2020-11-20T11:32:35.395 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.377+0000 7f8d29ba1700 1 Processor -- start 2020-11-20T11:32:35.395 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.377+0000 7f8d29ba1700 1 -- start start 2020-11-20T11:32:35.395 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.377+0000 7f8d29ba1700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8d24067a10 0x7f8d240adf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:35.395 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.377+0000 7f8d29ba1700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d240a2080 0x7f8d2413e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:35.396 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.377+0000 7f8d29ba1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d240a2460 0x7f8d240a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:35.396 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.377+0000 7f8d29ba1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2413d5d0 con 0x7f8d240a2460 2020-11-20T11:32:35.396 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.377+0000 7f8d29ba1700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2413d870 con 0x7f8d240a2080 2020-11-20T11:32:35.396 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.377+0000 7f8d29ba1700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2413db10 con 0x7f8d24067a10 2020-11-20T11:32:35.396 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.378+0000 7f8d293a0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d240a2460 0x7f8d240a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:35.397 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.378+0000 7f8d293a0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d240a2460 0x7f8d240a5590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41040/0 (socket says 172.21.15.23:41040) 2020-11-20T11:32:35.397 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.378+0000 7f8d293a0700 1 -- 172.21.15.23:0/3566235398 learned_addr learned my addr 172.21.15.23:0/3566235398 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:35.397 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.378+0000 7f8d23fff700 1 --2- 172.21.15.23:0/3566235398 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d240a2080 0x7f8d2413e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:35.397 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.378+0000 7f8d28b9f700 1 --2- 172.21.15.23:0/3566235398 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8d24067a10 0x7f8d240adf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:35.398 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.378+0000 7f8d293a0700 1 -- 172.21.15.23:0/3566235398 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8d24067a10 msgr2=0x7f8d240adf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:35.398 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.378+0000 7f8d293a0700 1 --2- 172.21.15.23:0/3566235398 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8d24067a10 0x7f8d240adf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:35.398 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.378+0000 7f8d293a0700 1 -- 172.21.15.23:0/3566235398 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d240a2080 msgr2=0x7f8d2413e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:35.398 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.378+0000 7f8d293a0700 1 --2- 172.21.15.23:0/3566235398 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d240a2080 0x7f8d2413e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:35.398 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.378+0000 7f8d293a0700 1 -- 172.21.15.23:0/3566235398 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2413d2c0 con 0x7f8d240a2460 2020-11-20T11:32:35.399 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.378+0000 7f8d293a0700 1 --2- 172.21.15.23:0/3566235398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d240a2460 0x7f8d240a5590 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f8d1c00cab0 tx=0x7f8d1c007dc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:35.399 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.378+0000 7f8d20ff9700 1 -- 172.21.15.23:0/3566235398 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c005c90 con 0x7f8d240a2460 2020-11-20T11:32:35.399 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.378+0000 7f8d20ff9700 1 -- 172.21.15.23:0/3566235398 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d1c005df0 con 0x7f8d240a2460 2020-11-20T11:32:35.399 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.378+0000 7f8d29ba1700 1 -- 172.21.15.23:0/3566235398 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d2413ec10 con 0x7f8d240a2460 2020-11-20T11:32:35.399 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.378+0000 7f8d20ff9700 1 -- 172.21.15.23:0/3566235398 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c007360 con 0x7f8d240a2460 2020-11-20T11:32:35.399 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.378+0000 7f8d29ba1700 1 -- 172.21.15.23:0/3566235398 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d240a5c40 con 0x7f8d240a2460 2020-11-20T11:32:35.400 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.379+0000 7f8d29ba1700 1 -- 172.21.15.23:0/3566235398 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d240aa760 con 0x7f8d240a2460 2020-11-20T11:32:35.400 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.379+0000 7f8d20ff9700 1 -- 172.21.15.23:0/3566235398 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8d1c010030 con 0x7f8d240a2460 2020-11-20T11:32:35.400 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.380+0000 7f8d20ff9700 1 --2- 172.21.15.23:0/3566235398 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8cfc056d60 0x7f8cfc059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:35.400 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.380+0000 7f8d20ff9700 1 -- 172.21.15.23:0/3566235398 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f8d1c070150 con 0x7f8d240a2460 2020-11-20T11:32:35.400 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.380+0000 7f8d28b9f700 1 --2- 172.21.15.23:0/3566235398 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8cfc056d60 0x7f8cfc059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:35.402 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.380+0000 7f8d28b9f700 1 --2- 172.21.15.23:0/3566235398 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8cfc056d60 0x7f8cfc059220 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f8d180035e0 tx=0x7f8d18009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:35.403 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.383+0000 7f8d20ff9700 1 -- 172.21.15.23:0/3566235398 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8d1c0453e0 con 0x7f8d240a2460 2020-11-20T11:32:35.546 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.532+0000 7f8d29ba1700 1 -- 172.21.15.23:0/3566235398 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f8d2413dcd0 con 0x7f8d240a2460 2020-11-20T11:32:35.547 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.532+0000 7f8d20ff9700 1 -- 172.21.15.23:0/3566235398 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8d1c048a00 con 0x7f8d240a2460 2020-11-20T11:32:35.547 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.533+0000 7f8d29ba1700 1 -- 172.21.15.23:0/3566235398 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8cfc056d60 msgr2=0x7f8cfc059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:35.547 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.533+0000 7f8d29ba1700 1 --2- 172.21.15.23:0/3566235398 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8cfc056d60 0x7f8cfc059220 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f8d180035e0 tx=0x7f8d18009040).stop 2020-11-20T11:32:35.547 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.533+0000 7f8d29ba1700 1 -- 172.21.15.23:0/3566235398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d240a2460 msgr2=0x7f8d240a5590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:35.547 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.534+0000 7f8d29ba1700 1 --2- 172.21.15.23:0/3566235398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d240a2460 0x7f8d240a5590 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f8d1c00cab0 tx=0x7f8d1c007dc0).stop 2020-11-20T11:32:35.548 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.534+0000 7f8d29ba1700 1 -- 172.21.15.23:0/3566235398 shutdown_connections 2020-11-20T11:32:35.548 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.534+0000 7f8d29ba1700 1 --2- 172.21.15.23:0/3566235398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d240a2460 0x7f8d240a5590 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:35.548 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.534+0000 7f8d29ba1700 1 --2- 172.21.15.23:0/3566235398 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8cfc056d60 0x7f8cfc059220 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:35.549 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.534+0000 7f8d29ba1700 1 --2- 172.21.15.23:0/3566235398 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8d24067a10 0x7f8d240adf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:35.549 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.534+0000 7f8d29ba1700 1 --2- 172.21.15.23:0/3566235398 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d240a2080 0x7f8d2413e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:35.549 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.534+0000 7f8d29ba1700 1 -- 172.21.15.23:0/3566235398 >> 172.21.15.23:0/3566235398 conn(0x7f8d24063980 msgr2=0x7f8d24067e40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:35.550 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.534+0000 7f8d29ba1700 1 -- 172.21.15.23:0/3566235398 shutdown_connections 2020-11-20T11:32:35.550 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.534+0000 7f8d29ba1700 1 -- 172.21.15.23:0/3566235398 wait complete. 2020-11-20T11:32:35.551 INFO:teuthology.orchestra.run.smithi023.stdout:55834574854 2020-11-20T11:32:35.552 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.4 2020-11-20T11:32:35.553 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2020-11-20T11:32:35.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.668+0000 7fcc64bfc700 1 Processor -- start 2020-11-20T11:32:35.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.668+0000 7fcc64bfc700 1 -- start start 2020-11-20T11:32:35.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.669+0000 7fcc64bfc700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fcc60060a80 0x7fcc60061950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:35.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.669+0000 7fcc64bfc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fcc600a9d40 0x7fcc600aa1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:35.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.669+0000 7fcc64bfc700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc60067bc0 con 0x7fcc60137920 2020-11-20T11:32:35.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.669+0000 7fcc64bfc700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc60067e60 con 0x7fcc600a9d40 2020-11-20T11:32:35.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.669+0000 7fcc64bfc700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc60068100 con 0x7fcc60060a80 2020-11-20T11:32:35.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.669+0000 7fcc5dd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fcc600a9d40 0x7fcc600aa1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:35.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.669+0000 7fcc5ed9d700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fcc60137920 0x7fcc60137d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55298/0 (socket says 172.21.15.23:55298) 2020-11-20T11:32:35.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.669+0000 7fcc5ed9d700 1 -- 172.21.15.23:0/3473498979 learned_addr learned my addr 172.21.15.23:0/3473498979 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:35.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.669+0000 7fcc5e59c700 1 --2- 172.21.15.23:0/3473498979 >> v2:172.21.15.155:3300/0 conn(0x7fcc60060a80 0x7fcc60061950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:35.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.669+0000 7fcc5dd9b700 1 -- 172.21.15.23:0/3473498979 >> v2:172.21.15.155:3300/0 conn(0x7fcc60060a80 msgr2=0x7fcc60061950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:35.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.669+0000 7fcc5dd9b700 1 --2- 172.21.15.23:0/3473498979 >> v2:172.21.15.155:3300/0 conn(0x7fcc60060a80 0x7fcc60061950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:35.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.669+0000 7fcc5dd9b700 1 -- 172.21.15.23:0/3473498979 >> v1:172.21.15.23:6789/0 conn(0x7fcc60137920 legacy=0x7fcc60137d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:35.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.669+0000 7fcc5dd9b700 1 -- 172.21.15.23:0/3473498979 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc60060e60 con 0x7fcc600a9d40 2020-11-20T11:32:35.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.669+0000 7fcc5d59a700 1 -- 172.21.15.23:0/3473498979 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2555153429 0 0) 0x7fcc60067bc0 con 0x7fcc60137920 2020-11-20T11:32:35.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.669+0000 7fcc5dd9b700 1 --2- 172.21.15.23:0/3473498979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc600a9d40 0x7fcc600aa1c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fcc600b01e0 tx=0x7fcc50006e30).ready entity=mon.0 client_cookie=1de4433572602e47 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:35.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.670+0000 7fcc5d59a700 1 -- 172.21.15.23:0/3473498979 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc500097b0 con 0x7fcc600a9d40 2020-11-20T11:32:35.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.670+0000 7fcc5d59a700 1 -- 172.21.15.23:0/3473498979 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc50000e90 con 0x7fcc600a9d40 2020-11-20T11:32:35.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.670+0000 7fcc5d59a700 1 -- 172.21.15.23:0/3473498979 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc50009b20 con 0x7fcc600a9d40 2020-11-20T11:32:35.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.670+0000 7fcc64bfc700 1 -- 172.21.15.23:0/3473498979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc600a9d40 msgr2=0x7fcc600aa1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:35.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.670+0000 7fcc64bfc700 1 --2- 172.21.15.23:0/3473498979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc600a9d40 0x7fcc600aa1c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fcc600b01e0 tx=0x7fcc50006e30).stop 2020-11-20T11:32:35.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.670+0000 7fcc64bfc700 1 -- 172.21.15.23:0/3473498979 shutdown_connections 2020-11-20T11:32:35.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.670+0000 7fcc64bfc700 1 --2- 172.21.15.23:0/3473498979 >> v2:172.21.15.155:3300/0 conn(0x7fcc60060a80 0x7fcc60061950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:35.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.670+0000 7fcc64bfc700 1 --2- 172.21.15.23:0/3473498979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc600a9d40 0x7fcc600aa1c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:35.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.670+0000 7fcc64bfc700 1 -- 172.21.15.23:0/3473498979 >> 172.21.15.23:0/3473498979 conn(0x7fcc60066730 msgr2=0x7fcc600601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:35.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.670+0000 7fcc64bfc700 1 -- 172.21.15.23:0/3473498979 shutdown_connections 2020-11-20T11:32:35.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.670+0000 7fcc64bfc700 1 -- 172.21.15.23:0/3473498979 wait complete. 2020-11-20T11:32:35.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.671+0000 7fcc64bfc700 1 Processor -- start 2020-11-20T11:32:35.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.671+0000 7fcc64bfc700 1 -- start start 2020-11-20T11:32:35.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.672+0000 7fcc64bfc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc60060a80 0x7fcc600a5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:35.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.672+0000 7fcc64bfc700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcc600a9d40 0x7fcc600a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:35.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.672+0000 7fcc64bfc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc60137920 0x7fcc600a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:35.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.672+0000 7fcc64bfc700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc600a3330 con 0x7fcc60060a80 2020-11-20T11:32:35.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.672+0000 7fcc64bfc700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc600a3590 con 0x7fcc60137920 2020-11-20T11:32:35.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.672+0000 7fcc64bfc700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc600a3830 con 0x7fcc600a9d40 2020-11-20T11:32:35.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.672+0000 7fcc5e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc60060a80 0x7fcc600a5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:35.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.672+0000 7fcc5e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc60060a80 0x7fcc600a5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41050/0 (socket says 172.21.15.23:41050) 2020-11-20T11:32:35.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.672+0000 7fcc5e59c700 1 -- 172.21.15.23:0/319959325 learned_addr learned my addr 172.21.15.23:0/319959325 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:35.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.672+0000 7fcc5e59c700 1 -- 172.21.15.23:0/319959325 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcc600a9d40 msgr2=0x7fcc600a4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:35.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.672+0000 7fcc5e59c700 1 --2- 172.21.15.23:0/319959325 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcc600a9d40 0x7fcc600a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:32:35.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.672+0000 7fcc5e59c700 1 -- 172.21.15.23:0/319959325 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc60137920 msgr2=0x7fcc600a5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:35.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.672+0000 7fcc5e59c700 1 --2- 172.21.15.23:0/319959325 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc60137920 0x7fcc600a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:32:35.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.672+0000 7fcc5e59c700 1 -- 172.21.15.23:0/319959325 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc6014a180 con 0x7fcc60060a80 2020-11-20T11:32:35.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.672+0000 7fcc5e59c700 1 --2- 172.21.15.23:0/319959325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc60060a80 0x7fcc600a5ba0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fcc4800db30 tx=0x7fcc48008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:35.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.672+0000 7fcc567fc700 1 -- 172.21.15.23:0/319959325 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc480097f0 con 0x7fcc60060a80 2020-11-20T11:32:35.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.672+0000 7fcc567fc700 1 -- 172.21.15.23:0/319959325 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc48009950 con 0x7fcc60060a80 2020-11-20T11:32:35.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.672+0000 7fcc64bfc700 1 -- 172.21.15.23:0/319959325 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc6014a430 con 0x7fcc60060a80 2020-11-20T11:32:35.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.672+0000 7fcc567fc700 1 -- 172.21.15.23:0/319959325 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc48009cd0 con 0x7fcc60060a80 2020-11-20T11:32:35.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.672+0000 7fcc64bfc700 1 -- 172.21.15.23:0/319959325 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc6014a970 con 0x7fcc60060a80 2020-11-20T11:32:35.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.673+0000 7fcc64bfc700 1 -- 172.21.15.23:0/319959325 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc60135770 con 0x7fcc60060a80 2020-11-20T11:32:35.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.673+0000 7fcc567fc700 1 -- 172.21.15.23:0/319959325 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fcc48013690 con 0x7fcc60060a80 2020-11-20T11:32:35.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.674+0000 7fcc567fc700 1 --2- 172.21.15.23:0/319959325 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcc3c056d60 0x7fcc3c059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:35.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.674+0000 7fcc567fc700 1 -- 172.21.15.23:0/319959325 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fcc48070140 con 0x7fcc60060a80 2020-11-20T11:32:35.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.674+0000 7fcc5dd9b700 1 --2- 172.21.15.23:0/319959325 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcc3c056d60 0x7fcc3c059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:35.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.674+0000 7fcc5dd9b700 1 --2- 172.21.15.23:0/319959325 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcc3c056d60 0x7fcc3c059220 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fcc6013bee0 tx=0x7fcc50015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:35.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.678+0000 7fcc567fc700 1 -- 172.21.15.23:0/319959325 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fcc48045430 con 0x7fcc60060a80 2020-11-20T11:32:35.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.825+0000 7fcc64bfc700 1 -- 172.21.15.23:0/319959325 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fcc600a39f0 con 0x7fcc60060a80 2020-11-20T11:32:35.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.825+0000 7fcc567fc700 1 -- 172.21.15.23:0/319959325 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcc48048a50 con 0x7fcc60060a80 2020-11-20T11:32:35.828 INFO:teuthology.orchestra.run.smithi023.stdout:55834574855 2020-11-20T11:32:35.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.826+0000 7fcc64bfc700 1 -- 172.21.15.23:0/319959325 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcc3c056d60 msgr2=0x7fcc3c059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:35.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.826+0000 7fcc64bfc700 1 --2- 172.21.15.23:0/319959325 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcc3c056d60 0x7fcc3c059220 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fcc6013bee0 tx=0x7fcc50015000).stop 2020-11-20T11:32:35.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.826+0000 7fcc64bfc700 1 -- 172.21.15.23:0/319959325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc60060a80 msgr2=0x7fcc600a5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:35.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.826+0000 7fcc64bfc700 1 --2- 172.21.15.23:0/319959325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc60060a80 0x7fcc600a5ba0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fcc4800db30 tx=0x7fcc48008040).stop 2020-11-20T11:32:35.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.827+0000 7fcc64bfc700 1 -- 172.21.15.23:0/319959325 shutdown_connections 2020-11-20T11:32:35.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.827+0000 7fcc64bfc700 1 --2- 172.21.15.23:0/319959325 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcc60137920 0x7fcc600a5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:32:35.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.827+0000 7fcc64bfc700 1 --2- 172.21.15.23:0/319959325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcc60060a80 0x7fcc600a5ba0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:35.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.827+0000 7fcc64bfc700 1 --2- 172.21.15.23:0/319959325 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcc3c056d60 0x7fcc3c059220 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:35.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.827+0000 7fcc64bfc700 1 --2- 172.21.15.23:0/319959325 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcc600a9d40 0x7fcc600a4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:32:35.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.827+0000 7fcc64bfc700 1 -- 172.21.15.23:0/319959325 >> 172.21.15.23:0/319959325 conn(0x7fcc60066730 msgr2=0x7fcc60136930 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:35.832 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.827+0000 7fcc64bfc700 1 -- 172.21.15.23:0/319959325 shutdown_connections 2020-11-20T11:32:35.832 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.827+0000 7fcc64bfc700 1 -- 172.21.15.23:0/319959325 wait complete. 2020-11-20T11:32:35.841 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.5 2020-11-20T11:32:35.841 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2020-11-20T11:32:35.960 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.957+0000 7f64e759e700 1 Processor -- start 2020-11-20T11:32:35.960 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.957+0000 7f64e759e700 1 -- start start 2020-11-20T11:32:35.966 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.957+0000 7f64e759e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64e80ace70 0x7f64e813a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:35.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.957+0000 7f64e759e700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f64e80ad2f0 0x7f64e813ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:35.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.957+0000 7f64e759e700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e80ad860 con 0x7f64e80a9040 2020-11-20T11:32:35.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.957+0000 7f64e759e700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64e80aa8a0 con 0x7f64e80ad2f0 2020-11-20T11:32:35.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.957+0000 7f64e759e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e80aaa90 con 0x7f64e80ace70 2020-11-20T11:32:35.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.958+0000 7f64e6d9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f64e80ad2f0 0x7f64e813ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:35.968 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.958+0000 7f64e659c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f64e80a9040 0x7f64e80aaf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55310/0 (socket says 172.21.15.23:55310) 2020-11-20T11:32:35.968 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.958+0000 7f64e659c700 1 -- 172.21.15.23:0/71541200 learned_addr learned my addr 172.21.15.23:0/71541200 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:35.968 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.958+0000 7f64e5d9b700 1 --2- 172.21.15.23:0/71541200 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64e80ace70 0x7f64e813a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:35.969 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.958+0000 7f64e6d9d700 1 -- 172.21.15.23:0/71541200 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64e80ace70 msgr2=0x7f64e813a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:35.969 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.958+0000 7f64e6d9d700 1 --2- 172.21.15.23:0/71541200 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64e80ace70 0x7f64e813a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:35.970 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.958+0000 7f64e6d9d700 1 -- 172.21.15.23:0/71541200 >> v1:172.21.15.23:6789/0 conn(0x7f64e80a9040 legacy=0x7f64e80aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:35.970 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.958+0000 7f64e559a700 1 -- 172.21.15.23:0/71541200 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3031862351 0 0) 0x7f64e80ad860 con 0x7f64e80a9040 2020-11-20T11:32:35.970 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.958+0000 7f64e6d9d700 1 -- 172.21.15.23:0/71541200 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e80a9420 con 0x7f64e80ad2f0 2020-11-20T11:32:35.971 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.958+0000 7f64e6d9d700 1 --2- 172.21.15.23:0/71541200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64e80ad2f0 0x7f64e813ad20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f64e0009ce0 tx=0x7f64e0009220).ready entity=mon.0 client_cookie=4ec610f61ff0209 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:35.971 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.958+0000 7f64e559a700 1 -- 172.21.15.23:0/71541200 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e000ba40 con 0x7f64e80ad2f0 2020-11-20T11:32:35.971 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.958+0000 7f64e559a700 1 -- 172.21.15.23:0/71541200 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e000bba0 con 0x7f64e80ad2f0 2020-11-20T11:32:35.971 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.958+0000 7f64e559a700 1 -- 172.21.15.23:0/71541200 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e0011400 con 0x7f64e80ad2f0 2020-11-20T11:32:35.972 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.959+0000 7f64e759e700 1 -- 172.21.15.23:0/71541200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64e80ad2f0 msgr2=0x7f64e813ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:35.972 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.959+0000 7f64e759e700 1 --2- 172.21.15.23:0/71541200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64e80ad2f0 0x7f64e813ad20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f64e0009ce0 tx=0x7f64e0009220).stop 2020-11-20T11:32:35.972 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.959+0000 7f64e759e700 1 -- 172.21.15.23:0/71541200 shutdown_connections 2020-11-20T11:32:35.973 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.959+0000 7f64e759e700 1 --2- 172.21.15.23:0/71541200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64e80ad2f0 0x7f64e813ad20 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:35.973 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.959+0000 7f64e759e700 1 --2- 172.21.15.23:0/71541200 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64e80ace70 0x7f64e813a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:35.974 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.959+0000 7f64e759e700 1 -- 172.21.15.23:0/71541200 >> 172.21.15.23:0/71541200 conn(0x7f64e80683b0 msgr2=0x7f64e80b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:35.974 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.959+0000 7f64e759e700 1 -- 172.21.15.23:0/71541200 shutdown_connections 2020-11-20T11:32:35.974 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.959+0000 7f64e759e700 1 -- 172.21.15.23:0/71541200 wait complete. 2020-11-20T11:32:35.975 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.960+0000 7f64e759e700 1 Processor -- start 2020-11-20T11:32:35.975 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.960+0000 7f64e759e700 1 -- start start 2020-11-20T11:32:35.975 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.960+0000 7f64e759e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64e80a9040 0x7f64e813d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:35.976 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.960+0000 7f64e759e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64e80ace70 0x7f64e80a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:35.976 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.960+0000 7f64e759e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64e80ad2f0 0x7f64e8149c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:35.976 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.960+0000 7f64e759e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e80a0380 con 0x7f64e80ad2f0 2020-11-20T11:32:35.976 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.960+0000 7f64e759e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e809f9a0 con 0x7f64e80ace70 2020-11-20T11:32:35.977 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.960+0000 7f64e759e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e809fc40 con 0x7f64e80a9040 2020-11-20T11:32:35.977 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.960+0000 7f64e6d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64e80ad2f0 0x7f64e8149c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:35.977 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.960+0000 7f64e6d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64e80ad2f0 0x7f64e8149c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41066/0 (socket says 172.21.15.23:41066) 2020-11-20T11:32:35.978 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.960+0000 7f64e6d9d700 1 -- 172.21.15.23:0/1547506763 learned_addr learned my addr 172.21.15.23:0/1547506763 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:35.978 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.961+0000 7f64e5d9b700 1 --2- 172.21.15.23:0/1547506763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64e80ace70 0x7f64e80a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:35.978 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.961+0000 7f64e659c700 1 --2- 172.21.15.23:0/1547506763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64e80a9040 0x7f64e813d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:35.979 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.961+0000 7f64e6d9d700 1 -- 172.21.15.23:0/1547506763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64e80a9040 msgr2=0x7f64e813d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:35.979 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.961+0000 7f64e6d9d700 1 --2- 172.21.15.23:0/1547506763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64e80a9040 0x7f64e813d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:35.979 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.961+0000 7f64e6d9d700 1 -- 172.21.15.23:0/1547506763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64e80ace70 msgr2=0x7f64e80a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:35.979 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.961+0000 7f64e6d9d700 1 --2- 172.21.15.23:0/1547506763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64e80ace70 0x7f64e80a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:35.980 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.961+0000 7f64e6d9d700 1 -- 172.21.15.23:0/1547506763 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e809fee0 con 0x7f64e80ad2f0 2020-11-20T11:32:35.980 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.961+0000 7f64e659c700 1 --2- 172.21.15.23:0/1547506763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64e80a9040 0x7f64e813d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:35.980 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.961+0000 7f64e6d9d700 1 --2- 172.21.15.23:0/1547506763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64e80ad2f0 0x7f64e8149c70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f64e000b800 tx=0x7f64e0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:35.980 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.961+0000 7f64d67fc700 1 -- 172.21.15.23:0/1547506763 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e000b4a0 con 0x7f64e80ad2f0 2020-11-20T11:32:35.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.961+0000 7f64d67fc700 1 -- 172.21.15.23:0/1547506763 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e0010a50 con 0x7f64e80ad2f0 2020-11-20T11:32:35.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.961+0000 7f64e759e700 1 -- 172.21.15.23:0/1547506763 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64e813e500 con 0x7f64e80ad2f0 2020-11-20T11:32:35.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.961+0000 7f64d67fc700 1 -- 172.21.15.23:0/1547506763 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e0010dd0 con 0x7f64e80ad2f0 2020-11-20T11:32:35.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.961+0000 7f64e759e700 1 -- 172.21.15.23:0/1547506763 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64e80a1f00 con 0x7f64e80ad2f0 2020-11-20T11:32:35.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.962+0000 7f64e759e700 1 -- 172.21.15.23:0/1547506763 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64e80aa570 con 0x7f64e80ad2f0 2020-11-20T11:32:35.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.962+0000 7f64d67fc700 1 -- 172.21.15.23:0/1547506763 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f64e0017040 con 0x7f64e80ad2f0 2020-11-20T11:32:35.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.963+0000 7f64d67fc700 1 --2- 172.21.15.23:0/1547506763 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f64bc056d60 0x7f64bc059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:35.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.963+0000 7f64d67fc700 1 -- 172.21.15.23:0/1547506763 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f64e0071cd0 con 0x7f64e80ad2f0 2020-11-20T11:32:35.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.963+0000 7f64e659c700 1 --2- 172.21.15.23:0/1547506763 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f64bc056d60 0x7f64bc059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:35.983 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.963+0000 7f64e659c700 1 --2- 172.21.15.23:0/1547506763 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f64bc056d60 0x7f64bc059220 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f64dc001550 tx=0x7f64dc009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:35.983 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:35.966+0000 7f64d67fc700 1 -- 172.21.15.23:0/1547506763 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f64e0046f60 con 0x7f64e80ad2f0 2020-11-20T11:32:36.118 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.114+0000 7f64e759e700 1 -- 172.21.15.23:0/1547506763 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f64e8057dc0 con 0x7f64e80ad2f0 2020-11-20T11:32:36.118 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.114+0000 7f64d67fc700 1 -- 172.21.15.23:0/1547506763 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f64e0015070 con 0x7f64e80ad2f0 2020-11-20T11:32:36.119 INFO:teuthology.orchestra.run.smithi023.stdout:60129542150 2020-11-20T11:32:36.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.116+0000 7f64e759e700 1 -- 172.21.15.23:0/1547506763 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f64bc056d60 msgr2=0x7f64bc059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:36.121 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.116+0000 7f64e759e700 1 --2- 172.21.15.23:0/1547506763 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f64bc056d60 0x7f64bc059220 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f64dc001550 tx=0x7f64dc009040).stop 2020-11-20T11:32:36.121 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.116+0000 7f64e759e700 1 -- 172.21.15.23:0/1547506763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64e80ad2f0 msgr2=0x7f64e8149c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:36.122 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.116+0000 7f64e759e700 1 --2- 172.21.15.23:0/1547506763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64e80ad2f0 0x7f64e8149c70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f64e000b800 tx=0x7f64e0016000).stop 2020-11-20T11:32:36.122 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.116+0000 7f64e759e700 1 -- 172.21.15.23:0/1547506763 shutdown_connections 2020-11-20T11:32:36.122 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.116+0000 7f64e759e700 1 --2- 172.21.15.23:0/1547506763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64e80ad2f0 0x7f64e8149c70 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:36.123 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.116+0000 7f64e759e700 1 --2- 172.21.15.23:0/1547506763 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f64bc056d60 0x7f64bc059220 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:36.123 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.116+0000 7f64e759e700 1 --2- 172.21.15.23:0/1547506763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64e80a9040 0x7f64e813d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:36.124 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.116+0000 7f64e759e700 1 --2- 172.21.15.23:0/1547506763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64e80ace70 0x7f64e80a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:36.124 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.116+0000 7f64e759e700 1 -- 172.21.15.23:0/1547506763 >> 172.21.15.23:0/1547506763 conn(0x7f64e80683b0 msgr2=0x7f64e80aae50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:36.124 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.116+0000 7f64e759e700 1 -- 172.21.15.23:0/1547506763 shutdown_connections 2020-11-20T11:32:36.125 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.116+0000 7f64e759e700 1 -- 172.21.15.23:0/1547506763 wait complete. 2020-11-20T11:32:36.130 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.6 2020-11-20T11:32:36.131 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2020-11-20T11:32:36.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.249+0000 7fc12ee4a700 1 Processor -- start 2020-11-20T11:32:36.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.250+0000 7fc12ee4a700 1 -- start start 2020-11-20T11:32:36.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.250+0000 7fc12ee4a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc1280a2080 0x7fc1280aef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:36.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.250+0000 7fc12ee4a700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1280aaab0 con 0x7fc1280a2460 2020-11-20T11:32:36.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.250+0000 7fc12ee4a700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1280aad50 con 0x7fc128067a10 2020-11-20T11:32:36.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.250+0000 7fc12ee4a700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc12813aa50 con 0x7fc1280a2080 2020-11-20T11:32:36.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.250+0000 7fc12de48700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc1280a2080 0x7fc1280aef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:36.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.250+0000 7fc12d647700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fc1280a2460 0x7fc128137df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55322/0 (socket says 172.21.15.23:55322) 2020-11-20T11:32:36.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.250+0000 7fc12de48700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc1280a2080 0x7fc1280aef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41072/0 (socket says 172.21.15.23:41072) 2020-11-20T11:32:36.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.250+0000 7fc12d647700 1 -- 172.21.15.23:0/3972200383 learned_addr learned my addr 172.21.15.23:0/3972200383 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:36.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.251+0000 7fc12de48700 1 -- 172.21.15.23:0/3972200383 >> v1:172.21.15.155:6789/0 conn(0x7fc128067a10 legacy=0x7fc1281453c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:36.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.251+0000 7fc12de48700 1 -- 172.21.15.23:0/3972200383 >> v1:172.21.15.23:6789/0 conn(0x7fc1280a2460 legacy=0x7fc128137df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:36.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.251+0000 7fc12de48700 1 -- 172.21.15.23:0/3972200383 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12813b920 con 0x7fc1280a2080 2020-11-20T11:32:36.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.251+0000 7fc12ce46700 1 -- 172.21.15.23:0/3972200383 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 78168267 0 0) 0x7fc1280aaab0 con 0x7fc1280a2460 2020-11-20T11:32:36.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.251+0000 7fc12de48700 1 --2- 172.21.15.23:0/3972200383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1280a2080 0x7fc1280aef10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fc11800b9d0 tx=0x7fc118006dd0).ready entity=mon.0 client_cookie=22183e58c2e9f55a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:36.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.251+0000 7fc12ce46700 1 -- 172.21.15.23:0/3972200383 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc118006030 con 0x7fc1280a2080 2020-11-20T11:32:36.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.251+0000 7fc12ce46700 1 -- 172.21.15.23:0/3972200383 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc118006190 con 0x7fc1280a2080 2020-11-20T11:32:36.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.251+0000 7fc12ce46700 1 -- 172.21.15.23:0/3972200383 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc118006510 con 0x7fc1280a2080 2020-11-20T11:32:36.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.251+0000 7fc12ee4a700 1 -- 172.21.15.23:0/3972200383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1280a2080 msgr2=0x7fc1280aef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:36.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.251+0000 7fc12ee4a700 1 --2- 172.21.15.23:0/3972200383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1280a2080 0x7fc1280aef10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fc11800b9d0 tx=0x7fc118006dd0).stop 2020-11-20T11:32:36.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.251+0000 7fc12ee4a700 1 -- 172.21.15.23:0/3972200383 shutdown_connections 2020-11-20T11:32:36.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.251+0000 7fc12ee4a700 1 --2- 172.21.15.23:0/3972200383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1280a2080 0x7fc1280aef10 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:36.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.251+0000 7fc12ee4a700 1 -- 172.21.15.23:0/3972200383 >> 172.21.15.23:0/3972200383 conn(0x7fc128063980 msgr2=0x7fc128063d90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:36.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.252+0000 7fc12ee4a700 1 -- 172.21.15.23:0/3972200383 shutdown_connections 2020-11-20T11:32:36.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.252+0000 7fc12ee4a700 1 -- 172.21.15.23:0/3972200383 wait complete. 2020-11-20T11:32:36.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.252+0000 7fc12ee4a700 1 Processor -- start 2020-11-20T11:32:36.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.253+0000 7fc12ee4a700 1 -- start start 2020-11-20T11:32:36.266 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.253+0000 7fc12ee4a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc128067a10 0x7fc128149f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:36.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.253+0000 7fc12ee4a700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1280a2080 0x7fc1280a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:36.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.253+0000 7fc12ee4a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1280a2460 0x7fc1280a8710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:36.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.253+0000 7fc12ee4a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12814aee0 con 0x7fc1280a2460 2020-11-20T11:32:36.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.253+0000 7fc12ee4a700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1280a52b0 con 0x7fc128067a10 2020-11-20T11:32:36.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.253+0000 7fc12ee4a700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1280a5550 con 0x7fc1280a2080 2020-11-20T11:32:36.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.253+0000 7fc12e649700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1280a2460 0x7fc1280a8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:36.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.253+0000 7fc12e649700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1280a2460 0x7fc1280a8710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41082/0 (socket says 172.21.15.23:41082) 2020-11-20T11:32:36.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.253+0000 7fc12e649700 1 -- 172.21.15.23:0/2571672012 learned_addr learned my addr 172.21.15.23:0/2571672012 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:36.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.253+0000 7fc12de48700 1 --2- 172.21.15.23:0/2571672012 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc128067a10 0x7fc128149f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:36.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.253+0000 7fc12d647700 1 --2- 172.21.15.23:0/2571672012 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1280a2080 0x7fc1280a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:36.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.254+0000 7fc12e649700 1 -- 172.21.15.23:0/2571672012 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1280a2080 msgr2=0x7fc1280a59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:36.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.254+0000 7fc12e649700 1 --2- 172.21.15.23:0/2571672012 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1280a2080 0x7fc1280a59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:36.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.254+0000 7fc12e649700 1 -- 172.21.15.23:0/2571672012 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc128067a10 msgr2=0x7fc128149f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:36.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.254+0000 7fc12e649700 1 --2- 172.21.15.23:0/2571672012 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc128067a10 0x7fc128149f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:36.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.254+0000 7fc12e649700 1 -- 172.21.15.23:0/2571672012 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1280a92e0 con 0x7fc1280a2460 2020-11-20T11:32:36.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.254+0000 7fc12de48700 1 --2- 172.21.15.23:0/2571672012 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc128067a10 0x7fc128149f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:36.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.254+0000 7fc12d647700 1 --2- 172.21.15.23:0/2571672012 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1280a2080 0x7fc1280a59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:36.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.254+0000 7fc12e649700 1 --2- 172.21.15.23:0/2571672012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1280a2460 0x7fc1280a8710 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fc12000ca60 tx=0x7fc1200023f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:36.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.254+0000 7fc115ffb700 1 -- 172.21.15.23:0/2571672012 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12000a760 con 0x7fc1280a2460 2020-11-20T11:32:36.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.254+0000 7fc115ffb700 1 -- 172.21.15.23:0/2571672012 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc12000a8e0 con 0x7fc1280a2460 2020-11-20T11:32:36.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.254+0000 7fc12ee4a700 1 -- 172.21.15.23:0/2571672012 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1280a6300 con 0x7fc1280a2460 2020-11-20T11:32:36.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.254+0000 7fc115ffb700 1 -- 172.21.15.23:0/2571672012 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12000aca0 con 0x7fc1280a2460 2020-11-20T11:32:36.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.254+0000 7fc12ee4a700 1 -- 172.21.15.23:0/2571672012 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1280a6f10 con 0x7fc1280a2460 2020-11-20T11:32:36.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.255+0000 7fc12ee4a700 1 -- 172.21.15.23:0/2571672012 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1280ac780 con 0x7fc1280a2460 2020-11-20T11:32:36.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.255+0000 7fc115ffb700 1 -- 172.21.15.23:0/2571672012 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc120010070 con 0x7fc1280a2460 2020-11-20T11:32:36.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.255+0000 7fc115ffb700 1 --2- 172.21.15.23:0/2571672012 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc100056d60 0x7fc100059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:36.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.255+0000 7fc115ffb700 1 -- 172.21.15.23:0/2571672012 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fc12006fee0 con 0x7fc1280a2460 2020-11-20T11:32:36.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.256+0000 7fc12de48700 1 --2- 172.21.15.23:0/2571672012 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc100056d60 0x7fc100059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:36.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.256+0000 7fc12de48700 1 --2- 172.21.15.23:0/2571672012 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc100056d60 0x7fc100059220 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fc11800b9d0 tx=0x7fc118015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:36.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.259+0000 7fc115ffb700 1 -- 172.21.15.23:0/2571672012 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc120045280 con 0x7fc1280a2460 2020-11-20T11:32:36.415 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.412+0000 7fc12ee4a700 1 -- 172.21.15.23:0/2571672012 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fc128057dc0 con 0x7fc1280a2460 2020-11-20T11:32:36.416 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.412+0000 7fc115ffb700 1 -- 172.21.15.23:0/2571672012 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc1200488a0 con 0x7fc1280a2460 2020-11-20T11:32:36.416 INFO:teuthology.orchestra.run.smithi023.stdout:55834574854 2020-11-20T11:32:36.418 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.413+0000 7fc12ee4a700 1 -- 172.21.15.23:0/2571672012 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc100056d60 msgr2=0x7fc100059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:36.418 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.413+0000 7fc12ee4a700 1 --2- 172.21.15.23:0/2571672012 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc100056d60 0x7fc100059220 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fc11800b9d0 tx=0x7fc118015000).stop 2020-11-20T11:32:36.418 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.414+0000 7fc12ee4a700 1 -- 172.21.15.23:0/2571672012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1280a2460 msgr2=0x7fc1280a8710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:36.419 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.414+0000 7fc12ee4a700 1 --2- 172.21.15.23:0/2571672012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1280a2460 0x7fc1280a8710 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fc12000ca60 tx=0x7fc1200023f0).stop 2020-11-20T11:32:36.419 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.414+0000 7fc12ee4a700 1 -- 172.21.15.23:0/2571672012 shutdown_connections 2020-11-20T11:32:36.419 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.414+0000 7fc12ee4a700 1 --2- 172.21.15.23:0/2571672012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1280a2460 0x7fc1280a8710 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:36.419 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.414+0000 7fc12ee4a700 1 --2- 172.21.15.23:0/2571672012 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc128067a10 0x7fc128149f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:36.419 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.414+0000 7fc12ee4a700 1 --2- 172.21.15.23:0/2571672012 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc100056d60 0x7fc100059220 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:36.420 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.414+0000 7fc12ee4a700 1 --2- 172.21.15.23:0/2571672012 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1280a2080 0x7fc1280a59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:36.420 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.414+0000 7fc12ee4a700 1 -- 172.21.15.23:0/2571672012 >> 172.21.15.23:0/2571672012 conn(0x7fc128063980 msgr2=0x7fc1280abda0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:36.420 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.414+0000 7fc12ee4a700 1 -- 172.21.15.23:0/2571672012 shutdown_connections 2020-11-20T11:32:36.420 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.414+0000 7fc12ee4a700 1 -- 172.21.15.23:0/2571672012 wait complete. 2020-11-20T11:32:36.428 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.7 2020-11-20T11:32:36.428 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2020-11-20T11:32:36.429 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-20T11:32:36.544 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.541+0000 7fa301eac700 1 Processor -- start 2020-11-20T11:32:36.544 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.541+0000 7fa301eac700 1 -- start start 2020-11-20T11:32:36.546 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.542+0000 7fa301eac700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa2fc0aaab0 0x7fa2fc13ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:36.547 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.542+0000 7fa301eac700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa2fc13aeb0 0x7fa2fc0a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:36.547 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.542+0000 7fa301eac700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2fc1388b0 con 0x7fa2fc0aae90 2020-11-20T11:32:36.547 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.542+0000 7fa301eac700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2fc13e600 con 0x7fa2fc0aaab0 2020-11-20T11:32:36.547 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.542+0000 7fa301eac700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2fc13e8a0 con 0x7fa2fc13aeb0 2020-11-20T11:32:36.548 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.542+0000 7fa2fb7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa2fc0aaab0 0x7fa2fc13ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:36.548 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.542+0000 7fa2faffd700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fa2fc0aae90 0x7fa2fc1381a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55336/0 (socket says 172.21.15.23:55336) 2020-11-20T11:32:36.548 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.542+0000 7fa2faffd700 1 -- 172.21.15.23:0/1711505637 learned_addr learned my addr 172.21.15.23:0/1711505637 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:36.548 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.542+0000 7fa2fbfff700 1 --2- 172.21.15.23:0/1711505637 >> v2:172.21.15.155:3300/0 conn(0x7fa2fc13aeb0 0x7fa2fc0a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:36.549 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.542+0000 7fa2fb7fe700 1 -- 172.21.15.23:0/1711505637 >> v2:172.21.15.155:3300/0 conn(0x7fa2fc13aeb0 msgr2=0x7fa2fc0a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:36.549 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.542+0000 7fa2fb7fe700 1 --2- 172.21.15.23:0/1711505637 >> v2:172.21.15.155:3300/0 conn(0x7fa2fc13aeb0 0x7fa2fc0a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:36.550 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.542+0000 7fa2fb7fe700 1 -- 172.21.15.23:0/1711505637 >> v1:172.21.15.23:6789/0 conn(0x7fa2fc0aae90 legacy=0x7fa2fc1381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:36.550 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.542+0000 7fa2fb7fe700 1 -- 172.21.15.23:0/1711505637 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2fc0ab6d0 con 0x7fa2fc0aaab0 2020-11-20T11:32:36.550 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.542+0000 7fa2fa7fc700 1 -- 172.21.15.23:0/1711505637 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4025252279 0 0) 0x7fa2fc1388b0 con 0x7fa2fc0aae90 2020-11-20T11:32:36.550 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.542+0000 7fa2fb7fe700 1 --2- 172.21.15.23:0/1711505637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2fc0aaab0 0x7fa2fc13ed40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fa2fc062f40 tx=0x7fa2ec006e30).ready entity=mon.0 client_cookie=f3ddf53d12aae6ea server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:36.551 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.543+0000 7fa2fa7fc700 1 -- 172.21.15.23:0/1711505637 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec0097b0 con 0x7fa2fc0aaab0 2020-11-20T11:32:36.551 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.543+0000 7fa2fa7fc700 1 -- 172.21.15.23:0/1711505637 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2ec000e90 con 0x7fa2fc0aaab0 2020-11-20T11:32:36.551 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.543+0000 7fa2fa7fc700 1 -- 172.21.15.23:0/1711505637 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec009b20 con 0x7fa2fc0aaab0 2020-11-20T11:32:36.551 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.543+0000 7fa301eac700 1 -- 172.21.15.23:0/1711505637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2fc0aaab0 msgr2=0x7fa2fc13ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:36.552 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.543+0000 7fa301eac700 1 --2- 172.21.15.23:0/1711505637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2fc0aaab0 0x7fa2fc13ed40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fa2fc062f40 tx=0x7fa2ec006e30).stop 2020-11-20T11:32:36.552 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.543+0000 7fa301eac700 1 -- 172.21.15.23:0/1711505637 shutdown_connections 2020-11-20T11:32:36.552 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.543+0000 7fa301eac700 1 --2- 172.21.15.23:0/1711505637 >> v2:172.21.15.155:3300/0 conn(0x7fa2fc13aeb0 0x7fa2fc0a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:36.552 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.543+0000 7fa301eac700 1 --2- 172.21.15.23:0/1711505637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2fc0aaab0 0x7fa2fc13ed40 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:36.552 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.543+0000 7fa301eac700 1 -- 172.21.15.23:0/1711505637 >> 172.21.15.23:0/1711505637 conn(0x7fa2fc0678e0 msgr2=0x7fa2fc067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:36.553 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.543+0000 7fa301eac700 1 -- 172.21.15.23:0/1711505637 shutdown_connections 2020-11-20T11:32:36.553 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.543+0000 7fa301eac700 1 -- 172.21.15.23:0/1711505637 wait complete. 2020-11-20T11:32:36.555 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.544+0000 7fa301eac700 1 Processor -- start 2020-11-20T11:32:36.556 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.544+0000 7fa301eac700 1 -- start start 2020-11-20T11:32:36.556 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.545+0000 7fa301eac700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2fc0aaab0 0x7fa2fc142190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:36.556 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.545+0000 7fa301eac700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2fc0aae90 0x7fa2fc0a7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:36.556 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.545+0000 7fa301eac700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2fc0a7f60 0x7fa2fc14a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:36.557 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.545+0000 7fa301eac700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2fc0a7820 con 0x7fa2fc0aaab0 2020-11-20T11:32:36.557 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.545+0000 7fa301eac700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2fc0a5420 con 0x7fa2fc0aae90 2020-11-20T11:32:36.557 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.545+0000 7fa301eac700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2fc0a56c0 con 0x7fa2fc0a7f60 2020-11-20T11:32:36.557 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.545+0000 7fa2fb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2fc0aaab0 0x7fa2fc142190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:36.557 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.545+0000 7fa2fb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2fc0aaab0 0x7fa2fc142190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41092/0 (socket says 172.21.15.23:41092) 2020-11-20T11:32:36.558 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.545+0000 7fa2fb7fe700 1 -- 172.21.15.23:0/4235572280 learned_addr learned my addr 172.21.15.23:0/4235572280 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:36.558 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.545+0000 7fa2faffd700 1 --2- 172.21.15.23:0/4235572280 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2fc0aae90 0x7fa2fc0a7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:36.558 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.545+0000 7fa2fbfff700 1 --2- 172.21.15.23:0/4235572280 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2fc0a7f60 0x7fa2fc14a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:36.558 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.545+0000 7fa2fb7fe700 1 -- 172.21.15.23:0/4235572280 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2fc0a7f60 msgr2=0x7fa2fc14a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:36.558 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.545+0000 7fa2fb7fe700 1 --2- 172.21.15.23:0/4235572280 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2fc0a7f60 0x7fa2fc14a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:36.559 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.545+0000 7fa2fb7fe700 1 -- 172.21.15.23:0/4235572280 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2fc0aae90 msgr2=0x7fa2fc0a7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:36.559 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.545+0000 7fa2fb7fe700 1 --2- 172.21.15.23:0/4235572280 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2fc0aae90 0x7fa2fc0a7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:36.559 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.545+0000 7fa2fb7fe700 1 -- 172.21.15.23:0/4235572280 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2fc0a7520 con 0x7fa2fc0aaab0 2020-11-20T11:32:36.559 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.545+0000 7fa2fb7fe700 1 --2- 172.21.15.23:0/4235572280 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2fc0aaab0 0x7fa2fc142190 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fa2ec00b190 tx=0x7fa2ec015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:36.560 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.545+0000 7fa2db7fe700 1 -- 172.21.15.23:0/4235572280 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec011c40 con 0x7fa2fc0aaab0 2020-11-20T11:32:36.560 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.545+0000 7fa2db7fe700 1 -- 172.21.15.23:0/4235572280 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2ec011da0 con 0x7fa2fc0aaab0 2020-11-20T11:32:36.560 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.545+0000 7fa301eac700 1 -- 172.21.15.23:0/4235572280 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2fc0a5b60 con 0x7fa2fc0aaab0 2020-11-20T11:32:36.560 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.545+0000 7fa2db7fe700 1 -- 172.21.15.23:0/4235572280 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec01e690 con 0x7fa2fc0aaab0 2020-11-20T11:32:36.560 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.546+0000 7fa301eac700 1 -- 172.21.15.23:0/4235572280 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2fc14a9b0 con 0x7fa2fc0aaab0 2020-11-20T11:32:36.561 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.546+0000 7fa301eac700 1 -- 172.21.15.23:0/4235572280 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2fc05c7d0 con 0x7fa2fc0aaab0 2020-11-20T11:32:36.561 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.547+0000 7fa2db7fe700 1 -- 172.21.15.23:0/4235572280 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa2ec025030 con 0x7fa2fc0aaab0 2020-11-20T11:32:36.561 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.547+0000 7fa2db7fe700 1 --2- 172.21.15.23:0/4235572280 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa2d4056d60 0x7fa2d4059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:36.561 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.547+0000 7fa2db7fe700 1 -- 172.21.15.23:0/4235572280 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fa2ec071550 con 0x7fa2fc0aaab0 2020-11-20T11:32:36.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.547+0000 7fa2faffd700 1 --2- 172.21.15.23:0/4235572280 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa2d4056d60 0x7fa2d4059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:36.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.548+0000 7fa2faffd700 1 --2- 172.21.15.23:0/4235572280 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa2d4056d60 0x7fa2d4059220 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fa2fc147e60 tx=0x7fa2f0009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:36.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.551+0000 7fa2db7fe700 1 -- 172.21.15.23:0/4235572280 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa2ec0468f0 con 0x7fa2fc0aaab0 2020-11-20T11:32:36.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.695+0000 7fa301eac700 1 -- 172.21.15.23:0/4235572280 --> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa2fc0a5880 con 0x7fa2d4056d60 2020-11-20T11:32:36.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.699+0000 7fa2db7fe700 1 -- 172.21.15.23:0/4235572280 <== mgr.4105 v2:172.21.15.155:6832/22693 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+82153 (secure 0 0 0) 0x7fa2fc0a5880 con 0x7fa2d4056d60 2020-11-20T11:32:36.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.701+0000 7fa301eac700 1 -- 172.21.15.23:0/4235572280 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa2d4056d60 msgr2=0x7fa2d4059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:36.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.701+0000 7fa301eac700 1 --2- 172.21.15.23:0/4235572280 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa2d4056d60 0x7fa2d4059220 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fa2fc147e60 tx=0x7fa2f0009040).stop 2020-11-20T11:32:36.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.701+0000 7fa301eac700 1 -- 172.21.15.23:0/4235572280 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2fc0aaab0 msgr2=0x7fa2fc142190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:36.707 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.701+0000 7fa301eac700 1 --2- 172.21.15.23:0/4235572280 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2fc0aaab0 0x7fa2fc142190 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fa2ec00b190 tx=0x7fa2ec015000).stop 2020-11-20T11:32:36.707 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.701+0000 7fa301eac700 1 -- 172.21.15.23:0/4235572280 shutdown_connections 2020-11-20T11:32:36.708 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.701+0000 7fa301eac700 1 --2- 172.21.15.23:0/4235572280 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa2d4056d60 0x7fa2d4059220 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:36.708 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.701+0000 7fa301eac700 1 --2- 172.21.15.23:0/4235572280 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2fc0a7f60 0x7fa2fc14a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:36.708 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.701+0000 7fa301eac700 1 --2- 172.21.15.23:0/4235572280 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2fc0aaab0 0x7fa2fc142190 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:36.709 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.701+0000 7fa301eac700 1 --2- 172.21.15.23:0/4235572280 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2fc0aae90 0x7fa2fc0a7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:36.709 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.701+0000 7fa301eac700 1 -- 172.21.15.23:0/4235572280 >> 172.21.15.23:0/4235572280 conn(0x7fa2fc0678e0 msgr2=0x7fa2fc0adc80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:36.710 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.701+0000 7fa301eac700 1 -- 172.21.15.23:0/4235572280 shutdown_connections 2020-11-20T11:32:36.710 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.701+0000 7fa301eac700 1 -- 172.21.15.23:0/4235572280 wait complete. 2020-11-20T11:32:36.710 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2020-11-20T11:32:36.711 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:32:36.715 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2020-11-20T11:32:35.525404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2304,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40,"ondisk_log_size":40,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":66,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":41152,"kb_used_data":2112,"kb_used_omap":0,"kb_used_meta":38976,"kb_avail":754933568,"statfs":{"total":773094113280,"available":773051973632,"internally_reserved":0,"allocated":2162688,"data_stored":862848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":39911424},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2304,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002508"},"pg_stats":[{"pgid":"4.2","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.480587+0000","last_change":"2020-11-20T11:32:25.125195+0000","last_active":"2020-11-20T11:32:25.480587+0000","last_peered":"2020-11-20T11:32:25.480587+0000","last_clean":"2020-11-20T11:32:25.480587+0000","last_became_active":"2020-11-20T11:32:25.124627+0000","last_became_peered":"2020-11-20T11:32:25.124627+0000","last_unstale":"2020-11-20T11:32:25.480587+0000","last_undegraded":"2020-11-20T11:32:25.480587+0000","last_fullsized":"2020-11-20T11:32:25.480587+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"16","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.480586+0000","last_change":"2020-11-20T11:32:19.482604+0000","last_active":"2020-11-20T11:32:25.480586+0000","last_peered":"2020-11-20T11:32:25.480586+0000","last_clean":"2020-11-20T11:32:25.480586+0000","last_became_active":"2020-11-20T11:32:19.482197+0000","last_became_peered":"2020-11-20T11:32:19.482197+0000","last_unstale":"2020-11-20T11:32:25.480586+0000","last_undegraded":"2020-11-20T11:32:25.480586+0000","last_fullsized":"2020-11-20T11:32:25.480586+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"24'6","reported_seq":"19","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.494434+0000","last_change":"2020-11-20T11:32:22.098576+0000","last_active":"2020-11-20T11:32:25.494434+0000","last_peered":"2020-11-20T11:32:25.494434+0000","last_clean":"2020-11-20T11:32:25.494434+0000","last_became_active":"2020-11-20T11:32:22.098264+0000","last_became_peered":"2020-11-20T11:32:22.098264+0000","last_unstale":"2020-11-20T11:32:25.494434+0000","last_undegraded":"2020-11-20T11:32:25.494434+0000","last_fullsized":"2020-11-20T11:32:25.494434+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.483873+0000","last_change":"2020-11-20T11:32:25.479776+0000","last_active":"2020-11-20T11:32:25.483873+0000","last_peered":"2020-11-20T11:32:25.483873+0000","last_clean":"2020-11-20T11:32:25.483873+0000","last_became_active":"2020-11-20T11:32:25.479350+0000","last_became_peered":"2020-11-20T11:32:25.479350+0000","last_unstale":"2020-11-20T11:32:25.483873+0000","last_undegraded":"2020-11-20T11:32:25.483873+0000","last_fullsized":"2020-11-20T11:32:25.483873+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"16","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.479488+0000","last_change":"2020-11-20T11:32:18.924328+0000","last_active":"2020-11-20T11:32:25.479488+0000","last_peered":"2020-11-20T11:32:25.479488+0000","last_clean":"2020-11-20T11:32:25.479488+0000","last_became_active":"2020-11-20T11:32:18.923879+0000","last_became_peered":"2020-11-20T11:32:18.923879+0000","last_unstale":"2020-11-20T11:32:25.479488+0000","last_undegraded":"2020-11-20T11:32:25.479488+0000","last_fullsized":"2020-11-20T11:32:25.479488+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"24'5","reported_seq":"18","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.488715+0000","last_change":"2020-11-20T11:32:22.093559+0000","last_active":"2020-11-20T11:32:25.488715+0000","last_peered":"2020-11-20T11:32:25.488715+0000","last_clean":"2020-11-20T11:32:25.488715+0000","last_became_active":"2020-11-20T11:32:22.093123+0000","last_became_peered":"2020-11-20T11:32:22.093123+0000","last_unstale":"2020-11-20T11:32:25.488715+0000","last_undegraded":"2020-11-20T11:32:25.488715+0000","last_fullsized":"2020-11-20T11:32:25.488715+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.481419+0000","last_change":"2020-11-20T11:32:25.125052+0000","last_active":"2020-11-20T11:32:25.481419+0000","last_peered":"2020-11-20T11:32:25.481419+0000","last_clean":"2020-11-20T11:32:25.481419+0000","last_became_active":"2020-11-20T11:32:25.124401+0000","last_became_peered":"2020-11-20T11:32:25.124401+0000","last_unstale":"2020-11-20T11:32:25.481419+0000","last_undegraded":"2020-11-20T11:32:25.481419+0000","last_fullsized":"2020-11-20T11:32:25.481419+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"15","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.474417+0000","last_change":"2020-11-20T11:32:18.927388+0000","last_active":"2020-11-20T11:32:25.474417+0000","last_peered":"2020-11-20T11:32:25.474417+0000","last_clean":"2020-11-20T11:32:25.474417+0000","last_became_active":"2020-11-20T11:32:18.926981+0000","last_became_peered":"2020-11-20T11:32:18.926981+0000","last_unstale":"2020-11-20T11:32:25.474417+0000","last_undegraded":"2020-11-20T11:32:25.474417+0000","last_fullsized":"2020-11-20T11:32:25.474417+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"24'8","reported_seq":"21","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.492278+0000","last_change":"2020-11-20T11:32:22.088128+0000","last_active":"2020-11-20T11:32:25.492278+0000","last_peered":"2020-11-20T11:32:25.492278+0000","last_clean":"2020-11-20T11:32:25.492278+0000","last_became_active":"2020-11-20T11:32:22.087723+0000","last_became_peered":"2020-11-20T11:32:22.087723+0000","last_unstale":"2020-11-20T11:32:25.492278+0000","last_undegraded":"2020-11-20T11:32:25.492278+0000","last_fullsized":"2020-11-20T11:32:25.492278+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.495311+0000","last_change":"2020-11-20T11:32:25.492514+0000","last_active":"2020-11-20T11:32:25.495311+0000","last_peered":"2020-11-20T11:32:25.495311+0000","last_clean":"2020-11-20T11:32:25.495311+0000","last_became_active":"2020-11-20T11:32:25.491657+0000","last_became_peered":"2020-11-20T11:32:25.491657+0000","last_unstale":"2020-11-20T11:32:25.495311+0000","last_undegraded":"2020-11-20T11:32:25.495311+0000","last_fullsized":"2020-11-20T11:32:25.495311+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"16","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.479536+0000","last_change":"2020-11-20T11:32:18.927044+0000","last_active":"2020-11-20T11:32:25.479536+0000","last_peered":"2020-11-20T11:32:25.479536+0000","last_clean":"2020-11-20T11:32:25.479536+0000","last_became_active":"2020-11-20T11:32:18.926711+0000","last_became_peered":"2020-11-20T11:32:18.926711+0000","last_unstale":"2020-11-20T11:32:25.479536+0000","last_undegraded":"2020-11-20T11:32:25.479536+0000","last_fullsized":"2020-11-20T11:32:25.479536+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'2","reported_seq":"20","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.488044+0000","last_change":"2020-11-20T11:32:14.446049+0000","last_active":"2020-11-20T11:32:25.488044+0000","last_peered":"2020-11-20T11:32:25.488044+0000","last_clean":"2020-11-20T11:32:25.488044+0000","last_became_active":"2020-11-20T11:32:14.445477+0000","last_became_peered":"2020-11-20T11:32:14.445477+0000","last_unstale":"2020-11-20T11:32:25.488044+0000","last_undegraded":"2020-11-20T11:32:25.488044+0000","last_fullsized":"2020-11-20T11:32:25.488044+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:10.515989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:10.515989+0000","last_clean_scrub_stamp":"2020-11-20T11:32:10.515989+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"24'1","reported_seq":"14","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.488459+0000","last_change":"2020-11-20T11:32:22.088160+0000","last_active":"2020-11-20T11:32:25.488459+0000","last_peered":"2020-11-20T11:32:25.488459+0000","last_clean":"2020-11-20T11:32:25.488459+0000","last_became_active":"2020-11-20T11:32:22.087777+0000","last_became_peered":"2020-11-20T11:32:22.087777+0000","last_unstale":"2020-11-20T11:32:25.488459+0000","last_undegraded":"2020-11-20T11:32:25.488459+0000","last_fullsized":"2020-11-20T11:32:25.488459+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.493247+0000","last_change":"2020-11-20T11:32:25.491667+0000","last_active":"2020-11-20T11:32:25.493247+0000","last_peered":"2020-11-20T11:32:25.493247+0000","last_clean":"2020-11-20T11:32:25.493247+0000","last_became_active":"2020-11-20T11:32:25.491100+0000","last_became_peered":"2020-11-20T11:32:25.491100+0000","last_unstale":"2020-11-20T11:32:25.493247+0000","last_undegraded":"2020-11-20T11:32:25.493247+0000","last_fullsized":"2020-11-20T11:32:25.493247+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"15","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.488045+0000","last_change":"2020-11-20T11:32:18.924600+0000","last_active":"2020-11-20T11:32:25.488045+0000","last_peered":"2020-11-20T11:32:25.488045+0000","last_clean":"2020-11-20T11:32:25.488045+0000","last_became_active":"2020-11-20T11:32:18.924324+0000","last_became_peered":"2020-11-20T11:32:18.924324+0000","last_unstale":"2020-11-20T11:32:25.488045+0000","last_undegraded":"2020-11-20T11:32:25.488045+0000","last_fullsized":"2020-11-20T11:32:25.488045+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"13","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.476386+0000","last_change":"2020-11-20T11:32:22.088716+0000","last_active":"2020-11-20T11:32:25.476386+0000","last_peered":"2020-11-20T11:32:25.476386+0000","last_clean":"2020-11-20T11:32:25.476386+0000","last_became_active":"2020-11-20T11:32:22.088414+0000","last_became_peered":"2020-11-20T11:32:22.088414+0000","last_unstale":"2020-11-20T11:32:25.476386+0000","last_undegraded":"2020-11-20T11:32:25.476386+0000","last_fullsized":"2020-11-20T11:32:25.476386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.493332+0000","last_change":"2020-11-20T11:32:25.491507+0000","last_active":"2020-11-20T11:32:25.493332+0000","last_peered":"2020-11-20T11:32:25.493332+0000","last_clean":"2020-11-20T11:32:25.493332+0000","last_became_active":"2020-11-20T11:32:25.490917+0000","last_became_peered":"2020-11-20T11:32:25.490917+0000","last_unstale":"2020-11-20T11:32:25.493332+0000","last_undegraded":"2020-11-20T11:32:25.493332+0000","last_fullsized":"2020-11-20T11:32:25.493332+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"15","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.488121+0000","last_change":"2020-11-20T11:32:19.482719+0000","last_active":"2020-11-20T11:32:25.488121+0000","last_peered":"2020-11-20T11:32:25.488121+0000","last_clean":"2020-11-20T11:32:25.488121+0000","last_became_active":"2020-11-20T11:32:19.482359+0000","last_became_peered":"2020-11-20T11:32:19.482359+0000","last_unstale":"2020-11-20T11:32:25.488121+0000","last_undegraded":"2020-11-20T11:32:25.488121+0000","last_fullsized":"2020-11-20T11:32:25.488121+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"24'8","reported_seq":"21","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.497370+0000","last_change":"2020-11-20T11:32:22.093099+0000","last_active":"2020-11-20T11:32:25.497370+0000","last_peered":"2020-11-20T11:32:25.497370+0000","last_clean":"2020-11-20T11:32:25.497370+0000","last_became_active":"2020-11-20T11:32:22.092619+0000","last_became_peered":"2020-11-20T11:32:22.092619+0000","last_unstale":"2020-11-20T11:32:25.497370+0000","last_undegraded":"2020-11-20T11:32:25.497370+0000","last_fullsized":"2020-11-20T11:32:25.497370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1535,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.493059+0000","last_change":"2020-11-20T11:32:25.492150+0000","last_active":"2020-11-20T11:32:25.493059+0000","last_peered":"2020-11-20T11:32:25.493059+0000","last_clean":"2020-11-20T11:32:25.493059+0000","last_became_active":"2020-11-20T11:32:25.491099+0000","last_became_peered":"2020-11-20T11:32:25.491099+0000","last_unstale":"2020-11-20T11:32:25.493059+0000","last_undegraded":"2020-11-20T11:32:25.493059+0000","last_fullsized":"2020-11-20T11:32:25.493059+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"24'6","reported_seq":"19","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.492963+0000","last_change":"2020-11-20T11:32:22.461697+0000","last_active":"2020-11-20T11:32:25.492963+0000","last_peered":"2020-11-20T11:32:25.492963+0000","last_clean":"2020-11-20T11:32:25.492963+0000","last_became_active":"2020-11-20T11:32:22.461331+0000","last_became_peered":"2020-11-20T11:32:22.461331+0000","last_unstale":"2020-11-20T11:32:25.492963+0000","last_undegraded":"2020-11-20T11:32:25.492963+0000","last_fullsized":"2020-11-20T11:32:25.492963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"16","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.480381+0000","last_change":"2020-11-20T11:32:18.923886+0000","last_active":"2020-11-20T11:32:25.480381+0000","last_peered":"2020-11-20T11:32:25.480381+0000","last_clean":"2020-11-20T11:32:25.480381+0000","last_became_active":"2020-11-20T11:32:18.923448+0000","last_became_peered":"2020-11-20T11:32:18.923448+0000","last_unstale":"2020-11-20T11:32:25.480381+0000","last_undegraded":"2020-11-20T11:32:25.480381+0000","last_fullsized":"2020-11-20T11:32:25.480381+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.498234+0000","last_change":"2020-11-20T11:32:25.491585+0000","last_active":"2020-11-20T11:32:25.498234+0000","last_peered":"2020-11-20T11:32:25.498234+0000","last_clean":"2020-11-20T11:32:25.498234+0000","last_became_active":"2020-11-20T11:32:25.491038+0000","last_became_peered":"2020-11-20T11:32:25.491038+0000","last_unstale":"2020-11-20T11:32:25.498234+0000","last_undegraded":"2020-11-20T11:32:25.498234+0000","last_fullsized":"2020-11-20T11:32:25.498234+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"15","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.473625+0000","last_change":"2020-11-20T11:32:18.924052+0000","last_active":"2020-11-20T11:32:25.473625+0000","last_peered":"2020-11-20T11:32:25.473625+0000","last_clean":"2020-11-20T11:32:25.473625+0000","last_became_active":"2020-11-20T11:32:18.923668+0000","last_became_peered":"2020-11-20T11:32:18.923668+0000","last_unstale":"2020-11-20T11:32:25.473625+0000","last_undegraded":"2020-11-20T11:32:25.473625+0000","last_fullsized":"2020-11-20T11:32:25.473625+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"25'4","reported_seq":"17","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:32:27.326193+0000","last_change":"2020-11-20T11:32:22.093019+0000","last_active":"2020-11-20T11:32:27.326193+0000","last_peered":"2020-11-20T11:32:27.326193+0000","last_clean":"2020-11-20T11:32:27.326193+0000","last_became_active":"2020-11-20T11:32:22.092604+0000","last_became_peered":"2020-11-20T11:32:22.092604+0000","last_unstale":"2020-11-20T11:32:27.326193+0000","last_undegraded":"2020-11-20T11:32:27.326193+0000","last_fullsized":"2020-11-20T11:32:27.326193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":8},{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":2304,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":6,"up_from":14,"seq":60129542151,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5064,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366776,"statfs":{"total":96636764160,"available":96631578624,"internally_reserved":0,"allocated":262144,"data_stored":107280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.232}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2030000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.415}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.337}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.621}]}]},{"osd":3,"up_from":14,"seq":60129542152,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":278528,"data_stored":108015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.871}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0149999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1140000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1679999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5132,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366708,"statfs":{"total":96636764160,"available":96631508992,"internally_reserved":0,"allocated":266240,"data_stored":107314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0960000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5196,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366644,"statfs":{"total":96636764160,"available":96631443456,"internally_reserved":0,"allocated":266240,"data_stored":107314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2350000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.373}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.331}]}]},{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":270336,"data_stored":107855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2949999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.361}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.325}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":270336,"data_stored":107440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.042}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0660000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.236}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.099}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.123}]}]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5204,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366636,"statfs":{"total":96636764160,"available":96631435264,"internally_reserved":0,"allocated":274432,"data_stored":108815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0640000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4019999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.26}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3779999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542151,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5140,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366700,"statfs":{"total":96636764160,"available":96631500800,"internally_reserved":0,"allocated":274432,"data_stored":108815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.458}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3080000000000001}]}]}]}} 2020-11-20T11:32:36.719 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-20T11:32:36.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.836+0000 7fbdc993b700 1 Processor -- start 2020-11-20T11:32:36.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.836+0000 7fbdc993b700 1 -- start start 2020-11-20T11:32:36.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.837+0000 7fbdc993b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbdc40aa1c0 0x7fbdc40ac290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:36.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.837+0000 7fbdc993b700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdc4067bc0 con 0x7fbdc4060a80 2020-11-20T11:32:36.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.837+0000 7fbdc993b700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdc4067e60 con 0x7fbdc40a9d40 2020-11-20T11:32:36.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.837+0000 7fbdc993b700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc40aa630 con 0x7fbdc40aa1c0 2020-11-20T11:32:36.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.837+0000 7fbdc2ffd700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fbdc4060a80 0x7fbdc4061950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55350/0 (socket says 172.21.15.23:55350) 2020-11-20T11:32:36.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.837+0000 7fbdc2ffd700 1 -- 172.21.15.23:0/3750756084 learned_addr learned my addr 172.21.15.23:0/3750756084 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:36.850 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.837+0000 7fbdc37fe700 1 --2- 172.21.15.23:0/3750756084 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbdc40aa1c0 0x7fbdc40ac290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:36.850 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.837+0000 7fbdc1ffb700 1 -- 172.21.15.23:0/3750756084 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673590675 0 0) 0x7fbdc4067bc0 con 0x7fbdc4060a80 2020-11-20T11:32:36.850 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.837+0000 7fbdc1ffb700 1 -- 172.21.15.23:0/3750756084 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdb0003280 con 0x7fbdc4060a80 2020-11-20T11:32:36.851 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.837+0000 7fbdc1ffb700 1 -- 172.21.15.23:0/3750756084 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3336270581 0 0) 0x7fbdb0003280 con 0x7fbdc4060a80 2020-11-20T11:32:36.851 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.837+0000 7fbdc1ffb700 1 -- 172.21.15.23:0/3750756084 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbdc40aa1c0 msgr2=0x7fbdc40ac290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:36.851 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.837+0000 7fbdc1ffb700 1 --2- 172.21.15.23:0/3750756084 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbdc40aa1c0 0x7fbdc40ac290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:36.852 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.837+0000 7fbdc1ffb700 1 -- 172.21.15.23:0/3750756084 >> v1:172.21.15.155:6789/0 conn(0x7fbdc40a9d40 legacy=0x7fbdc40abb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:36.852 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.837+0000 7fbdc1ffb700 1 -- 172.21.15.23:0/3750756084 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc4068100 con 0x7fbdc4060a80 2020-11-20T11:32:36.853 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.837+0000 7fbdc1ffb700 1 -- 172.21.15.23:0/3750756084 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7fbdb4001520 con 0x7fbdc4060a80 2020-11-20T11:32:36.853 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.837+0000 7fbdc1ffb700 1 -- 172.21.15.23:0/3750756084 >> v1:172.21.15.23:6789/0 conn(0x7fbdc4060a80 legacy=0x7fbdc4061950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:36.853 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.837+0000 7fbdc1ffb700 1 --2- 172.21.15.23:0/3750756084 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbdc40aa1c0 0x7fbdb00037a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:36.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.837+0000 7fbdc1ffb700 1 --2- 172.21.15.23:0/3750756084 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdb0003c90 0x7fbdb0004520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:36.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.837+0000 7fbdc1ffb700 1 --2- 172.21.15.23:0/3750756084 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbdb0004a10 0x7fbdb0004e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:36.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.837+0000 7fbdc1ffb700 1 -- 172.21.15.23:0/3750756084 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb00041b0 con 0x7fbdb0003c90 2020-11-20T11:32:36.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.837+0000 7fbdc1ffb700 1 -- 172.21.15.23:0/3750756084 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb0004320 con 0x7fbdb0004a10 2020-11-20T11:32:36.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.837+0000 7fbdc1ffb700 1 -- 172.21.15.23:0/3750756084 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb000b770 con 0x7fbdc40aa1c0 2020-11-20T11:32:36.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.838+0000 7fbdc27fc700 1 --2- 172.21.15.23:0/3750756084 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdb0003c90 0x7fbdb0004520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:36.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.838+0000 7fbdc2ffd700 1 --2- 172.21.15.23:0/3750756084 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbdc40aa1c0 0x7fbdb00037a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:36.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.838+0000 7fbdc37fe700 1 --2- 172.21.15.23:0/3750756084 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbdb0004a10 0x7fbdb0004e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:36.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.838+0000 7fbdc27fc700 1 -- 172.21.15.23:0/3750756084 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbdc40aa1c0 msgr2=0x7fbdb00037a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:36.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.838+0000 7fbdc27fc700 1 --2- 172.21.15.23:0/3750756084 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbdc40aa1c0 0x7fbdb00037a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:36.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.838+0000 7fbdc27fc700 1 -- 172.21.15.23:0/3750756084 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbdb0004a10 msgr2=0x7fbdb0004e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:36.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.838+0000 7fbdc27fc700 1 --2- 172.21.15.23:0/3750756084 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbdb0004a10 0x7fbdb0004e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:36.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.838+0000 7fbdc27fc700 1 -- 172.21.15.23:0/3750756084 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb0003280 con 0x7fbdb0003c90 2020-11-20T11:32:36.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.838+0000 7fbdc27fc700 1 --2- 172.21.15.23:0/3750756084 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdb0003c90 0x7fbdb0004520 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fbdb8001ac0 tx=0x7fbdb8007b90).ready entity=mon.0 client_cookie=f36623e11fe64dd7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:36.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.838+0000 7fbdc1ffb700 1 -- 172.21.15.23:0/3750756084 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb800c800 con 0x7fbdb0003c90 2020-11-20T11:32:36.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.838+0000 7fbdc1ffb700 1 -- 172.21.15.23:0/3750756084 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdb800c960 con 0x7fbdb0003c90 2020-11-20T11:32:36.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.838+0000 7fbdc1ffb700 1 -- 172.21.15.23:0/3750756084 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb800cd00 con 0x7fbdb0003c90 2020-11-20T11:32:36.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.839+0000 7fbdc993b700 1 -- 172.21.15.23:0/3750756084 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdb0003c90 msgr2=0x7fbdb0004520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:36.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.839+0000 7fbdc993b700 1 --2- 172.21.15.23:0/3750756084 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdb0003c90 0x7fbdb0004520 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fbdb8001ac0 tx=0x7fbdb8007b90).stop 2020-11-20T11:32:36.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.839+0000 7fbdc37fe700 1 -- 172.21.15.23:0/3750756084 reap_dead start 2020-11-20T11:32:36.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.839+0000 7fbdc993b700 1 -- 172.21.15.23:0/3750756084 shutdown_connections 2020-11-20T11:32:36.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.839+0000 7fbdc993b700 1 -- 172.21.15.23:0/3750756084 >> 172.21.15.23:0/3750756084 conn(0x7fbdc4066730 msgr2=0x7fbdc40601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:36.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.839+0000 7fbdc993b700 1 -- 172.21.15.23:0/3750756084 shutdown_connections 2020-11-20T11:32:36.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.839+0000 7fbdc993b700 1 -- 172.21.15.23:0/3750756084 wait complete. 2020-11-20T11:32:36.859 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.840+0000 7fbdc993b700 1 Processor -- start 2020-11-20T11:32:36.859 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.840+0000 7fbdc993b700 1 -- start start 2020-11-20T11:32:36.859 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.840+0000 7fbdc993b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbdb0003c90 0x7fbdc413da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:36.859 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.840+0000 7fbdc993b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdc414ab90 0x7fbdc413c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:36.859 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.840+0000 7fbdc993b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbdc40a9360 0x7fbdc413cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:36.860 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.840+0000 7fbdc993b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc414a1b0 con 0x7fbdc414ab90 2020-11-20T11:32:36.860 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.840+0000 7fbdc993b700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc414a450 con 0x7fbdc40a9360 2020-11-20T11:32:36.860 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.840+0000 7fbdc993b700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc414a6f0 con 0x7fbdb0003c90 2020-11-20T11:32:36.860 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.841+0000 7fbdc27fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdc414ab90 0x7fbdc413c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:36.860 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.841+0000 7fbdc27fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdc414ab90 0x7fbdc413c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41116/0 (socket says 172.21.15.23:41116) 2020-11-20T11:32:36.861 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.841+0000 7fbdc27fc700 1 -- 172.21.15.23:0/2226003937 learned_addr learned my addr 172.21.15.23:0/2226003937 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:36.861 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.841+0000 7fbdc2ffd700 1 --2- 172.21.15.23:0/2226003937 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbdb0003c90 0x7fbdc413da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:36.861 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.841+0000 7fbdc37fe700 1 --2- 172.21.15.23:0/2226003937 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbdc40a9360 0x7fbdc413cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:36.861 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.841+0000 7fbdc27fc700 1 -- 172.21.15.23:0/2226003937 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbdb0003c90 msgr2=0x7fbdc413da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:36.861 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.841+0000 7fbdc27fc700 1 --2- 172.21.15.23:0/2226003937 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbdb0003c90 0x7fbdc413da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:36.862 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.841+0000 7fbdc27fc700 1 -- 172.21.15.23:0/2226003937 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbdc40a9360 msgr2=0x7fbdc413cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:36.862 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.841+0000 7fbdc27fc700 1 --2- 172.21.15.23:0/2226003937 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbdc40a9360 0x7fbdc413cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:36.862 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.841+0000 7fbdc27fc700 1 -- 172.21.15.23:0/2226003937 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc413c650 con 0x7fbdc414ab90 2020-11-20T11:32:36.862 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.841+0000 7fbdc27fc700 1 --2- 172.21.15.23:0/2226003937 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdc414ab90 0x7fbdc413c160 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fbdb00042f0 tx=0x7fbdb8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:36.862 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.841+0000 7fbdaaffd700 1 -- 172.21.15.23:0/2226003937 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb801e470 con 0x7fbdc414ab90 2020-11-20T11:32:36.863 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.841+0000 7fbdaaffd700 1 -- 172.21.15.23:0/2226003937 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdb801e5d0 con 0x7fbdc414ab90 2020-11-20T11:32:36.863 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.841+0000 7fbdc993b700 1 -- 172.21.15.23:0/2226003937 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdc4149ea0 con 0x7fbdc414ab90 2020-11-20T11:32:36.863 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.841+0000 7fbdaaffd700 1 -- 172.21.15.23:0/2226003937 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb801e970 con 0x7fbdc414ab90 2020-11-20T11:32:36.863 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.841+0000 7fbdc993b700 1 -- 172.21.15.23:0/2226003937 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdc40a6c60 con 0x7fbdc414ab90 2020-11-20T11:32:36.863 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.842+0000 7fbdaaffd700 1 -- 172.21.15.23:0/2226003937 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbdb8025030 con 0x7fbdc414ab90 2020-11-20T11:32:36.864 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.842+0000 7fbdc993b700 1 -- 172.21.15.23:0/2226003937 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdc414a8b0 con 0x7fbdc414ab90 2020-11-20T11:32:36.864 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.842+0000 7fbdaaffd700 1 --2- 172.21.15.23:0/2226003937 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbda0056d60 0x7fbda0059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:36.864 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.843+0000 7fbdaaffd700 1 -- 172.21.15.23:0/2226003937 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fbdb8071b00 con 0x7fbdc414ab90 2020-11-20T11:32:36.864 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.843+0000 7fbdc2ffd700 1 --2- 172.21.15.23:0/2226003937 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbda0056d60 0x7fbda0059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:36.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.843+0000 7fbdc2ffd700 1 --2- 172.21.15.23:0/2226003937 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbda0056d60 0x7fbda0059220 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fbdb4001650 tx=0x7fbdb4009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:36.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.846+0000 7fbdaaffd700 1 -- 172.21.15.23:0/2226003937 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fbdb8046df0 con 0x7fbdc414ab90 2020-11-20T11:32:36.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.988+0000 7fbdc993b700 1 -- 172.21.15.23:0/2226003937 --> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbdc4044710 con 0x7fbda0056d60 2020-11-20T11:32:36.995 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.991+0000 7fbdaaffd700 1 -- 172.21.15.23:0/2226003937 <== mgr.4105 v2:172.21.15.155:6832/22693 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+82153 (secure 0 0 0) 0x7fbdc4044710 con 0x7fbda0056d60 2020-11-20T11:32:36.995 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:32:36.999 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.993+0000 7fbdc993b700 1 -- 172.21.15.23:0/2226003937 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbda0056d60 msgr2=0x7fbda0059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:36.999 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.993+0000 7fbdc993b700 1 --2- 172.21.15.23:0/2226003937 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbda0056d60 0x7fbda0059220 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fbdb4001650 tx=0x7fbdb4009040).stop 2020-11-20T11:32:37.000 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.993+0000 7fbdc993b700 1 -- 172.21.15.23:0/2226003937 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdc414ab90 msgr2=0x7fbdc413c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:37.000 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.993+0000 7fbdc993b700 1 --2- 172.21.15.23:0/2226003937 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdc414ab90 0x7fbdc413c160 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fbdb00042f0 tx=0x7fbdb8016000).stop 2020-11-20T11:32:37.000 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.993+0000 7fbdc993b700 1 -- 172.21.15.23:0/2226003937 shutdown_connections 2020-11-20T11:32:37.001 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.993+0000 7fbdc993b700 1 --2- 172.21.15.23:0/2226003937 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbdc40a9360 0x7fbdc413cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:37.001 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.993+0000 7fbdc993b700 1 --2- 172.21.15.23:0/2226003937 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbda0056d60 0x7fbda0059220 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:37.002 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.993+0000 7fbdc993b700 1 --2- 172.21.15.23:0/2226003937 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbdb0003c90 0x7fbdc413da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:37.002 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.993+0000 7fbdc993b700 1 --2- 172.21.15.23:0/2226003937 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdc414ab90 0x7fbdc413c160 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:37.003 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.993+0000 7fbdc993b700 1 -- 172.21.15.23:0/2226003937 >> 172.21.15.23:0/2226003937 conn(0x7fbdc4066730 msgr2=0x7fbdc413b560 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:37.003 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.993+0000 7fbdc993b700 1 -- 172.21.15.23:0/2226003937 shutdown_connections 2020-11-20T11:32:37.004 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:36.993+0000 7fbdc993b700 1 -- 172.21.15.23:0/2226003937 wait complete. 2020-11-20T11:32:37.004 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2020-11-20T11:32:37.007 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2020-11-20T11:32:35.525404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2304,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40,"ondisk_log_size":40,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":66,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":41152,"kb_used_data":2112,"kb_used_omap":0,"kb_used_meta":38976,"kb_avail":754933568,"statfs":{"total":773094113280,"available":773051973632,"internally_reserved":0,"allocated":2162688,"data_stored":862848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":39911424},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2304,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002508"},"pg_stats":[{"pgid":"4.2","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.480587+0000","last_change":"2020-11-20T11:32:25.125195+0000","last_active":"2020-11-20T11:32:25.480587+0000","last_peered":"2020-11-20T11:32:25.480587+0000","last_clean":"2020-11-20T11:32:25.480587+0000","last_became_active":"2020-11-20T11:32:25.124627+0000","last_became_peered":"2020-11-20T11:32:25.124627+0000","last_unstale":"2020-11-20T11:32:25.480587+0000","last_undegraded":"2020-11-20T11:32:25.480587+0000","last_fullsized":"2020-11-20T11:32:25.480587+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"16","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.480586+0000","last_change":"2020-11-20T11:32:19.482604+0000","last_active":"2020-11-20T11:32:25.480586+0000","last_peered":"2020-11-20T11:32:25.480586+0000","last_clean":"2020-11-20T11:32:25.480586+0000","last_became_active":"2020-11-20T11:32:19.482197+0000","last_became_peered":"2020-11-20T11:32:19.482197+0000","last_unstale":"2020-11-20T11:32:25.480586+0000","last_undegraded":"2020-11-20T11:32:25.480586+0000","last_fullsized":"2020-11-20T11:32:25.480586+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"24'6","reported_seq":"19","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.494434+0000","last_change":"2020-11-20T11:32:22.098576+0000","last_active":"2020-11-20T11:32:25.494434+0000","last_peered":"2020-11-20T11:32:25.494434+0000","last_clean":"2020-11-20T11:32:25.494434+0000","last_became_active":"2020-11-20T11:32:22.098264+0000","last_became_peered":"2020-11-20T11:32:22.098264+0000","last_unstale":"2020-11-20T11:32:25.494434+0000","last_undegraded":"2020-11-20T11:32:25.494434+0000","last_fullsized":"2020-11-20T11:32:25.494434+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.483873+0000","last_change":"2020-11-20T11:32:25.479776+0000","last_active":"2020-11-20T11:32:25.483873+0000","last_peered":"2020-11-20T11:32:25.483873+0000","last_clean":"2020-11-20T11:32:25.483873+0000","last_became_active":"2020-11-20T11:32:25.479350+0000","last_became_peered":"2020-11-20T11:32:25.479350+0000","last_unstale":"2020-11-20T11:32:25.483873+0000","last_undegraded":"2020-11-20T11:32:25.483873+0000","last_fullsized":"2020-11-20T11:32:25.483873+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"16","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.479488+0000","last_change":"2020-11-20T11:32:18.924328+0000","last_active":"2020-11-20T11:32:25.479488+0000","last_peered":"2020-11-20T11:32:25.479488+0000","last_clean":"2020-11-20T11:32:25.479488+0000","last_became_active":"2020-11-20T11:32:18.923879+0000","last_became_peered":"2020-11-20T11:32:18.923879+0000","last_unstale":"2020-11-20T11:32:25.479488+0000","last_undegraded":"2020-11-20T11:32:25.479488+0000","last_fullsized":"2020-11-20T11:32:25.479488+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"24'5","reported_seq":"18","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.488715+0000","last_change":"2020-11-20T11:32:22.093559+0000","last_active":"2020-11-20T11:32:25.488715+0000","last_peered":"2020-11-20T11:32:25.488715+0000","last_clean":"2020-11-20T11:32:25.488715+0000","last_became_active":"2020-11-20T11:32:22.093123+0000","last_became_peered":"2020-11-20T11:32:22.093123+0000","last_unstale":"2020-11-20T11:32:25.488715+0000","last_undegraded":"2020-11-20T11:32:25.488715+0000","last_fullsized":"2020-11-20T11:32:25.488715+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.481419+0000","last_change":"2020-11-20T11:32:25.125052+0000","last_active":"2020-11-20T11:32:25.481419+0000","last_peered":"2020-11-20T11:32:25.481419+0000","last_clean":"2020-11-20T11:32:25.481419+0000","last_became_active":"2020-11-20T11:32:25.124401+0000","last_became_peered":"2020-11-20T11:32:25.124401+0000","last_unstale":"2020-11-20T11:32:25.481419+0000","last_undegraded":"2020-11-20T11:32:25.481419+0000","last_fullsized":"2020-11-20T11:32:25.481419+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"15","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.474417+0000","last_change":"2020-11-20T11:32:18.927388+0000","last_active":"2020-11-20T11:32:25.474417+0000","last_peered":"2020-11-20T11:32:25.474417+0000","last_clean":"2020-11-20T11:32:25.474417+0000","last_became_active":"2020-11-20T11:32:18.926981+0000","last_became_peered":"2020-11-20T11:32:18.926981+0000","last_unstale":"2020-11-20T11:32:25.474417+0000","last_undegraded":"2020-11-20T11:32:25.474417+0000","last_fullsized":"2020-11-20T11:32:25.474417+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"24'8","reported_seq":"21","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.492278+0000","last_change":"2020-11-20T11:32:22.088128+0000","last_active":"2020-11-20T11:32:25.492278+0000","last_peered":"2020-11-20T11:32:25.492278+0000","last_clean":"2020-11-20T11:32:25.492278+0000","last_became_active":"2020-11-20T11:32:22.087723+0000","last_became_peered":"2020-11-20T11:32:22.087723+0000","last_unstale":"2020-11-20T11:32:25.492278+0000","last_undegraded":"2020-11-20T11:32:25.492278+0000","last_fullsized":"2020-11-20T11:32:25.492278+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.495311+0000","last_change":"2020-11-20T11:32:25.492514+0000","last_active":"2020-11-20T11:32:25.495311+0000","last_peered":"2020-11-20T11:32:25.495311+0000","last_clean":"2020-11-20T11:32:25.495311+0000","last_became_active":"2020-11-20T11:32:25.491657+0000","last_became_peered":"2020-11-20T11:32:25.491657+0000","last_unstale":"2020-11-20T11:32:25.495311+0000","last_undegraded":"2020-11-20T11:32:25.495311+0000","last_fullsized":"2020-11-20T11:32:25.495311+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"16","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.479536+0000","last_change":"2020-11-20T11:32:18.927044+0000","last_active":"2020-11-20T11:32:25.479536+0000","last_peered":"2020-11-20T11:32:25.479536+0000","last_clean":"2020-11-20T11:32:25.479536+0000","last_became_active":"2020-11-20T11:32:18.926711+0000","last_became_peered":"2020-11-20T11:32:18.926711+0000","last_unstale":"2020-11-20T11:32:25.479536+0000","last_undegraded":"2020-11-20T11:32:25.479536+0000","last_fullsized":"2020-11-20T11:32:25.479536+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'2","reported_seq":"20","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.488044+0000","last_change":"2020-11-20T11:32:14.446049+0000","last_active":"2020-11-20T11:32:25.488044+0000","last_peered":"2020-11-20T11:32:25.488044+0000","last_clean":"2020-11-20T11:32:25.488044+0000","last_became_active":"2020-11-20T11:32:14.445477+0000","last_became_peered":"2020-11-20T11:32:14.445477+0000","last_unstale":"2020-11-20T11:32:25.488044+0000","last_undegraded":"2020-11-20T11:32:25.488044+0000","last_fullsized":"2020-11-20T11:32:25.488044+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:10.515989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:10.515989+0000","last_clean_scrub_stamp":"2020-11-20T11:32:10.515989+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"24'1","reported_seq":"14","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.488459+0000","last_change":"2020-11-20T11:32:22.088160+0000","last_active":"2020-11-20T11:32:25.488459+0000","last_peered":"2020-11-20T11:32:25.488459+0000","last_clean":"2020-11-20T11:32:25.488459+0000","last_became_active":"2020-11-20T11:32:22.087777+0000","last_became_peered":"2020-11-20T11:32:22.087777+0000","last_unstale":"2020-11-20T11:32:25.488459+0000","last_undegraded":"2020-11-20T11:32:25.488459+0000","last_fullsized":"2020-11-20T11:32:25.488459+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.493247+0000","last_change":"2020-11-20T11:32:25.491667+0000","last_active":"2020-11-20T11:32:25.493247+0000","last_peered":"2020-11-20T11:32:25.493247+0000","last_clean":"2020-11-20T11:32:25.493247+0000","last_became_active":"2020-11-20T11:32:25.491100+0000","last_became_peered":"2020-11-20T11:32:25.491100+0000","last_unstale":"2020-11-20T11:32:25.493247+0000","last_undegraded":"2020-11-20T11:32:25.493247+0000","last_fullsized":"2020-11-20T11:32:25.493247+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"15","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.488045+0000","last_change":"2020-11-20T11:32:18.924600+0000","last_active":"2020-11-20T11:32:25.488045+0000","last_peered":"2020-11-20T11:32:25.488045+0000","last_clean":"2020-11-20T11:32:25.488045+0000","last_became_active":"2020-11-20T11:32:18.924324+0000","last_became_peered":"2020-11-20T11:32:18.924324+0000","last_unstale":"2020-11-20T11:32:25.488045+0000","last_undegraded":"2020-11-20T11:32:25.488045+0000","last_fullsized":"2020-11-20T11:32:25.488045+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"13","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.476386+0000","last_change":"2020-11-20T11:32:22.088716+0000","last_active":"2020-11-20T11:32:25.476386+0000","last_peered":"2020-11-20T11:32:25.476386+0000","last_clean":"2020-11-20T11:32:25.476386+0000","last_became_active":"2020-11-20T11:32:22.088414+0000","last_became_peered":"2020-11-20T11:32:22.088414+0000","last_unstale":"2020-11-20T11:32:25.476386+0000","last_undegraded":"2020-11-20T11:32:25.476386+0000","last_fullsized":"2020-11-20T11:32:25.476386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.493332+0000","last_change":"2020-11-20T11:32:25.491507+0000","last_active":"2020-11-20T11:32:25.493332+0000","last_peered":"2020-11-20T11:32:25.493332+0000","last_clean":"2020-11-20T11:32:25.493332+0000","last_became_active":"2020-11-20T11:32:25.490917+0000","last_became_peered":"2020-11-20T11:32:25.490917+0000","last_unstale":"2020-11-20T11:32:25.493332+0000","last_undegraded":"2020-11-20T11:32:25.493332+0000","last_fullsized":"2020-11-20T11:32:25.493332+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"15","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.488121+0000","last_change":"2020-11-20T11:32:19.482719+0000","last_active":"2020-11-20T11:32:25.488121+0000","last_peered":"2020-11-20T11:32:25.488121+0000","last_clean":"2020-11-20T11:32:25.488121+0000","last_became_active":"2020-11-20T11:32:19.482359+0000","last_became_peered":"2020-11-20T11:32:19.482359+0000","last_unstale":"2020-11-20T11:32:25.488121+0000","last_undegraded":"2020-11-20T11:32:25.488121+0000","last_fullsized":"2020-11-20T11:32:25.488121+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"24'8","reported_seq":"21","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.497370+0000","last_change":"2020-11-20T11:32:22.093099+0000","last_active":"2020-11-20T11:32:25.497370+0000","last_peered":"2020-11-20T11:32:25.497370+0000","last_clean":"2020-11-20T11:32:25.497370+0000","last_became_active":"2020-11-20T11:32:22.092619+0000","last_became_peered":"2020-11-20T11:32:22.092619+0000","last_unstale":"2020-11-20T11:32:25.497370+0000","last_undegraded":"2020-11-20T11:32:25.497370+0000","last_fullsized":"2020-11-20T11:32:25.497370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1535,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.493059+0000","last_change":"2020-11-20T11:32:25.492150+0000","last_active":"2020-11-20T11:32:25.493059+0000","last_peered":"2020-11-20T11:32:25.493059+0000","last_clean":"2020-11-20T11:32:25.493059+0000","last_became_active":"2020-11-20T11:32:25.491099+0000","last_became_peered":"2020-11-20T11:32:25.491099+0000","last_unstale":"2020-11-20T11:32:25.493059+0000","last_undegraded":"2020-11-20T11:32:25.493059+0000","last_fullsized":"2020-11-20T11:32:25.493059+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"24'6","reported_seq":"19","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.492963+0000","last_change":"2020-11-20T11:32:22.461697+0000","last_active":"2020-11-20T11:32:25.492963+0000","last_peered":"2020-11-20T11:32:25.492963+0000","last_clean":"2020-11-20T11:32:25.492963+0000","last_became_active":"2020-11-20T11:32:22.461331+0000","last_became_peered":"2020-11-20T11:32:22.461331+0000","last_unstale":"2020-11-20T11:32:25.492963+0000","last_undegraded":"2020-11-20T11:32:25.492963+0000","last_fullsized":"2020-11-20T11:32:25.492963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"16","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.480381+0000","last_change":"2020-11-20T11:32:18.923886+0000","last_active":"2020-11-20T11:32:25.480381+0000","last_peered":"2020-11-20T11:32:25.480381+0000","last_clean":"2020-11-20T11:32:25.480381+0000","last_became_active":"2020-11-20T11:32:18.923448+0000","last_became_peered":"2020-11-20T11:32:18.923448+0000","last_unstale":"2020-11-20T11:32:25.480381+0000","last_undegraded":"2020-11-20T11:32:25.480381+0000","last_fullsized":"2020-11-20T11:32:25.480381+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.498234+0000","last_change":"2020-11-20T11:32:25.491585+0000","last_active":"2020-11-20T11:32:25.498234+0000","last_peered":"2020-11-20T11:32:25.498234+0000","last_clean":"2020-11-20T11:32:25.498234+0000","last_became_active":"2020-11-20T11:32:25.491038+0000","last_became_peered":"2020-11-20T11:32:25.491038+0000","last_unstale":"2020-11-20T11:32:25.498234+0000","last_undegraded":"2020-11-20T11:32:25.498234+0000","last_fullsized":"2020-11-20T11:32:25.498234+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"15","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.473625+0000","last_change":"2020-11-20T11:32:18.924052+0000","last_active":"2020-11-20T11:32:25.473625+0000","last_peered":"2020-11-20T11:32:25.473625+0000","last_clean":"2020-11-20T11:32:25.473625+0000","last_became_active":"2020-11-20T11:32:18.923668+0000","last_became_peered":"2020-11-20T11:32:18.923668+0000","last_unstale":"2020-11-20T11:32:25.473625+0000","last_undegraded":"2020-11-20T11:32:25.473625+0000","last_fullsized":"2020-11-20T11:32:25.473625+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"25'4","reported_seq":"17","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:32:27.326193+0000","last_change":"2020-11-20T11:32:22.093019+0000","last_active":"2020-11-20T11:32:27.326193+0000","last_peered":"2020-11-20T11:32:27.326193+0000","last_clean":"2020-11-20T11:32:27.326193+0000","last_became_active":"2020-11-20T11:32:22.092604+0000","last_became_peered":"2020-11-20T11:32:22.092604+0000","last_unstale":"2020-11-20T11:32:27.326193+0000","last_undegraded":"2020-11-20T11:32:27.326193+0000","last_fullsized":"2020-11-20T11:32:27.326193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":8},{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":2304,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":6,"up_from":14,"seq":60129542151,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5064,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366776,"statfs":{"total":96636764160,"available":96631578624,"internally_reserved":0,"allocated":262144,"data_stored":107280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.232}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2030000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.415}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.337}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.621}]}]},{"osd":3,"up_from":14,"seq":60129542152,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":278528,"data_stored":108015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.871}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0149999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1140000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1679999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5132,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366708,"statfs":{"total":96636764160,"available":96631508992,"internally_reserved":0,"allocated":266240,"data_stored":107314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0960000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5196,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366644,"statfs":{"total":96636764160,"available":96631443456,"internally_reserved":0,"allocated":266240,"data_stored":107314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2350000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.373}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.331}]}]},{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":270336,"data_stored":107855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2949999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.361}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.325}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":270336,"data_stored":107440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.042}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0660000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.236}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.099}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.123}]}]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5204,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366636,"statfs":{"total":96636764160,"available":96631435264,"internally_reserved":0,"allocated":274432,"data_stored":108815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0640000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4019999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.26}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3779999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542151,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5140,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366700,"statfs":{"total":96636764160,"available":96631500800,"internally_reserved":0,"allocated":274432,"data_stored":108815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.458}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3080000000000001}]}]}]}} 2020-11-20T11:32:37.009 INFO:tasks.ceph.ceph_manager.ceph:clean! 2020-11-20T11:32:37.009 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2020-11-20T11:32:37.010 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2020-11-20T11:32:37.010 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2020-11-20T11:32:37.124 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.121+0000 7f29cd50b700 1 Processor -- start 2020-11-20T11:32:37.124 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.121+0000 7f29cd50b700 1 -- start start 2020-11-20T11:32:37.126 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.121+0000 7f29cd50b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f29c80ae760 0x7f29c813e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:37.127 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.121+0000 7f29cd50b700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f29c8065100 0x7f29c8145610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:37.127 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.121+0000 7f29cd50b700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c80661b0 con 0x7f29c80aeb40 2020-11-20T11:32:37.127 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.121+0000 7f29cd50b700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29c813ad60 con 0x7f29c80ae760 2020-11-20T11:32:37.127 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.121+0000 7f29cd50b700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29c8066c70 con 0x7f29c8065100 2020-11-20T11:32:37.128 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.122+0000 7f29c6ffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f29c80ae760 0x7f29c813e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:37.128 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.122+0000 7f29c67fc700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f29c80aeb40 0x7f29c8137ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55370/0 (socket says 172.21.15.23:55370) 2020-11-20T11:32:37.128 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.122+0000 7f29c67fc700 1 -- 172.21.15.23:0/1668033325 learned_addr learned my addr 172.21.15.23:0/1668033325 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:37.128 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.122+0000 7f29c77fe700 1 --2- 172.21.15.23:0/1668033325 >> v2:172.21.15.155:3300/0 conn(0x7f29c8065100 0x7f29c8145610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:37.129 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.122+0000 7f29c6ffd700 1 -- 172.21.15.23:0/1668033325 >> v2:172.21.15.155:3300/0 conn(0x7f29c8065100 msgr2=0x7f29c8145610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:37.129 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.122+0000 7f29c6ffd700 1 --2- 172.21.15.23:0/1668033325 >> v2:172.21.15.155:3300/0 conn(0x7f29c8065100 0x7f29c8145610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:37.129 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.122+0000 7f29c6ffd700 1 -- 172.21.15.23:0/1668033325 >> v1:172.21.15.23:6789/0 conn(0x7f29c80aeb40 legacy=0x7f29c8137ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:37.129 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.122+0000 7f29c6ffd700 1 -- 172.21.15.23:0/1668033325 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c813e470 con 0x7f29c80ae760 2020-11-20T11:32:37.129 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.122+0000 7f29c5ffb700 1 -- 172.21.15.23:0/1668033325 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 966564425 0 0) 0x7f29c80661b0 con 0x7f29c80aeb40 2020-11-20T11:32:37.130 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.122+0000 7f29c6ffd700 1 --2- 172.21.15.23:0/1668033325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c80ae760 0x7f29c813e910 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f29c80664c0 tx=0x7f29b0006e30).ready entity=mon.0 client_cookie=9625b0f930773a2e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:37.130 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.122+0000 7f29c5ffb700 1 -- 172.21.15.23:0/1668033325 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b00097b0 con 0x7f29c80ae760 2020-11-20T11:32:37.130 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.122+0000 7f29c5ffb700 1 -- 172.21.15.23:0/1668033325 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b0000e90 con 0x7f29c80ae760 2020-11-20T11:32:37.130 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.122+0000 7f29c5ffb700 1 -- 172.21.15.23:0/1668033325 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b0009b20 con 0x7f29c80ae760 2020-11-20T11:32:37.130 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.123+0000 7f29cd50b700 1 -- 172.21.15.23:0/1668033325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c80ae760 msgr2=0x7f29c813e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:37.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.123+0000 7f29cd50b700 1 --2- 172.21.15.23:0/1668033325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c80ae760 0x7f29c813e910 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f29c80664c0 tx=0x7f29b0006e30).stop 2020-11-20T11:32:37.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.123+0000 7f29cd50b700 1 -- 172.21.15.23:0/1668033325 shutdown_connections 2020-11-20T11:32:37.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.123+0000 7f29cd50b700 1 --2- 172.21.15.23:0/1668033325 >> v2:172.21.15.155:3300/0 conn(0x7f29c8065100 0x7f29c8145610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:37.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.123+0000 7f29cd50b700 1 --2- 172.21.15.23:0/1668033325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c80ae760 0x7f29c813e910 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:37.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.123+0000 7f29cd50b700 1 -- 172.21.15.23:0/1668033325 >> 172.21.15.23:0/1668033325 conn(0x7f29c8061d80 msgr2=0x7f29c80ae050 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:37.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.123+0000 7f29cd50b700 1 -- 172.21.15.23:0/1668033325 shutdown_connections 2020-11-20T11:32:37.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.123+0000 7f29cd50b700 1 -- 172.21.15.23:0/1668033325 wait complete. 2020-11-20T11:32:37.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.124+0000 7f29cd50b700 1 Processor -- start 2020-11-20T11:32:37.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.124+0000 7f29cd50b700 1 -- start start 2020-11-20T11:32:37.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.124+0000 7f29cd50b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29c8065100 0x7f29c80ad2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:37.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.124+0000 7f29cd50b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29c80ae760 0x7f29c814daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:37.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.124+0000 7f29cd50b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c80aeb40 0x7f29c813d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:37.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.124+0000 7f29cd50b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c813c190 con 0x7f29c80aeb40 2020-11-20T11:32:37.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.124+0000 7f29cd50b700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c81341a0 con 0x7f29c80ae760 2020-11-20T11:32:37.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.124+0000 7f29cd50b700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29c814e4d0 con 0x7f29c8065100 2020-11-20T11:32:37.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.124+0000 7f29c77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c80aeb40 0x7f29c813d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:37.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.125+0000 7f29c77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c80aeb40 0x7f29c813d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41130/0 (socket says 172.21.15.23:41130) 2020-11-20T11:32:37.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.125+0000 7f29c77fe700 1 -- 172.21.15.23:0/837954701 learned_addr learned my addr 172.21.15.23:0/837954701 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:37.137 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.125+0000 7f29c6ffd700 1 --2- 172.21.15.23:0/837954701 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29c8065100 0x7f29c80ad2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:37.137 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.125+0000 7f29c67fc700 1 --2- 172.21.15.23:0/837954701 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29c80ae760 0x7f29c814daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:37.137 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.125+0000 7f29c77fe700 1 -- 172.21.15.23:0/837954701 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29c8065100 msgr2=0x7f29c80ad2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:37.137 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.125+0000 7f29c77fe700 1 --2- 172.21.15.23:0/837954701 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29c8065100 0x7f29c80ad2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:37.137 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.125+0000 7f29c77fe700 1 -- 172.21.15.23:0/837954701 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29c80ae760 msgr2=0x7f29c814daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:37.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.125+0000 7f29c77fe700 1 --2- 172.21.15.23:0/837954701 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29c80ae760 0x7f29c814daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:37.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.125+0000 7f29c77fe700 1 -- 172.21.15.23:0/837954701 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c814e230 con 0x7f29c80aeb40 2020-11-20T11:32:37.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.125+0000 7f29c77fe700 1 --2- 172.21.15.23:0/837954701 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c80aeb40 0x7f29c813d700 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f29c000cb80 tx=0x7f29c0008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:37.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.125+0000 7f29beffd700 1 -- 172.21.15.23:0/837954701 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c0007740 con 0x7f29c80aeb40 2020-11-20T11:32:37.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.125+0000 7f29beffd700 1 -- 172.21.15.23:0/837954701 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29c00078a0 con 0x7f29c80aeb40 2020-11-20T11:32:37.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.125+0000 7f29cd50b700 1 -- 172.21.15.23:0/837954701 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29c814df90 con 0x7f29c80aeb40 2020-11-20T11:32:37.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.125+0000 7f29beffd700 1 -- 172.21.15.23:0/837954701 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c0007c20 con 0x7f29c80aeb40 2020-11-20T11:32:37.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.125+0000 7f29cd50b700 1 -- 172.21.15.23:0/837954701 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29c8138b30 con 0x7f29c80aeb40 2020-11-20T11:32:37.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.126+0000 7f29cd50b700 1 -- 172.21.15.23:0/837954701 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29c8060920 con 0x7f29c80aeb40 2020-11-20T11:32:37.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.126+0000 7f29beffd700 1 -- 172.21.15.23:0/837954701 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f29c001b460 con 0x7f29c80aeb40 2020-11-20T11:32:37.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.127+0000 7f29beffd700 1 --2- 172.21.15.23:0/837954701 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f29a4056d90 0x7f29a4059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:37.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.127+0000 7f29beffd700 1 -- 172.21.15.23:0/837954701 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f29c0070150 con 0x7f29c80aeb40 2020-11-20T11:32:37.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.127+0000 7f29c6ffd700 1 --2- 172.21.15.23:0/837954701 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f29a4056d90 0x7f29a4059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:37.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.127+0000 7f29c6ffd700 1 --2- 172.21.15.23:0/837954701 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f29a4056d90 0x7f29a4059250 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f29b0006e70 tx=0x7f29b0015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:37.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.130+0000 7f29beffd700 1 -- 172.21.15.23:0/837954701 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f29c00453e0 con 0x7f29c80aeb40 2020-11-20T11:32:37.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.336+0000 7f29cd50b700 1 -- 172.21.15.23:0/837954701 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f29c8138b30 con 0x7f29c80aeb40 2020-11-20T11:32:37.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.337+0000 7f29beffd700 1 -- 172.21.15.23:0/837954701 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+45 (secure 0 0 0) 0x7f29c8138b30 con 0x7f29c80aeb40 2020-11-20T11:32:37.340 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:32:37.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.338+0000 7f29cd50b700 1 -- 172.21.15.23:0/837954701 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f29a4056d90 msgr2=0x7f29a4059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:37.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.338+0000 7f29cd50b700 1 --2- 172.21.15.23:0/837954701 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f29a4056d90 0x7f29a4059250 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f29b0006e70 tx=0x7f29b0015000).stop 2020-11-20T11:32:37.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.338+0000 7f29cd50b700 1 -- 172.21.15.23:0/837954701 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c80aeb40 msgr2=0x7f29c813d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:37.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.338+0000 7f29cd50b700 1 --2- 172.21.15.23:0/837954701 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c80aeb40 0x7f29c813d700 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f29c000cb80 tx=0x7f29c0008040).stop 2020-11-20T11:32:37.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.338+0000 7f29cd50b700 1 -- 172.21.15.23:0/837954701 shutdown_connections 2020-11-20T11:32:37.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.338+0000 7f29cd50b700 1 --2- 172.21.15.23:0/837954701 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c80aeb40 0x7f29c813d700 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:37.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.338+0000 7f29cd50b700 1 --2- 172.21.15.23:0/837954701 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f29a4056d90 0x7f29a4059250 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:37.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.338+0000 7f29cd50b700 1 --2- 172.21.15.23:0/837954701 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f29c8065100 0x7f29c80ad2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:37.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.338+0000 7f29cd50b700 1 --2- 172.21.15.23:0/837954701 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f29c80ae760 0x7f29c814daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:37.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.338+0000 7f29cd50b700 1 -- 172.21.15.23:0/837954701 >> 172.21.15.23:0/837954701 conn(0x7f29c8061d80 msgr2=0x7f29c80677b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:37.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.339+0000 7f29cd50b700 1 -- 172.21.15.23:0/837954701 shutdown_connections 2020-11-20T11:32:37.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.339+0000 7f29cd50b700 1 -- 172.21.15.23:0/837954701 wait complete. 2020-11-20T11:32:37.352 INFO:teuthology.orchestra.run.smithi023.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-11-20T11:32:37.353 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2020-11-20T11:32:37.353 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T11:32:37.467 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.464+0000 7f02d427d700 1 Processor -- start 2020-11-20T11:32:37.467 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.464+0000 7f02d427d700 1 -- start start 2020-11-20T11:32:37.469 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.465+0000 7f02d427d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f02cc0a3210 0x7f02cc0a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:37.470 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.465+0000 7f02d427d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f02cc13e870 0x7f02cc13ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:37.470 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.465+0000 7f02d427d700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02cc0a9b00 con 0x7f02cc0a35f0 2020-11-20T11:32:37.470 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.465+0000 7f02d427d700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02cc0a1da0 con 0x7f02cc0a3210 2020-11-20T11:32:37.470 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.465+0000 7f02d427d700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc0aa710 con 0x7f02cc13e870 2020-11-20T11:32:37.470 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.465+0000 7f02d2019700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f02cc0a3210 0x7f02cc0a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:37.471 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.465+0000 7f02d1818700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f02cc0a35f0 0x7f02cc0a9f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55384/0 (socket says 172.21.15.23:55384) 2020-11-20T11:32:37.471 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.465+0000 7f02d1818700 1 -- 172.21.15.23:0/622028959 learned_addr learned my addr 172.21.15.23:0/622028959 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:37.471 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.465+0000 7f02d281a700 1 --2- 172.21.15.23:0/622028959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f02cc13e870 0x7f02cc13ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:37.471 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.465+0000 7f02d2019700 1 -- 172.21.15.23:0/622028959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f02cc13e870 msgr2=0x7f02cc13ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:37.471 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.465+0000 7f02d2019700 1 --2- 172.21.15.23:0/622028959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f02cc13e870 0x7f02cc13ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:37.472 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.465+0000 7f02d2019700 1 -- 172.21.15.23:0/622028959 >> v1:172.21.15.23:6789/0 conn(0x7f02cc0a35f0 legacy=0x7f02cc0a9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:37.472 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.465+0000 7f02d2019700 1 -- 172.21.15.23:0/622028959 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02cc0aa9b0 con 0x7f02cc0a3210 2020-11-20T11:32:37.472 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.465+0000 7f02d281a700 1 --2- 172.21.15.23:0/622028959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f02cc13e870 0x7f02cc13ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:37.472 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.466+0000 7f02d2019700 1 --2- 172.21.15.23:0/622028959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02cc0a3210 0x7f02cc0a18b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f02bc006fd0 tx=0x7f02bc006e30).ready entity=mon.0 client_cookie=f7dadc03a3dd7588 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:37.473 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.466+0000 7f02d1017700 1 -- 172.21.15.23:0/622028959 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02bc0097e0 con 0x7f02cc0a3210 2020-11-20T11:32:37.473 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.466+0000 7f02d1017700 1 -- 172.21.15.23:0/622028959 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02bc000e90 con 0x7f02cc0a3210 2020-11-20T11:32:37.473 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.466+0000 7f02d1017700 1 -- 172.21.15.23:0/622028959 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02bc009b50 con 0x7f02cc0a3210 2020-11-20T11:32:37.473 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.466+0000 7f02d427d700 1 -- 172.21.15.23:0/622028959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02cc0a3210 msgr2=0x7f02cc0a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:37.473 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.466+0000 7f02d427d700 1 --2- 172.21.15.23:0/622028959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02cc0a3210 0x7f02cc0a18b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f02bc006fd0 tx=0x7f02bc006e30).stop 2020-11-20T11:32:37.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.466+0000 7f02d427d700 1 -- 172.21.15.23:0/622028959 shutdown_connections 2020-11-20T11:32:37.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.466+0000 7f02d427d700 1 --2- 172.21.15.23:0/622028959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f02cc13e870 0x7f02cc13ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:37.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.466+0000 7f02d427d700 1 --2- 172.21.15.23:0/622028959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02cc0a3210 0x7f02cc0a18b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:37.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.466+0000 7f02d427d700 1 -- 172.21.15.23:0/622028959 >> 172.21.15.23:0/622028959 conn(0x7f02cc0601b0 msgr2=0x7f02cc0605c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:37.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.466+0000 7f02d427d700 1 -- 172.21.15.23:0/622028959 shutdown_connections 2020-11-20T11:32:37.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.466+0000 7f02d427d700 1 -- 172.21.15.23:0/622028959 wait complete. 2020-11-20T11:32:37.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.467+0000 7f02d427d700 1 Processor -- start 2020-11-20T11:32:37.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.467+0000 7f02d427d700 1 -- start start 2020-11-20T11:32:37.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.468+0000 7f02d427d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f02cc0a3210 0x7f02cc066490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:37.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.468+0000 7f02d427d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02cc0a35f0 0x7f02cc0a6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:37.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.468+0000 7f02d427d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f02cc13e870 0x7f02cc149f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:37.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.468+0000 7f02d427d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc0a7490 con 0x7f02cc0a35f0 2020-11-20T11:32:37.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.468+0000 7f02d427d700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc0a59a0 con 0x7f02cc0a3210 2020-11-20T11:32:37.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.468+0000 7f02d427d700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc0a5c40 con 0x7f02cc13e870 2020-11-20T11:32:37.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.468+0000 7f02d1818700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02cc0a35f0 0x7f02cc0a6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:37.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.468+0000 7f02d1818700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02cc0a35f0 0x7f02cc0a6fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41140/0 (socket says 172.21.15.23:41140) 2020-11-20T11:32:37.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.468+0000 7f02d1818700 1 -- 172.21.15.23:0/385233264 learned_addr learned my addr 172.21.15.23:0/385233264 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:37.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.468+0000 7f02d281a700 1 --2- 172.21.15.23:0/385233264 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f02cc13e870 0x7f02cc149f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:37.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.468+0000 7f02d1818700 1 -- 172.21.15.23:0/385233264 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f02cc13e870 msgr2=0x7f02cc149f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:37.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.468+0000 7f02d1818700 1 --2- 172.21.15.23:0/385233264 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f02cc13e870 0x7f02cc149f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:37.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.468+0000 7f02d1818700 1 -- 172.21.15.23:0/385233264 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f02cc0a3210 msgr2=0x7f02cc066490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:37.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.468+0000 7f02d1818700 1 --2- 172.21.15.23:0/385233264 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f02cc0a3210 0x7f02cc066490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:32:37.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.468+0000 7f02d1818700 1 -- 172.21.15.23:0/385233264 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02cc0a5ee0 con 0x7f02cc0a35f0 2020-11-20T11:32:37.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.468+0000 7f02d281a700 1 --2- 172.21.15.23:0/385233264 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f02cc13e870 0x7f02cc149f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:32:37.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.468+0000 7f02d1818700 1 --2- 172.21.15.23:0/385233264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02cc0a35f0 0x7f02cc0a6fa0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f02c800c710 tx=0x7f02c8007c80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:37.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.468+0000 7f02ba7fc700 1 -- 172.21.15.23:0/385233264 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c8007320 con 0x7f02cc0a35f0 2020-11-20T11:32:37.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.469+0000 7f02ba7fc700 1 -- 172.21.15.23:0/385233264 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c8007480 con 0x7f02cc0a35f0 2020-11-20T11:32:37.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.469+0000 7f02d427d700 1 -- 172.21.15.23:0/385233264 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02cc14a470 con 0x7f02cc0a35f0 2020-11-20T11:32:37.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.469+0000 7f02ba7fc700 1 -- 172.21.15.23:0/385233264 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c8005840 con 0x7f02cc0a35f0 2020-11-20T11:32:37.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.469+0000 7f02d427d700 1 -- 172.21.15.23:0/385233264 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02cc0a6380 con 0x7f02cc0a35f0 2020-11-20T11:32:37.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.469+0000 7f02d427d700 1 -- 172.21.15.23:0/385233264 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02cc062c40 con 0x7f02cc0a35f0 2020-11-20T11:32:37.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.470+0000 7f02ba7fc700 1 -- 172.21.15.23:0/385233264 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f02c8010030 con 0x7f02cc0a35f0 2020-11-20T11:32:37.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.470+0000 7f02ba7fc700 1 --2- 172.21.15.23:0/385233264 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f02a4056d60 0x7f02a4059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:37.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.470+0000 7f02ba7fc700 1 -- 172.21.15.23:0/385233264 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f02c806fe20 con 0x7f02cc0a35f0 2020-11-20T11:32:37.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.470+0000 7f02d2019700 1 --2- 172.21.15.23:0/385233264 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f02a4056d60 0x7f02a4059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:37.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.471+0000 7f02d2019700 1 --2- 172.21.15.23:0/385233264 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f02a4056d60 0x7f02a4059220 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f02bc00b190 tx=0x7f02bc015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:37.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.474+0000 7f02ba7fc700 1 -- 172.21.15.23:0/385233264 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f02c8045110 con 0x7f02cc0a35f0 2020-11-20T11:32:37.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.624+0000 7f02d427d700 1 -- 172.21.15.23:0/385233264 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02cc14a600 con 0x7f02cc0a35f0 2020-11-20T11:32:37.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.624+0000 7f02ba7fc700 1 -- 172.21.15.23:0/385233264 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17087 (secure 0 0 0) 0x7f02cc14a600 con 0x7f02cc0a35f0 2020-11-20T11:32:37.627 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:32:37.628 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":25,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","created":"2020-11-20T11:32:06.136679+0000","modified":"2020-11-20T11:32:26.470927+0000","last_up_change":"2020-11-20T11:32:13.536194+0000","last_in_change":"2020-11-20T11:32:13.536194+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T11:32:09.559868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T11:32:17.000782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T11:32:20.423341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T11:32:23.126273+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e9d219f5-e40e-4b2b-9468-deb9c4074a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":23083},{"type":"v1","addr":"172.21.15.23:6801","nonce":23083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":23083},{"type":"v1","addr":"172.21.15.23:6803","nonce":23083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":23083},{"type":"v1","addr":"172.21.15.23:6807","nonce":23083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":23083},{"type":"v1","addr":"172.21.15.23:6805","nonce":23083}]},"public_addr":"172.21.15.23:6801/23083","cluster_addr":"172.21.15.23:6803/23083","heartbeat_back_addr":"172.21.15.23:6807/23083","heartbeat_front_addr":"172.21.15.23:6805/23083","state":["exists","up"]},{"osd":1,"uuid":"2328adde-2108-4fe0-8138-f0d096a496a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":23099},{"type":"v1","addr":"172.21.15.23:6817","nonce":23099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":23099},{"type":"v1","addr":"172.21.15.23:6819","nonce":23099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":23099},{"type":"v1","addr":"172.21.15.23:6824","nonce":23099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":23099},{"type":"v1","addr":"172.21.15.23:6821","nonce":23099}]},"public_addr":"172.21.15.23:6817/23099","cluster_addr":"172.21.15.23:6819/23099","heartbeat_back_addr":"172.21.15.23:6824/23099","heartbeat_front_addr":"172.21.15.23:6821/23099","state":["exists","up"]},{"osd":2,"uuid":"13367708-59a2-4363-8a3a-ab0cb97145ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":23084},{"type":"v1","addr":"172.21.15.23:6809","nonce":23084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":23084},{"type":"v1","addr":"172.21.15.23:6811","nonce":23084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":23084},{"type":"v1","addr":"172.21.15.23:6815","nonce":23084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":23084},{"type":"v1","addr":"172.21.15.23:6813","nonce":23084}]},"public_addr":"172.21.15.23:6809/23084","cluster_addr":"172.21.15.23:6811/23084","heartbeat_back_addr":"172.21.15.23:6815/23084","heartbeat_front_addr":"172.21.15.23:6813/23084","state":["exists","up"]},{"osd":3,"uuid":"1f147dd7-9289-47b0-ad1c-4088618d6faa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6823","nonce":23114},{"type":"v1","addr":"172.21.15.23:6825","nonce":23114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":23114},{"type":"v1","addr":"172.21.15.23:6827","nonce":23114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":23114},{"type":"v1","addr":"172.21.15.23:6831","nonce":23114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":23114},{"type":"v1","addr":"172.21.15.23:6829","nonce":23114}]},"public_addr":"172.21.15.23:6825/23114","cluster_addr":"172.21.15.23:6827/23114","heartbeat_back_addr":"172.21.15.23:6831/23114","heartbeat_front_addr":"172.21.15.23:6829/23114","state":["exists","up"]},{"osd":4,"uuid":"d008b039-4bc5-4135-9678-1511e58efc29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":23280},{"type":"v1","addr":"172.21.15.155:6801","nonce":23280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":23280},{"type":"v1","addr":"172.21.15.155:6803","nonce":23280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":23280},{"type":"v1","addr":"172.21.15.155:6807","nonce":23280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":23280},{"type":"v1","addr":"172.21.15.155:6805","nonce":23280}]},"public_addr":"172.21.15.155:6801/23280","cluster_addr":"172.21.15.155:6803/23280","heartbeat_back_addr":"172.21.15.155:6807/23280","heartbeat_front_addr":"172.21.15.155:6805/23280","state":["exists","up"]},{"osd":5,"uuid":"56715e6f-cf6e-44a6-9e39-93ab1224095f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":23298},{"type":"v1","addr":"172.21.15.155:6817","nonce":23298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":23298},{"type":"v1","addr":"172.21.15.155:6819","nonce":23298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":23298},{"type":"v1","addr":"172.21.15.155:6823","nonce":23298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":23298},{"type":"v1","addr":"172.21.15.155:6821","nonce":23298}]},"public_addr":"172.21.15.155:6817/23298","cluster_addr":"172.21.15.155:6819/23298","heartbeat_back_addr":"172.21.15.155:6823/23298","heartbeat_front_addr":"172.21.15.155:6821/23298","state":["exists","up"]},{"osd":6,"uuid":"db1e61ca-8e14-4a1f-8607-6e01b148d61c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":23307},{"type":"v1","addr":"172.21.15.155:6825","nonce":23307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":23307},{"type":"v1","addr":"172.21.15.155:6827","nonce":23307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":23307},{"type":"v1","addr":"172.21.15.155:6831","nonce":23307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":23307},{"type":"v1","addr":"172.21.15.155:6829","nonce":23307}]},"public_addr":"172.21.15.155:6825/23307","cluster_addr":"172.21.15.155:6827/23307","heartbeat_back_addr":"172.21.15.155:6831/23307","heartbeat_front_addr":"172.21.15.155:6829/23307","state":["exists","up"]},{"osd":7,"uuid":"d44419be-ef89-462c-8cf1-3c6db2a6944b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":23281},{"type":"v1","addr":"172.21.15.155:6809","nonce":23281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":23281},{"type":"v1","addr":"172.21.15.155:6811","nonce":23281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":23281},{"type":"v1","addr":"172.21.15.155:6815","nonce":23281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":23281},{"type":"v1","addr":"172.21.15.155:6813","nonce":23281}]},"public_addr":"172.21.15.155:6809/23281","cluster_addr":"172.21.15.155:6811/23281","heartbeat_back_addr":"172.21.15.155:6815/23281","heartbeat_front_addr":"172.21.15.155:6813/23281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.486702+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.509396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.533414+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.509489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.467600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.513068+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.536636+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.531772+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T11:32:37.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.626+0000 7f02d427d700 1 -- 172.21.15.23:0/385233264 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f02a4056d60 msgr2=0x7f02a4059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:37.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.626+0000 7f02d427d700 1 --2- 172.21.15.23:0/385233264 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f02a4056d60 0x7f02a4059220 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f02bc00b190 tx=0x7f02bc015000).stop 2020-11-20T11:32:37.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.626+0000 7f02d427d700 1 -- 172.21.15.23:0/385233264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02cc0a35f0 msgr2=0x7f02cc0a6fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:37.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.626+0000 7f02d427d700 1 --2- 172.21.15.23:0/385233264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02cc0a35f0 0x7f02cc0a6fa0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f02c800c710 tx=0x7f02c8007c80).stop 2020-11-20T11:32:37.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.626+0000 7f02d427d700 1 -- 172.21.15.23:0/385233264 shutdown_connections 2020-11-20T11:32:37.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.626+0000 7f02d427d700 1 --2- 172.21.15.23:0/385233264 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f02a4056d60 0x7f02a4059220 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:37.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.626+0000 7f02d427d700 1 --2- 172.21.15.23:0/385233264 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f02cc13e870 0x7f02cc149f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:37.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.626+0000 7f02d427d700 1 --2- 172.21.15.23:0/385233264 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f02cc0a3210 0x7f02cc066490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:32:37.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.626+0000 7f02d427d700 1 --2- 172.21.15.23:0/385233264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f02cc0a35f0 0x7f02cc0a6fa0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:37.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.626+0000 7f02d427d700 1 -- 172.21.15.23:0/385233264 >> 172.21.15.23:0/385233264 conn(0x7f02cc0601b0 msgr2=0x7f02cc0a8bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:37.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.626+0000 7f02d427d700 1 -- 172.21.15.23:0/385233264 shutdown_connections 2020-11-20T11:32:37.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.626+0000 7f02d427d700 1 -- 172.21.15.23:0/385233264 wait complete. 2020-11-20T11:32:37.640 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T11:32:09.559868+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T11:32:17.000782+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T11:32:20.423341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T11:32:23.126273+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T11:32:37.641 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T11:32:37.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.755+0000 7f0e12d5f700 1 Processor -- start 2020-11-20T11:32:37.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.755+0000 7f0e12d5f700 1 -- start start 2020-11-20T11:32:37.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.756+0000 7f0e12d5f700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0e0c060a80 0x7f0e0c061950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:37.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.756+0000 7f0e12d5f700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0e0c0aa1c0 0x7f0e0c138030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:37.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.756+0000 7f0e12d5f700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e0c067bc0 con 0x7f0e0c0a9d40 2020-11-20T11:32:37.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.756+0000 7f0e12d5f700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e0c0aa630 con 0x7f0e0c060a80 2020-11-20T11:32:37.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.756+0000 7f0e12d5f700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c067e60 con 0x7f0e0c0aa1c0 2020-11-20T11:32:37.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.756+0000 7f0e10afb700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0e0c060a80 0x7f0e0c061950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:37.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.756+0000 7f0e112fc700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0e0c0aa1c0 0x7f0e0c138030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:37.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.756+0000 7f0e0bfff700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f0e0c0a9d40 0x7f0e0c137920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:33372/0 (socket says 172.21.15.23:33372) 2020-11-20T11:32:37.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.756+0000 7f0e0bfff700 1 -- 172.21.15.23:0/987609534 learned_addr learned my addr 172.21.15.23:0/987609534 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:37.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.757+0000 7f0e10afb700 1 -- 172.21.15.23:0/987609534 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0e0c0aa1c0 msgr2=0x7f0e0c138030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:37.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.757+0000 7f0e10afb700 1 --2- 172.21.15.23:0/987609534 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0e0c0aa1c0 0x7f0e0c138030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:37.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.757+0000 7f0e10afb700 1 -- 172.21.15.23:0/987609534 >> v1:172.21.15.155:6789/0 conn(0x7f0e0c0a9d40 legacy=0x7f0e0c137920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:37.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.757+0000 7f0e0b7fe700 1 -- 172.21.15.23:0/987609534 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4077971809 0 0) 0x7f0e0c067bc0 con 0x7f0e0c0a9d40 2020-11-20T11:32:37.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.757+0000 7f0e10afb700 1 -- 172.21.15.23:0/987609534 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e0c068100 con 0x7f0e0c060a80 2020-11-20T11:32:37.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.757+0000 7f0e112fc700 1 --2- 172.21.15.23:0/987609534 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0e0c0aa1c0 0x7f0e0c138030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:32:37.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.757+0000 7f0e10afb700 1 --2- 172.21.15.23:0/987609534 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e0c060a80 0x7f0e0c061950 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f0e00006fd0 tx=0x7f0e00006e30).ready entity=mon.1 client_cookie=6130bd6a8905ec39 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:37.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.757+0000 7f0e0b7fe700 1 -- 172.21.15.23:0/987609534 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e000097e0 con 0x7f0e0c060a80 2020-11-20T11:32:37.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.757+0000 7f0e0b7fe700 1 -- 172.21.15.23:0/987609534 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e00000e90 con 0x7f0e0c060a80 2020-11-20T11:32:37.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.757+0000 7f0e0b7fe700 1 -- 172.21.15.23:0/987609534 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e00009b50 con 0x7f0e0c060a80 2020-11-20T11:32:37.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.757+0000 7f0e12d5f700 1 -- 172.21.15.23:0/987609534 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e0c060a80 msgr2=0x7f0e0c061950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:37.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.757+0000 7f0e12d5f700 1 --2- 172.21.15.23:0/987609534 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e0c060a80 0x7f0e0c061950 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f0e00006fd0 tx=0x7f0e00006e30).stop 2020-11-20T11:32:37.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.757+0000 7f0e12d5f700 1 -- 172.21.15.23:0/987609534 shutdown_connections 2020-11-20T11:32:37.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.757+0000 7f0e12d5f700 1 --2- 172.21.15.23:0/987609534 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e0c060a80 0x7f0e0c061950 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:37.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.757+0000 7f0e12d5f700 1 --2- 172.21.15.23:0/987609534 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0e0c0aa1c0 0x7f0e0c138030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:37.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.757+0000 7f0e12d5f700 1 -- 172.21.15.23:0/987609534 >> 172.21.15.23:0/987609534 conn(0x7f0e0c066730 msgr2=0x7f0e0c0601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:37.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.758+0000 7f0e12d5f700 1 -- 172.21.15.23:0/987609534 shutdown_connections 2020-11-20T11:32:37.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.758+0000 7f0e12d5f700 1 -- 172.21.15.23:0/987609534 wait complete. 2020-11-20T11:32:37.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.758+0000 7f0e12d5f700 1 Processor -- start 2020-11-20T11:32:37.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.759+0000 7f0e12d5f700 1 -- start start 2020-11-20T11:32:37.770 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.759+0000 7f0e12d5f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e0c060a80 0x7f0e0c137fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:37.770 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.759+0000 7f0e12d5f700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0e0c0a9d40 0x7f0e0c0a5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:37.771 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.759+0000 7f0e12d5f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e0c0aa1c0 0x7f0e0c1348f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:37.771 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.759+0000 7f0e12d5f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c0a3580 con 0x7f0e0c0aa1c0 2020-11-20T11:32:37.771 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.759+0000 7f0e12d5f700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c0a3820 con 0x7f0e0c060a80 2020-11-20T11:32:37.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.759+0000 7f0e12d5f700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c0a5700 con 0x7f0e0c0a9d40 2020-11-20T11:32:37.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.759+0000 7f0e112fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e0c0aa1c0 0x7f0e0c1348f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:37.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.759+0000 7f0e112fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e0c0aa1c0 0x7f0e0c1348f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41158/0 (socket says 172.21.15.23:41158) 2020-11-20T11:32:37.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.759+0000 7f0e112fc700 1 -- 172.21.15.23:0/3499852853 learned_addr learned my addr 172.21.15.23:0/3499852853 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:37.773 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.760+0000 7f0e10afb700 1 --2- 172.21.15.23:0/3499852853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e0c060a80 0x7f0e0c137fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:37.773 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.760+0000 7f0e0bfff700 1 --2- 172.21.15.23:0/3499852853 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0e0c0a9d40 0x7f0e0c0a5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:37.773 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.760+0000 7f0e112fc700 1 -- 172.21.15.23:0/3499852853 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0e0c0a9d40 msgr2=0x7f0e0c0a5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:37.773 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.760+0000 7f0e112fc700 1 --2- 172.21.15.23:0/3499852853 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0e0c0a9d40 0x7f0e0c0a5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:37.773 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.760+0000 7f0e112fc700 1 -- 172.21.15.23:0/3499852853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e0c060a80 msgr2=0x7f0e0c137fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:37.774 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.760+0000 7f0e112fc700 1 --2- 172.21.15.23:0/3499852853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e0c060a80 0x7f0e0c137fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:37.774 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.760+0000 7f0e112fc700 1 -- 172.21.15.23:0/3499852853 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e0c0a60b0 con 0x7f0e0c0aa1c0 2020-11-20T11:32:37.774 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.760+0000 7f0e10afb700 1 --2- 172.21.15.23:0/3499852853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e0c060a80 0x7f0e0c137fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:37.774 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.760+0000 7f0e0bfff700 1 --2- 172.21.15.23:0/3499852853 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0e0c0a9d40 0x7f0e0c0a5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:37.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.760+0000 7f0e112fc700 1 --2- 172.21.15.23:0/3499852853 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e0c0aa1c0 0x7f0e0c1348f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f0e0400ef50 tx=0x7f0e04009220).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:37.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.760+0000 7f0e08ff9700 1 -- 172.21.15.23:0/3499852853 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e0400cce0 con 0x7f0e0c0aa1c0 2020-11-20T11:32:37.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.760+0000 7f0e08ff9700 1 -- 172.21.15.23:0/3499852853 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e0400ce40 con 0x7f0e0c0aa1c0 2020-11-20T11:32:37.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.760+0000 7f0e12d5f700 1 -- 172.21.15.23:0/3499852853 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e0c0a7930 con 0x7f0e0c0aa1c0 2020-11-20T11:32:37.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.760+0000 7f0e08ff9700 1 -- 172.21.15.23:0/3499852853 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e04012620 con 0x7f0e0c0aa1c0 2020-11-20T11:32:37.776 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.760+0000 7f0e12d5f700 1 -- 172.21.15.23:0/3499852853 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e0c0a85d0 con 0x7f0e0c0aa1c0 2020-11-20T11:32:37.776 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.761+0000 7f0e12d5f700 1 -- 172.21.15.23:0/3499852853 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e0c044710 con 0x7f0e0c0aa1c0 2020-11-20T11:32:37.776 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.761+0000 7f0e08ff9700 1 -- 172.21.15.23:0/3499852853 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0e04022070 con 0x7f0e0c0aa1c0 2020-11-20T11:32:37.776 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.761+0000 7f0e08ff9700 1 --2- 172.21.15.23:0/3499852853 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0de4056d60 0x7f0de4059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:37.776 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.762+0000 7f0e08ff9700 1 -- 172.21.15.23:0/3499852853 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f0e04070b20 con 0x7f0e0c0aa1c0 2020-11-20T11:32:37.777 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.762+0000 7f0e10afb700 1 --2- 172.21.15.23:0/3499852853 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0de4056d60 0x7f0de4059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:37.777 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.762+0000 7f0e10afb700 1 --2- 172.21.15.23:0/3499852853 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0de4056d60 0x7f0de4059220 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f0e00006e70 tx=0x7f0e00015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:37.777 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.765+0000 7f0e08ff9700 1 -- 172.21.15.23:0/3499852853 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0e04045d00 con 0x7f0e0c0aa1c0 2020-11-20T11:32:37.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.916+0000 7f0e12d5f700 1 -- 172.21.15.23:0/3499852853 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f0e0c05c280 con 0x7f0e0c0aa1c0 2020-11-20T11:32:37.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.916+0000 7f0e08ff9700 1 -- 172.21.15.23:0/3499852853 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v25) v1 ==== 110+0+10 (secure 0 0 0) 0x7f0e040160b0 con 0x7f0e0c0aa1c0 2020-11-20T11:32:37.919 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2020-11-20T11:32:37.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.917+0000 7f0e12d5f700 1 -- 172.21.15.23:0/3499852853 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0de4056d60 msgr2=0x7f0de4059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:37.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.917+0000 7f0e12d5f700 1 --2- 172.21.15.23:0/3499852853 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0de4056d60 0x7f0de4059220 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f0e00006e70 tx=0x7f0e00015000).stop 2020-11-20T11:32:37.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.917+0000 7f0e12d5f700 1 -- 172.21.15.23:0/3499852853 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e0c0aa1c0 msgr2=0x7f0e0c1348f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:37.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.917+0000 7f0e12d5f700 1 --2- 172.21.15.23:0/3499852853 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e0c0aa1c0 0x7f0e0c1348f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f0e0400ef50 tx=0x7f0e04009220).stop 2020-11-20T11:32:37.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.918+0000 7f0e12d5f700 1 -- 172.21.15.23:0/3499852853 shutdown_connections 2020-11-20T11:32:37.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.918+0000 7f0e12d5f700 1 --2- 172.21.15.23:0/3499852853 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e0c0aa1c0 0x7f0e0c1348f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:37.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.918+0000 7f0e12d5f700 1 --2- 172.21.15.23:0/3499852853 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0e0c060a80 0x7f0e0c137fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:37.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.918+0000 7f0e12d5f700 1 --2- 172.21.15.23:0/3499852853 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0de4056d60 0x7f0de4059220 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:37.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.918+0000 7f0e12d5f700 1 --2- 172.21.15.23:0/3499852853 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0e0c0a9d40 0x7f0e0c0a5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:37.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.918+0000 7f0e12d5f700 1 -- 172.21.15.23:0/3499852853 >> 172.21.15.23:0/3499852853 conn(0x7f0e0c066730 msgr2=0x7f0e0c141580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:37.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.918+0000 7f0e12d5f700 1 -- 172.21.15.23:0/3499852853 shutdown_connections 2020-11-20T11:32:37.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:37.918+0000 7f0e12d5f700 1 -- 172.21.15.23:0/3499852853 wait complete. 2020-11-20T11:32:37.932 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T11:32:38.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.046+0000 7f592433c700 1 Processor -- start 2020-11-20T11:32:38.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.046+0000 7f592433c700 1 -- start start 2020-11-20T11:32:38.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.047+0000 7f592433c700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f591c060a80 0x7f591c061950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:38.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.047+0000 7f592433c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f591c0aa1c0 0x7f591c138030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:38.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.047+0000 7f592433c700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f591c067bc0 con 0x7f591c0a9d40 2020-11-20T11:32:38.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.047+0000 7f592433c700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f591c0aa630 con 0x7f591c0aa1c0 2020-11-20T11:32:38.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.047+0000 7f592433c700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f591c067e60 con 0x7f591c060a80 2020-11-20T11:32:38.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.047+0000 7f59228d9700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f591c0aa1c0 0x7f591c138030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:38.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.047+0000 7f59218d7700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f591c0a9d40 0x7f591c137920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55414/0 (socket says 172.21.15.23:55414) 2020-11-20T11:32:38.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.047+0000 7f59218d7700 1 -- 172.21.15.23:0/617193547 learned_addr learned my addr 172.21.15.23:0/617193547 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:38.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.047+0000 7f59220d8700 1 --2- 172.21.15.23:0/617193547 >> v2:172.21.15.155:3300/0 conn(0x7f591c060a80 0x7f591c061950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:38.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.047+0000 7f59228d9700 1 -- 172.21.15.23:0/617193547 >> v2:172.21.15.155:3300/0 conn(0x7f591c060a80 msgr2=0x7f591c061950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:38.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.047+0000 7f59228d9700 1 --2- 172.21.15.23:0/617193547 >> v2:172.21.15.155:3300/0 conn(0x7f591c060a80 0x7f591c061950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.047+0000 7f59228d9700 1 -- 172.21.15.23:0/617193547 >> v1:172.21.15.23:6789/0 conn(0x7f591c0a9d40 legacy=0x7f591c137920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:38.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.047+0000 7f59210d6700 1 -- 172.21.15.23:0/617193547 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2510790465 0 0) 0x7f591c067bc0 con 0x7f591c0a9d40 2020-11-20T11:32:38.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.047+0000 7f59228d9700 1 -- 172.21.15.23:0/617193547 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f591c068100 con 0x7f591c0aa1c0 2020-11-20T11:32:38.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.047+0000 7f59228d9700 1 --2- 172.21.15.23:0/617193547 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f591c0aa1c0 0x7f591c138030 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f591c0b01e0 tx=0x7f5914009220).ready entity=mon.0 client_cookie=4b7c1cb9d494cfcb server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:38.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.047+0000 7f59210d6700 1 -- 172.21.15.23:0/617193547 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591400bbf0 con 0x7f591c0aa1c0 2020-11-20T11:32:38.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.047+0000 7f59210d6700 1 -- 172.21.15.23:0/617193547 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591400bd50 con 0x7f591c0aa1c0 2020-11-20T11:32:38.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.047+0000 7f59210d6700 1 -- 172.21.15.23:0/617193547 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5914011610 con 0x7f591c0aa1c0 2020-11-20T11:32:38.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.048+0000 7f592433c700 1 -- 172.21.15.23:0/617193547 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f591c0aa1c0 msgr2=0x7f591c138030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:38.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.048+0000 7f592433c700 1 --2- 172.21.15.23:0/617193547 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f591c0aa1c0 0x7f591c138030 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f591c0b01e0 tx=0x7f5914009220).stop 2020-11-20T11:32:38.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.048+0000 7f592433c700 1 -- 172.21.15.23:0/617193547 shutdown_connections 2020-11-20T11:32:38.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.048+0000 7f592433c700 1 --2- 172.21.15.23:0/617193547 >> v2:172.21.15.155:3300/0 conn(0x7f591c060a80 0x7f591c061950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.048+0000 7f592433c700 1 --2- 172.21.15.23:0/617193547 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f591c0aa1c0 0x7f591c138030 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.048+0000 7f592433c700 1 -- 172.21.15.23:0/617193547 >> 172.21.15.23:0/617193547 conn(0x7f591c066730 msgr2=0x7f591c0601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:38.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.048+0000 7f592433c700 1 -- 172.21.15.23:0/617193547 shutdown_connections 2020-11-20T11:32:38.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.048+0000 7f592433c700 1 -- 172.21.15.23:0/617193547 wait complete. 2020-11-20T11:32:38.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.049+0000 7f592433c700 1 Processor -- start 2020-11-20T11:32:38.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.049+0000 7f592433c700 1 -- start start 2020-11-20T11:32:38.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.049+0000 7f592433c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f591c060a80 0x7f591c13a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:38.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.049+0000 7f592433c700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f591c0a9d40 0x7f591c0a87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:38.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.049+0000 7f592433c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f591c0aa1c0 0x7f591c1349e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:38.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.049+0000 7f592433c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591c0a50e0 con 0x7f591c060a80 2020-11-20T11:32:38.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.049+0000 7f592433c700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591c0a5380 con 0x7f591c0aa1c0 2020-11-20T11:32:38.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.049+0000 7f592433c700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f591c0a8320 con 0x7f591c0a9d40 2020-11-20T11:32:38.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.050+0000 7f59220d8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f591c060a80 0x7f591c13a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:38.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.050+0000 7f59220d8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f591c060a80 0x7f591c13a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41168/0 (socket says 172.21.15.23:41168) 2020-11-20T11:32:38.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.050+0000 7f59220d8700 1 -- 172.21.15.23:0/2412414873 learned_addr learned my addr 172.21.15.23:0/2412414873 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:38.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.050+0000 7f59220d8700 1 -- 172.21.15.23:0/2412414873 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f591c0a9d40 msgr2=0x7f591c0a87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:38.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.050+0000 7f59228d9700 1 --2- 172.21.15.23:0/2412414873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f591c0aa1c0 0x7f591c1349e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:38.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.050+0000 7f59218d7700 1 --2- 172.21.15.23:0/2412414873 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f591c0a9d40 0x7f591c0a87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:38.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.050+0000 7f59220d8700 1 --2- 172.21.15.23:0/2412414873 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f591c0a9d40 0x7f591c0a87c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.050+0000 7f59220d8700 1 -- 172.21.15.23:0/2412414873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f591c0aa1c0 msgr2=0x7f591c1349e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:38.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.050+0000 7f59220d8700 1 --2- 172.21.15.23:0/2412414873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f591c0aa1c0 0x7f591c1349e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.050+0000 7f59220d8700 1 -- 172.21.15.23:0/2412414873 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f591c0a8cb0 con 0x7f591c060a80 2020-11-20T11:32:38.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.050+0000 7f59220d8700 1 --2- 172.21.15.23:0/2412414873 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f591c060a80 0x7f591c13a390 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f591800c900 tx=0x7f5918008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:38.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.050+0000 7f590a7fc700 1 -- 172.21.15.23:0/2412414873 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591800cea0 con 0x7f591c060a80 2020-11-20T11:32:38.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.050+0000 7f590a7fc700 1 -- 172.21.15.23:0/2412414873 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5918007740 con 0x7f591c060a80 2020-11-20T11:32:38.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.050+0000 7f592433c700 1 -- 172.21.15.23:0/2412414873 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f591c0a77a0 con 0x7f591c060a80 2020-11-20T11:32:38.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.050+0000 7f590a7fc700 1 -- 172.21.15.23:0/2412414873 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5918007ae0 con 0x7f591c060a80 2020-11-20T11:32:38.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.050+0000 7f592433c700 1 -- 172.21.15.23:0/2412414873 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f591c0a5760 con 0x7f591c060a80 2020-11-20T11:32:38.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.051+0000 7f592433c700 1 -- 172.21.15.23:0/2412414873 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f591c0a8f70 con 0x7f591c060a80 2020-11-20T11:32:38.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.051+0000 7f590a7fc700 1 -- 172.21.15.23:0/2412414873 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f591801b460 con 0x7f591c060a80 2020-11-20T11:32:38.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.052+0000 7f590a7fc700 1 --2- 172.21.15.23:0/2412414873 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f58f4056d60 0x7f58f4059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:38.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.052+0000 7f590a7fc700 1 -- 172.21.15.23:0/2412414873 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f5918070170 con 0x7f591c060a80 2020-11-20T11:32:38.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.052+0000 7f59218d7700 1 --2- 172.21.15.23:0/2412414873 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f58f4056d60 0x7f58f4059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:38.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.052+0000 7f59218d7700 1 --2- 172.21.15.23:0/2412414873 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f58f4056d60 0x7f58f4059220 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f591c0a43c0 tx=0x7f590c009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:38.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.055+0000 7f590a7fc700 1 -- 172.21.15.23:0/2412414873 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5918045400 con 0x7f591c060a80 2020-11-20T11:32:38.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.205+0000 7f592433c700 1 -- 172.21.15.23:0/2412414873 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f591c044710 con 0x7f591c060a80 2020-11-20T11:32:38.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.205+0000 7f590a7fc700 1 -- 172.21.15.23:0/2412414873 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v25) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5918048a20 con 0x7f591c060a80 2020-11-20T11:32:38.208 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 8 2020-11-20T11:32:38.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.206+0000 7f592433c700 1 -- 172.21.15.23:0/2412414873 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f58f4056d60 msgr2=0x7f58f4059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:38.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.206+0000 7f592433c700 1 --2- 172.21.15.23:0/2412414873 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f58f4056d60 0x7f58f4059220 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f591c0a43c0 tx=0x7f590c009040).stop 2020-11-20T11:32:38.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.207+0000 7f592433c700 1 -- 172.21.15.23:0/2412414873 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f591c060a80 msgr2=0x7f591c13a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:38.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.207+0000 7f592433c700 1 --2- 172.21.15.23:0/2412414873 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f591c060a80 0x7f591c13a390 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f591800c900 tx=0x7f5918008040).stop 2020-11-20T11:32:38.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.207+0000 7f592433c700 1 -- 172.21.15.23:0/2412414873 shutdown_connections 2020-11-20T11:32:38.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.207+0000 7f592433c700 1 --2- 172.21.15.23:0/2412414873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f591c0aa1c0 0x7f591c1349e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.207+0000 7f592433c700 1 --2- 172.21.15.23:0/2412414873 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f591c060a80 0x7f591c13a390 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.207+0000 7f592433c700 1 --2- 172.21.15.23:0/2412414873 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f58f4056d60 0x7f58f4059220 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.207+0000 7f592433c700 1 --2- 172.21.15.23:0/2412414873 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f591c0a9d40 0x7f591c0a87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.207+0000 7f592433c700 1 -- 172.21.15.23:0/2412414873 >> 172.21.15.23:0/2412414873 conn(0x7f591c066730 msgr2=0x7f591c13ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:38.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.207+0000 7f592433c700 1 -- 172.21.15.23:0/2412414873 shutdown_connections 2020-11-20T11:32:38.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.207+0000 7f592433c700 1 -- 172.21.15.23:0/2412414873 wait complete. 2020-11-20T11:32:38.221 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T11:32:38.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.334+0000 7f629c975700 1 Processor -- start 2020-11-20T11:32:38.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.334+0000 7f629c975700 1 -- start start 2020-11-20T11:32:38.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.335+0000 7f629c975700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6298060a80 0x7f6298061950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:38.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.335+0000 7f629c975700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f62980aa1c0 0x7f6298138030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:38.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.335+0000 7f629c975700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6298067bc0 con 0x7f62980a9d40 2020-11-20T11:32:38.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.335+0000 7f629c975700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6298067e60 con 0x7f62980aa1c0 2020-11-20T11:32:38.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.335+0000 7f629c975700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6298068100 con 0x7f6298060a80 2020-11-20T11:32:38.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.335+0000 7f6296d9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f62980aa1c0 0x7f6298138030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:38.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.335+0000 7f6295d9b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f62980a9d40 0x7f6298137920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55426/0 (socket says 172.21.15.23:55426) 2020-11-20T11:32:38.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.335+0000 7f6295d9b700 1 -- 172.21.15.23:0/3415999518 learned_addr learned my addr 172.21.15.23:0/3415999518 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:38.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.335+0000 7f629659c700 1 --2- 172.21.15.23:0/3415999518 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6298060a80 0x7f6298061950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:38.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.336+0000 7f6296d9d700 1 -- 172.21.15.23:0/3415999518 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6298060a80 msgr2=0x7f6298061950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:38.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.336+0000 7f6296d9d700 1 --2- 172.21.15.23:0/3415999518 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6298060a80 0x7f6298061950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.336+0000 7f6296d9d700 1 -- 172.21.15.23:0/3415999518 >> v1:172.21.15.23:6789/0 conn(0x7f62980a9d40 legacy=0x7f6298137920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:38.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.336+0000 7f629559a700 1 -- 172.21.15.23:0/3415999518 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2155843627 0 0) 0x7f6298067bc0 con 0x7f62980a9d40 2020-11-20T11:32:38.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.336+0000 7f6296d9d700 1 -- 172.21.15.23:0/3415999518 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6298060e60 con 0x7f62980aa1c0 2020-11-20T11:32:38.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.336+0000 7f6296d9d700 1 --2- 172.21.15.23:0/3415999518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62980aa1c0 0x7f6298138030 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f6290009ce0 tx=0x7f6290009220).ready entity=mon.0 client_cookie=bfd2e881dff48d88 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:38.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.336+0000 7f629559a700 1 -- 172.21.15.23:0/3415999518 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629000bbf0 con 0x7f62980aa1c0 2020-11-20T11:32:38.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.336+0000 7f629559a700 1 -- 172.21.15.23:0/3415999518 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f629000bd50 con 0x7f62980aa1c0 2020-11-20T11:32:38.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.336+0000 7f629559a700 1 -- 172.21.15.23:0/3415999518 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6290011610 con 0x7f62980aa1c0 2020-11-20T11:32:38.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.336+0000 7f629c975700 1 -- 172.21.15.23:0/3415999518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62980aa1c0 msgr2=0x7f6298138030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:38.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.336+0000 7f629c975700 1 --2- 172.21.15.23:0/3415999518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62980aa1c0 0x7f6298138030 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f6290009ce0 tx=0x7f6290009220).stop 2020-11-20T11:32:38.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.336+0000 7f629c975700 1 -- 172.21.15.23:0/3415999518 shutdown_connections 2020-11-20T11:32:38.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.336+0000 7f629c975700 1 --2- 172.21.15.23:0/3415999518 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6298060a80 0x7f6298061950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.336+0000 7f629c975700 1 --2- 172.21.15.23:0/3415999518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62980aa1c0 0x7f6298138030 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.336+0000 7f629c975700 1 -- 172.21.15.23:0/3415999518 >> 172.21.15.23:0/3415999518 conn(0x7f6298066730 msgr2=0x7f62980601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:38.345 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.336+0000 7f629c975700 1 -- 172.21.15.23:0/3415999518 shutdown_connections 2020-11-20T11:32:38.345 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.336+0000 7f629c975700 1 -- 172.21.15.23:0/3415999518 wait complete. 2020-11-20T11:32:38.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.337+0000 7f629c975700 1 Processor -- start 2020-11-20T11:32:38.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.337+0000 7f629c975700 1 -- start start 2020-11-20T11:32:38.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.338+0000 7f629c975700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6298060a80 0x7f62980a6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:38.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.338+0000 7f629c975700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62980a9d40 0x7f629813d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:38.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.338+0000 7f629c975700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f62980aa1c0 0x7f629813c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:38.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.338+0000 7f629c975700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629813e830 con 0x7f6298060a80 2020-11-20T11:32:38.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.338+0000 7f629c975700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629813cf80 con 0x7f62980a9d40 2020-11-20T11:32:38.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.338+0000 7f629c975700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f629813d280 con 0x7f62980aa1c0 2020-11-20T11:32:38.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.338+0000 7f629659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6298060a80 0x7f62980a6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:38.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.338+0000 7f629659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6298060a80 0x7f62980a6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41184/0 (socket says 172.21.15.23:41184) 2020-11-20T11:32:38.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.338+0000 7f629659c700 1 -- 172.21.15.23:0/1895876474 learned_addr learned my addr 172.21.15.23:0/1895876474 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:38.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.338+0000 7f6296d9d700 1 --2- 172.21.15.23:0/1895876474 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f62980aa1c0 0x7f629813c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:38.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.338+0000 7f6295d9b700 1 --2- 172.21.15.23:0/1895876474 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62980a9d40 0x7f629813d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:38.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.338+0000 7f629659c700 1 -- 172.21.15.23:0/1895876474 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f62980aa1c0 msgr2=0x7f629813c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:38.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.338+0000 7f629659c700 1 --2- 172.21.15.23:0/1895876474 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f62980aa1c0 0x7f629813c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.338+0000 7f629659c700 1 -- 172.21.15.23:0/1895876474 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62980a9d40 msgr2=0x7f629813d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:38.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.338+0000 7f629659c700 1 --2- 172.21.15.23:0/1895876474 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62980a9d40 0x7f629813d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.338+0000 7f629659c700 1 -- 172.21.15.23:0/1895876474 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f629813d4e0 con 0x7f6298060a80 2020-11-20T11:32:38.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.338+0000 7f629659c700 1 --2- 172.21.15.23:0/1895876474 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6298060a80 0x7f62980a6750 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f628800db30 tx=0x7f6288008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:38.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.339+0000 7f62867fc700 1 -- 172.21.15.23:0/1895876474 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628800de40 con 0x7f6298060a80 2020-11-20T11:32:38.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.339+0000 7f62867fc700 1 -- 172.21.15.23:0/1895876474 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62880097c0 con 0x7f6298060a80 2020-11-20T11:32:38.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.339+0000 7f629c975700 1 -- 172.21.15.23:0/1895876474 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6298137560 con 0x7f6298060a80 2020-11-20T11:32:38.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.339+0000 7f62867fc700 1 -- 172.21.15.23:0/1895876474 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6288009b40 con 0x7f6298060a80 2020-11-20T11:32:38.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.339+0000 7f629c975700 1 -- 172.21.15.23:0/1895876474 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f629813cb60 con 0x7f6298060a80 2020-11-20T11:32:38.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.339+0000 7f629c975700 1 -- 172.21.15.23:0/1895876474 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f629813d6e0 con 0x7f6298060a80 2020-11-20T11:32:38.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.340+0000 7f62867fc700 1 -- 172.21.15.23:0/1895876474 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6288013480 con 0x7f6298060a80 2020-11-20T11:32:38.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.340+0000 7f62867fc700 1 --2- 172.21.15.23:0/1895876474 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f626c056d60 0x7f626c059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:38.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.340+0000 7f62867fc700 1 -- 172.21.15.23:0/1895876474 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f62880702f0 con 0x7f6298060a80 2020-11-20T11:32:38.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.340+0000 7f6295d9b700 1 --2- 172.21.15.23:0/1895876474 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f626c056d60 0x7f626c059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:38.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.341+0000 7f6295d9b700 1 --2- 172.21.15.23:0/1895876474 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f626c056d60 0x7f626c059220 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f6298147bf0 tx=0x7f628c007a40).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:38.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.344+0000 7f62867fc700 1 -- 172.21.15.23:0/1895876474 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f629813d6e0 con 0x7f6298060a80 2020-11-20T11:32:38.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.496+0000 7f629c975700 1 -- 172.21.15.23:0/1895876474 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f629813d6e0 con 0x7f6298060a80 2020-11-20T11:32:38.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.496+0000 7f62867fc700 1 -- 172.21.15.23:0/1895876474 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+10 (secure 0 0 0) 0x7f629813d6e0 con 0x7f6298060a80 2020-11-20T11:32:38.499 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 8 2020-11-20T11:32:38.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.498+0000 7f629c975700 1 -- 172.21.15.23:0/1895876474 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f626c056d60 msgr2=0x7f626c059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:38.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.498+0000 7f629c975700 1 --2- 172.21.15.23:0/1895876474 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f626c056d60 0x7f626c059220 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f6298147bf0 tx=0x7f628c007a40).stop 2020-11-20T11:32:38.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.498+0000 7f629c975700 1 -- 172.21.15.23:0/1895876474 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6298060a80 msgr2=0x7f62980a6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:38.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.498+0000 7f629c975700 1 --2- 172.21.15.23:0/1895876474 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6298060a80 0x7f62980a6750 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f628800db30 tx=0x7f6288008040).stop 2020-11-20T11:32:38.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.498+0000 7f629c975700 1 -- 172.21.15.23:0/1895876474 shutdown_connections 2020-11-20T11:32:38.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.498+0000 7f629c975700 1 --2- 172.21.15.23:0/1895876474 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f626c056d60 0x7f626c059220 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.498+0000 7f629c975700 1 --2- 172.21.15.23:0/1895876474 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f62980aa1c0 0x7f629813c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.498+0000 7f629c975700 1 --2- 172.21.15.23:0/1895876474 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6298060a80 0x7f62980a6750 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.498+0000 7f629c975700 1 --2- 172.21.15.23:0/1895876474 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62980a9d40 0x7f629813d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.498+0000 7f629c975700 1 -- 172.21.15.23:0/1895876474 >> 172.21.15.23:0/1895876474 conn(0x7f6298066730 msgr2=0x7f62980601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:38.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.498+0000 7f629c975700 1 -- 172.21.15.23:0/1895876474 shutdown_connections 2020-11-20T11:32:38.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.498+0000 7f629c975700 1 -- 172.21.15.23:0/1895876474 wait complete. 2020-11-20T11:32:38.512 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T11:32:38.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.626+0000 7f7a5d594700 1 Processor -- start 2020-11-20T11:32:38.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.626+0000 7f7a5d594700 1 -- start start 2020-11-20T11:32:38.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.627+0000 7f7a5d594700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f7a580aae90 0x7f7a581381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:38.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.627+0000 7f7a5d594700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f7a5813aeb0 0x7f7a580a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:38.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.627+0000 7f7a5d594700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a5813e600 con 0x7f7a580aaab0 2020-11-20T11:32:38.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.627+0000 7f7a5d594700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a5813e8a0 con 0x7f7a5813aeb0 2020-11-20T11:32:38.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.627+0000 7f7a5d594700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a580ab6d0 con 0x7f7a580aae90 2020-11-20T11:32:38.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.627+0000 7f7a577fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f7a5813aeb0 0x7f7a580a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:38.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.627+0000 7f7a56ffd700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f7a580aaab0 0x7f7a5813ed40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55440/0 (socket says 172.21.15.23:55440) 2020-11-20T11:32:38.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.627+0000 7f7a56ffd700 1 -- 172.21.15.23:0/3677661414 learned_addr learned my addr 172.21.15.23:0/3677661414 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:38.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.627+0000 7f7a567fc700 1 --2- 172.21.15.23:0/3677661414 >> v2:172.21.15.155:3300/0 conn(0x7f7a580aae90 0x7f7a581381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:38.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.627+0000 7f7a577fe700 1 -- 172.21.15.23:0/3677661414 >> v2:172.21.15.155:3300/0 conn(0x7f7a580aae90 msgr2=0x7f7a581381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:38.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.627+0000 7f7a577fe700 1 --2- 172.21.15.23:0/3677661414 >> v2:172.21.15.155:3300/0 conn(0x7f7a580aae90 0x7f7a581381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.627+0000 7f7a577fe700 1 -- 172.21.15.23:0/3677661414 >> v1:172.21.15.23:6789/0 conn(0x7f7a580aaab0 legacy=0x7f7a5813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:38.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.627+0000 7f7a577fe700 1 -- 172.21.15.23:0/3677661414 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a58138690 con 0x7f7a5813aeb0 2020-11-20T11:32:38.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.627+0000 7f7a55ffb700 1 -- 172.21.15.23:0/3677661414 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2748802886 0 0) 0x7f7a5813e600 con 0x7f7a580aaab0 2020-11-20T11:32:38.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.628+0000 7f7a577fe700 1 --2- 172.21.15.23:0/3677661414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a5813aeb0 0x7f7a580a8c60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f7a580ab930 tx=0x7f7a50009220).ready entity=mon.0 client_cookie=a5927e69d30c865 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:38.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.628+0000 7f7a55ffb700 1 -- 172.21.15.23:0/3677661414 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a5000bbf0 con 0x7f7a5813aeb0 2020-11-20T11:32:38.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.628+0000 7f7a55ffb700 1 -- 172.21.15.23:0/3677661414 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a5000bd50 con 0x7f7a5813aeb0 2020-11-20T11:32:38.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.628+0000 7f7a55ffb700 1 -- 172.21.15.23:0/3677661414 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a50011610 con 0x7f7a5813aeb0 2020-11-20T11:32:38.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.628+0000 7f7a5d594700 1 -- 172.21.15.23:0/3677661414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a5813aeb0 msgr2=0x7f7a580a8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:38.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.628+0000 7f7a5d594700 1 --2- 172.21.15.23:0/3677661414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a5813aeb0 0x7f7a580a8c60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f7a580ab930 tx=0x7f7a50009220).stop 2020-11-20T11:32:38.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.628+0000 7f7a5d594700 1 -- 172.21.15.23:0/3677661414 shutdown_connections 2020-11-20T11:32:38.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.628+0000 7f7a5d594700 1 --2- 172.21.15.23:0/3677661414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a5813aeb0 0x7f7a580a8c60 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.628+0000 7f7a5d594700 1 --2- 172.21.15.23:0/3677661414 >> v2:172.21.15.155:3300/0 conn(0x7f7a580aae90 0x7f7a581381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.628+0000 7f7a5d594700 1 -- 172.21.15.23:0/3677661414 >> 172.21.15.23:0/3677661414 conn(0x7f7a580678e0 msgr2=0x7f7a58067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:38.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.628+0000 7f7a5d594700 1 -- 172.21.15.23:0/3677661414 shutdown_connections 2020-11-20T11:32:38.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.628+0000 7f7a5d594700 1 -- 172.21.15.23:0/3677661414 wait complete. 2020-11-20T11:32:38.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.629+0000 7f7a5d594700 1 Processor -- start 2020-11-20T11:32:38.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.629+0000 7f7a5d594700 1 -- start start 2020-11-20T11:32:38.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.630+0000 7f7a5d594700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a580aaab0 0x7f7a580a6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:38.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.630+0000 7f7a5d594700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7a580aae90 0x7f7a5813d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:38.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.630+0000 7f7a5d594700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7a5814e1b0 0x7f7a58134840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:38.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.630+0000 7f7a5d594700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a580a5920 con 0x7f7a580aaab0 2020-11-20T11:32:38.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.630+0000 7f7a5d594700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5813dc90 con 0x7f7a5814e1b0 2020-11-20T11:32:38.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.630+0000 7f7a5d594700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5814da70 con 0x7f7a580aae90 2020-11-20T11:32:38.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.630+0000 7f7a56ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a580aaab0 0x7f7a580a6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:38.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.630+0000 7f7a56ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a580aaab0 0x7f7a580a6590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41196/0 (socket says 172.21.15.23:41196) 2020-11-20T11:32:38.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.630+0000 7f7a56ffd700 1 -- 172.21.15.23:0/822261873 learned_addr learned my addr 172.21.15.23:0/822261873 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:38.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.630+0000 7f7a577fe700 1 --2- 172.21.15.23:0/822261873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7a5814e1b0 0x7f7a58134840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:38.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.630+0000 7f7a567fc700 1 --2- 172.21.15.23:0/822261873 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7a580aae90 0x7f7a5813d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:38.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.630+0000 7f7a56ffd700 1 -- 172.21.15.23:0/822261873 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7a580aae90 msgr2=0x7f7a5813d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:38.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.630+0000 7f7a56ffd700 1 --2- 172.21.15.23:0/822261873 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7a580aae90 0x7f7a5813d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.630+0000 7f7a56ffd700 1 -- 172.21.15.23:0/822261873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7a5814e1b0 msgr2=0x7f7a58134840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:38.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.630+0000 7f7a56ffd700 1 --2- 172.21.15.23:0/822261873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7a5814e1b0 0x7f7a58134840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.630+0000 7f7a56ffd700 1 -- 172.21.15.23:0/822261873 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a5814dd10 con 0x7f7a580aaab0 2020-11-20T11:32:38.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.631+0000 7f7a56ffd700 1 --2- 172.21.15.23:0/822261873 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a580aaab0 0x7f7a580a6590 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f7a4800c7f0 tx=0x7f7a48007b00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:38.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.631+0000 7f7a3effd700 1 -- 172.21.15.23:0/822261873 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a48007320 con 0x7f7a580aaab0 2020-11-20T11:32:38.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.631+0000 7f7a3effd700 1 -- 172.21.15.23:0/822261873 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a48007480 con 0x7f7a580aaab0 2020-11-20T11:32:38.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.631+0000 7f7a3effd700 1 -- 172.21.15.23:0/822261873 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a48005840 con 0x7f7a580aaab0 2020-11-20T11:32:38.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.631+0000 7f7a5d594700 1 -- 172.21.15.23:0/822261873 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a58134430 con 0x7f7a580aaab0 2020-11-20T11:32:38.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.631+0000 7f7a5d594700 1 -- 172.21.15.23:0/822261873 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a5813da00 con 0x7f7a580aaab0 2020-11-20T11:32:38.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.632+0000 7f7a5d594700 1 -- 172.21.15.23:0/822261873 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a58139590 con 0x7f7a580aaab0 2020-11-20T11:32:38.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.632+0000 7f7a3effd700 1 -- 172.21.15.23:0/822261873 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7a48010030 con 0x7f7a580aaab0 2020-11-20T11:32:38.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.632+0000 7f7a3effd700 1 --2- 172.21.15.23:0/822261873 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7a34056d60 0x7f7a34059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:38.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.632+0000 7f7a3effd700 1 -- 172.21.15.23:0/822261873 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f7a4806fd10 con 0x7f7a580aaab0 2020-11-20T11:32:38.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.633+0000 7f7a567fc700 1 --2- 172.21.15.23:0/822261873 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7a34056d60 0x7f7a34059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:38.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.633+0000 7f7a567fc700 1 --2- 172.21.15.23:0/822261873 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7a34056d60 0x7f7a34059220 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f7a58133120 tx=0x7f7a4c008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:38.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.636+0000 7f7a3effd700 1 -- 172.21.15.23:0/822261873 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7a48045000 con 0x7f7a580aaab0 2020-11-20T11:32:38.789 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.786+0000 7f7a5d594700 1 -- 172.21.15.23:0/822261873 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7a5805c7d0 con 0x7f7a580aaab0 2020-11-20T11:32:38.789 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.786+0000 7f7a3effd700 1 -- 172.21.15.23:0/822261873 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+10 (secure 0 0 0) 0x7f7a48048620 con 0x7f7a580aaab0 2020-11-20T11:32:38.789 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 8 2020-11-20T11:32:38.791 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.787+0000 7f7a5d594700 1 -- 172.21.15.23:0/822261873 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7a34056d60 msgr2=0x7f7a34059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:38.791 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.787+0000 7f7a5d594700 1 --2- 172.21.15.23:0/822261873 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7a34056d60 0x7f7a34059220 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f7a58133120 tx=0x7f7a4c008040).stop 2020-11-20T11:32:38.791 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.787+0000 7f7a5d594700 1 -- 172.21.15.23:0/822261873 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a580aaab0 msgr2=0x7f7a580a6590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:38.791 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.787+0000 7f7a5d594700 1 --2- 172.21.15.23:0/822261873 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a580aaab0 0x7f7a580a6590 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f7a4800c7f0 tx=0x7f7a48007b00).stop 2020-11-20T11:32:38.792 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.788+0000 7f7a5d594700 1 -- 172.21.15.23:0/822261873 shutdown_connections 2020-11-20T11:32:38.792 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.788+0000 7f7a5d594700 1 --2- 172.21.15.23:0/822261873 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7a5814e1b0 0x7f7a58134840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.792 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.788+0000 7f7a5d594700 1 --2- 172.21.15.23:0/822261873 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7a580aaab0 0x7f7a580a6590 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.792 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.788+0000 7f7a5d594700 1 --2- 172.21.15.23:0/822261873 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7a34056d60 0x7f7a34059220 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.792 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.788+0000 7f7a5d594700 1 --2- 172.21.15.23:0/822261873 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7a580aae90 0x7f7a5813d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.793 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.788+0000 7f7a5d594700 1 -- 172.21.15.23:0/822261873 >> 172.21.15.23:0/822261873 conn(0x7f7a580678e0 msgr2=0x7f7a580acf80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:38.793 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.788+0000 7f7a5d594700 1 -- 172.21.15.23:0/822261873 shutdown_connections 2020-11-20T11:32:38.793 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.788+0000 7f7a5d594700 1 -- 172.21.15.23:0/822261873 wait complete. 2020-11-20T11:32:38.802 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T11:32:38.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.914+0000 7fc874a52700 1 Processor -- start 2020-11-20T11:32:38.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.914+0000 7fc874a52700 1 -- start start 2020-11-20T11:32:38.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.914+0000 7fc874a52700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc8700ace70 0x7fc87013a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:38.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.914+0000 7fc874a52700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8700ad860 con 0x7fc8700a9040 2020-11-20T11:32:38.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.914+0000 7fc874a52700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8700aaa90 con 0x7fc8700ad2f0 2020-11-20T11:32:38.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.914+0000 7fc874a52700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8700aa8a0 con 0x7fc8700ace70 2020-11-20T11:32:38.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.914+0000 7fc86ed9d700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fc8700a9040 0x7fc8700aaf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55454/0 (socket says 172.21.15.23:55454) 2020-11-20T11:32:38.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.914+0000 7fc86ed9d700 1 -- 172.21.15.23:0/2923098953 learned_addr learned my addr 172.21.15.23:0/2923098953 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:38.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.915+0000 7fc86e59c700 1 --2- 172.21.15.23:0/2923098953 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc8700ace70 0x7fc87013a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:38.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.915+0000 7fc86dd9b700 1 -- 172.21.15.23:0/2923098953 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1599750659 0 0) 0x7fc8700ad860 con 0x7fc8700a9040 2020-11-20T11:32:38.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.915+0000 7fc86dd9b700 1 -- 172.21.15.23:0/2923098953 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc850003280 con 0x7fc8700a9040 2020-11-20T11:32:38.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.915+0000 7fc86dd9b700 1 -- 172.21.15.23:0/2923098953 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3976859406 0 0) 0x7fc850003280 con 0x7fc8700a9040 2020-11-20T11:32:38.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.915+0000 7fc86dd9b700 1 -- 172.21.15.23:0/2923098953 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc8700ace70 msgr2=0x7fc87013a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:38.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.915+0000 7fc86dd9b700 1 --2- 172.21.15.23:0/2923098953 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc8700ace70 0x7fc87013a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.915+0000 7fc86dd9b700 1 -- 172.21.15.23:0/2923098953 >> v1:172.21.15.155:6789/0 conn(0x7fc8700ad2f0 legacy=0x7fc87013ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:38.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.915+0000 7fc86dd9b700 1 -- 172.21.15.23:0/2923098953 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8700a9420 con 0x7fc8700a9040 2020-11-20T11:32:38.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.915+0000 7fc86dd9b700 1 -- 172.21.15.23:0/2923098953 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7fc8600016a0 con 0x7fc8700a9040 2020-11-20T11:32:38.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.915+0000 7fc86dd9b700 1 -- 172.21.15.23:0/2923098953 >> v1:172.21.15.23:6789/0 conn(0x7fc8700a9040 legacy=0x7fc8700aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:38.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.915+0000 7fc86dd9b700 1 --2- 172.21.15.23:0/2923098953 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc8500037a0 0x7fc850003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:38.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.915+0000 7fc86dd9b700 1 --2- 172.21.15.23:0/2923098953 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc850004520 0x7fc8500049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:38.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.915+0000 7fc86dd9b700 1 --2- 172.21.15.23:0/2923098953 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc8700ace70 0x7fc850004e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:38.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.915+0000 7fc86dd9b700 1 -- 172.21.15.23:0/2923098953 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8500053a0 con 0x7fc850004520 2020-11-20T11:32:38.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.915+0000 7fc86dd9b700 1 -- 172.21.15.23:0/2923098953 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8500042e0 con 0x7fc8500037a0 2020-11-20T11:32:38.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.915+0000 7fc86dd9b700 1 -- 172.21.15.23:0/2923098953 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc85000b770 con 0x7fc8700ace70 2020-11-20T11:32:38.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.915+0000 7fc86e59c700 1 --2- 172.21.15.23:0/2923098953 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc850004520 0x7fc8500049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:38.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.915+0000 7fc86ed9d700 1 --2- 172.21.15.23:0/2923098953 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc8500037a0 0x7fc850003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:38.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.915+0000 7fc86f59e700 1 --2- 172.21.15.23:0/2923098953 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc8700ace70 0x7fc850004e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:38.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.915+0000 7fc86e59c700 1 -- 172.21.15.23:0/2923098953 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc8700ace70 msgr2=0x7fc850004e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:38.928 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.915+0000 7fc86e59c700 1 --2- 172.21.15.23:0/2923098953 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc8700ace70 0x7fc850004e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.928 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.915+0000 7fc86e59c700 1 -- 172.21.15.23:0/2923098953 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc8500037a0 msgr2=0x7fc850003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:38.928 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.915+0000 7fc86e59c700 1 --2- 172.21.15.23:0/2923098953 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc8500037a0 0x7fc850003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.928 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.915+0000 7fc86e59c700 1 -- 172.21.15.23:0/2923098953 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc850003280 con 0x7fc850004520 2020-11-20T11:32:38.928 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.915+0000 7fc86f59e700 1 --2- 172.21.15.23:0/2923098953 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc8700ace70 0x7fc850004e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:32:38.929 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.915+0000 7fc86ed9d700 1 --2- 172.21.15.23:0/2923098953 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc8500037a0 0x7fc850003bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:38.929 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.915+0000 7fc86e59c700 1 --2- 172.21.15.23:0/2923098953 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc850004520 0x7fc8500049a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fc85800bc40 tx=0x7fc858006e30).ready entity=mon.0 client_cookie=411ca22bf9446c75 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:38.929 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.916+0000 7fc86dd9b700 1 -- 172.21.15.23:0/2923098953 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85800b350 con 0x7fc850004520 2020-11-20T11:32:38.929 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.916+0000 7fc86dd9b700 1 -- 172.21.15.23:0/2923098953 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc85800eab0 con 0x7fc850004520 2020-11-20T11:32:38.929 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.916+0000 7fc86dd9b700 1 -- 172.21.15.23:0/2923098953 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85800ee30 con 0x7fc850004520 2020-11-20T11:32:38.930 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.916+0000 7fc874a52700 1 -- 172.21.15.23:0/2923098953 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc850004520 msgr2=0x7fc8500049a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:38.930 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.916+0000 7fc874a52700 1 --2- 172.21.15.23:0/2923098953 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc850004520 0x7fc8500049a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fc85800bc40 tx=0x7fc858006e30).stop 2020-11-20T11:32:38.930 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.916+0000 7fc86f59e700 1 -- 172.21.15.23:0/2923098953 reap_dead start 2020-11-20T11:32:38.930 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.916+0000 7fc874a52700 1 -- 172.21.15.23:0/2923098953 shutdown_connections 2020-11-20T11:32:38.930 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.916+0000 7fc874a52700 1 -- 172.21.15.23:0/2923098953 >> 172.21.15.23:0/2923098953 conn(0x7fc8700683b0 msgr2=0x7fc8700b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:38.931 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.916+0000 7fc874a52700 1 -- 172.21.15.23:0/2923098953 shutdown_connections 2020-11-20T11:32:38.931 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.916+0000 7fc874a52700 1 -- 172.21.15.23:0/2923098953 wait complete. 2020-11-20T11:32:38.931 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.917+0000 7fc874a52700 1 Processor -- start 2020-11-20T11:32:38.931 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.917+0000 7fc874a52700 1 -- start start 2020-11-20T11:32:38.931 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.918+0000 7fc874a52700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc850004520 0x7fc870065da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:38.932 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.918+0000 7fc874a52700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc87013dbd0 0x7fc8700636b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:38.932 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.918+0000 7fc874a52700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc870063ba0 0x7fc870151690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:38.932 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.918+0000 7fc874a52700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87013e050 con 0x7fc870063ba0 2020-11-20T11:32:38.932 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.918+0000 7fc874a52700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8700a73d0 con 0x7fc87013dbd0 2020-11-20T11:32:38.932 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.918+0000 7fc874a52700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8700a7670 con 0x7fc850004520 2020-11-20T11:32:38.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.918+0000 7fc86f59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc870063ba0 0x7fc870151690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:38.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.918+0000 7fc86f59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc870063ba0 0x7fc870151690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41218/0 (socket says 172.21.15.23:41218) 2020-11-20T11:32:38.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.918+0000 7fc86f59e700 1 -- 172.21.15.23:0/4032696730 learned_addr learned my addr 172.21.15.23:0/4032696730 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:38.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.918+0000 7fc86e59c700 1 --2- 172.21.15.23:0/4032696730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc87013dbd0 0x7fc8700636b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:38.934 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.918+0000 7fc86ed9d700 1 --2- 172.21.15.23:0/4032696730 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc850004520 0x7fc870065da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:38.934 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.918+0000 7fc86f59e700 1 -- 172.21.15.23:0/4032696730 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc850004520 msgr2=0x7fc870065da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:38.934 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.918+0000 7fc86f59e700 1 --2- 172.21.15.23:0/4032696730 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc850004520 0x7fc870065da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.934 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.918+0000 7fc86f59e700 1 -- 172.21.15.23:0/4032696730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc87013dbd0 msgr2=0x7fc8700636b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:38.934 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.918+0000 7fc86f59e700 1 --2- 172.21.15.23:0/4032696730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc87013dbd0 0x7fc8700636b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:38.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.918+0000 7fc86f59e700 1 -- 172.21.15.23:0/4032696730 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8700a7910 con 0x7fc870063ba0 2020-11-20T11:32:38.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.918+0000 7fc86f59e700 1 --2- 172.21.15.23:0/4032696730 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc870063ba0 0x7fc870151690 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fc870063270 tx=0x7fc868009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:38.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.919+0000 7fc866ffd700 1 -- 172.21.15.23:0/4032696730 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86800f510 con 0x7fc870063ba0 2020-11-20T11:32:38.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.919+0000 7fc866ffd700 1 -- 172.21.15.23:0/4032696730 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc86800f670 con 0x7fc870063ba0 2020-11-20T11:32:38.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.919+0000 7fc874a52700 1 -- 172.21.15.23:0/4032696730 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc870067680 con 0x7fc870063ba0 2020-11-20T11:32:38.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.919+0000 7fc866ffd700 1 -- 172.21.15.23:0/4032696730 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86800fa10 con 0x7fc870063ba0 2020-11-20T11:32:38.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.919+0000 7fc874a52700 1 -- 172.21.15.23:0/4032696730 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc870151b80 con 0x7fc870063ba0 2020-11-20T11:32:38.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.920+0000 7fc874a52700 1 -- 172.21.15.23:0/4032696730 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc870057dc0 con 0x7fc870063ba0 2020-11-20T11:32:38.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.920+0000 7fc866ffd700 1 -- 172.21.15.23:0/4032696730 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc868013420 con 0x7fc870063ba0 2020-11-20T11:32:38.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.920+0000 7fc866ffd700 1 --2- 172.21.15.23:0/4032696730 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc84c056d60 0x7fc84c059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:38.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.920+0000 7fc866ffd700 1 -- 172.21.15.23:0/4032696730 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fc86806fc50 con 0x7fc870063ba0 2020-11-20T11:32:38.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.920+0000 7fc86ed9d700 1 --2- 172.21.15.23:0/4032696730 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc84c056d60 0x7fc84c059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:38.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.921+0000 7fc86ed9d700 1 --2- 172.21.15.23:0/4032696730 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc84c056d60 0x7fc84c059220 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fc860001150 tx=0x7fc860000f90).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:38.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:38.924+0000 7fc866ffd700 1 -- 172.21.15.23:0/4032696730 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc868044f60 con 0x7fc870063ba0 2020-11-20T11:32:39.127 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:39.123+0000 7fc874a52700 1 -- 172.21.15.23:0/4032696730 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc870044710 con 0x7fc870063ba0 2020-11-20T11:32:39.128 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:39.124+0000 7fc866ffd700 1 -- 172.21.15.23:0/4032696730 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+2128 (secure 0 0 0) 0x7fc868048580 con 0x7fc870063ba0 2020-11-20T11:32:39.128 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:32:39.129 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4223,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/564012815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":564012815},{"type":"v1","addr":"172.21.15.155:6835","nonce":564012815}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T11:32:25.469149+0000","modified":"2020-11-20T11:32:28.118344+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.23:6833/919963501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":919963501},{"type":"v1","addr":"172.21.15.23:6833","nonce":919963501}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T11:32:39.130 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:39.125+0000 7fc874a52700 1 -- 172.21.15.23:0/4032696730 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc84c056d60 msgr2=0x7fc84c059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:39.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:39.125+0000 7fc874a52700 1 --2- 172.21.15.23:0/4032696730 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc84c056d60 0x7fc84c059220 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fc860001150 tx=0x7fc860000f90).stop 2020-11-20T11:32:39.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:39.125+0000 7fc874a52700 1 -- 172.21.15.23:0/4032696730 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc870063ba0 msgr2=0x7fc870151690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:39.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:39.125+0000 7fc874a52700 1 --2- 172.21.15.23:0/4032696730 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc870063ba0 0x7fc870151690 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fc870063270 tx=0x7fc868009040).stop 2020-11-20T11:32:39.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:39.125+0000 7fc874a52700 1 -- 172.21.15.23:0/4032696730 shutdown_connections 2020-11-20T11:32:39.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:39.125+0000 7fc874a52700 1 --2- 172.21.15.23:0/4032696730 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc870063ba0 0x7fc870151690 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:39.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:39.125+0000 7fc874a52700 1 --2- 172.21.15.23:0/4032696730 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc84c056d60 0x7fc84c059220 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:39.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:39.125+0000 7fc874a52700 1 --2- 172.21.15.23:0/4032696730 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc850004520 0x7fc870065da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:39.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:39.125+0000 7fc874a52700 1 --2- 172.21.15.23:0/4032696730 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc87013dbd0 0x7fc8700636b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:39.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:39.125+0000 7fc874a52700 1 -- 172.21.15.23:0/4032696730 >> 172.21.15.23:0/4032696730 conn(0x7fc8700683b0 msgr2=0x7fc87013a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:39.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:39.125+0000 7fc874a52700 1 -- 172.21.15.23:0/4032696730 shutdown_connections 2020-11-20T11:32:39.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:39.125+0000 7fc874a52700 1 -- 172.21.15.23:0/4032696730 wait complete. 2020-11-20T11:32:39.136 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 7 2020-11-20T11:32:39.341 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T11:32:25.469149+0000', 'modified': '2020-11-20T11:32:28.118344+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4257}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4257': {'gid': 4257, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.23:6833/919963501', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6832', 'nonce': 919963501}, {'type': 'v1', 'addr': '172.21.15.23:6833', 'nonce': 919963501}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T11:32:39.342 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T11:32:39.342 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T11:32:39.477 INFO:teuthology.orchestra.run.smithi023.stdout:{"cluster_fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","whoami":0,"id":4257,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":13.994772292,"mdsmap_epoch":7,"osdmap_epoch":25,"osdmap_epoch_barrier":25,"uptime":20.442092911} 2020-11-20T11:32:39.477 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", "fs_name": "cephfs", "id": 4257, "mdsmap_epoch": 7, "osdmap_epoch": 25, "osdmap_epoch_barrier": 25, "rank_uptime": 13.994772292, "state": "up:active", "uptime": 20.442092911, "want_state": "up:active", "whoami": 0 } 2020-11-20T11:32:39.478 INFO:teuthology.run_tasks:Running task ceph-fuse... 2020-11-20T11:32:39.502 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2020-11-20T11:32:39.503 INFO:tasks.ceph_fuse:config is None 2020-11-20T11:32:39.503 INFO:tasks.ceph_fuse:new config is {'client.0': None} 2020-11-20T11:32:39.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T11:32:39.504 INFO:teuthology.orchestra.run.smithi023:> ip netns list 2020-11-20T11:32:39.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T11:32:39.520 INFO:teuthology.orchestra.run.smithi023:> sudo ip link delete ceph-brx 2020-11-20T11:32:39.586 INFO:teuthology.orchestra.run.smithi023.stderr:Cannot find device "ceph-brx" 2020-11-20T11:32:39.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-11-20T11:32:39.588 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2020-11-20T11:32:39.588 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2020-11-20T11:32:39.589 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi023.front.sepia.ceph.com 2020-11-20T11:32:39.589 INFO:tasks.cephfs.mount:self.client.name = client.0 2020-11-20T11:32:39.589 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2020-11-20T11:32:39.590 INFO:tasks.cephfs.mount:self.cephfs_name = None 2020-11-20T11:32:39.590 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2020-11-20T11:32:39.590 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2020-11-20T11:32:39.591 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2020-11-20T11:32:39.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T11:32:39.591 INFO:teuthology.orchestra.run.smithi023:> ip addr 2020-11-20T11:32:39.642 INFO:teuthology.orchestra.run.smithi023.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2020-11-20T11:32:39.643 INFO:teuthology.orchestra.run.smithi023.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2020-11-20T11:32:39.643 INFO:teuthology.orchestra.run.smithi023.stdout: inet 127.0.0.1/8 scope host lo 2020-11-20T11:32:39.643 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2020-11-20T11:32:39.643 INFO:teuthology.orchestra.run.smithi023.stdout: inet6 ::1/128 scope host 2020-11-20T11:32:39.644 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2020-11-20T11:32:39.644 INFO:teuthology.orchestra.run.smithi023.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T11:32:39.644 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:6b:fd:ae brd ff:ff:ff:ff:ff:ff 2020-11-20T11:32:39.644 INFO:teuthology.orchestra.run.smithi023.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2020-11-20T11:32:39.644 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:59:ce:42 brd ff:ff:ff:ff:ff:ff 2020-11-20T11:32:39.645 INFO:teuthology.orchestra.run.smithi023.stdout: inet 172.21.15.23/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2020-11-20T11:32:39.645 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft 42804sec preferred_lft 42804sec 2020-11-20T11:32:39.645 INFO:teuthology.orchestra.run.smithi023.stdout: inet6 fe80::ec4:7aff:fe59:ce42/64 scope link 2020-11-20T11:32:39.645 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2020-11-20T11:32:39.645 INFO:teuthology.orchestra.run.smithi023.stdout:4: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T11:32:39.646 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:6b:fd:af brd ff:ff:ff:ff:ff:ff 2020-11-20T11:32:39.646 INFO:teuthology.orchestra.run.smithi023.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2020-11-20T11:32:39.646 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:59:ce:43 brd ff:ff:ff:ff:ff:ff 2020-11-20T11:32:39.646 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2020-11-20T11:32:39.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T11:32:39.647 INFO:teuthology.orchestra.run.smithi023:> (cd / && exec sudo bash -c ' 2020-11-20T11:32:39.648 INFO:teuthology.orchestra.run.smithi023:> set -e 2020-11-20T11:32:39.648 INFO:teuthology.orchestra.run.smithi023:> sudo ip link add name ceph-brx type bridge 2020-11-20T11:32:39.648 INFO:teuthology.orchestra.run.smithi023:> sudo ip addr flush dev ceph-brx 2020-11-20T11:32:39.649 INFO:teuthology.orchestra.run.smithi023:> sudo ip link set ceph-brx up 2020-11-20T11:32:39.649 INFO:teuthology.orchestra.run.smithi023:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2020-11-20T11:32:39.649 INFO:teuthology.orchestra.run.smithi023:> ') 2020-11-20T11:32:39.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T11:32:39.813 INFO:teuthology.orchestra.run.smithi023:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2020-11-20T11:32:39.850 INFO:teuthology.orchestra.run.smithi023.stdout:1 2020-11-20T11:32:39.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T11:32:39.851 INFO:teuthology.orchestra.run.smithi023:> route 2020-11-20T11:32:39.920 INFO:teuthology.orchestra.run.smithi023.stdout:Kernel IP routing table 2020-11-20T11:32:39.920 INFO:teuthology.orchestra.run.smithi023.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2020-11-20T11:32:39.920 INFO:teuthology.orchestra.run.smithi023.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2020-11-20T11:32:39.921 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2020-11-20T11:32:39.921 INFO:teuthology.orchestra.run.smithi023.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2020-11-20T11:32:39.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T11:32:39.922 INFO:teuthology.orchestra.run.smithi023:> (cd / && exec sudo bash -c ' 2020-11-20T11:32:39.922 INFO:teuthology.orchestra.run.smithi023:> set -e 2020-11-20T11:32:39.923 INFO:teuthology.orchestra.run.smithi023:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2020-11-20T11:32:39.923 INFO:teuthology.orchestra.run.smithi023:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2020-11-20T11:32:39.923 INFO:teuthology.orchestra.run.smithi023:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2020-11-20T11:32:39.923 INFO:teuthology.orchestra.run.smithi023:> ') 2020-11-20T11:32:40.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T11:32:40.175 INFO:teuthology.orchestra.run.smithi023:> ip netns list 2020-11-20T11:32:40.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T11:32:40.192 INFO:teuthology.orchestra.run.smithi023:> ip netns list-id 2020-11-20T11:32:40.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T11:32:40.246 INFO:teuthology.orchestra.run.smithi023:> (cd / && exec sudo bash -c ' 2020-11-20T11:32:40.246 INFO:teuthology.orchestra.run.smithi023:> set -e 2020-11-20T11:32:40.247 INFO:teuthology.orchestra.run.smithi023:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2020-11-20T11:32:40.247 INFO:teuthology.orchestra.run.smithi023:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2020-11-20T11:32:40.247 INFO:teuthology.orchestra.run.smithi023:> ') 2020-11-20T11:32:40.335 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2020-11-20T11:32:40.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T11:32:40.336 INFO:teuthology.orchestra.run.smithi023:> (cd / && exec sudo bash -c ' 2020-11-20T11:32:40.336 INFO:teuthology.orchestra.run.smithi023:> set -e 2020-11-20T11:32:40.336 INFO:teuthology.orchestra.run.smithi023:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2020-11-20T11:32:40.337 INFO:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2020-11-20T11:32:40.337 INFO:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2020-11-20T11:32:40.337 INFO:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2020-11-20T11:32:40.338 INFO:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2020-11-20T11:32:40.338 INFO:teuthology.orchestra.run.smithi023:> ') 2020-11-20T11:32:40.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T11:32:40.519 INFO:teuthology.orchestra.run.smithi023:> (cd / && exec sudo bash -c ' 2020-11-20T11:32:40.520 INFO:teuthology.orchestra.run.smithi023:> set -e 2020-11-20T11:32:40.520 INFO:teuthology.orchestra.run.smithi023:> sudo ip link set brx.0 up 2020-11-20T11:32:40.520 INFO:teuthology.orchestra.run.smithi023:> sudo ip link set dev brx.0 master ceph-brx 2020-11-20T11:32:40.521 INFO:teuthology.orchestra.run.smithi023:> ') 2020-11-20T11:32:40.618 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T11:32:40.736 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.733+0000 7fc1531b7700 1 Processor -- start 2020-11-20T11:32:40.736 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.733+0000 7fc1531b7700 1 -- start start 2020-11-20T11:32:40.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.734+0000 7fc1531b7700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc14c137410 0x7fc14c137870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:40.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.734+0000 7fc1531b7700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc14c062ad0 con 0x7fc14c0a6e10 2020-11-20T11:32:40.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.734+0000 7fc1531b7700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc14c0a63e0 con 0x7fc14c133dc0 2020-11-20T11:32:40.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.734+0000 7fc1531b7700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc14c0a61f0 con 0x7fc14c137410 2020-11-20T11:32:40.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.734+0000 7fc150f53700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fc14c0a6e10 0x7fc14c061850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55476/0 (socket says 172.21.15.23:55476) 2020-11-20T11:32:40.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.734+0000 7fc150f53700 1 -- 172.21.15.23:0/1350469134 learned_addr learned my addr 172.21.15.23:0/1350469134 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:40.744 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.734+0000 7fc14bfff700 1 --2- 172.21.15.23:0/1350469134 >> v2:172.21.15.155:3300/0 conn(0x7fc14c137410 0x7fc14c137870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:40.744 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.734+0000 7fc14b7fe700 1 -- 172.21.15.23:0/1350469134 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2760302687 0 0) 0x7fc14c062ad0 con 0x7fc14c0a6e10 2020-11-20T11:32:40.744 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.734+0000 7fc14b7fe700 1 -- 172.21.15.23:0/1350469134 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc130003280 con 0x7fc14c0a6e10 2020-11-20T11:32:40.744 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.735+0000 7fc14b7fe700 1 -- 172.21.15.23:0/1350469134 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262731652 0 0) 0x7fc14c0a63e0 con 0x7fc14c133dc0 2020-11-20T11:32:40.744 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.735+0000 7fc14b7fe700 1 -- 172.21.15.23:0/1350469134 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc14c062ad0 con 0x7fc14c133dc0 2020-11-20T11:32:40.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.735+0000 7fc14b7fe700 1 -- 172.21.15.23:0/1350469134 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 724212572 0 0) 0x7fc130003280 con 0x7fc14c0a6e10 2020-11-20T11:32:40.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.735+0000 7fc14b7fe700 1 -- 172.21.15.23:0/1350469134 >> v2:172.21.15.155:3300/0 conn(0x7fc14c137410 msgr2=0x7fc14c137870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:40.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.735+0000 7fc14b7fe700 1 --2- 172.21.15.23:0/1350469134 >> v2:172.21.15.155:3300/0 conn(0x7fc14c137410 0x7fc14c137870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:40.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.735+0000 7fc14b7fe700 1 -- 172.21.15.23:0/1350469134 >> v1:172.21.15.155:6789/0 conn(0x7fc14c133dc0 legacy=0x7fc14c1341d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:40.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.735+0000 7fc14b7fe700 1 -- 172.21.15.23:0/1350469134 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc14c0a6680 con 0x7fc14c0a6e10 2020-11-20T11:32:40.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.735+0000 7fc14b7fe700 1 -- 172.21.15.23:0/1350469134 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7fc140001520 con 0x7fc14c0a6e10 2020-11-20T11:32:40.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.735+0000 7fc14b7fe700 1 -- 172.21.15.23:0/1350469134 >> v1:172.21.15.23:6789/0 conn(0x7fc14c0a6e10 legacy=0x7fc14c061850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:40.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.735+0000 7fc14b7fe700 1 --2- 172.21.15.23:0/1350469134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc130003900 0x7fc130003d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:40.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.735+0000 7fc14b7fe700 1 --2- 172.21.15.23:0/1350469134 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc130004520 0x7fc1300049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:40.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.735+0000 7fc14b7fe700 1 --2- 172.21.15.23:0/1350469134 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc130004e90 0x7fc13000b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:40.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.735+0000 7fc14b7fe700 1 -- 172.21.15.23:0/1350469134 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1300053c0 con 0x7fc130003900 2020-11-20T11:32:40.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.735+0000 7fc14b7fe700 1 -- 172.21.15.23:0/1350469134 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13000bc60 con 0x7fc130004e90 2020-11-20T11:32:40.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.735+0000 7fc14b7fe700 1 -- 172.21.15.23:0/1350469134 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc13000be20 con 0x7fc130004520 2020-11-20T11:32:40.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.735+0000 7fc150f53700 1 --2- 172.21.15.23:0/1350469134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc130003900 0x7fc130003d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:40.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.735+0000 7fc14bfff700 1 --2- 172.21.15.23:0/1350469134 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc130004520 0x7fc1300049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:40.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.735+0000 7fc151754700 1 --2- 172.21.15.23:0/1350469134 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc130004e90 0x7fc13000b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:40.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.735+0000 7fc150f53700 1 -- 172.21.15.23:0/1350469134 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc130004520 msgr2=0x7fc1300049a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:40.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.735+0000 7fc150f53700 1 --2- 172.21.15.23:0/1350469134 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc130004520 0x7fc1300049a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:40.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.735+0000 7fc150f53700 1 -- 172.21.15.23:0/1350469134 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc130004e90 msgr2=0x7fc13000b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:40.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.735+0000 7fc150f53700 1 --2- 172.21.15.23:0/1350469134 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc130004e90 0x7fc13000b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:40.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.735+0000 7fc150f53700 1 -- 172.21.15.23:0/1350469134 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc130003280 con 0x7fc130003900 2020-11-20T11:32:40.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.735+0000 7fc14bfff700 1 --2- 172.21.15.23:0/1350469134 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc130004520 0x7fc1300049a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:40.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.735+0000 7fc151754700 1 --2- 172.21.15.23:0/1350469134 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc130004e90 0x7fc13000b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:40.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.735+0000 7fc151754700 1 -- 172.21.15.23:0/1350469134 reap_dead start 2020-11-20T11:32:40.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.735+0000 7fc150f53700 1 --2- 172.21.15.23:0/1350469134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc130003900 0x7fc130003d20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fc14000c420 tx=0x7fc14c0a6810).ready entity=mon.0 client_cookie=2542c81ef1b65f7b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:40.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.735+0000 7fc14b7fe700 1 -- 172.21.15.23:0/1350469134 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14000ce00 con 0x7fc130003900 2020-11-20T11:32:40.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.735+0000 7fc14b7fe700 1 -- 172.21.15.23:0/1350469134 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1400072e0 con 0x7fc130003900 2020-11-20T11:32:40.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.735+0000 7fc14b7fe700 1 -- 172.21.15.23:0/1350469134 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc140005880 con 0x7fc130003900 2020-11-20T11:32:40.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.736+0000 7fc1531b7700 1 -- 172.21.15.23:0/1350469134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc130003900 msgr2=0x7fc130003d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:40.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.736+0000 7fc1531b7700 1 --2- 172.21.15.23:0/1350469134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc130003900 0x7fc130003d20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fc14000c420 tx=0x7fc14c0a6810).stop 2020-11-20T11:32:40.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.736+0000 7fc1531b7700 1 -- 172.21.15.23:0/1350469134 shutdown_connections 2020-11-20T11:32:40.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.736+0000 7fc1531b7700 1 --2- 172.21.15.23:0/1350469134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc130003900 0x7fc130003d20 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:40.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.736+0000 7fc1531b7700 1 -- 172.21.15.23:0/1350469134 >> 172.21.15.23:0/1350469134 conn(0x7fc14c060c80 msgr2=0x7fc14c0acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:40.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.736+0000 7fc1531b7700 1 -- 172.21.15.23:0/1350469134 shutdown_connections 2020-11-20T11:32:40.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.736+0000 7fc1531b7700 1 -- 172.21.15.23:0/1350469134 wait complete. 2020-11-20T11:32:40.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.737+0000 7fc1531b7700 1 Processor -- start 2020-11-20T11:32:40.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.737+0000 7fc1531b7700 1 -- start start 2020-11-20T11:32:40.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.738+0000 7fc1531b7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc130003900 0x7fc14c14e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:40.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.738+0000 7fc1531b7700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc14c0a9a30 0x7fc14c14a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:40.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.738+0000 7fc1531b7700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc14c1321e0 0x7fc14c132640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:40.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.738+0000 7fc1531b7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14c14b110 con 0x7fc130003900 2020-11-20T11:32:40.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.738+0000 7fc1531b7700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14c0a9eb0 con 0x7fc14c1321e0 2020-11-20T11:32:40.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.738+0000 7fc1531b7700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc14c12bfa0 con 0x7fc14c0a9a30 2020-11-20T11:32:40.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.738+0000 7fc150f53700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc130003900 0x7fc14c14e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:40.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.738+0000 7fc150f53700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc130003900 0x7fc14c14e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41236/0 (socket says 172.21.15.23:41236) 2020-11-20T11:32:40.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.738+0000 7fc150f53700 1 -- 172.21.15.23:0/2574440629 learned_addr learned my addr 172.21.15.23:0/2574440629 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:40.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.738+0000 7fc14bfff700 1 --2- 172.21.15.23:0/2574440629 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc14c0a9a30 0x7fc14c14a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:40.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.738+0000 7fc151754700 1 --2- 172.21.15.23:0/2574440629 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc14c1321e0 0x7fc14c132640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:40.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.738+0000 7fc150f53700 1 -- 172.21.15.23:0/2574440629 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc14c0a9a30 msgr2=0x7fc14c14a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:40.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.738+0000 7fc150f53700 1 --2- 172.21.15.23:0/2574440629 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc14c0a9a30 0x7fc14c14a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:40.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.738+0000 7fc150f53700 1 -- 172.21.15.23:0/2574440629 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc14c1321e0 msgr2=0x7fc14c132640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:40.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.738+0000 7fc150f53700 1 --2- 172.21.15.23:0/2574440629 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc14c1321e0 0x7fc14c132640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:40.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.738+0000 7fc150f53700 1 -- 172.21.15.23:0/2574440629 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc14c14a650 con 0x7fc130003900 2020-11-20T11:32:40.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.738+0000 7fc14bfff700 1 --2- 172.21.15.23:0/2574440629 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc14c0a9a30 0x7fc14c14a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:40.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.738+0000 7fc151754700 1 --2- 172.21.15.23:0/2574440629 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc14c1321e0 0x7fc14c132640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:40.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.738+0000 7fc150f53700 1 --2- 172.21.15.23:0/2574440629 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc130003900 0x7fc14c14e290 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fc140000f50 tx=0x7fc140015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:40.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.738+0000 7fc148ff9700 1 -- 172.21.15.23:0/2574440629 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc140007520 con 0x7fc130003900 2020-11-20T11:32:40.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.738+0000 7fc148ff9700 1 -- 172.21.15.23:0/2574440629 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc140001420 con 0x7fc130003900 2020-11-20T11:32:40.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.739+0000 7fc148ff9700 1 -- 172.21.15.23:0/2574440629 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc140014ad0 con 0x7fc130003900 2020-11-20T11:32:40.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.739+0000 7fc1531b7700 1 -- 172.21.15.23:0/2574440629 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc14c14d8b0 con 0x7fc130003900 2020-11-20T11:32:40.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.739+0000 7fc1531b7700 1 -- 172.21.15.23:0/2574440629 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc14c138380 con 0x7fc130003900 2020-11-20T11:32:40.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.739+0000 7fc1531b7700 1 -- 172.21.15.23:0/2574440629 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc14c05c520 con 0x7fc130003900 2020-11-20T11:32:40.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.740+0000 7fc148ff9700 1 -- 172.21.15.23:0/2574440629 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc140025020 con 0x7fc130003900 2020-11-20T11:32:40.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.740+0000 7fc148ff9700 1 --2- 172.21.15.23:0/2574440629 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc124056d90 0x7fc124059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:40.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.740+0000 7fc148ff9700 1 -- 172.21.15.23:0/2574440629 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fc140071660 con 0x7fc130003900 2020-11-20T11:32:40.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.741+0000 7fc14bfff700 1 --2- 172.21.15.23:0/2574440629 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc124056d90 0x7fc124059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:40.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.741+0000 7fc14bfff700 1 --2- 172.21.15.23:0/2574440629 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc124056d90 0x7fc124059250 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fc138006d40 tx=0x7fc138008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:40.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.744+0000 7fc148ff9700 1 -- 172.21.15.23:0/2574440629 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc1400468c0 con 0x7fc130003900 2020-11-20T11:32:40.897 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.895+0000 7fc1531b7700 1 -- 172.21.15.23:0/2574440629 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc14c14a650 con 0x7fc130003900 2020-11-20T11:32:40.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.896+0000 7fc148ff9700 1 -- 172.21.15.23:0/2574440629 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17087 (secure 0 0 0) 0x7fc14001a020 con 0x7fc130003900 2020-11-20T11:32:40.899 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:32:40.899 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":25,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","created":"2020-11-20T11:32:06.136679+0000","modified":"2020-11-20T11:32:26.470927+0000","last_up_change":"2020-11-20T11:32:13.536194+0000","last_in_change":"2020-11-20T11:32:13.536194+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T11:32:09.559868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T11:32:17.000782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T11:32:20.423341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T11:32:23.126273+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e9d219f5-e40e-4b2b-9468-deb9c4074a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":23083},{"type":"v1","addr":"172.21.15.23:6801","nonce":23083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":23083},{"type":"v1","addr":"172.21.15.23:6803","nonce":23083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":23083},{"type":"v1","addr":"172.21.15.23:6807","nonce":23083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":23083},{"type":"v1","addr":"172.21.15.23:6805","nonce":23083}]},"public_addr":"172.21.15.23:6801/23083","cluster_addr":"172.21.15.23:6803/23083","heartbeat_back_addr":"172.21.15.23:6807/23083","heartbeat_front_addr":"172.21.15.23:6805/23083","state":["exists","up"]},{"osd":1,"uuid":"2328adde-2108-4fe0-8138-f0d096a496a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":23099},{"type":"v1","addr":"172.21.15.23:6817","nonce":23099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":23099},{"type":"v1","addr":"172.21.15.23:6819","nonce":23099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":23099},{"type":"v1","addr":"172.21.15.23:6824","nonce":23099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":23099},{"type":"v1","addr":"172.21.15.23:6821","nonce":23099}]},"public_addr":"172.21.15.23:6817/23099","cluster_addr":"172.21.15.23:6819/23099","heartbeat_back_addr":"172.21.15.23:6824/23099","heartbeat_front_addr":"172.21.15.23:6821/23099","state":["exists","up"]},{"osd":2,"uuid":"13367708-59a2-4363-8a3a-ab0cb97145ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":23084},{"type":"v1","addr":"172.21.15.23:6809","nonce":23084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":23084},{"type":"v1","addr":"172.21.15.23:6811","nonce":23084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":23084},{"type":"v1","addr":"172.21.15.23:6815","nonce":23084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":23084},{"type":"v1","addr":"172.21.15.23:6813","nonce":23084}]},"public_addr":"172.21.15.23:6809/23084","cluster_addr":"172.21.15.23:6811/23084","heartbeat_back_addr":"172.21.15.23:6815/23084","heartbeat_front_addr":"172.21.15.23:6813/23084","state":["exists","up"]},{"osd":3,"uuid":"1f147dd7-9289-47b0-ad1c-4088618d6faa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6823","nonce":23114},{"type":"v1","addr":"172.21.15.23:6825","nonce":23114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":23114},{"type":"v1","addr":"172.21.15.23:6827","nonce":23114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":23114},{"type":"v1","addr":"172.21.15.23:6831","nonce":23114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":23114},{"type":"v1","addr":"172.21.15.23:6829","nonce":23114}]},"public_addr":"172.21.15.23:6825/23114","cluster_addr":"172.21.15.23:6827/23114","heartbeat_back_addr":"172.21.15.23:6831/23114","heartbeat_front_addr":"172.21.15.23:6829/23114","state":["exists","up"]},{"osd":4,"uuid":"d008b039-4bc5-4135-9678-1511e58efc29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":23280},{"type":"v1","addr":"172.21.15.155:6801","nonce":23280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":23280},{"type":"v1","addr":"172.21.15.155:6803","nonce":23280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":23280},{"type":"v1","addr":"172.21.15.155:6807","nonce":23280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":23280},{"type":"v1","addr":"172.21.15.155:6805","nonce":23280}]},"public_addr":"172.21.15.155:6801/23280","cluster_addr":"172.21.15.155:6803/23280","heartbeat_back_addr":"172.21.15.155:6807/23280","heartbeat_front_addr":"172.21.15.155:6805/23280","state":["exists","up"]},{"osd":5,"uuid":"56715e6f-cf6e-44a6-9e39-93ab1224095f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":23298},{"type":"v1","addr":"172.21.15.155:6817","nonce":23298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":23298},{"type":"v1","addr":"172.21.15.155:6819","nonce":23298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":23298},{"type":"v1","addr":"172.21.15.155:6823","nonce":23298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":23298},{"type":"v1","addr":"172.21.15.155:6821","nonce":23298}]},"public_addr":"172.21.15.155:6817/23298","cluster_addr":"172.21.15.155:6819/23298","heartbeat_back_addr":"172.21.15.155:6823/23298","heartbeat_front_addr":"172.21.15.155:6821/23298","state":["exists","up"]},{"osd":6,"uuid":"db1e61ca-8e14-4a1f-8607-6e01b148d61c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":23307},{"type":"v1","addr":"172.21.15.155:6825","nonce":23307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":23307},{"type":"v1","addr":"172.21.15.155:6827","nonce":23307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":23307},{"type":"v1","addr":"172.21.15.155:6831","nonce":23307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":23307},{"type":"v1","addr":"172.21.15.155:6829","nonce":23307}]},"public_addr":"172.21.15.155:6825/23307","cluster_addr":"172.21.15.155:6827/23307","heartbeat_back_addr":"172.21.15.155:6831/23307","heartbeat_front_addr":"172.21.15.155:6829/23307","state":["exists","up"]},{"osd":7,"uuid":"d44419be-ef89-462c-8cf1-3c6db2a6944b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":23281},{"type":"v1","addr":"172.21.15.155:6809","nonce":23281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":23281},{"type":"v1","addr":"172.21.15.155:6811","nonce":23281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":23281},{"type":"v1","addr":"172.21.15.155:6815","nonce":23281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":23281},{"type":"v1","addr":"172.21.15.155:6813","nonce":23281}]},"public_addr":"172.21.15.155:6809/23281","cluster_addr":"172.21.15.155:6811/23281","heartbeat_back_addr":"172.21.15.155:6815/23281","heartbeat_front_addr":"172.21.15.155:6813/23281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.486702+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.509396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.533414+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.509489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.467600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.513068+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.536636+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.531772+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T11:32:40.900 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.897+0000 7fc1531b7700 1 -- 172.21.15.23:0/2574440629 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc124056d90 msgr2=0x7fc124059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:40.901 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.897+0000 7fc1531b7700 1 --2- 172.21.15.23:0/2574440629 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc124056d90 0x7fc124059250 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fc138006d40 tx=0x7fc138008040).stop 2020-11-20T11:32:40.901 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.897+0000 7fc1531b7700 1 -- 172.21.15.23:0/2574440629 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc130003900 msgr2=0x7fc14c14e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:40.901 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.897+0000 7fc1531b7700 1 --2- 172.21.15.23:0/2574440629 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc130003900 0x7fc14c14e290 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fc140000f50 tx=0x7fc140015000).stop 2020-11-20T11:32:40.901 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.897+0000 7fc1531b7700 1 -- 172.21.15.23:0/2574440629 shutdown_connections 2020-11-20T11:32:40.901 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.897+0000 7fc1531b7700 1 --2- 172.21.15.23:0/2574440629 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc14c1321e0 0x7fc14c132640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:40.902 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.897+0000 7fc1531b7700 1 --2- 172.21.15.23:0/2574440629 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc130003900 0x7fc14c14e290 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:40.902 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.897+0000 7fc1531b7700 1 --2- 172.21.15.23:0/2574440629 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc124056d90 0x7fc124059250 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:40.902 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.897+0000 7fc1531b7700 1 --2- 172.21.15.23:0/2574440629 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc14c0a9a30 0x7fc14c14a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:40.902 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.897+0000 7fc1531b7700 1 -- 172.21.15.23:0/2574440629 >> 172.21.15.23:0/2574440629 conn(0x7fc14c060c80 msgr2=0x7fc14c061850 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:40.902 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.898+0000 7fc1531b7700 1 -- 172.21.15.23:0/2574440629 shutdown_connections 2020-11-20T11:32:40.903 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:40.898+0000 7fc1531b7700 1 -- 172.21.15.23:0/2574440629 wait complete. 2020-11-20T11:32:40.913 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T11:32:09.559868+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T11:32:17.000782+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T11:32:20.423341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T11:32:23.126273+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T11:32:40.914 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T11:32:41.031 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.028+0000 7fc20a213700 1 Processor -- start 2020-11-20T11:32:41.032 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.028+0000 7fc20a213700 1 -- start start 2020-11-20T11:32:41.034 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.029+0000 7fc20a213700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc204137870 0x7fc2041344d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:41.034 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.029+0000 7fc20a213700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc204062ad0 con 0x7fc204137410 2020-11-20T11:32:41.034 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.029+0000 7fc20a213700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2040a63e0 con 0x7fc2040a6e10 2020-11-20T11:32:41.034 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.029+0000 7fc20a213700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2040a61f0 con 0x7fc204137870 2020-11-20T11:32:41.035 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.029+0000 7fc202ffd700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fc204137410 0x7fc204133dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55498/0 (socket says 172.21.15.23:55498) 2020-11-20T11:32:41.035 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.029+0000 7fc202ffd700 1 -- 172.21.15.23:0/3148898046 learned_addr learned my addr 172.21.15.23:0/3148898046 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:41.035 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.029+0000 7fc203fff700 1 --2- 172.21.15.23:0/3148898046 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc204137870 0x7fc2041344d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:41.035 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.030+0000 7fc2027fc700 1 -- 172.21.15.23:0/3148898046 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2006709022 0 0) 0x7fc204062ad0 con 0x7fc204137410 2020-11-20T11:32:41.035 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.030+0000 7fc2027fc700 1 -- 172.21.15.23:0/3148898046 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1f0004390 con 0x7fc204137410 2020-11-20T11:32:41.036 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.030+0000 7fc2027fc700 1 -- 172.21.15.23:0/3148898046 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1929508608 0 0) 0x7fc2040a63e0 con 0x7fc2040a6e10 2020-11-20T11:32:41.036 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.030+0000 7fc2027fc700 1 -- 172.21.15.23:0/3148898046 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc204062ad0 con 0x7fc2040a6e10 2020-11-20T11:32:41.036 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.030+0000 7fc203fff700 1 -- 172.21.15.23:0/3148898046 >> v1:172.21.15.155:6789/0 conn(0x7fc2040a6e10 legacy=0x7fc204061850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:41.036 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.030+0000 7fc203fff700 1 -- 172.21.15.23:0/3148898046 >> v1:172.21.15.23:6789/0 conn(0x7fc204137410 legacy=0x7fc204133dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:41.036 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.030+0000 7fc203fff700 1 -- 172.21.15.23:0/3148898046 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2040a6680 con 0x7fc204137870 2020-11-20T11:32:41.036 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.030+0000 7fc203fff700 1 --2- 172.21.15.23:0/3148898046 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc204137870 0x7fc2041344d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fc2040a6110 tx=0x7fc1fc009220).ready entity=mon.2 client_cookie=be3799ec6fd9542b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:41.037 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.030+0000 7fc2027fc700 1 -- 172.21.15.23:0/3148898046 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1fc00b990 con 0x7fc204137870 2020-11-20T11:32:41.037 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.030+0000 7fc2027fc700 1 -- 172.21.15.23:0/3148898046 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1fc00baf0 con 0x7fc204137870 2020-11-20T11:32:41.037 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.030+0000 7fc2027fc700 1 -- 172.21.15.23:0/3148898046 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1fc00be70 con 0x7fc204137870 2020-11-20T11:32:41.037 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.031+0000 7fc20a213700 1 -- 172.21.15.23:0/3148898046 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc204137870 msgr2=0x7fc2041344d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:41.038 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.031+0000 7fc20a213700 1 --2- 172.21.15.23:0/3148898046 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc204137870 0x7fc2041344d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fc2040a6110 tx=0x7fc1fc009220).stop 2020-11-20T11:32:41.038 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.031+0000 7fc20a213700 1 -- 172.21.15.23:0/3148898046 shutdown_connections 2020-11-20T11:32:41.038 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.031+0000 7fc20a213700 1 --2- 172.21.15.23:0/3148898046 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc204137870 0x7fc2041344d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:41.038 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.031+0000 7fc20a213700 1 -- 172.21.15.23:0/3148898046 >> 172.21.15.23:0/3148898046 conn(0x7fc204060c80 msgr2=0x7fc2040acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:41.038 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.031+0000 7fc20a213700 1 -- 172.21.15.23:0/3148898046 shutdown_connections 2020-11-20T11:32:41.038 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.031+0000 7fc20a213700 1 -- 172.21.15.23:0/3148898046 wait complete. 2020-11-20T11:32:41.041 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.032+0000 7fc20a213700 1 Processor -- start 2020-11-20T11:32:41.041 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.032+0000 7fc20a213700 1 -- start start 2020-11-20T11:32:41.041 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.032+0000 7fc20a213700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc2040a6e10 0x7fc20409df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:41.042 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.032+0000 7fc20a213700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc204137410 0x7fc20409f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:41.042 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.032+0000 7fc20a213700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc204137870 0x7fc2040aba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:41.042 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.032+0000 7fc20a213700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20409e400 con 0x7fc2040a6e10 2020-11-20T11:32:41.042 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.032+0000 7fc20a213700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc204134ff0 con 0x7fc204137410 2020-11-20T11:32:41.042 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.032+0000 7fc20a213700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2040ad210 con 0x7fc204137870 2020-11-20T11:32:41.043 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.033+0000 7fc2037fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc2040a6e10 0x7fc20409df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:41.043 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.033+0000 7fc2037fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc2040a6e10 0x7fc20409df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41250/0 (socket says 172.21.15.23:41250) 2020-11-20T11:32:41.043 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.033+0000 7fc2037fe700 1 -- 172.21.15.23:0/634542811 learned_addr learned my addr 172.21.15.23:0/634542811 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:41.043 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.033+0000 7fc203fff700 1 --2- 172.21.15.23:0/634542811 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc204137870 0x7fc2040aba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:41.043 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.033+0000 7fc202ffd700 1 --2- 172.21.15.23:0/634542811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc204137410 0x7fc20409f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:41.044 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.033+0000 7fc2037fe700 1 -- 172.21.15.23:0/634542811 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc204137870 msgr2=0x7fc2040aba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:41.044 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.033+0000 7fc2037fe700 1 --2- 172.21.15.23:0/634542811 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc204137870 0x7fc2040aba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:41.044 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.033+0000 7fc2037fe700 1 -- 172.21.15.23:0/634542811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc204137410 msgr2=0x7fc20409f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:41.044 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.033+0000 7fc2037fe700 1 --2- 172.21.15.23:0/634542811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc204137410 0x7fc20409f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:41.044 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.033+0000 7fc2037fe700 1 -- 172.21.15.23:0/634542811 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2040ad4b0 con 0x7fc2040a6e10 2020-11-20T11:32:41.045 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.033+0000 7fc2037fe700 1 --2- 172.21.15.23:0/634542811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc2040a6e10 0x7fc20409df10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fc1f400c710 tx=0x7fc1f4007ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:41.045 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.033+0000 7fc1e37fe700 1 -- 172.21.15.23:0/634542811 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f4010510 con 0x7fc2040a6e10 2020-11-20T11:32:41.045 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.033+0000 7fc1e37fe700 1 -- 172.21.15.23:0/634542811 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1f4010670 con 0x7fc2040a6e10 2020-11-20T11:32:41.045 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.033+0000 7fc20a213700 1 -- 172.21.15.23:0/634542811 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2040ad750 con 0x7fc2040a6e10 2020-11-20T11:32:41.045 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.033+0000 7fc1e37fe700 1 -- 172.21.15.23:0/634542811 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f4010a10 con 0x7fc2040a6e10 2020-11-20T11:32:41.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.033+0000 7fc20a213700 1 -- 172.21.15.23:0/634542811 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2040ac540 con 0x7fc2040a6e10 2020-11-20T11:32:41.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.034+0000 7fc20a213700 1 -- 172.21.15.23:0/634542811 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc20405c520 con 0x7fc2040a6e10 2020-11-20T11:32:41.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.034+0000 7fc1e37fe700 1 -- 172.21.15.23:0/634542811 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc1f4010070 con 0x7fc2040a6e10 2020-11-20T11:32:41.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.035+0000 7fc1e37fe700 1 --2- 172.21.15.23:0/634542811 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc1dc056d60 0x7fc1dc059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:41.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.035+0000 7fc1e37fe700 1 -- 172.21.15.23:0/634542811 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fc1f406fc10 con 0x7fc2040a6e10 2020-11-20T11:32:41.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.035+0000 7fc202ffd700 1 --2- 172.21.15.23:0/634542811 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc1dc056d60 0x7fc1dc059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:41.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.035+0000 7fc202ffd700 1 --2- 172.21.15.23:0/634542811 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc1dc056d60 0x7fc1dc059220 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fc20413a4a0 tx=0x7fc1f8009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:41.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.038+0000 7fc1e37fe700 1 -- 172.21.15.23:0/634542811 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc1f4044f00 con 0x7fc2040a6e10 2020-11-20T11:32:41.191 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.188+0000 7fc20a213700 1 -- 172.21.15.23:0/634542811 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fc204044710 con 0x7fc2040a6e10 2020-11-20T11:32:41.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.189+0000 7fc1e37fe700 1 -- 172.21.15.23:0/634542811 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v25) v1 ==== 110+0+10 (secure 0 0 0) 0x7fc1f4048520 con 0x7fc2040a6e10 2020-11-20T11:32:41.192 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2020-11-20T11:32:41.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.190+0000 7fc20a213700 1 -- 172.21.15.23:0/634542811 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc1dc056d60 msgr2=0x7fc1dc059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:41.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.190+0000 7fc20a213700 1 --2- 172.21.15.23:0/634542811 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc1dc056d60 0x7fc1dc059220 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fc20413a4a0 tx=0x7fc1f8009040).stop 2020-11-20T11:32:41.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.190+0000 7fc20a213700 1 -- 172.21.15.23:0/634542811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc2040a6e10 msgr2=0x7fc20409df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:41.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.190+0000 7fc20a213700 1 --2- 172.21.15.23:0/634542811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc2040a6e10 0x7fc20409df10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fc1f400c710 tx=0x7fc1f4007ad0).stop 2020-11-20T11:32:41.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.190+0000 7fc20a213700 1 -- 172.21.15.23:0/634542811 shutdown_connections 2020-11-20T11:32:41.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.190+0000 7fc20a213700 1 --2- 172.21.15.23:0/634542811 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc1dc056d60 0x7fc1dc059220 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:41.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.190+0000 7fc20a213700 1 --2- 172.21.15.23:0/634542811 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc204137870 0x7fc2040aba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:41.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.190+0000 7fc20a213700 1 --2- 172.21.15.23:0/634542811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc2040a6e10 0x7fc20409df10 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:41.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.190+0000 7fc20a213700 1 --2- 172.21.15.23:0/634542811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc204137410 0x7fc20409f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:41.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.190+0000 7fc20a213700 1 -- 172.21.15.23:0/634542811 >> 172.21.15.23:0/634542811 conn(0x7fc204060c80 msgr2=0x7fc20413bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:41.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.190+0000 7fc20a213700 1 -- 172.21.15.23:0/634542811 shutdown_connections 2020-11-20T11:32:41.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.190+0000 7fc20a213700 1 -- 172.21.15.23:0/634542811 wait complete. 2020-11-20T11:32:41.204 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T11:32:41.321 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.318+0000 7fb0a7ecc700 1 Processor -- start 2020-11-20T11:32:41.321 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.318+0000 7fb0a7ecc700 1 -- start start 2020-11-20T11:32:41.324 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.319+0000 7fb0a7ecc700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0a00ad9e0 0x7fb0a013af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:41.324 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.319+0000 7fb0a7ecc700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a013be80 con 0x7fb0a00a6a70 2020-11-20T11:32:41.325 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.319+0000 7fb0a7ecc700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a00a6f10 con 0x7fb0a00ad600 2020-11-20T11:32:41.325 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.319+0000 7fb0a7ecc700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a00a71b0 con 0x7fb0a00ad9e0 2020-11-20T11:32:41.325 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.319+0000 7fb0a6469700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fb0a00a6a70 0x7fb0a013b770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55512/0 (socket says 172.21.15.23:55512) 2020-11-20T11:32:41.325 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.319+0000 7fb0a6469700 1 -- 172.21.15.23:0/273305371 learned_addr learned my addr 172.21.15.23:0/273305371 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:41.325 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.319+0000 7fb0a5467700 1 --2- 172.21.15.23:0/273305371 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0a00ad9e0 0x7fb0a013af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:41.326 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.319+0000 7fb0a4c66700 1 -- 172.21.15.23:0/273305371 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 921912266 0 0) 0x7fb0a013be80 con 0x7fb0a00a6a70 2020-11-20T11:32:41.326 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.320+0000 7fb0a4c66700 1 -- 172.21.15.23:0/273305371 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb088003280 con 0x7fb0a00a6a70 2020-11-20T11:32:41.326 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.320+0000 7fb0a4c66700 1 -- 172.21.15.23:0/273305371 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157185205 0 0) 0x7fb0a00a6f10 con 0x7fb0a00ad600 2020-11-20T11:32:41.326 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.320+0000 7fb0a4c66700 1 -- 172.21.15.23:0/273305371 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0a013be80 con 0x7fb0a00ad600 2020-11-20T11:32:41.327 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.320+0000 7fb0a4c66700 1 -- 172.21.15.23:0/273305371 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2271501420 0 0) 0x7fb088003280 con 0x7fb0a00a6a70 2020-11-20T11:32:41.327 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.320+0000 7fb0a4c66700 1 -- 172.21.15.23:0/273305371 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0a00ad9e0 msgr2=0x7fb0a013af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:41.327 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.320+0000 7fb0a4c66700 1 --2- 172.21.15.23:0/273305371 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0a00ad9e0 0x7fb0a013af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:41.327 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.320+0000 7fb0a4c66700 1 -- 172.21.15.23:0/273305371 >> v1:172.21.15.155:6789/0 conn(0x7fb0a00ad600 legacy=0x7fb0a00a90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:41.327 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.320+0000 7fb0a4c66700 1 -- 172.21.15.23:0/273305371 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a00a4da0 con 0x7fb0a00a6a70 2020-11-20T11:32:41.327 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.320+0000 7fb0a4c66700 1 -- 172.21.15.23:0/273305371 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7fb0980018a0 con 0x7fb0a00a6a70 2020-11-20T11:32:41.328 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.320+0000 7fb0a4c66700 1 -- 172.21.15.23:0/273305371 >> v1:172.21.15.23:6789/0 conn(0x7fb0a00a6a70 legacy=0x7fb0a013b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:41.328 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.320+0000 7fb0a4c66700 1 --2- 172.21.15.23:0/273305371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb088003930 0x7fb088003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:41.328 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.320+0000 7fb0a4c66700 1 --2- 172.21.15.23:0/273305371 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0a00ad9e0 0x7fb088004520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:41.328 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.320+0000 7fb0a4c66700 1 --2- 172.21.15.23:0/273305371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb088004a10 0x7fb088004e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:41.329 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.320+0000 7fb0a4c66700 1 -- 172.21.15.23:0/273305371 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a00a71b0 con 0x7fb088003930 2020-11-20T11:32:41.329 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.320+0000 7fb0a4c66700 1 -- 172.21.15.23:0/273305371 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb088005380 con 0x7fb088004a10 2020-11-20T11:32:41.329 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.320+0000 7fb0a4c66700 1 -- 172.21.15.23:0/273305371 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb08800b7a0 con 0x7fb0a00ad9e0 2020-11-20T11:32:41.329 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.320+0000 7fb0a5c68700 1 --2- 172.21.15.23:0/273305371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb088003930 0x7fb088003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:41.329 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.320+0000 7fb0a5467700 1 --2- 172.21.15.23:0/273305371 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0a00ad9e0 0x7fb088004520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:41.330 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.320+0000 7fb0a6469700 1 --2- 172.21.15.23:0/273305371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb088004a10 0x7fb088004e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:41.330 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.320+0000 7fb0a5c68700 1 -- 172.21.15.23:0/273305371 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0a00ad9e0 msgr2=0x7fb088004520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:41.330 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.320+0000 7fb0a5c68700 1 --2- 172.21.15.23:0/273305371 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0a00ad9e0 0x7fb088004520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:41.330 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.320+0000 7fb0a5c68700 1 -- 172.21.15.23:0/273305371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb088004a10 msgr2=0x7fb088004e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:41.330 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.320+0000 7fb0a5c68700 1 --2- 172.21.15.23:0/273305371 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb088004a10 0x7fb088004e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:41.330 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.320+0000 7fb0a5c68700 1 -- 172.21.15.23:0/273305371 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb088003280 con 0x7fb088003930 2020-11-20T11:32:41.331 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.320+0000 7fb0a5c68700 1 --2- 172.21.15.23:0/273305371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb088003930 0x7fb088003d50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fb090007e90 tx=0x7fb090007bd0).ready entity=mon.0 client_cookie=ac8f451bb1c9caa4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:41.331 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.321+0000 7fb0a4c66700 1 -- 172.21.15.23:0/273305371 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb090007020 con 0x7fb088003930 2020-11-20T11:32:41.331 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.321+0000 7fb0a4c66700 1 -- 172.21.15.23:0/273305371 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb090007180 con 0x7fb088003930 2020-11-20T11:32:41.331 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.321+0000 7fb0a4c66700 1 -- 172.21.15.23:0/273305371 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09000c860 con 0x7fb088003930 2020-11-20T11:32:41.331 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.321+0000 7fb0a7ecc700 1 -- 172.21.15.23:0/273305371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb088003930 msgr2=0x7fb088003d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:41.332 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.321+0000 7fb0a7ecc700 1 --2- 172.21.15.23:0/273305371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb088003930 0x7fb088003d50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fb090007e90 tx=0x7fb090007bd0).stop 2020-11-20T11:32:41.332 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.321+0000 7fb0a6469700 1 -- 172.21.15.23:0/273305371 reap_dead start 2020-11-20T11:32:41.332 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.321+0000 7fb0a7ecc700 1 -- 172.21.15.23:0/273305371 shutdown_connections 2020-11-20T11:32:41.332 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.321+0000 7fb0a7ecc700 1 -- 172.21.15.23:0/273305371 >> 172.21.15.23:0/273305371 conn(0x7fb0a0060c80 msgr2=0x7fb0a00acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:41.332 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.321+0000 7fb0a7ecc700 1 -- 172.21.15.23:0/273305371 shutdown_connections 2020-11-20T11:32:41.333 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.321+0000 7fb0a7ecc700 1 -- 172.21.15.23:0/273305371 wait complete. 2020-11-20T11:32:41.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.322+0000 7fb0a7ecc700 1 Processor -- start 2020-11-20T11:32:41.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.322+0000 7fb0a7ecc700 1 -- start start 2020-11-20T11:32:41.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.323+0000 7fb0a7ecc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0a0114fa0 0x7fb0a0065aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:41.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.323+0000 7fb0a7ecc700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0a014a0c0 0x7fb0a014a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:41.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.323+0000 7fb0a7ecc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0a01470e0 0x7fb0a0147540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:41.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.323+0000 7fb0a7ecc700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a0133be0 con 0x7fb0a01470e0 2020-11-20T11:32:41.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.323+0000 7fb0a7ecc700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a0066f70 con 0x7fb0a0114fa0 2020-11-20T11:32:41.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.323+0000 7fb0a7ecc700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a0066590 con 0x7fb0a014a0c0 2020-11-20T11:32:41.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.323+0000 7fb0a6469700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0a01470e0 0x7fb0a0147540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:41.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.323+0000 7fb0a6469700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0a01470e0 0x7fb0a0147540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41274/0 (socket says 172.21.15.23:41274) 2020-11-20T11:32:41.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.323+0000 7fb0a6469700 1 -- 172.21.15.23:0/2483527839 learned_addr learned my addr 172.21.15.23:0/2483527839 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:41.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.323+0000 7fb0a5467700 1 --2- 172.21.15.23:0/2483527839 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0a014a0c0 0x7fb0a014a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:41.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.323+0000 7fb0a5c68700 1 --2- 172.21.15.23:0/2483527839 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0a0114fa0 0x7fb0a0065aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:41.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.323+0000 7fb0a6469700 1 -- 172.21.15.23:0/2483527839 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0a014a0c0 msgr2=0x7fb0a014a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:41.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.323+0000 7fb0a6469700 1 --2- 172.21.15.23:0/2483527839 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0a014a0c0 0x7fb0a014a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:41.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.323+0000 7fb0a6469700 1 -- 172.21.15.23:0/2483527839 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0a0114fa0 msgr2=0x7fb0a0065aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:41.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.323+0000 7fb0a6469700 1 --2- 172.21.15.23:0/2483527839 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0a0114fa0 0x7fb0a0065aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:41.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.323+0000 7fb0a6469700 1 -- 172.21.15.23:0/2483527839 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a0066830 con 0x7fb0a01470e0 2020-11-20T11:32:41.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.323+0000 7fb0a5467700 1 --2- 172.21.15.23:0/2483527839 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0a014a0c0 0x7fb0a014a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:32:41.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.323+0000 7fb0a6469700 1 --2- 172.21.15.23:0/2483527839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0a01470e0 0x7fb0a0147540 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fb0a005c830 tx=0x7fb0a01379c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:41.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.323+0000 7fb08dffb700 1 -- 172.21.15.23:0/2483527839 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0980018a0 con 0x7fb0a01470e0 2020-11-20T11:32:41.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.323+0000 7fb08dffb700 1 -- 172.21.15.23:0/2483527839 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb09800f9c0 con 0x7fb0a01470e0 2020-11-20T11:32:41.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.323+0000 7fb0a7ecc700 1 -- 172.21.15.23:0/2483527839 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0a0066ad0 con 0x7fb0a01470e0 2020-11-20T11:32:41.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.323+0000 7fb08dffb700 1 -- 172.21.15.23:0/2483527839 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09800fd60 con 0x7fb0a01470e0 2020-11-20T11:32:41.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.324+0000 7fb0a7ecc700 1 -- 172.21.15.23:0/2483527839 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0a0065f90 con 0x7fb0a01470e0 2020-11-20T11:32:41.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.325+0000 7fb0a7ecc700 1 -- 172.21.15.23:0/2483527839 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0a0066c60 con 0x7fb0a01470e0 2020-11-20T11:32:41.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.325+0000 7fb08dffb700 1 -- 172.21.15.23:0/2483527839 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb098012560 con 0x7fb0a01470e0 2020-11-20T11:32:41.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.325+0000 7fb08dffb700 1 --2- 172.21.15.23:0/2483527839 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb078056d90 0x7fb078059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:41.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.325+0000 7fb08dffb700 1 -- 172.21.15.23:0/2483527839 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fb09806f890 con 0x7fb0a01470e0 2020-11-20T11:32:41.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.325+0000 7fb0a5c68700 1 --2- 172.21.15.23:0/2483527839 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb078056d90 0x7fb078059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:41.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.326+0000 7fb0a5c68700 1 --2- 172.21.15.23:0/2483527839 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb078056d90 0x7fb078059250 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fb0a005eac0 tx=0x7fb090018000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:41.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.329+0000 7fb08dffb700 1 -- 172.21.15.23:0/2483527839 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb098044b80 con 0x7fb0a01470e0 2020-11-20T11:32:41.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.481+0000 7fb0a7ecc700 1 -- 172.21.15.23:0/2483527839 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fb0a005c520 con 0x7fb0a01470e0 2020-11-20T11:32:41.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.481+0000 7fb08dffb700 1 -- 172.21.15.23:0/2483527839 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v25) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb098019030 con 0x7fb0a01470e0 2020-11-20T11:32:41.484 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 8 2020-11-20T11:32:41.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.483+0000 7fb0a7ecc700 1 -- 172.21.15.23:0/2483527839 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb078056d90 msgr2=0x7fb078059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:41.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.483+0000 7fb0a7ecc700 1 --2- 172.21.15.23:0/2483527839 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb078056d90 0x7fb078059250 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fb0a005eac0 tx=0x7fb090018000).stop 2020-11-20T11:32:41.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.483+0000 7fb0a7ecc700 1 -- 172.21.15.23:0/2483527839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0a01470e0 msgr2=0x7fb0a0147540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:41.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.483+0000 7fb0a7ecc700 1 --2- 172.21.15.23:0/2483527839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0a01470e0 0x7fb0a0147540 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fb0a005c830 tx=0x7fb0a01379c0).stop 2020-11-20T11:32:41.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.483+0000 7fb0a7ecc700 1 -- 172.21.15.23:0/2483527839 shutdown_connections 2020-11-20T11:32:41.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.483+0000 7fb0a7ecc700 1 --2- 172.21.15.23:0/2483527839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0a01470e0 0x7fb0a0147540 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:41.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.483+0000 7fb0a7ecc700 1 --2- 172.21.15.23:0/2483527839 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb0a0114fa0 0x7fb0a0065aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:41.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.483+0000 7fb0a7ecc700 1 --2- 172.21.15.23:0/2483527839 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb078056d90 0x7fb078059250 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:41.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.483+0000 7fb0a7ecc700 1 --2- 172.21.15.23:0/2483527839 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0a014a0c0 0x7fb0a014a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:41.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.483+0000 7fb0a7ecc700 1 -- 172.21.15.23:0/2483527839 >> 172.21.15.23:0/2483527839 conn(0x7fb0a0060c80 msgr2=0x7fb0a00aa140 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:41.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.483+0000 7fb0a7ecc700 1 -- 172.21.15.23:0/2483527839 shutdown_connections 2020-11-20T11:32:41.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.483+0000 7fb0a7ecc700 1 -- 172.21.15.23:0/2483527839 wait complete. 2020-11-20T11:32:41.497 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T11:32:41.616 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.613+0000 7fb3aa4e1700 1 Processor -- start 2020-11-20T11:32:41.616 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.613+0000 7fb3aa4e1700 1 -- start start 2020-11-20T11:32:41.618 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.613+0000 7fb3aa4e1700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb3a40ae760 0x7fb3a413e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:41.618 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.613+0000 7fb3aa4e1700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a40661b0 con 0x7fb3a40aeb40 2020-11-20T11:32:41.618 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.613+0000 7fb3aa4e1700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a413ad60 con 0x7fb3a4065100 2020-11-20T11:32:41.619 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.613+0000 7fb3aa4e1700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3a4066c70 con 0x7fb3a40ae760 2020-11-20T11:32:41.619 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.614+0000 7fb3a3fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb3a40ae760 0x7fb3a413e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:41.619 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.614+0000 7fb3a37fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fb3a40aeb40 0x7fb3a4137ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55532/0 (socket says 172.21.15.23:55532) 2020-11-20T11:32:41.619 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.614+0000 7fb3a37fe700 1 -- 172.21.15.23:0/1010126445 learned_addr learned my addr 172.21.15.23:0/1010126445 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:41.619 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.614+0000 7fb3a3fff700 1 -- 172.21.15.23:0/1010126445 >> v1:172.21.15.155:6789/0 conn(0x7fb3a4065100 legacy=0x7fb3a4145610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:41.620 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.614+0000 7fb3a3fff700 1 -- 172.21.15.23:0/1010126445 >> v1:172.21.15.23:6789/0 conn(0x7fb3a40aeb40 legacy=0x7fb3a4137ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:41.620 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.614+0000 7fb3a3fff700 1 -- 172.21.15.23:0/1010126445 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a413e470 con 0x7fb3a40ae760 2020-11-20T11:32:41.620 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.614+0000 7fb3a2ffd700 1 -- 172.21.15.23:0/1010126445 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454286780 0 0) 0x7fb3a40661b0 con 0x7fb3a40aeb40 2020-11-20T11:32:41.620 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.614+0000 7fb3a3fff700 1 --2- 172.21.15.23:0/1010126445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3a40ae760 0x7fb3a413e910 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fb398006fd0 tx=0x7fb398006e30).ready entity=mon.0 client_cookie=9c0cab56975977c3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:41.620 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.614+0000 7fb3a2ffd700 1 -- 172.21.15.23:0/1010126445 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3980097a0 con 0x7fb3a40ae760 2020-11-20T11:32:41.621 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.614+0000 7fb3a2ffd700 1 -- 172.21.15.23:0/1010126445 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb398000e90 con 0x7fb3a40ae760 2020-11-20T11:32:41.621 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.614+0000 7fb3a2ffd700 1 -- 172.21.15.23:0/1010126445 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb398009b10 con 0x7fb3a40ae760 2020-11-20T11:32:41.621 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.615+0000 7fb3aa4e1700 1 -- 172.21.15.23:0/1010126445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3a40ae760 msgr2=0x7fb3a413e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:41.621 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.615+0000 7fb3aa4e1700 1 --2- 172.21.15.23:0/1010126445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3a40ae760 0x7fb3a413e910 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fb398006fd0 tx=0x7fb398006e30).stop 2020-11-20T11:32:41.621 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.615+0000 7fb3aa4e1700 1 -- 172.21.15.23:0/1010126445 shutdown_connections 2020-11-20T11:32:41.622 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.615+0000 7fb3aa4e1700 1 --2- 172.21.15.23:0/1010126445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3a40ae760 0x7fb3a413e910 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:41.622 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.615+0000 7fb3aa4e1700 1 -- 172.21.15.23:0/1010126445 >> 172.21.15.23:0/1010126445 conn(0x7fb3a4061d80 msgr2=0x7fb3a40ae050 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:41.622 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.615+0000 7fb3aa4e1700 1 -- 172.21.15.23:0/1010126445 shutdown_connections 2020-11-20T11:32:41.622 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.615+0000 7fb3aa4e1700 1 -- 172.21.15.23:0/1010126445 wait complete. 2020-11-20T11:32:41.625 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.616+0000 7fb3aa4e1700 1 Processor -- start 2020-11-20T11:32:41.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.616+0000 7fb3aa4e1700 1 -- start start 2020-11-20T11:32:41.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.617+0000 7fb3aa4e1700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3a4065100 0x7fb3a413dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:41.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.617+0000 7fb3aa4e1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3a40ae760 0x7fb3a4140670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:41.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.617+0000 7fb3aa4e1700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3a40aeb40 0x7fb3a40ac420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:41.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.617+0000 7fb3aa4e1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a40ada30 con 0x7fb3a40ae760 2020-11-20T11:32:41.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.617+0000 7fb3aa4e1700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a40a5e10 con 0x7fb3a4065100 2020-11-20T11:32:41.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.617+0000 7fb3aa4e1700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a40a60b0 con 0x7fb3a40aeb40 2020-11-20T11:32:41.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.617+0000 7fb3a37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3a40ae760 0x7fb3a4140670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:41.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.617+0000 7fb3a37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3a40ae760 0x7fb3a4140670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41286/0 (socket says 172.21.15.23:41286) 2020-11-20T11:32:41.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.617+0000 7fb3a37fe700 1 -- 172.21.15.23:0/1380211920 learned_addr learned my addr 172.21.15.23:0/1380211920 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:41.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.617+0000 7fb3a3fff700 1 --2- 172.21.15.23:0/1380211920 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3a4065100 0x7fb3a413dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:41.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.617+0000 7fb3a8a7e700 1 --2- 172.21.15.23:0/1380211920 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3a40aeb40 0x7fb3a40ac420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:41.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.617+0000 7fb3a37fe700 1 -- 172.21.15.23:0/1380211920 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3a40aeb40 msgr2=0x7fb3a40ac420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:41.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.617+0000 7fb3a37fe700 1 --2- 172.21.15.23:0/1380211920 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3a40aeb40 0x7fb3a40ac420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:41.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.617+0000 7fb3a37fe700 1 -- 172.21.15.23:0/1380211920 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3a4065100 msgr2=0x7fb3a413dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:41.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.617+0000 7fb3a37fe700 1 --2- 172.21.15.23:0/1380211920 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3a4065100 0x7fb3a413dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:41.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.617+0000 7fb3a37fe700 1 -- 172.21.15.23:0/1380211920 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a413d8a0 con 0x7fb3a40ae760 2020-11-20T11:32:41.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.617+0000 7fb3a3fff700 1 --2- 172.21.15.23:0/1380211920 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3a4065100 0x7fb3a413dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:32:41.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.617+0000 7fb3a8a7e700 1 --2- 172.21.15.23:0/1380211920 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3a40aeb40 0x7fb3a40ac420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:32:41.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.617+0000 7fb3a37fe700 1 --2- 172.21.15.23:0/1380211920 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3a40ae760 0x7fb3a4140670 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fb39400c760 tx=0x7fb394007ca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:41.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.617+0000 7fb383fff700 1 -- 172.21.15.23:0/1380211920 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3940072e0 con 0x7fb3a40ae760 2020-11-20T11:32:41.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.617+0000 7fb383fff700 1 -- 172.21.15.23:0/1380211920 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb394007440 con 0x7fb3a40ae760 2020-11-20T11:32:41.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.617+0000 7fb3aa4e1700 1 -- 172.21.15.23:0/1380211920 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3a4140db0 con 0x7fb3a40ae760 2020-11-20T11:32:41.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.617+0000 7fb383fff700 1 -- 172.21.15.23:0/1380211920 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb394005860 con 0x7fb3a40ae760 2020-11-20T11:32:41.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.617+0000 7fb3aa4e1700 1 -- 172.21.15.23:0/1380211920 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a40a5890 con 0x7fb3a40ae760 2020-11-20T11:32:41.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.618+0000 7fb3aa4e1700 1 -- 172.21.15.23:0/1380211920 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3a4062760 con 0x7fb3a40ae760 2020-11-20T11:32:41.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.618+0000 7fb383fff700 1 -- 172.21.15.23:0/1380211920 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb39401a060 con 0x7fb3a40ae760 2020-11-20T11:32:41.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.619+0000 7fb383fff700 1 --2- 172.21.15.23:0/1380211920 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb37c056d60 0x7fb37c059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:41.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.619+0000 7fb383fff700 1 -- 172.21.15.23:0/1380211920 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fb39406f4c0 con 0x7fb3a40ae760 2020-11-20T11:32:41.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.619+0000 7fb3a3fff700 1 --2- 172.21.15.23:0/1380211920 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb37c056d60 0x7fb37c059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:41.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.619+0000 7fb3a3fff700 1 --2- 172.21.15.23:0/1380211920 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb37c056d60 0x7fb37c059220 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fb398006e70 tx=0x7fb398015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:41.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.622+0000 7fb383fff700 1 -- 172.21.15.23:0/1380211920 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb3940447b0 con 0x7fb3a40ae760 2020-11-20T11:32:41.774 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.771+0000 7fb3aa4e1700 1 -- 172.21.15.23:0/1380211920 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb3a413da30 con 0x7fb3a40ae760 2020-11-20T11:32:41.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.772+0000 7fb383fff700 1 -- 172.21.15.23:0/1380211920 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+10 (secure 0 0 0) 0x7fb394047dd0 con 0x7fb3a40ae760 2020-11-20T11:32:41.775 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 8 2020-11-20T11:32:41.776 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.773+0000 7fb3aa4e1700 1 -- 172.21.15.23:0/1380211920 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb37c056d60 msgr2=0x7fb37c059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:41.777 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.773+0000 7fb3aa4e1700 1 --2- 172.21.15.23:0/1380211920 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb37c056d60 0x7fb37c059220 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fb398006e70 tx=0x7fb398015000).stop 2020-11-20T11:32:41.777 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.773+0000 7fb3aa4e1700 1 -- 172.21.15.23:0/1380211920 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3a40ae760 msgr2=0x7fb3a4140670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:41.777 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.773+0000 7fb3aa4e1700 1 --2- 172.21.15.23:0/1380211920 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3a40ae760 0x7fb3a4140670 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fb39400c760 tx=0x7fb394007ca0).stop 2020-11-20T11:32:41.777 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.773+0000 7fb3aa4e1700 1 -- 172.21.15.23:0/1380211920 shutdown_connections 2020-11-20T11:32:41.777 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.773+0000 7fb3aa4e1700 1 --2- 172.21.15.23:0/1380211920 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb37c056d60 0x7fb37c059220 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:41.778 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.773+0000 7fb3aa4e1700 1 --2- 172.21.15.23:0/1380211920 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3a40aeb40 0x7fb3a40ac420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:41.778 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.773+0000 7fb3aa4e1700 1 --2- 172.21.15.23:0/1380211920 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3a4065100 0x7fb3a413dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:41.778 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.773+0000 7fb3aa4e1700 1 --2- 172.21.15.23:0/1380211920 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3a40ae760 0x7fb3a4140670 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:41.778 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.773+0000 7fb3aa4e1700 1 -- 172.21.15.23:0/1380211920 >> 172.21.15.23:0/1380211920 conn(0x7fb3a4061d80 msgr2=0x7fb3a40aa9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:41.779 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.773+0000 7fb3aa4e1700 1 -- 172.21.15.23:0/1380211920 shutdown_connections 2020-11-20T11:32:41.779 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.773+0000 7fb3aa4e1700 1 -- 172.21.15.23:0/1380211920 wait complete. 2020-11-20T11:32:41.787 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T11:32:41.904 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.901+0000 7fa492634700 1 Processor -- start 2020-11-20T11:32:41.904 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.901+0000 7fa492634700 1 -- start start 2020-11-20T11:32:41.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.902+0000 7fa492634700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa48c0aae90 0x7fa48c1381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:41.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.902+0000 7fa492634700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa48c13e600 con 0x7fa48c13aeb0 2020-11-20T11:32:41.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.902+0000 7fa492634700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa48c13e8a0 con 0x7fa48c0aaab0 2020-11-20T11:32:41.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.902+0000 7fa492634700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa48c0ab6d0 con 0x7fa48c0aae90 2020-11-20T11:32:41.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.902+0000 7fa490bd1700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fa48c13aeb0 0x7fa48c0a8c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55544/0 (socket says 172.21.15.23:55544) 2020-11-20T11:32:41.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.902+0000 7fa490bd1700 1 -- 172.21.15.23:0/2680951770 learned_addr learned my addr 172.21.15.23:0/2680951770 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:41.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.902+0000 7fa48b7fe700 1 --2- 172.21.15.23:0/2680951770 >> v2:172.21.15.155:3300/0 conn(0x7fa48c0aae90 0x7fa48c1381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:41.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.902+0000 7fa48affd700 1 -- 172.21.15.23:0/2680951770 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 281032018 0 0) 0x7fa48c13e600 con 0x7fa48c13aeb0 2020-11-20T11:32:41.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.902+0000 7fa48affd700 1 -- 172.21.15.23:0/2680951770 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa478003280 con 0x7fa48c13aeb0 2020-11-20T11:32:41.909 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.902+0000 7fa48affd700 1 -- 172.21.15.23:0/2680951770 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 896095708 0 0) 0x7fa48c13e8a0 con 0x7fa48c0aaab0 2020-11-20T11:32:41.909 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.903+0000 7fa48affd700 1 -- 172.21.15.23:0/2680951770 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa48c13e600 con 0x7fa48c0aaab0 2020-11-20T11:32:41.909 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.903+0000 7fa48b7fe700 1 -- 172.21.15.23:0/2680951770 >> v1:172.21.15.155:6789/0 conn(0x7fa48c0aaab0 legacy=0x7fa48c13ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:41.909 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.903+0000 7fa48affd700 1 -- 172.21.15.23:0/2680951770 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1500022246 0 0) 0x7fa478003280 con 0x7fa48c13aeb0 2020-11-20T11:32:41.910 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.903+0000 7fa48b7fe700 1 -- 172.21.15.23:0/2680951770 >> v1:172.21.15.23:6789/0 conn(0x7fa48c13aeb0 legacy=0x7fa48c0a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:41.910 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.903+0000 7fa48b7fe700 1 -- 172.21.15.23:0/2680951770 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa48c0a9370 con 0x7fa48c0aae90 2020-11-20T11:32:41.910 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.903+0000 7fa48b7fe700 1 --2- 172.21.15.23:0/2680951770 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa48c0aae90 0x7fa48c1381a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fa480006fd0 tx=0x7fa480006e30).ready entity=mon.1 client_cookie=d3a2bd9b7414cb06 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:41.910 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.903+0000 7fa48affd700 1 -- 172.21.15.23:0/2680951770 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4840018a0 con 0x7fa48c0aae90 2020-11-20T11:32:41.910 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.903+0000 7fa48affd700 1 -- 172.21.15.23:0/2680951770 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa480000e90 con 0x7fa48c0aae90 2020-11-20T11:32:41.911 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.903+0000 7fa48affd700 1 -- 172.21.15.23:0/2680951770 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa480011610 con 0x7fa48c0aae90 2020-11-20T11:32:41.911 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.903+0000 7fa492634700 1 -- 172.21.15.23:0/2680951770 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa48c0aae90 msgr2=0x7fa48c1381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:41.911 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.903+0000 7fa492634700 1 --2- 172.21.15.23:0/2680951770 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa48c0aae90 0x7fa48c1381a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fa480006fd0 tx=0x7fa480006e30).stop 2020-11-20T11:32:41.911 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.903+0000 7fa492634700 1 -- 172.21.15.23:0/2680951770 shutdown_connections 2020-11-20T11:32:41.912 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.903+0000 7fa492634700 1 --2- 172.21.15.23:0/2680951770 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa48c0aae90 0x7fa48c1381a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:41.912 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.903+0000 7fa492634700 1 -- 172.21.15.23:0/2680951770 >> 172.21.15.23:0/2680951770 conn(0x7fa48c0678e0 msgr2=0x7fa48c067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:41.912 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.903+0000 7fa492634700 1 -- 172.21.15.23:0/2680951770 shutdown_connections 2020-11-20T11:32:41.912 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.903+0000 7fa492634700 1 -- 172.21.15.23:0/2680951770 wait complete. 2020-11-20T11:32:41.912 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.904+0000 7fa492634700 1 Processor -- start 2020-11-20T11:32:41.915 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.905+0000 7fa492634700 1 -- start start 2020-11-20T11:32:41.915 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.905+0000 7fa492634700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa48c0aaab0 0x7fa48c14a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:41.916 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.905+0000 7fa492634700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa48c0aae90 0x7fa48c1359d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:41.916 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.905+0000 7fa492634700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa48c14d870 0x7fa48c14dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:41.916 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.905+0000 7fa492634700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa48c131490 con 0x7fa48c0aae90 2020-11-20T11:32:41.916 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.905+0000 7fa492634700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa48c14a780 con 0x7fa48c14d870 2020-11-20T11:32:41.916 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.905+0000 7fa492634700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa48c14aa10 con 0x7fa48c0aaab0 2020-11-20T11:32:41.917 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.905+0000 7fa48b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa48c0aae90 0x7fa48c1359d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:41.917 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.905+0000 7fa48b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa48c0aae90 0x7fa48c1359d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41300/0 (socket says 172.21.15.23:41300) 2020-11-20T11:32:41.917 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.905+0000 7fa48b7fe700 1 -- 172.21.15.23:0/3025363445 learned_addr learned my addr 172.21.15.23:0/3025363445 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:41.917 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.905+0000 7fa490bd1700 1 --2- 172.21.15.23:0/3025363445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa48c14d870 0x7fa48c14dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:41.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.906+0000 7fa48bfff700 1 --2- 172.21.15.23:0/3025363445 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa48c0aaab0 0x7fa48c14a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:41.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.906+0000 7fa48b7fe700 1 -- 172.21.15.23:0/3025363445 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa48c0aaab0 msgr2=0x7fa48c14a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:41.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.906+0000 7fa48b7fe700 1 --2- 172.21.15.23:0/3025363445 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa48c0aaab0 0x7fa48c14a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:41.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.906+0000 7fa48b7fe700 1 -- 172.21.15.23:0/3025363445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa48c14d870 msgr2=0x7fa48c14dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:41.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.906+0000 7fa48b7fe700 1 --2- 172.21.15.23:0/3025363445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa48c14d870 0x7fa48c14dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:41.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.906+0000 7fa48b7fe700 1 -- 172.21.15.23:0/3025363445 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa48c14acb0 con 0x7fa48c0aae90 2020-11-20T11:32:41.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.906+0000 7fa490bd1700 1 --2- 172.21.15.23:0/3025363445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa48c14d870 0x7fa48c14dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:32:41.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.906+0000 7fa48b7fe700 1 --2- 172.21.15.23:0/3025363445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa48c0aae90 0x7fa48c1359d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fa480006e70 tx=0x7fa480015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:41.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.906+0000 7fa46bfff700 1 -- 172.21.15.23:0/3025363445 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa480010b80 con 0x7fa48c0aae90 2020-11-20T11:32:41.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.906+0000 7fa46bfff700 1 -- 172.21.15.23:0/3025363445 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa480010ce0 con 0x7fa48c0aae90 2020-11-20T11:32:41.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.906+0000 7fa492634700 1 -- 172.21.15.23:0/3025363445 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa48c14afb0 con 0x7fa48c0aae90 2020-11-20T11:32:41.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.906+0000 7fa46bfff700 1 -- 172.21.15.23:0/3025363445 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48001e690 con 0x7fa48c0aae90 2020-11-20T11:32:41.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.906+0000 7fa492634700 1 -- 172.21.15.23:0/3025363445 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa48c135ec0 con 0x7fa48c0aae90 2020-11-20T11:32:41.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.907+0000 7fa492634700 1 -- 172.21.15.23:0/3025363445 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa48c1360b0 con 0x7fa48c0aae90 2020-11-20T11:32:41.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.907+0000 7fa46bfff700 1 -- 172.21.15.23:0/3025363445 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa480025030 con 0x7fa48c0aae90 2020-11-20T11:32:41.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.907+0000 7fa46bfff700 1 --2- 172.21.15.23:0/3025363445 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa464056d90 0x7fa464059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:41.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.907+0000 7fa46bfff700 1 -- 172.21.15.23:0/3025363445 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fa4800710f0 con 0x7fa48c0aae90 2020-11-20T11:32:41.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.908+0000 7fa48bfff700 1 --2- 172.21.15.23:0/3025363445 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa464056d90 0x7fa464059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:41.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.908+0000 7fa48bfff700 1 --2- 172.21.15.23:0/3025363445 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa464056d90 0x7fa464059250 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fa47c000c00 tx=0x7fa47c009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:41.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:41.911+0000 7fa46bfff700 1 -- 172.21.15.23:0/3025363445 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa480046490 con 0x7fa48c0aae90 2020-11-20T11:32:42.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.059+0000 7fa492634700 1 -- 172.21.15.23:0/3025363445 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa48c05c7d0 con 0x7fa48c0aae90 2020-11-20T11:32:42.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.059+0000 7fa46bfff700 1 -- 172.21.15.23:0/3025363445 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+10 (secure 0 0 0) 0x7fa480023070 con 0x7fa48c0aae90 2020-11-20T11:32:42.062 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 8 2020-11-20T11:32:42.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.060+0000 7fa492634700 1 -- 172.21.15.23:0/3025363445 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa464056d90 msgr2=0x7fa464059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:42.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.060+0000 7fa492634700 1 --2- 172.21.15.23:0/3025363445 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa464056d90 0x7fa464059250 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fa47c000c00 tx=0x7fa47c009040).stop 2020-11-20T11:32:42.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.060+0000 7fa492634700 1 -- 172.21.15.23:0/3025363445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa48c0aae90 msgr2=0x7fa48c1359d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:42.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.060+0000 7fa492634700 1 --2- 172.21.15.23:0/3025363445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa48c0aae90 0x7fa48c1359d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fa480006e70 tx=0x7fa480015000).stop 2020-11-20T11:32:42.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.060+0000 7fa492634700 1 -- 172.21.15.23:0/3025363445 shutdown_connections 2020-11-20T11:32:42.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.060+0000 7fa492634700 1 --2- 172.21.15.23:0/3025363445 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa48c14d870 0x7fa48c14dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:42.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.060+0000 7fa492634700 1 --2- 172.21.15.23:0/3025363445 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa464056d90 0x7fa464059250 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:42.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.060+0000 7fa492634700 1 --2- 172.21.15.23:0/3025363445 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa48c0aaab0 0x7fa48c14a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:42.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.060+0000 7fa492634700 1 --2- 172.21.15.23:0/3025363445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa48c0aae90 0x7fa48c1359d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:42.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.060+0000 7fa492634700 1 -- 172.21.15.23:0/3025363445 >> 172.21.15.23:0/3025363445 conn(0x7fa48c0678e0 msgr2=0x7fa48c13a250 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:42.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.060+0000 7fa492634700 1 -- 172.21.15.23:0/3025363445 shutdown_connections 2020-11-20T11:32:42.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.061+0000 7fa492634700 1 -- 172.21.15.23:0/3025363445 wait complete. 2020-11-20T11:32:42.074 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2020-11-20T11:32:42.075 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T11:32:42.191 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.186+0000 7f6238ee0700 1 Processor -- start 2020-11-20T11:32:42.191 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.186+0000 7f6238ee0700 1 -- start start 2020-11-20T11:32:42.191 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.187+0000 7f6238ee0700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f623405bc80 0x7f623413bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:42.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.187+0000 7f6238ee0700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6234067fd0 0x7f62340a9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:42.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.187+0000 7f6238ee0700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62340a8c20 con 0x7f62340a55c0 2020-11-20T11:32:42.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.187+0000 7f6238ee0700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62340a9e20 con 0x7f623405bc80 2020-11-20T11:32:42.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.187+0000 7f6238ee0700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62340684e0 con 0x7f6234067fd0 2020-11-20T11:32:42.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.187+0000 7f623259c700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f623405bc80 0x7f623413bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:42.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.187+0000 7f6232d9d700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f62340a55c0 0x7f62340a6ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:33532/0 (socket says 172.21.15.23:33532) 2020-11-20T11:32:42.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.187+0000 7f6232d9d700 1 -- 172.21.15.23:0/3269524892 learned_addr learned my addr 172.21.15.23:0/3269524892 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:42.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.187+0000 7f6231d9b700 1 --2- 172.21.15.23:0/3269524892 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6234067fd0 0x7f62340a9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:42.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.187+0000 7f623259c700 1 -- 172.21.15.23:0/3269524892 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6234067fd0 msgr2=0x7f62340a9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:42.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.187+0000 7f623259c700 1 --2- 172.21.15.23:0/3269524892 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6234067fd0 0x7f62340a9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:42.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.187+0000 7f623259c700 1 -- 172.21.15.23:0/3269524892 >> v1:172.21.15.155:6789/0 conn(0x7f62340a55c0 legacy=0x7f62340a6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:42.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.187+0000 7f623259c700 1 -- 172.21.15.23:0/3269524892 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f623405c060 con 0x7f623405bc80 2020-11-20T11:32:42.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.187+0000 7f6231d9b700 1 --2- 172.21.15.23:0/3269524892 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6234067fd0 0x7f62340a9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:32:42.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.187+0000 7f623259c700 1 --2- 172.21.15.23:0/3269524892 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f623405bc80 0x7f623413bfa0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f62340a8db0 tx=0x7f6224006e30).ready entity=mon.1 client_cookie=7cee345b01a77bb2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:42.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.187+0000 7f623159a700 1 -- 172.21.15.23:0/3269524892 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62240097b0 con 0x7f623405bc80 2020-11-20T11:32:42.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.187+0000 7f623159a700 1 -- 172.21.15.23:0/3269524892 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6224000e90 con 0x7f623405bc80 2020-11-20T11:32:42.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.188+0000 7f623159a700 1 -- 172.21.15.23:0/3269524892 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6224009b20 con 0x7f623405bc80 2020-11-20T11:32:42.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.188+0000 7f6238ee0700 1 -- 172.21.15.23:0/3269524892 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f623405bc80 msgr2=0x7f623413bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:42.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.188+0000 7f6238ee0700 1 --2- 172.21.15.23:0/3269524892 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f623405bc80 0x7f623413bfa0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f62340a8db0 tx=0x7f6224006e30).stop 2020-11-20T11:32:42.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.188+0000 7f6238ee0700 1 -- 172.21.15.23:0/3269524892 shutdown_connections 2020-11-20T11:32:42.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.188+0000 7f6238ee0700 1 --2- 172.21.15.23:0/3269524892 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f623405bc80 0x7f623413bfa0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:42.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.188+0000 7f6238ee0700 1 --2- 172.21.15.23:0/3269524892 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6234067fd0 0x7f62340a9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:42.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.188+0000 7f6238ee0700 1 -- 172.21.15.23:0/3269524892 >> 172.21.15.23:0/3269524892 conn(0x7f62340ad7a0 msgr2=0x7f62340adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:42.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.188+0000 7f6238ee0700 1 -- 172.21.15.23:0/3269524892 shutdown_connections 2020-11-20T11:32:42.197 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.188+0000 7f6238ee0700 1 -- 172.21.15.23:0/3269524892 wait complete. 2020-11-20T11:32:42.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.189+0000 7f6238ee0700 1 Processor -- start 2020-11-20T11:32:42.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.189+0000 7f6238ee0700 1 -- start start 2020-11-20T11:32:42.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.190+0000 7f6238ee0700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f623405bc80 0x7f623413b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:42.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.190+0000 7f6238ee0700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6234067fd0 0x7f62340ac9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:42.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.190+0000 7f6238ee0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62340a55c0 0x7f6234067270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:42.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.190+0000 7f6238ee0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6234067760 con 0x7f62340a55c0 2020-11-20T11:32:42.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.190+0000 7f6238ee0700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6234139dd0 con 0x7f6234067fd0 2020-11-20T11:32:42.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.190+0000 7f6238ee0700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f623413a070 con 0x7f623405bc80 2020-11-20T11:32:42.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.190+0000 7f6232d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62340a55c0 0x7f6234067270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:42.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.190+0000 7f623259c700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f623405bc80 0x7f623413b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:42.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.190+0000 7f6232d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62340a55c0 0x7f6234067270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41316/0 (socket says 172.21.15.23:41316) 2020-11-20T11:32:42.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.190+0000 7f6232d9d700 1 -- 172.21.15.23:0/333083222 learned_addr learned my addr 172.21.15.23:0/333083222 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:42.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.190+0000 7f6231d9b700 1 --2- 172.21.15.23:0/333083222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6234067fd0 0x7f62340ac9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:42.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.190+0000 7f6232d9d700 1 -- 172.21.15.23:0/333083222 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f623405bc80 msgr2=0x7f623413b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:42.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.190+0000 7f6232d9d700 1 --2- 172.21.15.23:0/333083222 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f623405bc80 0x7f623413b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:42.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.190+0000 7f6232d9d700 1 -- 172.21.15.23:0/333083222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6234067fd0 msgr2=0x7f62340ac9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:42.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.190+0000 7f6232d9d700 1 --2- 172.21.15.23:0/333083222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6234067fd0 0x7f62340ac9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:42.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.190+0000 7f6232d9d700 1 -- 172.21.15.23:0/333083222 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f623413a310 con 0x7f62340a55c0 2020-11-20T11:32:42.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.190+0000 7f6232d9d700 1 --2- 172.21.15.23:0/333083222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62340a55c0 0x7f6234067270 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f622c00ca10 tx=0x7f622c0013a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:42.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.194+0000 7f62227fc700 1 -- 172.21.15.23:0/333083222 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f622c005bc0 con 0x7f62340a55c0 2020-11-20T11:32:42.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.194+0000 7f62227fc700 1 -- 172.21.15.23:0/333083222 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f622c005d20 con 0x7f62340a55c0 2020-11-20T11:32:42.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.194+0000 7f6238ee0700 1 -- 172.21.15.23:0/333083222 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6234065780 con 0x7f62340a55c0 2020-11-20T11:32:42.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.194+0000 7f62227fc700 1 -- 172.21.15.23:0/333083222 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f622c007320 con 0x7f62340a55c0 2020-11-20T11:32:42.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.194+0000 7f6238ee0700 1 -- 172.21.15.23:0/333083222 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62340a1db0 con 0x7f62340a55c0 2020-11-20T11:32:42.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.195+0000 7f6238ee0700 1 -- 172.21.15.23:0/333083222 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f623413a4a0 con 0x7f62340a55c0 2020-11-20T11:32:42.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.195+0000 7f62227fc700 1 -- 172.21.15.23:0/333083222 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f622c01b060 con 0x7f62340a55c0 2020-11-20T11:32:42.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.195+0000 7f62227fc700 1 --2- 172.21.15.23:0/333083222 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6208056d90 0x7f6208059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:42.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.196+0000 7f62227fc700 1 -- 172.21.15.23:0/333083222 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f622c06fac0 con 0x7f62340a55c0 2020-11-20T11:32:42.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.196+0000 7f623259c700 1 --2- 172.21.15.23:0/333083222 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6208056d90 0x7f6208059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:42.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.196+0000 7f623259c700 1 --2- 172.21.15.23:0/333083222 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6208056d90 0x7f6208059250 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f6224006e70 tx=0x7f6224015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:42.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.199+0000 7f62227fc700 1 -- 172.21.15.23:0/333083222 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f622c044c40 con 0x7f62340a55c0 2020-11-20T11:32:42.397 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.394+0000 7f6238ee0700 1 -- 172.21.15.23:0/333083222 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6234062fe0 con 0x7f62340a55c0 2020-11-20T11:32:42.398 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.395+0000 7f62227fc700 1 -- 172.21.15.23:0/333083222 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+2128 (secure 0 0 0) 0x7f622c019030 con 0x7f62340a55c0 2020-11-20T11:32:42.398 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:32:42.399 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4223,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/564012815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":564012815},{"type":"v1","addr":"172.21.15.155:6835","nonce":564012815}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T11:32:25.469149+0000","modified":"2020-11-20T11:32:28.118344+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.23:6833/919963501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":919963501},{"type":"v1","addr":"172.21.15.23:6833","nonce":919963501}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T11:32:42.400 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.396+0000 7f6238ee0700 1 -- 172.21.15.23:0/333083222 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6208056d90 msgr2=0x7f6208059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:42.400 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.396+0000 7f6238ee0700 1 --2- 172.21.15.23:0/333083222 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6208056d90 0x7f6208059250 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f6224006e70 tx=0x7f6224015000).stop 2020-11-20T11:32:42.400 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.396+0000 7f6238ee0700 1 -- 172.21.15.23:0/333083222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62340a55c0 msgr2=0x7f6234067270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:42.400 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.396+0000 7f6238ee0700 1 --2- 172.21.15.23:0/333083222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62340a55c0 0x7f6234067270 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f622c00ca10 tx=0x7f622c0013a0).stop 2020-11-20T11:32:42.401 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.396+0000 7f6238ee0700 1 -- 172.21.15.23:0/333083222 shutdown_connections 2020-11-20T11:32:42.401 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.396+0000 7f6238ee0700 1 --2- 172.21.15.23:0/333083222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62340a55c0 0x7f6234067270 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:42.401 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.396+0000 7f6238ee0700 1 --2- 172.21.15.23:0/333083222 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6208056d90 0x7f6208059250 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:42.401 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.396+0000 7f6238ee0700 1 --2- 172.21.15.23:0/333083222 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f623405bc80 0x7f623413b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:42.401 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.396+0000 7f6238ee0700 1 --2- 172.21.15.23:0/333083222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6234067fd0 0x7f62340ac9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:42.402 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.396+0000 7f6238ee0700 1 -- 172.21.15.23:0/333083222 >> 172.21.15.23:0/333083222 conn(0x7f62340ad7a0 msgr2=0x7f6234138b40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:42.402 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.397+0000 7f6238ee0700 1 -- 172.21.15.23:0/333083222 shutdown_connections 2020-11-20T11:32:42.402 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:42.397+0000 7f6238ee0700 1 -- 172.21.15.23:0/333083222 wait complete. 2020-11-20T11:32:42.402 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 7 2020-11-20T11:32:42.613 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2020-11-20T11:32:25.469149+0000', 'modified': '2020-11-20T11:32:28.118344+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4257}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4257': {'gid': 4257, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.23:6833/919963501', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6832', 'nonce': 919963501}, {'type': 'v1', 'addr': '172.21.15.23:6833', 'nonce': 919963501}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138297136906239, 'flags': 0}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2020-11-20T11:32:42.613 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-11-20T11:32:42.614 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2020-11-20T11:32:42.750 INFO:teuthology.orchestra.run.smithi023.stdout:{"cluster_fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","whoami":0,"id":4257,"want_state":"up:active","state":"up:active","fs_name":"cephfs","rank_uptime":17.267866729000001,"mdsmap_epoch":7,"osdmap_epoch":25,"osdmap_epoch_barrier":25,"uptime":23.715187568000001} 2020-11-20T11:32:42.750 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", "fs_name": "cephfs", "id": 4257, "mdsmap_epoch": 7, "osdmap_epoch": 25, "osdmap_epoch_barrier": 25, "rank_uptime": 17.267866729, "state": "up:active", "uptime": 23.715187568, "want_state": "up:active", "whoami": 0 } 2020-11-20T11:32:42.751 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2020-11-20T11:32:42.751 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2020-11-20T11:32:42.751 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T11:32:42.752 INFO:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p -v /home/ubuntu/cephtest/mnt.0) 2020-11-20T11:32:42.768 INFO:teuthology.orchestra.run.smithi023.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2020-11-20T11:32:42.768 INFO:teuthology.orchestra.run.smithi023:> sudo modprobe fuse 2020-11-20T11:32:42.862 INFO:teuthology.orchestra.run:Running command with timeout 30 2020-11-20T11:32:42.863 INFO:teuthology.orchestra.run.smithi023:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2020-11-20T11:32:42.891 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T11:32:42.892 INFO:teuthology.orchestra.run.smithi023:> ls /sys/fs/fuse/connections 2020-11-20T11:32:42.946 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2020-11-20T11:32:42.947 INFO:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --id 0 /home/ubuntu/cephtest/mnt.0) 2020-11-20T11:32:42.989 INFO:teuthology.orchestra.run.smithi023:> sudo modprobe fuse 2020-11-20T11:32:43.013 INFO:teuthology.orchestra.run:Running command with timeout 30 2020-11-20T11:32:43.014 INFO:teuthology.orchestra.run.smithi023:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2020-11-20T11:32:43.078 INFO:teuthology.orchestra.run.smithi023.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted on /sys/fs/fuse/connections. 2020-11-20T11:32:43.079 DEBUG:teuthology.orchestra.run:got remote process result: 32 2020-11-20T11:32:43.080 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T11:32:43.080 INFO:teuthology.orchestra.run.smithi023:> ls /sys/fs/fuse/connections 2020-11-20T11:32:43.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi023.stderr:2020-11-20T11:32:43.114+0000 7f9a5d3ba240 -1 init, newargv = 0x562448923f10 newargc=15 2020-11-20T11:32:43.117 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi023.stderr:ceph-fuse[28386]: starting ceph client 2020-11-20T11:32:43.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi023.stderr:ceph-fuse[28386]: starting fuse 2020-11-20T11:32:43.131 INFO:teuthology.orchestra.run.smithi023.stdout:47 2020-11-20T11:32:43.132 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2020-11-20T11:32:43.132 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2020-11-20T11:32:43.133 INFO:teuthology.orchestra.run.smithi023:> import glob 2020-11-20T11:32:43.133 INFO:teuthology.orchestra.run.smithi023:> import re 2020-11-20T11:32:43.133 INFO:teuthology.orchestra.run.smithi023:> import os 2020-11-20T11:32:43.134 INFO:teuthology.orchestra.run.smithi023:> import subprocess 2020-11-20T11:32:43.134 INFO:teuthology.orchestra.run.smithi023:> 2020-11-20T11:32:43.134 INFO:teuthology.orchestra.run.smithi023:> def _find_admin_socket(client_name): 2020-11-20T11:32:43.134 INFO:teuthology.orchestra.run.smithi023:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2020-11-20T11:32:43.135 INFO:teuthology.orchestra.run.smithi023:> files = glob.glob(asok_path) 2020-11-20T11:32:43.135 INFO:teuthology.orchestra.run.smithi023:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2020-11-20T11:32:43.135 INFO:teuthology.orchestra.run.smithi023:> 2020-11-20T11:32:43.136 INFO:teuthology.orchestra.run.smithi023:> # Given a non-glob path, it better be there 2020-11-20T11:32:43.136 INFO:teuthology.orchestra.run.smithi023:> if "*" not in asok_path: 2020-11-20T11:32:43.136 INFO:teuthology.orchestra.run.smithi023:> assert(len(files) == 1) 2020-11-20T11:32:43.137 INFO:teuthology.orchestra.run.smithi023:> return files[0] 2020-11-20T11:32:43.137 INFO:teuthology.orchestra.run.smithi023:> 2020-11-20T11:32:43.138 INFO:teuthology.orchestra.run.smithi023:> for f in files: 2020-11-20T11:32:43.138 INFO:teuthology.orchestra.run.smithi023:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2020-11-20T11:32:43.138 INFO:teuthology.orchestra.run.smithi023:> if os.path.exists("/proc/{0}".format(pid)): 2020-11-20T11:32:43.139 INFO:teuthology.orchestra.run.smithi023:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2020-11-20T11:32:43.139 INFO:teuthology.orchestra.run.smithi023:> contents = proc_f.read() 2020-11-20T11:32:43.139 INFO:teuthology.orchestra.run.smithi023:> if mountpoint in contents: 2020-11-20T11:32:43.140 INFO:teuthology.orchestra.run.smithi023:> return f 2020-11-20T11:32:43.140 INFO:teuthology.orchestra.run.smithi023:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2020-11-20T11:32:43.140 INFO:teuthology.orchestra.run.smithi023:> 2020-11-20T11:32:43.141 INFO:teuthology.orchestra.run.smithi023:> print(_find_admin_socket("client.0")) 2020-11-20T11:32:43.141 INFO:teuthology.orchestra.run.smithi023:> ' 2020-11-20T11:32:43.253 INFO:teuthology.orchestra.run.smithi023.stdout:/var/run/ceph/ceph-client.0.28386.asok 2020-11-20T11:32:43.435 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.28386.asok 2020-11-20T11:32:43.435 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T11:32:43.436 INFO:teuthology.orchestra.run.smithi023:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.28386.asok status 2020-11-20T11:32:43.585 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:32:43.585 INFO:teuthology.orchestra.run.smithi023.stdout: "metadata": { 2020-11-20T11:32:43.585 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_sha1": "b86e691bf191566922a8f08e2a5abf64fde8cfcb", 2020-11-20T11:32:43.585 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_version": "ceph version 16.0.0-7397-gb86e691b (b86e691bf191566922a8f08e2a5abf64fde8cfcb) pacific (dev)", 2020-11-20T11:32:43.586 INFO:teuthology.orchestra.run.smithi023.stdout: "entity_id": "0", 2020-11-20T11:32:43.586 INFO:teuthology.orchestra.run.smithi023.stdout: "hostname": "smithi023", 2020-11-20T11:32:43.586 INFO:teuthology.orchestra.run.smithi023.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2020-11-20T11:32:43.586 INFO:teuthology.orchestra.run.smithi023.stdout: "pid": "28386", 2020-11-20T11:32:43.586 INFO:teuthology.orchestra.run.smithi023.stdout: "root": "/" 2020-11-20T11:32:43.587 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:43.587 INFO:teuthology.orchestra.run.smithi023.stdout: "dentry_count": 0, 2020-11-20T11:32:43.587 INFO:teuthology.orchestra.run.smithi023.stdout: "dentry_pinned_count": 0, 2020-11-20T11:32:43.587 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 4659, 2020-11-20T11:32:43.587 INFO:teuthology.orchestra.run.smithi023.stdout: "inst": { 2020-11-20T11:32:43.588 INFO:teuthology.orchestra.run.smithi023.stdout: "name": { 2020-11-20T11:32:43.588 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "client", 2020-11-20T11:32:43.588 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 4659 2020-11-20T11:32:43.588 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:43.588 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": { 2020-11-20T11:32:43.589 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:32:43.589 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "192.168.0.1:0", 2020-11-20T11:32:43.589 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 21670065 2020-11-20T11:32:43.589 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:32:43.589 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:43.590 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": { 2020-11-20T11:32:43.590 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:32:43.590 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "192.168.0.1:0", 2020-11-20T11:32:43.590 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 21670065 2020-11-20T11:32:43.590 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:43.591 INFO:teuthology.orchestra.run.smithi023.stdout: "inst_str": "client.4659 192.168.0.1:0/21670065", 2020-11-20T11:32:43.591 INFO:teuthology.orchestra.run.smithi023.stdout: "addr_str": "192.168.0.1:0/21670065", 2020-11-20T11:32:43.591 INFO:teuthology.orchestra.run.smithi023.stdout: "inode_count": 1, 2020-11-20T11:32:43.591 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_epoch": 7, 2020-11-20T11:32:43.591 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_epoch": 25, 2020-11-20T11:32:43.591 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_epoch_barrier": 0, 2020-11-20T11:32:43.592 INFO:teuthology.orchestra.run.smithi023.stdout: "blocklisted": false, 2020-11-20T11:32:43.592 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs" 2020-11-20T11:32:43.592 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:32:43.594 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T11:32:43.594 INFO:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2020-11-20T11:32:43.595 INFO:teuthology.orchestra.run.smithi023:> ' -- /home/ubuntu/cephtest/mnt.0) 2020-11-20T11:32:43.610 INFO:teuthology.orchestra.run.smithi023.stdout:fuseblk 2020-11-20T11:32:43.610 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2020-11-20T11:32:43.611 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-11-20T11:32:43.611 INFO:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2020-11-20T11:32:43.676 INFO:teuthology.run_tasks:Running task mon_thrash... 2020-11-20T11:32:43.690 INFO:tasks.mon_thrash:Beginning mon_thrash... 2020-11-20T11:32:43.690 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T11:32:43.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.828+0000 7f281286b700 1 Processor -- start 2020-11-20T11:32:43.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.828+0000 7f281286b700 1 -- start start 2020-11-20T11:32:43.833 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.828+0000 7f281286b700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f280c065100 0x7f280c146620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:43.834 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.829+0000 7f281286b700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f280c0661b0 con 0x7f280c0aeb40 2020-11-20T11:32:43.834 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.829+0000 7f281286b700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f280c13ad60 con 0x7f280c0ae760 2020-11-20T11:32:43.834 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.829+0000 7f281286b700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f280c066c70 con 0x7f280c065100 2020-11-20T11:32:43.834 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.829+0000 7f280b7fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f280c0aeb40 0x7f280c137ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55592/0 (socket says 172.21.15.23:55592) 2020-11-20T11:32:43.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.829+0000 7f280b7fe700 1 -- 172.21.15.23:0/528346848 learned_addr learned my addr 172.21.15.23:0/528346848 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:43.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.829+0000 7f2810e08700 1 --2- 172.21.15.23:0/528346848 >> v2:172.21.15.155:3300/0 conn(0x7f280c065100 0x7f280c146620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:43.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.829+0000 7f280affd700 1 -- 172.21.15.23:0/528346848 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3357679495 0 0) 0x7f280c0661b0 con 0x7f280c0aeb40 2020-11-20T11:32:43.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.829+0000 7f280affd700 1 -- 172.21.15.23:0/528346848 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27f8003280 con 0x7f280c0aeb40 2020-11-20T11:32:43.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.829+0000 7f280affd700 1 -- 172.21.15.23:0/528346848 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4078140874 0 0) 0x7f280c13ad60 con 0x7f280c0ae760 2020-11-20T11:32:43.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.829+0000 7f280affd700 1 -- 172.21.15.23:0/528346848 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f280c0661b0 con 0x7f280c0ae760 2020-11-20T11:32:43.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.829+0000 7f280affd700 1 -- 172.21.15.23:0/528346848 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3022969043 0 0) 0x7f27f8003280 con 0x7f280c0aeb40 2020-11-20T11:32:43.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.829+0000 7f2810e08700 1 -- 172.21.15.23:0/528346848 >> v1:172.21.15.155:6789/0 conn(0x7f280c0ae760 legacy=0x7f280c13e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:43.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.829+0000 7f2810e08700 1 -- 172.21.15.23:0/528346848 >> v1:172.21.15.23:6789/0 conn(0x7f280c0aeb40 legacy=0x7f280c137ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:43.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.829+0000 7f2810e08700 1 -- 172.21.15.23:0/528346848 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f280c13e470 con 0x7f280c065100 2020-11-20T11:32:43.837 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.829+0000 7f2810e08700 1 --2- 172.21.15.23:0/528346848 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f280c065100 0x7f280c146620 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f2804009ce0 tx=0x7f2804009220).ready entity=mon.1 client_cookie=3f3dde8d70c58192 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:43.837 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.830+0000 7f280affd700 1 -- 172.21.15.23:0/528346848 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27fc001520 con 0x7f280c065100 2020-11-20T11:32:43.837 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.830+0000 7f280affd700 1 -- 172.21.15.23:0/528346848 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f280400b680 con 0x7f280c065100 2020-11-20T11:32:43.837 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.830+0000 7f280affd700 1 -- 172.21.15.23:0/528346848 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280400ba00 con 0x7f280c065100 2020-11-20T11:32:43.837 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.830+0000 7f281286b700 1 -- 172.21.15.23:0/528346848 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f280c065100 msgr2=0x7f280c146620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:43.838 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.830+0000 7f281286b700 1 --2- 172.21.15.23:0/528346848 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f280c065100 0x7f280c146620 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f2804009ce0 tx=0x7f2804009220).stop 2020-11-20T11:32:43.838 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.830+0000 7f281286b700 1 -- 172.21.15.23:0/528346848 shutdown_connections 2020-11-20T11:32:43.838 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.830+0000 7f281286b700 1 --2- 172.21.15.23:0/528346848 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f280c065100 0x7f280c146620 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:43.838 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.830+0000 7f281286b700 1 -- 172.21.15.23:0/528346848 >> 172.21.15.23:0/528346848 conn(0x7f280c061d80 msgr2=0x7f280c0ae050 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:43.838 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.830+0000 7f281286b700 1 -- 172.21.15.23:0/528346848 shutdown_connections 2020-11-20T11:32:43.839 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.830+0000 7f281286b700 1 -- 172.21.15.23:0/528346848 wait complete. 2020-11-20T11:32:43.839 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.831+0000 7f281286b700 1 Processor -- start 2020-11-20T11:32:43.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.831+0000 7f281286b700 1 -- start start 2020-11-20T11:32:43.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.832+0000 7f281286b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f280c065100 0x7f280c1410a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:43.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.832+0000 7f281286b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f280c0ae760 0x7f280c13a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:43.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.832+0000 7f281286b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f280c0aeb40 0x7f280c0a3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:43.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.832+0000 7f281286b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f280c0a3670 con 0x7f280c0ae760 2020-11-20T11:32:43.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.832+0000 7f281286b700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f280c0a1e40 con 0x7f280c0aeb40 2020-11-20T11:32:43.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.832+0000 7f281286b700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f280c0a20e0 con 0x7f280c065100 2020-11-20T11:32:43.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.832+0000 7f280b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f280c0ae760 0x7f280c13a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:43.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.832+0000 7f280b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f280c0ae760 0x7f280c13a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41346/0 (socket says 172.21.15.23:41346) 2020-11-20T11:32:43.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.832+0000 7f280b7fe700 1 -- 172.21.15.23:0/1579035578 learned_addr learned my addr 172.21.15.23:0/1579035578 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:43.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.832+0000 7f2810e08700 1 --2- 172.21.15.23:0/1579035578 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f280c0aeb40 0x7f280c0a3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:43.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.832+0000 7f280bfff700 1 --2- 172.21.15.23:0/1579035578 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f280c065100 0x7f280c1410a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:43.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.832+0000 7f280b7fe700 1 -- 172.21.15.23:0/1579035578 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f280c065100 msgr2=0x7f280c1410a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:43.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.832+0000 7f280b7fe700 1 --2- 172.21.15.23:0/1579035578 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f280c065100 0x7f280c1410a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:43.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.832+0000 7f280b7fe700 1 -- 172.21.15.23:0/1579035578 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f280c0aeb40 msgr2=0x7f280c0a3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:43.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.832+0000 7f280b7fe700 1 --2- 172.21.15.23:0/1579035578 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f280c0aeb40 0x7f280c0a3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:43.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.832+0000 7f280b7fe700 1 -- 172.21.15.23:0/1579035578 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f280c0a1ac0 con 0x7f280c0ae760 2020-11-20T11:32:43.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.832+0000 7f280b7fe700 1 --2- 172.21.15.23:0/1579035578 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f280c0ae760 0x7f280c13a6c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f27fc00cae0 tx=0x7f27fc001620).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:43.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.833+0000 7f27ebfff700 1 -- 172.21.15.23:0/1579035578 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27fc007e30 con 0x7f280c0ae760 2020-11-20T11:32:43.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.833+0000 7f27ebfff700 1 -- 172.21.15.23:0/1579035578 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27fc007320 con 0x7f280c0ae760 2020-11-20T11:32:43.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.833+0000 7f281286b700 1 -- 172.21.15.23:0/1579035578 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f280c0a45e0 con 0x7f280c0ae760 2020-11-20T11:32:43.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.833+0000 7f27ebfff700 1 -- 172.21.15.23:0/1579035578 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27fc005860 con 0x7f280c0ae760 2020-11-20T11:32:43.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.833+0000 7f281286b700 1 -- 172.21.15.23:0/1579035578 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f280c1421f0 con 0x7f280c0ae760 2020-11-20T11:32:43.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.834+0000 7f281286b700 1 -- 172.21.15.23:0/1579035578 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f280c0a22a0 con 0x7f280c0ae760 2020-11-20T11:32:43.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.834+0000 7f27ebfff700 1 -- 172.21.15.23:0/1579035578 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f27fc010030 con 0x7f280c0ae760 2020-11-20T11:32:43.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.834+0000 7f27ebfff700 1 --2- 172.21.15.23:0/1579035578 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f27e4056d60 0x7f27e4059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:43.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.834+0000 7f27ebfff700 1 -- 172.21.15.23:0/1579035578 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f27fc06fec0 con 0x7f280c0ae760 2020-11-20T11:32:43.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.834+0000 7f280bfff700 1 --2- 172.21.15.23:0/1579035578 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f27e4056d60 0x7f27e4059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:43.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.835+0000 7f280bfff700 1 --2- 172.21.15.23:0/1579035578 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f27e4056d60 0x7f27e4059220 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f2800003580 tx=0x7f2800009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:43.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.838+0000 7f27ebfff700 1 -- 172.21.15.23:0/1579035578 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f27fc0451b0 con 0x7f280c0ae760 2020-11-20T11:32:43.994 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.991+0000 7f281286b700 1 -- 172.21.15.23:0/1579035578 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f280c0adde0 con 0x7f280c0ae760 2020-11-20T11:32:43.995 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.992+0000 7f27ebfff700 1 -- 172.21.15.23:0/1579035578 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17087 (secure 0 0 0) 0x7f27fc0487d0 con 0x7f280c0ae760 2020-11-20T11:32:43.995 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:32:43.996 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":25,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","created":"2020-11-20T11:32:06.136679+0000","modified":"2020-11-20T11:32:26.470927+0000","last_up_change":"2020-11-20T11:32:13.536194+0000","last_in_change":"2020-11-20T11:32:13.536194+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T11:32:09.559868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T11:32:17.000782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T11:32:20.423341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T11:32:23.126273+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e9d219f5-e40e-4b2b-9468-deb9c4074a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":23083},{"type":"v1","addr":"172.21.15.23:6801","nonce":23083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":23083},{"type":"v1","addr":"172.21.15.23:6803","nonce":23083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":23083},{"type":"v1","addr":"172.21.15.23:6807","nonce":23083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":23083},{"type":"v1","addr":"172.21.15.23:6805","nonce":23083}]},"public_addr":"172.21.15.23:6801/23083","cluster_addr":"172.21.15.23:6803/23083","heartbeat_back_addr":"172.21.15.23:6807/23083","heartbeat_front_addr":"172.21.15.23:6805/23083","state":["exists","up"]},{"osd":1,"uuid":"2328adde-2108-4fe0-8138-f0d096a496a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":23099},{"type":"v1","addr":"172.21.15.23:6817","nonce":23099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":23099},{"type":"v1","addr":"172.21.15.23:6819","nonce":23099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":23099},{"type":"v1","addr":"172.21.15.23:6824","nonce":23099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":23099},{"type":"v1","addr":"172.21.15.23:6821","nonce":23099}]},"public_addr":"172.21.15.23:6817/23099","cluster_addr":"172.21.15.23:6819/23099","heartbeat_back_addr":"172.21.15.23:6824/23099","heartbeat_front_addr":"172.21.15.23:6821/23099","state":["exists","up"]},{"osd":2,"uuid":"13367708-59a2-4363-8a3a-ab0cb97145ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":23084},{"type":"v1","addr":"172.21.15.23:6809","nonce":23084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":23084},{"type":"v1","addr":"172.21.15.23:6811","nonce":23084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":23084},{"type":"v1","addr":"172.21.15.23:6815","nonce":23084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":23084},{"type":"v1","addr":"172.21.15.23:6813","nonce":23084}]},"public_addr":"172.21.15.23:6809/23084","cluster_addr":"172.21.15.23:6811/23084","heartbeat_back_addr":"172.21.15.23:6815/23084","heartbeat_front_addr":"172.21.15.23:6813/23084","state":["exists","up"]},{"osd":3,"uuid":"1f147dd7-9289-47b0-ad1c-4088618d6faa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6823","nonce":23114},{"type":"v1","addr":"172.21.15.23:6825","nonce":23114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":23114},{"type":"v1","addr":"172.21.15.23:6827","nonce":23114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":23114},{"type":"v1","addr":"172.21.15.23:6831","nonce":23114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":23114},{"type":"v1","addr":"172.21.15.23:6829","nonce":23114}]},"public_addr":"172.21.15.23:6825/23114","cluster_addr":"172.21.15.23:6827/23114","heartbeat_back_addr":"172.21.15.23:6831/23114","heartbeat_front_addr":"172.21.15.23:6829/23114","state":["exists","up"]},{"osd":4,"uuid":"d008b039-4bc5-4135-9678-1511e58efc29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":23280},{"type":"v1","addr":"172.21.15.155:6801","nonce":23280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":23280},{"type":"v1","addr":"172.21.15.155:6803","nonce":23280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":23280},{"type":"v1","addr":"172.21.15.155:6807","nonce":23280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":23280},{"type":"v1","addr":"172.21.15.155:6805","nonce":23280}]},"public_addr":"172.21.15.155:6801/23280","cluster_addr":"172.21.15.155:6803/23280","heartbeat_back_addr":"172.21.15.155:6807/23280","heartbeat_front_addr":"172.21.15.155:6805/23280","state":["exists","up"]},{"osd":5,"uuid":"56715e6f-cf6e-44a6-9e39-93ab1224095f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":23298},{"type":"v1","addr":"172.21.15.155:6817","nonce":23298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":23298},{"type":"v1","addr":"172.21.15.155:6819","nonce":23298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":23298},{"type":"v1","addr":"172.21.15.155:6823","nonce":23298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":23298},{"type":"v1","addr":"172.21.15.155:6821","nonce":23298}]},"public_addr":"172.21.15.155:6817/23298","cluster_addr":"172.21.15.155:6819/23298","heartbeat_back_addr":"172.21.15.155:6823/23298","heartbeat_front_addr":"172.21.15.155:6821/23298","state":["exists","up"]},{"osd":6,"uuid":"db1e61ca-8e14-4a1f-8607-6e01b148d61c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":23307},{"type":"v1","addr":"172.21.15.155:6825","nonce":23307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":23307},{"type":"v1","addr":"172.21.15.155:6827","nonce":23307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":23307},{"type":"v1","addr":"172.21.15.155:6831","nonce":23307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":23307},{"type":"v1","addr":"172.21.15.155:6829","nonce":23307}]},"public_addr":"172.21.15.155:6825/23307","cluster_addr":"172.21.15.155:6827/23307","heartbeat_back_addr":"172.21.15.155:6831/23307","heartbeat_front_addr":"172.21.15.155:6829/23307","state":["exists","up"]},{"osd":7,"uuid":"d44419be-ef89-462c-8cf1-3c6db2a6944b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":23281},{"type":"v1","addr":"172.21.15.155:6809","nonce":23281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":23281},{"type":"v1","addr":"172.21.15.155:6811","nonce":23281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":23281},{"type":"v1","addr":"172.21.15.155:6815","nonce":23281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":23281},{"type":"v1","addr":"172.21.15.155:6813","nonce":23281}]},"public_addr":"172.21.15.155:6809/23281","cluster_addr":"172.21.15.155:6811/23281","heartbeat_back_addr":"172.21.15.155:6815/23281","heartbeat_front_addr":"172.21.15.155:6813/23281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.486702+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.509396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.533414+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.509489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.467600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.513068+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.536636+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.531772+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T11:32:43.997 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.994+0000 7f281286b700 1 -- 172.21.15.23:0/1579035578 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f27e4056d60 msgr2=0x7f27e4059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:43.997 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.994+0000 7f281286b700 1 --2- 172.21.15.23:0/1579035578 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f27e4056d60 0x7f27e4059220 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f2800003580 tx=0x7f2800009040).stop 2020-11-20T11:32:43.997 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.994+0000 7f281286b700 1 -- 172.21.15.23:0/1579035578 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f280c0ae760 msgr2=0x7f280c13a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:43.997 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.994+0000 7f281286b700 1 --2- 172.21.15.23:0/1579035578 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f280c0ae760 0x7f280c13a6c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f27fc00cae0 tx=0x7f27fc001620).stop 2020-11-20T11:32:43.998 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.994+0000 7f281286b700 1 -- 172.21.15.23:0/1579035578 shutdown_connections 2020-11-20T11:32:43.998 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.994+0000 7f281286b700 1 --2- 172.21.15.23:0/1579035578 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f280c0aeb40 0x7f280c0a3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:43.998 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.994+0000 7f281286b700 1 --2- 172.21.15.23:0/1579035578 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f27e4056d60 0x7f27e4059220 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:43.998 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.994+0000 7f281286b700 1 --2- 172.21.15.23:0/1579035578 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f280c065100 0x7f280c1410a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:43.998 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.994+0000 7f281286b700 1 --2- 172.21.15.23:0/1579035578 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f280c0ae760 0x7f280c13a6c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:43.999 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.994+0000 7f281286b700 1 -- 172.21.15.23:0/1579035578 >> 172.21.15.23:0/1579035578 conn(0x7f280c061d80 msgr2=0x7f280c1372e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:43.999 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.994+0000 7f281286b700 1 -- 172.21.15.23:0/1579035578 shutdown_connections 2020-11-20T11:32:43.999 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:43.994+0000 7f281286b700 1 -- 172.21.15.23:0/1579035578 wait complete. 2020-11-20T11:32:44.008 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T11:32:09.559868+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T11:32:17.000782+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T11:32:20.423341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T11:32:23.126273+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T11:32:44.009 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T11:32:44.126 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.123+0000 7f311b59e700 1 Processor -- start 2020-11-20T11:32:44.127 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.123+0000 7f311b59e700 1 -- start start 2020-11-20T11:32:44.129 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.124+0000 7f311b59e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f311c0a9040 0x7f311c0aaf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:44.129 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.124+0000 7f311b59e700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f311c0ace70 0x7f311c13de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:44.129 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.124+0000 7f311b59e700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311c0ad860 con 0x7f311c0ad2f0 2020-11-20T11:32:44.129 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.124+0000 7f311b59e700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f311c0aa8a0 con 0x7f311c0ace70 2020-11-20T11:32:44.130 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.124+0000 7f311b59e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311c0aaa90 con 0x7f311c0a9040 2020-11-20T11:32:44.130 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.124+0000 7f3119d9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f311c0ace70 0x7f311c13de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:44.130 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.124+0000 7f3119d9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f311c0ace70 0x7f311c13de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41354/0 (socket says 172.21.15.23:41354) 2020-11-20T11:32:44.130 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.124+0000 7f311ad9d700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f311c0ad2f0 0x7f311c13e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55608/0 (socket says 172.21.15.23:55608) 2020-11-20T11:32:44.130 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.124+0000 7f3119d9b700 1 -- 172.21.15.23:0/883419739 learned_addr learned my addr 172.21.15.23:0/883419739 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:44.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.125+0000 7f311a59c700 1 --2- 172.21.15.23:0/883419739 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f311c0a9040 0x7f311c0aaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:44.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.125+0000 7f3119d9b700 1 -- 172.21.15.23:0/883419739 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f311c0a9040 msgr2=0x7f311c0aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:44.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.125+0000 7f3119d9b700 1 --2- 172.21.15.23:0/883419739 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f311c0a9040 0x7f311c0aaf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:44.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.125+0000 7f3119d9b700 1 -- 172.21.15.23:0/883419739 >> v1:172.21.15.23:6789/0 conn(0x7f311c0ad2f0 legacy=0x7f311c13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:44.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.125+0000 7f3119d9b700 1 -- 172.21.15.23:0/883419739 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311c0a9420 con 0x7f311c0ace70 2020-11-20T11:32:44.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.125+0000 7f3119d9b700 1 --2- 172.21.15.23:0/883419739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f311c0ace70 0x7f311c13de00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f3110009a10 tx=0x7f3110006ef0).ready entity=mon.0 client_cookie=493f50ec92b7bb98 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:44.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.125+0000 7f311959a700 1 -- 172.21.15.23:0/883419739 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311000b860 con 0x7f311c0ace70 2020-11-20T11:32:44.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.125+0000 7f311959a700 1 -- 172.21.15.23:0/883419739 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3110000e90 con 0x7f311c0ace70 2020-11-20T11:32:44.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.125+0000 7f311959a700 1 -- 172.21.15.23:0/883419739 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311000bbd0 con 0x7f311c0ace70 2020-11-20T11:32:44.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.125+0000 7f311b59e700 1 -- 172.21.15.23:0/883419739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f311c0ace70 msgr2=0x7f311c13de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:44.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.125+0000 7f311b59e700 1 --2- 172.21.15.23:0/883419739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f311c0ace70 0x7f311c13de00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f3110009a10 tx=0x7f3110006ef0).stop 2020-11-20T11:32:44.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.125+0000 7f311b59e700 1 -- 172.21.15.23:0/883419739 shutdown_connections 2020-11-20T11:32:44.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.125+0000 7f311b59e700 1 --2- 172.21.15.23:0/883419739 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f311c0a9040 0x7f311c0aaf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:44.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.125+0000 7f311b59e700 1 --2- 172.21.15.23:0/883419739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f311c0ace70 0x7f311c13de00 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:44.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.125+0000 7f311b59e700 1 -- 172.21.15.23:0/883419739 >> 172.21.15.23:0/883419739 conn(0x7f311c0683b0 msgr2=0x7f311c0b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:44.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.126+0000 7f311b59e700 1 -- 172.21.15.23:0/883419739 shutdown_connections 2020-11-20T11:32:44.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.126+0000 7f311b59e700 1 -- 172.21.15.23:0/883419739 wait complete. 2020-11-20T11:32:44.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.126+0000 7f311b59e700 1 Processor -- start 2020-11-20T11:32:44.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.126+0000 7f311b59e700 1 -- start start 2020-11-20T11:32:44.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.127+0000 7f311b59e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f311c0a9040 0x7f311c0a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:44.137 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.127+0000 7f311b59e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f311c0ace70 0x7f311c14a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:44.137 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.127+0000 7f311b59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f311c0ad2f0 0x7f311c13f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:44.137 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.127+0000 7f311b59e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311c14ab80 con 0x7f311c0ad2f0 2020-11-20T11:32:44.137 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.127+0000 7f311b59e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311c14a670 con 0x7f311c0ace70 2020-11-20T11:32:44.137 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.127+0000 7f311b59e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311c0a7620 con 0x7f311c0a9040 2020-11-20T11:32:44.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.127+0000 7f311ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f311c0ad2f0 0x7f311c13f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:44.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.127+0000 7f311ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f311c0ad2f0 0x7f311c13f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41364/0 (socket says 172.21.15.23:41364) 2020-11-20T11:32:44.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.127+0000 7f311ad9d700 1 -- 172.21.15.23:0/3772860540 learned_addr learned my addr 172.21.15.23:0/3772860540 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:44.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.127+0000 7f311a59c700 1 --2- 172.21.15.23:0/3772860540 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f311c0a9040 0x7f311c0a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:44.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.127+0000 7f3119d9b700 1 --2- 172.21.15.23:0/3772860540 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f311c0ace70 0x7f311c14a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:44.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.127+0000 7f311ad9d700 1 -- 172.21.15.23:0/3772860540 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f311c0a9040 msgr2=0x7f311c0a06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:44.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.127+0000 7f311ad9d700 1 --2- 172.21.15.23:0/3772860540 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f311c0a9040 0x7f311c0a06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:44.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.127+0000 7f311ad9d700 1 -- 172.21.15.23:0/3772860540 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f311c0ace70 msgr2=0x7f311c14a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:44.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.127+0000 7f311ad9d700 1 --2- 172.21.15.23:0/3772860540 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f311c0ace70 0x7f311c14a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:44.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.127+0000 7f311ad9d700 1 -- 172.21.15.23:0/3772860540 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311c14a8e0 con 0x7f311c0ad2f0 2020-11-20T11:32:44.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.128+0000 7f311ad9d700 1 --2- 172.21.15.23:0/3772860540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f311c0ad2f0 0x7f311c13f4b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f311400cac0 tx=0x7f31140013a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:44.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.128+0000 7f310a7fc700 1 -- 172.21.15.23:0/3772860540 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3114005b00 con 0x7f311c0ad2f0 2020-11-20T11:32:44.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.128+0000 7f310a7fc700 1 -- 172.21.15.23:0/3772860540 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3114005c60 con 0x7f311c0ad2f0 2020-11-20T11:32:44.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.128+0000 7f311b59e700 1 -- 172.21.15.23:0/3772860540 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f311c13f010 con 0x7f311c0ad2f0 2020-11-20T11:32:44.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.128+0000 7f310a7fc700 1 -- 172.21.15.23:0/3772860540 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3114007110 con 0x7f311c0ad2f0 2020-11-20T11:32:44.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.128+0000 7f311b59e700 1 -- 172.21.15.23:0/3772860540 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f311c13f9a0 con 0x7f311c0ad2f0 2020-11-20T11:32:44.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.129+0000 7f311b59e700 1 -- 172.21.15.23:0/3772860540 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f311c13f1a0 con 0x7f311c0ad2f0 2020-11-20T11:32:44.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.129+0000 7f310a7fc700 1 -- 172.21.15.23:0/3772860540 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3114010030 con 0x7f311c0ad2f0 2020-11-20T11:32:44.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.129+0000 7f310a7fc700 1 --2- 172.21.15.23:0/3772860540 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f30f0056d60 0x7f30f0059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:44.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.129+0000 7f310a7fc700 1 -- 172.21.15.23:0/3772860540 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f311406ffe0 con 0x7f311c0ad2f0 2020-11-20T11:32:44.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.130+0000 7f311a59c700 1 --2- 172.21.15.23:0/3772860540 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f30f0056d60 0x7f30f0059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:44.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.130+0000 7f311a59c700 1 --2- 172.21.15.23:0/3772860540 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f30f0056d60 0x7f30f0059220 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f310c006fd0 tx=0x7f310c008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:44.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.133+0000 7f310a7fc700 1 -- 172.21.15.23:0/3772860540 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f31140452d0 con 0x7f311c0ad2f0 2020-11-20T11:32:44.287 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.284+0000 7f311b59e700 1 -- 172.21.15.23:0/3772860540 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f311c057dc0 con 0x7f311c0ad2f0 2020-11-20T11:32:44.288 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.284+0000 7f310a7fc700 1 -- 172.21.15.23:0/3772860540 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v25) v1 ==== 110+0+10 (secure 0 0 0) 0x7f31140488f0 con 0x7f311c0ad2f0 2020-11-20T11:32:44.288 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2020-11-20T11:32:44.290 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.285+0000 7f311b59e700 1 -- 172.21.15.23:0/3772860540 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f30f0056d60 msgr2=0x7f30f0059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:44.290 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.285+0000 7f311b59e700 1 --2- 172.21.15.23:0/3772860540 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f30f0056d60 0x7f30f0059220 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f310c006fd0 tx=0x7f310c008040).stop 2020-11-20T11:32:44.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.285+0000 7f311b59e700 1 -- 172.21.15.23:0/3772860540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f311c0ad2f0 msgr2=0x7f311c13f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:44.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.285+0000 7f311b59e700 1 --2- 172.21.15.23:0/3772860540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f311c0ad2f0 0x7f311c13f4b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f311400cac0 tx=0x7f31140013a0).stop 2020-11-20T11:32:44.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.285+0000 7f311b59e700 1 -- 172.21.15.23:0/3772860540 shutdown_connections 2020-11-20T11:32:44.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.285+0000 7f311b59e700 1 --2- 172.21.15.23:0/3772860540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f311c0ad2f0 0x7f311c13f4b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:44.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.285+0000 7f311b59e700 1 --2- 172.21.15.23:0/3772860540 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f30f0056d60 0x7f30f0059220 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:44.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.285+0000 7f311b59e700 1 --2- 172.21.15.23:0/3772860540 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f311c0a9040 0x7f311c0a06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:44.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.285+0000 7f311b59e700 1 --2- 172.21.15.23:0/3772860540 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f311c0ace70 0x7f311c14a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:44.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.285+0000 7f311b59e700 1 -- 172.21.15.23:0/3772860540 >> 172.21.15.23:0/3772860540 conn(0x7f311c0683b0 msgr2=0x7f311c137440 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:44.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.286+0000 7f311b59e700 1 -- 172.21.15.23:0/3772860540 shutdown_connections 2020-11-20T11:32:44.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.286+0000 7f311b59e700 1 -- 172.21.15.23:0/3772860540 wait complete. 2020-11-20T11:32:44.300 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T11:32:44.417 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.414+0000 7f912c7c0700 1 Processor -- start 2020-11-20T11:32:44.417 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.414+0000 7f912c7c0700 1 -- start start 2020-11-20T11:32:44.419 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.415+0000 7f912c7c0700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f912413aeb0 0x7f91240a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:44.420 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.415+0000 7f912c7c0700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91241388b0 con 0x7f91240aaab0 2020-11-20T11:32:44.420 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.415+0000 7f912c7c0700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f912413e600 con 0x7f91240aae90 2020-11-20T11:32:44.420 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.415+0000 7f912c7c0700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f912413e8a0 con 0x7f912413aeb0 2020-11-20T11:32:44.420 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.415+0000 7f912ad5d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f912413aeb0 0x7f91240a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:44.420 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.415+0000 7f912ad5d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f912413aeb0 0x7f91240a8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41372/0 (socket says 172.21.15.23:41372) 2020-11-20T11:32:44.421 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.415+0000 7f912ad5d700 1 -- 172.21.15.23:0/2060132393 learned_addr learned my addr 172.21.15.23:0/2060132393 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:44.421 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.415+0000 7f912a55c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f91240aaab0 0x7f912413ed40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55618/0 (socket says 172.21.15.23:55618) 2020-11-20T11:32:44.421 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.416+0000 7f912ad5d700 1 -- 172.21.15.23:0/2060132393 >> v1:172.21.15.155:6789/0 conn(0x7f91240aae90 legacy=0x7f91241381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:44.421 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.416+0000 7f912ad5d700 1 -- 172.21.15.23:0/2060132393 >> v1:172.21.15.23:6789/0 conn(0x7f91240aaab0 legacy=0x7f912413ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:44.422 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.416+0000 7f912ad5d700 1 -- 172.21.15.23:0/2060132393 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91240ab6d0 con 0x7f912413aeb0 2020-11-20T11:32:44.426 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.416+0000 7f912955a700 1 -- 172.21.15.23:0/2060132393 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1248507023 0 0) 0x7f91241388b0 con 0x7f91240aaab0 2020-11-20T11:32:44.426 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.416+0000 7f912ad5d700 1 --2- 172.21.15.23:0/2060132393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f912413aeb0 0x7f91240a8c60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f911c009d90 tx=0x7f911c0092d0).ready entity=mon.0 client_cookie=c22e0f40b321c1d0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:44.426 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.416+0000 7f912955a700 1 -- 172.21.15.23:0/2060132393 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911c00bbb0 con 0x7f912413aeb0 2020-11-20T11:32:44.427 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.416+0000 7f912955a700 1 -- 172.21.15.23:0/2060132393 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f911c00bd10 con 0x7f912413aeb0 2020-11-20T11:32:44.427 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.416+0000 7f912955a700 1 -- 172.21.15.23:0/2060132393 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911c011610 con 0x7f912413aeb0 2020-11-20T11:32:44.427 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.416+0000 7f912c7c0700 1 -- 172.21.15.23:0/2060132393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f912413aeb0 msgr2=0x7f91240a8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:44.427 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.416+0000 7f912c7c0700 1 --2- 172.21.15.23:0/2060132393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f912413aeb0 0x7f91240a8c60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f911c009d90 tx=0x7f911c0092d0).stop 2020-11-20T11:32:44.427 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.416+0000 7f912c7c0700 1 -- 172.21.15.23:0/2060132393 shutdown_connections 2020-11-20T11:32:44.428 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.416+0000 7f912c7c0700 1 --2- 172.21.15.23:0/2060132393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f912413aeb0 0x7f91240a8c60 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:44.428 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.416+0000 7f912c7c0700 1 -- 172.21.15.23:0/2060132393 >> 172.21.15.23:0/2060132393 conn(0x7f91240678e0 msgr2=0x7f9124067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:44.428 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.416+0000 7f912c7c0700 1 -- 172.21.15.23:0/2060132393 shutdown_connections 2020-11-20T11:32:44.429 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.416+0000 7f912c7c0700 1 -- 172.21.15.23:0/2060132393 wait complete. 2020-11-20T11:32:44.431 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.417+0000 7f912c7c0700 1 Processor -- start 2020-11-20T11:32:44.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.417+0000 7f912c7c0700 1 -- start start 2020-11-20T11:32:44.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.418+0000 7f912c7c0700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f91240aaab0 0x7f91240a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:44.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.418+0000 7f912c7c0700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f91240aae90 0x7f91240a74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:44.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.418+0000 7f912c7c0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f912409f830 0x7f91240a09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:44.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.418+0000 7f912c7c0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91240a8330 con 0x7f912409f830 2020-11-20T11:32:44.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.418+0000 7f912c7c0700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912409fdb0 con 0x7f91240aaab0 2020-11-20T11:32:44.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.418+0000 7f912c7c0700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91240a0050 con 0x7f91240aae90 2020-11-20T11:32:44.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.418+0000 7f912ad5d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f912409f830 0x7f91240a09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:44.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.418+0000 7f912ad5d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f912409f830 0x7f91240a09c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41378/0 (socket says 172.21.15.23:41378) 2020-11-20T11:32:44.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.418+0000 7f912ad5d700 1 -- 172.21.15.23:0/2814006423 learned_addr learned my addr 172.21.15.23:0/2814006423 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:44.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.418+0000 7f912a55c700 1 --2- 172.21.15.23:0/2814006423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f91240aaab0 0x7f91240a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:44.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.418+0000 7f9129d5b700 1 --2- 172.21.15.23:0/2814006423 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f91240aae90 0x7f91240a74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:44.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.418+0000 7f912ad5d700 1 -- 172.21.15.23:0/2814006423 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f91240aae90 msgr2=0x7f91240a74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:44.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.418+0000 7f912ad5d700 1 --2- 172.21.15.23:0/2814006423 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f91240aae90 0x7f91240a74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:44.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.418+0000 7f912ad5d700 1 -- 172.21.15.23:0/2814006423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f91240aaab0 msgr2=0x7f91240a2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:44.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.418+0000 7f912ad5d700 1 --2- 172.21.15.23:0/2814006423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f91240aaab0 0x7f91240a2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:44.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.418+0000 7f912ad5d700 1 -- 172.21.15.23:0/2814006423 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91240a02f0 con 0x7f912409f830 2020-11-20T11:32:44.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.419+0000 7f912ad5d700 1 --2- 172.21.15.23:0/2814006423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f912409f830 0x7f91240a09c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f911c009130 tx=0x7f911c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:44.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.419+0000 7f91127fc700 1 -- 172.21.15.23:0/2814006423 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911c011e30 con 0x7f912409f830 2020-11-20T11:32:44.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.419+0000 7f91127fc700 1 -- 172.21.15.23:0/2814006423 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f911c010b40 con 0x7f912409f830 2020-11-20T11:32:44.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.419+0000 7f912c7c0700 1 -- 172.21.15.23:0/2814006423 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91240a0eb0 con 0x7f912409f830 2020-11-20T11:32:44.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.419+0000 7f91127fc700 1 -- 172.21.15.23:0/2814006423 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911c01c470 con 0x7f912409f830 2020-11-20T11:32:44.437 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.419+0000 7f912c7c0700 1 -- 172.21.15.23:0/2814006423 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91241374f0 con 0x7f912409f830 2020-11-20T11:32:44.437 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.420+0000 7f912c7c0700 1 -- 172.21.15.23:0/2814006423 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f912405c7d0 con 0x7f912409f830 2020-11-20T11:32:44.437 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.420+0000 7f91127fc700 1 -- 172.21.15.23:0/2814006423 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f911c017070 con 0x7f912409f830 2020-11-20T11:32:44.437 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.420+0000 7f91127fc700 1 --2- 172.21.15.23:0/2814006423 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f90fc056d60 0x7f90fc059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:44.437 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.420+0000 7f91127fc700 1 -- 172.21.15.23:0/2814006423 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f911c072410 con 0x7f912409f830 2020-11-20T11:32:44.438 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.421+0000 7f912a55c700 1 --2- 172.21.15.23:0/2814006423 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f90fc056d60 0x7f90fc059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:44.438 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.421+0000 7f912a55c700 1 --2- 172.21.15.23:0/2814006423 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f90fc056d60 0x7f90fc059220 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f91240a7c50 tx=0x7f9120007180).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:44.438 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.424+0000 7f91127fc700 1 -- 172.21.15.23:0/2814006423 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f911c0476e0 con 0x7f912409f830 2020-11-20T11:32:44.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.572+0000 7f912c7c0700 1 -- 172.21.15.23:0/2814006423 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f9124044710 con 0x7f912409f830 2020-11-20T11:32:44.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.572+0000 7f91127fc700 1 -- 172.21.15.23:0/2814006423 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v25) v1 ==== 92+0+10 (secure 0 0 0) 0x7f911c015020 con 0x7f912409f830 2020-11-20T11:32:44.576 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 8 2020-11-20T11:32:44.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.574+0000 7f912c7c0700 1 -- 172.21.15.23:0/2814006423 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f90fc056d60 msgr2=0x7f90fc059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:44.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.574+0000 7f912c7c0700 1 --2- 172.21.15.23:0/2814006423 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f90fc056d60 0x7f90fc059220 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f91240a7c50 tx=0x7f9120007180).stop 2020-11-20T11:32:44.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.574+0000 7f912c7c0700 1 -- 172.21.15.23:0/2814006423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f912409f830 msgr2=0x7f91240a09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:44.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.574+0000 7f912c7c0700 1 --2- 172.21.15.23:0/2814006423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f912409f830 0x7f91240a09c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f911c009130 tx=0x7f911c016000).stop 2020-11-20T11:32:44.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.574+0000 7f912c7c0700 1 -- 172.21.15.23:0/2814006423 shutdown_connections 2020-11-20T11:32:44.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.574+0000 7f912c7c0700 1 --2- 172.21.15.23:0/2814006423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f912409f830 0x7f91240a09c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:44.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.574+0000 7f912c7c0700 1 --2- 172.21.15.23:0/2814006423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f91240aaab0 0x7f91240a2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:44.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.574+0000 7f912c7c0700 1 --2- 172.21.15.23:0/2814006423 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f90fc056d60 0x7f90fc059220 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:44.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.574+0000 7f912c7c0700 1 --2- 172.21.15.23:0/2814006423 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f91240aae90 0x7f91240a74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:44.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.574+0000 7f912c7c0700 1 -- 172.21.15.23:0/2814006423 >> 172.21.15.23:0/2814006423 conn(0x7f91240678e0 msgr2=0x7f91240af260 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:44.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.574+0000 7f912c7c0700 1 -- 172.21.15.23:0/2814006423 shutdown_connections 2020-11-20T11:32:44.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.574+0000 7f912c7c0700 1 -- 172.21.15.23:0/2814006423 wait complete. 2020-11-20T11:32:44.588 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T11:32:44.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.701+0000 7f12a26bc700 1 Processor -- start 2020-11-20T11:32:44.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.701+0000 7f12a26bc700 1 -- start start 2020-11-20T11:32:44.708 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.702+0000 7f12a26bc700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f129c065100 0x7f129c146620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:44.708 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.702+0000 7f12a26bc700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f129c0661b0 con 0x7f129c0ae760 2020-11-20T11:32:44.709 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.702+0000 7f12a26bc700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f129c13ad60 con 0x7f129c0aeb40 2020-11-20T11:32:44.709 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.702+0000 7f12a26bc700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f129c066c70 con 0x7f129c065100 2020-11-20T11:32:44.709 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.702+0000 7f129bfff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f129c0ae760 0x7f129c13e910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55634/0 (socket says 172.21.15.23:55634) 2020-11-20T11:32:44.709 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.702+0000 7f129bfff700 1 -- 172.21.15.23:0/2938573661 learned_addr learned my addr 172.21.15.23:0/2938573661 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:44.710 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.703+0000 7f12a0c59700 1 --2- 172.21.15.23:0/2938573661 >> v2:172.21.15.155:3300/0 conn(0x7f129c065100 0x7f129c146620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:44.710 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.703+0000 7f129affd700 1 -- 172.21.15.23:0/2938573661 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 459170005 0 0) 0x7f129c0661b0 con 0x7f129c0ae760 2020-11-20T11:32:44.710 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.703+0000 7f129affd700 1 -- 172.21.15.23:0/2938573661 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1288003280 con 0x7f129c0ae760 2020-11-20T11:32:44.710 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.703+0000 7f129affd700 1 -- 172.21.15.23:0/2938573661 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3151796132 0 0) 0x7f1288003280 con 0x7f129c0ae760 2020-11-20T11:32:44.710 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.703+0000 7f129affd700 1 -- 172.21.15.23:0/2938573661 >> v2:172.21.15.155:3300/0 conn(0x7f129c065100 msgr2=0x7f129c146620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:44.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.703+0000 7f129affd700 1 --2- 172.21.15.23:0/2938573661 >> v2:172.21.15.155:3300/0 conn(0x7f129c065100 0x7f129c146620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:44.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.703+0000 7f129affd700 1 -- 172.21.15.23:0/2938573661 >> v1:172.21.15.155:6789/0 conn(0x7f129c0aeb40 legacy=0x7f129c137ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:44.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.703+0000 7f129affd700 1 -- 172.21.15.23:0/2938573661 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f129c13e470 con 0x7f129c0ae760 2020-11-20T11:32:44.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.703+0000 7f129affd700 1 -- 172.21.15.23:0/2938573661 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f1290001520 con 0x7f129c0ae760 2020-11-20T11:32:44.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.703+0000 7f129affd700 1 -- 172.21.15.23:0/2938573661 >> v1:172.21.15.23:6789/0 conn(0x7f129c0ae760 legacy=0x7f129c13e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:44.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.703+0000 7f129affd700 1 --2- 172.21.15.23:0/2938573661 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1288003770 0x7f1288003b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:44.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.703+0000 7f129affd700 1 --2- 172.21.15.23:0/2938573661 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1288004520 0x7f12880049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:44.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.703+0000 7f129affd700 1 --2- 172.21.15.23:0/2938573661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1288004e90 0x7f128800b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:44.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.703+0000 7f129affd700 1 -- 172.21.15.23:0/2938573661 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129c066c70 con 0x7f1288004e90 2020-11-20T11:32:44.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.703+0000 7f129affd700 1 -- 172.21.15.23:0/2938573661 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12880042a0 con 0x7f1288004520 2020-11-20T11:32:44.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.703+0000 7f129affd700 1 -- 172.21.15.23:0/2938573661 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f128800bc60 con 0x7f1288003770 2020-11-20T11:32:44.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.703+0000 7f12a0c59700 1 --2- 172.21.15.23:0/2938573661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1288004e90 0x7f128800b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:44.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.703+0000 7f129bfff700 1 --2- 172.21.15.23:0/2938573661 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1288003770 0x7f1288003b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:44.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.703+0000 7f129b7fe700 1 --2- 172.21.15.23:0/2938573661 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1288004520 0x7f12880049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:44.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.703+0000 7f12a0c59700 1 -- 172.21.15.23:0/2938573661 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1288003770 msgr2=0x7f1288003b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:44.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.703+0000 7f12a0c59700 1 --2- 172.21.15.23:0/2938573661 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1288003770 0x7f1288003b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:44.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.703+0000 7f12a0c59700 1 -- 172.21.15.23:0/2938573661 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1288004520 msgr2=0x7f12880049a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:44.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.703+0000 7f12a0c59700 1 --2- 172.21.15.23:0/2938573661 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1288004520 0x7f12880049a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:44.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.703+0000 7f12a0c59700 1 -- 172.21.15.23:0/2938573661 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1288003280 con 0x7f1288004e90 2020-11-20T11:32:44.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.704+0000 7f12a0c59700 1 -- 172.21.15.23:0/2938573661 reap_dead start 2020-11-20T11:32:44.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.704+0000 7f12a0c59700 1 --2- 172.21.15.23:0/2938573661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1288004e90 0x7f128800b770 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f1294009ec0 tx=0x7f1294009220).ready entity=mon.0 client_cookie=e24b3b30e90ca7c6 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:44.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.704+0000 7f129affd700 1 -- 172.21.15.23:0/2938573661 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1294009610 con 0x7f1288004e90 2020-11-20T11:32:44.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.704+0000 7f129affd700 1 -- 172.21.15.23:0/2938573661 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f129400ec80 con 0x7f1288004e90 2020-11-20T11:32:44.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.704+0000 7f129affd700 1 -- 172.21.15.23:0/2938573661 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12940123f0 con 0x7f1288004e90 2020-11-20T11:32:44.716 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.704+0000 7f12a26bc700 1 -- 172.21.15.23:0/2938573661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1288004e90 msgr2=0x7f128800b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:44.716 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.704+0000 7f12a26bc700 1 --2- 172.21.15.23:0/2938573661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1288004e90 0x7f128800b770 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f1294009ec0 tx=0x7f1294009220).stop 2020-11-20T11:32:44.716 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.704+0000 7f12a26bc700 1 -- 172.21.15.23:0/2938573661 shutdown_connections 2020-11-20T11:32:44.716 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.704+0000 7f12a26bc700 1 --2- 172.21.15.23:0/2938573661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1288004e90 0x7f128800b770 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:44.716 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.704+0000 7f12a26bc700 1 -- 172.21.15.23:0/2938573661 >> 172.21.15.23:0/2938573661 conn(0x7f129c061d80 msgr2=0x7f129c0ae050 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:44.717 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.704+0000 7f12a26bc700 1 -- 172.21.15.23:0/2938573661 shutdown_connections 2020-11-20T11:32:44.717 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.704+0000 7f12a26bc700 1 -- 172.21.15.23:0/2938573661 wait complete. 2020-11-20T11:32:44.717 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.705+0000 7f12a26bc700 1 Processor -- start 2020-11-20T11:32:44.717 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.705+0000 7f12a26bc700 1 -- start start 2020-11-20T11:32:44.719 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.706+0000 7f12a26bc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1288004e90 0x7f129c13d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:44.720 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.706+0000 7f12a26bc700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f129c0a18b0 0x7f129c14e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:44.720 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.706+0000 7f12a26bc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f129c0a3400 0x7f129c0a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:44.720 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.706+0000 7f12a26bc700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129c14e680 con 0x7f129c0a3400 2020-11-20T11:32:44.720 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.706+0000 7f12a26bc700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129c14d7b0 con 0x7f1288004e90 2020-11-20T11:32:44.721 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.706+0000 7f12a26bc700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f129c14da50 con 0x7f129c0a18b0 2020-11-20T11:32:44.721 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.706+0000 7f12a0c59700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f129c0a3400 0x7f129c0a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:44.721 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.706+0000 7f12a0c59700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f129c0a3400 0x7f129c0a53c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41398/0 (socket says 172.21.15.23:41398) 2020-11-20T11:32:44.721 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.706+0000 7f12a0c59700 1 -- 172.21.15.23:0/816283050 learned_addr learned my addr 172.21.15.23:0/816283050 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:44.721 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.706+0000 7f129bfff700 1 --2- 172.21.15.23:0/816283050 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1288004e90 0x7f129c13d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:44.722 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.706+0000 7f129b7fe700 1 --2- 172.21.15.23:0/816283050 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f129c0a18b0 0x7f129c14e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:44.722 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.707+0000 7f12a0c59700 1 -- 172.21.15.23:0/816283050 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f129c0a18b0 msgr2=0x7f129c14e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:44.722 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.707+0000 7f12a0c59700 1 --2- 172.21.15.23:0/816283050 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f129c0a18b0 0x7f129c14e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:44.722 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.707+0000 7f12a0c59700 1 -- 172.21.15.23:0/816283050 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1288004e90 msgr2=0x7f129c13d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:44.722 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.707+0000 7f12a0c59700 1 --2- 172.21.15.23:0/816283050 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1288004e90 0x7f129c13d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:44.723 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.707+0000 7f12a0c59700 1 -- 172.21.15.23:0/816283050 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f129c14dcf0 con 0x7f129c0a3400 2020-11-20T11:32:44.723 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.707+0000 7f12a0c59700 1 --2- 172.21.15.23:0/816283050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f129c0a3400 0x7f129c0a53c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f1294008000 tx=0x7f1294019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:44.723 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.707+0000 7f127bfff700 1 -- 172.21.15.23:0/816283050 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1294021070 con 0x7f129c0a3400 2020-11-20T11:32:44.723 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.707+0000 7f127bfff700 1 -- 172.21.15.23:0/816283050 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1294012e10 con 0x7f129c0a3400 2020-11-20T11:32:44.723 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.707+0000 7f12a26bc700 1 -- 172.21.15.23:0/816283050 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f129c13df30 con 0x7f129c0a3400 2020-11-20T11:32:44.724 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.707+0000 7f127bfff700 1 -- 172.21.15.23:0/816283050 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129401b680 con 0x7f129c0a3400 2020-11-20T11:32:44.724 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.707+0000 7f12a26bc700 1 -- 172.21.15.23:0/816283050 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f129c0a5af0 con 0x7f129c0a3400 2020-11-20T11:32:44.724 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.708+0000 7f12a26bc700 1 -- 172.21.15.23:0/816283050 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f129c062760 con 0x7f129c0a3400 2020-11-20T11:32:44.724 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.708+0000 7f127bfff700 1 -- 172.21.15.23:0/816283050 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f129402d070 con 0x7f129c0a3400 2020-11-20T11:32:44.724 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.708+0000 7f127bfff700 1 --2- 172.21.15.23:0/816283050 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1274056d60 0x7f1274059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:44.725 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.708+0000 7f127bfff700 1 -- 172.21.15.23:0/816283050 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f1294071b80 con 0x7f129c0a3400 2020-11-20T11:32:44.725 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.709+0000 7f129bfff700 1 --2- 172.21.15.23:0/816283050 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1274056d60 0x7f1274059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:44.725 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.709+0000 7f129bfff700 1 --2- 172.21.15.23:0/816283050 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1274056d60 0x7f1274059220 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f12900013f0 tx=0x7f129c13aef0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:44.725 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.712+0000 7f127bfff700 1 -- 172.21.15.23:0/816283050 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1294046eb0 con 0x7f129c0a3400 2020-11-20T11:32:44.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.862+0000 7f12a26bc700 1 -- 172.21.15.23:0/816283050 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f129c14de80 con 0x7f129c0a3400 2020-11-20T11:32:44.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.862+0000 7f127bfff700 1 -- 172.21.15.23:0/816283050 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+10 (secure 0 0 0) 0x7f1294016020 con 0x7f129c0a3400 2020-11-20T11:32:44.866 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 8 2020-11-20T11:32:44.867 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.864+0000 7f12a26bc700 1 -- 172.21.15.23:0/816283050 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1274056d60 msgr2=0x7f1274059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:44.867 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.864+0000 7f12a26bc700 1 --2- 172.21.15.23:0/816283050 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1274056d60 0x7f1274059220 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f12900013f0 tx=0x7f129c13aef0).stop 2020-11-20T11:32:44.867 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.864+0000 7f12a26bc700 1 -- 172.21.15.23:0/816283050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f129c0a3400 msgr2=0x7f129c0a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:44.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.864+0000 7f12a26bc700 1 --2- 172.21.15.23:0/816283050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f129c0a3400 0x7f129c0a53c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f1294008000 tx=0x7f1294019000).stop 2020-11-20T11:32:44.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.864+0000 7f12a26bc700 1 -- 172.21.15.23:0/816283050 shutdown_connections 2020-11-20T11:32:44.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.864+0000 7f12a26bc700 1 --2- 172.21.15.23:0/816283050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f129c0a3400 0x7f129c0a53c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:44.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.864+0000 7f12a26bc700 1 --2- 172.21.15.23:0/816283050 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1288004e90 0x7f129c13d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:44.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.864+0000 7f12a26bc700 1 --2- 172.21.15.23:0/816283050 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1274056d60 0x7f1274059220 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:44.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.864+0000 7f12a26bc700 1 --2- 172.21.15.23:0/816283050 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f129c0a18b0 0x7f129c14e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:44.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.864+0000 7f12a26bc700 1 -- 172.21.15.23:0/816283050 >> 172.21.15.23:0/816283050 conn(0x7f129c061d80 msgr2=0x7f129c0ae030 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:44.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.864+0000 7f12a26bc700 1 -- 172.21.15.23:0/816283050 shutdown_connections 2020-11-20T11:32:44.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.864+0000 7f12a26bc700 1 -- 172.21.15.23:0/816283050 wait complete. 2020-11-20T11:32:44.878 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T11:32:44.994 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.991+0000 7f68bf5ef700 1 Processor -- start 2020-11-20T11:32:44.995 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.991+0000 7f68bf5ef700 1 -- start start 2020-11-20T11:32:44.997 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.992+0000 7f68bf5ef700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f68b80aae90 0x7f68b81381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:44.997 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.992+0000 7f68bf5ef700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f68b813aeb0 0x7f68b80a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:44.998 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.992+0000 7f68bf5ef700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b813e600 con 0x7f68b80aaab0 2020-11-20T11:32:44.998 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.992+0000 7f68bf5ef700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68b813e8a0 con 0x7f68b813aeb0 2020-11-20T11:32:44.998 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.992+0000 7f68bf5ef700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68b80ab6d0 con 0x7f68b80aae90 2020-11-20T11:32:44.998 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.992+0000 7f68bd38b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f68b80aaab0 0x7f68b813ed40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55652/0 (socket says 172.21.15.23:55652) 2020-11-20T11:32:44.999 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.992+0000 7f68bd38b700 1 -- 172.21.15.23:0/1188216037 learned_addr learned my addr 172.21.15.23:0/1188216037 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:44.999 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.992+0000 7f68bdb8c700 1 --2- 172.21.15.23:0/1188216037 >> v2:172.21.15.23:3300/0 conn(0x7f68b813aeb0 0x7f68b80a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:44.999 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.992+0000 7f68bcb8a700 1 --2- 172.21.15.23:0/1188216037 >> v2:172.21.15.155:3300/0 conn(0x7f68b80aae90 0x7f68b81381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:44.999 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.993+0000 7f68a7fff700 1 -- 172.21.15.23:0/1188216037 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4212882616 0 0) 0x7f68b813e600 con 0x7f68b80aaab0 2020-11-20T11:32:44.999 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.993+0000 7f68a7fff700 1 -- 172.21.15.23:0/1188216037 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68a0003280 con 0x7f68b80aaab0 2020-11-20T11:32:45.000 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.993+0000 7f68bdb8c700 1 -- 172.21.15.23:0/1188216037 >> v2:172.21.15.155:3300/0 conn(0x7f68b80aae90 msgr2=0x7f68b81381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:45.000 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.993+0000 7f68bdb8c700 1 --2- 172.21.15.23:0/1188216037 >> v2:172.21.15.155:3300/0 conn(0x7f68b80aae90 0x7f68b81381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.000 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.993+0000 7f68bdb8c700 1 -- 172.21.15.23:0/1188216037 >> v1:172.21.15.23:6789/0 conn(0x7f68b80aaab0 legacy=0x7f68b813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:45.000 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.993+0000 7f68bdb8c700 1 -- 172.21.15.23:0/1188216037 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b8138690 con 0x7f68b813aeb0 2020-11-20T11:32:45.001 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.993+0000 7f68bdb8c700 1 --2- 172.21.15.23:0/1188216037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b813aeb0 0x7f68b80a8c60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f68b80ab930 tx=0x7f68b0009220).ready entity=mon.0 client_cookie=99aaac474180cc56 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:45.001 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.993+0000 7f68a7fff700 1 -- 172.21.15.23:0/1188216037 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b000b950 con 0x7f68b813aeb0 2020-11-20T11:32:45.001 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.993+0000 7f68a7fff700 1 -- 172.21.15.23:0/1188216037 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68b000bab0 con 0x7f68b813aeb0 2020-11-20T11:32:45.001 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.993+0000 7f68a7fff700 1 -- 172.21.15.23:0/1188216037 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b000be30 con 0x7f68b813aeb0 2020-11-20T11:32:45.002 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.993+0000 7f68bf5ef700 1 -- 172.21.15.23:0/1188216037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b813aeb0 msgr2=0x7f68b80a8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:45.002 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.993+0000 7f68bf5ef700 1 --2- 172.21.15.23:0/1188216037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b813aeb0 0x7f68b80a8c60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f68b80ab930 tx=0x7f68b0009220).stop 2020-11-20T11:32:45.002 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.993+0000 7f68bf5ef700 1 -- 172.21.15.23:0/1188216037 shutdown_connections 2020-11-20T11:32:45.002 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.993+0000 7f68bf5ef700 1 --2- 172.21.15.23:0/1188216037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b813aeb0 0x7f68b80a8c60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.003 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.993+0000 7f68bf5ef700 1 --2- 172.21.15.23:0/1188216037 >> v2:172.21.15.155:3300/0 conn(0x7f68b80aae90 0x7f68b81381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.003 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.993+0000 7f68bf5ef700 1 -- 172.21.15.23:0/1188216037 >> 172.21.15.23:0/1188216037 conn(0x7f68b80678e0 msgr2=0x7f68b8067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:45.003 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.994+0000 7f68bf5ef700 1 -- 172.21.15.23:0/1188216037 shutdown_connections 2020-11-20T11:32:45.003 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.994+0000 7f68bf5ef700 1 -- 172.21.15.23:0/1188216037 wait complete. 2020-11-20T11:32:45.003 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.994+0000 7f68bf5ef700 1 Processor -- start 2020-11-20T11:32:45.004 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.994+0000 7f68bf5ef700 1 -- start start 2020-11-20T11:32:45.006 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.995+0000 7f68bf5ef700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b80aaab0 0x7f68b80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:45.006 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.995+0000 7f68bf5ef700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68b80aae90 0x7f68b8134b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:45.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.995+0000 7f68bf5ef700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68b80a56b0 0x7f68b80a5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:45.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.995+0000 7f68bf5ef700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b80a73e0 con 0x7f68b80aaab0 2020-11-20T11:32:45.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.995+0000 7f68bf5ef700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b80a5fe0 con 0x7f68b80aae90 2020-11-20T11:32:45.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.995+0000 7f68bf5ef700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b80a6780 con 0x7f68b80a56b0 2020-11-20T11:32:45.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.995+0000 7f68bd38b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b80aaab0 0x7f68b80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:45.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.995+0000 7f68bd38b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b80aaab0 0x7f68b80a61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41408/0 (socket says 172.21.15.23:41408) 2020-11-20T11:32:45.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.995+0000 7f68bd38b700 1 -- 172.21.15.23:0/1544223718 learned_addr learned my addr 172.21.15.23:0/1544223718 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:45.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.995+0000 7f68bdb8c700 1 --2- 172.21.15.23:0/1544223718 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68b80a56b0 0x7f68b80a5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:45.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.995+0000 7f68bcb8a700 1 --2- 172.21.15.23:0/1544223718 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68b80aae90 0x7f68b8134b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:45.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.996+0000 7f68bd38b700 1 -- 172.21.15.23:0/1544223718 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68b80a56b0 msgr2=0x7f68b80a5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:45.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.996+0000 7f68bd38b700 1 --2- 172.21.15.23:0/1544223718 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68b80a56b0 0x7f68b80a5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.996+0000 7f68bd38b700 1 -- 172.21.15.23:0/1544223718 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68b80aae90 msgr2=0x7f68b8134b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:45.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.996+0000 7f68bd38b700 1 --2- 172.21.15.23:0/1544223718 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68b80aae90 0x7f68b8134b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.996+0000 7f68bd38b700 1 -- 172.21.15.23:0/1544223718 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b80a6a20 con 0x7f68b80aaab0 2020-11-20T11:32:45.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.996+0000 7f68bcb8a700 1 --2- 172.21.15.23:0/1544223718 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68b80aae90 0x7f68b8134b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:45.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.996+0000 7f68bdb8c700 1 --2- 172.21.15.23:0/1544223718 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68b80a56b0 0x7f68b80a5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:45.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.996+0000 7f68bd38b700 1 --2- 172.21.15.23:0/1544223718 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b80aaab0 0x7f68b80a61e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f68b400c770 tx=0x7f68b4007b10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:45.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.996+0000 7f68a57fa700 1 -- 172.21.15.23:0/1544223718 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b4007320 con 0x7f68b80aaab0 2020-11-20T11:32:45.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.996+0000 7f68a57fa700 1 -- 172.21.15.23:0/1544223718 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68b4007480 con 0x7f68b80aaab0 2020-11-20T11:32:45.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.996+0000 7f68bf5ef700 1 -- 172.21.15.23:0/1544223718 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68b80a5210 con 0x7f68b80aaab0 2020-11-20T11:32:45.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.996+0000 7f68a57fa700 1 -- 172.21.15.23:0/1544223718 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b4005840 con 0x7f68b80aaab0 2020-11-20T11:32:45.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.996+0000 7f68bf5ef700 1 -- 172.21.15.23:0/1544223718 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b80b03e0 con 0x7f68b80aaab0 2020-11-20T11:32:45.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.997+0000 7f68bf5ef700 1 -- 172.21.15.23:0/1544223718 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68b805c7b0 con 0x7f68b80aaab0 2020-11-20T11:32:45.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.997+0000 7f68a57fa700 1 -- 172.21.15.23:0/1544223718 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f68b4010030 con 0x7f68b80aaab0 2020-11-20T11:32:45.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.997+0000 7f68a57fa700 1 --2- 172.21.15.23:0/1544223718 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6890056d60 0x7f6890059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:45.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.997+0000 7f68a57fa700 1 -- 172.21.15.23:0/1544223718 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f68b406ff00 con 0x7f68b80aaab0 2020-11-20T11:32:45.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.998+0000 7f68bcb8a700 1 --2- 172.21.15.23:0/1544223718 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6890056d60 0x7f6890059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:45.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:44.998+0000 7f68bcb8a700 1 --2- 172.21.15.23:0/1544223718 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6890056d60 0x7f6890059220 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f68b80b0f30 tx=0x7f68a8008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:45.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.001+0000 7f68a57fa700 1 -- 172.21.15.23:0/1544223718 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f68b4045270 con 0x7f68b80aaab0 2020-11-20T11:32:45.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.151+0000 7f68bf5ef700 1 -- 172.21.15.23:0/1544223718 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f68b80a53a0 con 0x7f68b80aaab0 2020-11-20T11:32:45.155 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.152+0000 7f68a57fa700 1 -- 172.21.15.23:0/1544223718 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+10 (secure 0 0 0) 0x7f68b4048890 con 0x7f68b80aaab0 2020-11-20T11:32:45.155 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 8 2020-11-20T11:32:45.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.153+0000 7f68bf5ef700 1 -- 172.21.15.23:0/1544223718 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6890056d60 msgr2=0x7f6890059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:45.157 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.153+0000 7f68bf5ef700 1 --2- 172.21.15.23:0/1544223718 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6890056d60 0x7f6890059220 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f68b80b0f30 tx=0x7f68a8008040).stop 2020-11-20T11:32:45.157 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.153+0000 7f68bf5ef700 1 -- 172.21.15.23:0/1544223718 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b80aaab0 msgr2=0x7f68b80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:45.157 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.153+0000 7f68bf5ef700 1 --2- 172.21.15.23:0/1544223718 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b80aaab0 0x7f68b80a61e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f68b400c770 tx=0x7f68b4007b10).stop 2020-11-20T11:32:45.157 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.153+0000 7f68bf5ef700 1 -- 172.21.15.23:0/1544223718 shutdown_connections 2020-11-20T11:32:45.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.153+0000 7f68bf5ef700 1 --2- 172.21.15.23:0/1544223718 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6890056d60 0x7f6890059220 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.153+0000 7f68bf5ef700 1 --2- 172.21.15.23:0/1544223718 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68b80a56b0 0x7f68b80a5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.153+0000 7f68bf5ef700 1 --2- 172.21.15.23:0/1544223718 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68b80aaab0 0x7f68b80a61e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.153+0000 7f68bf5ef700 1 --2- 172.21.15.23:0/1544223718 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68b80aae90 0x7f68b8134b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.153+0000 7f68bf5ef700 1 -- 172.21.15.23:0/1544223718 >> 172.21.15.23:0/1544223718 conn(0x7f68b80678e0 msgr2=0x7f68b8137ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:45.159 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.153+0000 7f68bf5ef700 1 -- 172.21.15.23:0/1544223718 shutdown_connections 2020-11-20T11:32:45.159 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.153+0000 7f68bf5ef700 1 -- 172.21.15.23:0/1544223718 wait complete. 2020-11-20T11:32:45.167 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2020-11-20T11:32:45.168 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-20T11:32:45.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.280+0000 7effd40c6700 1 Processor -- start 2020-11-20T11:32:45.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.280+0000 7effd40c6700 1 -- start start 2020-11-20T11:32:45.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.280+0000 7effd40c6700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7effcc0aaab0 0x7effcc13ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:45.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.280+0000 7effd40c6700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7effcc13aeb0 0x7effcc0a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:45.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.280+0000 7effd40c6700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effcc1388b0 con 0x7effcc0aae90 2020-11-20T11:32:45.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.280+0000 7effd40c6700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effcc13e600 con 0x7effcc13aeb0 2020-11-20T11:32:45.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.280+0000 7effd40c6700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effcc13e8a0 con 0x7effcc0aaab0 2020-11-20T11:32:45.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.281+0000 7effd1e62700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7effcc0aaab0 0x7effcc13ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:45.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.281+0000 7effd2663700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7effcc13aeb0 0x7effcc0a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:45.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.281+0000 7effd2663700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7effcc13aeb0 0x7effcc0a8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:34314/0 (socket says 172.21.15.23:34314) 2020-11-20T11:32:45.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.281+0000 7effd1661700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7effcc0aae90 0x7effcc1381a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:33644/0 (socket says 172.21.15.23:33644) 2020-11-20T11:32:45.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.281+0000 7effd1e62700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7effcc0aaab0 0x7effcc13ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:51656/0 (socket says 172.21.15.23:51656) 2020-11-20T11:32:45.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.281+0000 7effd2663700 1 -- 172.21.15.23:0/2760855415 learned_addr learned my addr 172.21.15.23:0/2760855415 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:45.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.281+0000 7effd2663700 1 -- 172.21.15.23:0/2760855415 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7effcc0aaab0 msgr2=0x7effcc13ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:45.287 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.281+0000 7effd2663700 1 --2- 172.21.15.23:0/2760855415 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7effcc0aaab0 0x7effcc13ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.287 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.281+0000 7effd2663700 1 -- 172.21.15.23:0/2760855415 >> v1:172.21.15.155:6789/0 conn(0x7effcc0aae90 legacy=0x7effcc1381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:45.287 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.281+0000 7effd2663700 1 -- 172.21.15.23:0/2760855415 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effcc0ab6d0 con 0x7effcc13aeb0 2020-11-20T11:32:45.287 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.281+0000 7effd0e60700 1 -- 172.21.15.23:0/2760855415 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502459903 0 0) 0x7effcc1388b0 con 0x7effcc0aae90 2020-11-20T11:32:45.287 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.281+0000 7effd1e62700 1 --2- 172.21.15.23:0/2760855415 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7effcc0aaab0 0x7effcc13ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:32:45.288 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.281+0000 7effd2663700 1 --2- 172.21.15.23:0/2760855415 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effcc13aeb0 0x7effcc0a8c60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7effc4009d90 tx=0x7effc40092d0).ready entity=mon.1 client_cookie=ac14a020cbe9a898 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:45.288 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.281+0000 7effd0e60700 1 -- 172.21.15.23:0/2760855415 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc400bbf0 con 0x7effcc13aeb0 2020-11-20T11:32:45.288 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.282+0000 7effd0e60700 1 -- 172.21.15.23:0/2760855415 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effc400bd50 con 0x7effcc13aeb0 2020-11-20T11:32:45.288 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.282+0000 7effd0e60700 1 -- 172.21.15.23:0/2760855415 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc4011610 con 0x7effcc13aeb0 2020-11-20T11:32:45.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.282+0000 7effd40c6700 1 -- 172.21.15.23:0/2760855415 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effcc13aeb0 msgr2=0x7effcc0a8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:45.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.282+0000 7effd40c6700 1 --2- 172.21.15.23:0/2760855415 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effcc13aeb0 0x7effcc0a8c60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7effc4009d90 tx=0x7effc40092d0).stop 2020-11-20T11:32:45.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.282+0000 7effd40c6700 1 -- 172.21.15.23:0/2760855415 shutdown_connections 2020-11-20T11:32:45.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.282+0000 7effd40c6700 1 --2- 172.21.15.23:0/2760855415 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effcc13aeb0 0x7effcc0a8c60 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.282+0000 7effd40c6700 1 --2- 172.21.15.23:0/2760855415 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7effcc0aaab0 0x7effcc13ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.282+0000 7effd40c6700 1 -- 172.21.15.23:0/2760855415 >> 172.21.15.23:0/2760855415 conn(0x7effcc0678e0 msgr2=0x7effcc067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:45.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.282+0000 7effd40c6700 1 -- 172.21.15.23:0/2760855415 shutdown_connections 2020-11-20T11:32:45.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.282+0000 7effd40c6700 1 -- 172.21.15.23:0/2760855415 wait complete. 2020-11-20T11:32:45.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.283+0000 7effd40c6700 1 Processor -- start 2020-11-20T11:32:45.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.283+0000 7effd40c6700 1 -- start start 2020-11-20T11:32:45.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.283+0000 7effd40c6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effcc0aaab0 0x7effcc0a7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:45.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.284+0000 7effd40c6700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effcc0aae90 0x7effcc09e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:45.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.284+0000 7effd40c6700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7effcc09ee80 0x7effcc09e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:45.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.284+0000 7effd40c6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effcc0a02b0 con 0x7effcc0aaab0 2020-11-20T11:32:45.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.284+0000 7effd40c6700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effcc09f390 con 0x7effcc0aae90 2020-11-20T11:32:45.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.284+0000 7effd40c6700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effcc09f630 con 0x7effcc09ee80 2020-11-20T11:32:45.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.284+0000 7effd1e62700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effcc0aaab0 0x7effcc0a7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:45.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.284+0000 7effd1e62700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effcc0aaab0 0x7effcc0a7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41422/0 (socket says 172.21.15.23:41422) 2020-11-20T11:32:45.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.284+0000 7effd1e62700 1 -- 172.21.15.23:0/2353083557 learned_addr learned my addr 172.21.15.23:0/2353083557 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:45.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.284+0000 7effd2663700 1 --2- 172.21.15.23:0/2353083557 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7effcc09ee80 0x7effcc09e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:45.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.284+0000 7effd1661700 1 --2- 172.21.15.23:0/2353083557 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effcc0aae90 0x7effcc09e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:45.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.284+0000 7effd1e62700 1 -- 172.21.15.23:0/2353083557 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7effcc09ee80 msgr2=0x7effcc09e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:45.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.284+0000 7effd1e62700 1 --2- 172.21.15.23:0/2353083557 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7effcc09ee80 0x7effcc09e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.284+0000 7effd1e62700 1 -- 172.21.15.23:0/2353083557 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effcc0aae90 msgr2=0x7effcc09e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:45.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.284+0000 7effd1e62700 1 --2- 172.21.15.23:0/2353083557 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effcc0aae90 0x7effcc09e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.284+0000 7effd1e62700 1 -- 172.21.15.23:0/2353083557 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effcc09f8d0 con 0x7effcc0aaab0 2020-11-20T11:32:45.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.284+0000 7effd1e62700 1 --2- 172.21.15.23:0/2353083557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effcc0aaab0 0x7effcc0a7db0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7effc800ef50 tx=0x7effc8006e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:45.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.284+0000 7effb9ffb700 1 -- 172.21.15.23:0/2353083557 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc800cb40 con 0x7effcc0aaab0 2020-11-20T11:32:45.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.284+0000 7effb9ffb700 1 -- 172.21.15.23:0/2353083557 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effc800cca0 con 0x7effcc0aaab0 2020-11-20T11:32:45.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.284+0000 7effd40c6700 1 -- 172.21.15.23:0/2353083557 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effcc0b0f80 con 0x7effcc0aaab0 2020-11-20T11:32:45.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.284+0000 7effb9ffb700 1 -- 172.21.15.23:0/2353083557 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8012600 con 0x7effcc0aaab0 2020-11-20T11:32:45.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.285+0000 7effd40c6700 1 -- 172.21.15.23:0/2353083557 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effcc0a1430 con 0x7effcc0aaab0 2020-11-20T11:32:45.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.285+0000 7effd40c6700 1 -- 172.21.15.23:0/2353083557 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effcc05c7d0 con 0x7effcc0aaab0 2020-11-20T11:32:45.298 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.286+0000 7effb9ffb700 1 -- 172.21.15.23:0/2353083557 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7effc8012970 con 0x7effcc0aaab0 2020-11-20T11:32:45.298 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.286+0000 7effb9ffb700 1 --2- 172.21.15.23:0/2353083557 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7effa4056d60 0x7effa4059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:45.298 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.286+0000 7effb9ffb700 1 -- 172.21.15.23:0/2353083557 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7effc8070670 con 0x7effcc0aaab0 2020-11-20T11:32:45.298 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.286+0000 7effd1661700 1 --2- 172.21.15.23:0/2353083557 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7effa4056d60 0x7effa4059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:45.298 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.286+0000 7effd1661700 1 --2- 172.21.15.23:0/2353083557 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7effa4056d60 0x7effa4059220 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7effbc007840 tx=0x7effbc007a60).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:45.299 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.290+0000 7effb9ffb700 1 -- 172.21.15.23:0/2353083557 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7effc8045960 con 0x7effcc0aaab0 2020-11-20T11:32:45.437 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.435+0000 7effd40c6700 1 -- 172.21.15.23:0/2353083557 --> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7effcc044710 con 0x7effa4056d60 2020-11-20T11:32:45.442 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.438+0000 7effb9ffb700 1 -- 172.21.15.23:0/2353083557 <== mgr.4105 v2:172.21.15.155:6832/22693 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+82213 (secure 0 0 0) 0x7effcc044710 con 0x7effa4056d60 2020-11-20T11:32:45.442 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:32:45.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.440+0000 7effd40c6700 1 -- 172.21.15.23:0/2353083557 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7effa4056d60 msgr2=0x7effa4059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:45.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.440+0000 7effd40c6700 1 --2- 172.21.15.23:0/2353083557 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7effa4056d60 0x7effa4059220 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7effbc007840 tx=0x7effbc007a60).stop 2020-11-20T11:32:45.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.440+0000 7effd40c6700 1 -- 172.21.15.23:0/2353083557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effcc0aaab0 msgr2=0x7effcc0a7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:45.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.440+0000 7effd40c6700 1 --2- 172.21.15.23:0/2353083557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effcc0aaab0 0x7effcc0a7db0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7effc800ef50 tx=0x7effc8006e30).stop 2020-11-20T11:32:45.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.440+0000 7effd40c6700 1 -- 172.21.15.23:0/2353083557 shutdown_connections 2020-11-20T11:32:45.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.440+0000 7effd40c6700 1 --2- 172.21.15.23:0/2353083557 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7effa4056d60 0x7effa4059220 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.440+0000 7effd40c6700 1 --2- 172.21.15.23:0/2353083557 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7effcc09ee80 0x7effcc09e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.440+0000 7effd40c6700 1 --2- 172.21.15.23:0/2353083557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effcc0aaab0 0x7effcc0a7db0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.440+0000 7effd40c6700 1 --2- 172.21.15.23:0/2353083557 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effcc0aae90 0x7effcc09e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.440+0000 7effd40c6700 1 -- 172.21.15.23:0/2353083557 >> 172.21.15.23:0/2353083557 conn(0x7effcc0678e0 msgr2=0x7effcc0acf80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:45.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.440+0000 7effd40c6700 1 -- 172.21.15.23:0/2353083557 shutdown_connections 2020-11-20T11:32:45.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.440+0000 7effd40c6700 1 -- 172.21.15.23:0/2353083557 wait complete. 2020-11-20T11:32:45.446 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2020-11-20T11:32:45.454 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2020-11-20T11:32:43.527218+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2304,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40,"ondisk_log_size":40,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":66,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":41152,"kb_used_data":2112,"kb_used_omap":0,"kb_used_meta":38976,"kb_avail":754933568,"statfs":{"total":773094113280,"available":773051973632,"internally_reserved":0,"allocated":2162688,"data_stored":862848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":39911424},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002954"},"pg_stats":[{"pgid":"4.2","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.480587+0000","last_change":"2020-11-20T11:32:25.125195+0000","last_active":"2020-11-20T11:32:25.480587+0000","last_peered":"2020-11-20T11:32:25.480587+0000","last_clean":"2020-11-20T11:32:25.480587+0000","last_became_active":"2020-11-20T11:32:25.124627+0000","last_became_peered":"2020-11-20T11:32:25.124627+0000","last_unstale":"2020-11-20T11:32:25.480587+0000","last_undegraded":"2020-11-20T11:32:25.480587+0000","last_fullsized":"2020-11-20T11:32:25.480587+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"16","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.480586+0000","last_change":"2020-11-20T11:32:19.482604+0000","last_active":"2020-11-20T11:32:25.480586+0000","last_peered":"2020-11-20T11:32:25.480586+0000","last_clean":"2020-11-20T11:32:25.480586+0000","last_became_active":"2020-11-20T11:32:19.482197+0000","last_became_peered":"2020-11-20T11:32:19.482197+0000","last_unstale":"2020-11-20T11:32:25.480586+0000","last_undegraded":"2020-11-20T11:32:25.480586+0000","last_fullsized":"2020-11-20T11:32:25.480586+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"24'6","reported_seq":"19","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.494434+0000","last_change":"2020-11-20T11:32:22.098576+0000","last_active":"2020-11-20T11:32:25.494434+0000","last_peered":"2020-11-20T11:32:25.494434+0000","last_clean":"2020-11-20T11:32:25.494434+0000","last_became_active":"2020-11-20T11:32:22.098264+0000","last_became_peered":"2020-11-20T11:32:22.098264+0000","last_unstale":"2020-11-20T11:32:25.494434+0000","last_undegraded":"2020-11-20T11:32:25.494434+0000","last_fullsized":"2020-11-20T11:32:25.494434+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.483873+0000","last_change":"2020-11-20T11:32:25.479776+0000","last_active":"2020-11-20T11:32:25.483873+0000","last_peered":"2020-11-20T11:32:25.483873+0000","last_clean":"2020-11-20T11:32:25.483873+0000","last_became_active":"2020-11-20T11:32:25.479350+0000","last_became_peered":"2020-11-20T11:32:25.479350+0000","last_unstale":"2020-11-20T11:32:25.483873+0000","last_undegraded":"2020-11-20T11:32:25.483873+0000","last_fullsized":"2020-11-20T11:32:25.483873+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"16","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.479488+0000","last_change":"2020-11-20T11:32:18.924328+0000","last_active":"2020-11-20T11:32:25.479488+0000","last_peered":"2020-11-20T11:32:25.479488+0000","last_clean":"2020-11-20T11:32:25.479488+0000","last_became_active":"2020-11-20T11:32:18.923879+0000","last_became_peered":"2020-11-20T11:32:18.923879+0000","last_unstale":"2020-11-20T11:32:25.479488+0000","last_undegraded":"2020-11-20T11:32:25.479488+0000","last_fullsized":"2020-11-20T11:32:25.479488+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"24'5","reported_seq":"18","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.488715+0000","last_change":"2020-11-20T11:32:22.093559+0000","last_active":"2020-11-20T11:32:25.488715+0000","last_peered":"2020-11-20T11:32:25.488715+0000","last_clean":"2020-11-20T11:32:25.488715+0000","last_became_active":"2020-11-20T11:32:22.093123+0000","last_became_peered":"2020-11-20T11:32:22.093123+0000","last_unstale":"2020-11-20T11:32:25.488715+0000","last_undegraded":"2020-11-20T11:32:25.488715+0000","last_fullsized":"2020-11-20T11:32:25.488715+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.481419+0000","last_change":"2020-11-20T11:32:25.125052+0000","last_active":"2020-11-20T11:32:25.481419+0000","last_peered":"2020-11-20T11:32:25.481419+0000","last_clean":"2020-11-20T11:32:25.481419+0000","last_became_active":"2020-11-20T11:32:25.124401+0000","last_became_peered":"2020-11-20T11:32:25.124401+0000","last_unstale":"2020-11-20T11:32:25.481419+0000","last_undegraded":"2020-11-20T11:32:25.481419+0000","last_fullsized":"2020-11-20T11:32:25.481419+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"15","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.474417+0000","last_change":"2020-11-20T11:32:18.927388+0000","last_active":"2020-11-20T11:32:25.474417+0000","last_peered":"2020-11-20T11:32:25.474417+0000","last_clean":"2020-11-20T11:32:25.474417+0000","last_became_active":"2020-11-20T11:32:18.926981+0000","last_became_peered":"2020-11-20T11:32:18.926981+0000","last_unstale":"2020-11-20T11:32:25.474417+0000","last_undegraded":"2020-11-20T11:32:25.474417+0000","last_fullsized":"2020-11-20T11:32:25.474417+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"24'8","reported_seq":"21","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.492278+0000","last_change":"2020-11-20T11:32:22.088128+0000","last_active":"2020-11-20T11:32:25.492278+0000","last_peered":"2020-11-20T11:32:25.492278+0000","last_clean":"2020-11-20T11:32:25.492278+0000","last_became_active":"2020-11-20T11:32:22.087723+0000","last_became_peered":"2020-11-20T11:32:22.087723+0000","last_unstale":"2020-11-20T11:32:25.492278+0000","last_undegraded":"2020-11-20T11:32:25.492278+0000","last_fullsized":"2020-11-20T11:32:25.492278+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.495311+0000","last_change":"2020-11-20T11:32:25.492514+0000","last_active":"2020-11-20T11:32:25.495311+0000","last_peered":"2020-11-20T11:32:25.495311+0000","last_clean":"2020-11-20T11:32:25.495311+0000","last_became_active":"2020-11-20T11:32:25.491657+0000","last_became_peered":"2020-11-20T11:32:25.491657+0000","last_unstale":"2020-11-20T11:32:25.495311+0000","last_undegraded":"2020-11-20T11:32:25.495311+0000","last_fullsized":"2020-11-20T11:32:25.495311+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"16","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.479536+0000","last_change":"2020-11-20T11:32:18.927044+0000","last_active":"2020-11-20T11:32:25.479536+0000","last_peered":"2020-11-20T11:32:25.479536+0000","last_clean":"2020-11-20T11:32:25.479536+0000","last_became_active":"2020-11-20T11:32:18.926711+0000","last_became_peered":"2020-11-20T11:32:18.926711+0000","last_unstale":"2020-11-20T11:32:25.479536+0000","last_undegraded":"2020-11-20T11:32:25.479536+0000","last_fullsized":"2020-11-20T11:32:25.479536+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'2","reported_seq":"20","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.488044+0000","last_change":"2020-11-20T11:32:14.446049+0000","last_active":"2020-11-20T11:32:25.488044+0000","last_peered":"2020-11-20T11:32:25.488044+0000","last_clean":"2020-11-20T11:32:25.488044+0000","last_became_active":"2020-11-20T11:32:14.445477+0000","last_became_peered":"2020-11-20T11:32:14.445477+0000","last_unstale":"2020-11-20T11:32:25.488044+0000","last_undegraded":"2020-11-20T11:32:25.488044+0000","last_fullsized":"2020-11-20T11:32:25.488044+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:10.515989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:10.515989+0000","last_clean_scrub_stamp":"2020-11-20T11:32:10.515989+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"24'1","reported_seq":"14","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.488459+0000","last_change":"2020-11-20T11:32:22.088160+0000","last_active":"2020-11-20T11:32:25.488459+0000","last_peered":"2020-11-20T11:32:25.488459+0000","last_clean":"2020-11-20T11:32:25.488459+0000","last_became_active":"2020-11-20T11:32:22.087777+0000","last_became_peered":"2020-11-20T11:32:22.087777+0000","last_unstale":"2020-11-20T11:32:25.488459+0000","last_undegraded":"2020-11-20T11:32:25.488459+0000","last_fullsized":"2020-11-20T11:32:25.488459+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.493247+0000","last_change":"2020-11-20T11:32:25.491667+0000","last_active":"2020-11-20T11:32:25.493247+0000","last_peered":"2020-11-20T11:32:25.493247+0000","last_clean":"2020-11-20T11:32:25.493247+0000","last_became_active":"2020-11-20T11:32:25.491100+0000","last_became_peered":"2020-11-20T11:32:25.491100+0000","last_unstale":"2020-11-20T11:32:25.493247+0000","last_undegraded":"2020-11-20T11:32:25.493247+0000","last_fullsized":"2020-11-20T11:32:25.493247+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"15","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.488045+0000","last_change":"2020-11-20T11:32:18.924600+0000","last_active":"2020-11-20T11:32:25.488045+0000","last_peered":"2020-11-20T11:32:25.488045+0000","last_clean":"2020-11-20T11:32:25.488045+0000","last_became_active":"2020-11-20T11:32:18.924324+0000","last_became_peered":"2020-11-20T11:32:18.924324+0000","last_unstale":"2020-11-20T11:32:25.488045+0000","last_undegraded":"2020-11-20T11:32:25.488045+0000","last_fullsized":"2020-11-20T11:32:25.488045+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"13","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.476386+0000","last_change":"2020-11-20T11:32:22.088716+0000","last_active":"2020-11-20T11:32:25.476386+0000","last_peered":"2020-11-20T11:32:25.476386+0000","last_clean":"2020-11-20T11:32:25.476386+0000","last_became_active":"2020-11-20T11:32:22.088414+0000","last_became_peered":"2020-11-20T11:32:22.088414+0000","last_unstale":"2020-11-20T11:32:25.476386+0000","last_undegraded":"2020-11-20T11:32:25.476386+0000","last_fullsized":"2020-11-20T11:32:25.476386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.493332+0000","last_change":"2020-11-20T11:32:25.491507+0000","last_active":"2020-11-20T11:32:25.493332+0000","last_peered":"2020-11-20T11:32:25.493332+0000","last_clean":"2020-11-20T11:32:25.493332+0000","last_became_active":"2020-11-20T11:32:25.490917+0000","last_became_peered":"2020-11-20T11:32:25.490917+0000","last_unstale":"2020-11-20T11:32:25.493332+0000","last_undegraded":"2020-11-20T11:32:25.493332+0000","last_fullsized":"2020-11-20T11:32:25.493332+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"15","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.488121+0000","last_change":"2020-11-20T11:32:19.482719+0000","last_active":"2020-11-20T11:32:25.488121+0000","last_peered":"2020-11-20T11:32:25.488121+0000","last_clean":"2020-11-20T11:32:25.488121+0000","last_became_active":"2020-11-20T11:32:19.482359+0000","last_became_peered":"2020-11-20T11:32:19.482359+0000","last_unstale":"2020-11-20T11:32:25.488121+0000","last_undegraded":"2020-11-20T11:32:25.488121+0000","last_fullsized":"2020-11-20T11:32:25.488121+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"24'8","reported_seq":"21","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.497370+0000","last_change":"2020-11-20T11:32:22.093099+0000","last_active":"2020-11-20T11:32:25.497370+0000","last_peered":"2020-11-20T11:32:25.497370+0000","last_clean":"2020-11-20T11:32:25.497370+0000","last_became_active":"2020-11-20T11:32:22.092619+0000","last_became_peered":"2020-11-20T11:32:22.092619+0000","last_unstale":"2020-11-20T11:32:25.497370+0000","last_undegraded":"2020-11-20T11:32:25.497370+0000","last_fullsized":"2020-11-20T11:32:25.497370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1535,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.493059+0000","last_change":"2020-11-20T11:32:25.492150+0000","last_active":"2020-11-20T11:32:25.493059+0000","last_peered":"2020-11-20T11:32:25.493059+0000","last_clean":"2020-11-20T11:32:25.493059+0000","last_became_active":"2020-11-20T11:32:25.491099+0000","last_became_peered":"2020-11-20T11:32:25.491099+0000","last_unstale":"2020-11-20T11:32:25.493059+0000","last_undegraded":"2020-11-20T11:32:25.493059+0000","last_fullsized":"2020-11-20T11:32:25.493059+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"24'6","reported_seq":"19","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.492963+0000","last_change":"2020-11-20T11:32:22.461697+0000","last_active":"2020-11-20T11:32:25.492963+0000","last_peered":"2020-11-20T11:32:25.492963+0000","last_clean":"2020-11-20T11:32:25.492963+0000","last_became_active":"2020-11-20T11:32:22.461331+0000","last_became_peered":"2020-11-20T11:32:22.461331+0000","last_unstale":"2020-11-20T11:32:25.492963+0000","last_undegraded":"2020-11-20T11:32:25.492963+0000","last_fullsized":"2020-11-20T11:32:25.492963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"16","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.480381+0000","last_change":"2020-11-20T11:32:18.923886+0000","last_active":"2020-11-20T11:32:25.480381+0000","last_peered":"2020-11-20T11:32:25.480381+0000","last_clean":"2020-11-20T11:32:25.480381+0000","last_became_active":"2020-11-20T11:32:18.923448+0000","last_became_peered":"2020-11-20T11:32:18.923448+0000","last_unstale":"2020-11-20T11:32:25.480381+0000","last_undegraded":"2020-11-20T11:32:25.480381+0000","last_fullsized":"2020-11-20T11:32:25.480381+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.498234+0000","last_change":"2020-11-20T11:32:25.491585+0000","last_active":"2020-11-20T11:32:25.498234+0000","last_peered":"2020-11-20T11:32:25.498234+0000","last_clean":"2020-11-20T11:32:25.498234+0000","last_became_active":"2020-11-20T11:32:25.491038+0000","last_became_peered":"2020-11-20T11:32:25.491038+0000","last_unstale":"2020-11-20T11:32:25.498234+0000","last_undegraded":"2020-11-20T11:32:25.498234+0000","last_fullsized":"2020-11-20T11:32:25.498234+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"15","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.473625+0000","last_change":"2020-11-20T11:32:18.924052+0000","last_active":"2020-11-20T11:32:25.473625+0000","last_peered":"2020-11-20T11:32:25.473625+0000","last_clean":"2020-11-20T11:32:25.473625+0000","last_became_active":"2020-11-20T11:32:18.923668+0000","last_became_peered":"2020-11-20T11:32:18.923668+0000","last_unstale":"2020-11-20T11:32:25.473625+0000","last_undegraded":"2020-11-20T11:32:25.473625+0000","last_fullsized":"2020-11-20T11:32:25.473625+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"25'4","reported_seq":"17","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:32:27.326193+0000","last_change":"2020-11-20T11:32:22.093019+0000","last_active":"2020-11-20T11:32:27.326193+0000","last_peered":"2020-11-20T11:32:27.326193+0000","last_clean":"2020-11-20T11:32:27.326193+0000","last_became_active":"2020-11-20T11:32:22.092604+0000","last_became_peered":"2020-11-20T11:32:22.092604+0000","last_unstale":"2020-11-20T11:32:27.326193+0000","last_undegraded":"2020-11-20T11:32:27.326193+0000","last_fullsized":"2020-11-20T11:32:27.326193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":8},{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":2304,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":6,"up_from":14,"seq":60129542152,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5064,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366776,"statfs":{"total":96636764160,"available":96631578624,"internally_reserved":0,"allocated":262144,"data_stored":107280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0029999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.075}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.103}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.125}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1739999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542153,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":278528,"data_stored":108015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94999999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3680000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4810000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.181}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5089999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6259999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5132,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366708,"statfs":{"total":96636764160,"available":96631508992,"internally_reserved":0,"allocated":266240,"data_stored":107314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4239999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2430000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3779999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4610000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5196,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366644,"statfs":{"total":96636764160,"available":96631443456,"internally_reserved":0,"allocated":266240,"data_stored":107314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.044}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93300000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.155}]}]},{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":270336,"data_stored":107855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0489999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85999999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99399999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":270336,"data_stored":107440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93999999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.052}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.079}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.196}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.171}]}]},{"osd":1,"up_from":14,"seq":60129542151,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5204,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366636,"statfs":{"total":96636764160,"available":96631435264,"internally_reserved":0,"allocated":274432,"data_stored":108815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79200000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.464}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4870000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3959999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4410000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542152,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5140,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366700,"statfs":{"total":96636764160,"available":96631500800,"internally_reserved":0,"allocated":274432,"data_stored":108815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.458}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3080000000000001}]}]}]}} 2020-11-20T11:32:45.456 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-20T11:32:45.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.569+0000 7f85a80f2700 1 Processor -- start 2020-11-20T11:32:45.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.569+0000 7f85a80f2700 1 -- start start 2020-11-20T11:32:45.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.569+0000 7f85a80f2700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f85a00aae90 0x7f85a01381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:45.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.569+0000 7f85a80f2700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f85a013aeb0 0x7f85a00a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:45.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.569+0000 7f85a80f2700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85a013e600 con 0x7f85a00aaab0 2020-11-20T11:32:45.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.569+0000 7f85a80f2700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85a013e8a0 con 0x7f85a013aeb0 2020-11-20T11:32:45.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.569+0000 7f85a80f2700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85a00ab6d0 con 0x7f85a00aae90 2020-11-20T11:32:45.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.570+0000 7f85a5e8e700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f85a00aaab0 0x7f85a013ed40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55680/0 (socket says 172.21.15.23:55680) 2020-11-20T11:32:45.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.570+0000 7f85a5e8e700 1 -- 172.21.15.23:0/4173050389 learned_addr learned my addr 172.21.15.23:0/4173050389 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:45.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.570+0000 7f85a668f700 1 --2- 172.21.15.23:0/4173050389 >> v2:172.21.15.155:3300/0 conn(0x7f85a013aeb0 0x7f85a00a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:45.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.570+0000 7f85a568d700 1 --2- 172.21.15.23:0/4173050389 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f85a00aae90 0x7f85a01381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:45.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.570+0000 7f85a4e8c700 1 -- 172.21.15.23:0/4173050389 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018497680 0 0) 0x7f85a013e600 con 0x7f85a00aaab0 2020-11-20T11:32:45.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.570+0000 7f85a4e8c700 1 -- 172.21.15.23:0/4173050389 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8588003280 con 0x7f85a00aaab0 2020-11-20T11:32:45.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.570+0000 7f85a4e8c700 1 -- 172.21.15.23:0/4173050389 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1496616427 0 0) 0x7f8588003280 con 0x7f85a00aaab0 2020-11-20T11:32:45.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.570+0000 7f85a4e8c700 1 -- 172.21.15.23:0/4173050389 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f85a00aae90 msgr2=0x7f85a01381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:45.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.570+0000 7f85a4e8c700 1 --2- 172.21.15.23:0/4173050389 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f85a00aae90 0x7f85a01381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.570+0000 7f85a4e8c700 1 -- 172.21.15.23:0/4173050389 >> v2:172.21.15.155:3300/0 conn(0x7f85a013aeb0 msgr2=0x7f85a00a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:45.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.570+0000 7f85a4e8c700 1 --2- 172.21.15.23:0/4173050389 >> v2:172.21.15.155:3300/0 conn(0x7f85a013aeb0 0x7f85a00a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.570+0000 7f85a4e8c700 1 -- 172.21.15.23:0/4173050389 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a0138690 con 0x7f85a00aaab0 2020-11-20T11:32:45.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.570+0000 7f85a4e8c700 1 -- 172.21.15.23:0/4173050389 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f8590001520 con 0x7f85a00aaab0 2020-11-20T11:32:45.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.570+0000 7f85a4e8c700 1 -- 172.21.15.23:0/4173050389 >> v1:172.21.15.23:6789/0 conn(0x7f85a00aaab0 legacy=0x7f85a013ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:45.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.570+0000 7f85a4e8c700 1 --2- 172.21.15.23:0/4173050389 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8588003770 0x7f8588003b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:45.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.570+0000 7f85a4e8c700 1 --2- 172.21.15.23:0/4173050389 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8588004520 0x7f85880049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:45.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.570+0000 7f85a4e8c700 1 --2- 172.21.15.23:0/4173050389 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f85a00aae90 0x7f8588004e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:45.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.570+0000 7f85a4e8c700 1 -- 172.21.15.23:0/4173050389 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a013e8a0 con 0x7f8588003770 2020-11-20T11:32:45.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.570+0000 7f85a4e8c700 1 -- 172.21.15.23:0/4173050389 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a00ab6d0 con 0x7f8588004520 2020-11-20T11:32:45.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.570+0000 7f85a4e8c700 1 -- 172.21.15.23:0/4173050389 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85880042b0 con 0x7f85a00aae90 2020-11-20T11:32:45.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.570+0000 7f85a5e8e700 1 --2- 172.21.15.23:0/4173050389 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8588003770 0x7f8588003b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:45.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.570+0000 7f85a568d700 1 --2- 172.21.15.23:0/4173050389 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8588004520 0x7f85880049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:45.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.570+0000 7f85a668f700 1 --2- 172.21.15.23:0/4173050389 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f85a00aae90 0x7f8588004e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:45.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.571+0000 7f85a5e8e700 1 -- 172.21.15.23:0/4173050389 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f85a00aae90 msgr2=0x7f8588004e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:45.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.571+0000 7f85a5e8e700 1 --2- 172.21.15.23:0/4173050389 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f85a00aae90 0x7f8588004e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.571+0000 7f85a5e8e700 1 -- 172.21.15.23:0/4173050389 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8588004520 msgr2=0x7f85880049a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:45.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.571+0000 7f85a5e8e700 1 --2- 172.21.15.23:0/4173050389 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8588004520 0x7f85880049a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.571+0000 7f85a5e8e700 1 -- 172.21.15.23:0/4173050389 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8588003280 con 0x7f8588003770 2020-11-20T11:32:45.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.571+0000 7f85a668f700 1 --2- 172.21.15.23:0/4173050389 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f85a00aae90 0x7f8588004e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:45.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.571+0000 7f85a5e8e700 1 --2- 172.21.15.23:0/4173050389 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8588003770 0x7f8588003b90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f859000c440 tx=0x7f8590007840).ready entity=mon.0 client_cookie=117b3d7b9e164bc8 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:45.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.571+0000 7f85a4e8c700 1 -- 172.21.15.23:0/4173050389 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859000ce40 con 0x7f8588003770 2020-11-20T11:32:45.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.571+0000 7f85a4e8c700 1 -- 172.21.15.23:0/4173050389 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85900072e0 con 0x7f8588003770 2020-11-20T11:32:45.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.571+0000 7f85a4e8c700 1 -- 172.21.15.23:0/4173050389 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8590005860 con 0x7f8588003770 2020-11-20T11:32:45.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.571+0000 7f85a80f2700 1 -- 172.21.15.23:0/4173050389 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8588003770 msgr2=0x7f8588003b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:45.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.571+0000 7f85a80f2700 1 --2- 172.21.15.23:0/4173050389 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8588003770 0x7f8588003b90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f859000c440 tx=0x7f8590007840).stop 2020-11-20T11:32:45.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.571+0000 7f85a668f700 1 -- 172.21.15.23:0/4173050389 reap_dead start 2020-11-20T11:32:45.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.571+0000 7f85a80f2700 1 -- 172.21.15.23:0/4173050389 shutdown_connections 2020-11-20T11:32:45.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.571+0000 7f85a80f2700 1 -- 172.21.15.23:0/4173050389 >> 172.21.15.23:0/4173050389 conn(0x7f85a00678e0 msgr2=0x7f85a0067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:45.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.571+0000 7f85a80f2700 1 -- 172.21.15.23:0/4173050389 shutdown_connections 2020-11-20T11:32:45.584 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.571+0000 7f85a80f2700 1 -- 172.21.15.23:0/4173050389 wait complete. 2020-11-20T11:32:45.584 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.572+0000 7f85a80f2700 1 Processor -- start 2020-11-20T11:32:45.584 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.572+0000 7f85a80f2700 1 -- start start 2020-11-20T11:32:45.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.573+0000 7f85a80f2700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8588003770 0x7f85a013dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:45.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.573+0000 7f85a80f2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85a013cad0 0x7f85a00a61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:45.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.573+0000 7f85a80f2700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85a014db00 0x7f85a014df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:45.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.573+0000 7f85a80f2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a013e250 con 0x7f85a013cad0 2020-11-20T11:32:45.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.573+0000 7f85a80f2700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a01334a0 con 0x7f85a014db00 2020-11-20T11:32:45.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.573+0000 7f85a80f2700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85a013d620 con 0x7f8588003770 2020-11-20T11:32:45.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.573+0000 7f85a568d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85a013cad0 0x7f85a00a61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:45.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.573+0000 7f85a568d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85a013cad0 0x7f85a00a61a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41444/0 (socket says 172.21.15.23:41444) 2020-11-20T11:32:45.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.573+0000 7f85a568d700 1 -- 172.21.15.23:0/619651680 learned_addr learned my addr 172.21.15.23:0/619651680 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:45.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.573+0000 7f85a5e8e700 1 --2- 172.21.15.23:0/619651680 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8588003770 0x7f85a013dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:45.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.573+0000 7f85a668f700 1 --2- 172.21.15.23:0/619651680 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85a014db00 0x7f85a014df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:45.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.573+0000 7f85a568d700 1 -- 172.21.15.23:0/619651680 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8588003770 msgr2=0x7f85a013dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:45.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.573+0000 7f85a568d700 1 --2- 172.21.15.23:0/619651680 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8588003770 0x7f85a013dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.573+0000 7f85a568d700 1 -- 172.21.15.23:0/619651680 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85a014db00 msgr2=0x7f85a014df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:45.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.573+0000 7f85a568d700 1 --2- 172.21.15.23:0/619651680 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85a014db00 0x7f85a014df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.573+0000 7f85a568d700 1 -- 172.21.15.23:0/619651680 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a013d8c0 con 0x7f85a013cad0 2020-11-20T11:32:45.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.574+0000 7f85a568d700 1 --2- 172.21.15.23:0/619651680 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85a013cad0 0x7f85a00a61a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f859c011cc0 tx=0x7f859c006e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:45.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.574+0000 7f858dffb700 1 -- 172.21.15.23:0/619651680 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859c00f9e0 con 0x7f85a013cad0 2020-11-20T11:32:45.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.574+0000 7f858dffb700 1 -- 172.21.15.23:0/619651680 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f859c000e90 con 0x7f85a013cad0 2020-11-20T11:32:45.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.574+0000 7f85a80f2700 1 -- 172.21.15.23:0/619651680 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85a00a6690 con 0x7f85a013cad0 2020-11-20T11:32:45.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.574+0000 7f858dffb700 1 -- 172.21.15.23:0/619651680 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859c00fd50 con 0x7f85a013cad0 2020-11-20T11:32:45.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.574+0000 7f85a80f2700 1 -- 172.21.15.23:0/619651680 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85a013d110 con 0x7f85a013cad0 2020-11-20T11:32:45.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.575+0000 7f85a80f2700 1 -- 172.21.15.23:0/619651680 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85a00ae960 con 0x7f85a013cad0 2020-11-20T11:32:45.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.575+0000 7f858dffb700 1 -- 172.21.15.23:0/619651680 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f859c014650 con 0x7f85a013cad0 2020-11-20T11:32:45.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.575+0000 7f858dffb700 1 --2- 172.21.15.23:0/619651680 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8578056d90 0x7f8578059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:45.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.575+0000 7f858dffb700 1 -- 172.21.15.23:0/619651680 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f859c0707e0 con 0x7f85a013cad0 2020-11-20T11:32:45.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.576+0000 7f85a5e8e700 1 --2- 172.21.15.23:0/619651680 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8578056d90 0x7f8578059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:45.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.576+0000 7f85a5e8e700 1 --2- 172.21.15.23:0/619651680 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8578056d90 0x7f8578059250 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f8590008fd0 tx=0x7f8590015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:45.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.579+0000 7f858dffb700 1 -- 172.21.15.23:0/619651680 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f859c045ad0 con 0x7f85a013cad0 2020-11-20T11:32:45.728 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.725+0000 7f85a80f2700 1 -- 172.21.15.23:0/619651680 --> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f85a005c7d0 con 0x7f8578056d90 2020-11-20T11:32:45.731 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.728+0000 7f858dffb700 1 -- 172.21.15.23:0/619651680 <== mgr.4105 v2:172.21.15.155:6832/22693 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+82220 (secure 0 0 0) 0x7f85a005c7d0 con 0x7f8578056d90 2020-11-20T11:32:45.732 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:32:45.733 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.730+0000 7f85a80f2700 1 -- 172.21.15.23:0/619651680 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8578056d90 msgr2=0x7f8578059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:45.733 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.730+0000 7f85a80f2700 1 --2- 172.21.15.23:0/619651680 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8578056d90 0x7f8578059250 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f8590008fd0 tx=0x7f8590015000).stop 2020-11-20T11:32:45.734 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.730+0000 7f85a80f2700 1 -- 172.21.15.23:0/619651680 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85a013cad0 msgr2=0x7f85a00a61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:45.734 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.730+0000 7f85a80f2700 1 --2- 172.21.15.23:0/619651680 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85a013cad0 0x7f85a00a61a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f859c011cc0 tx=0x7f859c006e30).stop 2020-11-20T11:32:45.734 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.730+0000 7f85a80f2700 1 -- 172.21.15.23:0/619651680 shutdown_connections 2020-11-20T11:32:45.734 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.730+0000 7f85a80f2700 1 --2- 172.21.15.23:0/619651680 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f85a014db00 0x7f85a014df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.735 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.730+0000 7f85a80f2700 1 --2- 172.21.15.23:0/619651680 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8578056d90 0x7f8578059250 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.735 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.730+0000 7f85a80f2700 1 --2- 172.21.15.23:0/619651680 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8588003770 0x7f85a013dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.735 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.730+0000 7f85a80f2700 1 --2- 172.21.15.23:0/619651680 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85a013cad0 0x7f85a00a61a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.735 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.730+0000 7f85a80f2700 1 -- 172.21.15.23:0/619651680 >> 172.21.15.23:0/619651680 conn(0x7f85a00678e0 msgr2=0x7f85a00b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:45.735 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.730+0000 7f85a80f2700 1 -- 172.21.15.23:0/619651680 shutdown_connections 2020-11-20T11:32:45.735 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.730+0000 7f85a80f2700 1 -- 172.21.15.23:0/619651680 wait complete. 2020-11-20T11:32:45.736 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2020-11-20T11:32:45.744 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2020-11-20T11:32:45.527607+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3529,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":66,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":41152,"kb_used_data":2112,"kb_used_omap":0,"kb_used_meta":38976,"kb_avail":754933568,"statfs":{"total":773094113280,"available":773051973632,"internally_reserved":0,"allocated":2162688,"data_stored":865298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":39911424},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1225,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002793"},"pg_stats":[{"pgid":"4.2","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.480587+0000","last_change":"2020-11-20T11:32:25.125195+0000","last_active":"2020-11-20T11:32:25.480587+0000","last_peered":"2020-11-20T11:32:25.480587+0000","last_clean":"2020-11-20T11:32:25.480587+0000","last_became_active":"2020-11-20T11:32:25.124627+0000","last_became_peered":"2020-11-20T11:32:25.124627+0000","last_unstale":"2020-11-20T11:32:25.480587+0000","last_undegraded":"2020-11-20T11:32:25.480587+0000","last_fullsized":"2020-11-20T11:32:25.480587+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"16","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.480586+0000","last_change":"2020-11-20T11:32:19.482604+0000","last_active":"2020-11-20T11:32:25.480586+0000","last_peered":"2020-11-20T11:32:25.480586+0000","last_clean":"2020-11-20T11:32:25.480586+0000","last_became_active":"2020-11-20T11:32:19.482197+0000","last_became_peered":"2020-11-20T11:32:19.482197+0000","last_unstale":"2020-11-20T11:32:25.480586+0000","last_undegraded":"2020-11-20T11:32:25.480586+0000","last_fullsized":"2020-11-20T11:32:25.480586+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"24'6","reported_seq":"19","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.494434+0000","last_change":"2020-11-20T11:32:22.098576+0000","last_active":"2020-11-20T11:32:25.494434+0000","last_peered":"2020-11-20T11:32:25.494434+0000","last_clean":"2020-11-20T11:32:25.494434+0000","last_became_active":"2020-11-20T11:32:22.098264+0000","last_became_peered":"2020-11-20T11:32:22.098264+0000","last_unstale":"2020-11-20T11:32:25.494434+0000","last_undegraded":"2020-11-20T11:32:25.494434+0000","last_fullsized":"2020-11-20T11:32:25.494434+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.483873+0000","last_change":"2020-11-20T11:32:25.479776+0000","last_active":"2020-11-20T11:32:25.483873+0000","last_peered":"2020-11-20T11:32:25.483873+0000","last_clean":"2020-11-20T11:32:25.483873+0000","last_became_active":"2020-11-20T11:32:25.479350+0000","last_became_peered":"2020-11-20T11:32:25.479350+0000","last_unstale":"2020-11-20T11:32:25.483873+0000","last_undegraded":"2020-11-20T11:32:25.483873+0000","last_fullsized":"2020-11-20T11:32:25.483873+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"16","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.479488+0000","last_change":"2020-11-20T11:32:18.924328+0000","last_active":"2020-11-20T11:32:25.479488+0000","last_peered":"2020-11-20T11:32:25.479488+0000","last_clean":"2020-11-20T11:32:25.479488+0000","last_became_active":"2020-11-20T11:32:18.923879+0000","last_became_peered":"2020-11-20T11:32:18.923879+0000","last_unstale":"2020-11-20T11:32:25.479488+0000","last_undegraded":"2020-11-20T11:32:25.479488+0000","last_fullsized":"2020-11-20T11:32:25.479488+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"24'5","reported_seq":"18","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.488715+0000","last_change":"2020-11-20T11:32:22.093559+0000","last_active":"2020-11-20T11:32:25.488715+0000","last_peered":"2020-11-20T11:32:25.488715+0000","last_clean":"2020-11-20T11:32:25.488715+0000","last_became_active":"2020-11-20T11:32:22.093123+0000","last_became_peered":"2020-11-20T11:32:22.093123+0000","last_unstale":"2020-11-20T11:32:25.488715+0000","last_undegraded":"2020-11-20T11:32:25.488715+0000","last_fullsized":"2020-11-20T11:32:25.488715+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.481419+0000","last_change":"2020-11-20T11:32:25.125052+0000","last_active":"2020-11-20T11:32:25.481419+0000","last_peered":"2020-11-20T11:32:25.481419+0000","last_clean":"2020-11-20T11:32:25.481419+0000","last_became_active":"2020-11-20T11:32:25.124401+0000","last_became_peered":"2020-11-20T11:32:25.124401+0000","last_unstale":"2020-11-20T11:32:25.481419+0000","last_undegraded":"2020-11-20T11:32:25.481419+0000","last_fullsized":"2020-11-20T11:32:25.481419+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"15","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.474417+0000","last_change":"2020-11-20T11:32:18.927388+0000","last_active":"2020-11-20T11:32:25.474417+0000","last_peered":"2020-11-20T11:32:25.474417+0000","last_clean":"2020-11-20T11:32:25.474417+0000","last_became_active":"2020-11-20T11:32:18.926981+0000","last_became_peered":"2020-11-20T11:32:18.926981+0000","last_unstale":"2020-11-20T11:32:25.474417+0000","last_undegraded":"2020-11-20T11:32:25.474417+0000","last_fullsized":"2020-11-20T11:32:25.474417+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"24'8","reported_seq":"21","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.492278+0000","last_change":"2020-11-20T11:32:22.088128+0000","last_active":"2020-11-20T11:32:25.492278+0000","last_peered":"2020-11-20T11:32:25.492278+0000","last_clean":"2020-11-20T11:32:25.492278+0000","last_became_active":"2020-11-20T11:32:22.087723+0000","last_became_peered":"2020-11-20T11:32:22.087723+0000","last_unstale":"2020-11-20T11:32:25.492278+0000","last_undegraded":"2020-11-20T11:32:25.492278+0000","last_fullsized":"2020-11-20T11:32:25.492278+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.495311+0000","last_change":"2020-11-20T11:32:25.492514+0000","last_active":"2020-11-20T11:32:25.495311+0000","last_peered":"2020-11-20T11:32:25.495311+0000","last_clean":"2020-11-20T11:32:25.495311+0000","last_became_active":"2020-11-20T11:32:25.491657+0000","last_became_peered":"2020-11-20T11:32:25.491657+0000","last_unstale":"2020-11-20T11:32:25.495311+0000","last_undegraded":"2020-11-20T11:32:25.495311+0000","last_fullsized":"2020-11-20T11:32:25.495311+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"16","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.479536+0000","last_change":"2020-11-20T11:32:18.927044+0000","last_active":"2020-11-20T11:32:25.479536+0000","last_peered":"2020-11-20T11:32:25.479536+0000","last_clean":"2020-11-20T11:32:25.479536+0000","last_became_active":"2020-11-20T11:32:18.926711+0000","last_became_peered":"2020-11-20T11:32:18.926711+0000","last_unstale":"2020-11-20T11:32:25.479536+0000","last_undegraded":"2020-11-20T11:32:25.479536+0000","last_fullsized":"2020-11-20T11:32:25.479536+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'2","reported_seq":"20","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.488044+0000","last_change":"2020-11-20T11:32:14.446049+0000","last_active":"2020-11-20T11:32:25.488044+0000","last_peered":"2020-11-20T11:32:25.488044+0000","last_clean":"2020-11-20T11:32:25.488044+0000","last_became_active":"2020-11-20T11:32:14.445477+0000","last_became_peered":"2020-11-20T11:32:14.445477+0000","last_unstale":"2020-11-20T11:32:25.488044+0000","last_undegraded":"2020-11-20T11:32:25.488044+0000","last_fullsized":"2020-11-20T11:32:25.488044+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:10.515989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:10.515989+0000","last_clean_scrub_stamp":"2020-11-20T11:32:10.515989+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"24'1","reported_seq":"14","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.488459+0000","last_change":"2020-11-20T11:32:22.088160+0000","last_active":"2020-11-20T11:32:25.488459+0000","last_peered":"2020-11-20T11:32:25.488459+0000","last_clean":"2020-11-20T11:32:25.488459+0000","last_became_active":"2020-11-20T11:32:22.087777+0000","last_became_peered":"2020-11-20T11:32:22.087777+0000","last_unstale":"2020-11-20T11:32:25.488459+0000","last_undegraded":"2020-11-20T11:32:25.488459+0000","last_fullsized":"2020-11-20T11:32:25.488459+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.493247+0000","last_change":"2020-11-20T11:32:25.491667+0000","last_active":"2020-11-20T11:32:25.493247+0000","last_peered":"2020-11-20T11:32:25.493247+0000","last_clean":"2020-11-20T11:32:25.493247+0000","last_became_active":"2020-11-20T11:32:25.491100+0000","last_became_peered":"2020-11-20T11:32:25.491100+0000","last_unstale":"2020-11-20T11:32:25.493247+0000","last_undegraded":"2020-11-20T11:32:25.493247+0000","last_fullsized":"2020-11-20T11:32:25.493247+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"15","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.488045+0000","last_change":"2020-11-20T11:32:18.924600+0000","last_active":"2020-11-20T11:32:25.488045+0000","last_peered":"2020-11-20T11:32:25.488045+0000","last_clean":"2020-11-20T11:32:25.488045+0000","last_became_active":"2020-11-20T11:32:18.924324+0000","last_became_peered":"2020-11-20T11:32:18.924324+0000","last_unstale":"2020-11-20T11:32:25.488045+0000","last_undegraded":"2020-11-20T11:32:25.488045+0000","last_fullsized":"2020-11-20T11:32:25.488045+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"13","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.476386+0000","last_change":"2020-11-20T11:32:22.088716+0000","last_active":"2020-11-20T11:32:25.476386+0000","last_peered":"2020-11-20T11:32:25.476386+0000","last_clean":"2020-11-20T11:32:25.476386+0000","last_became_active":"2020-11-20T11:32:22.088414+0000","last_became_peered":"2020-11-20T11:32:22.088414+0000","last_unstale":"2020-11-20T11:32:25.476386+0000","last_undegraded":"2020-11-20T11:32:25.476386+0000","last_fullsized":"2020-11-20T11:32:25.476386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.493332+0000","last_change":"2020-11-20T11:32:25.491507+0000","last_active":"2020-11-20T11:32:25.493332+0000","last_peered":"2020-11-20T11:32:25.493332+0000","last_clean":"2020-11-20T11:32:25.493332+0000","last_became_active":"2020-11-20T11:32:25.490917+0000","last_became_peered":"2020-11-20T11:32:25.490917+0000","last_unstale":"2020-11-20T11:32:25.493332+0000","last_undegraded":"2020-11-20T11:32:25.493332+0000","last_fullsized":"2020-11-20T11:32:25.493332+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"15","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.488121+0000","last_change":"2020-11-20T11:32:19.482719+0000","last_active":"2020-11-20T11:32:25.488121+0000","last_peered":"2020-11-20T11:32:25.488121+0000","last_clean":"2020-11-20T11:32:25.488121+0000","last_became_active":"2020-11-20T11:32:19.482359+0000","last_became_peered":"2020-11-20T11:32:19.482359+0000","last_unstale":"2020-11-20T11:32:25.488121+0000","last_undegraded":"2020-11-20T11:32:25.488121+0000","last_fullsized":"2020-11-20T11:32:25.488121+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"25'11","reported_seq":"24","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:32:43.119744+0000","last_change":"2020-11-20T11:32:22.093099+0000","last_active":"2020-11-20T11:32:43.119744+0000","last_peered":"2020-11-20T11:32:43.119744+0000","last_clean":"2020-11-20T11:32:43.119744+0000","last_became_active":"2020-11-20T11:32:22.092619+0000","last_became_peered":"2020-11-20T11:32:22.092619+0000","last_unstale":"2020-11-20T11:32:43.119744+0000","last_undegraded":"2020-11-20T11:32:43.119744+0000","last_fullsized":"2020-11-20T11:32:43.119744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2760,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.493059+0000","last_change":"2020-11-20T11:32:25.492150+0000","last_active":"2020-11-20T11:32:25.493059+0000","last_peered":"2020-11-20T11:32:25.493059+0000","last_clean":"2020-11-20T11:32:25.493059+0000","last_became_active":"2020-11-20T11:32:25.491099+0000","last_became_peered":"2020-11-20T11:32:25.491099+0000","last_unstale":"2020-11-20T11:32:25.493059+0000","last_undegraded":"2020-11-20T11:32:25.493059+0000","last_fullsized":"2020-11-20T11:32:25.493059+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"24'6","reported_seq":"19","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.492963+0000","last_change":"2020-11-20T11:32:22.461697+0000","last_active":"2020-11-20T11:32:25.492963+0000","last_peered":"2020-11-20T11:32:25.492963+0000","last_clean":"2020-11-20T11:32:25.492963+0000","last_became_active":"2020-11-20T11:32:22.461331+0000","last_became_peered":"2020-11-20T11:32:22.461331+0000","last_unstale":"2020-11-20T11:32:25.492963+0000","last_undegraded":"2020-11-20T11:32:25.492963+0000","last_fullsized":"2020-11-20T11:32:25.492963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"16","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.480381+0000","last_change":"2020-11-20T11:32:18.923886+0000","last_active":"2020-11-20T11:32:25.480381+0000","last_peered":"2020-11-20T11:32:25.480381+0000","last_clean":"2020-11-20T11:32:25.480381+0000","last_became_active":"2020-11-20T11:32:18.923448+0000","last_became_peered":"2020-11-20T11:32:18.923448+0000","last_unstale":"2020-11-20T11:32:25.480381+0000","last_undegraded":"2020-11-20T11:32:25.480381+0000","last_fullsized":"2020-11-20T11:32:25.480381+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":"12","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.498234+0000","last_change":"2020-11-20T11:32:25.491585+0000","last_active":"2020-11-20T11:32:25.498234+0000","last_peered":"2020-11-20T11:32:25.498234+0000","last_clean":"2020-11-20T11:32:25.498234+0000","last_became_active":"2020-11-20T11:32:25.491038+0000","last_became_peered":"2020-11-20T11:32:25.491038+0000","last_unstale":"2020-11-20T11:32:25.498234+0000","last_undegraded":"2020-11-20T11:32:25.498234+0000","last_fullsized":"2020-11-20T11:32:25.498234+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"15","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.473625+0000","last_change":"2020-11-20T11:32:18.924052+0000","last_active":"2020-11-20T11:32:25.473625+0000","last_peered":"2020-11-20T11:32:25.473625+0000","last_clean":"2020-11-20T11:32:25.473625+0000","last_became_active":"2020-11-20T11:32:18.923668+0000","last_became_peered":"2020-11-20T11:32:18.923668+0000","last_unstale":"2020-11-20T11:32:25.473625+0000","last_undegraded":"2020-11-20T11:32:25.473625+0000","last_fullsized":"2020-11-20T11:32:25.473625+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"25'4","reported_seq":"17","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:32:27.326193+0000","last_change":"2020-11-20T11:32:22.093019+0000","last_active":"2020-11-20T11:32:27.326193+0000","last_peered":"2020-11-20T11:32:27.326193+0000","last_clean":"2020-11-20T11:32:27.326193+0000","last_became_active":"2020-11-20T11:32:22.092604+0000","last_became_peered":"2020-11-20T11:32:22.092604+0000","last_unstale":"2020-11-20T11:32:27.326193+0000","last_undegraded":"2020-11-20T11:32:27.326193+0000","last_fullsized":"2020-11-20T11:32:27.326193+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":8},{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":3529,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":6,"up_from":14,"seq":60129542153,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5064,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366776,"statfs":{"total":96636764160,"available":96631578624,"internally_reserved":0,"allocated":262144,"data_stored":107280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.173}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3109999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.337}]}]},{"osd":3,"up_from":14,"seq":60129542154,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":278528,"data_stored":108015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0760000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.752}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4530000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4239999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2569999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5132,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366708,"statfs":{"total":96636764160,"available":96631508992,"internally_reserved":0,"allocated":266240,"data_stored":107314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4239999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2430000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3779999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4610000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5196,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366644,"statfs":{"total":96636764160,"available":96631443456,"internally_reserved":0,"allocated":266240,"data_stored":107314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.044}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93300000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.155}]}]},{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":270336,"data_stored":107855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0489999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85999999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99399999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5136,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366704,"statfs":{"total":96636764160,"available":96631504896,"internally_reserved":0,"allocated":270336,"data_stored":107440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93999999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.052}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.079}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.196}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.171}]}]},{"osd":1,"up_from":14,"seq":60129542152,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5204,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366636,"statfs":{"total":96636764160,"available":96631435264,"internally_reserved":0,"allocated":274432,"data_stored":110040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.028}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.159}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2390000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542153,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5140,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366700,"statfs":{"total":96636764160,"available":96631500800,"internally_reserved":0,"allocated":274432,"data_stored":110040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.755}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0389999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.091}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2270000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3049999999999999}]}]}]}} 2020-11-20T11:32:45.746 INFO:tasks.mon_thrash.ceph_manager:clean! 2020-11-20T11:32:45.747 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T11:32:45.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.858+0000 7f8cb2e06700 1 Processor -- start 2020-11-20T11:32:45.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.858+0000 7f8cb2e06700 1 -- start start 2020-11-20T11:32:45.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.859+0000 7f8cb2e06700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8cac0aaab0 0x7f8cac13ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:45.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.859+0000 7f8cb2e06700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f8cac0aae90 0x7f8cac1381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:45.870 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.859+0000 7f8cb2e06700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cac13e600 con 0x7f8cac13aeb0 2020-11-20T11:32:45.870 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.859+0000 7f8cb2e06700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cac13e8a0 con 0x7f8cac0aae90 2020-11-20T11:32:45.870 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.859+0000 7f8cb2e06700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac0ab6d0 con 0x7f8cac0aaab0 2020-11-20T11:32:45.870 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.859+0000 7f8cb0ba2700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8cac0aaab0 0x7f8cac13ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:45.871 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.859+0000 7f8cabfff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f8cac0aae90 0x7f8cac1381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:45.871 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.859+0000 7f8cb13a3700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f8cac13aeb0 0x7f8cac0a8c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:33678/0 (socket says 172.21.15.23:33678) 2020-11-20T11:32:45.871 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.859+0000 7f8cabfff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f8cac0aae90 0x7f8cac1381a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:34348/0 (socket says 172.21.15.23:34348) 2020-11-20T11:32:45.871 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.859+0000 7f8cb13a3700 1 -- 172.21.15.23:0/1162379108 learned_addr learned my addr 172.21.15.23:0/1162379108 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:45.871 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.860+0000 7f8cab7fe700 1 -- 172.21.15.23:0/1162379108 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3919311802 0 0) 0x7f8cac13e600 con 0x7f8cac13aeb0 2020-11-20T11:32:45.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.860+0000 7f8cab7fe700 1 -- 172.21.15.23:0/1162379108 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c90003280 con 0x7f8cac13aeb0 2020-11-20T11:32:45.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.860+0000 7f8cb0ba2700 1 -- 172.21.15.23:0/1162379108 >> v2:172.21.15.155:3300/0 conn(0x7f8cac0aae90 msgr2=0x7f8cac1381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:45.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.860+0000 7f8cb0ba2700 1 --2- 172.21.15.23:0/1162379108 >> v2:172.21.15.155:3300/0 conn(0x7f8cac0aae90 0x7f8cac1381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.860+0000 7f8cb0ba2700 1 -- 172.21.15.23:0/1162379108 >> v1:172.21.15.155:6789/0 conn(0x7f8cac13aeb0 legacy=0x7f8cac0a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:45.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.860+0000 7f8cb0ba2700 1 -- 172.21.15.23:0/1162379108 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cac0a9370 con 0x7f8cac0aaab0 2020-11-20T11:32:45.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.860+0000 7f8cabfff700 1 --2- 172.21.15.23:0/1162379108 >> v2:172.21.15.155:3300/0 conn(0x7f8cac0aae90 0x7f8cac1381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:32:45.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.860+0000 7f8cb0ba2700 1 --2- 172.21.15.23:0/1162379108 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8cac0aaab0 0x7f8cac13ed40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f8ca0006fd0 tx=0x7f8ca0006e30).ready entity=mon.2 client_cookie=c3ab4b513711919c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:45.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.860+0000 7f8cab7fe700 1 -- 172.21.15.23:0/1162379108 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0009630 con 0x7f8cac0aaab0 2020-11-20T11:32:45.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.860+0000 7f8cab7fe700 1 -- 172.21.15.23:0/1162379108 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca0000e90 con 0x7f8cac0aaab0 2020-11-20T11:32:45.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.860+0000 7f8cab7fe700 1 -- 172.21.15.23:0/1162379108 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca00099a0 con 0x7f8cac0aaab0 2020-11-20T11:32:45.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.861+0000 7f8cb2e06700 1 -- 172.21.15.23:0/1162379108 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8cac0aaab0 msgr2=0x7f8cac13ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:45.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.861+0000 7f8cb2e06700 1 --2- 172.21.15.23:0/1162379108 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8cac0aaab0 0x7f8cac13ed40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f8ca0006fd0 tx=0x7f8ca0006e30).stop 2020-11-20T11:32:45.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.861+0000 7f8cb2e06700 1 -- 172.21.15.23:0/1162379108 shutdown_connections 2020-11-20T11:32:45.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.861+0000 7f8cb2e06700 1 --2- 172.21.15.23:0/1162379108 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8cac0aaab0 0x7f8cac13ed40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.861+0000 7f8cb2e06700 1 --2- 172.21.15.23:0/1162379108 >> v2:172.21.15.155:3300/0 conn(0x7f8cac0aae90 0x7f8cac1381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.861+0000 7f8cb2e06700 1 -- 172.21.15.23:0/1162379108 >> 172.21.15.23:0/1162379108 conn(0x7f8cac0678e0 msgr2=0x7f8cac067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:45.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.861+0000 7f8cb2e06700 1 -- 172.21.15.23:0/1162379108 shutdown_connections 2020-11-20T11:32:45.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.861+0000 7f8cb2e06700 1 -- 172.21.15.23:0/1162379108 wait complete. 2020-11-20T11:32:45.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.862+0000 7f8cb2e06700 1 Processor -- start 2020-11-20T11:32:45.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.862+0000 7f8cb2e06700 1 -- start start 2020-11-20T11:32:45.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.862+0000 7f8cb2e06700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8cac0aaab0 0x7f8cac09ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:45.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.862+0000 7f8cb2e06700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cac0aae90 0x7f8cac0b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:45.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.862+0000 7f8cb2e06700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8cac14e470 0x7f8cac13c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:45.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.863+0000 7f8cb2e06700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac14d860 con 0x7f8cac0aae90 2020-11-20T11:32:45.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.863+0000 7f8cb2e06700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac13d240 con 0x7f8cac0aaab0 2020-11-20T11:32:45.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.863+0000 7f8cb2e06700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac13c7a0 con 0x7f8cac14e470 2020-11-20T11:32:45.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.863+0000 7f8cabfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cac0aae90 0x7f8cac0b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:45.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.863+0000 7f8cabfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cac0aae90 0x7f8cac0b0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41458/0 (socket says 172.21.15.23:41458) 2020-11-20T11:32:45.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.863+0000 7f8cabfff700 1 -- 172.21.15.23:0/3939861149 learned_addr learned my addr 172.21.15.23:0/3939861149 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:45.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.863+0000 7f8cb0ba2700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8cac0aaab0 0x7f8cac09ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:45.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.863+0000 7f8cb13a3700 1 --2- 172.21.15.23:0/3939861149 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8cac14e470 0x7f8cac13c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:45.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.863+0000 7f8cabfff700 1 -- 172.21.15.23:0/3939861149 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8cac14e470 msgr2=0x7f8cac13c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:45.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.863+0000 7f8cabfff700 1 --2- 172.21.15.23:0/3939861149 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8cac14e470 0x7f8cac13c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.863+0000 7f8cabfff700 1 -- 172.21.15.23:0/3939861149 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8cac0aaab0 msgr2=0x7f8cac09ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:45.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.863+0000 7f8cabfff700 1 --2- 172.21.15.23:0/3939861149 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8cac0aaab0 0x7f8cac09ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:45.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.863+0000 7f8cabfff700 1 -- 172.21.15.23:0/3939861149 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cac14da90 con 0x7f8cac0aae90 2020-11-20T11:32:45.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.863+0000 7f8cabfff700 1 --2- 172.21.15.23:0/3939861149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cac0aae90 0x7f8cac0b0b40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f8c9800ecf0 tx=0x7f8c98006e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:45.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.863+0000 7f8ca8ff9700 1 -- 172.21.15.23:0/3939861149 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9800c980 con 0x7f8cac0aae90 2020-11-20T11:32:45.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.863+0000 7f8ca8ff9700 1 -- 172.21.15.23:0/3939861149 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c9800cae0 con 0x7f8cac0aae90 2020-11-20T11:32:45.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.863+0000 7f8cb2e06700 1 -- 172.21.15.23:0/3939861149 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cac14dd30 con 0x7f8cac0aae90 2020-11-20T11:32:45.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.863+0000 7f8ca8ff9700 1 -- 172.21.15.23:0/3939861149 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9800ce60 con 0x7f8cac0aae90 2020-11-20T11:32:45.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.863+0000 7f8cb2e06700 1 -- 172.21.15.23:0/3939861149 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cac0a81b0 con 0x7f8cac0aae90 2020-11-20T11:32:45.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.864+0000 7f8cb2e06700 1 -- 172.21.15.23:0/3939861149 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cac0a9080 con 0x7f8cac0aae90 2020-11-20T11:32:45.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.864+0000 7f8ca8ff9700 1 -- 172.21.15.23:0/3939861149 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8c98012600 con 0x7f8cac0aae90 2020-11-20T11:32:45.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.865+0000 7f8ca8ff9700 1 --2- 172.21.15.23:0/3939861149 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8c84056d60 0x7f8c84059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:45.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.865+0000 7f8ca8ff9700 1 -- 172.21.15.23:0/3939861149 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f8c98070520 con 0x7f8cac0aae90 2020-11-20T11:32:45.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.865+0000 7f8cb0ba2700 1 --2- 172.21.15.23:0/3939861149 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8c84056d60 0x7f8c84059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:45.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.865+0000 7f8cb0ba2700 1 --2- 172.21.15.23:0/3939861149 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8c84056d60 0x7f8c84059220 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f8ca0006e70 tx=0x7f8ca0015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:45.882 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:45.869+0000 7f8ca8ff9700 1 -- 172.21.15.23:0/3939861149 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8c98045810 con 0x7f8cac0aae90 2020-11-20T11:32:46.021 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.018+0000 7f8cb2e06700 1 -- 172.21.15.23:0/3939861149 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8cac05c7d0 con 0x7f8cac0aae90 2020-11-20T11:32:46.022 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.019+0000 7f8ca8ff9700 1 -- 172.21.15.23:0/3939861149 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17087 (secure 0 0 0) 0x7f8c98048e30 con 0x7f8cac0aae90 2020-11-20T11:32:46.022 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:32:46.023 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":25,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","created":"2020-11-20T11:32:06.136679+0000","modified":"2020-11-20T11:32:26.470927+0000","last_up_change":"2020-11-20T11:32:13.536194+0000","last_in_change":"2020-11-20T11:32:13.536194+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T11:32:09.559868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T11:32:17.000782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T11:32:20.423341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T11:32:23.126273+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e9d219f5-e40e-4b2b-9468-deb9c4074a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":23083},{"type":"v1","addr":"172.21.15.23:6801","nonce":23083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":23083},{"type":"v1","addr":"172.21.15.23:6803","nonce":23083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":23083},{"type":"v1","addr":"172.21.15.23:6807","nonce":23083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":23083},{"type":"v1","addr":"172.21.15.23:6805","nonce":23083}]},"public_addr":"172.21.15.23:6801/23083","cluster_addr":"172.21.15.23:6803/23083","heartbeat_back_addr":"172.21.15.23:6807/23083","heartbeat_front_addr":"172.21.15.23:6805/23083","state":["exists","up"]},{"osd":1,"uuid":"2328adde-2108-4fe0-8138-f0d096a496a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":23099},{"type":"v1","addr":"172.21.15.23:6817","nonce":23099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":23099},{"type":"v1","addr":"172.21.15.23:6819","nonce":23099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":23099},{"type":"v1","addr":"172.21.15.23:6824","nonce":23099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":23099},{"type":"v1","addr":"172.21.15.23:6821","nonce":23099}]},"public_addr":"172.21.15.23:6817/23099","cluster_addr":"172.21.15.23:6819/23099","heartbeat_back_addr":"172.21.15.23:6824/23099","heartbeat_front_addr":"172.21.15.23:6821/23099","state":["exists","up"]},{"osd":2,"uuid":"13367708-59a2-4363-8a3a-ab0cb97145ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":23084},{"type":"v1","addr":"172.21.15.23:6809","nonce":23084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":23084},{"type":"v1","addr":"172.21.15.23:6811","nonce":23084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":23084},{"type":"v1","addr":"172.21.15.23:6815","nonce":23084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":23084},{"type":"v1","addr":"172.21.15.23:6813","nonce":23084}]},"public_addr":"172.21.15.23:6809/23084","cluster_addr":"172.21.15.23:6811/23084","heartbeat_back_addr":"172.21.15.23:6815/23084","heartbeat_front_addr":"172.21.15.23:6813/23084","state":["exists","up"]},{"osd":3,"uuid":"1f147dd7-9289-47b0-ad1c-4088618d6faa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6823","nonce":23114},{"type":"v1","addr":"172.21.15.23:6825","nonce":23114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":23114},{"type":"v1","addr":"172.21.15.23:6827","nonce":23114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":23114},{"type":"v1","addr":"172.21.15.23:6831","nonce":23114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":23114},{"type":"v1","addr":"172.21.15.23:6829","nonce":23114}]},"public_addr":"172.21.15.23:6825/23114","cluster_addr":"172.21.15.23:6827/23114","heartbeat_back_addr":"172.21.15.23:6831/23114","heartbeat_front_addr":"172.21.15.23:6829/23114","state":["exists","up"]},{"osd":4,"uuid":"d008b039-4bc5-4135-9678-1511e58efc29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":23280},{"type":"v1","addr":"172.21.15.155:6801","nonce":23280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":23280},{"type":"v1","addr":"172.21.15.155:6803","nonce":23280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":23280},{"type":"v1","addr":"172.21.15.155:6807","nonce":23280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":23280},{"type":"v1","addr":"172.21.15.155:6805","nonce":23280}]},"public_addr":"172.21.15.155:6801/23280","cluster_addr":"172.21.15.155:6803/23280","heartbeat_back_addr":"172.21.15.155:6807/23280","heartbeat_front_addr":"172.21.15.155:6805/23280","state":["exists","up"]},{"osd":5,"uuid":"56715e6f-cf6e-44a6-9e39-93ab1224095f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":23298},{"type":"v1","addr":"172.21.15.155:6817","nonce":23298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":23298},{"type":"v1","addr":"172.21.15.155:6819","nonce":23298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":23298},{"type":"v1","addr":"172.21.15.155:6823","nonce":23298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":23298},{"type":"v1","addr":"172.21.15.155:6821","nonce":23298}]},"public_addr":"172.21.15.155:6817/23298","cluster_addr":"172.21.15.155:6819/23298","heartbeat_back_addr":"172.21.15.155:6823/23298","heartbeat_front_addr":"172.21.15.155:6821/23298","state":["exists","up"]},{"osd":6,"uuid":"db1e61ca-8e14-4a1f-8607-6e01b148d61c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":23307},{"type":"v1","addr":"172.21.15.155:6825","nonce":23307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":23307},{"type":"v1","addr":"172.21.15.155:6827","nonce":23307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":23307},{"type":"v1","addr":"172.21.15.155:6831","nonce":23307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":23307},{"type":"v1","addr":"172.21.15.155:6829","nonce":23307}]},"public_addr":"172.21.15.155:6825/23307","cluster_addr":"172.21.15.155:6827/23307","heartbeat_back_addr":"172.21.15.155:6831/23307","heartbeat_front_addr":"172.21.15.155:6829/23307","state":["exists","up"]},{"osd":7,"uuid":"d44419be-ef89-462c-8cf1-3c6db2a6944b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":23281},{"type":"v1","addr":"172.21.15.155:6809","nonce":23281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":23281},{"type":"v1","addr":"172.21.15.155:6811","nonce":23281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":23281},{"type":"v1","addr":"172.21.15.155:6815","nonce":23281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":23281},{"type":"v1","addr":"172.21.15.155:6813","nonce":23281}]},"public_addr":"172.21.15.155:6809/23281","cluster_addr":"172.21.15.155:6811/23281","heartbeat_back_addr":"172.21.15.155:6815/23281","heartbeat_front_addr":"172.21.15.155:6813/23281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.486702+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.509396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.533414+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.509489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.467600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.513068+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.536636+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.531772+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T11:32:46.024 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.021+0000 7f8cb2e06700 1 -- 172.21.15.23:0/3939861149 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8c84056d60 msgr2=0x7f8c84059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:46.024 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.021+0000 7f8cb2e06700 1 --2- 172.21.15.23:0/3939861149 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8c84056d60 0x7f8c84059220 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f8ca0006e70 tx=0x7f8ca0015000).stop 2020-11-20T11:32:46.024 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.021+0000 7f8cb2e06700 1 -- 172.21.15.23:0/3939861149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cac0aae90 msgr2=0x7f8cac0b0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:46.024 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.021+0000 7f8cb2e06700 1 --2- 172.21.15.23:0/3939861149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cac0aae90 0x7f8cac0b0b40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f8c9800ecf0 tx=0x7f8c98006e30).stop 2020-11-20T11:32:46.025 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.021+0000 7f8cb2e06700 1 -- 172.21.15.23:0/3939861149 shutdown_connections 2020-11-20T11:32:46.025 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.021+0000 7f8cb2e06700 1 --2- 172.21.15.23:0/3939861149 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8c84056d60 0x7f8c84059220 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:46.025 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.021+0000 7f8cb2e06700 1 --2- 172.21.15.23:0/3939861149 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8cac14e470 0x7f8cac13c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:46.025 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.021+0000 7f8cb2e06700 1 --2- 172.21.15.23:0/3939861149 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8cac0aaab0 0x7f8cac09ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:46.026 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.021+0000 7f8cb2e06700 1 --2- 172.21.15.23:0/3939861149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cac0aae90 0x7f8cac0b0b40 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:46.026 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.021+0000 7f8cb2e06700 1 -- 172.21.15.23:0/3939861149 >> 172.21.15.23:0/3939861149 conn(0x7f8cac0678e0 msgr2=0x7f8cac0acf60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:46.026 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.021+0000 7f8cb2e06700 1 -- 172.21.15.23:0/3939861149 shutdown_connections 2020-11-20T11:32:46.026 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.021+0000 7f8cb2e06700 1 -- 172.21.15.23:0/3939861149 wait complete. 2020-11-20T11:32:46.036 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-11-20T11:32:09.559868+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-11-20T11:32:17.000782+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2020-11-20T11:32:20.423341+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2020-11-20T11:32:23.126273+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2020-11-20T11:32:46.037 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2020-11-20T11:32:46.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.151+0000 7f12fee22700 1 Processor -- start 2020-11-20T11:32:46.155 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.152+0000 7f12fee22700 1 -- start start 2020-11-20T11:32:46.157 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.152+0000 7f12fee22700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f12f80aaab0 0x7f12f813ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:46.157 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.152+0000 7f12fee22700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12f80aae90 0x7f12f81381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:46.157 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.152+0000 7f12fee22700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f12f813aeb0 0x7f12f80a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:46.157 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.152+0000 7f12fee22700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12f813e600 con 0x7f12f813aeb0 2020-11-20T11:32:46.157 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.152+0000 7f12fee22700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12f813e8a0 con 0x7f12f80aaab0 2020-11-20T11:32:46.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.152+0000 7f12fee22700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f80ab6d0 con 0x7f12f80aae90 2020-11-20T11:32:46.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.152+0000 7f12fd3bf700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f12f813aeb0 0x7f12f80a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:46.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.152+0000 7f12fd3bf700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f12f813aeb0 0x7f12f80a8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41468/0 (socket says 172.21.15.23:41468) 2020-11-20T11:32:46.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.152+0000 7f12fd3bf700 1 -- 172.21.15.23:0/3420856788 learned_addr learned my addr 172.21.15.23:0/3420856788 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:46.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.153+0000 7f12f7fff700 1 --2- 172.21.15.23:0/3420856788 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12f80aae90 0x7f12f81381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:46.159 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.153+0000 7f12fcbbe700 1 --2- 172.21.15.23:0/3420856788 >> v2:172.21.15.155:3300/0 conn(0x7f12f80aaab0 0x7f12f813ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:46.159 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.153+0000 7f12fd3bf700 1 -- 172.21.15.23:0/3420856788 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12f80aae90 msgr2=0x7f12f81381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:46.159 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.153+0000 7f12fd3bf700 1 --2- 172.21.15.23:0/3420856788 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12f80aae90 0x7f12f81381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:46.159 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.153+0000 7f12fd3bf700 1 -- 172.21.15.23:0/3420856788 >> v2:172.21.15.155:3300/0 conn(0x7f12f80aaab0 msgr2=0x7f12f813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:46.159 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.153+0000 7f12fd3bf700 1 --2- 172.21.15.23:0/3420856788 >> v2:172.21.15.155:3300/0 conn(0x7f12f80aaab0 0x7f12f813ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:46.160 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.153+0000 7f12fd3bf700 1 -- 172.21.15.23:0/3420856788 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f813f230 con 0x7f12f813aeb0 2020-11-20T11:32:46.160 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.153+0000 7f12fd3bf700 1 --2- 172.21.15.23:0/3420856788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12f813aeb0 0x7f12f80a8c60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f12f80b23d0 tx=0x7f12f00092d0).ready entity=mon.0 client_cookie=6c4618492ab21768 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:46.160 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.153+0000 7f12f77fe700 1 -- 172.21.15.23:0/3420856788 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f000bbf0 con 0x7f12f813aeb0 2020-11-20T11:32:46.160 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.153+0000 7f12f77fe700 1 -- 172.21.15.23:0/3420856788 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f000bd50 con 0x7f12f813aeb0 2020-11-20T11:32:46.160 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.153+0000 7f12f77fe700 1 -- 172.21.15.23:0/3420856788 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f0011610 con 0x7f12f813aeb0 2020-11-20T11:32:46.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.153+0000 7f12fee22700 1 -- 172.21.15.23:0/3420856788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12f813aeb0 msgr2=0x7f12f80a8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:46.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.153+0000 7f12fee22700 1 --2- 172.21.15.23:0/3420856788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12f813aeb0 0x7f12f80a8c60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f12f80b23d0 tx=0x7f12f00092d0).stop 2020-11-20T11:32:46.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.153+0000 7f12fee22700 1 -- 172.21.15.23:0/3420856788 shutdown_connections 2020-11-20T11:32:46.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.153+0000 7f12fee22700 1 --2- 172.21.15.23:0/3420856788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12f813aeb0 0x7f12f80a8c60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:46.162 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.153+0000 7f12fee22700 1 --2- 172.21.15.23:0/3420856788 >> v2:172.21.15.155:3300/0 conn(0x7f12f80aaab0 0x7f12f813ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:46.163 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.153+0000 7f12fee22700 1 --2- 172.21.15.23:0/3420856788 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12f80aae90 0x7f12f81381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:46.163 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.153+0000 7f12fee22700 1 -- 172.21.15.23:0/3420856788 >> 172.21.15.23:0/3420856788 conn(0x7f12f80678e0 msgr2=0x7f12f8067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:46.163 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.154+0000 7f12fee22700 1 -- 172.21.15.23:0/3420856788 shutdown_connections 2020-11-20T11:32:46.163 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.154+0000 7f12fee22700 1 -- 172.21.15.23:0/3420856788 wait complete. 2020-11-20T11:32:46.167 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.154+0000 7f12fee22700 1 Processor -- start 2020-11-20T11:32:46.167 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.155+0000 7f12fee22700 1 -- start start 2020-11-20T11:32:46.167 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.155+0000 7f12fee22700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12f80aaab0 0x7f12f80a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:46.168 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.155+0000 7f12fee22700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12f80aae90 0x7f12f80a66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:46.168 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.155+0000 7f12fee22700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12f80a5c80 0x7f12f80a6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:46.168 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.155+0000 7f12fee22700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f809e0c0 con 0x7f12f80a5c80 2020-11-20T11:32:46.168 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.155+0000 7f12fee22700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f80a52f0 con 0x7f12f80aae90 2020-11-20T11:32:46.168 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.155+0000 7f12fee22700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f80a8400 con 0x7f12f80aaab0 2020-11-20T11:32:46.169 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.155+0000 7f12fd3bf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12f80a5c80 0x7f12f80a6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:46.169 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.155+0000 7f12fd3bf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12f80a5c80 0x7f12f80a6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41474/0 (socket says 172.21.15.23:41474) 2020-11-20T11:32:46.169 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.155+0000 7f12fd3bf700 1 -- 172.21.15.23:0/2080043710 learned_addr learned my addr 172.21.15.23:0/2080043710 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:46.169 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.156+0000 7f12fcbbe700 1 --2- 172.21.15.23:0/2080043710 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12f80aaab0 0x7f12f80a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:46.170 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.156+0000 7f12f7fff700 1 --2- 172.21.15.23:0/2080043710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12f80aae90 0x7f12f80a66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:46.170 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.156+0000 7f12fd3bf700 1 -- 172.21.15.23:0/2080043710 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12f80aaab0 msgr2=0x7f12f80a5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:46.170 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.156+0000 7f12fd3bf700 1 --2- 172.21.15.23:0/2080043710 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12f80aaab0 0x7f12f80a5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:46.170 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.156+0000 7f12fd3bf700 1 -- 172.21.15.23:0/2080043710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12f80aae90 msgr2=0x7f12f80a66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:46.170 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.156+0000 7f12fd3bf700 1 --2- 172.21.15.23:0/2080043710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12f80aae90 0x7f12f80a66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:46.171 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.156+0000 7f12fd3bf700 1 -- 172.21.15.23:0/2080043710 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f8149fb0 con 0x7f12f80a5c80 2020-11-20T11:32:46.171 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.156+0000 7f12fcbbe700 1 --2- 172.21.15.23:0/2080043710 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12f80aaab0 0x7f12f80a5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:46.171 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.156+0000 7f12fd3bf700 1 --2- 172.21.15.23:0/2080043710 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12f80a5c80 0x7f12f80a6bd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f12f0011000 tx=0x7f12f0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:46.171 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.156+0000 7f12f4ff9700 1 -- 172.21.15.23:0/2080043710 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f0009ea0 con 0x7f12f80a5c80 2020-11-20T11:32:46.171 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.156+0000 7f12f4ff9700 1 -- 172.21.15.23:0/2080043710 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f000bec0 con 0x7f12f80a5c80 2020-11-20T11:32:46.172 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.156+0000 7f12fee22700 1 -- 172.21.15.23:0/2080043710 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12f809ddb0 con 0x7f12f80a5c80 2020-11-20T11:32:46.172 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.156+0000 7f12f4ff9700 1 -- 172.21.15.23:0/2080043710 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f0010e10 con 0x7f12f80a5c80 2020-11-20T11:32:46.172 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.156+0000 7f12fee22700 1 -- 172.21.15.23:0/2080043710 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f814a5f0 con 0x7f12f80a5c80 2020-11-20T11:32:46.172 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.157+0000 7f12fee22700 1 -- 172.21.15.23:0/2080043710 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12f805c7d0 con 0x7f12f80a5c80 2020-11-20T11:32:46.172 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.157+0000 7f12f4ff9700 1 -- 172.21.15.23:0/2080043710 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f12f0023030 con 0x7f12f80a5c80 2020-11-20T11:32:46.173 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.157+0000 7f12f4ff9700 1 --2- 172.21.15.23:0/2080043710 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f12d0056d60 0x7f12d0059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:46.173 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.158+0000 7f12f4ff9700 1 -- 172.21.15.23:0/2080043710 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f12f00717b0 con 0x7f12f80a5c80 2020-11-20T11:32:46.174 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.158+0000 7f12fcbbe700 1 --2- 172.21.15.23:0/2080043710 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f12d0056d60 0x7f12d0059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:46.174 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.158+0000 7f12fcbbe700 1 --2- 172.21.15.23:0/2080043710 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f12d0056d60 0x7f12d0059220 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f12ec006fd0 tx=0x7f12ec008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:46.174 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.161+0000 7f12f4ff9700 1 -- 172.21.15.23:0/2080043710 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f12f0046bd0 con 0x7f12f80a5c80 2020-11-20T11:32:46.315 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.312+0000 7f12fee22700 1 -- 172.21.15.23:0/2080043710 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f12f8044710 con 0x7f12f80a5c80 2020-11-20T11:32:46.315 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.312+0000 7f12f4ff9700 1 -- 172.21.15.23:0/2080043710 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v25) v1 ==== 110+0+10 (secure 0 0 0) 0x7f12f0021070 con 0x7f12f80a5c80 2020-11-20T11:32:46.316 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2020-11-20T11:32:46.317 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.314+0000 7f12fee22700 1 -- 172.21.15.23:0/2080043710 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f12d0056d60 msgr2=0x7f12d0059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:46.317 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.314+0000 7f12fee22700 1 --2- 172.21.15.23:0/2080043710 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f12d0056d60 0x7f12d0059220 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f12ec006fd0 tx=0x7f12ec008040).stop 2020-11-20T11:32:46.317 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.314+0000 7f12fee22700 1 -- 172.21.15.23:0/2080043710 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12f80a5c80 msgr2=0x7f12f80a6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:46.318 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.314+0000 7f12fee22700 1 --2- 172.21.15.23:0/2080043710 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12f80a5c80 0x7f12f80a6bd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f12f0011000 tx=0x7f12f0016000).stop 2020-11-20T11:32:46.318 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.314+0000 7f12fee22700 1 -- 172.21.15.23:0/2080043710 shutdown_connections 2020-11-20T11:32:46.318 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.314+0000 7f12fee22700 1 --2- 172.21.15.23:0/2080043710 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12f80a5c80 0x7f12f80a6bd0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:46.318 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.314+0000 7f12fee22700 1 --2- 172.21.15.23:0/2080043710 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f12d0056d60 0x7f12d0059220 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:46.318 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.314+0000 7f12fee22700 1 --2- 172.21.15.23:0/2080043710 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12f80aaab0 0x7f12f80a5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:46.319 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.314+0000 7f12fee22700 1 --2- 172.21.15.23:0/2080043710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12f80aae90 0x7f12f80a66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:46.319 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.314+0000 7f12fee22700 1 -- 172.21.15.23:0/2080043710 >> 172.21.15.23:0/2080043710 conn(0x7f12f80678e0 msgr2=0x7f12f813eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:46.319 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.314+0000 7f12fee22700 1 -- 172.21.15.23:0/2080043710 shutdown_connections 2020-11-20T11:32:46.319 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.314+0000 7f12fee22700 1 -- 172.21.15.23:0/2080043710 wait complete. 2020-11-20T11:32:46.329 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-11-20T11:32:46.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.444+0000 7f3bb3fff700 1 Processor -- start 2020-11-20T11:32:46.497 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.444+0000 7f3bb3fff700 1 -- start start 2020-11-20T11:32:46.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.444+0000 7f3bb3fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3bb40a5be0 0x7f3bb4133bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:46.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.444+0000 7f3bb3fff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bb40a6080 con 0x7f3bb40ad7e0 2020-11-20T11:32:46.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.444+0000 7f3bb3fff700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bb40a6320 con 0x7f3bb40a8fb0 2020-11-20T11:32:46.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.444+0000 7f3bb3fff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bb40a71f0 con 0x7f3bb40a5be0 2020-11-20T11:32:46.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.444+0000 7f3bb27fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3bb40a5be0 0x7f3bb4133bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:46.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.445+0000 7f3bb2ffd700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f3bb40ad7e0 0x7f3bb40a6ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55728/0 (socket says 172.21.15.23:55728) 2020-11-20T11:32:46.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.445+0000 7f3bb27fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3bb40a5be0 0x7f3bb4133bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41478/0 (socket says 172.21.15.23:41478) 2020-11-20T11:32:46.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.445+0000 7f3bb2ffd700 1 -- 172.21.15.23:0/3538984052 learned_addr learned my addr 172.21.15.23:0/3538984052 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:46.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.445+0000 7f3bb27fc700 1 -- 172.21.15.23:0/3538984052 >> v1:172.21.15.155:6789/0 conn(0x7f3bb40a8fb0 legacy=0x7f3bb41340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:46.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.445+0000 7f3bb27fc700 1 -- 172.21.15.23:0/3538984052 >> v1:172.21.15.23:6789/0 conn(0x7f3bb40ad7e0 legacy=0x7f3bb40a6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:46.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.445+0000 7f3bb27fc700 1 -- 172.21.15.23:0/3538984052 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb40adbc0 con 0x7f3bb40a5be0 2020-11-20T11:32:46.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.445+0000 7f3bb1ffb700 1 -- 172.21.15.23:0/3538984052 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2391398558 0 0) 0x7f3bb40a6080 con 0x7f3bb40ad7e0 2020-11-20T11:32:46.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.445+0000 7f3bb27fc700 1 --2- 172.21.15.23:0/3538984052 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3bb40a5be0 0x7f3bb4133bd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f3ba8006fd0 tx=0x7f3ba8006e30).ready entity=mon.0 client_cookie=7a3d293da1ca6b8e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:46.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.445+0000 7f3bb1ffb700 1 -- 172.21.15.23:0/3538984052 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba80097e0 con 0x7f3bb40a5be0 2020-11-20T11:32:46.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.445+0000 7f3bb1ffb700 1 -- 172.21.15.23:0/3538984052 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba8000e90 con 0x7f3bb40a5be0 2020-11-20T11:32:46.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.445+0000 7f3bb1ffb700 1 -- 172.21.15.23:0/3538984052 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba8009b50 con 0x7f3bb40a5be0 2020-11-20T11:32:46.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.446+0000 7f3bb3fff700 1 -- 172.21.15.23:0/3538984052 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3bb40a5be0 msgr2=0x7f3bb4133bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:46.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.446+0000 7f3bb3fff700 1 --2- 172.21.15.23:0/3538984052 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3bb40a5be0 0x7f3bb4133bd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f3ba8006fd0 tx=0x7f3ba8006e30).stop 2020-11-20T11:32:46.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.446+0000 7f3bb3fff700 1 -- 172.21.15.23:0/3538984052 shutdown_connections 2020-11-20T11:32:46.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.446+0000 7f3bb3fff700 1 --2- 172.21.15.23:0/3538984052 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3bb40a5be0 0x7f3bb4133bd0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:46.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.446+0000 7f3bb3fff700 1 -- 172.21.15.23:0/3538984052 >> 172.21.15.23:0/3538984052 conn(0x7f3bb4061850 msgr2=0x7f3bb4061c60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:46.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.446+0000 7f3bb3fff700 1 -- 172.21.15.23:0/3538984052 shutdown_connections 2020-11-20T11:32:46.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.446+0000 7f3bb3fff700 1 -- 172.21.15.23:0/3538984052 wait complete. 2020-11-20T11:32:46.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.447+0000 7f3bb3fff700 1 Processor -- start 2020-11-20T11:32:46.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.447+0000 7f3bb3fff700 1 -- start start 2020-11-20T11:32:46.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.448+0000 7f3bb3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3bb40a5be0 0x7f3bb4131640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:46.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.448+0000 7f3bb3fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bb40a8fb0 0x7f3bb4131b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:46.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.448+0000 7f3bb3fff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3bb40ad7e0 0x7f3bb414dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:46.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.448+0000 7f3bb3fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb40a3980 con 0x7f3bb40a5be0 2020-11-20T11:32:46.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.448+0000 7f3bb3fff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb4132b00 con 0x7f3bb40a8fb0 2020-11-20T11:32:46.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.448+0000 7f3bb3fff700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb4132da0 con 0x7f3bb40ad7e0 2020-11-20T11:32:46.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.448+0000 7f3bb2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3bb40a5be0 0x7f3bb4131640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:46.508 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.448+0000 7f3bb2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3bb40a5be0 0x7f3bb4131640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41486/0 (socket says 172.21.15.23:41486) 2020-11-20T11:32:46.508 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.448+0000 7f3bb2ffd700 1 -- 172.21.15.23:0/1165838123 learned_addr learned my addr 172.21.15.23:0/1165838123 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:46.508 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.448+0000 7f3bb27fc700 1 --2- 172.21.15.23:0/1165838123 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bb40a8fb0 0x7f3bb4131b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:46.508 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.448+0000 7f3bb37fe700 1 --2- 172.21.15.23:0/1165838123 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3bb40ad7e0 0x7f3bb414dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:46.508 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.448+0000 7f3bb2ffd700 1 -- 172.21.15.23:0/1165838123 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3bb40ad7e0 msgr2=0x7f3bb414dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:46.508 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.448+0000 7f3bb2ffd700 1 --2- 172.21.15.23:0/1165838123 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3bb40ad7e0 0x7f3bb414dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:46.509 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.448+0000 7f3bb2ffd700 1 -- 172.21.15.23:0/1165838123 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bb40a8fb0 msgr2=0x7f3bb4131b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:46.509 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.448+0000 7f3bb2ffd700 1 --2- 172.21.15.23:0/1165838123 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bb40a8fb0 0x7f3bb4131b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:46.509 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.448+0000 7f3bb2ffd700 1 -- 172.21.15.23:0/1165838123 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb414d8c0 con 0x7f3bb40a5be0 2020-11-20T11:32:46.509 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.448+0000 7f3bb2ffd700 1 --2- 172.21.15.23:0/1165838123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3bb40a5be0 0x7f3bb4131640 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f3ba400c710 tx=0x7f3ba4007cd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:46.509 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.449+0000 7f3b9affd700 1 -- 172.21.15.23:0/1165838123 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba40072e0 con 0x7f3bb40a5be0 2020-11-20T11:32:46.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.449+0000 7f3b9affd700 1 -- 172.21.15.23:0/1165838123 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba4007440 con 0x7f3bb40a5be0 2020-11-20T11:32:46.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.449+0000 7f3bb3fff700 1 -- 172.21.15.23:0/1165838123 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bb41327a0 con 0x7f3bb40a5be0 2020-11-20T11:32:46.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.449+0000 7f3b9affd700 1 -- 172.21.15.23:0/1165838123 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba4005880 con 0x7f3bb40a5be0 2020-11-20T11:32:46.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.449+0000 7f3bb3fff700 1 -- 172.21.15.23:0/1165838123 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb414e250 con 0x7f3bb40a5be0 2020-11-20T11:32:46.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.449+0000 7f3bb3fff700 1 -- 172.21.15.23:0/1165838123 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bb414da50 con 0x7f3bb40a5be0 2020-11-20T11:32:46.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.450+0000 7f3b9affd700 1 -- 172.21.15.23:0/1165838123 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3ba4010070 con 0x7f3bb40a5be0 2020-11-20T11:32:46.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.450+0000 7f3b9affd700 1 --2- 172.21.15.23:0/1165838123 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3b90056d60 0x7f3b90059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:46.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.450+0000 7f3b9affd700 1 -- 172.21.15.23:0/1165838123 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f3ba406f720 con 0x7f3bb40a5be0 2020-11-20T11:32:46.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.450+0000 7f3bb27fc700 1 --2- 172.21.15.23:0/1165838123 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3b90056d60 0x7f3b90059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:46.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.451+0000 7f3bb27fc700 1 --2- 172.21.15.23:0/1165838123 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3b90056d60 0x7f3b90059220 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f3ba800b150 tx=0x7f3ba8015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:46.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.454+0000 7f3b9affd700 1 -- 172.21.15.23:0/1165838123 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3ba4044a90 con 0x7f3bb40a5be0 2020-11-20T11:32:46.606 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.603+0000 7f3bb3fff700 1 -- 172.21.15.23:0/1165838123 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f3bb4057dc0 con 0x7f3bb40a5be0 2020-11-20T11:32:46.606 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.603+0000 7f3b9affd700 1 -- 172.21.15.23:0/1165838123 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v25) v1 ==== 92+0+10 (secure 0 0 0) 0x7f3ba40480b0 con 0x7f3bb40a5be0 2020-11-20T11:32:46.606 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 8 2020-11-20T11:32:46.608 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.605+0000 7f3bb3fff700 1 -- 172.21.15.23:0/1165838123 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3b90056d60 msgr2=0x7f3b90059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:46.608 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.605+0000 7f3bb3fff700 1 --2- 172.21.15.23:0/1165838123 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3b90056d60 0x7f3b90059220 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f3ba800b150 tx=0x7f3ba8015000).stop 2020-11-20T11:32:46.609 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.605+0000 7f3bb3fff700 1 -- 172.21.15.23:0/1165838123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3bb40a5be0 msgr2=0x7f3bb4131640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:46.609 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.605+0000 7f3bb3fff700 1 --2- 172.21.15.23:0/1165838123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3bb40a5be0 0x7f3bb4131640 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f3ba400c710 tx=0x7f3ba4007cd0).stop 2020-11-20T11:32:46.609 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.605+0000 7f3bb3fff700 1 -- 172.21.15.23:0/1165838123 shutdown_connections 2020-11-20T11:32:46.609 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.605+0000 7f3bb3fff700 1 --2- 172.21.15.23:0/1165838123 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3b90056d60 0x7f3b90059220 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:46.610 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.605+0000 7f3bb3fff700 1 --2- 172.21.15.23:0/1165838123 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3bb40ad7e0 0x7f3bb414dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:46.610 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.605+0000 7f3bb3fff700 1 --2- 172.21.15.23:0/1165838123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3bb40a5be0 0x7f3bb4131640 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:46.610 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.605+0000 7f3bb3fff700 1 --2- 172.21.15.23:0/1165838123 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3bb40a8fb0 0x7f3bb4131b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:46.610 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.605+0000 7f3bb3fff700 1 -- 172.21.15.23:0/1165838123 >> 172.21.15.23:0/1165838123 conn(0x7f3bb4061850 msgr2=0x7f3bb4061c30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:46.610 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.605+0000 7f3bb3fff700 1 -- 172.21.15.23:0/1165838123 shutdown_connections 2020-11-20T11:32:46.611 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.605+0000 7f3bb3fff700 1 -- 172.21.15.23:0/1165838123 wait complete. 2020-11-20T11:32:46.620 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-11-20T11:32:46.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.746+0000 7f1eb4678700 1 Processor -- start 2020-11-20T11:32:46.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.746+0000 7f1eb4678700 1 -- start start 2020-11-20T11:32:46.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.747+0000 7f1eb4678700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1eac137410 0x7f1eac137870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:46.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.747+0000 7f1eb4678700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f1eac133dc0 0x7f1eac1341d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:46.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.747+0000 7f1eb4678700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eac062ad0 con 0x7f1eac0a6e10 2020-11-20T11:32:46.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.747+0000 7f1eb4678700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eac0a61f0 con 0x7f1eac137410 2020-11-20T11:32:46.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.747+0000 7f1eb4678700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eac061f60 con 0x7f1eac133dc0 2020-11-20T11:32:46.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.747+0000 7f1eb1c13700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1eac137410 0x7f1eac137870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:46.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.747+0000 7f1eb2414700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f1eac0a6e10 0x7f1eac061850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55742/0 (socket says 172.21.15.23:55742) 2020-11-20T11:32:46.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.747+0000 7f1eb1c13700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1eac137410 0x7f1eac137870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41494/0 (socket says 172.21.15.23:41494) 2020-11-20T11:32:46.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.747+0000 7f1eb2414700 1 -- 172.21.15.23:0/4226656481 learned_addr learned my addr 172.21.15.23:0/4226656481 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:46.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.747+0000 7f1eb2c15700 1 --2- 172.21.15.23:0/4226656481 >> v2:172.21.15.155:3300/0 conn(0x7f1eac133dc0 0x7f1eac1341d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:46.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.747+0000 7f1eb1c13700 1 -- 172.21.15.23:0/4226656481 >> v2:172.21.15.155:3300/0 conn(0x7f1eac133dc0 msgr2=0x7f1eac1341d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:46.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.747+0000 7f1eb1c13700 1 --2- 172.21.15.23:0/4226656481 >> v2:172.21.15.155:3300/0 conn(0x7f1eac133dc0 0x7f1eac1341d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:46.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.747+0000 7f1eb1c13700 1 -- 172.21.15.23:0/4226656481 >> v1:172.21.15.23:6789/0 conn(0x7f1eac0a6e10 legacy=0x7f1eac061850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:46.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.747+0000 7f1eb1412700 1 -- 172.21.15.23:0/4226656481 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1526646337 0 0) 0x7f1eac062ad0 con 0x7f1eac0a6e10 2020-11-20T11:32:46.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.747+0000 7f1eb1c13700 1 -- 172.21.15.23:0/4226656481 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eac0a63e0 con 0x7f1eac137410 2020-11-20T11:32:46.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.748+0000 7f1eb1c13700 1 --2- 172.21.15.23:0/4226656481 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1eac137410 0x7f1eac137870 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f1eac0a7480 tx=0x7f1e9c006e30).ready entity=mon.0 client_cookie=101041eb766ee2b4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:46.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.748+0000 7f1eb1412700 1 -- 172.21.15.23:0/4226656481 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c0097b0 con 0x7f1eac137410 2020-11-20T11:32:46.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.748+0000 7f1eb1412700 1 -- 172.21.15.23:0/4226656481 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e9c000e90 con 0x7f1eac137410 2020-11-20T11:32:46.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.748+0000 7f1eb1412700 1 -- 172.21.15.23:0/4226656481 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c009b20 con 0x7f1eac137410 2020-11-20T11:32:46.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.748+0000 7f1eb4678700 1 -- 172.21.15.23:0/4226656481 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1eac137410 msgr2=0x7f1eac137870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:46.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.748+0000 7f1eb4678700 1 --2- 172.21.15.23:0/4226656481 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1eac137410 0x7f1eac137870 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f1eac0a7480 tx=0x7f1e9c006e30).stop 2020-11-20T11:32:46.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.748+0000 7f1eb4678700 1 -- 172.21.15.23:0/4226656481 shutdown_connections 2020-11-20T11:32:46.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.748+0000 7f1eb4678700 1 --2- 172.21.15.23:0/4226656481 >> v2:172.21.15.155:3300/0 conn(0x7f1eac133dc0 0x7f1eac1341d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:46.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.748+0000 7f1eb4678700 1 --2- 172.21.15.23:0/4226656481 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1eac137410 0x7f1eac137870 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:46.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.748+0000 7f1eb4678700 1 -- 172.21.15.23:0/4226656481 >> 172.21.15.23:0/4226656481 conn(0x7f1eac060c80 msgr2=0x7f1eac0acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:46.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.748+0000 7f1eb4678700 1 -- 172.21.15.23:0/4226656481 shutdown_connections 2020-11-20T11:32:46.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.748+0000 7f1eb4678700 1 -- 172.21.15.23:0/4226656481 wait complete. 2020-11-20T11:32:46.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.749+0000 7f1eb4678700 1 Processor -- start 2020-11-20T11:32:46.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.749+0000 7f1eb4678700 1 -- start start 2020-11-20T11:32:46.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.750+0000 7f1eb4678700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1eac0a6e10 0x7f1eac065f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:46.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.750+0000 7f1eb4678700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1eac133dc0 0x7f1eac12fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:46.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.750+0000 7f1eb4678700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1eac137410 0x7f1eac146f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:46.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.750+0000 7f1eb4678700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac066990 con 0x7f1eac0a6e10 2020-11-20T11:32:46.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.750+0000 7f1eb4678700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac066bc0 con 0x7f1eac137410 2020-11-20T11:32:46.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.750+0000 7f1eb4678700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac066e60 con 0x7f1eac133dc0 2020-11-20T11:32:46.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.750+0000 7f1eb2414700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1eac0a6e10 0x7f1eac065f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:46.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.750+0000 7f1eb2414700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1eac0a6e10 0x7f1eac065f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41498/0 (socket says 172.21.15.23:41498) 2020-11-20T11:32:46.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.750+0000 7f1eb2414700 1 -- 172.21.15.23:0/239034359 learned_addr learned my addr 172.21.15.23:0/239034359 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:46.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.750+0000 7f1eb2c15700 1 --2- 172.21.15.23:0/239034359 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1eac137410 0x7f1eac146f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:46.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.750+0000 7f1eb1c13700 1 --2- 172.21.15.23:0/239034359 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1eac133dc0 0x7f1eac12fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:46.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.750+0000 7f1eb2414700 1 -- 172.21.15.23:0/239034359 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1eac133dc0 msgr2=0x7f1eac12fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:46.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.750+0000 7f1eb2414700 1 --2- 172.21.15.23:0/239034359 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1eac133dc0 0x7f1eac12fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:46.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.750+0000 7f1eb2414700 1 -- 172.21.15.23:0/239034359 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1eac137410 msgr2=0x7f1eac146f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:46.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.750+0000 7f1eb2414700 1 --2- 172.21.15.23:0/239034359 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1eac137410 0x7f1eac146f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:46.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.750+0000 7f1eb2414700 1 -- 172.21.15.23:0/239034359 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eac134ea0 con 0x7f1eac0a6e10 2020-11-20T11:32:46.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.750+0000 7f1eb2414700 1 --2- 172.21.15.23:0/239034359 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1eac0a6e10 0x7f1eac065f50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f1ea800c7c0 tx=0x7f1ea8007b00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:46.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.751+0000 7f1e9a7fc700 1 -- 172.21.15.23:0/239034359 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea80072e0 con 0x7f1eac0a6e10 2020-11-20T11:32:46.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.751+0000 7f1e9a7fc700 1 -- 172.21.15.23:0/239034359 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea8007440 con 0x7f1eac0a6e10 2020-11-20T11:32:46.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.751+0000 7f1eb4678700 1 -- 172.21.15.23:0/239034359 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eac064ef0 con 0x7f1eac0a6e10 2020-11-20T11:32:46.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.751+0000 7f1e9a7fc700 1 -- 172.21.15.23:0/239034359 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea8005840 con 0x7f1eac0a6e10 2020-11-20T11:32:46.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.751+0000 7f1eb4678700 1 -- 172.21.15.23:0/239034359 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eac147400 con 0x7f1eac0a6e10 2020-11-20T11:32:46.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.751+0000 7f1eb4678700 1 -- 172.21.15.23:0/239034359 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eac13b680 con 0x7f1eac0a6e10 2020-11-20T11:32:46.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.752+0000 7f1e9a7fc700 1 -- 172.21.15.23:0/239034359 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1ea8010070 con 0x7f1eac0a6e10 2020-11-20T11:32:46.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.752+0000 7f1e9a7fc700 1 --2- 172.21.15.23:0/239034359 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1e84056d60 0x7f1e84059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:46.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.752+0000 7f1e9a7fc700 1 -- 172.21.15.23:0/239034359 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f1ea806f7a0 con 0x7f1eac0a6e10 2020-11-20T11:32:46.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.752+0000 7f1eb1c13700 1 --2- 172.21.15.23:0/239034359 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1e84056d60 0x7f1e84059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:46.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.753+0000 7f1eb1c13700 1 --2- 172.21.15.23:0/239034359 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1e84056d60 0x7f1e84059220 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f1e9c006e70 tx=0x7f1e9c015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:46.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.756+0000 7f1e9a7fc700 1 -- 172.21.15.23:0/239034359 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1ea8044a90 con 0x7f1eac0a6e10 2020-11-20T11:32:46.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.904+0000 7f1eb4678700 1 -- 172.21.15.23:0/239034359 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1eac05c520 con 0x7f1eac0a6e10 2020-11-20T11:32:46.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.905+0000 7f1e9a7fc700 1 -- 172.21.15.23:0/239034359 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+10 (secure 0 0 0) 0x7f1ea8007c40 con 0x7f1eac0a6e10 2020-11-20T11:32:46.908 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 8 2020-11-20T11:32:46.909 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.906+0000 7f1eb4678700 1 -- 172.21.15.23:0/239034359 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1e84056d60 msgr2=0x7f1e84059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:46.910 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.906+0000 7f1eb4678700 1 --2- 172.21.15.23:0/239034359 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1e84056d60 0x7f1e84059220 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f1e9c006e70 tx=0x7f1e9c015000).stop 2020-11-20T11:32:46.910 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.906+0000 7f1eb4678700 1 -- 172.21.15.23:0/239034359 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1eac0a6e10 msgr2=0x7f1eac065f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:46.910 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.906+0000 7f1eb4678700 1 --2- 172.21.15.23:0/239034359 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1eac0a6e10 0x7f1eac065f50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f1ea800c7c0 tx=0x7f1ea8007b00).stop 2020-11-20T11:32:46.910 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.906+0000 7f1eb4678700 1 -- 172.21.15.23:0/239034359 shutdown_connections 2020-11-20T11:32:46.911 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.906+0000 7f1eb4678700 1 --2- 172.21.15.23:0/239034359 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1eac137410 0x7f1eac146f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:46.911 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.906+0000 7f1eb4678700 1 --2- 172.21.15.23:0/239034359 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1eac0a6e10 0x7f1eac065f50 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:46.911 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.906+0000 7f1eb4678700 1 --2- 172.21.15.23:0/239034359 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1e84056d60 0x7f1e84059220 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:46.912 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.906+0000 7f1eb4678700 1 --2- 172.21.15.23:0/239034359 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1eac133dc0 0x7f1eac12fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:46.913 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.906+0000 7f1eb4678700 1 -- 172.21.15.23:0/239034359 >> 172.21.15.23:0/239034359 conn(0x7f1eac060c80 msgr2=0x7f1eac139dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:46.913 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.906+0000 7f1eb4678700 1 -- 172.21.15.23:0/239034359 shutdown_connections 2020-11-20T11:32:46.913 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:46.906+0000 7f1eb4678700 1 -- 172.21.15.23:0/239034359 wait complete. 2020-11-20T11:32:46.921 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-11-20T11:32:47.040 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.038+0000 7f2a4b59e700 1 Processor -- start 2020-11-20T11:32:47.041 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.038+0000 7f2a4b59e700 1 -- start start 2020-11-20T11:32:47.043 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.038+0000 7f2a4b59e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a4c0a9040 0x7f2a4c0aaf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:47.043 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.038+0000 7f2a4b59e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f2a4c0ad2f0 0x7f2a4c13e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:47.043 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.038+0000 7f2a4b59e700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a4c0ad860 con 0x7f2a4c0ace70 2020-11-20T11:32:47.043 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.038+0000 7f2a4b59e700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a4c0aa8a0 con 0x7f2a4c0ad2f0 2020-11-20T11:32:47.044 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.038+0000 7f2a4b59e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4c0aaa90 con 0x7f2a4c0a9040 2020-11-20T11:32:47.044 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.039+0000 7f2a4ad9d700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f2a4c0ad2f0 0x7f2a4c13e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:47.044 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.039+0000 7f2a49d9b700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f2a4c0ace70 0x7f2a4c13de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:33730/0 (socket says 172.21.15.23:33730) 2020-11-20T11:32:47.044 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.039+0000 7f2a4ad9d700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f2a4c0ad2f0 0x7f2a4c13e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:34402/0 (socket says 172.21.15.23:34402) 2020-11-20T11:32:47.044 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.039+0000 7f2a49d9b700 1 -- 172.21.15.23:0/2032514290 learned_addr learned my addr 172.21.15.23:0/2032514290 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:47.045 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.039+0000 7f2a4a59c700 1 --2- 172.21.15.23:0/2032514290 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a4c0a9040 0x7f2a4c0aaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:47.045 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.039+0000 7f2a4ad9d700 1 -- 172.21.15.23:0/2032514290 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a4c0a9040 msgr2=0x7f2a4c0aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:47.045 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.039+0000 7f2a4ad9d700 1 --2- 172.21.15.23:0/2032514290 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a4c0a9040 0x7f2a4c0aaf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:47.045 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.039+0000 7f2a4ad9d700 1 -- 172.21.15.23:0/2032514290 >> v1:172.21.15.155:6789/0 conn(0x7f2a4c0ace70 legacy=0x7f2a4c13de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:47.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.039+0000 7f2a4ad9d700 1 -- 172.21.15.23:0/2032514290 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a4c0a9420 con 0x7f2a4c0ad2f0 2020-11-20T11:32:47.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.039+0000 7f2a4a59c700 1 --2- 172.21.15.23:0/2032514290 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a4c0a9040 0x7f2a4c0aaf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:32:47.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.039+0000 7f2a4ad9d700 1 --2- 172.21.15.23:0/2032514290 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a4c0ad2f0 0x7f2a4c13e510 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f2a44009ce0 tx=0x7f2a44009220).ready entity=mon.1 client_cookie=c942a23cc64b39eb server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:47.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.039+0000 7f2a4959a700 1 -- 172.21.15.23:0/2032514290 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a4400ba40 con 0x7f2a4c0ad2f0 2020-11-20T11:32:47.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.039+0000 7f2a4959a700 1 -- 172.21.15.23:0/2032514290 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a4400bba0 con 0x7f2a4c0ad2f0 2020-11-20T11:32:47.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.039+0000 7f2a4959a700 1 -- 172.21.15.23:0/2032514290 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a44011400 con 0x7f2a4c0ad2f0 2020-11-20T11:32:47.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.040+0000 7f2a4b59e700 1 -- 172.21.15.23:0/2032514290 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a4c0ad2f0 msgr2=0x7f2a4c13e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:47.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.040+0000 7f2a4b59e700 1 --2- 172.21.15.23:0/2032514290 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a4c0ad2f0 0x7f2a4c13e510 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f2a44009ce0 tx=0x7f2a44009220).stop 2020-11-20T11:32:47.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.040+0000 7f2a4b59e700 1 -- 172.21.15.23:0/2032514290 shutdown_connections 2020-11-20T11:32:47.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.040+0000 7f2a4b59e700 1 --2- 172.21.15.23:0/2032514290 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a4c0ad2f0 0x7f2a4c13e510 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:47.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.040+0000 7f2a4b59e700 1 --2- 172.21.15.23:0/2032514290 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a4c0a9040 0x7f2a4c0aaf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:47.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.040+0000 7f2a4b59e700 1 -- 172.21.15.23:0/2032514290 >> 172.21.15.23:0/2032514290 conn(0x7f2a4c0683b0 msgr2=0x7f2a4c0b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:47.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.040+0000 7f2a4b59e700 1 -- 172.21.15.23:0/2032514290 shutdown_connections 2020-11-20T11:32:47.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.040+0000 7f2a4b59e700 1 -- 172.21.15.23:0/2032514290 wait complete. 2020-11-20T11:32:47.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.041+0000 7f2a4b59e700 1 Processor -- start 2020-11-20T11:32:47.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.041+0000 7f2a4b59e700 1 -- start start 2020-11-20T11:32:47.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.042+0000 7f2a4b59e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a4c0a9040 0x7f2a4c09ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:47.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.042+0000 7f2a4b59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a4c0ace70 0x7f2a4c137400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:47.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.042+0000 7f2a4b59e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a4c0ad2f0 0x7f2a4c1378f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:47.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.042+0000 7f2a4b59e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4c13d6b0 con 0x7f2a4c0ace70 2020-11-20T11:32:47.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.042+0000 7f2a4b59e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4c09e650 con 0x7f2a4c0a9040 2020-11-20T11:32:47.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.042+0000 7f2a4b59e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4c09e8f0 con 0x7f2a4c0ad2f0 2020-11-20T11:32:47.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.042+0000 7f2a49d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a4c0ace70 0x7f2a4c137400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:47.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.042+0000 7f2a49d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a4c0ace70 0x7f2a4c137400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41512/0 (socket says 172.21.15.23:41512) 2020-11-20T11:32:47.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.042+0000 7f2a49d9b700 1 -- 172.21.15.23:0/3018086372 learned_addr learned my addr 172.21.15.23:0/3018086372 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:47.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.042+0000 7f2a4a59c700 1 --2- 172.21.15.23:0/3018086372 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a4c0a9040 0x7f2a4c09ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:47.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.042+0000 7f2a4ad9d700 1 --2- 172.21.15.23:0/3018086372 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a4c0ad2f0 0x7f2a4c1378f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:47.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.042+0000 7f2a49d9b700 1 -- 172.21.15.23:0/3018086372 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a4c0ad2f0 msgr2=0x7f2a4c1378f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:47.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.042+0000 7f2a49d9b700 1 --2- 172.21.15.23:0/3018086372 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a4c0ad2f0 0x7f2a4c1378f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:47.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.042+0000 7f2a49d9b700 1 -- 172.21.15.23:0/3018086372 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a4c0a9040 msgr2=0x7f2a4c09ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:47.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.042+0000 7f2a49d9b700 1 --2- 172.21.15.23:0/3018086372 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a4c0a9040 0x7f2a4c09ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:47.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.042+0000 7f2a49d9b700 1 -- 172.21.15.23:0/3018086372 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a4c137de0 con 0x7f2a4c0ace70 2020-11-20T11:32:47.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.042+0000 7f2a4a59c700 1 --2- 172.21.15.23:0/3018086372 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a4c0a9040 0x7f2a4c09ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:32:47.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.042+0000 7f2a4ad9d700 1 --2- 172.21.15.23:0/3018086372 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a4c0ad2f0 0x7f2a4c1378f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:32:47.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.042+0000 7f2a49d9b700 1 --2- 172.21.15.23:0/3018086372 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a4c0ace70 0x7f2a4c137400 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f2a4000c6e0 tx=0x7f2a40007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:47.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.042+0000 7f2a3a7fc700 1 -- 172.21.15.23:0/3018086372 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a40007320 con 0x7f2a4c0ace70 2020-11-20T11:32:47.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.042+0000 7f2a3a7fc700 1 -- 172.21.15.23:0/3018086372 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a40007480 con 0x7f2a4c0ace70 2020-11-20T11:32:47.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.042+0000 7f2a4b59e700 1 -- 172.21.15.23:0/3018086372 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a4c09d910 con 0x7f2a4c0ace70 2020-11-20T11:32:47.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.042+0000 7f2a3a7fc700 1 -- 172.21.15.23:0/3018086372 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a40005840 con 0x7f2a4c0ace70 2020-11-20T11:32:47.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.042+0000 7f2a4b59e700 1 -- 172.21.15.23:0/3018086372 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a4c09f4c0 con 0x7f2a4c0ace70 2020-11-20T11:32:47.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.043+0000 7f2a4b59e700 1 -- 172.21.15.23:0/3018086372 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a4c0aaf70 con 0x7f2a4c0ace70 2020-11-20T11:32:47.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.044+0000 7f2a3a7fc700 1 -- 172.21.15.23:0/3018086372 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2a40010030 con 0x7f2a4c0ace70 2020-11-20T11:32:47.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.044+0000 7f2a3a7fc700 1 --2- 172.21.15.23:0/3018086372 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2a20056d60 0x7f2a20059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:47.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.044+0000 7f2a3a7fc700 1 -- 172.21.15.23:0/3018086372 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f2a4006fcd0 con 0x7f2a4c0ace70 2020-11-20T11:32:47.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.044+0000 7f2a4a59c700 1 --2- 172.21.15.23:0/3018086372 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2a20056d60 0x7f2a20059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:47.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.044+0000 7f2a4a59c700 1 --2- 172.21.15.23:0/3018086372 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2a20056d60 0x7f2a20059220 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f2a3c006bb0 tx=0x7f2a3c008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:47.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.048+0000 7f2a3a7fc700 1 -- 172.21.15.23:0/3018086372 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2a40044fc0 con 0x7f2a4c0ace70 2020-11-20T11:32:47.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.198+0000 7f2a4b59e700 1 -- 172.21.15.23:0/3018086372 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2a4c05ceb0 con 0x7f2a4c0ace70 2020-11-20T11:32:47.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.199+0000 7f2a3a7fc700 1 -- 172.21.15.23:0/3018086372 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+10 (secure 0 0 0) 0x7f2a400485e0 con 0x7f2a4c0ace70 2020-11-20T11:32:47.202 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 8 2020-11-20T11:32:47.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.200+0000 7f2a4b59e700 1 -- 172.21.15.23:0/3018086372 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2a20056d60 msgr2=0x7f2a20059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:47.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.200+0000 7f2a4b59e700 1 --2- 172.21.15.23:0/3018086372 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2a20056d60 0x7f2a20059220 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f2a3c006bb0 tx=0x7f2a3c008040).stop 2020-11-20T11:32:47.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.200+0000 7f2a4b59e700 1 -- 172.21.15.23:0/3018086372 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a4c0ace70 msgr2=0x7f2a4c137400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:47.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.200+0000 7f2a4b59e700 1 --2- 172.21.15.23:0/3018086372 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a4c0ace70 0x7f2a4c137400 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f2a4000c6e0 tx=0x7f2a40007c50).stop 2020-11-20T11:32:47.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.200+0000 7f2a4b59e700 1 -- 172.21.15.23:0/3018086372 shutdown_connections 2020-11-20T11:32:47.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.200+0000 7f2a4b59e700 1 --2- 172.21.15.23:0/3018086372 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2a20056d60 0x7f2a20059220 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:47.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.200+0000 7f2a4b59e700 1 --2- 172.21.15.23:0/3018086372 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a4c0ad2f0 0x7f2a4c1378f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:47.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.200+0000 7f2a4b59e700 1 --2- 172.21.15.23:0/3018086372 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a4c0a9040 0x7f2a4c09ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:47.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.200+0000 7f2a4b59e700 1 --2- 172.21.15.23:0/3018086372 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a4c0ace70 0x7f2a4c137400 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:47.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.200+0000 7f2a4b59e700 1 -- 172.21.15.23:0/3018086372 >> 172.21.15.23:0/3018086372 conn(0x7f2a4c0683b0 msgr2=0x7f2a4c0af7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:47.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.201+0000 7f2a4b59e700 1 -- 172.21.15.23:0/3018086372 shutdown_connections 2020-11-20T11:32:47.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.201+0000 7f2a4b59e700 1 -- 172.21.15.23:0/3018086372 wait complete. 2020-11-20T11:32:47.215 DEBUG:tasks.mon_thrash:Yielding 2020-11-20T11:32:47.215 INFO:teuthology.run_tasks:Running task workunit... 2020-11-20T11:32:47.233 INFO:tasks.workunit:Pulling workunits from ref b86e691bf191566922a8f08e2a5abf64fde8cfcb 2020-11-20T11:32:47.235 INFO:tasks.workunit:Making a separate scratch dir for every client... 2020-11-20T11:32:47.235 INFO:tasks.workunit:timeout=3h 2020-11-20T11:32:47.235 INFO:tasks.workunit:cleanup=True 2020-11-20T11:32:47.236 INFO:teuthology.orchestra.run.smithi023:> stat -- /home/ubuntu/cephtest/mnt.0 2020-11-20T11:32:47.236 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T11:32:47.253 INFO:teuthology.orchestra.run.smithi023.stdout: File: /home/ubuntu/cephtest/mnt.0 2020-11-20T11:32:47.254 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2020-11-20T11:32:47.254 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 2fh/47d Inode: 1 Links: 2 2020-11-20T11:32:47.254 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2020-11-20T11:32:47.254 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:fusefs_t:s0 2020-11-20T11:32:47.255 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2020-11-20T11:32:47.255 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2020-11-20 11:32:25.471406096 +0000 2020-11-20T11:32:47.255 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2020-11-20 11:32:43.672695775 +0000 2020-11-20T11:32:47.256 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: - 2020-11-20T11:32:47.256 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2020-11-20T11:32:47.256 INFO:teuthology.orchestra.run.smithi023:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2020-11-20T11:32:47.324 INFO:teuthology.orchestra.run.smithi023:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout b86e691bf191566922a8f08e2a5abf64fde8cfcb 2020-11-20T11:32:47.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.349+0000 7f84fe7a6700 1 Processor -- start 2020-11-20T11:32:47.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.349+0000 7f84fe7a6700 1 -- start start 2020-11-20T11:32:47.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.350+0000 7f84fe7a6700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84f80aaab0 0x7f84f813ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:47.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.350+0000 7f84fe7a6700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f84f80aae90 0x7f84f81381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:47.354 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.350+0000 7f84fe7a6700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84f813e600 con 0x7f84f813aeb0 2020-11-20T11:32:47.354 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.350+0000 7f84fe7a6700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84f813e8a0 con 0x7f84f80aae90 2020-11-20T11:32:47.354 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.350+0000 7f84fe7a6700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84f80ab6d0 con 0x7f84f80aaab0 2020-11-20T11:32:47.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.350+0000 7f84f77fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f84f80aae90 0x7f84f81381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:47.360 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.350+0000 7f84f7fff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84f80aaab0 0x7f84f813ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:47.360 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.350+0000 7f84f7fff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84f80aaab0 0x7f84f813ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:51758/0 (socket says 172.21.15.23:51758) 2020-11-20T11:32:47.360 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.350+0000 7f84f7fff700 1 -- 172.21.15.23:0/3867374367 learned_addr learned my addr 172.21.15.23:0/3867374367 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:47.361 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.350+0000 7f84fcd43700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f84f813aeb0 0x7f84f80a8c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:33744/0 (socket says 172.21.15.23:33744) 2020-11-20T11:32:47.361 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.350+0000 7f84f77fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f84f80aae90 0x7f84f81381a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:34418/0 (socket says 172.21.15.23:34418) 2020-11-20T11:32:47.361 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.350+0000 7f84f7fff700 1 -- 172.21.15.23:0/3867374367 >> v2:172.21.15.155:3300/0 conn(0x7f84f80aae90 msgr2=0x7f84f81381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:47.361 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.350+0000 7f84f7fff700 1 --2- 172.21.15.23:0/3867374367 >> v2:172.21.15.155:3300/0 conn(0x7f84f80aae90 0x7f84f81381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:47.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.350+0000 7f84f7fff700 1 -- 172.21.15.23:0/3867374367 >> v1:172.21.15.155:6789/0 conn(0x7f84f813aeb0 legacy=0x7f84f80a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:47.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.350+0000 7f84f7fff700 1 -- 172.21.15.23:0/3867374367 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f80a9370 con 0x7f84f80aaab0 2020-11-20T11:32:47.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.351+0000 7f84f7fff700 1 --2- 172.21.15.23:0/3867374367 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84f80aaab0 0x7f84f813ed40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f84e0009a10 tx=0x7f84e0006ef0).ready entity=mon.2 client_cookie=40fbd9fb2959bccf server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:47.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.351+0000 7f84f6ffd700 1 -- 172.21.15.23:0/3867374367 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e000b860 con 0x7f84f80aaab0 2020-11-20T11:32:47.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.351+0000 7f84f6ffd700 1 -- 172.21.15.23:0/3867374367 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84e0000e90 con 0x7f84f80aaab0 2020-11-20T11:32:47.363 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.351+0000 7f84f6ffd700 1 -- 172.21.15.23:0/3867374367 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e000bbd0 con 0x7f84f80aaab0 2020-11-20T11:32:47.363 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.351+0000 7f84fe7a6700 1 -- 172.21.15.23:0/3867374367 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84f80aaab0 msgr2=0x7f84f813ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:47.363 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.351+0000 7f84fe7a6700 1 --2- 172.21.15.23:0/3867374367 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84f80aaab0 0x7f84f813ed40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f84e0009a10 tx=0x7f84e0006ef0).stop 2020-11-20T11:32:47.363 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.351+0000 7f84fe7a6700 1 -- 172.21.15.23:0/3867374367 shutdown_connections 2020-11-20T11:32:47.363 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.351+0000 7f84fe7a6700 1 --2- 172.21.15.23:0/3867374367 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84f80aaab0 0x7f84f813ed40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:47.364 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.351+0000 7f84fe7a6700 1 --2- 172.21.15.23:0/3867374367 >> v2:172.21.15.155:3300/0 conn(0x7f84f80aae90 0x7f84f81381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:47.364 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.351+0000 7f84fe7a6700 1 -- 172.21.15.23:0/3867374367 >> 172.21.15.23:0/3867374367 conn(0x7f84f80678e0 msgr2=0x7f84f8067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:47.364 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.351+0000 7f84fe7a6700 1 -- 172.21.15.23:0/3867374367 shutdown_connections 2020-11-20T11:32:47.364 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.351+0000 7f84fe7a6700 1 -- 172.21.15.23:0/3867374367 wait complete. 2020-11-20T11:32:47.364 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.352+0000 7f84fe7a6700 1 Processor -- start 2020-11-20T11:32:47.365 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.352+0000 7f84fe7a6700 1 -- start start 2020-11-20T11:32:47.365 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.353+0000 7f84fe7a6700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84f80aaab0 0x7f84f8132510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:47.365 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.353+0000 7f84fe7a6700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84f80aae90 0x7f84f814a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:47.365 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.353+0000 7f84fe7a6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84f813d700 0x7f84f814ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:47.366 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.353+0000 7f84fe7a6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f813dc80 con 0x7f84f813d700 2020-11-20T11:32:47.366 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.353+0000 7f84fe7a6700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f813df20 con 0x7f84f80aaab0 2020-11-20T11:32:47.366 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.353+0000 7f84fe7a6700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84f814a490 con 0x7f84f80aae90 2020-11-20T11:32:47.366 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.353+0000 7f84fcd43700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84f813d700 0x7f84f814ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:47.366 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.353+0000 7f84fcd43700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84f813d700 0x7f84f814ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41528/0 (socket says 172.21.15.23:41528) 2020-11-20T11:32:47.367 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.353+0000 7f84fcd43700 1 -- 172.21.15.23:0/2774050337 learned_addr learned my addr 172.21.15.23:0/2774050337 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:47.367 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.353+0000 7f84f7fff700 1 --2- 172.21.15.23:0/2774050337 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84f80aaab0 0x7f84f8132510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:47.367 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.353+0000 7f84f77fe700 1 --2- 172.21.15.23:0/2774050337 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84f80aae90 0x7f84f814a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:47.367 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.354+0000 7f84fcd43700 1 -- 172.21.15.23:0/2774050337 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84f80aae90 msgr2=0x7f84f814a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:47.367 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.354+0000 7f84fcd43700 1 --2- 172.21.15.23:0/2774050337 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84f80aae90 0x7f84f814a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:47.368 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.354+0000 7f84fcd43700 1 -- 172.21.15.23:0/2774050337 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84f80aaab0 msgr2=0x7f84f8132510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:47.368 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.354+0000 7f84fcd43700 1 --2- 172.21.15.23:0/2774050337 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84f80aaab0 0x7f84f8132510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:47.368 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.354+0000 7f84fcd43700 1 -- 172.21.15.23:0/2774050337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f813e160 con 0x7f84f813d700 2020-11-20T11:32:47.368 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.354+0000 7f84fcd43700 1 --2- 172.21.15.23:0/2774050337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84f813d700 0x7f84f814ae20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f84f000ca90 tx=0x7f84f00013a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:47.369 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.354+0000 7f84effff700 1 -- 172.21.15.23:0/2774050337 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f00058f0 con 0x7f84f813d700 2020-11-20T11:32:47.369 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.354+0000 7f84fe7a6700 1 -- 172.21.15.23:0/2774050337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84f8132be0 con 0x7f84f813d700 2020-11-20T11:32:47.369 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.354+0000 7f84effff700 1 -- 172.21.15.23:0/2774050337 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84f0005a50 con 0x7f84f813d700 2020-11-20T11:32:47.369 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.354+0000 7f84effff700 1 -- 172.21.15.23:0/2774050337 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f0005dd0 con 0x7f84f813d700 2020-11-20T11:32:47.369 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.354+0000 7f84fe7a6700 1 -- 172.21.15.23:0/2774050337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84f81317d0 con 0x7f84f813d700 2020-11-20T11:32:47.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.355+0000 7f84fe7a6700 1 -- 172.21.15.23:0/2774050337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84f805c7d0 con 0x7f84f813d700 2020-11-20T11:32:47.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.355+0000 7f84effff700 1 -- 172.21.15.23:0/2774050337 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f84f0010030 con 0x7f84f813d700 2020-11-20T11:32:47.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.356+0000 7f84effff700 1 --2- 172.21.15.23:0/2774050337 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f84d4056d90 0x7f84d4059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:47.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.356+0000 7f84effff700 1 -- 172.21.15.23:0/2774050337 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f84f006fe60 con 0x7f84f813d700 2020-11-20T11:32:47.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.356+0000 7f84f7fff700 1 --2- 172.21.15.23:0/2774050337 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f84d4056d90 0x7f84d4059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:47.371 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.356+0000 7f84f7fff700 1 --2- 172.21.15.23:0/2774050337 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f84d4056d90 0x7f84d4059250 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f84e0006f30 tx=0x7f84e0016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:47.371 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.359+0000 7f84effff700 1 -- 172.21.15.23:0/2774050337 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f84f0045150 con 0x7f84f813d700 2020-11-20T11:32:47.425 INFO:tasks.workunit.client.0.smithi023.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2020-11-20T11:32:47.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.562+0000 7f84fe7a6700 1 -- 172.21.15.23:0/2774050337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84f814a650 con 0x7f84f813d700 2020-11-20T11:32:47.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.563+0000 7f84effff700 1 -- 172.21.15.23:0/2774050337 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+2128 (secure 0 0 0) 0x7f84f0048770 con 0x7f84f813d700 2020-11-20T11:32:47.566 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:32:47.567 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4223,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/564012815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":564012815},{"type":"v1","addr":"172.21.15.155:6835","nonce":564012815}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T11:32:25.469149+0000","modified":"2020-11-20T11:32:28.118344+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.23:6833/919963501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":919963501},{"type":"v1","addr":"172.21.15.23:6833","nonce":919963501}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T11:32:47.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.564+0000 7f84fe7a6700 1 -- 172.21.15.23:0/2774050337 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f84d4056d90 msgr2=0x7f84d4059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:47.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.564+0000 7f84fe7a6700 1 --2- 172.21.15.23:0/2774050337 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f84d4056d90 0x7f84d4059250 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f84e0006f30 tx=0x7f84e0016000).stop 2020-11-20T11:32:47.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.564+0000 7f84fe7a6700 1 -- 172.21.15.23:0/2774050337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84f813d700 msgr2=0x7f84f814ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:47.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.564+0000 7f84fe7a6700 1 --2- 172.21.15.23:0/2774050337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84f813d700 0x7f84f814ae20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f84f000ca90 tx=0x7f84f00013a0).stop 2020-11-20T11:32:47.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.564+0000 7f84fe7a6700 1 -- 172.21.15.23:0/2774050337 shutdown_connections 2020-11-20T11:32:47.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.564+0000 7f84fe7a6700 1 --2- 172.21.15.23:0/2774050337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84f813d700 0x7f84f814ae20 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:47.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.564+0000 7f84fe7a6700 1 --2- 172.21.15.23:0/2774050337 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84f80aaab0 0x7f84f8132510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:47.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.564+0000 7f84fe7a6700 1 --2- 172.21.15.23:0/2774050337 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f84d4056d90 0x7f84d4059250 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:47.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.564+0000 7f84fe7a6700 1 --2- 172.21.15.23:0/2774050337 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84f80aae90 0x7f84f814a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:47.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.565+0000 7f84fe7a6700 1 -- 172.21.15.23:0/2774050337 >> 172.21.15.23:0/2774050337 conn(0x7f84f80678e0 msgr2=0x7f84f81405c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:47.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.565+0000 7f84fe7a6700 1 -- 172.21.15.23:0/2774050337 shutdown_connections 2020-11-20T11:32:47.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.565+0000 7f84fe7a6700 1 -- 172.21.15.23:0/2774050337 wait complete. 2020-11-20T11:32:47.570 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 7 2020-11-20T11:32:47.579 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2020-11-20T11:32:47.579 INFO:tasks.mon_thrash.mon_thrasher:seed: 1605871965, revive delay: 20.0, thrash delay: 10.0 thrash many: False, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2020-11-20T11:32:47.579 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:32:47.580 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:32:47.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.690+0000 7fe7fa584700 1 Processor -- start 2020-11-20T11:32:47.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.690+0000 7fe7fa584700 1 -- start start 2020-11-20T11:32:47.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.690+0000 7fe7fa584700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe7f40aa1c0 0x7fe7f40ac290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:47.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.690+0000 7fe7fa584700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f4067bc0 con 0x7fe7f4060a80 2020-11-20T11:32:47.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.690+0000 7fe7fa584700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f4067e60 con 0x7fe7f40a9d40 2020-11-20T11:32:47.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.690+0000 7fe7fa584700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7f40aa630 con 0x7fe7f40aa1c0 2020-11-20T11:32:47.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.691+0000 7fe7f8b21700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe7f40aa1c0 0x7fe7f40ac290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:47.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.691+0000 7fe7f3fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fe7f4060a80 0x7fe7f4061950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55784/0 (socket says 172.21.15.23:55784) 2020-11-20T11:32:47.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.691+0000 7fe7f3fff700 1 -- 172.21.15.23:0/2104431471 learned_addr learned my addr 172.21.15.23:0/2104431471 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:47.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.691+0000 7fe7f8b21700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe7f40aa1c0 0x7fe7f40ac290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41538/0 (socket says 172.21.15.23:41538) 2020-11-20T11:32:47.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.691+0000 7fe7f8b21700 1 -- 172.21.15.23:0/2104431471 >> v1:172.21.15.155:6789/0 conn(0x7fe7f40a9d40 legacy=0x7fe7f40abb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:47.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.691+0000 7fe7f8b21700 1 -- 172.21.15.23:0/2104431471 >> v1:172.21.15.23:6789/0 conn(0x7fe7f4060a80 legacy=0x7fe7f4061950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:47.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.691+0000 7fe7f8b21700 1 -- 172.21.15.23:0/2104431471 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f4068100 con 0x7fe7f40aa1c0 2020-11-20T11:32:47.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.691+0000 7fe7f8b21700 1 --2- 172.21.15.23:0/2104431471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7f40aa1c0 0x7fe7f40ac290 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fe7ec009ce0 tx=0x7fe7ec009220).ready entity=mon.0 client_cookie=29f59f2c5174350e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:47.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.691+0000 7fe7f2ffd700 1 -- 172.21.15.23:0/2104431471 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec00bbf0 con 0x7fe7f40aa1c0 2020-11-20T11:32:47.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.691+0000 7fe7f2ffd700 1 -- 172.21.15.23:0/2104431471 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7ec00bd50 con 0x7fe7f40aa1c0 2020-11-20T11:32:47.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.691+0000 7fe7f2ffd700 1 -- 172.21.15.23:0/2104431471 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec011610 con 0x7fe7f40aa1c0 2020-11-20T11:32:47.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.692+0000 7fe7fa584700 1 -- 172.21.15.23:0/2104431471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7f40aa1c0 msgr2=0x7fe7f40ac290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:47.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.692+0000 7fe7fa584700 1 --2- 172.21.15.23:0/2104431471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7f40aa1c0 0x7fe7f40ac290 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fe7ec009ce0 tx=0x7fe7ec009220).stop 2020-11-20T11:32:47.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.692+0000 7fe7fa584700 1 -- 172.21.15.23:0/2104431471 shutdown_connections 2020-11-20T11:32:47.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.692+0000 7fe7fa584700 1 --2- 172.21.15.23:0/2104431471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7f40aa1c0 0x7fe7f40ac290 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:47.699 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.692+0000 7fe7fa584700 1 -- 172.21.15.23:0/2104431471 >> 172.21.15.23:0/2104431471 conn(0x7fe7f4066730 msgr2=0x7fe7f40601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:47.699 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.692+0000 7fe7fa584700 1 -- 172.21.15.23:0/2104431471 shutdown_connections 2020-11-20T11:32:47.699 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.692+0000 7fe7fa584700 1 -- 172.21.15.23:0/2104431471 wait complete. 2020-11-20T11:32:47.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.693+0000 7fe7fa584700 1 Processor -- start 2020-11-20T11:32:47.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.693+0000 7fe7fa584700 1 -- start start 2020-11-20T11:32:47.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.693+0000 7fe7fa584700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe7f4060a80 0x7fe7f40a6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:47.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.693+0000 7fe7fa584700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe7f40a9d40 0x7fe7f414a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:47.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.693+0000 7fe7fa584700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7f40aa1c0 0x7fe7f40ad720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:47.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.693+0000 7fe7fa584700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f414aed0 con 0x7fe7f40aa1c0 2020-11-20T11:32:47.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.693+0000 7fe7fa584700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f40acfe0 con 0x7fe7f4060a80 2020-11-20T11:32:47.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.693+0000 7fe7fa584700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f40ad280 con 0x7fe7f40a9d40 2020-11-20T11:32:47.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.694+0000 7fe7f8b21700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7f40aa1c0 0x7fe7f40ad720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:47.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.694+0000 7fe7f8b21700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7f40aa1c0 0x7fe7f40ad720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41544/0 (socket says 172.21.15.23:41544) 2020-11-20T11:32:47.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.694+0000 7fe7f8b21700 1 -- 172.21.15.23:0/2514944044 learned_addr learned my addr 172.21.15.23:0/2514944044 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:47.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.694+0000 7fe7f3fff700 1 --2- 172.21.15.23:0/2514944044 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe7f4060a80 0x7fe7f40a6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:47.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.694+0000 7fe7f37fe700 1 --2- 172.21.15.23:0/2514944044 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe7f40a9d40 0x7fe7f414a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:47.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.694+0000 7fe7f8b21700 1 -- 172.21.15.23:0/2514944044 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe7f40a9d40 msgr2=0x7fe7f414a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:47.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.694+0000 7fe7f8b21700 1 --2- 172.21.15.23:0/2514944044 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe7f40a9d40 0x7fe7f414a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:47.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.694+0000 7fe7f8b21700 1 -- 172.21.15.23:0/2514944044 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe7f4060a80 msgr2=0x7fe7f40a6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:47.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.694+0000 7fe7f8b21700 1 --2- 172.21.15.23:0/2514944044 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe7f4060a80 0x7fe7f40a6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:47.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.694+0000 7fe7f8b21700 1 -- 172.21.15.23:0/2514944044 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f4138a10 con 0x7fe7f40aa1c0 2020-11-20T11:32:47.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.694+0000 7fe7f8b21700 1 --2- 172.21.15.23:0/2514944044 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7f40aa1c0 0x7fe7f40ad720 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fe7ec009260 tx=0x7fe7ec016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:47.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.694+0000 7fe7d3fff700 1 -- 172.21.15.23:0/2514944044 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec00bec0 con 0x7fe7f40aa1c0 2020-11-20T11:32:47.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.694+0000 7fe7d3fff700 1 -- 172.21.15.23:0/2514944044 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7ec009e20 con 0x7fe7f40aa1c0 2020-11-20T11:32:47.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.694+0000 7fe7fa584700 1 -- 172.21.15.23:0/2514944044 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7f414aae0 con 0x7fe7f40aa1c0 2020-11-20T11:32:47.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.694+0000 7fe7d3fff700 1 -- 172.21.15.23:0/2514944044 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec010cf0 con 0x7fe7f40aa1c0 2020-11-20T11:32:47.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.694+0000 7fe7fa584700 1 -- 172.21.15.23:0/2514944044 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7f40ade50 con 0x7fe7f40aa1c0 2020-11-20T11:32:47.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.695+0000 7fe7fa584700 1 -- 172.21.15.23:0/2514944044 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7f40ad440 con 0x7fe7f40aa1c0 2020-11-20T11:32:47.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.695+0000 7fe7d3fff700 1 -- 172.21.15.23:0/2514944044 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe7ec017070 con 0x7fe7f40aa1c0 2020-11-20T11:32:47.707 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.696+0000 7fe7d3fff700 1 --2- 172.21.15.23:0/2514944044 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe7cc056d90 0x7fe7cc059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:47.707 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.696+0000 7fe7d3fff700 1 -- 172.21.15.23:0/2514944044 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fe7ec072760 con 0x7fe7f40aa1c0 2020-11-20T11:32:47.707 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.696+0000 7fe7f3fff700 1 --2- 172.21.15.23:0/2514944044 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe7cc056d90 0x7fe7cc059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:47.707 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.696+0000 7fe7f3fff700 1 --2- 172.21.15.23:0/2514944044 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe7cc056d90 0x7fe7cc059250 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fe7e8004610 tx=0x7fe7e8007bc0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:47.708 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.699+0000 7fe7d3fff700 1 -- 172.21.15.23:0/2514944044 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe7ec047a50 con 0x7fe7f40aa1c0 2020-11-20T11:32:47.903 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.900+0000 7fe7fa584700 1 -- 172.21.15.23:0/2514944044 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe7f40ae070 con 0x7fe7f40aa1c0 2020-11-20T11:32:47.904 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.901+0000 7fe7d3fff700 1 -- 172.21.15.23:0/2514944044 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1477 (secure 0 0 0) 0x7fe7ec015020 con 0x7fe7f40aa1c0 2020-11-20T11:32:47.904 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":41,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:32:47.905 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.902+0000 7fe7fa584700 1 -- 172.21.15.23:0/2514944044 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe7cc056d90 msgr2=0x7fe7cc059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:47.906 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.902+0000 7fe7fa584700 1 --2- 172.21.15.23:0/2514944044 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe7cc056d90 0x7fe7cc059250 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fe7e8004610 tx=0x7fe7e8007bc0).stop 2020-11-20T11:32:47.906 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.902+0000 7fe7fa584700 1 -- 172.21.15.23:0/2514944044 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7f40aa1c0 msgr2=0x7fe7f40ad720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:47.906 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.902+0000 7fe7fa584700 1 --2- 172.21.15.23:0/2514944044 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7f40aa1c0 0x7fe7f40ad720 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fe7ec009260 tx=0x7fe7ec016000).stop 2020-11-20T11:32:47.906 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.902+0000 7fe7fa584700 1 -- 172.21.15.23:0/2514944044 shutdown_connections 2020-11-20T11:32:47.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.902+0000 7fe7fa584700 1 --2- 172.21.15.23:0/2514944044 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7f40aa1c0 0x7fe7f40ad720 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:47.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.902+0000 7fe7fa584700 1 --2- 172.21.15.23:0/2514944044 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe7f4060a80 0x7fe7f40a6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:47.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.903+0000 7fe7fa584700 1 --2- 172.21.15.23:0/2514944044 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe7cc056d90 0x7fe7cc059250 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:47.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.903+0000 7fe7fa584700 1 --2- 172.21.15.23:0/2514944044 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe7f40a9d40 0x7fe7f414a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:47.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.903+0000 7fe7fa584700 1 -- 172.21.15.23:0/2514944044 >> 172.21.15.23:0/2514944044 conn(0x7fe7f4066730 msgr2=0x7fe7f4137890 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:47.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.903+0000 7fe7fa584700 1 -- 172.21.15.23:0/2514944044 shutdown_connections 2020-11-20T11:32:47.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:47.903+0000 7fe7fa584700 1 -- 172.21.15.23:0/2514944044 wait complete. 2020-11-20T11:32:47.917 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:32:47.917 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2020-11-20T11:32:47.917 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:32:48.032 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.029+0000 7f431dcd6700 1 Processor -- start 2020-11-20T11:32:48.032 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.029+0000 7f431dcd6700 1 -- start start 2020-11-20T11:32:48.034 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.029+0000 7f431dcd6700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43180ad600 0x7f43180658b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.034 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.029+0000 7f431dcd6700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f43180ad9e0 0x7f431813c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.034 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.029+0000 7f431dcd6700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f431813cd00 0x7f43180a6b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.034 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.029+0000 7f431dcd6700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43180a8b90 con 0x7f431813cd00 2020-11-20T11:32:48.035 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.029+0000 7f431dcd6700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4318065da0 con 0x7f43180ad9e0 2020-11-20T11:32:48.035 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.029+0000 7f431dcd6700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43180a83b0 con 0x7f43180ad600 2020-11-20T11:32:48.035 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.030+0000 7f4317fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f431813cd00 0x7f43180a6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:48.035 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.030+0000 7f4317fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f431813cd00 0x7f43180a6b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41552/0 (socket says 172.21.15.23:41552) 2020-11-20T11:32:48.035 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.030+0000 7f4317fff700 1 -- 172.21.15.23:0/388812763 learned_addr learned my addr 172.21.15.23:0/388812763 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:48.036 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.030+0000 7f4316ffd700 1 --2- 172.21.15.23:0/388812763 >> v2:172.21.15.155:3300/0 conn(0x7f43180ad9e0 0x7f431813c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:48.036 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.030+0000 7f43177fe700 1 --2- 172.21.15.23:0/388812763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43180ad600 0x7f43180658b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:48.036 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.030+0000 7f4317fff700 1 -- 172.21.15.23:0/388812763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43180ad600 msgr2=0x7f43180658b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:48.036 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.030+0000 7f4317fff700 1 --2- 172.21.15.23:0/388812763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43180ad600 0x7f43180658b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:48.037 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.030+0000 7f4317fff700 1 -- 172.21.15.23:0/388812763 >> v2:172.21.15.155:3300/0 conn(0x7f43180ad9e0 msgr2=0x7f431813c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:48.037 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.030+0000 7f4317fff700 1 --2- 172.21.15.23:0/388812763 >> v2:172.21.15.155:3300/0 conn(0x7f43180ad9e0 0x7f431813c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:48.037 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.030+0000 7f4317fff700 1 -- 172.21.15.23:0/388812763 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43180a8650 con 0x7f431813cd00 2020-11-20T11:32:48.037 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.030+0000 7f4317fff700 1 --2- 172.21.15.23:0/388812763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f431813cd00 0x7f43180a6b80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f43180628c0 tx=0x7f4308006ef0).ready entity=mon.0 client_cookie=1fd163a3d8841bb2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:48.038 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.030+0000 7f43167fc700 1 -- 172.21.15.23:0/388812763 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430800b860 con 0x7f431813cd00 2020-11-20T11:32:48.038 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.030+0000 7f43167fc700 1 -- 172.21.15.23:0/388812763 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4308000e90 con 0x7f431813cd00 2020-11-20T11:32:48.038 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.030+0000 7f43167fc700 1 -- 172.21.15.23:0/388812763 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430800bbd0 con 0x7f431813cd00 2020-11-20T11:32:48.038 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.031+0000 7f431dcd6700 1 -- 172.21.15.23:0/388812763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f431813cd00 msgr2=0x7f43180a6b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:48.038 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.031+0000 7f431dcd6700 1 --2- 172.21.15.23:0/388812763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f431813cd00 0x7f43180a6b80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f43180628c0 tx=0x7f4308006ef0).stop 2020-11-20T11:32:48.039 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.031+0000 7f431dcd6700 1 -- 172.21.15.23:0/388812763 shutdown_connections 2020-11-20T11:32:48.039 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.031+0000 7f431dcd6700 1 --2- 172.21.15.23:0/388812763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f431813cd00 0x7f43180a6b80 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:48.039 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.031+0000 7f431dcd6700 1 --2- 172.21.15.23:0/388812763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43180ad600 0x7f43180658b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:48.039 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.031+0000 7f431dcd6700 1 --2- 172.21.15.23:0/388812763 >> v2:172.21.15.155:3300/0 conn(0x7f43180ad9e0 0x7f431813c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:48.039 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.031+0000 7f431dcd6700 1 -- 172.21.15.23:0/388812763 >> 172.21.15.23:0/388812763 conn(0x7f4318060c80 msgr2=0x7f43180acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:48.040 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.031+0000 7f431dcd6700 1 -- 172.21.15.23:0/388812763 shutdown_connections 2020-11-20T11:32:48.040 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.031+0000 7f431dcd6700 1 -- 172.21.15.23:0/388812763 wait complete. 2020-11-20T11:32:48.043 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.032+0000 7f431dcd6700 1 Processor -- start 2020-11-20T11:32:48.043 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.032+0000 7f431dcd6700 1 -- start start 2020-11-20T11:32:48.043 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.032+0000 7f431dcd6700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43180ad600 0x7f43180a2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.043 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.032+0000 7f431dcd6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43180ad9e0 0x7f431813aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.043 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.032+0000 7f431dcd6700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f431813cd00 0x7f4318137470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.044 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.032+0000 7f431dcd6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43180aad20 con 0x7f43180ad9e0 2020-11-20T11:32:48.044 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.032+0000 7f431dcd6700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431813b340 con 0x7f43180ad600 2020-11-20T11:32:48.044 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.032+0000 7f431dcd6700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4318136d30 con 0x7f431813cd00 2020-11-20T11:32:48.044 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.032+0000 7f4316ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43180ad9e0 0x7f431813aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:48.044 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.033+0000 7f4316ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43180ad9e0 0x7f431813aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41554/0 (socket says 172.21.15.23:41554) 2020-11-20T11:32:48.045 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.033+0000 7f4316ffd700 1 -- 172.21.15.23:0/3230036325 learned_addr learned my addr 172.21.15.23:0/3230036325 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:48.045 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.033+0000 7f4317fff700 1 --2- 172.21.15.23:0/3230036325 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f431813cd00 0x7f4318137470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:48.045 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.033+0000 7f43177fe700 1 --2- 172.21.15.23:0/3230036325 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43180ad600 0x7f43180a2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:48.045 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.033+0000 7f4316ffd700 1 -- 172.21.15.23:0/3230036325 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f431813cd00 msgr2=0x7f4318137470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:48.045 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.033+0000 7f4316ffd700 1 --2- 172.21.15.23:0/3230036325 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f431813cd00 0x7f4318137470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:48.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.033+0000 7f4316ffd700 1 -- 172.21.15.23:0/3230036325 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43180ad600 msgr2=0x7f43180a2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:48.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.033+0000 7f4316ffd700 1 --2- 172.21.15.23:0/3230036325 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43180ad600 0x7f43180a2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:48.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.033+0000 7f4316ffd700 1 -- 172.21.15.23:0/3230036325 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4318136fd0 con 0x7f43180ad9e0 2020-11-20T11:32:48.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.033+0000 7f4317fff700 1 --2- 172.21.15.23:0/3230036325 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f431813cd00 0x7f4318137470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:48.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.033+0000 7f4316ffd700 1 --2- 172.21.15.23:0/3230036325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43180ad9e0 0x7f431813aaa0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f430000db30 tx=0x7f4300008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:48.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.033+0000 7f42f77fe700 1 -- 172.21.15.23:0/3230036325 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4300009760 con 0x7f43180ad9e0 2020-11-20T11:32:48.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.033+0000 7f42f77fe700 1 -- 172.21.15.23:0/3230036325 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43000098c0 con 0x7f43180ad9e0 2020-11-20T11:32:48.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.033+0000 7f42f77fe700 1 -- 172.21.15.23:0/3230036325 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4300009c40 con 0x7f43180ad9e0 2020-11-20T11:32:48.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.033+0000 7f431dcd6700 1 -- 172.21.15.23:0/3230036325 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43180a32d0 con 0x7f43180ad9e0 2020-11-20T11:32:48.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.033+0000 7f431dcd6700 1 -- 172.21.15.23:0/3230036325 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43181331b0 con 0x7f43180ad9e0 2020-11-20T11:32:48.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.034+0000 7f431dcd6700 1 --2- 172.21.15.23:0/3230036325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4318137960 0x7f43180ac440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.034+0000 7f431dcd6700 1 -- 172.21.15.23:0/3230036325 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431813b1d0 con 0x7f4318137960 2020-11-20T11:32:48.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.034+0000 7f43177fe700 1 --2- 172.21.15.23:0/3230036325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4318137960 0x7f43180ac440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:48.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.034+0000 7f43177fe700 1 -- 172.21.15.23:0/3230036325 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f431805c520 con 0x7f4318137960 2020-11-20T11:32:48.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.034+0000 7f42f77fe700 1 -- 172.21.15.23:0/3230036325 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4300013480 con 0x7f43180ad9e0 2020-11-20T11:32:48.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.034+0000 7f43177fe700 1 --2- 172.21.15.23:0/3230036325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4318137960 0x7f43180ac440 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f4310010bf0 tx=0x7f4310008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:48.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.035+0000 7f42f77fe700 1 --2- 172.21.15.23:0/3230036325 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f42f0056d90 0x7f42f0059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.035+0000 7f42f77fe700 1 -- 172.21.15.23:0/3230036325 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f4300070350 con 0x7f43180ad9e0 2020-11-20T11:32:48.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.035+0000 7f42f77fe700 1 -- 172.21.15.23:0/3230036325 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431000eae0 con 0x7f4318137960 2020-11-20T11:32:48.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.035+0000 7f42f77fe700 1 -- 172.21.15.23:0/3230036325 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f431000c030 con 0x7f4318137960 2020-11-20T11:32:48.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.035+0000 7f42f77fe700 1 -- 172.21.15.23:0/3230036325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4318137960 msgr2=0x7f43180ac440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:48.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.035+0000 7f42f77fe700 1 --2- 172.21.15.23:0/3230036325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4318137960 0x7f43180ac440 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f4310010bf0 tx=0x7f4310008040).stop 2020-11-20T11:32:48.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.035+0000 7f42f77fe700 1 -- 172.21.15.23:0/3230036325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4318137960 msgr2=0x7f43180ac440 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:32:48.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.035+0000 7f42f77fe700 1 --2- 172.21.15.23:0/3230036325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4318137960 0x7f43180ac440 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:48.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.035+0000 7f4317fff700 1 --2- 172.21.15.23:0/3230036325 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f42f0056d90 0x7f42f0059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:48.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.035+0000 7f4317fff700 1 --2- 172.21.15.23:0/3230036325 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f42f0056d90 0x7f42f0059250 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f431813d100 tx=0x7f4308015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:48.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.041+0000 7f431dcd6700 1 --2- 172.21.15.23:0/3230036325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43180ac930 0x7f43180a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.041+0000 7f431dcd6700 1 -- 172.21.15.23:0/3230036325 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4318044710 con 0x7f43180ac930 2020-11-20T11:32:48.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.042+0000 7f43177fe700 1 --2- 172.21.15.23:0/3230036325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43180ac930 0x7f43180a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:48.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.042+0000 7f43177fe700 1 -- 172.21.15.23:0/3230036325 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f43180687a0 con 0x7f43180ac930 2020-11-20T11:32:48.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.042+0000 7f43177fe700 1 --2- 172.21.15.23:0/3230036325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43180ac930 0x7f43180a18b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f4310010bf0 tx=0x7f4310008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:48.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.042+0000 7f42f77fe700 1 -- 172.21.15.23:0/3230036325 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43100226f0 con 0x7f43180ac930 2020-11-20T11:32:48.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.042+0000 7f42f77fe700 1 -- 172.21.15.23:0/3230036325 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4785 (secure 0 0 0) 0x7f4310026070 con 0x7f43180ac930 2020-11-20T11:32:48.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.042+0000 7f42f77fe700 1 -- 172.21.15.23:0/3230036325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43180ac930 msgr2=0x7f43180a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:48.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.042+0000 7f42f77fe700 1 --2- 172.21.15.23:0/3230036325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43180ac930 0x7f43180a18b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f4310010bf0 tx=0x7f4310008040).stop 2020-11-20T11:32:48.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.042+0000 7f42f77fe700 1 -- 172.21.15.23:0/3230036325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43180ac930 msgr2=0x7f43180a18b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:32:48.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.042+0000 7f42f77fe700 1 --2- 172.21.15.23:0/3230036325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43180ac930 0x7f43180a18b0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:48.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.042+0000 7f431dcd6700 1 -- 172.21.15.23:0/3230036325 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f42f0056d90 msgr2=0x7f42f0059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:48.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.043+0000 7f431dcd6700 1 --2- 172.21.15.23:0/3230036325 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f42f0056d90 0x7f42f0059250 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f431813d100 tx=0x7f4308015000).stop 2020-11-20T11:32:48.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.043+0000 7f431dcd6700 1 -- 172.21.15.23:0/3230036325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43180ad9e0 msgr2=0x7f431813aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:48.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.043+0000 7f431dcd6700 1 --2- 172.21.15.23:0/3230036325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43180ad9e0 0x7f431813aaa0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f430000db30 tx=0x7f4300008040).stop 2020-11-20T11:32:48.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.043+0000 7f4317fff700 1 -- 172.21.15.23:0/3230036325 reap_dead start 2020-11-20T11:32:48.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.043+0000 7f4317fff700 1 -- 172.21.15.23:0/3230036325 reap_dead start 2020-11-20T11:32:48.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.043+0000 7f431dcd6700 1 -- 172.21.15.23:0/3230036325 shutdown_connections 2020-11-20T11:32:48.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.043+0000 7f431dcd6700 1 -- 172.21.15.23:0/3230036325 >> 172.21.15.23:0/3230036325 conn(0x7f4318060c80 msgr2=0x7f43180a77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:48.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.043+0000 7f431dcd6700 1 -- 172.21.15.23:0/3230036325 shutdown_connections 2020-11-20T11:32:48.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.043+0000 7f431dcd6700 1 -- 172.21.15.23:0/3230036325 wait complete. 2020-11-20T11:32:48.056 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:32:48.057 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:32:48.057 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:32:48.057 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:32:48.057 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 4, 2020-11-20T11:32:48.057 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:32:48.058 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:32:48.058 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:32:48.058 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:32:48.058 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:32:48.058 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 41, 2020-11-20T11:32:48.059 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:32:48.059 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:32:48.059 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:32:48.059 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:32:48.059 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:32:48.060 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:32:48.060 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:32:48.060 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:32:48.060 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:32:48.061 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:32:48.061 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:32:48.061 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:32:48.061 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:32:48.061 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:32:48.062 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:32:48.062 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:32:48.062 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:32:48.062 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:32:48.062 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:32:48.063 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:32:48.063 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:32:48.063 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:32:48.063 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:32:48.063 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.063 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:32:48.064 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:32:48.064 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:32:48.064 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:32:48.064 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:32:48.064 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:32:48.065 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:32:48.065 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:32:48.065 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:32:48.065 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:32:48.065 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:32:48.066 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:32:48.066 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:32:48.066 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:32:48.066 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:32:48.066 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:32:48.067 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:32:48.067 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:32:48.067 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:32:48.067 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:32:48.067 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:32:48.067 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:32:48.068 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:32:48.068 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:32:48.068 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:32:48.068 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.069 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:32:48.069 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.069 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:32:48.069 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:32:48.069 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:32:48.070 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:32:48.070 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.070 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:32:48.070 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:32:48.070 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:32:48.071 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.071 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.071 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:32:48.071 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:32:48.071 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:32:48.072 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:32:48.072 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:32:48.072 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.072 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:32:48.072 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:32:48.073 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:32:48.073 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:32:48.073 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:32:48.073 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.073 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.074 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:32:48.074 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:32:48.074 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:32:48.074 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:32:48.074 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.075 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:32:48.075 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:32:48.075 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:32:48.075 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.075 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.075 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:32:48.076 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:32:48.076 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:32:48.076 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:32:48.076 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:32:48.076 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.077 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:32:48.077 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:32:48.077 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:32:48.077 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:32:48.077 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:32:48.078 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.078 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.078 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:32:48.078 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:32:48.078 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:32:48.078 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:32:48.079 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.079 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:32:48.079 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:32:48.079 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:32:48.079 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.080 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.080 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:32:48.080 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:32:48.080 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:32:48.080 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:32:48.081 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:32:48.081 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.081 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:32:48.081 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:32:48.081 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:32:48.082 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:32:48.082 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:32:48.082 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:32:48.082 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:32:48.082 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.082 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:32:48.083 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:32:48.083 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.083 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:32:48.083 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:32:48.083 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:32:48.084 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:32:48.084 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:32:48.084 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:32:48.084 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.084 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:32:48.085 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:32:48.085 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:32:48.085 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:32:48.085 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:32:48.085 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:32:48.085 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.086 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:32:48.086 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:32:48.086 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:32:48.086 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:32:48.086 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:32:48.087 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:32:48.087 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.087 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:32:48.087 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:32:48.087 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:32:48.088 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:32:48.088 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:32:48.088 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:32:48.088 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.088 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:32:48.089 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:32:48.089 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:32:48.089 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:32:48.089 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:32:48.089 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.089 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:32:48.090 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:32:48.091 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:32:48.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.203+0000 7fa33e0b6700 1 Processor -- start 2020-11-20T11:32:48.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.203+0000 7fa33e0b6700 1 -- start start 2020-11-20T11:32:48.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.204+0000 7fa33e0b6700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa3380ae760 0x7fa33813e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.204+0000 7fa33e0b6700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa3380aeb40 0x7fa338137ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.204+0000 7fa33e0b6700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3380661b0 con 0x7fa338065100 2020-11-20T11:32:48.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.204+0000 7fa33e0b6700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa33813ad60 con 0x7fa3380ae760 2020-11-20T11:32:48.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.204+0000 7fa33e0b6700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa338066c70 con 0x7fa3380aeb40 2020-11-20T11:32:48.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.204+0000 7fa3377fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa3380ae760 0x7fa33813e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:48.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.204+0000 7fa3377fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa3380ae760 0x7fa33813e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41566/0 (socket says 172.21.15.23:41566) 2020-11-20T11:32:48.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.204+0000 7fa337fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fa338065100 0x7fa338062b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55820/0 (socket says 172.21.15.23:55820) 2020-11-20T11:32:48.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.204+0000 7fa3377fe700 1 -- 172.21.15.23:0/2752340712 learned_addr learned my addr 172.21.15.23:0/2752340712 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:48.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.204+0000 7fa336ffd700 1 --2- 172.21.15.23:0/2752340712 >> v2:172.21.15.155:3300/0 conn(0x7fa3380aeb40 0x7fa338137ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:48.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.204+0000 7fa3377fe700 1 -- 172.21.15.23:0/2752340712 >> v2:172.21.15.155:3300/0 conn(0x7fa3380aeb40 msgr2=0x7fa338137ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:48.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.204+0000 7fa3377fe700 1 --2- 172.21.15.23:0/2752340712 >> v2:172.21.15.155:3300/0 conn(0x7fa3380aeb40 0x7fa338137ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:48.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.204+0000 7fa3377fe700 1 -- 172.21.15.23:0/2752340712 >> v1:172.21.15.23:6789/0 conn(0x7fa338065100 legacy=0x7fa338062b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:48.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.204+0000 7fa3377fe700 1 -- 172.21.15.23:0/2752340712 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa33813e470 con 0x7fa3380ae760 2020-11-20T11:32:48.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.204+0000 7fa3367fc700 1 -- 172.21.15.23:0/2752340712 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 251473567 0 0) 0x7fa3380661b0 con 0x7fa338065100 2020-11-20T11:32:48.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.204+0000 7fa336ffd700 1 --2- 172.21.15.23:0/2752340712 >> v2:172.21.15.155:3300/0 conn(0x7fa3380aeb40 0x7fa338137ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:48.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.204+0000 7fa3377fe700 1 --2- 172.21.15.23:0/2752340712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3380ae760 0x7fa33813e910 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fa3380af0c0 tx=0x7fa328006ef0).ready entity=mon.0 client_cookie=dd02c3904ce2ec76 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:48.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.204+0000 7fa3367fc700 1 -- 172.21.15.23:0/2752340712 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32800b860 con 0x7fa3380ae760 2020-11-20T11:32:48.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.205+0000 7fa3367fc700 1 -- 172.21.15.23:0/2752340712 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa328000e90 con 0x7fa3380ae760 2020-11-20T11:32:48.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.205+0000 7fa3367fc700 1 -- 172.21.15.23:0/2752340712 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32800bbd0 con 0x7fa3380ae760 2020-11-20T11:32:48.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.205+0000 7fa33e0b6700 1 -- 172.21.15.23:0/2752340712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3380ae760 msgr2=0x7fa33813e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:48.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.205+0000 7fa33e0b6700 1 --2- 172.21.15.23:0/2752340712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3380ae760 0x7fa33813e910 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fa3380af0c0 tx=0x7fa328006ef0).stop 2020-11-20T11:32:48.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.205+0000 7fa33e0b6700 1 -- 172.21.15.23:0/2752340712 shutdown_connections 2020-11-20T11:32:48.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.205+0000 7fa33e0b6700 1 --2- 172.21.15.23:0/2752340712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3380ae760 0x7fa33813e910 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:48.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.205+0000 7fa33e0b6700 1 --2- 172.21.15.23:0/2752340712 >> v2:172.21.15.155:3300/0 conn(0x7fa3380aeb40 0x7fa338137ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:48.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.205+0000 7fa33e0b6700 1 -- 172.21.15.23:0/2752340712 >> 172.21.15.23:0/2752340712 conn(0x7fa338061d80 msgr2=0x7fa3380ae050 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:48.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.205+0000 7fa33e0b6700 1 -- 172.21.15.23:0/2752340712 shutdown_connections 2020-11-20T11:32:48.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.205+0000 7fa33e0b6700 1 -- 172.21.15.23:0/2752340712 wait complete. 2020-11-20T11:32:48.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.206+0000 7fa33e0b6700 1 Processor -- start 2020-11-20T11:32:48.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.206+0000 7fa33e0b6700 1 -- start start 2020-11-20T11:32:48.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.207+0000 7fa33e0b6700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa338065100 0x7fa33813dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.207+0000 7fa33e0b6700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3380ae760 0x7fa3380a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.207+0000 7fa33e0b6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3380aeb40 0x7fa3380a2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.207+0000 7fa33e0b6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3380a75b0 con 0x7fa3380aeb40 2020-11-20T11:32:48.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.207+0000 7fa33e0b6700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3380a7810 con 0x7fa338065100 2020-11-20T11:32:48.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.207+0000 7fa33e0b6700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3380a7ab0 con 0x7fa3380ae760 2020-11-20T11:32:48.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.207+0000 7fa337fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3380aeb40 0x7fa3380a2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:48.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.207+0000 7fa337fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3380aeb40 0x7fa3380a2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41576/0 (socket says 172.21.15.23:41576) 2020-11-20T11:32:48.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.207+0000 7fa337fff700 1 -- 172.21.15.23:0/4134365450 learned_addr learned my addr 172.21.15.23:0/4134365450 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:48.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.207+0000 7fa3377fe700 1 --2- 172.21.15.23:0/4134365450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa338065100 0x7fa33813dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:48.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.207+0000 7fa336ffd700 1 --2- 172.21.15.23:0/4134365450 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3380ae760 0x7fa3380a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:48.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.207+0000 7fa337fff700 1 -- 172.21.15.23:0/4134365450 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3380ae760 msgr2=0x7fa3380a63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:48.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.207+0000 7fa337fff700 1 --2- 172.21.15.23:0/4134365450 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3380ae760 0x7fa3380a63f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:48.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.207+0000 7fa337fff700 1 -- 172.21.15.23:0/4134365450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa338065100 msgr2=0x7fa33813dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:48.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.207+0000 7fa337fff700 1 --2- 172.21.15.23:0/4134365450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa338065100 0x7fa33813dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:48.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.207+0000 7fa337fff700 1 -- 172.21.15.23:0/4134365450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3380a5950 con 0x7fa3380aeb40 2020-11-20T11:32:48.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.207+0000 7fa3377fe700 1 --2- 172.21.15.23:0/4134365450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa338065100 0x7fa33813dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:32:48.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.207+0000 7fa337fff700 1 --2- 172.21.15.23:0/4134365450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3380aeb40 0x7fa3380a2920 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fa33000cae0 tx=0x7fa330001360).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:48.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.207+0000 7fa3177fe700 1 -- 172.21.15.23:0/4134365450 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa330005b00 con 0x7fa3380aeb40 2020-11-20T11:32:48.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.207+0000 7fa3177fe700 1 -- 172.21.15.23:0/4134365450 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa330005c60 con 0x7fa3380aeb40 2020-11-20T11:32:48.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.207+0000 7fa33e0b6700 1 -- 172.21.15.23:0/4134365450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3380a7f50 con 0x7fa3380aeb40 2020-11-20T11:32:48.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.207+0000 7fa3177fe700 1 -- 172.21.15.23:0/4134365450 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa330007110 con 0x7fa3380aeb40 2020-11-20T11:32:48.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.208+0000 7fa33e0b6700 1 -- 172.21.15.23:0/4134365450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3380a6b20 con 0x7fa3380aeb40 2020-11-20T11:32:48.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.208+0000 7fa33e0b6700 1 --2- 172.21.15.23:0/4134365450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3380a2e70 0x7fa338134410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.208+0000 7fa33e0b6700 1 -- 172.21.15.23:0/4134365450 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa33805c280 con 0x7fa3380a2e70 2020-11-20T11:32:48.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.208+0000 7fa3377fe700 1 --2- 172.21.15.23:0/4134365450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3380a2e70 0x7fa338134410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:48.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.209+0000 7fa3177fe700 1 -- 172.21.15.23:0/4134365450 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa330010030 con 0x7fa3380aeb40 2020-11-20T11:32:48.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.209+0000 7fa3377fe700 1 -- 172.21.15.23:0/4134365450 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3380a7cb0 con 0x7fa3380a2e70 2020-11-20T11:32:48.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.209+0000 7fa3377fe700 1 --2- 172.21.15.23:0/4134365450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3380a2e70 0x7fa338134410 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fa3380af0c0 tx=0x7fa328016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:48.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.209+0000 7fa3177fe700 1 --2- 172.21.15.23:0/4134365450 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa310056d60 0x7fa310059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.209+0000 7fa3177fe700 1 -- 172.21.15.23:0/4134365450 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fa330070080 con 0x7fa3380aeb40 2020-11-20T11:32:48.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.209+0000 7fa3177fe700 1 -- 172.21.15.23:0/4134365450 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa328021070 con 0x7fa3380a2e70 2020-11-20T11:32:48.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.209+0000 7fa336ffd700 1 --2- 172.21.15.23:0/4134365450 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa310056d60 0x7fa310059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:48.225 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.210+0000 7fa3177fe700 1 -- 172.21.15.23:0/4134365450 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fa328011bd0 con 0x7fa3380a2e70 2020-11-20T11:32:48.225 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.210+0000 7fa3177fe700 1 -- 172.21.15.23:0/4134365450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3380a2e70 msgr2=0x7fa338134410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:48.225 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.210+0000 7fa3177fe700 1 --2- 172.21.15.23:0/4134365450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3380a2e70 0x7fa338134410 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fa3380af0c0 tx=0x7fa328016000).stop 2020-11-20T11:32:48.225 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.210+0000 7fa3177fe700 1 -- 172.21.15.23:0/4134365450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3380a2e70 msgr2=0x7fa338134410 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:32:48.225 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.210+0000 7fa3177fe700 1 --2- 172.21.15.23:0/4134365450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3380a2e70 0x7fa338134410 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:48.226 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.210+0000 7fa336ffd700 1 --2- 172.21.15.23:0/4134365450 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa310056d60 0x7fa310059220 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fa32c006c10 tx=0x7fa32c008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:48.227 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.216+0000 7fa33e0b6700 1 --2- 172.21.15.23:0/4134365450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa338134900 0x7fa3380ace30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.228 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.216+0000 7fa33e0b6700 1 -- 172.21.15.23:0/4134365450 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa338062760 con 0x7fa338134900 2020-11-20T11:32:48.228 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.217+0000 7fa3377fe700 1 --2- 172.21.15.23:0/4134365450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa338134900 0x7fa3380ace30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:48.228 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.217+0000 7fa3377fe700 1 -- 172.21.15.23:0/4134365450 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa33805c7d0 con 0x7fa338134900 2020-11-20T11:32:48.228 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.217+0000 7fa3377fe700 1 --2- 172.21.15.23:0/4134365450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa338134900 0x7fa3380ace30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fa328006f30 tx=0x7fa32801d000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:48.228 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.217+0000 7fa3177fe700 1 -- 172.21.15.23:0/4134365450 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32801c740 con 0x7fa338134900 2020-11-20T11:32:48.229 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.218+0000 7fa3177fe700 1 -- 172.21.15.23:0/4134365450 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4443 (secure 0 0 0) 0x7fa3280119c0 con 0x7fa338134900 2020-11-20T11:32:48.229 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.218+0000 7fa3177fe700 1 -- 172.21.15.23:0/4134365450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa338134900 msgr2=0x7fa3380ace30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:48.229 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.218+0000 7fa3177fe700 1 --2- 172.21.15.23:0/4134365450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa338134900 0x7fa3380ace30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fa328006f30 tx=0x7fa32801d000).stop 2020-11-20T11:32:48.229 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.218+0000 7fa3177fe700 1 -- 172.21.15.23:0/4134365450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa338134900 msgr2=0x7fa3380ace30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:32:48.229 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.218+0000 7fa3177fe700 1 --2- 172.21.15.23:0/4134365450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa338134900 0x7fa3380ace30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:48.230 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.218+0000 7fa33e0b6700 1 -- 172.21.15.23:0/4134365450 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa310056d60 msgr2=0x7fa310059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:48.230 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.218+0000 7fa33e0b6700 1 --2- 172.21.15.23:0/4134365450 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa310056d60 0x7fa310059220 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fa32c006c10 tx=0x7fa32c008040).stop 2020-11-20T11:32:48.230 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.218+0000 7fa33e0b6700 1 -- 172.21.15.23:0/4134365450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3380aeb40 msgr2=0x7fa3380a2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:48.230 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.218+0000 7fa33e0b6700 1 --2- 172.21.15.23:0/4134365450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3380aeb40 0x7fa3380a2920 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fa33000cae0 tx=0x7fa330001360).stop 2020-11-20T11:32:48.230 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.218+0000 7fa337fff700 1 -- 172.21.15.23:0/4134365450 reap_dead start 2020-11-20T11:32:48.231 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.218+0000 7fa337fff700 1 -- 172.21.15.23:0/4134365450 reap_dead start 2020-11-20T11:32:48.231 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.218+0000 7fa33e0b6700 1 -- 172.21.15.23:0/4134365450 shutdown_connections 2020-11-20T11:32:48.231 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.218+0000 7fa33e0b6700 1 -- 172.21.15.23:0/4134365450 >> 172.21.15.23:0/4134365450 conn(0x7fa338061d80 msgr2=0x7fa3380632c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:48.231 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.218+0000 7fa33e0b6700 1 -- 172.21.15.23:0/4134365450 shutdown_connections 2020-11-20T11:32:48.231 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.218+0000 7fa33e0b6700 1 -- 172.21.15.23:0/4134365450 wait complete. 2020-11-20T11:32:48.232 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:32:48.232 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:32:48.232 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:32:48.232 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:32:48.232 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 4, 2020-11-20T11:32:48.233 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:32:48.233 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:32:48.233 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:32:48.233 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:32:48.233 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:32:48.234 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 42, 2020-11-20T11:32:48.234 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:32:48.235 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:32:48.235 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:32:48.235 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:32:48.235 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:32:48.235 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:32:48.236 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:32:48.236 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:32:48.236 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:32:48.236 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:32:48.236 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:32:48.236 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:32:48.237 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:32:48.237 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:32:48.237 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:32:48.237 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:32:48.238 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:32:48.238 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:32:48.238 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:32:48.238 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:32:48.239 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:32:48.239 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:32:48.239 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:32:48.239 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.239 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:32:48.240 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:32:48.240 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:32:48.240 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:32:48.240 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:32:48.240 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:32:48.241 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:32:48.241 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:32:48.241 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:32:48.241 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:32:48.241 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:32:48.242 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:32:48.242 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:32:48.242 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:32:48.242 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:32:48.242 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:32:48.243 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:32:48.243 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:32:48.243 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:32:48.243 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:32:48.243 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:32:48.243 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:32:48.244 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:32:48.244 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:32:48.244 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:32:48.245 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.245 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:32:48.245 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.245 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:32:48.245 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:32:48.246 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:32:48.246 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:32:48.246 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.246 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:32:48.246 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:32:48.247 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:32:48.247 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.247 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.247 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:32:48.247 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:32:48.248 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:32:48.248 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:32:48.248 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:32:48.248 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.248 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:32:48.248 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:32:48.249 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:32:48.249 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:32:48.249 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:32:48.249 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.250 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.250 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:32:48.250 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:32:48.250 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:32:48.251 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:32:48.251 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.251 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:32:48.251 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:32:48.251 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:32:48.252 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.252 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.252 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:32:48.252 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:32:48.252 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:32:48.252 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:32:48.253 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:32:48.253 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.253 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:32:48.253 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:32:48.253 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:32:48.254 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:32:48.254 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:32:48.254 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.254 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.254 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:32:48.255 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:32:48.255 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:32:48.256 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:32:48.256 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.256 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:32:48.256 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:32:48.256 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:32:48.257 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.257 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.257 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:32:48.257 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:32:48.258 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:32:48.258 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:32:48.258 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:32:48.258 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.258 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:32:48.259 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:32:48.259 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:32:48.259 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:32:48.259 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:32:48.259 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:32:48.259 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:32:48.260 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.260 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:32:48.260 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:32:48.260 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.260 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:32:48.261 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:32:48.261 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:32:48.261 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:32:48.261 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:32:48.261 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:32:48.262 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.262 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:32:48.262 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:32:48.262 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:32:48.262 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:32:48.263 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:32:48.263 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:32:48.263 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.263 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:32:48.263 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:32:48.263 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 4 2020-11-20T11:32:48.264 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:32:48.264 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:32:48.264 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.264 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:32:48.264 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:32:48.266 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:32:48.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.378+0000 7f2a47fff700 1 Processor -- start 2020-11-20T11:32:48.381 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.378+0000 7f2a47fff700 1 -- start start 2020-11-20T11:32:48.384 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.378+0000 7f2a47fff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f2a480a2460 0x7f2a48137df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.384 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.378+0000 7f2a47fff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a480aaab0 con 0x7f2a480a2080 2020-11-20T11:32:48.385 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.378+0000 7f2a47fff700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a480aad50 con 0x7f2a48067a10 2020-11-20T11:32:48.385 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.378+0000 7f2a47fff700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a4813aa50 con 0x7f2a480a2460 2020-11-20T11:32:48.385 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.378+0000 7f2a46ffd700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f2a480a2080 0x7f2a480aef10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:55834/0 (socket says 172.21.15.23:55834) 2020-11-20T11:32:48.385 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.378+0000 7f2a46ffd700 1 -- 172.21.15.23:0/3994590673 learned_addr learned my addr 172.21.15.23:0/3994590673 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:32:48.386 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.379+0000 7f2a467fc700 1 --2- 172.21.15.23:0/3994590673 >> v2:172.21.15.155:3300/0 conn(0x7f2a480a2460 0x7f2a48137df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:48.386 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.379+0000 7f2a45ffb700 1 -- 172.21.15.23:0/3994590673 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3494328052 0 0) 0x7f2a480aaab0 con 0x7f2a480a2080 2020-11-20T11:32:48.386 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.379+0000 7f2a45ffb700 1 -- 172.21.15.23:0/3994590673 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a34003280 con 0x7f2a480a2080 2020-11-20T11:32:48.386 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.379+0000 7f2a45ffb700 1 -- 172.21.15.23:0/3994590673 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 558096783 0 0) 0x7f2a480aad50 con 0x7f2a48067a10 2020-11-20T11:32:48.386 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.379+0000 7f2a45ffb700 1 -- 172.21.15.23:0/3994590673 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a480aaab0 con 0x7f2a48067a10 2020-11-20T11:32:48.387 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.379+0000 7f2a45ffb700 1 -- 172.21.15.23:0/3994590673 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 504062100 0 0) 0x7f2a34003280 con 0x7f2a480a2080 2020-11-20T11:32:48.387 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.379+0000 7f2a45ffb700 1 -- 172.21.15.23:0/3994590673 >> v2:172.21.15.155:3300/0 conn(0x7f2a480a2460 msgr2=0x7f2a48137df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:48.387 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.379+0000 7f2a45ffb700 1 --2- 172.21.15.23:0/3994590673 >> v2:172.21.15.155:3300/0 conn(0x7f2a480a2460 0x7f2a48137df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:48.387 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.379+0000 7f2a45ffb700 1 -- 172.21.15.23:0/3994590673 >> v1:172.21.15.155:6789/0 conn(0x7f2a48067a10 legacy=0x7f2a481453c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:48.387 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.379+0000 7f2a45ffb700 1 -- 172.21.15.23:0/3994590673 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a4813b920 con 0x7f2a480a2080 2020-11-20T11:32:48.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.379+0000 7f2a45ffb700 1 -- 172.21.15.23:0/3994590673 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f2a3c0016a0 con 0x7f2a480a2080 2020-11-20T11:32:48.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.379+0000 7f2a45ffb700 1 -- 172.21.15.23:0/3994590673 >> v1:172.21.15.23:6789/0 conn(0x7f2a480a2080 legacy=0x7f2a480aef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:48.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.379+0000 7f2a45ffb700 1 --2- 172.21.15.23:0/3994590673 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a34003950 0x7f2a34003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.379+0000 7f2a45ffb700 1 --2- 172.21.15.23:0/3994590673 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a34004520 0x7f2a340049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.379+0000 7f2a45ffb700 1 --2- 172.21.15.23:0/3994590673 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a34004e90 0x7f2a3400b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.379+0000 7f2a45ffb700 1 -- 172.21.15.23:0/3994590673 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3400bc60 con 0x7f2a34004e90 2020-11-20T11:32:48.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.379+0000 7f2a45ffb700 1 -- 172.21.15.23:0/3994590673 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3400bda0 con 0x7f2a34004520 2020-11-20T11:32:48.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.379+0000 7f2a45ffb700 1 -- 172.21.15.23:0/3994590673 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3400bf10 con 0x7f2a34003950 2020-11-20T11:32:48.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.379+0000 7f2a467fc700 1 --2- 172.21.15.23:0/3994590673 >> v2:172.21.15.155:3300/0 conn(0x7f2a480a2460 0x7f2a48137df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:32:48.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.379+0000 7f2a477fe700 1 --2- 172.21.15.23:0/3994590673 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a34004e90 0x7f2a3400b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:48.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.379+0000 7f2a46ffd700 1 --2- 172.21.15.23:0/3994590673 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a34003950 0x7f2a34003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:48.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.379+0000 7f2a467fc700 1 --2- 172.21.15.23:0/3994590673 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a34004520 0x7f2a340049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:48.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.380+0000 7f2a477fe700 1 -- 172.21.15.23:0/3994590673 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a34003950 msgr2=0x7f2a34003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:48.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.380+0000 7f2a477fe700 1 --2- 172.21.15.23:0/3994590673 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a34003950 0x7f2a34003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:48.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.380+0000 7f2a477fe700 1 -- 172.21.15.23:0/3994590673 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a34004520 msgr2=0x7f2a340049a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:48.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.380+0000 7f2a477fe700 1 --2- 172.21.15.23:0/3994590673 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a34004520 0x7f2a340049a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:48.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.380+0000 7f2a477fe700 1 -- 172.21.15.23:0/3994590673 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a34003280 con 0x7f2a34004e90 2020-11-20T11:32:48.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.380+0000 7f2a467fc700 1 --2- 172.21.15.23:0/3994590673 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a34004520 0x7f2a340049a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:32:48.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.380+0000 7f2a477fe700 1 -- 172.21.15.23:0/3994590673 reap_dead start 2020-11-20T11:32:48.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.380+0000 7f2a477fe700 1 --2- 172.21.15.23:0/3994590673 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a34004e90 0x7f2a3400b770 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f2a40002090 tx=0x7f2a40007f50).ready entity=mon.0 client_cookie=69475bb0f28dc866 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:48.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.380+0000 7f2a45ffb700 1 -- 172.21.15.23:0/3994590673 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a4000cbf0 con 0x7f2a34004e90 2020-11-20T11:32:48.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.380+0000 7f2a45ffb700 1 -- 172.21.15.23:0/3994590673 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a4000cd50 con 0x7f2a34004e90 2020-11-20T11:32:48.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.380+0000 7f2a45ffb700 1 -- 172.21.15.23:0/3994590673 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a40006ae0 con 0x7f2a34004e90 2020-11-20T11:32:48.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.380+0000 7f2a47fff700 1 -- 172.21.15.23:0/3994590673 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a34004e90 msgr2=0x7f2a3400b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:48.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.380+0000 7f2a47fff700 1 --2- 172.21.15.23:0/3994590673 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a34004e90 0x7f2a3400b770 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f2a40002090 tx=0x7f2a40007f50).stop 2020-11-20T11:32:48.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.380+0000 7f2a47fff700 1 -- 172.21.15.23:0/3994590673 shutdown_connections 2020-11-20T11:32:48.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.380+0000 7f2a47fff700 1 --2- 172.21.15.23:0/3994590673 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a34004e90 0x7f2a3400b770 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:48.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.380+0000 7f2a47fff700 1 -- 172.21.15.23:0/3994590673 >> 172.21.15.23:0/3994590673 conn(0x7f2a48063980 msgr2=0x7f2a48063d90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:48.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.380+0000 7f2a47fff700 1 -- 172.21.15.23:0/3994590673 shutdown_connections 2020-11-20T11:32:48.394 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.380+0000 7f2a47fff700 1 -- 172.21.15.23:0/3994590673 wait complete. 2020-11-20T11:32:48.394 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.381+0000 7f2a47fff700 1 Processor -- start 2020-11-20T11:32:48.394 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.382+0000 7f2a47fff700 1 -- start start 2020-11-20T11:32:48.402 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.382+0000 7f2a47fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a34004e90 0x7f2a480a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.402 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.382+0000 7f2a47fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a480a8a00 0x7f2a4813d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.403 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.382+0000 7f2a47fff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a480a6300 0x7f2a480a6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.403 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.382+0000 7f2a47fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a480a78e0 con 0x7f2a34004e90 2020-11-20T11:32:48.403 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.382+0000 7f2a47fff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a480a7b80 con 0x7f2a480a8a00 2020-11-20T11:32:48.403 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.382+0000 7f2a47fff700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4813be50 con 0x7f2a480a6300 2020-11-20T11:32:48.403 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.382+0000 7f2a46ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a34004e90 0x7f2a480a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:48.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.382+0000 7f2a46ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a34004e90 0x7f2a480a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41596/0 (socket says 172.21.15.23:41596) 2020-11-20T11:32:48.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.382+0000 7f2a46ffd700 1 -- 172.21.15.23:0/2524065920 learned_addr learned my addr 172.21.15.23:0/2524065920 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:48.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.382+0000 7f2a477fe700 1 --2- 172.21.15.23:0/2524065920 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a480a6300 0x7f2a480a6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:48.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.382+0000 7f2a467fc700 1 --2- 172.21.15.23:0/2524065920 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a480a8a00 0x7f2a4813d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:48.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.382+0000 7f2a46ffd700 1 -- 172.21.15.23:0/2524065920 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a480a6300 msgr2=0x7f2a480a6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:48.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.382+0000 7f2a46ffd700 1 --2- 172.21.15.23:0/2524065920 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a480a6300 0x7f2a480a6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:48.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.382+0000 7f2a46ffd700 1 -- 172.21.15.23:0/2524065920 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a480a8a00 msgr2=0x7f2a4813d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:48.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.382+0000 7f2a46ffd700 1 --2- 172.21.15.23:0/2524065920 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a480a8a00 0x7f2a4813d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:48.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.382+0000 7f2a46ffd700 1 -- 172.21.15.23:0/2524065920 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a480a59f0 con 0x7f2a34004e90 2020-11-20T11:32:48.407 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.382+0000 7f2a467fc700 1 --2- 172.21.15.23:0/2524065920 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2a480a8a00 0x7f2a4813d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:32:48.407 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.383+0000 7f2a46ffd700 1 --2- 172.21.15.23:0/2524065920 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a34004e90 0x7f2a480a4d00 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f2a3c00fb80 tx=0x7f2a3c009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:48.407 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.383+0000 7f2a2effd700 1 -- 172.21.15.23:0/2524065920 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c0070d0 con 0x7f2a34004e90 2020-11-20T11:32:48.407 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.383+0000 7f2a2effd700 1 -- 172.21.15.23:0/2524065920 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a3c007250 con 0x7f2a34004e90 2020-11-20T11:32:48.407 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.383+0000 7f2a47fff700 1 -- 172.21.15.23:0/2524065920 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a4813d800 con 0x7f2a34004e90 2020-11-20T11:32:48.408 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.383+0000 7f2a2effd700 1 -- 172.21.15.23:0/2524065920 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c0073c0 con 0x7f2a34004e90 2020-11-20T11:32:48.408 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.383+0000 7f2a47fff700 1 -- 172.21.15.23:0/2524065920 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a480a85a0 con 0x7f2a34004e90 2020-11-20T11:32:48.408 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.384+0000 7f2a47fff700 1 --2- 172.21.15.23:0/2524065920 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a480a6c50 0x7f2a481359f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.408 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.384+0000 7f2a47fff700 1 -- 172.21.15.23:0/2524065920 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a480a5b80 con 0x7f2a480a6c50 2020-11-20T11:32:48.408 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.384+0000 7f2a467fc700 1 --2- 172.21.15.23:0/2524065920 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a480a6c50 0x7f2a481359f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:48.410 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.384+0000 7f2a2effd700 1 -- 172.21.15.23:0/2524065920 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2a3c01a030 con 0x7f2a34004e90 2020-11-20T11:32:48.410 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.384+0000 7f2a467fc700 1 -- 172.21.15.23:0/2524065920 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a480a7d40 con 0x7f2a480a6c50 2020-11-20T11:32:48.410 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.384+0000 7f2a467fc700 1 --2- 172.21.15.23:0/2524065920 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a480a6c50 0x7f2a481359f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f2a4813ce10 tx=0x7f2a38008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:48.410 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.384+0000 7f2a2effd700 1 --2- 172.21.15.23:0/2524065920 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2a24056d60 0x7f2a24059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.411 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.384+0000 7f2a2effd700 1 -- 172.21.15.23:0/2524065920 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f2a3c06f8f0 con 0x7f2a34004e90 2020-11-20T11:32:48.411 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.385+0000 7f2a2effd700 1 -- 172.21.15.23:0/2524065920 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3800f710 con 0x7f2a480a6c50 2020-11-20T11:32:48.411 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.385+0000 7f2a477fe700 1 --2- 172.21.15.23:0/2524065920 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2a24056d60 0x7f2a24059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:48.411 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.385+0000 7f2a2effd700 1 -- 172.21.15.23:0/2524065920 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f2a3800f870 con 0x7f2a480a6c50 2020-11-20T11:32:48.411 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.385+0000 7f2a2effd700 1 -- 172.21.15.23:0/2524065920 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a480a6c50 msgr2=0x7f2a481359f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:48.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.385+0000 7f2a2effd700 1 --2- 172.21.15.23:0/2524065920 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a480a6c50 0x7f2a481359f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f2a4813ce10 tx=0x7f2a38008040).stop 2020-11-20T11:32:48.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.385+0000 7f2a2effd700 1 -- 172.21.15.23:0/2524065920 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a480a6c50 msgr2=0x7f2a481359f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:32:48.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.385+0000 7f2a2effd700 1 --2- 172.21.15.23:0/2524065920 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a480a6c50 0x7f2a481359f0 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0x7f2a4813ce10 tx=0x7f2a38008040).stop 2020-11-20T11:32:48.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.385+0000 7f2a477fe700 1 --2- 172.21.15.23:0/2524065920 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2a24056d60 0x7f2a24059220 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f2a34005500 tx=0x7f2a40015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:48.413 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.391+0000 7f2a47fff700 1 --2- 172.21.15.23:0/2524065920 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a48135ee0 0x7f2a481362f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.413 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.391+0000 7f2a47fff700 1 -- 172.21.15.23:0/2524065920 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a480a7a20 con 0x7f2a48135ee0 2020-11-20T11:32:48.413 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.392+0000 7f2a467fc700 1 --2- 172.21.15.23:0/2524065920 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a48135ee0 0x7f2a481362f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:48.413 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.392+0000 7f2a467fc700 1 -- 172.21.15.23:0/2524065920 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2a48057dc0 con 0x7f2a48135ee0 2020-11-20T11:32:48.413 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.392+0000 7f2a467fc700 1 --2- 172.21.15.23:0/2524065920 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a48135ee0 0x7f2a481362f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f2a3801a000 tx=0x7f2a3801d000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:48.414 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.392+0000 7f2a2effd700 1 -- 172.21.15.23:0/2524065920 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a38015690 con 0x7f2a48135ee0 2020-11-20T11:32:48.414 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.393+0000 7f2a2effd700 1 -- 172.21.15.23:0/2524065920 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4783 (secure 0 0 0) 0x7f2a38000e90 con 0x7f2a48135ee0 2020-11-20T11:32:48.414 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.393+0000 7f2a2effd700 1 -- 172.21.15.23:0/2524065920 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a48135ee0 msgr2=0x7f2a481362f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:48.414 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.393+0000 7f2a2effd700 1 --2- 172.21.15.23:0/2524065920 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a48135ee0 0x7f2a481362f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f2a3801a000 tx=0x7f2a3801d000).stop 2020-11-20T11:32:48.414 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.393+0000 7f2a2effd700 1 -- 172.21.15.23:0/2524065920 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a48135ee0 msgr2=0x7f2a481362f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:32:48.415 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.393+0000 7f2a2effd700 1 --2- 172.21.15.23:0/2524065920 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2a48135ee0 0x7f2a481362f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:48.415 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.393+0000 7f2a47fff700 1 -- 172.21.15.23:0/2524065920 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2a24056d60 msgr2=0x7f2a24059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:48.415 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.393+0000 7f2a47fff700 1 --2- 172.21.15.23:0/2524065920 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2a24056d60 0x7f2a24059220 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f2a34005500 tx=0x7f2a40015000).stop 2020-11-20T11:32:48.415 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.393+0000 7f2a47fff700 1 -- 172.21.15.23:0/2524065920 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a34004e90 msgr2=0x7f2a480a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:48.415 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.393+0000 7f2a47fff700 1 --2- 172.21.15.23:0/2524065920 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a34004e90 0x7f2a480a4d00 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f2a3c00fb80 tx=0x7f2a3c009040).stop 2020-11-20T11:32:48.416 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.393+0000 7f2a477fe700 1 -- 172.21.15.23:0/2524065920 reap_dead start 2020-11-20T11:32:48.416 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.393+0000 7f2a47fff700 1 -- 172.21.15.23:0/2524065920 shutdown_connections 2020-11-20T11:32:48.416 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.393+0000 7f2a47fff700 1 --2- 172.21.15.23:0/2524065920 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a34004e90 0x7f2a480a4d00 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:48.416 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.393+0000 7f2a47fff700 1 -- 172.21.15.23:0/2524065920 >> 172.21.15.23:0/2524065920 conn(0x7f2a48063980 msgr2=0x7f2a480a18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:48.416 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.393+0000 7f2a47fff700 1 -- 172.21.15.23:0/2524065920 shutdown_connections 2020-11-20T11:32:48.417 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.393+0000 7f2a47fff700 1 -- 172.21.15.23:0/2524065920 wait complete. 2020-11-20T11:32:48.417 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:32:48.417 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:32:48.417 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:32:48.418 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:32:48.418 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 4, 2020-11-20T11:32:48.418 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:32:48.418 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:32:48.418 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:32:48.419 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:32:48.419 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:32:48.419 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 42, 2020-11-20T11:32:48.419 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:32:48.419 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:32:48.420 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:32:48.420 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:32:48.420 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:32:48.420 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:32:48.420 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:32:48.421 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:32:48.421 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:32:48.421 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:32:48.421 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:32:48.421 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:32:48.422 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:32:48.422 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:32:48.422 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:32:48.422 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:32:48.422 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:32:48.423 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:32:48.423 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:32:48.423 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:32:48.423 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:32:48.423 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:32:48.424 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:32:48.424 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.424 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:32:48.424 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:32:48.424 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:32:48.425 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:32:48.425 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:32:48.425 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:32:48.425 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:32:48.425 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:32:48.426 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:32:48.426 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:32:48.426 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:32:48.426 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:32:48.426 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:32:48.427 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:32:48.427 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:32:48.427 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:32:48.427 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:32:48.427 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:32:48.427 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:32:48.428 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:32:48.428 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:32:48.428 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:32:48.428 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:32:48.428 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:32:48.429 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:32:48.429 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.429 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:32:48.429 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.429 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:32:48.430 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:32:48.430 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:32:48.430 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:32:48.430 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.430 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:32:48.431 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:32:48.431 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:32:48.431 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.431 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.431 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:32:48.432 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:32:48.432 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:32:48.432 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:32:48.432 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:32:48.432 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.433 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:32:48.433 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:32:48.434 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:32:48.434 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:32:48.434 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:32:48.434 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.434 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.435 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:32:48.435 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:32:48.435 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:32:48.435 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:32:48.435 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.436 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:32:48.436 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:32:48.436 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:32:48.436 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.436 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.437 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:32:48.437 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:32:48.437 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:32:48.437 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:32:48.438 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:32:48.438 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.438 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:32:48.438 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:32:48.438 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:32:48.439 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:32:48.439 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:32:48.439 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.439 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.439 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:32:48.440 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:32:48.440 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:32:48.440 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:32:48.440 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.440 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:32:48.441 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:32:48.441 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:32:48.441 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.441 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.441 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:32:48.442 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:32:48.442 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:32:48.442 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:32:48.442 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:32:48.442 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.443 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:32:48.443 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:32:48.443 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:32:48.443 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:32:48.443 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:32:48.444 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:32:48.444 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:32:48.444 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.444 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:32:48.444 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:32:48.445 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.445 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:32:48.445 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:32:48.445 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:32:48.445 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:32:48.446 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:32:48.446 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:32:48.446 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.446 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:32:48.446 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:32:48.446 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:32:48.447 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:32:48.447 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:32:48.447 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:32:48.447 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.447 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:32:48.448 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:32:48.448 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:32:48.448 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:32:48.448 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:32:48.448 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:32:48.449 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.449 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:32:48.449 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:32:48.449 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:32:48.449 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:32:48.450 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:32:48.450 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:32:48.450 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:32:48.450 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:32:48.450 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:32:48.451 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:32:48.451 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:32:48.451 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:32:48.451 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:32:48.451 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:32:48.452 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:32:48.453 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2020-11-20T11:32:48.454 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2020-11-20T11:32:48.454 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2020-11-20T11:32:48.454 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2020-11-20T11:32:48.455 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2020-11-20T11:32:48.455 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T11:32:48.483 INFO:tasks.ceph.mon.a:Stopped 2020-11-20T11:32:48.483 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2020-11-20T11:32:48.483 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:32:48.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.646+0000 7fb88d217700 1 Processor -- start 2020-11-20T11:32:48.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.646+0000 7fb88d217700 1 -- start start 2020-11-20T11:32:48.661 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.647+0000 7fb88d217700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8880a63e0 0x7fb88813bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.661 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.647+0000 7fb88d217700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb8880a67c0 0x7fb8880aa250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.647+0000 7fb88d217700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8880a15f0 con 0x7fb8880a0e00 2020-11-20T11:32:48.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.647+0000 7fb88d217700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8880a9b10 con 0x7fb8880a67c0 2020-11-20T11:32:48.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.647+0000 7fb88d217700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8880a9db0 con 0x7fb8880a63e0 2020-11-20T11:32:48.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.647+0000 7fb88ca16700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7fb8880a0e00 legacy=0x7fb88813c430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2020-11-20T11:32:48.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.647+0000 7fb8877fe700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7fb8880a67c0 msgr2=0x7fb8880aa250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:32:48.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.647+0000 7fb8877fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb8880a67c0 0x7fb8880aa250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:32:48.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.647+0000 7fb887fff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8880a63e0 0x7fb88813bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:48.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.647+0000 7fb887fff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8880a63e0 0x7fb88813bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:51878/0 (socket says 172.21.15.23:51878) 2020-11-20T11:32:48.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.647+0000 7fb887fff700 1 -- 172.21.15.23:0/3516509836 learned_addr learned my addr 172.21.15.23:0/3516509836 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:48.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.647+0000 7fb887fff700 1 -- 172.21.15.23:0/3516509836 >> v2:172.21.15.23:3300/0 conn(0x7fb8880a67c0 msgr2=0x7fb8880aa250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:32:48.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.647+0000 7fb887fff700 1 --2- 172.21.15.23:0/3516509836 >> v2:172.21.15.23:3300/0 conn(0x7fb8880a67c0 0x7fb8880aa250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:32:48.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.647+0000 7fb887fff700 1 -- 172.21.15.23:0/3516509836 >> v1:172.21.15.23:6789/0 conn(0x7fb8880a0e00 legacy=0x7fb88813c430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:32:48.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.647+0000 7fb887fff700 1 -- 172.21.15.23:0/3516509836 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8881361a0 con 0x7fb8880a63e0 2020-11-20T11:32:48.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.647+0000 7fb887fff700 1 --2- 172.21.15.23:0/3516509836 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8880a63e0 0x7fb88813bdb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fb878009a10 tx=0x7fb878006ef0).ready entity=mon.2 client_cookie=48029c7ca6d641f3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:48.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.648+0000 7fb886ffd700 1 -- 172.21.15.23:0/3516509836 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87800b820 con 0x7fb8880a63e0 2020-11-20T11:32:48.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.648+0000 7fb886ffd700 1 -- 172.21.15.23:0/3516509836 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb878000e90 con 0x7fb8880a63e0 2020-11-20T11:32:48.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.648+0000 7fb886ffd700 1 -- 172.21.15.23:0/3516509836 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87800bb90 con 0x7fb8880a63e0 2020-11-20T11:32:48.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.648+0000 7fb88d217700 1 -- 172.21.15.23:0/3516509836 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8880a63e0 msgr2=0x7fb88813bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:48.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.648+0000 7fb88d217700 1 --2- 172.21.15.23:0/3516509836 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8880a63e0 0x7fb88813bdb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fb878009a10 tx=0x7fb878006ef0).stop 2020-11-20T11:32:48.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.648+0000 7fb88d217700 1 -- 172.21.15.23:0/3516509836 shutdown_connections 2020-11-20T11:32:48.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.648+0000 7fb88d217700 1 --2- 172.21.15.23:0/3516509836 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8880a63e0 0x7fb88813bdb0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:48.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.648+0000 7fb88d217700 1 --2- 172.21.15.23:0/3516509836 >> v2:172.21.15.23:3300/0 conn(0x7fb8880a67c0 0x7fb8880aa250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:32:48.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.648+0000 7fb88d217700 1 -- 172.21.15.23:0/3516509836 >> 172.21.15.23:0/3516509836 conn(0x7fb888061850 msgr2=0x7fb888061c60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:48.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.648+0000 7fb88d217700 1 -- 172.21.15.23:0/3516509836 shutdown_connections 2020-11-20T11:32:48.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.648+0000 7fb88d217700 1 -- 172.21.15.23:0/3516509836 wait complete. 2020-11-20T11:32:48.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.649+0000 7fb88d217700 1 Processor -- start 2020-11-20T11:32:48.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.649+0000 7fb88d217700 1 -- start start 2020-11-20T11:32:48.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.650+0000 7fb88d217700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8880a0e00 0x7fb888067040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.650+0000 7fb88d217700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8880a63e0 0x7fb888064ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.650+0000 7fb88d217700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8880a67c0 0x7fb8880ad110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.650+0000 7fb88d217700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88813ff30 con 0x7fb8880a67c0 2020-11-20T11:32:48.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.650+0000 7fb88d217700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb888065580 con 0x7fb8880a63e0 2020-11-20T11:32:48.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.650+0000 7fb88d217700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb888065820 con 0x7fb8880a0e00 2020-11-20T11:32:48.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.650+0000 7fb88ca16700 1 -- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8880a67c0 msgr2=0x7fb8880ad110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:32:48.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.650+0000 7fb88ca16700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8880a67c0 0x7fb8880ad110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:32:48.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.650+0000 7fb8877fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8880a63e0 0x7fb888064ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:48.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.650+0000 7fb887fff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8880a0e00 0x7fb888067040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:48.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.650+0000 7fb8877fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8880a63e0 0x7fb888064ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:34542/0 (socket says 172.21.15.23:34542) 2020-11-20T11:32:48.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.650+0000 7fb887fff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8880a0e00 0x7fb888067040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:51884/0 (socket says 172.21.15.23:51884) 2020-11-20T11:32:48.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.650+0000 7fb8877fe700 1 -- 172.21.15.23:0/3361927728 learned_addr learned my addr 172.21.15.23:0/3361927728 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:48.675 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.651+0000 7fb8877fe700 1 -- 172.21.15.23:0/3361927728 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8880a0e00 msgr2=0x7fb888067040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:48.675 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.651+0000 7fb8877fe700 1 --2- 172.21.15.23:0/3361927728 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8880a0e00 0x7fb888067040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:48.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.651+0000 7fb8877fe700 1 -- 172.21.15.23:0/3361927728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8880a67c0 msgr2=0x7fb8880ad110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:32:48.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.651+0000 7fb8877fe700 1 --2- 172.21.15.23:0/3361927728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8880a67c0 0x7fb8880ad110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:32:48.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.651+0000 7fb8877fe700 1 -- 172.21.15.23:0/3361927728 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb888065ac0 con 0x7fb8880a63e0 2020-11-20T11:32:48.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.651+0000 7fb8877fe700 1 --2- 172.21.15.23:0/3361927728 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8880a63e0 0x7fb888064ce0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fb87c009730 tx=0x7fb87c006e40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:48.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.651+0000 7fb867fff700 1 -- 172.21.15.23:0/3361927728 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c0050d0 con 0x7fb8880a63e0 2020-11-20T11:32:48.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.651+0000 7fb867fff700 1 -- 172.21.15.23:0/3361927728 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87c005250 con 0x7fb8880a63e0 2020-11-20T11:32:48.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.651+0000 7fb88d217700 1 -- 172.21.15.23:0/3361927728 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8880651d0 con 0x7fb8880a63e0 2020-11-20T11:32:48.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.651+0000 7fb867fff700 1 -- 172.21.15.23:0/3361927728 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c003850 con 0x7fb8880a63e0 2020-11-20T11:32:48.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.651+0000 7fb88d217700 1 -- 172.21.15.23:0/3361927728 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8880ad7c0 con 0x7fb8880a63e0 2020-11-20T11:32:48.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.652+0000 7fb88d217700 1 -- 172.21.15.23:0/3361927728 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8880a3bf0 con 0x7fb8880a63e0 2020-11-20T11:32:48.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.652+0000 7fb867fff700 1 -- 172.21.15.23:0/3361927728 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb87c00f070 con 0x7fb8880a63e0 2020-11-20T11:32:48.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.653+0000 7fb867fff700 1 --2- 172.21.15.23:0/3361927728 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb860056d10 0x7fb8600591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:48.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.653+0000 7fb867fff700 1 -- 172.21.15.23:0/3361927728 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fb87c06f690 con 0x7fb8880a63e0 2020-11-20T11:32:48.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.653+0000 7fb887fff700 1 --2- 172.21.15.23:0/3361927728 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb860056d10 0x7fb8600591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:48.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.653+0000 7fb887fff700 1 --2- 172.21.15.23:0/3361927728 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb860056d10 0x7fb8600591d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fb878006f30 tx=0x7fb878016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:48.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.657+0000 7fb867fff700 1 -- 172.21.15.23:0/3361927728 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb87c044a30 con 0x7fb8880a63e0 2020-11-20T11:32:48.866 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.863+0000 7fb88d217700 1 -- 172.21.15.23:0/3361927728 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb888057dc0 con 0x7fb8880a63e0 2020-11-20T11:32:48.867 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.864+0000 7fb867fff700 1 -- 172.21.15.23:0/3361927728 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1477 (secure 0 0 0) 0x7fb87c048050 con 0x7fb8880a63e0 2020-11-20T11:32:48.868 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":42,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:32:48.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.865+0000 7fb88d217700 1 -- 172.21.15.23:0/3361927728 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb860056d10 msgr2=0x7fb8600591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:48.870 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.865+0000 7fb88d217700 1 --2- 172.21.15.23:0/3361927728 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb860056d10 0x7fb8600591d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fb878006f30 tx=0x7fb878016000).stop 2020-11-20T11:32:48.870 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.865+0000 7fb88d217700 1 -- 172.21.15.23:0/3361927728 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8880a63e0 msgr2=0x7fb888064ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:48.870 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.865+0000 7fb88d217700 1 --2- 172.21.15.23:0/3361927728 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8880a63e0 0x7fb888064ce0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fb87c009730 tx=0x7fb87c006e40).stop 2020-11-20T11:32:48.871 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.865+0000 7fb88d217700 1 -- 172.21.15.23:0/3361927728 shutdown_connections 2020-11-20T11:32:48.871 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.865+0000 7fb88d217700 1 --2- 172.21.15.23:0/3361927728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8880a67c0 0x7fb8880ad110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:32:48.871 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.865+0000 7fb88d217700 1 --2- 172.21.15.23:0/3361927728 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb860056d10 0x7fb8600591d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:48.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.865+0000 7fb88d217700 1 --2- 172.21.15.23:0/3361927728 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8880a0e00 0x7fb888067040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:48.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.865+0000 7fb88d217700 1 --2- 172.21.15.23:0/3361927728 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8880a63e0 0x7fb888064ce0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:48.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.866+0000 7fb88d217700 1 -- 172.21.15.23:0/3361927728 >> 172.21.15.23:0/3361927728 conn(0x7fb888061850 msgr2=0x7fb8880a7a10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:48.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.866+0000 7fb88d217700 1 -- 172.21.15.23:0/3361927728 shutdown_connections 2020-11-20T11:32:48.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:48.866+0000 7fb88d217700 1 -- 172.21.15.23:0/3361927728 wait complete. 2020-11-20T11:32:51.882 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:32:52.018 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.015+0000 7fb44e08d700 1 Processor -- start 2020-11-20T11:32:52.019 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.015+0000 7fb44e08d700 1 -- start start 2020-11-20T11:32:52.020 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.016+0000 7fb44e08d700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb448060a80 0x7fb448061950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:52.021 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.016+0000 7fb44e08d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb4480aa1c0 0x7fb448138030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:52.021 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.016+0000 7fb44e08d700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb448067bc0 con 0x7fb4480a9d40 2020-11-20T11:32:52.021 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.016+0000 7fb44e08d700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4480aa630 con 0x7fb448060a80 2020-11-20T11:32:52.021 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.016+0000 7fb44e08d700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb448067e60 con 0x7fb4480aa1c0 2020-11-20T11:32:52.021 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.016+0000 7fb446ffd700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7fb4480a9d40 legacy=0x7fb448137920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2020-11-20T11:32:52.022 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.017+0000 7fb4477fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb448060a80 0x7fb448061950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:52.022 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.017+0000 7fb447fff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb4480aa1c0 0x7fb448138030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:52.022 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.017+0000 7fb4477fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb448060a80 0x7fb448061950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:34548/0 (socket says 172.21.15.23:34548) 2020-11-20T11:32:52.022 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.017+0000 7fb447fff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb4480aa1c0 0x7fb448138030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:51896/0 (socket says 172.21.15.23:51896) 2020-11-20T11:32:52.023 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.017+0000 7fb4477fe700 1 -- 172.21.15.23:0/1200839118 learned_addr learned my addr 172.21.15.23:0/1200839118 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:52.023 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.017+0000 7fb447fff700 1 -- 172.21.15.23:0/1200839118 >> v2:172.21.15.155:3300/0 conn(0x7fb448060a80 msgr2=0x7fb448061950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:32:52.023 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.017+0000 7fb447fff700 1 --2- 172.21.15.23:0/1200839118 >> v2:172.21.15.155:3300/0 conn(0x7fb448060a80 0x7fb448061950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:52.023 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.017+0000 7fb447fff700 1 -- 172.21.15.23:0/1200839118 >> v1:172.21.15.23:6789/0 conn(0x7fb4480a9d40 legacy=0x7fb448137920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:32:52.023 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.017+0000 7fb447fff700 1 -- 172.21.15.23:0/1200839118 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb448068100 con 0x7fb4480aa1c0 2020-11-20T11:32:52.027 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.017+0000 7fb4477fe700 1 --2- 172.21.15.23:0/1200839118 >> v2:172.21.15.155:3300/0 conn(0x7fb448060a80 0x7fb448061950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:32:52.027 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.017+0000 7fb447fff700 1 --2- 172.21.15.23:0/1200839118 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb4480aa1c0 0x7fb448138030 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fb440009ce0 tx=0x7fb440009220).ready entity=mon.2 client_cookie=1b9d45d0f6307f64 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:52.027 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.017+0000 7fb4467fc700 1 -- 172.21.15.23:0/1200839118 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44000ba00 con 0x7fb4480aa1c0 2020-11-20T11:32:52.027 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.017+0000 7fb4467fc700 1 -- 172.21.15.23:0/1200839118 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb44000bb60 con 0x7fb4480aa1c0 2020-11-20T11:32:52.027 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.017+0000 7fb4467fc700 1 -- 172.21.15.23:0/1200839118 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb440011400 con 0x7fb4480aa1c0 2020-11-20T11:32:52.028 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.018+0000 7fb44e08d700 1 -- 172.21.15.23:0/1200839118 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb4480aa1c0 msgr2=0x7fb448138030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:52.028 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.018+0000 7fb44e08d700 1 --2- 172.21.15.23:0/1200839118 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb4480aa1c0 0x7fb448138030 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fb440009ce0 tx=0x7fb440009220).stop 2020-11-20T11:32:52.028 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.018+0000 7fb44e08d700 1 -- 172.21.15.23:0/1200839118 shutdown_connections 2020-11-20T11:32:52.028 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.018+0000 7fb44e08d700 1 --2- 172.21.15.23:0/1200839118 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb4480aa1c0 0x7fb448138030 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:52.029 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.018+0000 7fb44e08d700 1 --2- 172.21.15.23:0/1200839118 >> v2:172.21.15.155:3300/0 conn(0x7fb448060a80 0x7fb448061950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:52.029 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.018+0000 7fb44e08d700 1 -- 172.21.15.23:0/1200839118 >> 172.21.15.23:0/1200839118 conn(0x7fb448066730 msgr2=0x7fb4480601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:52.029 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.018+0000 7fb44e08d700 1 -- 172.21.15.23:0/1200839118 shutdown_connections 2020-11-20T11:32:52.029 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.018+0000 7fb44e08d700 1 -- 172.21.15.23:0/1200839118 wait complete. 2020-11-20T11:32:52.029 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.019+0000 7fb44e08d700 1 Processor -- start 2020-11-20T11:32:52.030 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.019+0000 7fb44e08d700 1 -- start start 2020-11-20T11:32:52.030 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.020+0000 7fb44e08d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb448060a80 0x7fb4480a87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:52.030 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.020+0000 7fb44e08d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4480a9d40 0x7fb448136c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:52.030 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.020+0000 7fb44e08d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4480aa1c0 0x7fb448137150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:52.030 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.020+0000 7fb44e08d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4480a7320 con 0x7fb4480aa1c0 2020-11-20T11:32:52.031 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.020+0000 7fb44e08d700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb448136870 con 0x7fb4480a9d40 2020-11-20T11:32:52.031 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.020+0000 7fb44e08d700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb448137640 con 0x7fb448060a80 2020-11-20T11:32:52.031 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.020+0000 7fb447fff700 1 -- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4480aa1c0 msgr2=0x7fb448137150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:32:52.031 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.020+0000 7fb447fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4480aa1c0 0x7fb448137150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:32:52.031 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.020+0000 7fb446ffd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4480a9d40 0x7fb448136c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:52.032 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.020+0000 7fb446ffd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4480a9d40 0x7fb448136c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:34556/0 (socket says 172.21.15.23:34556) 2020-11-20T11:32:52.032 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.020+0000 7fb446ffd700 1 -- 172.21.15.23:0/645357868 learned_addr learned my addr 172.21.15.23:0/645357868 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:52.032 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.020+0000 7fb4477fe700 1 --2- 172.21.15.23:0/645357868 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb448060a80 0x7fb4480a87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:52.032 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.020+0000 7fb446ffd700 1 -- 172.21.15.23:0/645357868 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb448060a80 msgr2=0x7fb4480a87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:52.032 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.020+0000 7fb446ffd700 1 --2- 172.21.15.23:0/645357868 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb448060a80 0x7fb4480a87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:52.033 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.020+0000 7fb446ffd700 1 -- 172.21.15.23:0/645357868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4480aa1c0 msgr2=0x7fb448137150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:32:52.033 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.020+0000 7fb446ffd700 1 --2- 172.21.15.23:0/645357868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4480aa1c0 0x7fb448137150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:32:52.033 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.020+0000 7fb446ffd700 1 -- 172.21.15.23:0/645357868 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb448138b30 con 0x7fb4480a9d40 2020-11-20T11:32:52.033 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.021+0000 7fb446ffd700 1 --2- 172.21.15.23:0/645357868 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4480a9d40 0x7fb448136c60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fb43c0097f0 tx=0x7fb43c006ea0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:52.033 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.021+0000 7fb4277fe700 1 -- 172.21.15.23:0/645357868 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43c006170 con 0x7fb4480a9d40 2020-11-20T11:32:52.034 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.021+0000 7fb4277fe700 1 -- 172.21.15.23:0/645357868 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb43c0062f0 con 0x7fb4480a9d40 2020-11-20T11:32:52.034 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.021+0000 7fb44e08d700 1 -- 172.21.15.23:0/645357868 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4480a7080 con 0x7fb4480a9d40 2020-11-20T11:32:52.034 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.021+0000 7fb4277fe700 1 -- 172.21.15.23:0/645357868 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43c003920 con 0x7fb4480a9d40 2020-11-20T11:32:52.034 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.021+0000 7fb44e08d700 1 -- 172.21.15.23:0/645357868 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb448135d30 con 0x7fb4480a9d40 2020-11-20T11:32:52.034 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.022+0000 7fb44e08d700 1 -- 172.21.15.23:0/645357868 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4480a6ea0 con 0x7fb4480a9d40 2020-11-20T11:32:52.035 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.022+0000 7fb4277fe700 1 -- 172.21.15.23:0/645357868 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb43c017030 con 0x7fb4480a9d40 2020-11-20T11:32:52.035 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.022+0000 7fb4277fe700 1 --2- 172.21.15.23:0/645357868 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb420056d10 0x7fb4200591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:52.035 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.023+0000 7fb4277fe700 1 -- 172.21.15.23:0/645357868 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fb43c06f1c0 con 0x7fb4480a9d40 2020-11-20T11:32:52.035 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.023+0000 7fb4477fe700 1 --2- 172.21.15.23:0/645357868 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb420056d10 0x7fb4200591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:52.035 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.023+0000 7fb4477fe700 1 --2- 172.21.15.23:0/645357868 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb420056d10 0x7fb4200591d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fb438006a30 tx=0x7fb438006ef0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:52.036 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.027+0000 7fb4277fe700 1 -- 172.21.15.23:0/645357868 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb43c044560 con 0x7fb4480a9d40 2020-11-20T11:32:52.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.232+0000 7fb44e08d700 1 -- 172.21.15.23:0/645357868 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb4480a6310 con 0x7fb4480a9d40 2020-11-20T11:32:52.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.232+0000 7fb4277fe700 1 -- 172.21.15.23:0/645357868 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1477 (secure 0 0 0) 0x7fb43c047b80 con 0x7fb4480a9d40 2020-11-20T11:32:52.236 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":46,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:32:52.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.234+0000 7fb44e08d700 1 -- 172.21.15.23:0/645357868 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb420056d10 msgr2=0x7fb4200591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:52.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.234+0000 7fb44e08d700 1 --2- 172.21.15.23:0/645357868 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb420056d10 0x7fb4200591d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fb438006a30 tx=0x7fb438006ef0).stop 2020-11-20T11:32:52.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.234+0000 7fb44e08d700 1 -- 172.21.15.23:0/645357868 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4480a9d40 msgr2=0x7fb448136c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:52.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.234+0000 7fb44e08d700 1 --2- 172.21.15.23:0/645357868 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4480a9d40 0x7fb448136c60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fb43c0097f0 tx=0x7fb43c006ea0).stop 2020-11-20T11:32:52.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.234+0000 7fb44e08d700 1 -- 172.21.15.23:0/645357868 shutdown_connections 2020-11-20T11:32:52.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.234+0000 7fb44e08d700 1 --2- 172.21.15.23:0/645357868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb4480aa1c0 0x7fb448137150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:32:52.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.234+0000 7fb44e08d700 1 --2- 172.21.15.23:0/645357868 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb420056d10 0x7fb4200591d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:52.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.234+0000 7fb44e08d700 1 --2- 172.21.15.23:0/645357868 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb448060a80 0x7fb4480a87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:52.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.234+0000 7fb44e08d700 1 --2- 172.21.15.23:0/645357868 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb4480a9d40 0x7fb448136c60 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:52.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.234+0000 7fb44e08d700 1 -- 172.21.15.23:0/645357868 >> 172.21.15.23:0/645357868 conn(0x7fb448066730 msgr2=0x7fb44813ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:52.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.234+0000 7fb44e08d700 1 -- 172.21.15.23:0/645357868 shutdown_connections 2020-11-20T11:32:52.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:52.234+0000 7fb44e08d700 1 -- 172.21.15.23:0/645357868 wait complete. 2020-11-20T11:32:55.249 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:32:55.376 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.373+0000 7f7cac045700 1 Processor -- start 2020-11-20T11:32:55.377 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.374+0000 7f7cac045700 1 -- start start 2020-11-20T11:32:55.379 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.374+0000 7f7cac045700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f7ca40aaab0 0x7f7ca413ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:55.379 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.374+0000 7f7cac045700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f7ca413aeb0 0x7f7ca40a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:55.379 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.374+0000 7f7cac045700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ca41388b0 con 0x7f7ca40aae90 2020-11-20T11:32:55.379 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.374+0000 7f7cac045700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ca413e600 con 0x7f7ca413aeb0 2020-11-20T11:32:55.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.374+0000 7f7cac045700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ca413e8a0 con 0x7f7ca40aaab0 2020-11-20T11:32:55.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.374+0000 7f7caa5e2700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f7ca413aeb0 msgr2=0x7f7ca40a8c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:32:55.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.374+0000 7f7ca95e0700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f7ca40aae90 legacy=0x7f7ca41381a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2020-11-20T11:32:55.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.374+0000 7f7caa5e2700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f7ca413aeb0 0x7f7ca40a8c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:32:55.381 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.375+0000 7f7ca9de1700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f7ca40aaab0 0x7f7ca413ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:55.381 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.375+0000 7f7ca9de1700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f7ca40aaab0 0x7f7ca413ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:34562/0 (socket says 172.21.15.23:34562) 2020-11-20T11:32:55.381 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.375+0000 7f7ca9de1700 1 -- 172.21.15.23:0/3895308077 learned_addr learned my addr 172.21.15.23:0/3895308077 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:55.381 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.375+0000 7f7ca9de1700 1 -- 172.21.15.23:0/3895308077 >> v2:172.21.15.23:3300/0 conn(0x7f7ca413aeb0 msgr2=0x7f7ca40a8c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:32:55.381 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.375+0000 7f7ca9de1700 1 --2- 172.21.15.23:0/3895308077 >> v2:172.21.15.23:3300/0 conn(0x7f7ca413aeb0 0x7f7ca40a8c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:32:55.382 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.375+0000 7f7ca9de1700 1 -- 172.21.15.23:0/3895308077 >> v1:172.21.15.23:6789/0 conn(0x7f7ca40aae90 legacy=0x7f7ca41381a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:32:55.382 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.375+0000 7f7ca9de1700 1 -- 172.21.15.23:0/3895308077 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca40ab6d0 con 0x7f7ca40aaab0 2020-11-20T11:32:55.382 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.375+0000 7f7ca9de1700 1 --2- 172.21.15.23:0/3895308077 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ca40aaab0 0x7f7ca413ed40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f7ca40b23d0 tx=0x7f7c94006ef0).ready entity=mon.1 client_cookie=474006345bafb6b6 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:55.382 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.375+0000 7f7ca8ddf700 1 -- 172.21.15.23:0/3895308077 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9400b820 con 0x7f7ca40aaab0 2020-11-20T11:32:55.382 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.375+0000 7f7ca8ddf700 1 -- 172.21.15.23:0/3895308077 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c94000e90 con 0x7f7ca40aaab0 2020-11-20T11:32:55.382 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.375+0000 7f7ca8ddf700 1 -- 172.21.15.23:0/3895308077 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9400bb90 con 0x7f7ca40aaab0 2020-11-20T11:32:55.383 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.376+0000 7f7cac045700 1 -- 172.21.15.23:0/3895308077 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ca40aaab0 msgr2=0x7f7ca413ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:55.383 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.376+0000 7f7cac045700 1 --2- 172.21.15.23:0/3895308077 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ca40aaab0 0x7f7ca413ed40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f7ca40b23d0 tx=0x7f7c94006ef0).stop 2020-11-20T11:32:55.383 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.376+0000 7f7cac045700 1 -- 172.21.15.23:0/3895308077 shutdown_connections 2020-11-20T11:32:55.383 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.376+0000 7f7cac045700 1 --2- 172.21.15.23:0/3895308077 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ca40aaab0 0x7f7ca413ed40 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:55.383 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.376+0000 7f7cac045700 1 --2- 172.21.15.23:0/3895308077 >> v2:172.21.15.23:3300/0 conn(0x7f7ca413aeb0 0x7f7ca40a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:32:55.384 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.376+0000 7f7cac045700 1 -- 172.21.15.23:0/3895308077 >> 172.21.15.23:0/3895308077 conn(0x7f7ca40678e0 msgr2=0x7f7ca4067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:55.384 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.376+0000 7f7cac045700 1 -- 172.21.15.23:0/3895308077 shutdown_connections 2020-11-20T11:32:55.384 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.376+0000 7f7cac045700 1 -- 172.21.15.23:0/3895308077 wait complete. 2020-11-20T11:32:55.387 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.377+0000 7f7cac045700 1 Processor -- start 2020-11-20T11:32:55.387 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.377+0000 7f7cac045700 1 -- start start 2020-11-20T11:32:55.387 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.378+0000 7f7cac045700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ca40aaab0 0x7f7ca41368e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:55.387 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.378+0000 7f7cac045700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ca40aae90 0x7f7ca413c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:55.387 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.378+0000 7f7cac045700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ca40a5330 0x7f7ca4135890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:55.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.378+0000 7f7cac045700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca40a5790 con 0x7f7ca40a5330 2020-11-20T11:32:55.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.378+0000 7f7cac045700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca413d1e0 con 0x7f7ca40aaab0 2020-11-20T11:32:55.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.378+0000 7f7cac045700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca413cca0 con 0x7f7ca40aae90 2020-11-20T11:32:55.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.378+0000 7f7caa5e2700 1 -- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ca40a5330 msgr2=0x7f7ca4135890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:32:55.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.378+0000 7f7caa5e2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ca40a5330 0x7f7ca4135890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:32:55.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.378+0000 7f7ca9de1700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ca40aaab0 0x7f7ca41368e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:55.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.378+0000 7f7ca95e0700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ca40aae90 0x7f7ca413c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:55.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.378+0000 7f7ca9de1700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ca40aaab0 0x7f7ca41368e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:34568/0 (socket says 172.21.15.23:34568) 2020-11-20T11:32:55.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.378+0000 7f7ca9de1700 1 -- 172.21.15.23:0/3228175672 learned_addr learned my addr 172.21.15.23:0/3228175672 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:55.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.378+0000 7f7ca95e0700 1 -- 172.21.15.23:0/3228175672 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ca40aaab0 msgr2=0x7f7ca41368e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:55.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.378+0000 7f7ca95e0700 1 --2- 172.21.15.23:0/3228175672 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ca40aaab0 0x7f7ca41368e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:55.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.378+0000 7f7ca95e0700 1 -- 172.21.15.23:0/3228175672 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ca40a5330 msgr2=0x7f7ca4135890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:32:55.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.378+0000 7f7ca95e0700 1 --2- 172.21.15.23:0/3228175672 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ca40a5330 0x7f7ca4135890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:32:55.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.379+0000 7f7ca95e0700 1 -- 172.21.15.23:0/3228175672 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca413cf40 con 0x7f7ca40aae90 2020-11-20T11:32:55.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.379+0000 7f7ca9de1700 1 --2- 172.21.15.23:0/3228175672 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ca40aaab0 0x7f7ca41368e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:32:55.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.379+0000 7f7ca95e0700 1 --2- 172.21.15.23:0/3228175672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ca40aae90 0x7f7ca413c600 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f7ca000bfd0 tx=0x7f7ca0006dc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:55.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.379+0000 7f7c91ffb700 1 -- 172.21.15.23:0/3228175672 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca00050d0 con 0x7f7ca40aae90 2020-11-20T11:32:55.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.379+0000 7f7c91ffb700 1 -- 172.21.15.23:0/3228175672 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ca0005250 con 0x7f7ca40aae90 2020-11-20T11:32:55.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.379+0000 7f7cac045700 1 -- 172.21.15.23:0/3228175672 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ca4135d80 con 0x7f7ca40aae90 2020-11-20T11:32:55.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.379+0000 7f7c91ffb700 1 -- 172.21.15.23:0/3228175672 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca0003850 con 0x7f7ca40aae90 2020-11-20T11:32:55.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.379+0000 7f7cac045700 1 -- 172.21.15.23:0/3228175672 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ca41335f0 con 0x7f7ca40aae90 2020-11-20T11:32:55.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.380+0000 7f7cac045700 1 -- 172.21.15.23:0/3228175672 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ca413caf0 con 0x7f7ca40aae90 2020-11-20T11:32:55.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.380+0000 7f7c91ffb700 1 -- 172.21.15.23:0/3228175672 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7ca000f070 con 0x7f7ca40aae90 2020-11-20T11:32:55.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.381+0000 7f7c91ffb700 1 --2- 172.21.15.23:0/3228175672 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7c7c056d10 0x7f7c7c0591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:55.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.381+0000 7f7c91ffb700 1 -- 172.21.15.23:0/3228175672 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f7ca006f820 con 0x7f7ca40aae90 2020-11-20T11:32:55.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.381+0000 7f7ca9de1700 1 --2- 172.21.15.23:0/3228175672 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7c7c056d10 0x7f7c7c0591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:55.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.381+0000 7f7ca9de1700 1 --2- 172.21.15.23:0/3228175672 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7c7c056d10 0x7f7c7c0591d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f7c94006b30 tx=0x7f7c94016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:32:55.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.385+0000 7f7c91ffb700 1 -- 172.21.15.23:0/3228175672 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7ca0044b90 con 0x7f7ca40aae90 2020-11-20T11:32:55.596 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.593+0000 7f7cac045700 1 -- 172.21.15.23:0/3228175672 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7ca405c7d0 con 0x7f7ca40aae90 2020-11-20T11:32:55.597 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.594+0000 7f7c91ffb700 1 -- 172.21.15.23:0/3228175672 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1477 (secure 0 0 0) 0x7f7ca00481b0 con 0x7f7ca40aae90 2020-11-20T11:32:55.597 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":49,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:32:55.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.596+0000 7f7cac045700 1 -- 172.21.15.23:0/3228175672 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7c7c056d10 msgr2=0x7f7c7c0591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:55.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.596+0000 7f7cac045700 1 --2- 172.21.15.23:0/3228175672 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7c7c056d10 0x7f7c7c0591d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f7c94006b30 tx=0x7f7c94016000).stop 2020-11-20T11:32:55.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.596+0000 7f7cac045700 1 -- 172.21.15.23:0/3228175672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ca40aae90 msgr2=0x7f7ca413c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:32:55.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.596+0000 7f7cac045700 1 --2- 172.21.15.23:0/3228175672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ca40aae90 0x7f7ca413c600 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f7ca000bfd0 tx=0x7f7ca0006dc0).stop 2020-11-20T11:32:55.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.596+0000 7f7cac045700 1 -- 172.21.15.23:0/3228175672 shutdown_connections 2020-11-20T11:32:55.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.596+0000 7f7cac045700 1 --2- 172.21.15.23:0/3228175672 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ca40a5330 0x7f7ca4135890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:32:55.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.596+0000 7f7cac045700 1 --2- 172.21.15.23:0/3228175672 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ca40aaab0 0x7f7ca41368e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:55.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.596+0000 7f7cac045700 1 --2- 172.21.15.23:0/3228175672 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7c7c056d10 0x7f7c7c0591d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:55.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.596+0000 7f7cac045700 1 --2- 172.21.15.23:0/3228175672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ca40aae90 0x7f7ca413c600 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:32:55.601 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.596+0000 7f7cac045700 1 -- 172.21.15.23:0/3228175672 >> 172.21.15.23:0/3228175672 conn(0x7f7ca40678e0 msgr2=0x7f7ca41405e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:32:55.601 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.596+0000 7f7cac045700 1 -- 172.21.15.23:0/3228175672 shutdown_connections 2020-11-20T11:32:55.601 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:55.596+0000 7f7cac045700 1 -- 172.21.15.23:0/3228175672 wait complete. 2020-11-20T11:32:58.611 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:32:58.738 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:58.735+0000 7fad6dca0700 1 Processor -- start 2020-11-20T11:32:58.739 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:58.735+0000 7fad6dca0700 1 -- start start 2020-11-20T11:32:58.740 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:58.736+0000 7fad6dca0700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fad680aaab0 0x7fad6813ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:58.740 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:58.736+0000 7fad6dca0700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fad680aae90 0x7fad681381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:32:58.740 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:58.736+0000 7fad6dca0700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad6813e600 con 0x7fad6813aeb0 2020-11-20T11:32:58.741 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:58.736+0000 7fad6dca0700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad6813e8a0 con 0x7fad680aae90 2020-11-20T11:32:58.741 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:58.736+0000 7fad6dca0700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad680ab6d0 con 0x7fad680aaab0 2020-11-20T11:32:58.741 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:58.736+0000 7fad66ffd700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7fad680aae90 msgr2=0x7fad681381a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:32:58.741 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:58.736+0000 7fad66ffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fad680aae90 0x7fad681381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:32:58.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:58.736+0000 7fad677fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fad680aaab0 0x7fad6813ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:58.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:58.736+0000 7fad677fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fad680aaab0 0x7fad6813ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:34580/0 (socket says 172.21.15.23:34580) 2020-11-20T11:32:58.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:58.736+0000 7fad677fe700 1 -- 172.21.15.23:0/1417949287 learned_addr learned my addr 172.21.15.23:0/1417949287 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:32:58.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:58.737+0000 7fad677fe700 1 -- 172.21.15.23:0/1417949287 >> v2:172.21.15.155:3300/0 conn(0x7fad680aaab0 msgr2=0x7fad6813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:32:58.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:58.737+0000 7fad677fe700 1 -- 172.21.15.23:0/1417949287 >> v2:172.21.15.155:3300/0 conn(0x7fad680aaab0 msgr2=0x7fad6813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:32:58.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:58.737+0000 7fad677fe700 1 --2- 172.21.15.23:0/1417949287 >> v2:172.21.15.155:3300/0 conn(0x7fad680aaab0 0x7fad6813ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:32:58.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:58.737+0000 7fad677fe700 1 --2- 172.21.15.23:0/1417949287 >> v2:172.21.15.155:3300/0 conn(0x7fad680aaab0 0x7fad6813ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:32:58.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:58.937+0000 7fad66ffd700 1 -- 172.21.15.23:0/1417949287 >> v2:172.21.15.23:3300/0 conn(0x7fad680aae90 msgr2=0x7fad681381a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:32:58.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:58.937+0000 7fad66ffd700 1 --2- 172.21.15.23:0/1417949287 >> v2:172.21.15.23:3300/0 conn(0x7fad680aae90 0x7fad681381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:32:58.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:58.938+0000 7fad677fe700 1 --2- 172.21.15.23:0/1417949287 >> v2:172.21.15.155:3300/0 conn(0x7fad680aaab0 0x7fad6813ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:58.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:58.938+0000 7fad677fe700 1 -- 172.21.15.23:0/1417949287 >> v2:172.21.15.155:3300/0 conn(0x7fad680aaab0 msgr2=0x7fad6813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:32:58.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:58.938+0000 7fad677fe700 1 -- 172.21.15.23:0/1417949287 >> v2:172.21.15.155:3300/0 conn(0x7fad680aaab0 msgr2=0x7fad6813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:32:58.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:58.938+0000 7fad677fe700 1 --2- 172.21.15.23:0/1417949287 >> v2:172.21.15.155:3300/0 conn(0x7fad680aaab0 0x7fad6813ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:32:58.942 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:58.938+0000 7fad677fe700 1 --2- 172.21.15.23:0/1417949287 >> v2:172.21.15.155:3300/0 conn(0x7fad680aaab0 0x7fad6813ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:32:59.332 INFO:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:32:59.335 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:32:59.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:59.338+0000 7fad66ffd700 1 -- 172.21.15.23:0/1417949287 >> v2:172.21.15.23:3300/0 conn(0x7fad680aae90 msgr2=0x7fad681381a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:32:59.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:59.338+0000 7fad66ffd700 1 --2- 172.21.15.23:0/1417949287 >> v2:172.21.15.23:3300/0 conn(0x7fad680aae90 0x7fad681381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:32:59.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:59.339+0000 7fad677fe700 1 --2- 172.21.15.23:0/1417949287 >> v2:172.21.15.155:3300/0 conn(0x7fad680aaab0 0x7fad6813ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:32:59.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:59.339+0000 7fad677fe700 1 -- 172.21.15.23:0/1417949287 >> v2:172.21.15.155:3300/0 conn(0x7fad680aaab0 msgr2=0x7fad6813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:32:59.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:59.339+0000 7fad677fe700 1 -- 172.21.15.23:0/1417949287 >> v2:172.21.15.155:3300/0 conn(0x7fad680aaab0 msgr2=0x7fad6813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:32:59.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:59.339+0000 7fad677fe700 1 --2- 172.21.15.23:0/1417949287 >> v2:172.21.15.155:3300/0 conn(0x7fad680aaab0 0x7fad6813ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:32:59.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:32:59.339+0000 7fad677fe700 1 --2- 172.21.15.23:0/1417949287 >> v2:172.21.15.155:3300/0 conn(0x7fad680aaab0 0x7fad6813ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:33:00.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:00.139+0000 7fad66ffd700 1 -- 172.21.15.23:0/1417949287 >> v2:172.21.15.23:3300/0 conn(0x7fad680aae90 msgr2=0x7fad681381a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:33:00.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:00.139+0000 7fad66ffd700 1 --2- 172.21.15.23:0/1417949287 >> v2:172.21.15.23:3300/0 conn(0x7fad680aae90 0x7fad681381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:33:00.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:00.140+0000 7fad677fe700 1 --2- 172.21.15.23:0/1417949287 >> v2:172.21.15.155:3300/0 conn(0x7fad680aaab0 0x7fad6813ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:00.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:00.141+0000 7fad677fe700 1 -- 172.21.15.23:0/1417949287 >> v2:172.21.15.155:3300/0 conn(0x7fad680aaab0 msgr2=0x7fad6813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:33:00.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:00.141+0000 7fad677fe700 1 -- 172.21.15.23:0/1417949287 >> v2:172.21.15.155:3300/0 conn(0x7fad680aaab0 msgr2=0x7fad6813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:33:00.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:00.141+0000 7fad677fe700 1 --2- 172.21.15.23:0/1417949287 >> v2:172.21.15.155:3300/0 conn(0x7fad680aaab0 0x7fad6813ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:33:00.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:00.141+0000 7fad677fe700 1 --2- 172.21.15.23:0/1417949287 >> v2:172.21.15.155:3300/0 conn(0x7fad680aaab0 0x7fad6813ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:33:01.740 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:01.736+0000 7fad657fa700 1 -- 172.21.15.23:0/1417949287 >> v2:172.21.15.155:3300/0 conn(0x7fad680aaab0 msgr2=0x7fad6813ed40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:33:01.740 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:01.736+0000 7fad657fa700 1 --2- 172.21.15.23:0/1417949287 >> v2:172.21.15.155:3300/0 conn(0x7fad680aaab0 0x7fad6813ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:01.740 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:01.736+0000 7fad657fa700 1 -- 172.21.15.23:0/1417949287 >> v2:172.21.15.23:3300/0 conn(0x7fad680aae90 msgr2=0x7fad681381a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:33:01.740 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:01.736+0000 7fad657fa700 1 --2- 172.21.15.23:0/1417949287 >> v2:172.21.15.23:3300/0 conn(0x7fad680aae90 0x7fad681381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:33:01.741 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:01.736+0000 7fad657fa700 1 -- 172.21.15.23:0/1417949287 >> v1:172.21.15.155:6789/0 conn(0x7fad6813aeb0 legacy=0x7fad680a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:01.741 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:01.736+0000 7fad657fa700 1 --2- 172.21.15.23:0/1417949287 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad48007010 0x7fad480094d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:01.741 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:01.736+0000 7fad657fa700 1 -- 172.21.15.23:0/1417949287 --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad680a9370 con 0x7fad480033c0 2020-11-20T11:33:01.741 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:01.736+0000 7fad657fa700 1 -- 172.21.15.23:0/1417949287 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad6813e8a0 con 0x7fad48007010 2020-11-20T11:33:01.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:01.736+0000 7fad677fe700 1 -- 172.21.15.23:0/1417949287 >> v1:172.21.15.23:6789/0 conn(0x7fad480033c0 legacy=0x7fad48005840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2020-11-20T11:33:01.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:01.737+0000 7fad66ffd700 1 --2- 172.21.15.23:0/1417949287 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad48007010 0x7fad480094d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:01.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:01.737+0000 7fad66ffd700 1 -- 172.21.15.23:0/1417949287 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad48007010 msgr2=0x7fad480094d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:33:01.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:01.737+0000 7fad66ffd700 1 -- 172.21.15.23:0/1417949287 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad48007010 msgr2=0x7fad480094d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:33:01.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:01.737+0000 7fad66ffd700 1 --2- 172.21.15.23:0/1417949287 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad48007010 0x7fad480094d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:33:01.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:01.737+0000 7fad66ffd700 1 --2- 172.21.15.23:0/1417949287 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad48007010 0x7fad480094d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:33:01.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:01.937+0000 7fad677fe700 1 -- 172.21.15.23:0/1417949287 >> v1:172.21.15.23:6789/0 conn(0x7fad480033c0 legacy=0x7fad48005840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2020-11-20T11:33:01.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:01.938+0000 7fad66ffd700 1 --2- 172.21.15.23:0/1417949287 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad48007010 0x7fad480094d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:01.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:01.938+0000 7fad66ffd700 1 -- 172.21.15.23:0/1417949287 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad48007010 msgr2=0x7fad480094d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:33:01.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:01.938+0000 7fad66ffd700 1 -- 172.21.15.23:0/1417949287 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad48007010 msgr2=0x7fad480094d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:33:01.942 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:01.938+0000 7fad66ffd700 1 --2- 172.21.15.23:0/1417949287 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad48007010 0x7fad480094d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:33:01.942 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:01.938+0000 7fad66ffd700 1 --2- 172.21.15.23:0/1417949287 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad48007010 0x7fad480094d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:33:02.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:02.338+0000 7fad677fe700 1 -- 172.21.15.23:0/1417949287 >> v1:172.21.15.23:6789/0 conn(0x7fad480033c0 legacy=0x7fad48005840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2020-11-20T11:33:02.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:02.339+0000 7fad66ffd700 1 --2- 172.21.15.23:0/1417949287 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad48007010 0x7fad480094d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:02.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:02.339+0000 7fad66ffd700 1 -- 172.21.15.23:0/1417949287 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad48007010 msgr2=0x7fad480094d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:33:02.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:02.339+0000 7fad66ffd700 1 -- 172.21.15.23:0/1417949287 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad48007010 msgr2=0x7fad480094d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:33:02.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:02.339+0000 7fad66ffd700 1 --2- 172.21.15.23:0/1417949287 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad48007010 0x7fad480094d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:33:02.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:02.339+0000 7fad66ffd700 1 --2- 172.21.15.23:0/1417949287 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad48007010 0x7fad480094d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:33:03.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.139+0000 7fad677fe700 1 -- 172.21.15.23:0/1417949287 >> v1:172.21.15.23:6789/0 conn(0x7fad480033c0 legacy=0x7fad48005840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2020-11-20T11:33:03.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.141+0000 7fad66ffd700 1 --2- 172.21.15.23:0/1417949287 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad48007010 0x7fad480094d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:03.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.141+0000 7fad66ffd700 1 -- 172.21.15.23:0/1417949287 >> v1:172.21.15.23:6789/0 conn(0x7fad480033c0 legacy=0x7fad48005840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:33:03.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.141+0000 7fad66ffd700 1 -- 172.21.15.23:0/1417949287 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4800ac50 con 0x7fad48007010 2020-11-20T11:33:03.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.142+0000 7fad66ffd700 1 --2- 172.21.15.23:0/1417949287 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad48007010 0x7fad480094d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fad5c0036e0 tx=0x7fad5c008040).ready entity=mon.2 client_cookie=2f7e3a0a5230da8d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:03.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.142+0000 7fad667fc700 1 -- 172.21.15.23:0/1417949287 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5c003c90 con 0x7fad48007010 2020-11-20T11:33:03.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.142+0000 7fad667fc700 1 -- 172.21.15.23:0/1417949287 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad5c003df0 con 0x7fad48007010 2020-11-20T11:33:03.148 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.142+0000 7fad667fc700 1 -- 172.21.15.23:0/1417949287 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5c005cb0 con 0x7fad48007010 2020-11-20T11:33:03.152 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.143+0000 7fad6dca0700 1 -- 172.21.15.23:0/1417949287 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad48007010 msgr2=0x7fad480094d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:03.152 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.143+0000 7fad6dca0700 1 --2- 172.21.15.23:0/1417949287 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad48007010 0x7fad480094d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fad5c0036e0 tx=0x7fad5c008040).stop 2020-11-20T11:33:03.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.143+0000 7fad67fff700 1 -- 172.21.15.23:0/1417949287 reap_dead start 2020-11-20T11:33:03.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.143+0000 7fad6dca0700 1 -- 172.21.15.23:0/1417949287 shutdown_connections 2020-11-20T11:33:03.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.143+0000 7fad6dca0700 1 -- 172.21.15.23:0/1417949287 >> 172.21.15.23:0/1417949287 conn(0x7fad680678e0 msgr2=0x7fad68067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:33:03.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.143+0000 7fad6dca0700 1 -- 172.21.15.23:0/1417949287 shutdown_connections 2020-11-20T11:33:03.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.143+0000 7fad6dca0700 1 -- 172.21.15.23:0/1417949287 wait complete. 2020-11-20T11:33:03.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.144+0000 7fad6dca0700 1 Processor -- start 2020-11-20T11:33:03.155 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.144+0000 7fad6dca0700 1 -- start start 2020-11-20T11:33:03.155 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.145+0000 7fad6dca0700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fad48007010 0x7fad6813c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:03.155 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.145+0000 7fad6dca0700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad68134d20 0x7fad680b0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:03.155 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.145+0000 7fad6dca0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad680b0af0 0x7fad6814d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:03.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.145+0000 7fad6dca0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad680a5b50 con 0x7fad680b0af0 2020-11-20T11:33:03.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.145+0000 7fad6dca0700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad681351a0 con 0x7fad48007010 2020-11-20T11:33:03.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.145+0000 7fad6dca0700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad6813c320 con 0x7fad68134d20 2020-11-20T11:33:03.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.145+0000 7fad67fff700 1 -- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad680b0af0 msgr2=0x7fad6814d8f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:33:03.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.145+0000 7fad67fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad680b0af0 0x7fad6814d8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:33:03.157 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.145+0000 7fad677fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fad48007010 0x7fad6813c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:03.157 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.145+0000 7fad66ffd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad68134d20 0x7fad680b0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:03.157 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.145+0000 7fad677fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fad48007010 0x7fad6813c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:34610/0 (socket says 172.21.15.23:34610) 2020-11-20T11:33:03.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.145+0000 7fad677fe700 1 -- 172.21.15.23:0/2275054357 learned_addr learned my addr 172.21.15.23:0/2275054357 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:33:03.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.146+0000 7fad677fe700 1 -- 172.21.15.23:0/2275054357 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad68134d20 msgr2=0x7fad680b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:03.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.146+0000 7fad677fe700 1 --2- 172.21.15.23:0/2275054357 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad68134d20 0x7fad680b0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:03.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.146+0000 7fad677fe700 1 -- 172.21.15.23:0/2275054357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad680b0af0 msgr2=0x7fad6814d8f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:33:03.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.146+0000 7fad677fe700 1 --2- 172.21.15.23:0/2275054357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad680b0af0 0x7fad6814d8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:33:03.159 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.146+0000 7fad677fe700 1 -- 172.21.15.23:0/2275054357 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad680a5410 con 0x7fad48007010 2020-11-20T11:33:03.159 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.146+0000 7fad66ffd700 1 --2- 172.21.15.23:0/2275054357 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad68134d20 0x7fad680b0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:33:03.159 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.146+0000 7fad677fe700 1 --2- 172.21.15.23:0/2275054357 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fad48007010 0x7fad6813c720 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fad58012cb0 tx=0x7fad58008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:03.159 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.146+0000 7fad477fe700 1 -- 172.21.15.23:0/2275054357 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5800f9b0 con 0x7fad48007010 2020-11-20T11:33:03.159 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.146+0000 7fad477fe700 1 -- 172.21.15.23:0/2275054357 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad58006e90 con 0x7fad48007010 2020-11-20T11:33:03.160 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.146+0000 7fad6dca0700 1 -- 172.21.15.23:0/2275054357 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad680a56b0 con 0x7fad48007010 2020-11-20T11:33:03.160 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.146+0000 7fad477fe700 1 -- 172.21.15.23:0/2275054357 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5800fd20 con 0x7fad48007010 2020-11-20T11:33:03.160 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.146+0000 7fad6dca0700 1 -- 172.21.15.23:0/2275054357 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad6814dde0 con 0x7fad48007010 2020-11-20T11:33:03.160 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.147+0000 7fad6dca0700 1 -- 172.21.15.23:0/2275054357 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad6805c7d0 con 0x7fad48007010 2020-11-20T11:33:03.160 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.147+0000 7fad477fe700 1 -- 172.21.15.23:0/2275054357 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fad58025070 con 0x7fad48007010 2020-11-20T11:33:03.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.148+0000 7fad477fe700 1 --2- 172.21.15.23:0/2275054357 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fad40056d10 0x7fad400591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:03.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.148+0000 7fad477fe700 1 -- 172.21.15.23:0/2275054357 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fad58000e40 con 0x7fad48007010 2020-11-20T11:33:03.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.148+0000 7fad66ffd700 1 --2- 172.21.15.23:0/2275054357 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fad40056d10 0x7fad400591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:03.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.149+0000 7fad66ffd700 1 --2- 172.21.15.23:0/2275054357 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fad40056d10 0x7fad400591d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fad68139640 tx=0x7fad5c008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:03.162 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.152+0000 7fad477fe700 1 -- 172.21.15.23:0/2275054357 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fad58045ce0 con 0x7fad48007010 2020-11-20T11:33:03.368 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.364+0000 7fad6dca0700 1 -- 172.21.15.23:0/2275054357 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fad68044710 con 0x7fad48007010 2020-11-20T11:33:03.369 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.364+0000 7fad477fe700 1 -- 172.21.15.23:0/2275054357 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1470 (secure 0 0 0) 0x7fad58049300 con 0x7fad48007010 2020-11-20T11:33:03.370 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":6,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:33:03.376 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.366+0000 7fad6dca0700 1 -- 172.21.15.23:0/2275054357 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fad40056d10 msgr2=0x7fad400591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:03.378 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.366+0000 7fad6dca0700 1 --2- 172.21.15.23:0/2275054357 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fad40056d10 0x7fad400591d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fad68139640 tx=0x7fad5c008040).stop 2020-11-20T11:33:03.379 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.366+0000 7fad6dca0700 1 -- 172.21.15.23:0/2275054357 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fad48007010 msgr2=0x7fad6813c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:03.379 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.366+0000 7fad6dca0700 1 --2- 172.21.15.23:0/2275054357 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fad48007010 0x7fad6813c720 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fad58012cb0 tx=0x7fad58008040).stop 2020-11-20T11:33:03.379 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.366+0000 7fad6dca0700 1 -- 172.21.15.23:0/2275054357 shutdown_connections 2020-11-20T11:33:03.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.366+0000 7fad6dca0700 1 --2- 172.21.15.23:0/2275054357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad680b0af0 0x7fad6814d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:33:03.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.366+0000 7fad6dca0700 1 --2- 172.21.15.23:0/2275054357 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fad48007010 0x7fad6813c720 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:03.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.366+0000 7fad6dca0700 1 --2- 172.21.15.23:0/2275054357 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fad40056d10 0x7fad400591d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:03.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.366+0000 7fad6dca0700 1 --2- 172.21.15.23:0/2275054357 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fad68134d20 0x7fad680b0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:03.381 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.366+0000 7fad6dca0700 1 -- 172.21.15.23:0/2275054357 >> 172.21.15.23:0/2275054357 conn(0x7fad680678e0 msgr2=0x7fad680add90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:33:03.381 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.366+0000 7fad6dca0700 1 -- 172.21.15.23:0/2275054357 shutdown_connections 2020-11-20T11:33:03.381 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.366+0000 7fad6dca0700 1 -- 172.21.15.23:0/2275054357 wait complete. 2020-11-20T11:33:03.383 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2020-11-20T11:33:03.383 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:33:03.517 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.498+0000 7f987586e700 1 Processor -- start 2020-11-20T11:33:03.518 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.498+0000 7f987586e700 1 -- start start 2020-11-20T11:33:03.518 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.498+0000 7f987586e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f98700a6a70 0x7f987013af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:03.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.498+0000 7f987586e700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f987013be80 con 0x7f98700ad600 2020-11-20T11:33:03.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.498+0000 7f987586e700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98700a6f10 con 0x7f98700ad9e0 2020-11-20T11:33:03.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.498+0000 7f987586e700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98700a71b0 con 0x7f98700a6a70 2020-11-20T11:33:03.520 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.498+0000 7f986effd700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f98700ad600 legacy=0x7f98700a90a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2020-11-20T11:33:03.521 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.499+0000 7f986f7fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f98700a6a70 0x7f987013af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:03.521 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.499+0000 7f986f7fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f98700a6a70 0x7f987013af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:34622/0 (socket says 172.21.15.23:34622) 2020-11-20T11:33:03.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.499+0000 7f986f7fe700 1 -- 172.21.15.23:0/2926478867 learned_addr learned my addr 172.21.15.23:0/2926478867 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:33:03.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.499+0000 7f986f7fe700 1 -- 172.21.15.23:0/2926478867 >> v1:172.21.15.155:6789/0 conn(0x7f98700ad9e0 legacy=0x7f987013b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:03.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.499+0000 7f986f7fe700 1 -- 172.21.15.23:0/2926478867 >> v1:172.21.15.23:6789/0 conn(0x7f98700ad600 legacy=0x7f98700a90a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:33:03.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.499+0000 7f986f7fe700 1 -- 172.21.15.23:0/2926478867 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98700a4da0 con 0x7f98700a6a70 2020-11-20T11:33:03.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.499+0000 7f986f7fe700 1 --2- 172.21.15.23:0/2926478867 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98700a6a70 0x7f987013af90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f9868009d90 tx=0x7f98680092d0).ready entity=mon.1 client_cookie=5ada89729782f809 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:03.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.499+0000 7f986dffb700 1 -- 172.21.15.23:0/2926478867 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986800bbb0 con 0x7f98700a6a70 2020-11-20T11:33:03.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.499+0000 7f986dffb700 1 -- 172.21.15.23:0/2926478867 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f986800bd10 con 0x7f98700a6a70 2020-11-20T11:33:03.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.499+0000 7f986dffb700 1 -- 172.21.15.23:0/2926478867 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9868011610 con 0x7f98700a6a70 2020-11-20T11:33:03.525 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.500+0000 7f987586e700 1 -- 172.21.15.23:0/2926478867 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98700a6a70 msgr2=0x7f987013af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:03.525 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.500+0000 7f987586e700 1 --2- 172.21.15.23:0/2926478867 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98700a6a70 0x7f987013af90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f9868009d90 tx=0x7f98680092d0).stop 2020-11-20T11:33:03.526 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.500+0000 7f987586e700 1 -- 172.21.15.23:0/2926478867 shutdown_connections 2020-11-20T11:33:03.526 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.500+0000 7f987586e700 1 --2- 172.21.15.23:0/2926478867 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98700a6a70 0x7f987013af90 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:03.526 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.500+0000 7f987586e700 1 -- 172.21.15.23:0/2926478867 >> 172.21.15.23:0/2926478867 conn(0x7f9870060c80 msgr2=0x7f98700acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:33:03.527 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.500+0000 7f987586e700 1 -- 172.21.15.23:0/2926478867 shutdown_connections 2020-11-20T11:33:03.527 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.500+0000 7f987586e700 1 -- 172.21.15.23:0/2926478867 wait complete. 2020-11-20T11:33:03.528 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.501+0000 7f987586e700 1 Processor -- start 2020-11-20T11:33:03.528 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.501+0000 7f987586e700 1 -- start start 2020-11-20T11:33:03.528 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.502+0000 7f987586e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f98700a6a70 0x7f9870061a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:03.529 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.502+0000 7f987586e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f98700ad600 0x7f98700a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:03.529 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.502+0000 7f987586e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98700ad9e0 0x7f987014d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:03.530 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.502+0000 7f987586e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98700a1f90 con 0x7f98700ad600 2020-11-20T11:33:03.530 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.502+0000 7f987586e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98700a2230 con 0x7f98700ad9e0 2020-11-20T11:33:03.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.502+0000 7f987586e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98700a24d0 con 0x7f98700a6a70 2020-11-20T11:33:03.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.502+0000 7f986e7fc700 1 -- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f98700ad600 msgr2=0x7f98700a3140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:33:03.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.502+0000 7f986e7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f98700ad600 0x7f98700a3140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:33:03.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.502+0000 7f986f7fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98700ad9e0 0x7f987014d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:03.533 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.502+0000 7f986f7fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98700ad9e0 0x7f987014d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:34628/0 (socket says 172.21.15.23:34628) 2020-11-20T11:33:03.533 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.502+0000 7f986f7fe700 1 -- 172.21.15.23:0/3570962222 learned_addr learned my addr 172.21.15.23:0/3570962222 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:33:03.534 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.502+0000 7f986effd700 1 --2- 172.21.15.23:0/3570962222 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f98700a6a70 0x7f9870061a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:03.534 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.502+0000 7f986f7fe700 1 -- 172.21.15.23:0/3570962222 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f98700a6a70 msgr2=0x7f9870061a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:03.534 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.502+0000 7f986f7fe700 1 --2- 172.21.15.23:0/3570962222 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f98700a6a70 0x7f9870061a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:03.535 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.502+0000 7f986f7fe700 1 -- 172.21.15.23:0/3570962222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f98700ad600 msgr2=0x7f98700a3140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:33:03.535 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.502+0000 7f986f7fe700 1 --2- 172.21.15.23:0/3570962222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f98700ad600 0x7f98700a3140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:33:03.536 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.502+0000 7f986f7fe700 1 -- 172.21.15.23:0/3570962222 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98700a3630 con 0x7f98700ad9e0 2020-11-20T11:33:03.536 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.502+0000 7f986f7fe700 1 --2- 172.21.15.23:0/3570962222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98700ad9e0 0x7f987014d6a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f9868009130 tx=0x7f9868016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:03.536 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.502+0000 7f9856ffd700 1 -- 172.21.15.23:0/3570962222 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986800be80 con 0x7f98700ad9e0 2020-11-20T11:33:03.537 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.502+0000 7f9856ffd700 1 -- 172.21.15.23:0/3570962222 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9868010b10 con 0x7f98700ad9e0 2020-11-20T11:33:03.537 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.502+0000 7f987586e700 1 -- 172.21.15.23:0/3570962222 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98700a1af0 con 0x7f98700ad9e0 2020-11-20T11:33:03.538 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.502+0000 7f9856ffd700 1 -- 172.21.15.23:0/3570962222 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9868010e90 con 0x7f98700ad9e0 2020-11-20T11:33:03.538 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.503+0000 7f987586e700 1 -- 172.21.15.23:0/3570962222 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f987014ddd0 con 0x7f98700ad9e0 2020-11-20T11:33:03.538 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.503+0000 7f987586e700 1 --2- 172.21.15.23:0/3570962222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9870134120 0x7f9870134500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:03.539 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.503+0000 7f987586e700 1 -- 172.21.15.23:0/3570962222 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98700a1e20 con 0x7f9870134120 2020-11-20T11:33:03.539 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.503+0000 7f986effd700 1 --2- 172.21.15.23:0/3570962222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9870134120 0x7f9870134500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:03.539 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.504+0000 7f9856ffd700 1 -- 172.21.15.23:0/3570962222 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9868017070 con 0x7f98700ad9e0 2020-11-20T11:33:03.540 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.504+0000 7f986effd700 1 -- 172.21.15.23:0/3570962222 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f98700a4f60 con 0x7f9870134120 2020-11-20T11:33:03.540 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.504+0000 7f986effd700 1 --2- 172.21.15.23:0/3570962222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9870134120 0x7f9870134500 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f98640036e0 tx=0x7f9864006e10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:03.541 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.504+0000 7f9856ffd700 1 --2- 172.21.15.23:0/3570962222 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f984c056d10 0x7f984c0591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:03.541 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.504+0000 7f9856ffd700 1 -- 172.21.15.23:0/3570962222 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f9868072420 con 0x7f98700ad9e0 2020-11-20T11:33:03.542 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.504+0000 7f9856ffd700 1 -- 172.21.15.23:0/3570962222 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986400e620 con 0x7f9870134120 2020-11-20T11:33:03.542 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.504+0000 7f986e7fc700 1 --2- 172.21.15.23:0/3570962222 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f984c056d10 0x7f984c0591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:03.542 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.504+0000 7f9856ffd700 1 -- 172.21.15.23:0/3570962222 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f986400e780 con 0x7f9870134120 2020-11-20T11:33:03.543 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.504+0000 7f9856ffd700 1 -- 172.21.15.23:0/3570962222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9870134120 msgr2=0x7f9870134500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:03.543 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.504+0000 7f9856ffd700 1 --2- 172.21.15.23:0/3570962222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9870134120 0x7f9870134500 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f98640036e0 tx=0x7f9864006e10).stop 2020-11-20T11:33:03.544 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.504+0000 7f9856ffd700 1 -- 172.21.15.23:0/3570962222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9870134120 msgr2=0x7f9870134500 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:33:03.544 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.504+0000 7f9856ffd700 1 --2- 172.21.15.23:0/3570962222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9870134120 0x7f9870134500 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:03.544 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.505+0000 7f986e7fc700 1 --2- 172.21.15.23:0/3570962222 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f984c056d10 0x7f984c0591d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f98700a1a80 tx=0x7f9860007810).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:03.545 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.511+0000 7f987586e700 1 --2- 172.21.15.23:0/3570962222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98701349f0 0x7f9870137310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:03.545 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.511+0000 7f987586e700 1 -- 172.21.15.23:0/3570962222 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98700a2370 con 0x7f98701349f0 2020-11-20T11:33:03.545 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.511+0000 7f986effd700 1 --2- 172.21.15.23:0/3570962222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98701349f0 0x7f9870137310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:03.546 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.512+0000 7f986effd700 1 -- 172.21.15.23:0/3570962222 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f98700a2690 con 0x7f98701349f0 2020-11-20T11:33:03.546 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.512+0000 7f986effd700 1 --2- 172.21.15.23:0/3570962222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98701349f0 0x7f9870137310 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f98640036e0 tx=0x7f9864013000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:03.547 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.512+0000 7f9856ffd700 1 -- 172.21.15.23:0/3570962222 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9864023480 con 0x7f98701349f0 2020-11-20T11:33:03.547 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.512+0000 7f9856ffd700 1 -- 172.21.15.23:0/3570962222 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4773 (secure 0 0 0) 0x7f986400e410 con 0x7f98701349f0 2020-11-20T11:33:03.547 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.512+0000 7f9856ffd700 1 -- 172.21.15.23:0/3570962222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98701349f0 msgr2=0x7f9870137310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:03.548 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.512+0000 7f9856ffd700 1 --2- 172.21.15.23:0/3570962222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98701349f0 0x7f9870137310 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f98640036e0 tx=0x7f9864013000).stop 2020-11-20T11:33:03.548 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.512+0000 7f9856ffd700 1 -- 172.21.15.23:0/3570962222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98701349f0 msgr2=0x7f9870137310 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:33:03.548 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.512+0000 7f9856ffd700 1 --2- 172.21.15.23:0/3570962222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98701349f0 0x7f9870137310 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0x7f98640036e0 tx=0x7f9864013000).stop 2020-11-20T11:33:03.549 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.512+0000 7f987586e700 1 -- 172.21.15.23:0/3570962222 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f984c056d10 msgr2=0x7f984c0591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:03.549 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.512+0000 7f987586e700 1 --2- 172.21.15.23:0/3570962222 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f984c056d10 0x7f984c0591d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f98700a1a80 tx=0x7f9860007810).stop 2020-11-20T11:33:03.550 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.512+0000 7f987586e700 1 -- 172.21.15.23:0/3570962222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98700ad9e0 msgr2=0x7f987014d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:03.550 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.512+0000 7f987586e700 1 --2- 172.21.15.23:0/3570962222 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f98700ad9e0 0x7f987014d6a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f9868009130 tx=0x7f9868016000).stop 2020-11-20T11:33:03.550 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.512+0000 7f986f7fe700 1 -- 172.21.15.23:0/3570962222 reap_dead start 2020-11-20T11:33:03.551 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.512+0000 7f986f7fe700 1 -- 172.21.15.23:0/3570962222 reap_dead start 2020-11-20T11:33:03.551 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.512+0000 7f987586e700 1 -- 172.21.15.23:0/3570962222 shutdown_connections 2020-11-20T11:33:03.552 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.512+0000 7f987586e700 1 -- 172.21.15.23:0/3570962222 >> 172.21.15.23:0/3570962222 conn(0x7f9870060c80 msgr2=0x7f98700ab070 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:33:03.552 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.512+0000 7f987586e700 1 -- 172.21.15.23:0/3570962222 shutdown_connections 2020-11-20T11:33:03.552 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.512+0000 7f987586e700 1 -- 172.21.15.23:0/3570962222 wait complete. 2020-11-20T11:33:03.553 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:33:03.553 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:33:03.554 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:33:03.554 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:33:03.555 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 6, 2020-11-20T11:33:03.555 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:33:03.555 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:33:03.556 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:33:03.556 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:03.557 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 0, 2020-11-20T11:33:03.557 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:33:03.557 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:33:03.557 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:33:03.557 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:33:03.558 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:33:03.558 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:33:03.559 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:33:03.559 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:33:03.559 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:33:03.560 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:33:03.560 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:33:03.561 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:03.561 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:33:03.561 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:33:03.562 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:33:03.562 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:33:03.562 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:33:03.563 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:33:03.563 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:33:03.563 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:33:03.564 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:33:03.564 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:33:03.565 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:03.565 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:03.565 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:33:03.566 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:33:03.566 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:33:03.567 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:33:03.567 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:33:03.567 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:33:03.568 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:33:03.568 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:33:03.568 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:33:03.569 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:33:03.569 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:33:03.570 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:33:03.570 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:33:03.570 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:33:03.570 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:33:03.570 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:33:03.571 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:33:03.571 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:33:03.571 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:33:03.571 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:33:03.571 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:33:03.572 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:33:03.572 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:33:03.572 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:03.572 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:33:03.572 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:03.573 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:33:03.573 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:03.573 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:33:03.573 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:33:03.573 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:33:03.574 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:33:03.574 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:03.574 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:33:03.574 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:33:03.574 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:03.574 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:03.575 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:03.575 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:33:03.575 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:33:03.575 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:03.575 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:03.576 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:03.576 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:03.576 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:33:03.576 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:33:03.576 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:33:03.577 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:33:03.577 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:33:03.577 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:03.577 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:03.577 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:33:03.578 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:33:03.578 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:33:03.578 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:33:03.578 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:03.578 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:33:03.579 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:33:03.579 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:03.579 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:03.579 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:03.579 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:33:03.580 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:33:03.580 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:03.580 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:03.580 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:03.580 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:03.581 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:33:03.581 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:33:03.581 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:33:03.581 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:33:03.581 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:33:03.582 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:03.582 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:03.582 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:33:03.582 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:33:03.582 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:33:03.582 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:33:03.583 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:03.583 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:33:03.583 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:33:03.583 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:03.583 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:03.584 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:03.584 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:33:03.584 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:33:03.584 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:03.584 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:03.585 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:03.585 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:03.585 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:33:03.585 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:33:03.585 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:33:03.586 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:33:03.586 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:33:03.586 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:03.586 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:03.586 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:03.587 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:33:03.587 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:33:03.587 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:03.587 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:03.587 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:03.587 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:33:03.588 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:03.588 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:03.588 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:33:03.588 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:03.588 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:03.589 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:03.589 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:33:03.589 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:03.589 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:03.589 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:33:03.590 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:03.590 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:03.590 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:03.590 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 4 2020-11-20T11:33:03.590 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:03.591 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:03.591 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:33:03.591 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:03.591 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:03.591 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:03.592 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 6 2020-11-20T11:33:03.592 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:03.592 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:03.592 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:33:03.592 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:03.592 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:03.593 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:03.593 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:33:03.593 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:03.593 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:03.593 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:03.594 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:33:03.594 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:33:03.595 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:33:03.718 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.715+0000 7f41a4c0e700 1 Processor -- start 2020-11-20T11:33:03.718 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.715+0000 7f41a4c0e700 1 -- start start 2020-11-20T11:33:03.721 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.716+0000 7f41a4c0e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f41a00ae760 0x7f41a013e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:03.722 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.716+0000 7f41a4c0e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41a0065100 0x7f41a0145610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:03.722 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.716+0000 7f41a4c0e700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a00661b0 con 0x7f41a00aeb40 2020-11-20T11:33:03.722 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.716+0000 7f41a4c0e700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41a013ad60 con 0x7f41a00ae760 2020-11-20T11:33:03.723 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.716+0000 7f41a4c0e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41a0066c70 con 0x7f41a0065100 2020-11-20T11:33:03.723 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.716+0000 7f419ed9d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41a0065100 0x7f41a0145610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:03.723 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.716+0000 7f419e59c700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f41a00ae760 0x7f41a013e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:03.724 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.716+0000 7f419e59c700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f41a00ae760 0x7f41a013e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:34636/0 (socket says 172.21.15.23:34636) 2020-11-20T11:33:03.724 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.716+0000 7f419ed9d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41a0065100 0x7f41a0145610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:51984/0 (socket says 172.21.15.23:51984) 2020-11-20T11:33:03.724 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.716+0000 7f419e59c700 1 -- 172.21.15.23:0/3417085573 learned_addr learned my addr 172.21.15.23:0/3417085573 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:33:03.724 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.716+0000 7f419ed9d700 1 -- 172.21.15.23:0/3417085573 >> v2:172.21.15.155:3300/0 conn(0x7f41a00ae760 msgr2=0x7f41a013e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:33:03.725 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.716+0000 7f419ed9d700 1 --2- 172.21.15.23:0/3417085573 >> v2:172.21.15.155:3300/0 conn(0x7f41a00ae760 0x7f41a013e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:03.725 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.716+0000 7f419ed9d700 1 -- 172.21.15.23:0/3417085573 >> v1:172.21.15.155:6789/0 conn(0x7f41a00aeb40 legacy=0x7f41a0137ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:03.726 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.716+0000 7f419ed9d700 1 -- 172.21.15.23:0/3417085573 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a013e470 con 0x7f41a0065100 2020-11-20T11:33:03.726 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.716+0000 7f419e59c700 1 --2- 172.21.15.23:0/3417085573 >> v2:172.21.15.155:3300/0 conn(0x7f41a00ae760 0x7f41a013e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:33:03.726 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.717+0000 7f419ed9d700 1 --2- 172.21.15.23:0/3417085573 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41a0065100 0x7f41a0145610 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f4198009ce0 tx=0x7f4198009220).ready entity=mon.2 client_cookie=8844c8531666b850 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:03.727 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.717+0000 7f419d59a700 1 -- 172.21.15.23:0/3417085573 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419800ba40 con 0x7f41a0065100 2020-11-20T11:33:03.727 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.717+0000 7f419d59a700 1 -- 172.21.15.23:0/3417085573 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f419800bba0 con 0x7f41a0065100 2020-11-20T11:33:03.728 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.717+0000 7f419d59a700 1 -- 172.21.15.23:0/3417085573 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4198011400 con 0x7f41a0065100 2020-11-20T11:33:03.728 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.717+0000 7f41a4c0e700 1 -- 172.21.15.23:0/3417085573 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41a0065100 msgr2=0x7f41a0145610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:03.729 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.717+0000 7f41a4c0e700 1 --2- 172.21.15.23:0/3417085573 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41a0065100 0x7f41a0145610 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f4198009ce0 tx=0x7f4198009220).stop 2020-11-20T11:33:03.729 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.717+0000 7f41a4c0e700 1 -- 172.21.15.23:0/3417085573 shutdown_connections 2020-11-20T11:33:03.730 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.717+0000 7f41a4c0e700 1 --2- 172.21.15.23:0/3417085573 >> v2:172.21.15.155:3300/0 conn(0x7f41a00ae760 0x7f41a013e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:03.730 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.717+0000 7f41a4c0e700 1 --2- 172.21.15.23:0/3417085573 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41a0065100 0x7f41a0145610 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:03.730 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.717+0000 7f41a4c0e700 1 -- 172.21.15.23:0/3417085573 >> 172.21.15.23:0/3417085573 conn(0x7f41a0061d80 msgr2=0x7f41a00ae050 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:33:03.731 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.717+0000 7f41a4c0e700 1 -- 172.21.15.23:0/3417085573 shutdown_connections 2020-11-20T11:33:03.731 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.717+0000 7f41a4c0e700 1 -- 172.21.15.23:0/3417085573 wait complete. 2020-11-20T11:33:03.732 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.718+0000 7f41a4c0e700 1 Processor -- start 2020-11-20T11:33:03.732 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.718+0000 7f41a4c0e700 1 -- start start 2020-11-20T11:33:03.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.719+0000 7f41a4c0e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41a0065100 0x7f41a013d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:03.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.719+0000 7f41a4c0e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41a00ae760 0x7f41a00a5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:03.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.719+0000 7f41a4c0e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41a00aeb40 0x7f41a014dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:03.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.719+0000 7f41a4c0e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a0138b80 con 0x7f41a00aeb40 2020-11-20T11:33:03.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.719+0000 7f41a4c0e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a00a6720 con 0x7f41a00ae760 2020-11-20T11:33:03.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.719+0000 7f41a4c0e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41a014d390 con 0x7f41a0065100 2020-11-20T11:33:03.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.719+0000 7f419ed9d700 1 -- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41a00aeb40 msgr2=0x7f41a014dad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:33:03.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.719+0000 7f419ed9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41a00aeb40 0x7f41a014dad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:33:03.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.719+0000 7f419e59c700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41a0065100 0x7f41a013d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:03.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.719+0000 7f419dd9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41a00ae760 0x7f41a00a5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:03.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.719+0000 7f419dd9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41a00ae760 0x7f41a00a5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:34644/0 (socket says 172.21.15.23:34644) 2020-11-20T11:33:03.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.719+0000 7f419e59c700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41a0065100 0x7f41a013d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:51986/0 (socket says 172.21.15.23:51986) 2020-11-20T11:33:03.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.719+0000 7f419dd9b700 1 -- 172.21.15.23:0/2336137439 learned_addr learned my addr 172.21.15.23:0/2336137439 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:33:03.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.719+0000 7f419e59c700 1 -- 172.21.15.23:0/2336137439 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41a00ae760 msgr2=0x7f41a00a5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:03.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.719+0000 7f419e59c700 1 --2- 172.21.15.23:0/2336137439 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41a00ae760 0x7f41a00a5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:03.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.719+0000 7f419e59c700 1 -- 172.21.15.23:0/2336137439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41a00aeb40 msgr2=0x7f41a014dad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:33:03.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.719+0000 7f419e59c700 1 --2- 172.21.15.23:0/2336137439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41a00aeb40 0x7f41a014dad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:33:03.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.720+0000 7f419e59c700 1 -- 172.21.15.23:0/2336137439 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a014d630 con 0x7f41a0065100 2020-11-20T11:33:03.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.720+0000 7f419dd9b700 1 --2- 172.21.15.23:0/2336137439 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41a00ae760 0x7f41a00a5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:33:03.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.720+0000 7f419e59c700 1 --2- 172.21.15.23:0/2336137439 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41a0065100 0x7f41a013d810 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f418800ec00 tx=0x7f4188006ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:03.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.720+0000 7f41967fc700 1 -- 172.21.15.23:0/2336137439 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418800c980 con 0x7f41a0065100 2020-11-20T11:33:03.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.720+0000 7f41967fc700 1 -- 172.21.15.23:0/2336137439 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f418800cae0 con 0x7f41a0065100 2020-11-20T11:33:03.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.720+0000 7f41a4c0e700 1 -- 172.21.15.23:0/2336137439 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41a00a63e0 con 0x7f41a0065100 2020-11-20T11:33:03.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.720+0000 7f41967fc700 1 -- 172.21.15.23:0/2336137439 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418800ce60 con 0x7f41a0065100 2020-11-20T11:33:03.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.720+0000 7f41a4c0e700 1 -- 172.21.15.23:0/2336137439 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41a00a1980 con 0x7f41a0065100 2020-11-20T11:33:03.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.721+0000 7f41a4c0e700 1 --2- 172.21.15.23:0/2336137439 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41a014a3a0 0x7f41a014a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:03.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.721+0000 7f41a4c0e700 1 -- 172.21.15.23:0/2336137439 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41a0149ee0 con 0x7f41a014a3a0 2020-11-20T11:33:03.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.721+0000 7f419dd9b700 1 --2- 172.21.15.23:0/2336137439 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41a014a3a0 0x7f41a014a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:03.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.721+0000 7f419dd9b700 1 -- 172.21.15.23:0/2336137439 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f41a0062760 con 0x7f41a014a3a0 2020-11-20T11:33:03.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.721+0000 7f41967fc700 1 -- 172.21.15.23:0/2336137439 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4188022070 con 0x7f41a0065100 2020-11-20T11:33:03.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.721+0000 7f419dd9b700 1 --2- 172.21.15.23:0/2336137439 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41a014a3a0 0x7f41a014a780 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f41a00adf20 tx=0x7f4190009040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:03.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.721+0000 7f41967fc700 1 --2- 172.21.15.23:0/2336137439 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f417c056d10 0x7f417c0591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:03.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.721+0000 7f41967fc700 1 -- 172.21.15.23:0/2336137439 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f41880705f0 con 0x7f41a0065100 2020-11-20T11:33:03.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.722+0000 7f41967fc700 1 -- 172.21.15.23:0/2336137439 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4190001d20 con 0x7f41a014a3a0 2020-11-20T11:33:03.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.722+0000 7f419ed9d700 1 --2- 172.21.15.23:0/2336137439 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f417c056d10 0x7f417c0591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:03.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.722+0000 7f41967fc700 1 -- 172.21.15.23:0/2336137439 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f4190007240 con 0x7f41a014a3a0 2020-11-20T11:33:03.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.722+0000 7f41967fc700 1 -- 172.21.15.23:0/2336137439 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41a014a3a0 msgr2=0x7f41a014a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:03.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.722+0000 7f41967fc700 1 --2- 172.21.15.23:0/2336137439 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41a014a3a0 0x7f41a014a780 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f41a00adf20 tx=0x7f4190009040).stop 2020-11-20T11:33:03.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.722+0000 7f41967fc700 1 -- 172.21.15.23:0/2336137439 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41a014a3a0 msgr2=0x7f41a014a780 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:33:03.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.722+0000 7f41967fc700 1 --2- 172.21.15.23:0/2336137439 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41a014a3a0 0x7f41a014a780 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0x7f41a00adf20 tx=0x7f4190009040).stop 2020-11-20T11:33:03.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.722+0000 7f419ed9d700 1 --2- 172.21.15.23:0/2336137439 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f417c056d10 0x7f417c0591d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f41a0066500 tx=0x7f4198016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:03.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.728+0000 7f41a4c0e700 1 --2- 172.21.15.23:0/2336137439 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41a00ace30 0x7f41a00ad240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:03.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.728+0000 7f41a4c0e700 1 -- 172.21.15.23:0/2336137439 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41a005c280 con 0x7f41a00ace30 2020-11-20T11:33:03.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.729+0000 7f419dd9b700 1 --2- 172.21.15.23:0/2336137439 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41a00ace30 0x7f41a00ad240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:03.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.729+0000 7f419dd9b700 1 -- 172.21.15.23:0/2336137439 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f41a014ac70 con 0x7f41a00ace30 2020-11-20T11:33:03.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.729+0000 7f419dd9b700 1 --2- 172.21.15.23:0/2336137439 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41a00ace30 0x7f41a00ad240 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f41a00adf20 tx=0x7f4190018000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:03.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.729+0000 7f41967fc700 1 -- 172.21.15.23:0/2336137439 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4190025410 con 0x7f41a00ace30 2020-11-20T11:33:03.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.730+0000 7f41967fc700 1 -- 172.21.15.23:0/2336137439 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4771 (secure 0 0 0) 0x7f4190007030 con 0x7f41a00ace30 2020-11-20T11:33:03.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.730+0000 7f41967fc700 1 -- 172.21.15.23:0/2336137439 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41a00ace30 msgr2=0x7f41a00ad240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:03.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.730+0000 7f41967fc700 1 --2- 172.21.15.23:0/2336137439 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41a00ace30 0x7f41a00ad240 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f41a00adf20 tx=0x7f4190018000).stop 2020-11-20T11:33:03.769 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.730+0000 7f41967fc700 1 -- 172.21.15.23:0/2336137439 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41a00ace30 msgr2=0x7f41a00ad240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:33:03.769 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.730+0000 7f41967fc700 1 --2- 172.21.15.23:0/2336137439 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41a00ace30 0x7f41a00ad240 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:03.770 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.730+0000 7f41a4c0e700 1 -- 172.21.15.23:0/2336137439 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f417c056d10 msgr2=0x7f417c0591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:03.770 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.730+0000 7f41a4c0e700 1 --2- 172.21.15.23:0/2336137439 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f417c056d10 0x7f417c0591d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f41a0066500 tx=0x7f4198016000).stop 2020-11-20T11:33:03.771 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.730+0000 7f41a4c0e700 1 -- 172.21.15.23:0/2336137439 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41a0065100 msgr2=0x7f41a013d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:03.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.730+0000 7f41a4c0e700 1 --2- 172.21.15.23:0/2336137439 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41a0065100 0x7f41a013d810 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f418800ec00 tx=0x7f4188006ef0).stop 2020-11-20T11:33:03.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.730+0000 7f419ed9d700 1 -- 172.21.15.23:0/2336137439 reap_dead start 2020-11-20T11:33:03.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.730+0000 7f419ed9d700 1 -- 172.21.15.23:0/2336137439 reap_dead start 2020-11-20T11:33:03.773 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.730+0000 7f41a4c0e700 1 -- 172.21.15.23:0/2336137439 shutdown_connections 2020-11-20T11:33:03.773 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.730+0000 7f41a4c0e700 1 -- 172.21.15.23:0/2336137439 >> 172.21.15.23:0/2336137439 conn(0x7f41a0061d80 msgr2=0x7f41a00632c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:33:03.774 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.730+0000 7f41a4c0e700 1 -- 172.21.15.23:0/2336137439 shutdown_connections 2020-11-20T11:33:03.774 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:03.730+0000 7f41a4c0e700 1 -- 172.21.15.23:0/2336137439 wait complete. 2020-11-20T11:33:03.775 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:33:03.776 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:33:03.776 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:33:03.777 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:33:03.778 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 6, 2020-11-20T11:33:03.778 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:33:03.778 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:33:03.778 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:33:03.779 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:03.779 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 0, 2020-11-20T11:33:03.779 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:33:03.780 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:33:03.780 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:33:03.781 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:33:03.781 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:33:03.781 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:33:03.781 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:33:03.782 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:33:03.782 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:33:03.782 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:33:03.783 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:33:03.783 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:03.783 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:33:03.784 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:33:03.784 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:33:03.784 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:33:03.784 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:33:03.784 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:33:03.785 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:33:03.785 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:33:03.785 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:33:03.785 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:33:03.785 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:03.786 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:03.786 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:33:03.787 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:33:03.787 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:33:03.787 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:33:03.787 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:33:03.788 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:33:03.788 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:33:03.789 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:33:03.789 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:33:03.789 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:33:03.790 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:33:03.790 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:33:03.790 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:33:03.791 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:33:03.791 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:33:03.791 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:33:03.792 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:33:03.792 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:33:03.792 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:33:03.793 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:33:03.793 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:33:03.794 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:33:03.794 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:33:03.794 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:03.795 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:33:03.795 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:03.795 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:33:03.796 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:03.796 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:33:03.797 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:33:03.797 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:33:03.797 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:33:03.798 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:03.798 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:33:03.798 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:33:03.799 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:03.799 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:03.799 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:03.799 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:33:03.800 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:33:03.800 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:03.800 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:03.800 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:03.800 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:03.801 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:33:03.801 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:33:03.801 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:33:03.801 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:33:03.801 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:33:03.802 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:03.802 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:03.802 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:33:03.802 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:33:03.802 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:33:03.803 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:33:03.803 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:03.803 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:33:03.803 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:33:03.804 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:03.804 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:03.804 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:03.804 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:33:03.804 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:33:03.805 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:03.805 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:03.805 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:03.805 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:03.806 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:33:03.806 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:33:03.806 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:33:03.807 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:33:03.807 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:33:03.807 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:03.807 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:03.808 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:33:03.808 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:33:03.808 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:33:03.809 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:33:03.809 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:03.809 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:33:03.810 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:33:03.810 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:03.810 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:03.810 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:03.811 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:33:03.811 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:33:03.811 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:03.811 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:03.811 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:03.812 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:03.812 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:33:03.812 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:33:03.812 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:33:03.812 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:33:03.813 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:33:03.813 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:03.813 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:03.813 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:03.813 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:33:03.814 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:33:03.814 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:03.814 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:03.814 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:03.814 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:33:03.815 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:03.815 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:03.815 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:33:03.815 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:03.815 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:03.816 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:03.816 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:33:03.816 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:03.816 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:03.816 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:33:03.817 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:03.817 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:03.817 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:03.817 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 4 2020-11-20T11:33:03.817 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:03.818 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:03.818 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:33:03.818 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:03.818 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:03.818 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:03.819 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:33:03.819 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:03.819 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:03.819 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:33:03.819 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:03.820 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:03.820 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:03.820 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:33:03.820 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:03.820 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:03.821 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:03.821 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:33:03.821 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:33:03.821 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2020-11-20T11:33:23.822 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2020-11-20T11:33:23.823 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2020-11-20T11:33:23.823 INFO:tasks.ceph.mon.a:Restarting daemon 2020-11-20T11:33:23.824 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2020-11-20T11:33:23.826 INFO:tasks.ceph.mon.a:Started 2020-11-20T11:33:23.826 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:33:23.827 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:33:23.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.980+0000 7fdb615f3700 1 Processor -- start 2020-11-20T11:33:23.983 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.980+0000 7fdb615f3700 1 -- start start 2020-11-20T11:33:23.986 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.981+0000 7fdb615f3700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fdb5c061380 0x7fdb5c13afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:23.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.981+0000 7fdb615f3700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5c0a1c30 con 0x7fdb5c0aacf0 2020-11-20T11:33:23.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.981+0000 7fdb615f3700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5c0a9db0 con 0x7fdb5c0a9810 2020-11-20T11:33:23.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.981+0000 7fdb615f3700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb5c0a24f0 con 0x7fdb5c061380 2020-11-20T11:33:23.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.981+0000 7fdb5b7fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fdb5c0aacf0 0x7fdb5c145490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:56018/0 (socket says 172.21.15.23:56018) 2020-11-20T11:33:23.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.981+0000 7fdb5affd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fdb5c061380 0x7fdb5c13afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:23.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.981+0000 7fdb5b7fe700 1 -- 172.21.15.23:0/2719565710 learned_addr learned my addr 172.21.15.23:0/2719565710 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:33:23.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.981+0000 7fdb5affd700 1 -- 172.21.15.23:0/2719565710 >> v2:172.21.15.23:3300/0 conn(0x7fdb5c061380 msgr2=0x7fdb5c13afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:33:23.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.981+0000 7fdb5affd700 1 -- 172.21.15.23:0/2719565710 >> v2:172.21.15.23:3300/0 conn(0x7fdb5c061380 msgr2=0x7fdb5c13afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:33:23.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.981+0000 7fdb5affd700 1 --2- 172.21.15.23:0/2719565710 >> v2:172.21.15.23:3300/0 conn(0x7fdb5c061380 0x7fdb5c13afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:33:23.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.981+0000 7fdb5affd700 1 --2- 172.21.15.23:0/2719565710 >> v2:172.21.15.23:3300/0 conn(0x7fdb5c061380 0x7fdb5c13afe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:33:23.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.981+0000 7fdb59ffb700 1 -- 172.21.15.23:0/2719565710 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3831859969 0 0) 0x7fdb5c0a9db0 con 0x7fdb5c0a9810 2020-11-20T11:33:23.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.981+0000 7fdb59ffb700 1 -- 172.21.15.23:0/2719565710 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb48003280 con 0x7fdb5c0a9810 2020-11-20T11:33:23.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.982+0000 7fdb59ffb700 1 -- 172.21.15.23:0/2719565710 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2505981486 0 0) 0x7fdb48003280 con 0x7fdb5c0a9810 2020-11-20T11:33:23.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.982+0000 7fdb59ffb700 1 -- 172.21.15.23:0/2719565710 >> v2:172.21.15.23:3300/0 conn(0x7fdb5c061380 msgr2=0x7fdb5c13afe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:33:23.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.982+0000 7fdb59ffb700 1 --2- 172.21.15.23:0/2719565710 >> v2:172.21.15.23:3300/0 conn(0x7fdb5c061380 0x7fdb5c13afe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:23.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.982+0000 7fdb59ffb700 1 -- 172.21.15.23:0/2719565710 >> v1:172.21.15.23:6789/0 conn(0x7fdb5c0aacf0 legacy=0x7fdb5c145490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:23.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.982+0000 7fdb59ffb700 1 -- 172.21.15.23:0/2719565710 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5c13ab40 con 0x7fdb5c0a9810 2020-11-20T11:33:23.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.982+0000 7fdb59ffb700 1 -- 172.21.15.23:0/2719565710 <== mon.1 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7fdb50001520 con 0x7fdb5c0a9810 2020-11-20T11:33:23.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.982+0000 7fdb59ffb700 1 -- 172.21.15.23:0/2719565710 >> v1:172.21.15.155:6789/0 conn(0x7fdb5c0a9810 legacy=0x7fdb5c137a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:23.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.982+0000 7fdb59ffb700 1 --2- 172.21.15.23:0/2719565710 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdb48003770 0x7fdb48003bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:23.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.982+0000 7fdb59ffb700 1 --2- 172.21.15.23:0/2719565710 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb48004520 0x7fdb480049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:23.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.982+0000 7fdb59ffb700 1 --2- 172.21.15.23:0/2719565710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdb48004e90 0x7fdb4800b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:23.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.982+0000 7fdb59ffb700 1 -- 172.21.15.23:0/2719565710 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb480042c0 con 0x7fdb48004520 2020-11-20T11:33:23.992 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.982+0000 7fdb59ffb700 1 -- 172.21.15.23:0/2719565710 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4800bc60 con 0x7fdb48004e90 2020-11-20T11:33:23.992 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.982+0000 7fdb59ffb700 1 -- 172.21.15.23:0/2719565710 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4800bda0 con 0x7fdb48003770 2020-11-20T11:33:23.992 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.982+0000 7fdb5affd700 1 --2- 172.21.15.23:0/2719565710 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdb48003770 0x7fdb48003bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:23.992 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.982+0000 7fdb5a7fc700 1 --2- 172.21.15.23:0/2719565710 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb48004520 0x7fdb480049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:23.992 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.982+0000 7fdb5b7fe700 1 --2- 172.21.15.23:0/2719565710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdb48004e90 0x7fdb4800b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:23.993 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.982+0000 7fdb5a7fc700 1 -- 172.21.15.23:0/2719565710 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdb48003770 msgr2=0x7fdb48003bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:33:23.993 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.982+0000 7fdb5a7fc700 1 --2- 172.21.15.23:0/2719565710 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdb48003770 0x7fdb48003bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:23.993 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.982+0000 7fdb5a7fc700 1 -- 172.21.15.23:0/2719565710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdb48004e90 msgr2=0x7fdb4800b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:33:23.993 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.982+0000 7fdb5a7fc700 1 --2- 172.21.15.23:0/2719565710 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdb48004e90 0x7fdb4800b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:23.994 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.982+0000 7fdb5b7fe700 1 -- 172.21.15.23:0/2719565710 reap_dead start 2020-11-20T11:33:23.994 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.982+0000 7fdb5a7fc700 1 -- 172.21.15.23:0/2719565710 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb48003280 con 0x7fdb48004520 2020-11-20T11:33:23.994 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.982+0000 7fdb5affd700 1 --2- 172.21.15.23:0/2719565710 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdb48003770 0x7fdb48003bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:33:23.994 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.982+0000 7fdb5a7fc700 1 --2- 172.21.15.23:0/2719565710 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb48004520 0x7fdb480049a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fdb5000c440 tx=0x7fdb5c0a9f40).ready entity=mon.0 client_cookie=8167ad307599f027 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:23.994 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.983+0000 7fdb59ffb700 1 -- 172.21.15.23:0/2719565710 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb50007030 con 0x7fdb48004520 2020-11-20T11:33:23.995 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.983+0000 7fdb59ffb700 1 -- 172.21.15.23:0/2719565710 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb50007190 con 0x7fdb48004520 2020-11-20T11:33:23.995 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.983+0000 7fdb59ffb700 1 -- 172.21.15.23:0/2719565710 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb50006530 con 0x7fdb48004520 2020-11-20T11:33:23.995 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.983+0000 7fdb615f3700 1 -- 172.21.15.23:0/2719565710 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb48004520 msgr2=0x7fdb480049a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:23.995 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.983+0000 7fdb615f3700 1 --2- 172.21.15.23:0/2719565710 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb48004520 0x7fdb480049a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fdb5000c440 tx=0x7fdb5c0a9f40).stop 2020-11-20T11:33:23.996 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.983+0000 7fdb615f3700 1 -- 172.21.15.23:0/2719565710 shutdown_connections 2020-11-20T11:33:23.996 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.983+0000 7fdb615f3700 1 --2- 172.21.15.23:0/2719565710 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb48004520 0x7fdb480049a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:23.996 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.983+0000 7fdb615f3700 1 -- 172.21.15.23:0/2719565710 >> 172.21.15.23:0/2719565710 conn(0x7fdb5c0ae120 msgr2=0x7fdb5c0aa0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:33:23.996 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.983+0000 7fdb615f3700 1 -- 172.21.15.23:0/2719565710 shutdown_connections 2020-11-20T11:33:23.996 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.984+0000 7fdb615f3700 1 -- 172.21.15.23:0/2719565710 wait complete. 2020-11-20T11:33:23.999 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.985+0000 7fdb615f3700 1 Processor -- start 2020-11-20T11:33:24.000 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.985+0000 7fdb615f3700 1 -- start start 2020-11-20T11:33:24.000 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.985+0000 7fdb615f3700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdb48004520 0x7fdb5c066c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.000 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.985+0000 7fdb615f3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb5c140ed0 0x7fdb5c0a5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.000 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.985+0000 7fdb615f3700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdb5c0a6250 0x7fdb5c14d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.001 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.985+0000 7fdb615f3700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c141350 con 0x7fdb5c140ed0 2020-11-20T11:33:24.001 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.985+0000 7fdb615f3700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c0a7990 con 0x7fdb48004520 2020-11-20T11:33:24.001 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.985+0000 7fdb615f3700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c0a6fb0 con 0x7fdb5c0a6250 2020-11-20T11:33:24.001 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.985+0000 7fdb5a7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb5c140ed0 0x7fdb5c0a5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.002 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.985+0000 7fdb5a7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb5c140ed0 0x7fdb5c0a5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41778/0 (socket says 172.21.15.23:41778) 2020-11-20T11:33:24.002 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.985+0000 7fdb5a7fc700 1 -- 172.21.15.23:0/1177237423 learned_addr learned my addr 172.21.15.23:0/1177237423 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:33:24.002 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.986+0000 7fdb5a7fc700 1 -- 172.21.15.23:0/1177237423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb5c140ed0 msgr2=0x7fdb5c0a5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2020-11-20T11:33:24.002 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.986+0000 7fdb5a7fc700 1 -- 172.21.15.23:0/1177237423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb5c140ed0 msgr2=0x7fdb5c0a5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:33:24.002 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.986+0000 7fdb5a7fc700 1 --2- 172.21.15.23:0/1177237423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb5c140ed0 0x7fdb5c0a5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:33:24.003 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.986+0000 7fdb5a7fc700 1 --2- 172.21.15.23:0/1177237423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb5c140ed0 0x7fdb5c0a5d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:33:24.003 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.986+0000 7fdb5affd700 1 --2- 172.21.15.23:0/1177237423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdb48004520 0x7fdb5c066c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.003 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.986+0000 7fdb5b7fe700 1 --2- 172.21.15.23:0/1177237423 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdb5c0a6250 0x7fdb5c14d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.003 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.986+0000 7fdb5affd700 1 -- 172.21.15.23:0/1177237423 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdb5c0a6250 msgr2=0x7fdb5c14d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:24.004 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.986+0000 7fdb5affd700 1 --2- 172.21.15.23:0/1177237423 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdb5c0a6250 0x7fdb5c14d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:24.004 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.986+0000 7fdb5affd700 1 -- 172.21.15.23:0/1177237423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb5c140ed0 msgr2=0x7fdb5c0a5d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:33:24.004 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.986+0000 7fdb5affd700 1 --2- 172.21.15.23:0/1177237423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb5c140ed0 0x7fdb5c0a5d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:24.004 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.986+0000 7fdb5affd700 1 -- 172.21.15.23:0/1177237423 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5c0a7250 con 0x7fdb48004520 2020-11-20T11:33:24.004 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.986+0000 7fdb5affd700 1 --2- 172.21.15.23:0/1177237423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdb48004520 0x7fdb5c066c40 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fdb5c045440 tx=0x7fdb4c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:24.005 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.986+0000 7fdb42ffd700 1 -- 172.21.15.23:0/1177237423 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c00ea60 con 0x7fdb48004520 2020-11-20T11:33:24.005 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.986+0000 7fdb42ffd700 1 -- 172.21.15.23:0/1177237423 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb4c00ebc0 con 0x7fdb48004520 2020-11-20T11:33:24.005 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.986+0000 7fdb615f3700 1 -- 172.21.15.23:0/1177237423 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb5c0a74f0 con 0x7fdb48004520 2020-11-20T11:33:24.005 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.986+0000 7fdb42ffd700 1 -- 172.21.15.23:0/1177237423 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c014470 con 0x7fdb48004520 2020-11-20T11:33:24.005 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.987+0000 7fdb615f3700 1 -- 172.21.15.23:0/1177237423 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb5c068600 con 0x7fdb48004520 2020-11-20T11:33:24.006 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.988+0000 7fdb40ff9700 1 -- 172.21.15.23:0/1177237423 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb2c004fd0 con 0x7fdb48004520 2020-11-20T11:33:24.006 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.988+0000 7fdb42ffd700 1 -- 172.21.15.23:0/1177237423 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fdb4c014820 con 0x7fdb48004520 2020-11-20T11:33:24.006 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.988+0000 7fdb42ffd700 1 --2- 172.21.15.23:0/1177237423 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fdb38056d60 0x7fdb38059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.006 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.988+0000 7fdb42ffd700 1 -- 172.21.15.23:0/1177237423 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fdb4c0705b0 con 0x7fdb48004520 2020-11-20T11:33:24.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.989+0000 7fdb5a7fc700 1 --2- 172.21.15.23:0/1177237423 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fdb38056d60 0x7fdb38059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.989+0000 7fdb5a7fc700 1 --2- 172.21.15.23:0/1177237423 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fdb38056d60 0x7fdb38059220 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fdb50007870 tx=0x7fdb50014000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:24.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:23.993+0000 7fdb42ffd700 1 -- 172.21.15.23:0/1177237423 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fdb4c0458a0 con 0x7fdb48004520 2020-11-20T11:33:24.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.201+0000 7fdb40ff9700 1 -- 172.21.15.23:0/1177237423 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fdb2c0016a0 con 0x7fdb48004520 2020-11-20T11:33:24.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.202+0000 7fdb42ffd700 1 -- 172.21.15.23:0/1177237423 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1477 (secure 0 0 0) 0x7fdb4c018020 con 0x7fdb48004520 2020-11-20T11:33:24.205 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:33:24.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.203+0000 7fdb40ff9700 1 -- 172.21.15.23:0/1177237423 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fdb38056d60 msgr2=0x7fdb38059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:24.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.203+0000 7fdb40ff9700 1 --2- 172.21.15.23:0/1177237423 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fdb38056d60 0x7fdb38059220 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fdb50007870 tx=0x7fdb50014000).stop 2020-11-20T11:33:24.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.203+0000 7fdb40ff9700 1 -- 172.21.15.23:0/1177237423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdb48004520 msgr2=0x7fdb5c066c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:24.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.203+0000 7fdb40ff9700 1 --2- 172.21.15.23:0/1177237423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdb48004520 0x7fdb5c066c40 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fdb5c045440 tx=0x7fdb4c008040).stop 2020-11-20T11:33:24.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.203+0000 7fdb40ff9700 1 -- 172.21.15.23:0/1177237423 shutdown_connections 2020-11-20T11:33:24.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.203+0000 7fdb40ff9700 1 --2- 172.21.15.23:0/1177237423 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fdb38056d60 0x7fdb38059220 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:24.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.203+0000 7fdb40ff9700 1 --2- 172.21.15.23:0/1177237423 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdb5c0a6250 0x7fdb5c14d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:24.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.203+0000 7fdb40ff9700 1 --2- 172.21.15.23:0/1177237423 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdb48004520 0x7fdb5c066c40 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:24.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.203+0000 7fdb40ff9700 1 --2- 172.21.15.23:0/1177237423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdb5c140ed0 0x7fdb5c0a5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:24.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.203+0000 7fdb40ff9700 1 -- 172.21.15.23:0/1177237423 >> 172.21.15.23:0/1177237423 conn(0x7fdb5c0ae120 msgr2=0x7fdb5c0a3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:33:24.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.203+0000 7fdb40ff9700 1 -- 172.21.15.23:0/1177237423 shutdown_connections 2020-11-20T11:33:24.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.204+0000 7fdb40ff9700 1 -- 172.21.15.23:0/1177237423 wait complete. 2020-11-20T11:33:24.218 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:33:24.218 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:33:24.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.333+0000 7f4ac8f0b700 1 Processor -- start 2020-11-20T11:33:24.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.333+0000 7f4ac8f0b700 1 -- start start 2020-11-20T11:33:24.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.334+0000 7f4ac8f0b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f4ac4061380 0x7f4ac40a20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.334+0000 7f4ac8f0b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ac40a25c0 0x7f4ac413c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.334+0000 7f4ac8f0b700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac40a1ae0 con 0x7f4ac413c9d0 2020-11-20T11:33:24.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.334+0000 7f4ac8f0b700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ac40a9270 con 0x7f4ac4061380 2020-11-20T11:33:24.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.334+0000 7f4ac8f0b700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac40a9510 con 0x7f4ac40a25c0 2020-11-20T11:33:24.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.334+0000 7f4ac37fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f4ac4061380 0x7f4ac40a20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.334+0000 7f4ac3fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f4ac413c9d0 0x7f4ac40a7fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:56042/0 (socket says 172.21.15.23:56042) 2020-11-20T11:33:24.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.334+0000 7f4ac37fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f4ac4061380 0x7f4ac40a20d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41788/0 (socket says 172.21.15.23:41788) 2020-11-20T11:33:24.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.334+0000 7f4ac3fff700 1 -- 172.21.15.23:0/1466134775 learned_addr learned my addr 172.21.15.23:0/1466134775 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:33:24.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.334+0000 7f4ac2ffd700 1 --2- 172.21.15.23:0/1466134775 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ac40a25c0 0x7f4ac413c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.334+0000 7f4ac37fe700 1 -- 172.21.15.23:0/1466134775 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ac40a25c0 msgr2=0x7f4ac413c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:33:24.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.334+0000 7f4ac37fe700 1 --2- 172.21.15.23:0/1466134775 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ac40a25c0 0x7f4ac413c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:24.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.334+0000 7f4ac37fe700 1 -- 172.21.15.23:0/1466134775 >> v1:172.21.15.23:6789/0 conn(0x7f4ac413c9d0 legacy=0x7f4ac40a7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:24.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.334+0000 7f4ac37fe700 1 -- 172.21.15.23:0/1466134775 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac4061760 con 0x7f4ac4061380 2020-11-20T11:33:24.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.334+0000 7f4ac37fe700 1 --2- 172.21.15.23:0/1466134775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ac4061380 0x7f4ac40a20d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f4ab4006fd0 tx=0x7f4ab4006e30).ready entity=mon.0 client_cookie=112e905c4b556384 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:24.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.335+0000 7f4ac27fc700 1 -- 172.21.15.23:0/1466134775 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab40097e0 con 0x7f4ac4061380 2020-11-20T11:33:24.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.335+0000 7f4ac27fc700 1 -- 172.21.15.23:0/1466134775 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ab4000e90 con 0x7f4ac4061380 2020-11-20T11:33:24.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.335+0000 7f4ac27fc700 1 -- 172.21.15.23:0/1466134775 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab4009b50 con 0x7f4ac4061380 2020-11-20T11:33:24.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.335+0000 7f4ac8f0b700 1 -- 172.21.15.23:0/1466134775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ac4061380 msgr2=0x7f4ac40a20d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:24.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.335+0000 7f4ac8f0b700 1 --2- 172.21.15.23:0/1466134775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ac4061380 0x7f4ac40a20d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f4ab4006fd0 tx=0x7f4ab4006e30).stop 2020-11-20T11:33:24.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.335+0000 7f4ac8f0b700 1 -- 172.21.15.23:0/1466134775 shutdown_connections 2020-11-20T11:33:24.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.335+0000 7f4ac8f0b700 1 --2- 172.21.15.23:0/1466134775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ac4061380 0x7f4ac40a20d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:24.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.335+0000 7f4ac8f0b700 1 --2- 172.21.15.23:0/1466134775 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ac40a25c0 0x7f4ac413c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:24.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.335+0000 7f4ac8f0b700 1 -- 172.21.15.23:0/1466134775 >> 172.21.15.23:0/1466134775 conn(0x7f4ac40ae0a0 msgr2=0x7f4ac40aa0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:33:24.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.335+0000 7f4ac8f0b700 1 -- 172.21.15.23:0/1466134775 shutdown_connections 2020-11-20T11:33:24.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.335+0000 7f4ac8f0b700 1 -- 172.21.15.23:0/1466134775 wait complete. 2020-11-20T11:33:24.346 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.337+0000 7f4ac8f0b700 1 Processor -- start 2020-11-20T11:33:24.346 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.337+0000 7f4ac8f0b700 1 -- start start 2020-11-20T11:33:24.346 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.338+0000 7f4ac8f0b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ac4061380 0x7f4ac409ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.346 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.338+0000 7f4ac8f0b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ac40a25c0 0x7f4ac414bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.338+0000 7f4ac8f0b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ac413c9d0 0x7f4ac4115270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.338+0000 7f4ac8f0b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac409fd30 con 0x7f4ac40a25c0 2020-11-20T11:33:24.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.338+0000 7f4ac8f0b700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac414c1a0 con 0x7f4ac413c9d0 2020-11-20T11:33:24.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.338+0000 7f4ac8f0b700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac414c440 con 0x7f4ac4061380 2020-11-20T11:33:24.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.338+0000 7f4ac2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ac40a25c0 0x7f4ac414bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.338+0000 7f4ac2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ac40a25c0 0x7f4ac414bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41796/0 (socket says 172.21.15.23:41796) 2020-11-20T11:33:24.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.338+0000 7f4ac2ffd700 1 -- 172.21.15.23:0/3197956038 learned_addr learned my addr 172.21.15.23:0/3197956038 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:33:24.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.338+0000 7f4ac37fe700 1 --2- 172.21.15.23:0/3197956038 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ac4061380 0x7f4ac409ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.338+0000 7f4ac3fff700 1 --2- 172.21.15.23:0/3197956038 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ac413c9d0 0x7f4ac4115270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.338+0000 7f4ac2ffd700 1 -- 172.21.15.23:0/3197956038 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ac4061380 msgr2=0x7f4ac409ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:24.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.338+0000 7f4ac2ffd700 1 --2- 172.21.15.23:0/3197956038 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ac4061380 0x7f4ac409ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:24.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.338+0000 7f4ac2ffd700 1 -- 172.21.15.23:0/3197956038 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ac413c9d0 msgr2=0x7f4ac4115270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:24.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.338+0000 7f4ac2ffd700 1 --2- 172.21.15.23:0/3197956038 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4ac413c9d0 0x7f4ac4115270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:24.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.338+0000 7f4ac2ffd700 1 -- 172.21.15.23:0/3197956038 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac414c6e0 con 0x7f4ac40a25c0 2020-11-20T11:33:24.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.338+0000 7f4ac37fe700 1 --2- 172.21.15.23:0/3197956038 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4ac4061380 0x7f4ac409ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:33:24.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.339+0000 7f4ac2ffd700 1 --2- 172.21.15.23:0/3197956038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ac40a25c0 0x7f4ac414bb00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f4ab800db30 tx=0x7f4ab8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:24.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.339+0000 7f4aa37fe700 1 -- 172.21.15.23:0/3197956038 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab8009760 con 0x7f4ac40a25c0 2020-11-20T11:33:24.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.339+0000 7f4aa37fe700 1 -- 172.21.15.23:0/3197956038 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ab80098c0 con 0x7f4ac40a25c0 2020-11-20T11:33:24.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.339+0000 7f4ac8f0b700 1 -- 172.21.15.23:0/3197956038 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ac4115760 con 0x7f4ac40a25c0 2020-11-20T11:33:24.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.339+0000 7f4aa37fe700 1 -- 172.21.15.23:0/3197956038 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab8009c40 con 0x7f4ac40a25c0 2020-11-20T11:33:24.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.339+0000 7f4ac8f0b700 1 -- 172.21.15.23:0/3197956038 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ac4068030 con 0x7f4ac40a25c0 2020-11-20T11:33:24.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.340+0000 7f4ac8f0b700 1 --2- 172.21.15.23:0/3197956038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ac4061980 0x7f4ac4148530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.340+0000 7f4ac8f0b700 1 -- 172.21.15.23:0/3197956038 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac414bff0 con 0x7f4ac4061980 2020-11-20T11:33:24.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.340+0000 7f4ac37fe700 1 --2- 172.21.15.23:0/3197956038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ac4061980 0x7f4ac4148530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.340+0000 7f4aa37fe700 1 -- 172.21.15.23:0/3197956038 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4ab8013480 con 0x7f4ac40a25c0 2020-11-20T11:33:24.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.340+0000 7f4ac37fe700 1 -- 172.21.15.23:0/3197956038 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4ac4069c70 con 0x7f4ac4061980 2020-11-20T11:33:24.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.341+0000 7f4ac37fe700 1 --2- 172.21.15.23:0/3197956038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ac4061980 0x7f4ac4148530 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f4ab4006fd0 tx=0x7f4ab4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:24.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.341+0000 7f4aa37fe700 1 --2- 172.21.15.23:0/3197956038 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4a9c056d60 0x7f4a9c059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.341+0000 7f4aa37fe700 1 -- 172.21.15.23:0/3197956038 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f4ab8070430 con 0x7f4ac40a25c0 2020-11-20T11:33:24.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.341+0000 7f4aa37fe700 1 -- 172.21.15.23:0/3197956038 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab4024070 con 0x7f4ac4061980 2020-11-20T11:33:24.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.341+0000 7f4ac3fff700 1 --2- 172.21.15.23:0/3197956038 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4a9c056d60 0x7f4a9c059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.341+0000 7f4aa37fe700 1 -- 172.21.15.23:0/3197956038 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f4ab4011c80 con 0x7f4ac4061980 2020-11-20T11:33:24.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.341+0000 7f4aa37fe700 1 -- 172.21.15.23:0/3197956038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ac4061980 msgr2=0x7f4ac4148530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:24.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.341+0000 7f4aa37fe700 1 --2- 172.21.15.23:0/3197956038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ac4061980 0x7f4ac4148530 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f4ab4006fd0 tx=0x7f4ab4015000).stop 2020-11-20T11:33:24.354 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.341+0000 7f4aa37fe700 1 -- 172.21.15.23:0/3197956038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ac4061980 msgr2=0x7f4ac4148530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:33:24.354 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.341+0000 7f4aa37fe700 1 --2- 172.21.15.23:0/3197956038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ac4061980 0x7f4ac4148530 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:24.354 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.341+0000 7f4ac3fff700 1 --2- 172.21.15.23:0/3197956038 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4a9c056d60 0x7f4a9c059220 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f4abc007fd0 tx=0x7f4abc009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:24.356 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.348+0000 7f4ac8f0b700 1 --2- 172.21.15.23:0/3197956038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ac40682b0 0x7f4ac4148a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.356 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.348+0000 7f4ac8f0b700 1 -- 172.21.15.23:0/3197956038 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac4148f10 con 0x7f4ac40682b0 2020-11-20T11:33:24.356 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.348+0000 7f4ac37fe700 1 --2- 172.21.15.23:0/3197956038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ac40682b0 0x7f4ac4148a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.356 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.348+0000 7f4ac37fe700 1 -- 172.21.15.23:0/3197956038 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4ac40a6a70 con 0x7f4ac40682b0 2020-11-20T11:33:24.357 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.348+0000 7f4ac37fe700 1 --2- 172.21.15.23:0/3197956038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ac40682b0 0x7f4ac4148a20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f4ac414c130 tx=0x7f4ab4020000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:24.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.349+0000 7f4aa37fe700 1 -- 172.21.15.23:0/3197956038 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab40286a0 con 0x7f4ac40682b0 2020-11-20T11:33:24.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.349+0000 7f4aa37fe700 1 -- 172.21.15.23:0/3197956038 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4275 (secure 0 0 0) 0x7f4ab4011a70 con 0x7f4ac40682b0 2020-11-20T11:33:24.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.349+0000 7f4aa37fe700 1 -- 172.21.15.23:0/3197956038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ac40682b0 msgr2=0x7f4ac4148a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:24.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.349+0000 7f4aa37fe700 1 --2- 172.21.15.23:0/3197956038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ac40682b0 0x7f4ac4148a20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f4ac414c130 tx=0x7f4ab4020000).stop 2020-11-20T11:33:24.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.349+0000 7f4aa37fe700 1 -- 172.21.15.23:0/3197956038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ac40682b0 msgr2=0x7f4ac4148a20 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:33:24.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.349+0000 7f4aa37fe700 1 --2- 172.21.15.23:0/3197956038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ac40682b0 0x7f4ac4148a20 secure :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0x7f4ac414c130 tx=0x7f4ab4020000).stop 2020-11-20T11:33:24.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.349+0000 7f4ac8f0b700 1 -- 172.21.15.23:0/3197956038 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4a9c056d60 msgr2=0x7f4a9c059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:24.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.349+0000 7f4ac8f0b700 1 --2- 172.21.15.23:0/3197956038 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4a9c056d60 0x7f4a9c059220 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f4abc007fd0 tx=0x7f4abc009040).stop 2020-11-20T11:33:24.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.349+0000 7f4ac8f0b700 1 -- 172.21.15.23:0/3197956038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ac40a25c0 msgr2=0x7f4ac414bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:24.360 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.349+0000 7f4ac8f0b700 1 --2- 172.21.15.23:0/3197956038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4ac40a25c0 0x7f4ac414bb00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f4ab800db30 tx=0x7f4ab8008040).stop 2020-11-20T11:33:24.360 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.349+0000 7f4ac3fff700 1 -- 172.21.15.23:0/3197956038 reap_dead start 2020-11-20T11:33:24.360 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.349+0000 7f4ac3fff700 1 -- 172.21.15.23:0/3197956038 reap_dead start 2020-11-20T11:33:24.360 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.349+0000 7f4ac8f0b700 1 -- 172.21.15.23:0/3197956038 shutdown_connections 2020-11-20T11:33:24.360 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.349+0000 7f4ac8f0b700 1 -- 172.21.15.23:0/3197956038 >> 172.21.15.23:0/3197956038 conn(0x7f4ac40ae0a0 msgr2=0x7f4ac40a3340 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:33:24.361 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.349+0000 7f4ac8f0b700 1 -- 172.21.15.23:0/3197956038 shutdown_connections 2020-11-20T11:33:24.361 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.349+0000 7f4ac8f0b700 1 -- 172.21.15.23:0/3197956038 wait complete. 2020-11-20T11:33:24.361 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:33:24.362 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:33:24.362 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:33:24.362 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:33:24.362 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 10, 2020-11-20T11:33:24.362 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:33:24.363 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:33:24.364 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:33:24.364 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:33:24.364 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:24.364 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 0, 2020-11-20T11:33:24.364 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:33:24.365 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:33:24.365 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:33:24.365 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:33:24.365 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:33:24.365 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:33:24.366 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:33:24.366 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:33:24.366 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:33:24.366 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:33:24.366 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:33:24.367 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:24.367 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:33:24.367 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:33:24.367 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:33:24.367 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:33:24.368 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:33:24.368 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:33:24.368 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:33:24.368 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:33:24.368 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:33:24.369 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:33:24.369 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:24.369 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.369 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:33:24.369 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:33:24.370 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:33:24.370 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:33:24.370 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:33:24.370 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:33:24.370 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:33:24.371 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:33:24.371 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:33:24.371 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:33:24.371 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:33:24.371 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:33:24.372 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:33:24.372 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:33:24.372 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:33:24.372 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:33:24.372 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:33:24.373 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:33:24.373 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:33:24.373 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:33:24.373 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:33:24.374 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:33:24.374 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:33:24.374 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:24.374 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:33:24.374 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.375 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:33:24.375 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.375 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:33:24.375 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:33:24.375 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:33:24.375 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:33:24.376 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.376 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:33:24.376 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:33:24.376 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:24.376 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.377 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.377 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:33:24.377 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:33:24.377 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:24.377 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:24.378 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:24.378 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.378 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:33:24.378 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:33:24.378 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:33:24.379 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:33:24.379 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:33:24.379 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.379 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.379 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:33:24.380 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:33:24.380 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:33:24.380 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:33:24.380 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.380 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:33:24.381 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:33:24.381 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:24.381 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.381 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.381 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:33:24.382 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:33:24.382 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:24.382 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:24.382 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:24.382 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.382 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:33:24.383 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:33:24.383 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:33:24.383 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:33:24.383 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:33:24.383 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.384 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.384 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:33:24.384 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:33:24.384 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:33:24.384 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:33:24.385 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.385 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:33:24.385 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:33:24.385 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:24.385 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.386 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.386 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:33:24.386 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:33:24.386 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:24.386 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:24.387 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:24.387 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.387 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:33:24.387 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:33:24.388 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:33:24.388 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:33:24.388 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:33:24.388 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:24.388 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:24.389 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.389 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:33:24.389 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:33:24.389 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.389 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:24.390 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:24.390 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:33:24.390 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:24.390 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:24.390 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:33:24.391 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.391 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:24.391 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:24.391 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:33:24.391 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:24.392 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:24.392 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.392 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:33:24.392 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:33:24.393 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:33:24.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.507+0000 7f2216da9700 1 Processor -- start 2020-11-20T11:33:24.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.508+0000 7f2216da9700 1 -- start start 2020-11-20T11:33:24.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.508+0000 7f2216da9700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f22100676a0 0x7f22100a89c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.508+0000 7f2216da9700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f2210065e50 0x7f22100a8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.508+0000 7f2216da9700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2210138360 con 0x7f22100aad30 2020-11-20T11:33:24.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.508+0000 7f2216da9700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2210067d10 con 0x7f2210065e50 2020-11-20T11:33:24.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.508+0000 7f2216da9700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2210066700 con 0x7f22100676a0 2020-11-20T11:33:24.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.508+0000 7f22165a8700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f22100aad30 0x7f22100ab190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:56060/0 (socket says 172.21.15.23:56060) 2020-11-20T11:33:24.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.508+0000 7f22165a8700 1 -- 172.21.15.23:0/2006859232 learned_addr learned my addr 172.21.15.23:0/2006859232 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:33:24.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.508+0000 7f2215da7700 1 --2- 172.21.15.23:0/2006859232 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f22100676a0 0x7f22100a89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.508+0000 7f22155a6700 1 --2- 172.21.15.23:0/2006859232 >> v2:172.21.15.155:3300/0 conn(0x7f2210065e50 0x7f22100a8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.509+0000 7f2214da5700 1 -- 172.21.15.23:0/2006859232 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1689486105 0 0) 0x7f2210138360 con 0x7f22100aad30 2020-11-20T11:33:24.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.509+0000 7f2214da5700 1 -- 172.21.15.23:0/2006859232 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21f8003280 con 0x7f22100aad30 2020-11-20T11:33:24.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.509+0000 7f2215da7700 1 -- 172.21.15.23:0/2006859232 >> v2:172.21.15.155:3300/0 conn(0x7f2210065e50 msgr2=0x7f22100a8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:33:24.515 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.509+0000 7f2215da7700 1 --2- 172.21.15.23:0/2006859232 >> v2:172.21.15.155:3300/0 conn(0x7f2210065e50 0x7f22100a8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:24.515 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.509+0000 7f2215da7700 1 -- 172.21.15.23:0/2006859232 >> v1:172.21.15.23:6789/0 conn(0x7f22100aad30 legacy=0x7f22100ab190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:24.515 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.509+0000 7f2215da7700 1 -- 172.21.15.23:0/2006859232 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2210138600 con 0x7f22100676a0 2020-11-20T11:33:24.515 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.509+0000 7f22155a6700 1 --2- 172.21.15.23:0/2006859232 >> v2:172.21.15.155:3300/0 conn(0x7f2210065e50 0x7f22100a8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:33:24.515 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.509+0000 7f2215da7700 1 --2- 172.21.15.23:0/2006859232 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f22100676a0 0x7f22100a89c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f220c006fd0 tx=0x7f220c006e30).ready entity=mon.2 client_cookie=840d56bb7f3f57b2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:24.516 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.509+0000 7f2214da5700 1 -- 172.21.15.23:0/2006859232 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f220c009630 con 0x7f22100676a0 2020-11-20T11:33:24.516 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.509+0000 7f2214da5700 1 -- 172.21.15.23:0/2006859232 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f220c000e90 con 0x7f22100676a0 2020-11-20T11:33:24.516 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.509+0000 7f2214da5700 1 -- 172.21.15.23:0/2006859232 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f220c0099a0 con 0x7f22100676a0 2020-11-20T11:33:24.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.510+0000 7f2216da9700 1 -- 172.21.15.23:0/2006859232 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f22100676a0 msgr2=0x7f22100a89c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:24.520 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.510+0000 7f2216da9700 1 --2- 172.21.15.23:0/2006859232 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f22100676a0 0x7f22100a89c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f220c006fd0 tx=0x7f220c006e30).stop 2020-11-20T11:33:24.520 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.510+0000 7f2216da9700 1 -- 172.21.15.23:0/2006859232 shutdown_connections 2020-11-20T11:33:24.520 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.510+0000 7f2216da9700 1 --2- 172.21.15.23:0/2006859232 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f22100676a0 0x7f22100a89c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:24.520 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.510+0000 7f2216da9700 1 --2- 172.21.15.23:0/2006859232 >> v2:172.21.15.155:3300/0 conn(0x7f2210065e50 0x7f22100a8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:24.520 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.510+0000 7f2216da9700 1 -- 172.21.15.23:0/2006859232 >> 172.21.15.23:0/2006859232 conn(0x7f22100612c0 msgr2=0x7f22100616d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:33:24.521 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.510+0000 7f2216da9700 1 -- 172.21.15.23:0/2006859232 shutdown_connections 2020-11-20T11:33:24.521 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.510+0000 7f2216da9700 1 -- 172.21.15.23:0/2006859232 wait complete. 2020-11-20T11:33:24.521 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.511+0000 7f2216da9700 1 Processor -- start 2020-11-20T11:33:24.521 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.512+0000 7f2216da9700 1 -- start start 2020-11-20T11:33:24.521 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.512+0000 7f2216da9700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2210065e50 0x7f22100a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.512+0000 7f2216da9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22100676a0 0x7f221014a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.513+0000 7f2216da9700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f22100aad30 0x7f2210133690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.513+0000 7f2216da9700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2210138f40 con 0x7f22100676a0 2020-11-20T11:33:24.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.513+0000 7f2216da9700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f221014b050 con 0x7f2210065e50 2020-11-20T11:33:24.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.513+0000 7f22155a6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22100676a0 0x7f221014a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.513+0000 7f2216da9700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f221014ab20 con 0x7f22100aad30 2020-11-20T11:33:24.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.513+0000 7f22155a6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22100676a0 0x7f221014a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41814/0 (socket says 172.21.15.23:41814) 2020-11-20T11:33:24.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.513+0000 7f22155a6700 1 -- 172.21.15.23:0/2232059533 learned_addr learned my addr 172.21.15.23:0/2232059533 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:33:24.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.513+0000 7f2215da7700 1 --2- 172.21.15.23:0/2232059533 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2210065e50 0x7f22100a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.513+0000 7f22155a6700 1 -- 172.21.15.23:0/2232059533 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f22100aad30 msgr2=0x7f2210133690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:24.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.513+0000 7f22155a6700 1 --2- 172.21.15.23:0/2232059533 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f22100aad30 0x7f2210133690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:33:24.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.513+0000 7f22155a6700 1 -- 172.21.15.23:0/2232059533 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2210065e50 msgr2=0x7f22100a56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:24.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.513+0000 7f22155a6700 1 --2- 172.21.15.23:0/2232059533 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2210065e50 0x7f22100a56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:24.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.513+0000 7f22155a6700 1 -- 172.21.15.23:0/2232059533 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f221014adb0 con 0x7f22100676a0 2020-11-20T11:33:24.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.513+0000 7f2215da7700 1 --2- 172.21.15.23:0/2232059533 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2210065e50 0x7f22100a56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:33:24.525 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.513+0000 7f22155a6700 1 --2- 172.21.15.23:0/2232059533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22100676a0 0x7f221014a630 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f220000ed20 tx=0x7f2200006e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:24.525 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.513+0000 7f21fdffb700 1 -- 172.21.15.23:0/2232059533 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f220000cbe0 con 0x7f22100676a0 2020-11-20T11:33:24.525 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.513+0000 7f21fdffb700 1 -- 172.21.15.23:0/2232059533 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f220000cd40 con 0x7f22100676a0 2020-11-20T11:33:24.525 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.513+0000 7f2216da9700 1 -- 172.21.15.23:0/2232059533 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22101331f0 con 0x7f22100676a0 2020-11-20T11:33:24.525 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.513+0000 7f21fdffb700 1 -- 172.21.15.23:0/2232059533 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2200012600 con 0x7f22100676a0 2020-11-20T11:33:24.526 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.514+0000 7f2216da9700 1 -- 172.21.15.23:0/2232059533 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2210133b80 con 0x7f22100676a0 2020-11-20T11:33:24.526 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.514+0000 7f21ef7fe700 1 --2- 172.21.15.23:0/2232059533 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21e4002a50 0x7f21e4004f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.526 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.514+0000 7f21ef7fe700 1 -- 172.21.15.23:0/2232059533 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e4007bb0 con 0x7f21e4002a50 2020-11-20T11:33:24.526 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.515+0000 7f2215da7700 1 --2- 172.21.15.23:0/2232059533 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21e4002a50 0x7f21e4004f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.526 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.515+0000 7f21fdffb700 1 -- 172.21.15.23:0/2232059533 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2200012970 con 0x7f22100676a0 2020-11-20T11:33:24.527 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.515+0000 7f2215da7700 1 -- 172.21.15.23:0/2232059533 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21e4007d70 con 0x7f21e4002a50 2020-11-20T11:33:24.527 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.515+0000 7f2215da7700 1 --2- 172.21.15.23:0/2232059533 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21e4002a50 0x7f21e4004f10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f220c006fd0 tx=0x7f220c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:24.527 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.515+0000 7f21fdffb700 1 --2- 172.21.15.23:0/2232059533 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f21e8056d60 0x7f21e8059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.527 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.515+0000 7f21fdffb700 1 -- 172.21.15.23:0/2232059533 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f2200070690 con 0x7f22100676a0 2020-11-20T11:33:24.527 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.516+0000 7f21fdffb700 1 -- 172.21.15.23:0/2232059533 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f220c0117f0 con 0x7f21e4002a50 2020-11-20T11:33:24.528 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.516+0000 7f22165a8700 1 --2- 172.21.15.23:0/2232059533 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f21e8056d60 0x7f21e8059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.528 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.516+0000 7f21fdffb700 1 -- 172.21.15.23:0/2232059533 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f220c011950 con 0x7f21e4002a50 2020-11-20T11:33:24.528 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.516+0000 7f21fdffb700 1 -- 172.21.15.23:0/2232059533 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21e4002a50 msgr2=0x7f21e4004f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:24.528 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.516+0000 7f21fdffb700 1 --2- 172.21.15.23:0/2232059533 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21e4002a50 0x7f21e4004f10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f220c006fd0 tx=0x7f220c015000).stop 2020-11-20T11:33:24.528 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.516+0000 7f21fdffb700 1 -- 172.21.15.23:0/2232059533 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21e4002a50 msgr2=0x7f21e4004f10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:33:24.529 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.516+0000 7f21fdffb700 1 --2- 172.21.15.23:0/2232059533 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21e4002a50 0x7f21e4004f10 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:24.529 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.516+0000 7f22165a8700 1 --2- 172.21.15.23:0/2232059533 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f21e8056d60 0x7f21e8059220 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f2208005960 tx=0x7f2208001420).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:24.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.529+0000 7f21ef7fe700 1 --2- 172.21.15.23:0/2232059533 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21e4006bf0 0x7f21e4007080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.529+0000 7f21ef7fe700 1 -- 172.21.15.23:0/2232059533 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e4007690 con 0x7f21e4006bf0 2020-11-20T11:33:24.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.529+0000 7f2215da7700 1 --2- 172.21.15.23:0/2232059533 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21e4006bf0 0x7f21e4007080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.533 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.529+0000 7f2215da7700 1 -- 172.21.15.23:0/2232059533 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f21e4007850 con 0x7f21e4006bf0 2020-11-20T11:33:24.533 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.530+0000 7f2215da7700 1 --2- 172.21.15.23:0/2232059533 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21e4006bf0 0x7f21e4007080 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f220c006e70 tx=0x7f220c01f000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:24.533 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.530+0000 7f21fdffb700 1 -- 172.21.15.23:0/2232059533 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f220c01e640 con 0x7f21e4006bf0 2020-11-20T11:33:24.533 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.530+0000 7f21fdffb700 1 -- 172.21.15.23:0/2232059533 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4783 (secure 0 0 0) 0x7f220c0115e0 con 0x7f21e4006bf0 2020-11-20T11:33:24.534 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.530+0000 7f21fdffb700 1 -- 172.21.15.23:0/2232059533 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21e4006bf0 msgr2=0x7f21e4007080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:24.535 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.530+0000 7f21fdffb700 1 --2- 172.21.15.23:0/2232059533 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21e4006bf0 0x7f21e4007080 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f220c006e70 tx=0x7f220c01f000).stop 2020-11-20T11:33:24.535 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.530+0000 7f21fdffb700 1 -- 172.21.15.23:0/2232059533 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21e4006bf0 msgr2=0x7f21e4007080 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:33:24.535 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.530+0000 7f21fdffb700 1 --2- 172.21.15.23:0/2232059533 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21e4006bf0 0x7f21e4007080 secure :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0x7f220c006e70 tx=0x7f220c01f000).stop 2020-11-20T11:33:24.535 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.530+0000 7f21ef7fe700 1 -- 172.21.15.23:0/2232059533 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f21e8056d60 msgr2=0x7f21e8059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:24.536 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.530+0000 7f21ef7fe700 1 --2- 172.21.15.23:0/2232059533 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f21e8056d60 0x7f21e8059220 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f2208005960 tx=0x7f2208001420).stop 2020-11-20T11:33:24.536 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.530+0000 7f21ef7fe700 1 -- 172.21.15.23:0/2232059533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22100676a0 msgr2=0x7f221014a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:24.536 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.530+0000 7f21ef7fe700 1 --2- 172.21.15.23:0/2232059533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22100676a0 0x7f221014a630 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f220000ed20 tx=0x7f2200006e30).stop 2020-11-20T11:33:24.536 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.530+0000 7f22165a8700 1 -- 172.21.15.23:0/2232059533 reap_dead start 2020-11-20T11:33:24.536 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.531+0000 7f21ef7fe700 1 -- 172.21.15.23:0/2232059533 shutdown_connections 2020-11-20T11:33:24.537 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.531+0000 7f21ef7fe700 1 --2- 172.21.15.23:0/2232059533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22100676a0 0x7f221014a630 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:24.537 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.531+0000 7f21ef7fe700 1 -- 172.21.15.23:0/2232059533 >> 172.21.15.23:0/2232059533 conn(0x7f22100612c0 msgr2=0x7f22100616a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:33:24.537 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.531+0000 7f21ef7fe700 1 -- 172.21.15.23:0/2232059533 shutdown_connections 2020-11-20T11:33:24.537 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.531+0000 7f21ef7fe700 1 -- 172.21.15.23:0/2232059533 wait complete. 2020-11-20T11:33:24.537 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:33:24.538 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:33:24.538 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:33:24.538 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:33:24.538 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 10, 2020-11-20T11:33:24.539 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:33:24.539 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:33:24.539 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:33:24.539 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:33:24.539 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:24.540 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 0, 2020-11-20T11:33:24.540 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:33:24.540 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:33:24.540 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:33:24.540 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:33:24.541 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:33:24.541 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:33:24.541 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:33:24.541 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:33:24.541 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:33:24.542 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:33:24.542 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:33:24.542 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:24.542 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:33:24.542 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:33:24.543 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:33:24.543 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:33:24.543 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:33:24.543 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:33:24.543 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:33:24.543 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:33:24.544 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:33:24.544 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:33:24.544 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:24.544 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.544 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:33:24.545 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:33:24.545 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:33:24.545 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:33:24.545 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:33:24.545 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:33:24.546 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:33:24.546 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:33:24.546 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:33:24.546 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:33:24.546 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:33:24.547 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:33:24.547 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:33:24.547 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:33:24.547 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:33:24.547 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:33:24.548 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:33:24.548 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:33:24.548 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:33:24.548 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:33:24.548 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:33:24.549 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:33:24.549 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:33:24.549 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:24.549 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:33:24.549 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.550 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:33:24.550 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.550 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:33:24.550 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:33:24.550 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:33:24.551 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:33:24.551 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.551 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:33:24.551 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:33:24.551 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:24.551 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.552 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.552 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:33:24.552 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:33:24.552 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:24.552 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:24.553 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:24.553 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.553 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:33:24.553 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:33:24.553 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:33:24.554 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:33:24.554 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:33:24.554 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.554 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.554 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:33:24.555 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:33:24.555 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:33:24.555 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:33:24.555 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.555 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:33:24.556 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:33:24.556 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:24.556 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.556 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.556 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:33:24.556 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:33:24.557 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:24.557 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:24.557 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:24.557 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.557 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:33:24.558 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:33:24.558 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:33:24.558 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:33:24.558 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:33:24.558 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.559 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.559 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:33:24.559 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:33:24.559 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:33:24.559 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:33:24.560 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.560 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:33:24.560 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:33:24.560 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:24.560 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.561 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.561 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:33:24.561 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:33:24.561 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:24.561 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:24.562 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:24.562 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.562 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:33:24.562 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:33:24.562 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:33:24.562 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:33:24.563 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:33:24.563 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:24.563 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:24.563 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.563 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:33:24.564 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:33:24.564 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.564 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:24.564 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:24.564 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:33:24.565 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:24.565 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:24.565 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:33:24.565 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.565 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:24.566 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:24.566 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:33:24.566 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:24.566 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:24.566 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:33:24.567 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.567 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:24.567 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:24.567 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 4 2020-11-20T11:33:24.567 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:24.568 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:24.568 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:33:24.568 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.568 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:24.568 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:24.568 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:33:24.569 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:24.569 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:24.569 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:33:24.569 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.569 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:24.570 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:24.570 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:33:24.570 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:24.570 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:24.570 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.571 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:33:24.571 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:33:24.572 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:33:24.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.687+0000 7f05748e1700 1 Processor -- start 2020-11-20T11:33:24.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.687+0000 7f05748e1700 1 -- start start 2020-11-20T11:33:24.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.687+0000 7f05748e1700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05700a9040 0x7f05700aaf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.687+0000 7f05748e1700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f05700ace70 0x7f057013de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.687+0000 7f05748e1700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05700ad860 con 0x7f05700ad2f0 2020-11-20T11:33:24.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.687+0000 7f05748e1700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05700aa8a0 con 0x7f05700ace70 2020-11-20T11:33:24.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.687+0000 7f05748e1700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05700aaa90 con 0x7f05700a9040 2020-11-20T11:33:24.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.687+0000 7f056effd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f05700ace70 0x7f057013de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.687+0000 7f056effd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f05700ace70 0x7f057013de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41824/0 (socket says 172.21.15.23:41824) 2020-11-20T11:33:24.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.687+0000 7f056effd700 1 -- 172.21.15.23:0/3996908760 learned_addr learned my addr 172.21.15.23:0/3996908760 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:33:24.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.688+0000 7f056f7fe700 1 --2- 172.21.15.23:0/3996908760 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05700a9040 0x7f05700aaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.688+0000 7f056effd700 1 -- 172.21.15.23:0/3996908760 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05700a9040 msgr2=0x7f05700aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:33:24.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.688+0000 7f056effd700 1 --2- 172.21.15.23:0/3996908760 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05700a9040 0x7f05700aaf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:24.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.688+0000 7f056effd700 1 -- 172.21.15.23:0/3996908760 >> v1:172.21.15.155:6789/0 conn(0x7f05700ad2f0 legacy=0x7f057013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:24.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.688+0000 7f056effd700 1 -- 172.21.15.23:0/3996908760 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05700a9420 con 0x7f05700ace70 2020-11-20T11:33:24.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.688+0000 7f056effd700 1 --2- 172.21.15.23:0/3996908760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05700ace70 0x7f057013de00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f0564009a10 tx=0x7f0564006ef0).ready entity=mon.0 client_cookie=3430f2aa2253dbf9 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:24.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.688+0000 7f056e7fc700 1 -- 172.21.15.23:0/3996908760 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056400b860 con 0x7f05700ace70 2020-11-20T11:33:24.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.688+0000 7f056e7fc700 1 -- 172.21.15.23:0/3996908760 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0564000e90 con 0x7f05700ace70 2020-11-20T11:33:24.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.688+0000 7f056e7fc700 1 -- 172.21.15.23:0/3996908760 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056400bbd0 con 0x7f05700ace70 2020-11-20T11:33:24.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.689+0000 7f05748e1700 1 -- 172.21.15.23:0/3996908760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05700ace70 msgr2=0x7f057013de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:24.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.689+0000 7f05748e1700 1 --2- 172.21.15.23:0/3996908760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05700ace70 0x7f057013de00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f0564009a10 tx=0x7f0564006ef0).stop 2020-11-20T11:33:24.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.689+0000 7f05748e1700 1 -- 172.21.15.23:0/3996908760 shutdown_connections 2020-11-20T11:33:24.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.689+0000 7f05748e1700 1 --2- 172.21.15.23:0/3996908760 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05700a9040 0x7f05700aaf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:24.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.689+0000 7f05748e1700 1 --2- 172.21.15.23:0/3996908760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05700ace70 0x7f057013de00 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:24.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.689+0000 7f05748e1700 1 -- 172.21.15.23:0/3996908760 >> 172.21.15.23:0/3996908760 conn(0x7f05700683b0 msgr2=0x7f05700b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:33:24.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.689+0000 7f05748e1700 1 -- 172.21.15.23:0/3996908760 shutdown_connections 2020-11-20T11:33:24.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.689+0000 7f05748e1700 1 -- 172.21.15.23:0/3996908760 wait complete. 2020-11-20T11:33:24.700 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.691+0000 7f05748e1700 1 Processor -- start 2020-11-20T11:33:24.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.691+0000 7f05748e1700 1 -- start start 2020-11-20T11:33:24.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.692+0000 7f05748e1700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05700a9040 0x7f057009f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.692+0000 7f05748e1700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05700ace70 0x7f05700a0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.692+0000 7f05748e1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05700ad2f0 0x7f05700a7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.692+0000 7f05748e1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057009f5d0 con 0x7f05700ad2f0 2020-11-20T11:33:24.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.692+0000 7f05748e1700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057009e860 con 0x7f05700a9040 2020-11-20T11:33:24.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.692+0000 7f05748e1700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f057009dad0 con 0x7f05700ace70 2020-11-20T11:33:24.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.692+0000 7f056ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05700ad2f0 0x7f05700a7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.692+0000 7f056ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05700ad2f0 0x7f05700a7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41834/0 (socket says 172.21.15.23:41834) 2020-11-20T11:33:24.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.692+0000 7f056ffff700 1 -- 172.21.15.23:0/776378582 learned_addr learned my addr 172.21.15.23:0/776378582 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:33:24.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.692+0000 7f056effd700 1 --2- 172.21.15.23:0/776378582 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05700ace70 0x7f05700a0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.692+0000 7f056f7fe700 1 --2- 172.21.15.23:0/776378582 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05700a9040 0x7f057009f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.692+0000 7f056ffff700 1 -- 172.21.15.23:0/776378582 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05700ace70 msgr2=0x7f05700a0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:24.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.692+0000 7f056ffff700 1 --2- 172.21.15.23:0/776378582 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05700ace70 0x7f05700a0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:24.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.692+0000 7f056ffff700 1 -- 172.21.15.23:0/776378582 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05700a9040 msgr2=0x7f057009f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:24.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.692+0000 7f056ffff700 1 --2- 172.21.15.23:0/776378582 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05700a9040 0x7f057009f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:24.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.692+0000 7f056ffff700 1 -- 172.21.15.23:0/776378582 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f057009dd70 con 0x7f05700ad2f0 2020-11-20T11:33:24.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.692+0000 7f056effd700 1 --2- 172.21.15.23:0/776378582 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05700ace70 0x7f05700a0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:33:24.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.692+0000 7f056f7fe700 1 --2- 172.21.15.23:0/776378582 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05700a9040 0x7f057009f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:33:24.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.692+0000 7f056ffff700 1 --2- 172.21.15.23:0/776378582 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05700ad2f0 0x7f05700a7430 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f056800ca90 tx=0x7f05680013a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:24.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.692+0000 7f054f7fe700 1 -- 172.21.15.23:0/776378582 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056800a550 con 0x7f05700ad2f0 2020-11-20T11:33:24.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.693+0000 7f054f7fe700 1 -- 172.21.15.23:0/776378582 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f056800a6b0 con 0x7f05700ad2f0 2020-11-20T11:33:24.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.693+0000 7f05748e1700 1 -- 172.21.15.23:0/776378582 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f057009e010 con 0x7f05700ad2f0 2020-11-20T11:33:24.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.693+0000 7f054f7fe700 1 -- 172.21.15.23:0/776378582 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056800aa30 con 0x7f05700ad2f0 2020-11-20T11:33:24.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.693+0000 7f05748e1700 1 -- 172.21.15.23:0/776378582 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05700a0e60 con 0x7f05700ad2f0 2020-11-20T11:33:24.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.694+0000 7f05748e1700 1 --2- 172.21.15.23:0/776378582 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05700a7920 0x7f0570149c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.707 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.694+0000 7f05748e1700 1 -- 172.21.15.23:0/776378582 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f057005cf20 con 0x7f05700a7920 2020-11-20T11:33:24.707 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.694+0000 7f056f7fe700 1 --2- 172.21.15.23:0/776378582 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05700a7920 0x7f0570149c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.707 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.694+0000 7f054f7fe700 1 -- 172.21.15.23:0/776378582 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0568019030 con 0x7f05700ad2f0 2020-11-20T11:33:24.707 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.694+0000 7f056f7fe700 1 -- 172.21.15.23:0/776378582 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f057009d950 con 0x7f05700a7920 2020-11-20T11:33:24.707 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.694+0000 7f054f7fe700 1 --2- 172.21.15.23:0/776378582 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0548056d90 0x7f0548059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.708 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.694+0000 7f056f7fe700 1 --2- 172.21.15.23:0/776378582 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05700a7920 0x7f0570149c80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f0560006fd0 tx=0x7f0560008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:24.708 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.695+0000 7f054f7fe700 1 -- 172.21.15.23:0/776378582 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f056806f620 con 0x7f05700ad2f0 2020-11-20T11:33:24.708 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.695+0000 7f054f7fe700 1 -- 172.21.15.23:0/776378582 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056000ea90 con 0x7f05700a7920 2020-11-20T11:33:24.708 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.695+0000 7f056effd700 1 --2- 172.21.15.23:0/776378582 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0548056d90 0x7f0548059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.708 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.695+0000 7f054f7fe700 1 -- 172.21.15.23:0/776378582 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f056000ebf0 con 0x7f05700a7920 2020-11-20T11:33:24.709 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.695+0000 7f054f7fe700 1 -- 172.21.15.23:0/776378582 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05700a7920 msgr2=0x7f0570149c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:24.709 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.695+0000 7f054f7fe700 1 --2- 172.21.15.23:0/776378582 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05700a7920 0x7f0570149c80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f0560006fd0 tx=0x7f0560008040).stop 2020-11-20T11:33:24.709 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.695+0000 7f054f7fe700 1 -- 172.21.15.23:0/776378582 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05700a7920 msgr2=0x7f0570149c80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:33:24.709 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.695+0000 7f054f7fe700 1 --2- 172.21.15.23:0/776378582 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05700a7920 0x7f0570149c80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:24.709 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.695+0000 7f056effd700 1 --2- 172.21.15.23:0/776378582 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0548056d90 0x7f0548059250 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f0564009a10 tx=0x7f0564016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:24.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.702+0000 7f05748e1700 1 --2- 172.21.15.23:0/776378582 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f057014a170 0x7f057014d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.702+0000 7f05748e1700 1 -- 172.21.15.23:0/776378582 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f057009e3d0 con 0x7f057014a170 2020-11-20T11:33:24.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.702+0000 7f056f7fe700 1 --2- 172.21.15.23:0/776378582 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f057014a170 0x7f057014d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.703+0000 7f056f7fe700 1 -- 172.21.15.23:0/776378582 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f057009e1a0 con 0x7f057014a170 2020-11-20T11:33:24.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.703+0000 7f056f7fe700 1 --2- 172.21.15.23:0/776378582 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f057014a170 0x7f057014d250 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f0560018000 tx=0x7f056001b000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:24.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.703+0000 7f054f7fe700 1 -- 172.21.15.23:0/776378582 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056001d8f0 con 0x7f057014a170 2020-11-20T11:33:24.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.703+0000 7f054f7fe700 1 -- 172.21.15.23:0/776378582 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4783 (secure 0 0 0) 0x7f056000e880 con 0x7f057014a170 2020-11-20T11:33:24.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.703+0000 7f054f7fe700 1 -- 172.21.15.23:0/776378582 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f057014a170 msgr2=0x7f057014d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:24.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.703+0000 7f054f7fe700 1 --2- 172.21.15.23:0/776378582 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f057014a170 0x7f057014d250 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f0560018000 tx=0x7f056001b000).stop 2020-11-20T11:33:24.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.703+0000 7f054f7fe700 1 -- 172.21.15.23:0/776378582 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f057014a170 msgr2=0x7f057014d250 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:33:24.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.703+0000 7f054f7fe700 1 --2- 172.21.15.23:0/776378582 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f057014a170 0x7f057014d250 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0x7f0560018000 tx=0x7f056001b000).stop 2020-11-20T11:33:24.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.704+0000 7f05748e1700 1 -- 172.21.15.23:0/776378582 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0548056d90 msgr2=0x7f0548059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:24.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.704+0000 7f05748e1700 1 --2- 172.21.15.23:0/776378582 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0548056d90 0x7f0548059250 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f0564009a10 tx=0x7f0564016000).stop 2020-11-20T11:33:24.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.704+0000 7f05748e1700 1 -- 172.21.15.23:0/776378582 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05700ad2f0 msgr2=0x7f05700a7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:24.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.704+0000 7f05748e1700 1 --2- 172.21.15.23:0/776378582 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05700ad2f0 0x7f05700a7430 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f056800ca90 tx=0x7f05680013a0).stop 2020-11-20T11:33:24.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.704+0000 7f056ffff700 1 -- 172.21.15.23:0/776378582 reap_dead start 2020-11-20T11:33:24.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.704+0000 7f056ffff700 1 -- 172.21.15.23:0/776378582 reap_dead start 2020-11-20T11:33:24.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.704+0000 7f05748e1700 1 -- 172.21.15.23:0/776378582 shutdown_connections 2020-11-20T11:33:24.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.704+0000 7f05748e1700 1 -- 172.21.15.23:0/776378582 >> 172.21.15.23:0/776378582 conn(0x7f05700683b0 msgr2=0x7f0570137580 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:33:24.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.704+0000 7f05748e1700 1 -- 172.21.15.23:0/776378582 shutdown_connections 2020-11-20T11:33:24.716 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.704+0000 7f05748e1700 1 -- 172.21.15.23:0/776378582 wait complete. 2020-11-20T11:33:24.716 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:33:24.716 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:33:24.716 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:33:24.716 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:33:24.717 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 10, 2020-11-20T11:33:24.717 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:33:24.717 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:33:24.717 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:33:24.718 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:33:24.718 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:24.718 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 0, 2020-11-20T11:33:24.718 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:33:24.718 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:33:24.718 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:33:24.719 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:33:24.719 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:33:24.719 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:33:24.719 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:33:24.719 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:33:24.720 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:33:24.720 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:33:24.720 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:33:24.720 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:24.720 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:33:24.721 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:33:24.721 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:33:24.721 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:33:24.721 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:33:24.721 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:33:24.722 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:33:24.722 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:33:24.722 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:33:24.722 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:33:24.722 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:24.723 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.723 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:33:24.723 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:33:24.723 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:33:24.723 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:33:24.724 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:33:24.724 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:33:24.724 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:33:24.724 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:33:24.724 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:33:24.725 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:33:24.725 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:33:24.725 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:33:24.725 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:33:24.725 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:33:24.726 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:33:24.726 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:33:24.726 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:33:24.726 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:33:24.726 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:33:24.727 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:33:24.727 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:33:24.727 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:33:24.727 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:33:24.727 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:24.727 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:33:24.728 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.728 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:33:24.728 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.728 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:33:24.728 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:33:24.729 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:33:24.729 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:33:24.729 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.729 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:33:24.729 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:33:24.730 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:24.730 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.730 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.730 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:33:24.730 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:33:24.731 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:24.731 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:24.731 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:24.731 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.731 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:33:24.732 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:33:24.732 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:33:24.732 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:33:24.732 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:33:24.732 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.733 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.733 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:33:24.733 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:33:24.733 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:33:24.733 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:33:24.734 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.734 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:33:24.734 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:33:24.734 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:24.734 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.734 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.735 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:33:24.735 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:33:24.735 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:24.735 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:24.735 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:24.736 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.736 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:33:24.736 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:33:24.736 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:33:24.736 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:33:24.737 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:33:24.737 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.737 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.737 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:33:24.737 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:33:24.738 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:33:24.738 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:33:24.738 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.738 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:33:24.738 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:33:24.739 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:24.739 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.739 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.739 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:33:24.740 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:33:24.740 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:24.740 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:24.740 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:24.740 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.741 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:33:24.741 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:33:24.741 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:33:24.741 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:33:24.741 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:33:24.742 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:24.742 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:24.742 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.742 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:33:24.742 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:33:24.743 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.743 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:24.743 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:24.743 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:33:24.743 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:24.743 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:24.744 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:33:24.744 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.744 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:24.744 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:24.744 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:33:24.745 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:24.745 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:24.745 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:33:24.745 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.745 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:24.746 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:24.746 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 4 2020-11-20T11:33:24.746 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:24.746 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:24.746 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:33:24.747 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.747 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:24.747 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:24.747 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:33:24.747 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:24.748 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:24.748 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:33:24.748 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:24.748 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:24.748 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:24.749 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:33:24.749 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:24.749 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:24.749 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:24.749 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:33:24.750 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:33:24.751 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2020-11-20T11:33:24.751 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2020-11-20T11:33:24.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.867+0000 7f9db9456700 1 Processor -- start 2020-11-20T11:33:24.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.867+0000 7f9db9456700 1 -- start start 2020-11-20T11:33:24.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.868+0000 7f9db9456700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9db40adb40 0x7f9db40a8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.868+0000 7f9db9456700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f9db41387f0 0x7f9db4138ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.868+0000 7f9db9456700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9db40a8fa0 con 0x7f9db40ad760 2020-11-20T11:33:24.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.868+0000 7f9db9456700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9db40a8370 con 0x7f9db41387f0 2020-11-20T11:33:24.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.868+0000 7f9db9456700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9db40a8610 con 0x7f9db40adb40 2020-11-20T11:33:24.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.868+0000 7f9db37fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f9db41387f0 0x7f9db4138ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.868+0000 7f9db37fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f9db41387f0 0x7f9db4138ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41846/0 (socket says 172.21.15.23:41846) 2020-11-20T11:33:24.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.868+0000 7f9db37fe700 1 -- 172.21.15.23:0/4026886157 learned_addr learned my addr 172.21.15.23:0/4026886157 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:33:24.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.868+0000 7f9db27fc700 1 --2- 172.21.15.23:0/4026886157 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9db40adb40 0x7f9db40a8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.869+0000 7f9db37fe700 1 -- 172.21.15.23:0/4026886157 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9db40adb40 msgr2=0x7f9db40a8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:33:24.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.869+0000 7f9db37fe700 1 --2- 172.21.15.23:0/4026886157 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9db40adb40 0x7f9db40a8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:24.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.869+0000 7f9db37fe700 1 -- 172.21.15.23:0/4026886157 >> v1:172.21.15.155:6789/0 conn(0x7f9db40ad760 legacy=0x7f9db40658a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:24.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.869+0000 7f9db1ffb700 1 -- 172.21.15.23:0/4026886157 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 498752808 0 0) 0x7f9da4001110 con 0x7f9db40ad760 2020-11-20T11:33:24.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.869+0000 7f9db37fe700 1 -- 172.21.15.23:0/4026886157 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db40a80b0 con 0x7f9db41387f0 2020-11-20T11:33:24.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.869+0000 7f9db37fe700 1 --2- 172.21.15.23:0/4026886157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9db41387f0 0x7f9db4138ca0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f9dac009d90 tx=0x7f9dac0092d0).ready entity=mon.0 client_cookie=519d1c557ceb4b87 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:24.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.869+0000 7f9db1ffb700 1 -- 172.21.15.23:0/4026886157 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dac00bbf0 con 0x7f9db41387f0 2020-11-20T11:33:24.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.869+0000 7f9db1ffb700 1 -- 172.21.15.23:0/4026886157 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dac00bd50 con 0x7f9db41387f0 2020-11-20T11:33:24.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.869+0000 7f9db1ffb700 1 -- 172.21.15.23:0/4026886157 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dac011610 con 0x7f9db41387f0 2020-11-20T11:33:24.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.869+0000 7f9db9456700 1 -- 172.21.15.23:0/4026886157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9db41387f0 msgr2=0x7f9db4138ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:24.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.869+0000 7f9db9456700 1 --2- 172.21.15.23:0/4026886157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9db41387f0 0x7f9db4138ca0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f9dac009d90 tx=0x7f9dac0092d0).stop 2020-11-20T11:33:24.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.869+0000 7f9db9456700 1 -- 172.21.15.23:0/4026886157 shutdown_connections 2020-11-20T11:33:24.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.869+0000 7f9db9456700 1 --2- 172.21.15.23:0/4026886157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9db41387f0 0x7f9db4138ca0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:24.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.869+0000 7f9db9456700 1 --2- 172.21.15.23:0/4026886157 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9db40adb40 0x7f9db40a8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:24.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.869+0000 7f9db9456700 1 -- 172.21.15.23:0/4026886157 >> 172.21.15.23:0/4026886157 conn(0x7f9db4061d30 msgr2=0x7f9db4062140 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:33:24.882 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.870+0000 7f9db9456700 1 -- 172.21.15.23:0/4026886157 shutdown_connections 2020-11-20T11:33:24.882 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.870+0000 7f9db9456700 1 -- 172.21.15.23:0/4026886157 wait complete. 2020-11-20T11:33:24.886 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.871+0000 7f9db9456700 1 Processor -- start 2020-11-20T11:33:24.886 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.871+0000 7f9db9456700 1 -- start start 2020-11-20T11:33:24.886 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.871+0000 7f9db9456700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9db40ad760 0x7f9db4139d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.887 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.871+0000 7f9db9456700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9db40adb40 0x7f9db413c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.887 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.871+0000 7f9db9456700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9db41387f0 0x7f9db413a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.887 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.871+0000 7f9db9456700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db413f0e0 con 0x7f9db41387f0 2020-11-20T11:33:24.888 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.871+0000 7f9db9456700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db413f340 con 0x7f9db40adb40 2020-11-20T11:33:24.888 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.871+0000 7f9db9456700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9db413f5e0 con 0x7f9db40ad760 2020-11-20T11:33:24.888 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.872+0000 7f9db37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9db41387f0 0x7f9db413a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.889 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.872+0000 7f9db2ffd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9db40ad760 0x7f9db4139d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.889 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.872+0000 7f9db37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9db41387f0 0x7f9db413a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41852/0 (socket says 172.21.15.23:41852) 2020-11-20T11:33:24.889 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.872+0000 7f9db37fe700 1 -- 172.21.15.23:0/1222073309 learned_addr learned my addr 172.21.15.23:0/1222073309 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:33:24.889 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.872+0000 7f9db27fc700 1 --2- 172.21.15.23:0/1222073309 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9db40adb40 0x7f9db413c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.890 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.872+0000 7f9db37fe700 1 -- 172.21.15.23:0/1222073309 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9db40ad760 msgr2=0x7f9db4139d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:24.890 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.872+0000 7f9db37fe700 1 --2- 172.21.15.23:0/1222073309 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9db40ad760 0x7f9db4139d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:24.890 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.872+0000 7f9db37fe700 1 -- 172.21.15.23:0/1222073309 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9db40adb40 msgr2=0x7f9db413c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:24.891 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.872+0000 7f9db37fe700 1 --2- 172.21.15.23:0/1222073309 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9db40adb40 0x7f9db413c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:24.891 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.872+0000 7f9db37fe700 1 -- 172.21.15.23:0/1222073309 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db40a4320 con 0x7f9db41387f0 2020-11-20T11:33:24.891 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.872+0000 7f9db27fc700 1 --2- 172.21.15.23:0/1222073309 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9db40adb40 0x7f9db413c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:33:24.891 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.872+0000 7f9db37fe700 1 --2- 172.21.15.23:0/1222073309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9db41387f0 0x7f9db413a410 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f9dac009310 tx=0x7f9dac016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:24.892 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.872+0000 7f9d9affd700 1 -- 172.21.15.23:0/1222073309 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dac00b780 con 0x7f9db41387f0 2020-11-20T11:33:24.892 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.872+0000 7f9d9affd700 1 -- 172.21.15.23:0/1222073309 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dac011e30 con 0x7f9db41387f0 2020-11-20T11:33:24.892 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.872+0000 7f9d9affd700 1 -- 172.21.15.23:0/1222073309 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dac010cf0 con 0x7f9db41387f0 2020-11-20T11:33:24.892 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.872+0000 7f9db9456700 1 -- 172.21.15.23:0/1222073309 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9db413c5a0 con 0x7f9db41387f0 2020-11-20T11:33:24.892 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.872+0000 7f9db9456700 1 -- 172.21.15.23:0/1222073309 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9db409f8f0 con 0x7f9db41387f0 2020-11-20T11:33:24.893 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.873+0000 7f9db9456700 1 -- 172.21.15.23:0/1222073309 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9db4057dc0 con 0x7f9db41387f0 2020-11-20T11:33:24.893 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.874+0000 7f9d9affd700 1 -- 172.21.15.23:0/1222073309 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9dac017070 con 0x7f9db41387f0 2020-11-20T11:33:24.893 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.874+0000 7f9d9affd700 1 --2- 172.21.15.23:0/1222073309 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9d90056d60 0x7f9d90059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:24.894 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.875+0000 7f9d9affd700 1 -- 172.21.15.23:0/1222073309 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f9dac04a290 con 0x7f9db41387f0 2020-11-20T11:33:24.894 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.875+0000 7f9db2ffd700 1 --2- 172.21.15.23:0/1222073309 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9d90056d60 0x7f9d90059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:24.894 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.875+0000 7f9db2ffd700 1 --2- 172.21.15.23:0/1222073309 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9d90056d60 0x7f9d90059220 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f9db4145480 tx=0x7f9da4009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:24.895 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:24.879+0000 7f9d9affd700 1 -- 172.21.15.23:0/1222073309 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9dac076060 con 0x7f9db41387f0 2020-11-20T11:33:25.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:25.081+0000 7f9db9456700 1 -- 172.21.15.23:0/1222073309 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f9db4065fb0 con 0x7f9db41387f0 2020-11-20T11:33:25.086 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:25.082+0000 7f9d9affd700 1 -- 172.21.15.23:0/1222073309 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f9dac015020 con 0x7f9db41387f0 2020-11-20T11:33:25.087 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:25.084+0000 7f9db9456700 1 -- 172.21.15.23:0/1222073309 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9d90056d60 msgr2=0x7f9d90059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:25.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:25.084+0000 7f9db9456700 1 --2- 172.21.15.23:0/1222073309 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9d90056d60 0x7f9d90059220 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f9db4145480 tx=0x7f9da4009040).stop 2020-11-20T11:33:25.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:25.084+0000 7f9db9456700 1 -- 172.21.15.23:0/1222073309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9db41387f0 msgr2=0x7f9db413a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:25.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:25.084+0000 7f9db9456700 1 --2- 172.21.15.23:0/1222073309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9db41387f0 0x7f9db413a410 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f9dac009310 tx=0x7f9dac016000).stop 2020-11-20T11:33:25.089 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:25.084+0000 7f9db9456700 1 -- 172.21.15.23:0/1222073309 shutdown_connections 2020-11-20T11:33:25.089 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:25.084+0000 7f9db9456700 1 --2- 172.21.15.23:0/1222073309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9db41387f0 0x7f9db413a410 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:25.089 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:25.084+0000 7f9db9456700 1 --2- 172.21.15.23:0/1222073309 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9d90056d60 0x7f9d90059220 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:25.089 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:25.084+0000 7f9db9456700 1 --2- 172.21.15.23:0/1222073309 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9db40ad760 0x7f9db4139d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:25.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:25.084+0000 7f9db9456700 1 --2- 172.21.15.23:0/1222073309 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9db40adb40 0x7f9db413c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:25.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:25.084+0000 7f9db9456700 1 -- 172.21.15.23:0/1222073309 >> 172.21.15.23:0/1222073309 conn(0x7f9db4061d30 msgr2=0x7f9db4068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:33:25.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:25.084+0000 7f9db9456700 1 -- 172.21.15.23:0/1222073309 shutdown_connections 2020-11-20T11:33:25.091 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:25.084+0000 7f9db9456700 1 -- 172.21.15.23:0/1222073309 wait complete. 2020-11-20T11:33:25.099 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2020-11-20T11:33:29.404 INFO:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:33:29.407 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:33:35.100 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:33:35.101 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:33:35.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.214+0000 7f647ee89700 1 Processor -- start 2020-11-20T11:33:35.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.214+0000 7f647ee89700 1 -- start start 2020-11-20T11:33:35.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.215+0000 7f647ee89700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64780aaab0 0x7f647813ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.215+0000 7f647ee89700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f64780aae90 0x7f64781381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.215+0000 7f647ee89700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f647813aeb0 0x7f64780a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.215+0000 7f647ee89700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f647813e600 con 0x7f647813aeb0 2020-11-20T11:33:35.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.215+0000 7f647ee89700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f647813e8a0 con 0x7f64780aae90 2020-11-20T11:33:35.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.215+0000 7f647ee89700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64780ab6d0 con 0x7f64780aaab0 2020-11-20T11:33:35.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.215+0000 7f647d426700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f647813aeb0 0x7f64780a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.215+0000 7f647d426700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f647813aeb0 0x7f64780a8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41860/0 (socket says 172.21.15.23:41860) 2020-11-20T11:33:35.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.215+0000 7f647d426700 1 -- 172.21.15.23:0/4084213322 learned_addr learned my addr 172.21.15.23:0/4084213322 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:33:35.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.215+0000 7f647cc25700 1 --2- 172.21.15.23:0/4084213322 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64780aaab0 0x7f647813ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.215+0000 7f6477fff700 1 --2- 172.21.15.23:0/4084213322 >> v2:172.21.15.155:3300/0 conn(0x7f64780aae90 0x7f64781381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.216+0000 7f647d426700 1 -- 172.21.15.23:0/4084213322 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64780aaab0 msgr2=0x7f647813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:33:35.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.216+0000 7f647d426700 1 --2- 172.21.15.23:0/4084213322 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64780aaab0 0x7f647813ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:35.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.216+0000 7f647d426700 1 -- 172.21.15.23:0/4084213322 >> v2:172.21.15.155:3300/0 conn(0x7f64780aae90 msgr2=0x7f64781381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:33:35.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.216+0000 7f647d426700 1 --2- 172.21.15.23:0/4084213322 >> v2:172.21.15.155:3300/0 conn(0x7f64780aae90 0x7f64781381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:35.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.216+0000 7f647d426700 1 -- 172.21.15.23:0/4084213322 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f647813f230 con 0x7f647813aeb0 2020-11-20T11:33:35.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.216+0000 7f647d426700 1 --2- 172.21.15.23:0/4084213322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f647813aeb0 0x7f64780a8c60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f6478062e20 tx=0x7f64700092d0).ready entity=mon.0 client_cookie=fafb690a3370901e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:35.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.216+0000 7f64777fe700 1 -- 172.21.15.23:0/4084213322 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f647000bb40 con 0x7f647813aeb0 2020-11-20T11:33:35.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.216+0000 7f64777fe700 1 -- 172.21.15.23:0/4084213322 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f647000bca0 con 0x7f647813aeb0 2020-11-20T11:33:35.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.216+0000 7f64777fe700 1 -- 172.21.15.23:0/4084213322 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6470011610 con 0x7f647813aeb0 2020-11-20T11:33:35.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.216+0000 7f647ee89700 1 -- 172.21.15.23:0/4084213322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f647813aeb0 msgr2=0x7f64780a8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:35.225 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.216+0000 7f647ee89700 1 --2- 172.21.15.23:0/4084213322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f647813aeb0 0x7f64780a8c60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f6478062e20 tx=0x7f64700092d0).stop 2020-11-20T11:33:35.225 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.216+0000 7f647ee89700 1 -- 172.21.15.23:0/4084213322 shutdown_connections 2020-11-20T11:33:35.225 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.216+0000 7f647ee89700 1 --2- 172.21.15.23:0/4084213322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f647813aeb0 0x7f64780a8c60 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:35.225 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.216+0000 7f647ee89700 1 --2- 172.21.15.23:0/4084213322 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f64780aaab0 0x7f647813ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:35.225 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.216+0000 7f647ee89700 1 --2- 172.21.15.23:0/4084213322 >> v2:172.21.15.155:3300/0 conn(0x7f64780aae90 0x7f64781381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:35.226 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.217+0000 7f647ee89700 1 -- 172.21.15.23:0/4084213322 >> 172.21.15.23:0/4084213322 conn(0x7f64780678e0 msgr2=0x7f6478067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:33:35.226 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.217+0000 7f647ee89700 1 -- 172.21.15.23:0/4084213322 shutdown_connections 2020-11-20T11:33:35.226 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.217+0000 7f647ee89700 1 -- 172.21.15.23:0/4084213322 wait complete. 2020-11-20T11:33:35.226 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.217+0000 7f647ee89700 1 Processor -- start 2020-11-20T11:33:35.226 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.218+0000 7f647ee89700 1 -- start start 2020-11-20T11:33:35.229 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.218+0000 7f647ee89700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64780aaab0 0x7f64780b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.229 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.218+0000 7f647ee89700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64780aae90 0x7f64780a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.229 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.218+0000 7f647ee89700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f647814a760 0x7f647813c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.229 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.218+0000 7f647ee89700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64780a6810 con 0x7f64780aaab0 2020-11-20T11:33:35.230 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.218+0000 7f647ee89700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647814a060 con 0x7f64780aae90 2020-11-20T11:33:35.230 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.218+0000 7f647ee89700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f647814a2c0 con 0x7f647814a760 2020-11-20T11:33:35.230 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.218+0000 7f647cc25700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64780aaab0 0x7f64780b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.230 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.218+0000 7f647cc25700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64780aaab0 0x7f64780b09c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41862/0 (socket says 172.21.15.23:41862) 2020-11-20T11:33:35.230 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.218+0000 7f647cc25700 1 -- 172.21.15.23:0/3900347897 learned_addr learned my addr 172.21.15.23:0/3900347897 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:33:35.231 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.219+0000 7f6477fff700 1 --2- 172.21.15.23:0/3900347897 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64780aae90 0x7f64780a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.231 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.219+0000 7f647d426700 1 --2- 172.21.15.23:0/3900347897 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f647814a760 0x7f647813c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.231 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.219+0000 7f647cc25700 1 -- 172.21.15.23:0/3900347897 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f647814a760 msgr2=0x7f647813c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:35.231 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.219+0000 7f647cc25700 1 --2- 172.21.15.23:0/3900347897 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f647814a760 0x7f647813c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:35.231 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.219+0000 7f647cc25700 1 -- 172.21.15.23:0/3900347897 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64780aae90 msgr2=0x7f64780a6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:35.232 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.219+0000 7f647cc25700 1 --2- 172.21.15.23:0/3900347897 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64780aae90 0x7f64780a6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:35.232 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.219+0000 7f647cc25700 1 -- 172.21.15.23:0/3900347897 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f647813d7f0 con 0x7f64780aaab0 2020-11-20T11:33:35.232 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.219+0000 7f647cc25700 1 --2- 172.21.15.23:0/3900347897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64780aaab0 0x7f64780b09c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f646c00eba0 tx=0x7f646c006e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:35.232 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.219+0000 7f6474ff9700 1 -- 172.21.15.23:0/3900347897 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646c00c850 con 0x7f64780aaab0 2020-11-20T11:33:35.232 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.219+0000 7f6474ff9700 1 -- 172.21.15.23:0/3900347897 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646c00c9b0 con 0x7f64780aaab0 2020-11-20T11:33:35.232 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.219+0000 7f647ee89700 1 -- 172.21.15.23:0/3900347897 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f647813c2b0 con 0x7f64780aaab0 2020-11-20T11:33:35.233 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.219+0000 7f6474ff9700 1 -- 172.21.15.23:0/3900347897 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646c00cd30 con 0x7f64780aaab0 2020-11-20T11:33:35.233 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.219+0000 7f647ee89700 1 -- 172.21.15.23:0/3900347897 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f647809ddb0 con 0x7f64780aaab0 2020-11-20T11:33:35.233 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.220+0000 7f6474ff9700 1 -- 172.21.15.23:0/3900347897 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f646c012600 con 0x7f64780aaab0 2020-11-20T11:33:35.233 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.220+0000 7f647ee89700 1 -- 172.21.15.23:0/3900347897 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f647814a4c0 con 0x7f64780aaab0 2020-11-20T11:33:35.233 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.221+0000 7f6474ff9700 1 --2- 172.21.15.23:0/3900347897 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6450056d60 0x7f6450059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.234 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.221+0000 7f6474ff9700 1 -- 172.21.15.23:0/3900347897 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f646c0703a0 con 0x7f64780aaab0 2020-11-20T11:33:35.234 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.221+0000 7f6477fff700 1 --2- 172.21.15.23:0/3900347897 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6450056d60 0x7f6450059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.234 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.221+0000 7f6477fff700 1 --2- 172.21.15.23:0/3900347897 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6450056d60 0x7f6450059220 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f647813e220 tx=0x7f6464006e30).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:35.234 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.225+0000 7f6474ff9700 1 -- 172.21.15.23:0/3900347897 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f647814a4c0 con 0x7f64780aaab0 2020-11-20T11:33:35.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.443+0000 7f647ee89700 1 -- 172.21.15.23:0/3900347897 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f647805c7d0 con 0x7f64780aaab0 2020-11-20T11:33:35.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.444+0000 7f6474ff9700 1 -- 172.21.15.23:0/3900347897 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7f646c045690 con 0x7f64780aaab0 2020-11-20T11:33:35.447 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:33:35.448 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.445+0000 7f647ee89700 1 -- 172.21.15.23:0/3900347897 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6450056d60 msgr2=0x7f6450059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:35.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.445+0000 7f647ee89700 1 --2- 172.21.15.23:0/3900347897 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6450056d60 0x7f6450059220 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f647813e220 tx=0x7f6464006e30).stop 2020-11-20T11:33:35.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.445+0000 7f647ee89700 1 -- 172.21.15.23:0/3900347897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64780aaab0 msgr2=0x7f64780b09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:35.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.445+0000 7f647ee89700 1 --2- 172.21.15.23:0/3900347897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64780aaab0 0x7f64780b09c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f646c00eba0 tx=0x7f646c006e30).stop 2020-11-20T11:33:35.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.445+0000 7f647ee89700 1 -- 172.21.15.23:0/3900347897 shutdown_connections 2020-11-20T11:33:35.450 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.445+0000 7f647ee89700 1 --2- 172.21.15.23:0/3900347897 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6450056d60 0x7f6450059220 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:35.450 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.446+0000 7f647ee89700 1 --2- 172.21.15.23:0/3900347897 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f647814a760 0x7f647813c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:35.450 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.446+0000 7f647ee89700 1 --2- 172.21.15.23:0/3900347897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f64780aaab0 0x7f64780b09c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:35.450 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.446+0000 7f647ee89700 1 --2- 172.21.15.23:0/3900347897 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f64780aae90 0x7f64780a6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:35.450 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.446+0000 7f647ee89700 1 -- 172.21.15.23:0/3900347897 >> 172.21.15.23:0/3900347897 conn(0x7f64780678e0 msgr2=0x7f64780aba00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:33:35.451 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.446+0000 7f647ee89700 1 -- 172.21.15.23:0/3900347897 shutdown_connections 2020-11-20T11:33:35.451 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.446+0000 7f647ee89700 1 -- 172.21.15.23:0/3900347897 wait complete. 2020-11-20T11:33:35.460 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:33:35.460 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2020-11-20T11:33:35.460 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:33:35.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.574+0000 7fcf72e3e700 1 Processor -- start 2020-11-20T11:33:35.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.574+0000 7fcf72e3e700 1 -- start start 2020-11-20T11:33:35.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.574+0000 7fcf72e3e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fcf6c0aaab0 0x7fcf6c13ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.574+0000 7fcf72e3e700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fcf6c0aae90 0x7fcf6c1381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.574+0000 7fcf72e3e700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf6c13e600 con 0x7fcf6c13aeb0 2020-11-20T11:33:35.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.574+0000 7fcf72e3e700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6c13e8a0 con 0x7fcf6c0aae90 2020-11-20T11:33:35.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.574+0000 7fcf72e3e700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6c0ab6d0 con 0x7fcf6c0aaab0 2020-11-20T11:33:35.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.575+0000 7fcf6bfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fcf6c0aae90 0x7fcf6c1381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.575+0000 7fcf6bfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fcf6c0aae90 0x7fcf6c1381a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41872/0 (socket says 172.21.15.23:41872) 2020-11-20T11:33:35.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.575+0000 7fcf6bfff700 1 -- 172.21.15.23:0/41379365 learned_addr learned my addr 172.21.15.23:0/41379365 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:33:35.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.575+0000 7fcf70bda700 1 --2- 172.21.15.23:0/41379365 >> v2:172.21.15.155:3300/0 conn(0x7fcf6c0aaab0 0x7fcf6c13ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.575+0000 7fcf6bfff700 1 -- 172.21.15.23:0/41379365 >> v2:172.21.15.155:3300/0 conn(0x7fcf6c0aaab0 msgr2=0x7fcf6c13ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:33:35.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.575+0000 7fcf6bfff700 1 --2- 172.21.15.23:0/41379365 >> v2:172.21.15.155:3300/0 conn(0x7fcf6c0aaab0 0x7fcf6c13ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:35.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.575+0000 7fcf6bfff700 1 -- 172.21.15.23:0/41379365 >> v1:172.21.15.155:6789/0 conn(0x7fcf6c13aeb0 legacy=0x7fcf6c0a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:35.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.575+0000 7fcf6bfff700 1 -- 172.21.15.23:0/41379365 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf6c0a9370 con 0x7fcf6c0aae90 2020-11-20T11:33:35.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.575+0000 7fcf70bda700 1 --2- 172.21.15.23:0/41379365 >> v2:172.21.15.155:3300/0 conn(0x7fcf6c0aaab0 0x7fcf6c13ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:33:35.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.575+0000 7fcf6bfff700 1 --2- 172.21.15.23:0/41379365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf6c0aae90 0x7fcf6c1381a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fcf6c0b23d0 tx=0x7fcf58006ef0).ready entity=mon.0 client_cookie=da87bce820eb055b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:35.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.575+0000 7fcf6b7fe700 1 -- 172.21.15.23:0/41379365 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf5800b860 con 0x7fcf6c0aae90 2020-11-20T11:33:35.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.575+0000 7fcf6b7fe700 1 -- 172.21.15.23:0/41379365 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf58000e90 con 0x7fcf6c0aae90 2020-11-20T11:33:35.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.575+0000 7fcf6b7fe700 1 -- 172.21.15.23:0/41379365 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf5800bbd0 con 0x7fcf6c0aae90 2020-11-20T11:33:35.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.576+0000 7fcf72e3e700 1 -- 172.21.15.23:0/41379365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf6c0aae90 msgr2=0x7fcf6c1381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:35.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.576+0000 7fcf72e3e700 1 --2- 172.21.15.23:0/41379365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf6c0aae90 0x7fcf6c1381a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fcf6c0b23d0 tx=0x7fcf58006ef0).stop 2020-11-20T11:33:35.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.576+0000 7fcf72e3e700 1 -- 172.21.15.23:0/41379365 shutdown_connections 2020-11-20T11:33:35.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.576+0000 7fcf72e3e700 1 --2- 172.21.15.23:0/41379365 >> v2:172.21.15.155:3300/0 conn(0x7fcf6c0aaab0 0x7fcf6c13ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:35.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.576+0000 7fcf72e3e700 1 --2- 172.21.15.23:0/41379365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf6c0aae90 0x7fcf6c1381a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:35.584 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.576+0000 7fcf72e3e700 1 -- 172.21.15.23:0/41379365 >> 172.21.15.23:0/41379365 conn(0x7fcf6c0678e0 msgr2=0x7fcf6c067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:33:35.584 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.576+0000 7fcf72e3e700 1 -- 172.21.15.23:0/41379365 shutdown_connections 2020-11-20T11:33:35.584 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.576+0000 7fcf72e3e700 1 -- 172.21.15.23:0/41379365 wait complete. 2020-11-20T11:33:35.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.577+0000 7fcf72e3e700 1 Processor -- start 2020-11-20T11:33:35.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.577+0000 7fcf72e3e700 1 -- start start 2020-11-20T11:33:35.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.577+0000 7fcf72e3e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf6c0aaab0 0x7fcf6c14a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.578+0000 7fcf72e3e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf6c0aae90 0x7fcf6c13db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.578+0000 7fcf72e3e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcf6c14dab0 0x7fcf6c14df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.578+0000 7fcf72e3e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c13d0c0 con 0x7fcf6c0aaab0 2020-11-20T11:33:35.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.578+0000 7fcf72e3e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c14ac90 con 0x7fcf6c0aae90 2020-11-20T11:33:35.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.578+0000 7fcf72e3e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c14af30 con 0x7fcf6c14dab0 2020-11-20T11:33:35.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.578+0000 7fcf70bda700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf6c0aaab0 0x7fcf6c14a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.578+0000 7fcf70bda700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf6c0aaab0 0x7fcf6c14a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41876/0 (socket says 172.21.15.23:41876) 2020-11-20T11:33:35.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.578+0000 7fcf70bda700 1 -- 172.21.15.23:0/2819825605 learned_addr learned my addr 172.21.15.23:0/2819825605 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:33:35.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.578+0000 7fcf713db700 1 --2- 172.21.15.23:0/2819825605 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcf6c14dab0 0x7fcf6c14df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.578+0000 7fcf6bfff700 1 --2- 172.21.15.23:0/2819825605 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf6c0aae90 0x7fcf6c13db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.578+0000 7fcf70bda700 1 -- 172.21.15.23:0/2819825605 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcf6c14dab0 msgr2=0x7fcf6c14df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:35.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.578+0000 7fcf70bda700 1 --2- 172.21.15.23:0/2819825605 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcf6c14dab0 0x7fcf6c14df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:35.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.578+0000 7fcf70bda700 1 -- 172.21.15.23:0/2819825605 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf6c0aae90 msgr2=0x7fcf6c13db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:35.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.578+0000 7fcf70bda700 1 --2- 172.21.15.23:0/2819825605 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf6c0aae90 0x7fcf6c13db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:35.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.578+0000 7fcf70bda700 1 -- 172.21.15.23:0/2819825605 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf6c14b1d0 con 0x7fcf6c0aaab0 2020-11-20T11:33:35.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.578+0000 7fcf70bda700 1 --2- 172.21.15.23:0/2819825605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf6c0aaab0 0x7fcf6c14a4e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fcf6000db60 tx=0x7fcf60008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:35.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.578+0000 7fcf68ff9700 1 -- 172.21.15.23:0/2819825605 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6000de70 con 0x7fcf6c0aaab0 2020-11-20T11:33:35.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.578+0000 7fcf68ff9700 1 -- 172.21.15.23:0/2819825605 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf600097c0 con 0x7fcf6c0aaab0 2020-11-20T11:33:35.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.578+0000 7fcf72e3e700 1 -- 172.21.15.23:0/2819825605 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf6c14aa30 con 0x7fcf6c0aaab0 2020-11-20T11:33:35.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.578+0000 7fcf68ff9700 1 -- 172.21.15.23:0/2819825605 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf60009b80 con 0x7fcf6c0aaab0 2020-11-20T11:33:35.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.578+0000 7fcf72e3e700 1 -- 172.21.15.23:0/2819825605 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf6c1360b0 con 0x7fcf6c0aaab0 2020-11-20T11:33:35.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.579+0000 7fcf72e3e700 1 --2- 172.21.15.23:0/2819825605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf6c13aeb0 0x7fcf6c14e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.593 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.579+0000 7fcf72e3e700 1 -- 172.21.15.23:0/2819825605 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c14b3d0 con 0x7fcf6c13aeb0 2020-11-20T11:33:35.593 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.579+0000 7fcf6bfff700 1 --2- 172.21.15.23:0/2819825605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf6c13aeb0 0x7fcf6c14e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.593 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.580+0000 7fcf68ff9700 1 -- 172.21.15.23:0/2819825605 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fcf60013480 con 0x7fcf6c0aaab0 2020-11-20T11:33:35.593 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.580+0000 7fcf6bfff700 1 -- 172.21.15.23:0/2819825605 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf6c05c7d0 con 0x7fcf6c13aeb0 2020-11-20T11:33:35.593 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.580+0000 7fcf6bfff700 1 --2- 172.21.15.23:0/2819825605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf6c13aeb0 0x7fcf6c14e400 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fcf6c0b23d0 tx=0x7fcf58016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:35.594 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.580+0000 7fcf68ff9700 1 --2- 172.21.15.23:0/2819825605 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcf44056d60 0x7fcf44059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.594 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.580+0000 7fcf68ff9700 1 -- 172.21.15.23:0/2819825605 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fcf60070320 con 0x7fcf6c0aaab0 2020-11-20T11:33:35.594 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.580+0000 7fcf68ff9700 1 -- 172.21.15.23:0/2819825605 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf58011e00 con 0x7fcf6c13aeb0 2020-11-20T11:33:35.594 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.580+0000 7fcf713db700 1 --2- 172.21.15.23:0/2819825605 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcf44056d60 0x7fcf44059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.594 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.580+0000 7fcf68ff9700 1 -- 172.21.15.23:0/2819825605 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fcf58026470 con 0x7fcf6c13aeb0 2020-11-20T11:33:35.595 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.580+0000 7fcf68ff9700 1 -- 172.21.15.23:0/2819825605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf6c13aeb0 msgr2=0x7fcf6c14e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:35.595 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.580+0000 7fcf68ff9700 1 --2- 172.21.15.23:0/2819825605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf6c13aeb0 0x7fcf6c14e400 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fcf6c0b23d0 tx=0x7fcf58016000).stop 2020-11-20T11:33:35.595 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.580+0000 7fcf68ff9700 1 -- 172.21.15.23:0/2819825605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf6c13aeb0 msgr2=0x7fcf6c14e400 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:33:35.595 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.580+0000 7fcf68ff9700 1 --2- 172.21.15.23:0/2819825605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf6c13aeb0 0x7fcf6c14e400 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:35.595 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.581+0000 7fcf713db700 1 --2- 172.21.15.23:0/2819825605 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcf44056d60 0x7fcf44059220 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fcf64004290 tx=0x7fcf64007f50).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:35.597 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.587+0000 7fcf72e3e700 1 --2- 172.21.15.23:0/2819825605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf6c0601b0 0x7fcf6c0ab9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.597 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.587+0000 7fcf72e3e700 1 -- 172.21.15.23:0/2819825605 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c14e8f0 con 0x7fcf6c0601b0 2020-11-20T11:33:35.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.587+0000 7fcf6bfff700 1 --2- 172.21.15.23:0/2819825605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf6c0601b0 0x7fcf6c0ab9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.587+0000 7fcf6bfff700 1 -- 172.21.15.23:0/2819825605 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcf6c05beb0 con 0x7fcf6c0601b0 2020-11-20T11:33:35.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.587+0000 7fcf6bfff700 1 --2- 172.21.15.23:0/2819825605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf6c0601b0 0x7fcf6c0ab9e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fcf58021000 tx=0x7fcf5801f070).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:35.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.588+0000 7fcf68ff9700 1 -- 172.21.15.23:0/2819825605 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf5801ea40 con 0x7fcf6c0601b0 2020-11-20T11:33:35.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.588+0000 7fcf68ff9700 1 -- 172.21.15.23:0/2819825605 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4276 (secure 0 0 0) 0x7fcf58011bf0 con 0x7fcf6c0601b0 2020-11-20T11:33:35.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.588+0000 7fcf68ff9700 1 -- 172.21.15.23:0/2819825605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf6c0601b0 msgr2=0x7fcf6c0ab9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:35.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.588+0000 7fcf68ff9700 1 --2- 172.21.15.23:0/2819825605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf6c0601b0 0x7fcf6c0ab9e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fcf58021000 tx=0x7fcf5801f070).stop 2020-11-20T11:33:35.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.588+0000 7fcf68ff9700 1 -- 172.21.15.23:0/2819825605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf6c0601b0 msgr2=0x7fcf6c0ab9e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:33:35.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.588+0000 7fcf68ff9700 1 --2- 172.21.15.23:0/2819825605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf6c0601b0 0x7fcf6c0ab9e0 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0x7fcf58021000 tx=0x7fcf5801f070).stop 2020-11-20T11:33:35.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.588+0000 7fcf72e3e700 1 -- 172.21.15.23:0/2819825605 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcf44056d60 msgr2=0x7fcf44059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:35.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.588+0000 7fcf72e3e700 1 --2- 172.21.15.23:0/2819825605 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcf44056d60 0x7fcf44059220 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fcf64004290 tx=0x7fcf64007f50).stop 2020-11-20T11:33:35.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.588+0000 7fcf72e3e700 1 -- 172.21.15.23:0/2819825605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf6c0aaab0 msgr2=0x7fcf6c14a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:35.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.588+0000 7fcf72e3e700 1 --2- 172.21.15.23:0/2819825605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf6c0aaab0 0x7fcf6c14a4e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fcf6000db60 tx=0x7fcf60008040).stop 2020-11-20T11:33:35.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.588+0000 7fcf713db700 1 -- 172.21.15.23:0/2819825605 reap_dead start 2020-11-20T11:33:35.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.588+0000 7fcf713db700 1 -- 172.21.15.23:0/2819825605 reap_dead start 2020-11-20T11:33:35.601 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.588+0000 7fcf72e3e700 1 -- 172.21.15.23:0/2819825605 shutdown_connections 2020-11-20T11:33:35.601 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.588+0000 7fcf72e3e700 1 -- 172.21.15.23:0/2819825605 >> 172.21.15.23:0/2819825605 conn(0x7fcf6c0678e0 msgr2=0x7fcf6c0acf80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:33:35.601 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.588+0000 7fcf72e3e700 1 -- 172.21.15.23:0/2819825605 shutdown_connections 2020-11-20T11:33:35.601 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.588+0000 7fcf72e3e700 1 -- 172.21.15.23:0/2819825605 wait complete. 2020-11-20T11:33:35.602 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:33:35.602 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:33:35.602 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:33:35.602 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:33:35.602 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 10, 2020-11-20T11:33:35.603 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:33:35.603 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:33:35.603 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:33:35.603 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:33:35.603 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:35.604 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 11, 2020-11-20T11:33:35.604 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:33:35.604 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:33:35.604 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:33:35.605 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:33:35.605 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:33:35.605 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:33:35.605 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:33:35.605 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:33:35.606 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:33:35.606 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:33:35.606 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:33:35.606 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:35.606 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:33:35.607 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:33:35.607 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:33:35.607 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:33:35.607 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:33:35.607 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:33:35.608 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:33:35.608 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:33:35.608 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:33:35.608 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:33:35.608 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:35.609 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.609 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:33:35.609 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:33:35.609 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:33:35.609 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:33:35.610 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:33:35.610 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:33:35.610 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:33:35.610 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:33:35.610 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:33:35.611 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:33:35.611 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:33:35.611 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:33:35.611 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:33:35.611 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:33:35.612 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:33:35.612 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:33:35.612 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:33:35.612 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:33:35.612 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:33:35.613 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:33:35.613 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:33:35.613 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:33:35.613 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:33:35.613 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:35.614 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:33:35.614 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.614 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:33:35.614 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.614 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:33:35.615 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:33:35.615 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:33:35.615 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:33:35.615 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.615 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:33:35.616 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:33:35.616 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:35.616 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.616 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.616 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:33:35.616 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:33:35.617 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:35.617 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:35.617 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:35.617 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.617 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:33:35.618 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:33:35.618 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:33:35.618 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:33:35.618 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:33:35.618 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.619 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.619 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:33:35.619 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:33:35.619 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:33:35.619 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:33:35.620 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.620 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:33:35.620 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:33:35.620 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:35.620 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.621 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.621 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:33:35.621 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:33:35.621 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:35.621 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:35.622 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:35.622 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.622 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:33:35.622 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:33:35.622 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:33:35.623 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:33:35.623 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:33:35.623 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.623 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.623 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:33:35.624 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:33:35.624 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:33:35.624 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:33:35.624 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.624 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:33:35.625 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:33:35.625 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:35.625 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.625 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.625 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:33:35.626 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:33:35.626 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:35.626 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:35.626 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:35.626 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.627 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:33:35.627 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:33:35.627 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:33:35.627 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:33:35.627 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:33:35.628 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:35.628 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:35.628 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.628 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:33:35.628 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:33:35.629 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.629 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:35.629 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:35.629 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:33:35.629 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:35.630 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:35.630 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:33:35.630 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.630 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:35.630 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:35.631 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:33:35.631 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:35.631 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:35.631 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.631 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:33:35.632 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:33:35.633 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:33:35.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.747+0000 7fa119ab3700 1 Processor -- start 2020-11-20T11:33:35.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.747+0000 7fa119ab3700 1 -- start start 2020-11-20T11:33:35.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.748+0000 7fa119ab3700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa1140aaab0 0x7fa11413ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.748+0000 7fa119ab3700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa11413aeb0 0x7fa1140a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.748+0000 7fa119ab3700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1141388b0 con 0x7fa1140aae90 2020-11-20T11:33:35.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.748+0000 7fa119ab3700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa11413e600 con 0x7fa11413aeb0 2020-11-20T11:33:35.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.748+0000 7fa119ab3700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa11413e8a0 con 0x7fa1140aaab0 2020-11-20T11:33:35.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.748+0000 7fa112ffd700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fa1140aae90 0x7fa1141381a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:56138/0 (socket says 172.21.15.23:56138) 2020-11-20T11:33:35.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.748+0000 7fa112ffd700 1 -- 172.21.15.23:0/3929104837 learned_addr learned my addr 172.21.15.23:0/3929104837 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:33:35.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.748+0000 7fa113fff700 1 --2- 172.21.15.23:0/3929104837 >> v2:172.21.15.23:3300/0 conn(0x7fa11413aeb0 0x7fa1140a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.748+0000 7fa1137fe700 1 --2- 172.21.15.23:0/3929104837 >> v2:172.21.15.155:3300/0 conn(0x7fa1140aaab0 0x7fa11413ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.748+0000 7fa1127fc700 1 -- 172.21.15.23:0/3929104837 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1368623864 0 0) 0x7fa1141388b0 con 0x7fa1140aae90 2020-11-20T11:33:35.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.748+0000 7fa1127fc700 1 -- 172.21.15.23:0/3929104837 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0f4003280 con 0x7fa1140aae90 2020-11-20T11:33:35.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.748+0000 7fa113fff700 1 -- 172.21.15.23:0/3929104837 >> v2:172.21.15.155:3300/0 conn(0x7fa1140aaab0 msgr2=0x7fa11413ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:33:35.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.748+0000 7fa113fff700 1 --2- 172.21.15.23:0/3929104837 >> v2:172.21.15.155:3300/0 conn(0x7fa1140aaab0 0x7fa11413ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:35.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.748+0000 7fa113fff700 1 -- 172.21.15.23:0/3929104837 >> v1:172.21.15.23:6789/0 conn(0x7fa1140aae90 legacy=0x7fa1141381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:35.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.748+0000 7fa113fff700 1 -- 172.21.15.23:0/3929104837 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1140ab6d0 con 0x7fa11413aeb0 2020-11-20T11:33:35.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.748+0000 7fa113fff700 1 --2- 172.21.15.23:0/3929104837 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa11413aeb0 0x7fa1140a8c60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fa1140b23d0 tx=0x7fa10c009220).ready entity=mon.0 client_cookie=63e6ce3b7c41be6b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:35.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.749+0000 7fa1127fc700 1 -- 172.21.15.23:0/3929104837 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10c00b950 con 0x7fa11413aeb0 2020-11-20T11:33:35.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.749+0000 7fa1127fc700 1 -- 172.21.15.23:0/3929104837 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa10c00bab0 con 0x7fa11413aeb0 2020-11-20T11:33:35.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.749+0000 7fa1127fc700 1 -- 172.21.15.23:0/3929104837 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10c00be30 con 0x7fa11413aeb0 2020-11-20T11:33:35.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.749+0000 7fa119ab3700 1 -- 172.21.15.23:0/3929104837 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa11413aeb0 msgr2=0x7fa1140a8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:35.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.749+0000 7fa119ab3700 1 --2- 172.21.15.23:0/3929104837 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa11413aeb0 0x7fa1140a8c60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fa1140b23d0 tx=0x7fa10c009220).stop 2020-11-20T11:33:35.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.749+0000 7fa119ab3700 1 -- 172.21.15.23:0/3929104837 shutdown_connections 2020-11-20T11:33:35.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.749+0000 7fa119ab3700 1 --2- 172.21.15.23:0/3929104837 >> v2:172.21.15.155:3300/0 conn(0x7fa1140aaab0 0x7fa11413ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:35.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.749+0000 7fa119ab3700 1 --2- 172.21.15.23:0/3929104837 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa11413aeb0 0x7fa1140a8c60 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:35.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.749+0000 7fa119ab3700 1 -- 172.21.15.23:0/3929104837 >> 172.21.15.23:0/3929104837 conn(0x7fa1140678e0 msgr2=0x7fa114067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:33:35.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.749+0000 7fa119ab3700 1 -- 172.21.15.23:0/3929104837 shutdown_connections 2020-11-20T11:33:35.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.749+0000 7fa119ab3700 1 -- 172.21.15.23:0/3929104837 wait complete. 2020-11-20T11:33:35.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.750+0000 7fa119ab3700 1 Processor -- start 2020-11-20T11:33:35.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.750+0000 7fa119ab3700 1 -- start start 2020-11-20T11:33:35.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.751+0000 7fa119ab3700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa1140aaab0 0x7fa114140cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.751+0000 7fa119ab3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1140aae90 0x7fa1140a6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.751+0000 7fa119ab3700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa1140a5790 0x7fa1140a5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.751+0000 7fa119ab3700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1140a6e10 con 0x7fa1140aae90 2020-11-20T11:33:35.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.751+0000 7fa119ab3700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1140a6b70 con 0x7fa1140aaab0 2020-11-20T11:33:35.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.751+0000 7fa119ab3700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1140a52f0 con 0x7fa1140a5790 2020-11-20T11:33:35.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.751+0000 7fa112ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1140aae90 0x7fa1140a6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.751+0000 7fa112ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1140aae90 0x7fa1140a6340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41896/0 (socket says 172.21.15.23:41896) 2020-11-20T11:33:35.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.751+0000 7fa112ffd700 1 -- 172.21.15.23:0/2742395090 learned_addr learned my addr 172.21.15.23:0/2742395090 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:33:35.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.751+0000 7fa1137fe700 1 --2- 172.21.15.23:0/2742395090 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa1140aaab0 0x7fa114140cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.751+0000 7fa113fff700 1 --2- 172.21.15.23:0/2742395090 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa1140a5790 0x7fa1140a5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.751+0000 7fa112ffd700 1 -- 172.21.15.23:0/2742395090 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa1140a5790 msgr2=0x7fa1140a5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:35.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.751+0000 7fa112ffd700 1 --2- 172.21.15.23:0/2742395090 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa1140a5790 0x7fa1140a5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:35.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.751+0000 7fa112ffd700 1 -- 172.21.15.23:0/2742395090 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa1140aaab0 msgr2=0x7fa114140cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:35.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.751+0000 7fa112ffd700 1 --2- 172.21.15.23:0/2742395090 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa1140aaab0 0x7fa114140cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:35.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.751+0000 7fa112ffd700 1 -- 172.21.15.23:0/2742395090 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa114149fe0 con 0x7fa1140aae90 2020-11-20T11:33:35.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.751+0000 7fa112ffd700 1 --2- 172.21.15.23:0/2742395090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1140aae90 0x7fa1140a6340 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fa10400c770 tx=0x7fa104007b10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:35.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.752+0000 7fa10b7fe700 1 -- 172.21.15.23:0/2742395090 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1040072e0 con 0x7fa1140aae90 2020-11-20T11:33:35.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.752+0000 7fa10b7fe700 1 -- 172.21.15.23:0/2742395090 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa104007440 con 0x7fa1140aae90 2020-11-20T11:33:35.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.752+0000 7fa119ab3700 1 -- 172.21.15.23:0/2742395090 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1140a6830 con 0x7fa1140aae90 2020-11-20T11:33:35.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.752+0000 7fa10b7fe700 1 -- 172.21.15.23:0/2742395090 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa104005840 con 0x7fa1140aae90 2020-11-20T11:33:35.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.752+0000 7fa119ab3700 1 -- 172.21.15.23:0/2742395090 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa11414a570 con 0x7fa1140aae90 2020-11-20T11:33:35.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.753+0000 7fa119ab3700 1 --2- 172.21.15.23:0/2742395090 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa11413aeb0 0x7fa11414a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.753+0000 7fa119ab3700 1 -- 172.21.15.23:0/2742395090 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1141411a0 con 0x7fa11413aeb0 2020-11-20T11:33:35.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.753+0000 7fa10b7fe700 1 -- 172.21.15.23:0/2742395090 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa10401a060 con 0x7fa1140aae90 2020-11-20T11:33:35.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.753+0000 7fa1137fe700 1 --2- 172.21.15.23:0/2742395090 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa11413aeb0 0x7fa11414a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.753+0000 7fa10b7fe700 1 --2- 172.21.15.23:0/2742395090 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa0f0056d60 0x7fa0f0059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.753+0000 7fa10b7fe700 1 -- 172.21.15.23:0/2742395090 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fa10406f840 con 0x7fa1140aae90 2020-11-20T11:33:35.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.753+0000 7fa1137fe700 1 -- 172.21.15.23:0/2742395090 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa1140a5170 con 0x7fa11413aeb0 2020-11-20T11:33:35.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.753+0000 7fa113fff700 1 --2- 172.21.15.23:0/2742395090 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa0f0056d60 0x7fa0f0059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.753+0000 7fa1137fe700 1 --2- 172.21.15.23:0/2742395090 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa11413aeb0 0x7fa11414a760 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fa0fc011c10 tx=0x7fa0fc006e30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:35.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.754+0000 7fa10b7fe700 1 -- 172.21.15.23:0/2742395090 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc00fb80 con 0x7fa11413aeb0 2020-11-20T11:33:35.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.754+0000 7fa113fff700 1 --2- 172.21.15.23:0/2742395090 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa0f0056d60 0x7fa0f0059220 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fa1140af230 tx=0x7fa10c016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:35.769 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.754+0000 7fa10b7fe700 1 -- 172.21.15.23:0/2742395090 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fa0fc00fce0 con 0x7fa11413aeb0 2020-11-20T11:33:35.769 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.754+0000 7fa10b7fe700 1 -- 172.21.15.23:0/2742395090 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa11413aeb0 msgr2=0x7fa11414a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:35.769 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.754+0000 7fa10b7fe700 1 --2- 172.21.15.23:0/2742395090 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa11413aeb0 0x7fa11414a760 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fa0fc011c10 tx=0x7fa0fc006e30).stop 2020-11-20T11:33:35.769 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.754+0000 7fa10b7fe700 1 -- 172.21.15.23:0/2742395090 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa11413aeb0 msgr2=0x7fa11414a760 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:33:35.770 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.754+0000 7fa10b7fe700 1 --2- 172.21.15.23:0/2742395090 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa11413aeb0 0x7fa11414a760 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0x7fa0fc011c10 tx=0x7fa0fc006e30).stop 2020-11-20T11:33:35.771 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.761+0000 7fa119ab3700 1 --2- 172.21.15.23:0/2742395090 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa1140601b0 0x7fa11414d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.761+0000 7fa119ab3700 1 -- 172.21.15.23:0/2742395090 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1140aa680 con 0x7fa1140601b0 2020-11-20T11:33:35.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.761+0000 7fa1137fe700 1 --2- 172.21.15.23:0/2742395090 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa1140601b0 0x7fa11414d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.761+0000 7fa1137fe700 1 -- 172.21.15.23:0/2742395090 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa11405c7d0 con 0x7fa1140601b0 2020-11-20T11:33:35.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.762+0000 7fa1137fe700 1 --2- 172.21.15.23:0/2742395090 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa1140601b0 0x7fa11414d5b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fa0fc018000 tx=0x7fa0fc01b000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:35.773 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.762+0000 7fa10b7fe700 1 -- 172.21.15.23:0/2742395090 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc006290 con 0x7fa1140601b0 2020-11-20T11:33:35.773 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.762+0000 7fa10b7fe700 1 -- 172.21.15.23:0/2742395090 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4784 (secure 0 0 0) 0x7fa0fc027070 con 0x7fa1140601b0 2020-11-20T11:33:35.773 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.762+0000 7fa10b7fe700 1 -- 172.21.15.23:0/2742395090 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa1140601b0 msgr2=0x7fa11414d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:35.773 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.762+0000 7fa10b7fe700 1 --2- 172.21.15.23:0/2742395090 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa1140601b0 0x7fa11414d5b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fa0fc018000 tx=0x7fa0fc01b000).stop 2020-11-20T11:33:35.773 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.762+0000 7fa10b7fe700 1 -- 172.21.15.23:0/2742395090 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa1140601b0 msgr2=0x7fa11414d5b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:33:35.774 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.762+0000 7fa10b7fe700 1 --2- 172.21.15.23:0/2742395090 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa1140601b0 0x7fa11414d5b0 secure :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0x7fa0fc018000 tx=0x7fa0fc01b000).stop 2020-11-20T11:33:35.774 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.762+0000 7fa119ab3700 1 -- 172.21.15.23:0/2742395090 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa0f0056d60 msgr2=0x7fa0f0059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:35.774 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.762+0000 7fa119ab3700 1 --2- 172.21.15.23:0/2742395090 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa0f0056d60 0x7fa0f0059220 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fa1140af230 tx=0x7fa10c016000).stop 2020-11-20T11:33:35.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.762+0000 7fa119ab3700 1 -- 172.21.15.23:0/2742395090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1140aae90 msgr2=0x7fa1140a6340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:35.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.762+0000 7fa119ab3700 1 --2- 172.21.15.23:0/2742395090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1140aae90 0x7fa1140a6340 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fa10400c770 tx=0x7fa104007b10).stop 2020-11-20T11:33:35.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.762+0000 7fa113fff700 1 -- 172.21.15.23:0/2742395090 reap_dead start 2020-11-20T11:33:35.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.762+0000 7fa113fff700 1 -- 172.21.15.23:0/2742395090 reap_dead start 2020-11-20T11:33:35.776 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.762+0000 7fa119ab3700 1 -- 172.21.15.23:0/2742395090 shutdown_connections 2020-11-20T11:33:35.776 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.762+0000 7fa119ab3700 1 -- 172.21.15.23:0/2742395090 >> 172.21.15.23:0/2742395090 conn(0x7fa1140678e0 msgr2=0x7fa1140ace30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:33:35.776 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.762+0000 7fa119ab3700 1 -- 172.21.15.23:0/2742395090 shutdown_connections 2020-11-20T11:33:35.777 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.762+0000 7fa119ab3700 1 -- 172.21.15.23:0/2742395090 wait complete. 2020-11-20T11:33:35.777 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:33:35.777 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:33:35.778 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:33:35.778 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:33:35.778 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 10, 2020-11-20T11:33:35.778 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:33:35.778 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:33:35.779 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:33:35.779 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:33:35.779 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:35.779 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 11, 2020-11-20T11:33:35.780 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:33:35.780 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:33:35.780 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:33:35.780 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:33:35.780 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:33:35.781 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:33:35.781 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:33:35.781 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:33:35.781 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:33:35.781 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:33:35.782 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:33:35.782 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:35.782 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:33:35.782 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:33:35.783 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:33:35.783 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:33:35.783 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:33:35.783 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:33:35.783 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:33:35.784 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:33:35.784 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:33:35.784 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:33:35.784 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:35.784 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.785 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:33:35.785 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:33:35.785 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:33:35.785 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:33:35.785 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:33:35.786 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:33:35.786 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:33:35.786 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:33:35.786 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:33:35.787 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:33:35.787 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:33:35.787 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:33:35.787 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:33:35.787 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:33:35.788 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:33:35.788 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:33:35.788 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:33:35.788 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:33:35.788 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:33:35.789 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:33:35.789 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:33:35.789 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:33:35.789 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:33:35.789 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:35.790 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:33:35.790 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.790 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:33:35.790 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.791 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:33:35.791 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:33:35.791 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:33:35.791 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:33:35.791 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.792 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:33:35.792 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:33:35.792 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:35.792 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.792 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.793 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:33:35.793 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:33:35.793 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:35.793 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:35.794 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:35.794 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.794 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:33:35.794 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:33:35.794 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:33:35.795 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:33:35.795 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:33:35.796 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.796 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.796 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:33:35.797 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:33:35.797 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:33:35.798 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:33:35.798 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.798 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:33:35.798 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:33:35.799 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:35.799 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.799 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.799 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:33:35.800 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:33:35.800 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:35.800 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:35.800 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:35.801 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.801 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:33:35.802 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:33:35.802 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:33:35.802 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:33:35.803 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:33:35.803 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.803 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.803 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:33:35.804 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:33:35.804 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:33:35.804 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:33:35.804 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.804 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:33:35.805 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:33:35.805 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:35.805 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.805 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.806 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:33:35.806 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:33:35.806 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:35.807 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:35.807 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:35.808 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.808 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:33:35.808 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:33:35.809 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:33:35.809 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:33:35.809 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:33:35.809 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:35.809 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:35.810 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.810 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:33:35.810 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:33:35.810 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.810 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:35.811 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:35.811 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:33:35.811 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:35.812 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:35.812 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:33:35.813 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.813 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:35.813 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:35.814 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:33:35.814 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:35.814 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:35.814 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:33:35.815 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.815 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:35.815 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:35.815 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 4 2020-11-20T11:33:35.815 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:35.816 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:35.816 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:33:35.816 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.816 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:35.817 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:35.817 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:33:35.818 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:35.818 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:35.818 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:33:35.819 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.819 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:35.820 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:35.820 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:33:35.820 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:35.821 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:35.821 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.821 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:33:35.821 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:33:35.823 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:33:35.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.937+0000 7fbd750d0700 1 Processor -- start 2020-11-20T11:33:35.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.937+0000 7fbd750d0700 1 -- start start 2020-11-20T11:33:35.943 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.937+0000 7fbd750d0700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd700aaab0 0x7fbd7013ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.943 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.937+0000 7fbd750d0700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fbd700aae90 0x7fbd701381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.943 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.937+0000 7fbd750d0700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fbd7013aeb0 0x7fbd700a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.943 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.937+0000 7fbd750d0700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd7013e600 con 0x7fbd700aae90 2020-11-20T11:33:35.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.937+0000 7fbd750d0700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd7013e8a0 con 0x7fbd7013aeb0 2020-11-20T11:33:35.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.938+0000 7fbd750d0700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd700ab6d0 con 0x7fbd700aaab0 2020-11-20T11:33:35.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.938+0000 7fbd6e59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fbd700aae90 0x7fbd701381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.938+0000 7fbd6e59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fbd700aae90 0x7fbd701381a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41906/0 (socket says 172.21.15.23:41906) 2020-11-20T11:33:35.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.938+0000 7fbd6e59c700 1 -- 172.21.15.23:0/4263441990 learned_addr learned my addr 172.21.15.23:0/4263441990 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:33:35.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.938+0000 7fbd6f59e700 1 --2- 172.21.15.23:0/4263441990 >> v2:172.21.15.155:3300/0 conn(0x7fbd7013aeb0 0x7fbd700a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.938+0000 7fbd6ed9d700 1 --2- 172.21.15.23:0/4263441990 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd700aaab0 0x7fbd7013ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.938+0000 7fbd6e59c700 1 -- 172.21.15.23:0/4263441990 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd700aaab0 msgr2=0x7fbd7013ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:33:35.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.938+0000 7fbd6e59c700 1 --2- 172.21.15.23:0/4263441990 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd700aaab0 0x7fbd7013ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:35.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.938+0000 7fbd6e59c700 1 -- 172.21.15.23:0/4263441990 >> v2:172.21.15.155:3300/0 conn(0x7fbd7013aeb0 msgr2=0x7fbd700a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:33:35.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.938+0000 7fbd6e59c700 1 --2- 172.21.15.23:0/4263441990 >> v2:172.21.15.155:3300/0 conn(0x7fbd7013aeb0 0x7fbd700a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:35.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.938+0000 7fbd6e59c700 1 -- 172.21.15.23:0/4263441990 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd7013f230 con 0x7fbd700aae90 2020-11-20T11:33:35.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.938+0000 7fbd6e59c700 1 --2- 172.21.15.23:0/4263441990 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd700aae90 0x7fbd701381a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fbd700ae570 tx=0x7fbd64006ef0).ready entity=mon.0 client_cookie=475ff502bc2035a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:35.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.938+0000 7fbd6dd9b700 1 -- 172.21.15.23:0/4263441990 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6400b680 con 0x7fbd700aae90 2020-11-20T11:33:35.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.938+0000 7fbd6dd9b700 1 -- 172.21.15.23:0/4263441990 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd64000e90 con 0x7fbd700aae90 2020-11-20T11:33:35.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.938+0000 7fbd6dd9b700 1 -- 172.21.15.23:0/4263441990 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6400b9f0 con 0x7fbd700aae90 2020-11-20T11:33:35.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.939+0000 7fbd750d0700 1 -- 172.21.15.23:0/4263441990 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd700aae90 msgr2=0x7fbd701381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:35.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.939+0000 7fbd750d0700 1 --2- 172.21.15.23:0/4263441990 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd700aae90 0x7fbd701381a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fbd700ae570 tx=0x7fbd64006ef0).stop 2020-11-20T11:33:35.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.939+0000 7fbd750d0700 1 -- 172.21.15.23:0/4263441990 shutdown_connections 2020-11-20T11:33:35.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.939+0000 7fbd750d0700 1 --2- 172.21.15.23:0/4263441990 >> v2:172.21.15.155:3300/0 conn(0x7fbd7013aeb0 0x7fbd700a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:35.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.939+0000 7fbd750d0700 1 --2- 172.21.15.23:0/4263441990 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd700aaab0 0x7fbd7013ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:35.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.939+0000 7fbd750d0700 1 --2- 172.21.15.23:0/4263441990 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd700aae90 0x7fbd701381a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:35.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.939+0000 7fbd750d0700 1 -- 172.21.15.23:0/4263441990 >> 172.21.15.23:0/4263441990 conn(0x7fbd700678e0 msgr2=0x7fbd70067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:33:35.949 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.939+0000 7fbd750d0700 1 -- 172.21.15.23:0/4263441990 shutdown_connections 2020-11-20T11:33:35.949 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.939+0000 7fbd750d0700 1 -- 172.21.15.23:0/4263441990 wait complete. 2020-11-20T11:33:35.949 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.940+0000 7fbd750d0700 1 Processor -- start 2020-11-20T11:33:35.949 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.940+0000 7fbd750d0700 1 -- start start 2020-11-20T11:33:35.952 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.941+0000 7fbd750d0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd700aaab0 0x7fbd7009ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.952 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.941+0000 7fbd750d0700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd700aae90 0x7fbd700a5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.952 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.941+0000 7fbd750d0700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd700a8100 0x7fbd7014a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.953 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.941+0000 7fbd750d0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd700a6920 con 0x7fbd700aaab0 2020-11-20T11:33:35.953 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.941+0000 7fbd750d0700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd700a5e00 con 0x7fbd700aae90 2020-11-20T11:33:35.953 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.941+0000 7fbd750d0700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd700a5b60 con 0x7fbd700a8100 2020-11-20T11:33:35.953 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.941+0000 7fbd6ed9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd700aaab0 0x7fbd7009ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.954 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.941+0000 7fbd6ed9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd700aaab0 0x7fbd7009ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41912/0 (socket says 172.21.15.23:41912) 2020-11-20T11:33:35.954 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.941+0000 7fbd6ed9d700 1 -- 172.21.15.23:0/1325433988 learned_addr learned my addr 172.21.15.23:0/1325433988 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:33:35.954 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.941+0000 7fbd6f59e700 1 --2- 172.21.15.23:0/1325433988 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd700a8100 0x7fbd7014a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.954 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.941+0000 7fbd6e59c700 1 --2- 172.21.15.23:0/1325433988 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd700aae90 0x7fbd700a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.954 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.941+0000 7fbd6ed9d700 1 -- 172.21.15.23:0/1325433988 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd700a8100 msgr2=0x7fbd7014a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:35.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.941+0000 7fbd6ed9d700 1 --2- 172.21.15.23:0/1325433988 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd700a8100 0x7fbd7014a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:35.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.941+0000 7fbd6ed9d700 1 -- 172.21.15.23:0/1325433988 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd700aae90 msgr2=0x7fbd700a5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:35.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.941+0000 7fbd6ed9d700 1 --2- 172.21.15.23:0/1325433988 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd700aae90 0x7fbd700a5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:35.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.941+0000 7fbd6ed9d700 1 -- 172.21.15.23:0/1325433988 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd700a5820 con 0x7fbd700aaab0 2020-11-20T11:33:35.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.941+0000 7fbd6e59c700 1 --2- 172.21.15.23:0/1325433988 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbd700aae90 0x7fbd700a5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:33:35.956 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.941+0000 7fbd6ed9d700 1 --2- 172.21.15.23:0/1325433988 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd700aaab0 0x7fbd7009ddb0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fbd6000eba0 tx=0x7fbd60006e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:35.956 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.941+0000 7fbd56ffd700 1 -- 172.21.15.23:0/1325433988 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6000ca60 con 0x7fbd700aaab0 2020-11-20T11:33:35.956 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.941+0000 7fbd56ffd700 1 -- 172.21.15.23:0/1325433988 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd6000cbc0 con 0x7fbd700aaab0 2020-11-20T11:33:35.956 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.941+0000 7fbd750d0700 1 -- 172.21.15.23:0/1325433988 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd7014a820 con 0x7fbd700aaab0 2020-11-20T11:33:35.957 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.941+0000 7fbd56ffd700 1 -- 172.21.15.23:0/1325433988 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd600123f0 con 0x7fbd700aaab0 2020-11-20T11:33:35.957 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.941+0000 7fbd750d0700 1 -- 172.21.15.23:0/1325433988 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd7013d2c0 con 0x7fbd700aaab0 2020-11-20T11:33:35.957 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.942+0000 7fbd750d0700 1 --2- 172.21.15.23:0/1325433988 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd7013aeb0 0x7fbd7013d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.957 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.942+0000 7fbd750d0700 1 -- 172.21.15.23:0/1325433988 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7013ec60 con 0x7fbd7013aeb0 2020-11-20T11:33:35.957 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.942+0000 7fbd6e59c700 1 --2- 172.21.15.23:0/1325433988 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd7013aeb0 0x7fbd7013d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.958 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.943+0000 7fbd56ffd700 1 -- 172.21.15.23:0/1325433988 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbd600127e0 con 0x7fbd700aaab0 2020-11-20T11:33:35.958 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.943+0000 7fbd6e59c700 1 -- 172.21.15.23:0/1325433988 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbd7005c7d0 con 0x7fbd7013aeb0 2020-11-20T11:33:35.958 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.943+0000 7fbd6e59c700 1 --2- 172.21.15.23:0/1325433988 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd7013aeb0 0x7fbd7013d510 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fbd700ae570 tx=0x7fbd64016000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:35.958 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.943+0000 7fbd56ffd700 1 --2- 172.21.15.23:0/1325433988 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbd4c056d60 0x7fbd4c059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.959 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.943+0000 7fbd56ffd700 1 -- 172.21.15.23:0/1325433988 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fbd600704e0 con 0x7fbd700aaab0 2020-11-20T11:33:35.959 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.943+0000 7fbd56ffd700 1 -- 172.21.15.23:0/1325433988 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd640119d0 con 0x7fbd7013aeb0 2020-11-20T11:33:35.959 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.943+0000 7fbd6f59e700 1 --2- 172.21.15.23:0/1325433988 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbd4c056d60 0x7fbd4c059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.959 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.943+0000 7fbd56ffd700 1 -- 172.21.15.23:0/1325433988 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fbd64011b30 con 0x7fbd7013aeb0 2020-11-20T11:33:35.959 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.943+0000 7fbd56ffd700 1 -- 172.21.15.23:0/1325433988 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd7013aeb0 msgr2=0x7fbd7013d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:35.960 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.943+0000 7fbd56ffd700 1 --2- 172.21.15.23:0/1325433988 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd7013aeb0 0x7fbd7013d510 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fbd700ae570 tx=0x7fbd64016000).stop 2020-11-20T11:33:35.960 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.943+0000 7fbd56ffd700 1 -- 172.21.15.23:0/1325433988 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd7013aeb0 msgr2=0x7fbd7013d510 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:33:35.960 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.943+0000 7fbd56ffd700 1 --2- 172.21.15.23:0/1325433988 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd7013aeb0 0x7fbd7013d510 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:35.960 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.943+0000 7fbd6f59e700 1 --2- 172.21.15.23:0/1325433988 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbd4c056d60 0x7fbd4c059220 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fbd70138d00 tx=0x7fbd68008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:35.962 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.950+0000 7fbd750d0700 1 --2- 172.21.15.23:0/1325433988 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd700601b0 0x7fbd7013da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:35.962 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.950+0000 7fbd750d0700 1 -- 172.21.15.23:0/1325433988 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd70044710 con 0x7fbd700601b0 2020-11-20T11:33:35.963 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.950+0000 7fbd6e59c700 1 --2- 172.21.15.23:0/1325433988 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd700601b0 0x7fbd7013da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:35.963 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.951+0000 7fbd6e59c700 1 -- 172.21.15.23:0/1325433988 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbd7005c280 con 0x7fbd700601b0 2020-11-20T11:33:35.963 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.951+0000 7fbd6e59c700 1 --2- 172.21.15.23:0/1325433988 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd700601b0 0x7fbd7013da00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fbd64006f30 tx=0x7fbd6401e000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:35.963 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.951+0000 7fbd56ffd700 1 -- 172.21.15.23:0/1325433988 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6401d8b0 con 0x7fbd700601b0 2020-11-20T11:33:35.963 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.951+0000 7fbd56ffd700 1 -- 172.21.15.23:0/1325433988 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4784 (secure 0 0 0) 0x7fbd640117c0 con 0x7fbd700601b0 2020-11-20T11:33:35.964 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.951+0000 7fbd56ffd700 1 -- 172.21.15.23:0/1325433988 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd700601b0 msgr2=0x7fbd7013da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:35.964 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.951+0000 7fbd56ffd700 1 --2- 172.21.15.23:0/1325433988 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd700601b0 0x7fbd7013da00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fbd64006f30 tx=0x7fbd6401e000).stop 2020-11-20T11:33:35.964 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.951+0000 7fbd56ffd700 1 -- 172.21.15.23:0/1325433988 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd700601b0 msgr2=0x7fbd7013da00 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:33:35.964 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.951+0000 7fbd56ffd700 1 --2- 172.21.15.23:0/1325433988 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbd700601b0 0x7fbd7013da00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:35.964 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.952+0000 7fbd750d0700 1 -- 172.21.15.23:0/1325433988 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbd4c056d60 msgr2=0x7fbd4c059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:35.965 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.952+0000 7fbd750d0700 1 --2- 172.21.15.23:0/1325433988 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbd4c056d60 0x7fbd4c059220 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fbd70138d00 tx=0x7fbd68008040).stop 2020-11-20T11:33:35.965 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.952+0000 7fbd750d0700 1 -- 172.21.15.23:0/1325433988 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd700aaab0 msgr2=0x7fbd7009ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:35.965 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.952+0000 7fbd750d0700 1 --2- 172.21.15.23:0/1325433988 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd700aaab0 0x7fbd7009ddb0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fbd6000eba0 tx=0x7fbd60006e30).stop 2020-11-20T11:33:35.965 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.952+0000 7fbd6f59e700 1 -- 172.21.15.23:0/1325433988 reap_dead start 2020-11-20T11:33:35.965 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.952+0000 7fbd6f59e700 1 -- 172.21.15.23:0/1325433988 reap_dead start 2020-11-20T11:33:35.966 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.952+0000 7fbd750d0700 1 -- 172.21.15.23:0/1325433988 shutdown_connections 2020-11-20T11:33:35.966 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.952+0000 7fbd750d0700 1 -- 172.21.15.23:0/1325433988 >> 172.21.15.23:0/1325433988 conn(0x7fbd700678e0 msgr2=0x7fbd70139a30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:33:35.966 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.952+0000 7fbd750d0700 1 -- 172.21.15.23:0/1325433988 shutdown_connections 2020-11-20T11:33:35.966 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:35.952+0000 7fbd750d0700 1 -- 172.21.15.23:0/1325433988 wait complete. 2020-11-20T11:33:35.967 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:33:35.967 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:33:35.967 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:33:35.967 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:33:35.967 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 10, 2020-11-20T11:33:35.968 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:33:35.968 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:33:35.968 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:33:35.968 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:33:35.968 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:35.969 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 11, 2020-11-20T11:33:35.969 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:33:35.969 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:33:35.969 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:33:35.969 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:33:35.970 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:33:35.970 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:33:35.970 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:33:35.970 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:33:35.970 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:33:35.971 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:33:35.971 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:33:35.971 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:35.971 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:33:35.971 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:33:35.972 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:33:35.972 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:33:35.972 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:33:35.973 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:33:35.973 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:33:35.973 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:33:35.973 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:33:35.973 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:33:35.974 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:35.974 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.974 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:33:35.974 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:33:35.974 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:33:35.975 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:33:35.975 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:33:35.975 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:33:35.975 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:33:35.975 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:33:35.976 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:33:35.976 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:33:35.976 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:33:35.976 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:33:35.976 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:33:35.977 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:33:35.977 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:33:35.977 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:33:35.977 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:33:35.978 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:33:35.978 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:33:35.978 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:33:35.978 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:33:35.978 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:33:35.979 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:33:35.979 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:35.979 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:33:35.979 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.979 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:33:35.980 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.980 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:33:35.980 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:33:35.980 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:33:35.981 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:33:35.981 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.981 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:33:35.982 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:33:35.982 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:35.983 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.983 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.983 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:33:35.983 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:33:35.983 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:35.984 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:35.984 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:35.984 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.984 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:33:35.985 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:33:35.985 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:33:35.985 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:33:35.985 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:33:35.985 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.986 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.986 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:33:35.986 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:33:35.986 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:33:35.986 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:33:35.987 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.987 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:33:35.987 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:33:35.987 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:35.987 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.988 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.988 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:33:35.988 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:33:35.988 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:35.988 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:35.989 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:35.989 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.989 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:33:35.989 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:33:35.989 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:33:35.990 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:33:35.990 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:33:35.990 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.991 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.991 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:33:35.991 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:33:35.991 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:33:35.992 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:33:35.992 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.992 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:33:35.992 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:33:35.992 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:35.993 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.997 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:35.997 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:33:35.998 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:33:35.998 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:35.998 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:35.998 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:35.998 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:35.999 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:33:35.999 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:33:35.999 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:33:35.999 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:33:35.999 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:33:36.000 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:36.000 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:36.000 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:36.000 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:33:36.000 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:33:36.001 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:36.001 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:36.001 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:36.001 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:33:36.001 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:36.002 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:36.002 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:33:36.002 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:36.002 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:36.002 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:36.003 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:33:36.003 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:36.003 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:36.003 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:33:36.003 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:36.004 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:36.004 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:36.004 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 4 2020-11-20T11:33:36.004 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:36.004 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:36.005 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:33:36.005 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:36.005 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:36.005 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:36.005 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:33:36.006 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:36.006 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:36.006 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:33:36.006 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:36.006 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:36.007 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:36.007 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:33:36.007 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:36.007 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:36.007 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:36.008 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:33:36.008 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:33:36.010 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2020-11-20T11:33:36.010 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2020-11-20T11:33:36.011 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2020-11-20T11:33:36.011 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2020-11-20T11:33:36.011 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2020-11-20T11:33:36.012 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T11:33:36.012 INFO:tasks.workunit.client.0.smithi023.stderr:Note: switching to 'b86e691bf191566922a8f08e2a5abf64fde8cfcb'. 2020-11-20T11:33:36.012 INFO:tasks.workunit.client.0.smithi023.stderr: 2020-11-20T11:33:36.012 INFO:tasks.workunit.client.0.smithi023.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2020-11-20T11:33:36.013 INFO:tasks.workunit.client.0.smithi023.stderr:changes and commit them, and you can discard any commits you make in this 2020-11-20T11:33:36.013 INFO:tasks.workunit.client.0.smithi023.stderr:state without impacting any branches by switching back to a branch. 2020-11-20T11:33:36.013 INFO:tasks.workunit.client.0.smithi023.stderr: 2020-11-20T11:33:36.013 INFO:tasks.workunit.client.0.smithi023.stderr:If you want to create a new branch to retain commits you create, you may 2020-11-20T11:33:36.014 INFO:tasks.workunit.client.0.smithi023.stderr:do so (now or later) by using -c with the switch command. Example: 2020-11-20T11:33:36.014 INFO:tasks.workunit.client.0.smithi023.stderr: 2020-11-20T11:33:36.014 INFO:tasks.workunit.client.0.smithi023.stderr: git switch -c 2020-11-20T11:33:36.014 INFO:tasks.workunit.client.0.smithi023.stderr: 2020-11-20T11:33:36.014 INFO:tasks.workunit.client.0.smithi023.stderr:Or undo this operation with: 2020-11-20T11:33:36.015 INFO:tasks.workunit.client.0.smithi023.stderr: 2020-11-20T11:33:36.015 INFO:tasks.workunit.client.0.smithi023.stderr: git switch - 2020-11-20T11:33:36.015 INFO:tasks.workunit.client.0.smithi023.stderr: 2020-11-20T11:33:36.015 INFO:tasks.workunit.client.0.smithi023.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2020-11-20T11:33:36.016 INFO:tasks.workunit.client.0.smithi023.stderr: 2020-11-20T11:33:36.016 INFO:tasks.workunit.client.0.smithi023.stderr:HEAD is now at b86e691bf1 Merge PR #37174 into wip-pdonnell-testing-20201120.053228 2020-11-20T11:33:36.017 INFO:teuthology.orchestra.run.smithi023:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2020-11-20T11:33:36.025 INFO:tasks.ceph.mon.b:Stopped 2020-11-20T11:33:36.025 INFO:tasks.mon_thrash:Sending STOP to mon a 2020-11-20T11:33:36.026 INFO:tasks.ceph.mon.a:Sent signal 19 2020-11-20T11:33:36.026 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2020-11-20T11:33:36.049 INFO:tasks.workunit.client.0.smithi023.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2020-11-20T11:33:36.051 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2020-11-20T11:33:36.052 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2020-11-20T11:33:36.701 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2020-11-20T11:33:36.752 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2020-11-20T11:33:36.789 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2020-11-20T11:33:36.791 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2020-11-20T11:33:36.792 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2020-11-20T11:33:36.832 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2020-11-20T11:33:36.833 INFO:teuthology.orchestra.run.smithi023:> set -ex 2020-11-20T11:33:36.833 INFO:teuthology.orchestra.run.smithi023:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2020-11-20T11:33:36.891 INFO:teuthology.orchestra.run.smithi023.stderr:+ dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2020-11-20T11:33:36.893 INFO:teuthology.orchestra.run.smithi023.stdout:mon/rbd_snaps_ops.shmon/osd.shmon/config.shmon/pg_autoscaler.shmon/test_mon_osdmap_prune.shmon/pool_ops.shmon/ping.pymon/crush_ops.shmon/test_mon_config_key.pymon/auth_caps.shmon/caps.shmon/test_config_key_caps.shpost-file.shtrue.shceph-helpers-root.shceph-tests/ceph-admin-commands.sherasure-code/bench.sherasure-code/encode-decode-non-regression.shrgw/test_rgw_reshard.pyrgw/test_rgw_throttle.shrgw/s3_bucket_quota.plrgw/test_rgw_gc_log.shrgw/test_rgw_obj.shrgw/s3_multipart_upload.plrgw/test_rgw_orphan_list.shrgw/s3_user_quota.plrgw/run-reshard.shrgw/run-s3tests.shcephtool/test_daemon.shcephtool/test_kvstore_tool.shcephtool/test.shsuites/cephfs_journal_tool_smoke.shsuites/iogen.shsuites/pjd.shsuites/dbench.shsuites/iozone-sync.shsuites/fsx.shsuites/fsstress.shsuites/fio.shsuites/iozone.shsuites/blogbench.shsuites/dbench-short.shsuites/bonnie.shsuites/fsync-tester.shsuites/ffsb.shlibcephfs/test.shcls/test_cls_cmpomap.shcls/test_cls_numops.shcls/test_cls_hello.shcls/test_cls_sdk.shcls/test_cls_rbd.shcls/test_cls_refcount.shcls/test_cls_2pc_queue.shcls/test_cls_lock.shcls/test_cls_journal.shcls/test_cls_log.shcls/test_cls_rgw.shosdc/stress_objectcacher.shrbd/merge_diff.shrbd/krbd_stable_pages_required.shrbd/rbd_mirror_ha.shrbd/diff_continuous.shrbd/test_librbd.shrbd/test_lock_fence.shrbd/qemu_rebuild_object_map.shrbd/qemu-iotests.shrbd/test_rbd_mirror.shrbd/krbd_udev_enumerate.shrbd/test_admin_socket.shrbd/krbd_wac.shrbd/notify_slave.shrbd/test_librbd_python.shrbd/rbd_mirror_bootstrap.shrbd/test_rbd_tasks.shrbd/rbd-ggate.shrbd/rbd_mirror_helpers.shrbd/test_rbdmap_RBDMAPFILE.shrbd/verify_pool.shrbd/cli_generic.shrbd/image_read.shrbd/rbd_mirror_fsx_compare.shrbd/read-flags.shrbd/map-unmap.shrbd/journal.shrbd/krbd_udev_netns.shrbd/huge-tickets.shrbd/qemu_dynamic_features.shrbd/simple_big.shrbd/krbd_latest_osdmap_on_map.shrbd/krbd_namespaces.shrbd/rbd_groups.shrbd/krbd_udev_symlinks.shrbd/rbd_mirror_stress.shrbd/permissions.shrbd/notify_master.shrbd/rbd_mirror_fsx_prepare.shrbd/import_export.shrbd/issue-20295.shrbd/map-snapshot-io.shrbd/concurrent.shrbd/qos.shrbd/krbd_huge_osdmap.shrbd/krbd_exclusive_option.shrbd/diff.shrbd/krbd_udev_netlink_enobufs.shrbd/rbd-nbd.shrbd/krbd_data_pool.shrbd/krbd_fallocate.shrbd/kernel.shrbd/rbd_mirror_journal.shrbd/rbd_mirror_snapshot.shkernel_untar_build.shmgr/test_localpool.shhadoop/terasort.shhadoop/repl.shhadoop/wordcount.shrestart/test-backtraces.pyfs/misc/mkpool_layout_vxattrs.shfs/misc/rstats.shfs/misc/subvolume.shfs/misc/filelock_interrupt.pyfs/misc/layout_vxattrs.shfs/misc/direct_io.pyfs/misc/filelock_deadlock.pyfs/misc/chmod.shfs/misc/multiple_rsync.shfs/misc/acl.shfs/misc/xattrs.shfs/misc/i_complete_vs_rename.shfs/misc/dirfrag.shfs/misc/trivial_sync.shfs/test_o_trunc.shfs/upgrade/volume_clientfs/test_python.shfs/multiclient_sync_read_eof.pyfs/quota/quota.shfs/snap-hierarchy.shfs/test_o_truncfs/norstats/kernel_untar_tar.shfs/snaps/snaptest-2.shfs/snaps/snaptest-xattrwb.shfs/snaps/snaptest-double-null.shfs/snaps/snaptest-0.shfs/snaps/snaptest-capwb.shfs/snaps/snaptest-git-ceph.shfs/snaps/untar_snap_rm.shfs/snaps/snaptest-intodir.shfs/snaps/snaptest-1.shfs/snaps/snaptest-dir-rename.shfs/snaps/snaptest-estale.shfs/snaps/snaptest-multiple-capsnaps.shfs/snaps/snaptest-upchildrealms.shfs/snaps/snaptest-parents.shfs/snaps/snaptest-snap-rename.shfs/snaps/snaptest-authwb.shfs/snaps/snaptest-realm-split.shfs/snaps/snap-rm-diff.shfs/snaps/snaptest-snap-rm-cmp.shfs/snaps/snaptest-hardlink.shobjectstore/test_fuse.shcephadm/test_adoption.shcephadm/test_dashboard_e2e.shcephadm/test_repos.shcephadm/create_iscsi_disks.shcephadm/test_cephadm.shcaps/mon_commands.shrest/test-restful.shrest/test_mgr_rest_api.pydirect_io/test_sync_iodirect_io/big.shdirect_io/test_short_dio_readdirect_io/misc.shdirect_io/direct_io_testrename/pri_nul.shrename/prepare.shrename/rem_nul.shrename/rem_pri.shrename/pri_rem.shrename/rem_rem.shrename/pri_pri.shrename/dir_pri_nul.shrename/dir_pri_pri.shrename/all.shrados/test_dedup_tool.shrados/test_large_omap_detection.pyrados/test_pool_access.shrados/test_rados_timeouts.shrados/stress_watch.shrados/clone.shrados/test_envlibrados_for_rocksdb.shrados/load-gen-mix.shrados/test.shrados/test_python.shrados/test_librados_build.shrados/test_hang.shrados/test_health_warnings.shrados/test_crash.shrados/load-gen-mix-small-long.shrados/test_cache_pool.shrados/test_alloc_hint.shrados/load-gen-big.shrados/load-gen-mostlyread.shrados/test_rados_tool.shrados/load-gen-mix-small.shrados/test_pool_quota.shrados/version_number_sanity.sh 2020-11-20T11:33:36.893 INFO:teuthology.orchestra.run.smithi023.stderr:9+1 records in 2020-11-20T11:33:36.894 INFO:teuthology.orchestra.run.smithi023.stderr:9+1 records out 2020-11-20T11:33:36.894 INFO:teuthology.orchestra.run.smithi023.stderr:4744 bytes (4.7 kB, 4.6 KiB) copied, 4.5798e-05 s, 104 MB/s 2020-11-20T11:33:36.895 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2020-11-20T11:33:36.897 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2020-11-20T11:33:36.898 INFO:teuthology.orchestra.run.smithi023:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b86e691bf191566922a8f08e2a5abf64fde8cfcb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2020-11-20T11:33:51.027 INFO:tasks.mon_thrash:Sending CONT to mon a 2020-11-20T11:33:51.028 INFO:tasks.ceph.mon.a:Sent signal 18 2020-11-20T11:33:51.028 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2020-11-20T11:33:51.028 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:33:51.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.153+0000 7fc35712f700 1 Processor -- start 2020-11-20T11:33:51.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.153+0000 7fc35712f700 1 -- start start 2020-11-20T11:33:51.157 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.153+0000 7fc35712f700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc3500ad600 0x7fc3500658b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:51.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.153+0000 7fc35712f700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3500ad9e0 0x7fc35013c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:51.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.153+0000 7fc35712f700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc35013cd00 0x7fc3500a6b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:51.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.153+0000 7fc35712f700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3500a8b90 con 0x7fc3500ad600 2020-11-20T11:33:51.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.153+0000 7fc35712f700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc350065da0 con 0x7fc35013cd00 2020-11-20T11:33:51.159 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.153+0000 7fc35712f700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3500a83b0 con 0x7fc3500ad9e0 2020-11-20T11:33:51.159 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.154+0000 7fc354ecb700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc3500ad600 0x7fc3500658b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:51.159 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.154+0000 7fc3556cc700 1 -- >> v2:172.21.15.155:3300/0 conn(0x7fc35013cd00 msgr2=0x7fc3500a6b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:33:51.159 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.154+0000 7fc354ecb700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc3500ad600 0x7fc3500658b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41960/0 (socket says 172.21.15.23:41960) 2020-11-20T11:33:51.160 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.154+0000 7fc354ecb700 1 -- 172.21.15.23:0/955364763 learned_addr learned my addr 172.21.15.23:0/955364763 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:33:51.160 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.154+0000 7fc3556cc700 1 --2- 172.21.15.23:0/955364763 >> v2:172.21.15.155:3300/0 conn(0x7fc35013cd00 0x7fc3500a6b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:33:51.160 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.154+0000 7fc354ecb700 1 -- 172.21.15.23:0/955364763 >> v2:172.21.15.23:3300/0 conn(0x7fc3500ad600 msgr2=0x7fc3500658b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:33:51.160 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.154+0000 7fc354ecb700 1 -- 172.21.15.23:0/955364763 >> v2:172.21.15.23:3300/0 conn(0x7fc3500ad600 msgr2=0x7fc3500658b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:33:51.160 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.154+0000 7fc34ffff700 1 --2- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3500ad9e0 0x7fc35013c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:51.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.154+0000 7fc354ecb700 1 --2- 172.21.15.23:0/955364763 >> v2:172.21.15.23:3300/0 conn(0x7fc3500ad600 0x7fc3500658b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:33:51.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.154+0000 7fc354ecb700 1 --2- 172.21.15.23:0/955364763 >> v2:172.21.15.23:3300/0 conn(0x7fc3500ad600 0x7fc3500658b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:33:51.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.154+0000 7fc34ffff700 1 -- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3500ad9e0 msgr2=0x7fc35013c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:33:51.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.154+0000 7fc34ffff700 1 -- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3500ad9e0 msgr2=0x7fc35013c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:33:51.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.154+0000 7fc34ffff700 1 --2- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3500ad9e0 0x7fc35013c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:33:51.162 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.154+0000 7fc34ffff700 1 --2- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3500ad9e0 0x7fc35013c810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:33:51.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.355+0000 7fc3556cc700 1 -- 172.21.15.23:0/955364763 >> v2:172.21.15.155:3300/0 conn(0x7fc35013cd00 msgr2=0x7fc3500a6b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:33:51.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.355+0000 7fc3556cc700 1 --2- 172.21.15.23:0/955364763 >> v2:172.21.15.155:3300/0 conn(0x7fc35013cd00 0x7fc3500a6b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:33:51.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.355+0000 7fc354ecb700 1 --2- 172.21.15.23:0/955364763 >> v2:172.21.15.23:3300/0 conn(0x7fc3500ad600 0x7fc3500658b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:51.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.355+0000 7fc354ecb700 1 -- 172.21.15.23:0/955364763 >> v2:172.21.15.23:3300/0 conn(0x7fc3500ad600 msgr2=0x7fc3500658b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:33:51.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.355+0000 7fc354ecb700 1 -- 172.21.15.23:0/955364763 >> v2:172.21.15.23:3300/0 conn(0x7fc3500ad600 msgr2=0x7fc3500658b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:33:51.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.355+0000 7fc354ecb700 1 --2- 172.21.15.23:0/955364763 >> v2:172.21.15.23:3300/0 conn(0x7fc3500ad600 0x7fc3500658b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:33:51.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.355+0000 7fc354ecb700 1 --2- 172.21.15.23:0/955364763 >> v2:172.21.15.23:3300/0 conn(0x7fc3500ad600 0x7fc3500658b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:33:51.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.355+0000 7fc34ffff700 1 --2- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3500ad9e0 0x7fc35013c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:51.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.355+0000 7fc34ffff700 1 -- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3500ad9e0 msgr2=0x7fc35013c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:33:51.360 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.355+0000 7fc34ffff700 1 -- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3500ad9e0 msgr2=0x7fc35013c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:33:51.360 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.355+0000 7fc34ffff700 1 --2- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3500ad9e0 0x7fc35013c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:33:51.360 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.355+0000 7fc34ffff700 1 --2- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3500ad9e0 0x7fc35013c810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:33:51.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.756+0000 7fc354ecb700 1 --2- 172.21.15.23:0/955364763 >> v2:172.21.15.23:3300/0 conn(0x7fc3500ad600 0x7fc3500658b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:51.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.756+0000 7fc3556cc700 1 -- 172.21.15.23:0/955364763 >> v2:172.21.15.155:3300/0 conn(0x7fc35013cd00 msgr2=0x7fc3500a6b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:33:51.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.756+0000 7fc3556cc700 1 --2- 172.21.15.23:0/955364763 >> v2:172.21.15.155:3300/0 conn(0x7fc35013cd00 0x7fc3500a6b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:33:51.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.756+0000 7fc354ecb700 1 -- 172.21.15.23:0/955364763 >> v2:172.21.15.23:3300/0 conn(0x7fc3500ad600 msgr2=0x7fc3500658b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:33:51.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.756+0000 7fc354ecb700 1 -- 172.21.15.23:0/955364763 >> v2:172.21.15.23:3300/0 conn(0x7fc3500ad600 msgr2=0x7fc3500658b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:33:51.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.756+0000 7fc354ecb700 1 --2- 172.21.15.23:0/955364763 >> v2:172.21.15.23:3300/0 conn(0x7fc3500ad600 0x7fc3500658b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:33:51.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.756+0000 7fc34ffff700 1 --2- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3500ad9e0 0x7fc35013c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:51.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.756+0000 7fc354ecb700 1 --2- 172.21.15.23:0/955364763 >> v2:172.21.15.23:3300/0 conn(0x7fc3500ad600 0x7fc3500658b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:33:51.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.756+0000 7fc34ffff700 1 -- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3500ad9e0 msgr2=0x7fc35013c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:33:51.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.756+0000 7fc34ffff700 1 -- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3500ad9e0 msgr2=0x7fc35013c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:33:51.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.756+0000 7fc34ffff700 1 --2- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3500ad9e0 0x7fc35013c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:33:51.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:51.756+0000 7fc34ffff700 1 --2- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3500ad9e0 0x7fc35013c810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:33:52.560 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:52.557+0000 7fc3556cc700 1 -- 172.21.15.23:0/955364763 >> v2:172.21.15.155:3300/0 conn(0x7fc35013cd00 msgr2=0x7fc3500a6b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:33:52.561 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:52.557+0000 7fc3556cc700 1 --2- 172.21.15.23:0/955364763 >> v2:172.21.15.155:3300/0 conn(0x7fc35013cd00 0x7fc3500a6b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:33:52.561 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:52.557+0000 7fc354ecb700 1 --2- 172.21.15.23:0/955364763 >> v2:172.21.15.23:3300/0 conn(0x7fc3500ad600 0x7fc3500658b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:52.561 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:52.557+0000 7fc354ecb700 1 -- 172.21.15.23:0/955364763 >> v2:172.21.15.23:3300/0 conn(0x7fc3500ad600 msgr2=0x7fc3500658b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:33:52.561 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:52.557+0000 7fc354ecb700 1 -- 172.21.15.23:0/955364763 >> v2:172.21.15.23:3300/0 conn(0x7fc3500ad600 msgr2=0x7fc3500658b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:33:52.561 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:52.557+0000 7fc354ecb700 1 --2- 172.21.15.23:0/955364763 >> v2:172.21.15.23:3300/0 conn(0x7fc3500ad600 0x7fc3500658b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:33:52.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:52.557+0000 7fc354ecb700 1 --2- 172.21.15.23:0/955364763 >> v2:172.21.15.23:3300/0 conn(0x7fc3500ad600 0x7fc3500658b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:33:52.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:52.557+0000 7fc34ffff700 1 --2- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3500ad9e0 0x7fc35013c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:52.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:52.557+0000 7fc34ffff700 1 -- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3500ad9e0 msgr2=0x7fc35013c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:33:52.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:52.557+0000 7fc34ffff700 1 -- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3500ad9e0 msgr2=0x7fc35013c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:33:52.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:52.557+0000 7fc34ffff700 1 --2- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3500ad9e0 0x7fc35013c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:33:52.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:52.558+0000 7fc34ffff700 1 --2- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3500ad9e0 0x7fc35013c810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:33:54.157 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:54.154+0000 7fc34e7fc700 1 -- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3500ad9e0 msgr2=0x7fc35013c810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:33:54.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:54.154+0000 7fc34e7fc700 1 --2- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3500ad9e0 0x7fc35013c810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:54.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:54.154+0000 7fc34e7fc700 1 -- 172.21.15.23:0/955364763 >> v2:172.21.15.155:3300/0 conn(0x7fc35013cd00 msgr2=0x7fc3500a6b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:33:54.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:54.154+0000 7fc34e7fc700 1 --2- 172.21.15.23:0/955364763 >> v2:172.21.15.155:3300/0 conn(0x7fc35013cd00 0x7fc3500a6b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:33:54.159 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:54.154+0000 7fc34e7fc700 1 -- 172.21.15.23:0/955364763 >> v2:172.21.15.23:3300/0 conn(0x7fc3500ad600 msgr2=0x7fc3500658b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:33:54.159 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:54.154+0000 7fc34e7fc700 1 --2- 172.21.15.23:0/955364763 >> v2:172.21.15.23:3300/0 conn(0x7fc3500ad600 0x7fc3500658b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:54.159 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:54.154+0000 7fc34e7fc700 1 -- 172.21.15.23:0/955364763 --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3500a8650 con 0x7fc338007050 2020-11-20T11:33:54.160 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:54.154+0000 7fc34e7fc700 1 -- 172.21.15.23:0/955364763 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc33800bdd0 con 0x7fc3380033e0 2020-11-20T11:33:54.160 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:54.154+0000 7fc354ecb700 1 -- 172.21.15.23:0/955364763 >> v1:172.21.15.155:6789/0 conn(0x7fc3380033e0 legacy=0x7fc338005820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:33:54.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:54.355+0000 7fc354ecb700 1 -- 172.21.15.23:0/955364763 >> v1:172.21.15.155:6789/0 conn(0x7fc3380033e0 legacy=0x7fc338005820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:33:54.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:54.756+0000 7fc354ecb700 1 -- 172.21.15.23:0/955364763 >> v1:172.21.15.155:6789/0 conn(0x7fc3380033e0 legacy=0x7fc338005820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:33:55.560 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:55.557+0000 7fc354ecb700 1 -- 172.21.15.23:0/955364763 >> v1:172.21.15.155:6789/0 conn(0x7fc3380033e0 legacy=0x7fc338005820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:33:56.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.045+0000 7fc34f7fe700 1 -- 172.21.15.23:0/955364763 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2782013218 0 0) 0x7fc3500a8650 con 0x7fc338007050 2020-11-20T11:33:56.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.045+0000 7fc34f7fe700 1 -- 172.21.15.23:0/955364763 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc334003280 con 0x7fc338007050 2020-11-20T11:33:56.050 INFO:tasks.workunit.client.0.smithi023.stderr:--2020-11-20 11:33:56-- http://download.ceph.com/qa/ffsb.tar.bz2 2020-11-20T11:33:56.050 INFO:tasks.workunit.client.0.smithi023.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2020-11-20T11:33:56.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.045+0000 7fc34f7fe700 1 -- 172.21.15.23:0/955364763 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1611730248 0 0) 0x7fc334003280 con 0x7fc338007050 2020-11-20T11:33:56.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.045+0000 7fc34f7fe700 1 -- 172.21.15.23:0/955364763 >> v1:172.21.15.155:6789/0 conn(0x7fc3380033e0 legacy=0x7fc338005820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:33:56.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.045+0000 7fc34f7fe700 1 -- 172.21.15.23:0/955364763 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc33800d030 con 0x7fc338007050 2020-11-20T11:33:56.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.045+0000 7fc34f7fe700 1 -- 172.21.15.23:0/955364763 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7fc33c00d3b0 con 0x7fc338007050 2020-11-20T11:33:56.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.046+0000 7fc34f7fe700 1 -- 172.21.15.23:0/955364763 >> v1:172.21.15.23:6789/0 conn(0x7fc338007050 legacy=0x7fc338009510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:56.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.046+0000 7fc3556cc700 1 -- 172.21.15.23:0/955364763 reap_dead start 2020-11-20T11:33:56.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.046+0000 7fc34f7fe700 1 --2- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3340037a0 0x7fc334003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:56.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.046+0000 7fc34f7fe700 1 --2- 172.21.15.23:0/955364763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc334004070 0x7fc334004540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:56.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.046+0000 7fc34f7fe700 1 --2- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc334004be0 0x7fc33400b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:56.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.046+0000 7fc34f7fe700 1 -- 172.21.15.23:0/955364763 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc334005250 con 0x7fc334004070 2020-11-20T11:33:56.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.046+0000 7fc34f7fe700 1 -- 172.21.15.23:0/955364763 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3340053c0 con 0x7fc3340037a0 2020-11-20T11:33:56.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.046+0000 7fc34f7fe700 1 -- 172.21.15.23:0/955364763 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc33400bcb0 con 0x7fc334004be0 2020-11-20T11:33:56.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.046+0000 7fc34ffff700 1 --2- 172.21.15.23:0/955364763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc334004070 0x7fc334004540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:56.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.046+0000 7fc354ecb700 1 -- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3340037a0 msgr2=0x7fc334003b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:33:56.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.046+0000 7fc354ecb700 1 --2- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3340037a0 0x7fc334003b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:33:56.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.046+0000 7fc3556cc700 1 --2- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc334004be0 0x7fc33400b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:56.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.046+0000 7fc34ffff700 1 -- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc334004be0 msgr2=0x7fc33400b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:33:56.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.046+0000 7fc34ffff700 1 --2- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc334004be0 0x7fc33400b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:56.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.046+0000 7fc34ffff700 1 -- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3340037a0 msgr2=0x7fc334003b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:33:56.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.046+0000 7fc34ffff700 1 --2- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3340037a0 0x7fc334003b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:33:56.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.046+0000 7fc34ffff700 1 -- 172.21.15.23:0/955364763 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc334003280 con 0x7fc334004070 2020-11-20T11:33:56.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.046+0000 7fc34ffff700 1 --2- 172.21.15.23:0/955364763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc334004070 0x7fc334004540 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fc33c006c40 tx=0x7fc33c008040).ready entity=mon.0 client_cookie=280d1fb2cb042d7a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:56.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.048+0000 7fc34f7fe700 1 -- 172.21.15.23:0/955364763 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc33c012ea0 con 0x7fc334004070 2020-11-20T11:33:56.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.048+0000 7fc34f7fe700 1 -- 172.21.15.23:0/955364763 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc33c00d980 con 0x7fc334004070 2020-11-20T11:33:56.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.048+0000 7fc34f7fe700 1 -- 172.21.15.23:0/955364763 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc33c015870 con 0x7fc334004070 2020-11-20T11:33:56.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.049+0000 7fc35712f700 1 -- 172.21.15.23:0/955364763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc334004070 msgr2=0x7fc334004540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:56.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.049+0000 7fc35712f700 1 --2- 172.21.15.23:0/955364763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc334004070 0x7fc334004540 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fc33c006c40 tx=0x7fc33c008040).stop 2020-11-20T11:33:56.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.049+0000 7fc35712f700 1 -- 172.21.15.23:0/955364763 shutdown_connections 2020-11-20T11:33:56.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.049+0000 7fc35712f700 1 --2- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc334004be0 0x7fc33400b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:56.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.049+0000 7fc35712f700 1 --2- 172.21.15.23:0/955364763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc334004070 0x7fc334004540 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:56.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.049+0000 7fc35712f700 1 --2- 172.21.15.23:0/955364763 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3340037a0 0x7fc334003b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:33:56.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.049+0000 7fc35712f700 1 -- 172.21.15.23:0/955364763 >> 172.21.15.23:0/955364763 conn(0x7fc350060c80 msgr2=0x7fc3500acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:33:56.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.049+0000 7fc35712f700 1 -- 172.21.15.23:0/955364763 shutdown_connections 2020-11-20T11:33:56.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.049+0000 7fc35712f700 1 -- 172.21.15.23:0/955364763 wait complete. 2020-11-20T11:33:56.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.050+0000 7fc35712f700 1 Processor -- start 2020-11-20T11:33:56.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.051+0000 7fc35712f700 1 -- start start 2020-11-20T11:33:56.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.051+0000 7fc35712f700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3340037a0 0x7fc350066df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:56.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.051+0000 7fc35712f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc334004070 0x7fc3500ac440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:56.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.051+0000 7fc35712f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc334004be0 0x7fc3501321d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:56.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.051+0000 7fc35712f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3500a5360 con 0x7fc334004be0 2020-11-20T11:33:56.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.051+0000 7fc35712f700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3500664d0 con 0x7fc334004070 2020-11-20T11:33:56.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.051+0000 7fc35712f700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3500a5550 con 0x7fc3340037a0 2020-11-20T11:33:56.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.051+0000 7fc34ffff700 1 -- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc334004070 msgr2=0x7fc3500ac440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:33:56.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.052+0000 7fc34ffff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc334004070 0x7fc3500ac440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:33:56.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.052+0000 7fc3556cc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc334004be0 0x7fc3501321d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:56.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.052+0000 7fc3556cc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc334004be0 0x7fc3501321d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42010/0 (socket says 172.21.15.23:42010) 2020-11-20T11:33:56.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.052+0000 7fc3556cc700 1 -- 172.21.15.23:0/2253109605 learned_addr learned my addr 172.21.15.23:0/2253109605 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:33:56.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.052+0000 7fc354ecb700 1 --2- 172.21.15.23:0/2253109605 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3340037a0 0x7fc350066df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:56.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.052+0000 7fc3556cc700 1 -- 172.21.15.23:0/2253109605 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3340037a0 msgr2=0x7fc350066df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:56.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.052+0000 7fc3556cc700 1 --2- 172.21.15.23:0/2253109605 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3340037a0 0x7fc350066df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:56.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.052+0000 7fc3556cc700 1 -- 172.21.15.23:0/2253109605 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc334004070 msgr2=0x7fc3500ac440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:33:56.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.052+0000 7fc3556cc700 1 --2- 172.21.15.23:0/2253109605 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc334004070 0x7fc3500ac440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:33:56.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.052+0000 7fc3556cc700 1 -- 172.21.15.23:0/2253109605 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3500a62d0 con 0x7fc334004be0 2020-11-20T11:33:56.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.052+0000 7fc3556cc700 1 --2- 172.21.15.23:0/2253109605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc334004be0 0x7fc3501321d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fc35013d100 tx=0x7fc348009640).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:56.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.052+0000 7fc34cff9700 1 -- 172.21.15.23:0/2253109605 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc348009e10 con 0x7fc334004be0 2020-11-20T11:33:56.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.052+0000 7fc34cff9700 1 -- 172.21.15.23:0/2253109605 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc34800b600 con 0x7fc334004be0 2020-11-20T11:33:56.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.052+0000 7fc35712f700 1 -- 172.21.15.23:0/2253109605 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3500672e0 con 0x7fc334004be0 2020-11-20T11:33:56.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.052+0000 7fc34cff9700 1 -- 172.21.15.23:0/2253109605 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc34800b9a0 con 0x7fc334004be0 2020-11-20T11:33:56.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.052+0000 7fc35712f700 1 -- 172.21.15.23:0/2253109605 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3501336e0 con 0x7fc334004be0 2020-11-20T11:33:56.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.054+0000 7fc35712f700 1 -- 172.21.15.23:0/2253109605 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3500a5710 con 0x7fc334004be0 2020-11-20T11:33:56.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.054+0000 7fc34cff9700 1 -- 172.21.15.23:0/2253109605 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc34801e070 con 0x7fc334004be0 2020-11-20T11:33:56.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.054+0000 7fc34cff9700 1 --2- 172.21.15.23:0/2253109605 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc328056d10 0x7fc3280591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:56.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.054+0000 7fc34cff9700 1 -- 172.21.15.23:0/2253109605 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fc348070ec0 con 0x7fc334004be0 2020-11-20T11:33:56.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.055+0000 7fc354ecb700 1 --2- 172.21.15.23:0/2253109605 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc328056d10 0x7fc3280591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:56.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.055+0000 7fc354ecb700 1 --2- 172.21.15.23:0/2253109605 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc328056d10 0x7fc3280591d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fc344006b50 tx=0x7fc344008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:56.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.058+0000 7fc34cff9700 1 -- 172.21.15.23:0/2253109605 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc3480461b0 con 0x7fc334004be0 2020-11-20T11:33:56.117 INFO:tasks.workunit.client.0.smithi023.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2020-11-20T11:33:56.178 INFO:tasks.workunit.client.0.smithi023.stderr:HTTP request sent, awaiting response... 200 OK 2020-11-20T11:33:56.179 INFO:tasks.workunit.client.0.smithi023.stderr:Length: 128931 (126K) [application/octet-stream] 2020-11-20T11:33:56.180 INFO:tasks.workunit.client.0.smithi023.stderr:Saving to: ‘ffsb.tar.bz2’ 2020-11-20T11:33:56.184 INFO:tasks.workunit.client.0.smithi023.stderr: 2020-11-20T11:33:56.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.269+0000 7fc35712f700 1 -- 172.21.15.23:0/2253109605 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc35005c520 con 0x7fc334004be0 2020-11-20T11:33:56.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.269+0000 7fc34cff9700 1 -- 172.21.15.23:0/2253109605 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1471 (secure 0 0 0) 0x7fc3480497d0 con 0x7fc334004be0 2020-11-20T11:33:56.273 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":14,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:33:56.274 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.271+0000 7fc35712f700 1 -- 172.21.15.23:0/2253109605 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc328056d10 msgr2=0x7fc3280591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:56.275 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.271+0000 7fc35712f700 1 --2- 172.21.15.23:0/2253109605 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc328056d10 0x7fc3280591d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fc344006b50 tx=0x7fc344008040).stop 2020-11-20T11:33:56.275 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.271+0000 7fc35712f700 1 -- 172.21.15.23:0/2253109605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc334004be0 msgr2=0x7fc3501321d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:56.275 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.271+0000 7fc35712f700 1 --2- 172.21.15.23:0/2253109605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc334004be0 0x7fc3501321d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fc35013d100 tx=0x7fc348009640).stop 2020-11-20T11:33:56.275 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.271+0000 7fc35712f700 1 -- 172.21.15.23:0/2253109605 shutdown_connections 2020-11-20T11:33:56.275 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.271+0000 7fc35712f700 1 --2- 172.21.15.23:0/2253109605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc334004be0 0x7fc3501321d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:56.276 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.271+0000 7fc35712f700 1 --2- 172.21.15.23:0/2253109605 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc328056d10 0x7fc3280591d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:56.276 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.271+0000 7fc35712f700 1 --2- 172.21.15.23:0/2253109605 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc3340037a0 0x7fc350066df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:56.276 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.271+0000 7fc35712f700 1 --2- 172.21.15.23:0/2253109605 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc334004070 0x7fc3500ac440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:33:56.276 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.271+0000 7fc35712f700 1 -- 172.21.15.23:0/2253109605 >> 172.21.15.23:0/2253109605 conn(0x7fc350060c80 msgr2=0x7fc3500aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:33:56.276 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.271+0000 7fc35712f700 1 -- 172.21.15.23:0/2253109605 shutdown_connections 2020-11-20T11:33:56.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.271+0000 7fc35712f700 1 -- 172.21.15.23:0/2253109605 wait complete. 2020-11-20T11:33:56.286 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2020-11-20T11:33:56.286 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:33:56.315 INFO:tasks.workunit.client.0.smithi023.stderr: 0K .......... .......... .......... .......... .......... 39% 371K 0s 2020-11-20T11:33:56.385 INFO:tasks.workunit.client.0.smithi023.stderr: 50K .......... .......... .......... .......... .......... 79% 715K 0s 2020-11-20T11:33:56.402 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.399+0000 7f8af4713700 1 Processor -- start 2020-11-20T11:33:56.402 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.399+0000 7f8af4713700 1 -- start start 2020-11-20T11:33:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.400+0000 7f8af4713700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8aec0a8df0 0x7f8aec138a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.400+0000 7f8af4713700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f8aec0a6ee0 0x7f8aec0a83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.400+0000 7f8af4713700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8aec065d10 0x7f8aec0a37a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.400+0000 7f8af4713700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aec0a7780 con 0x7f8aec0a8df0 2020-11-20T11:33:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.400+0000 7f8af4713700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aec0a7d20 con 0x7f8aec0a6ee0 2020-11-20T11:33:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.400+0000 7f8af4713700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec066190 con 0x7f8aec065d10 2020-11-20T11:33:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.400+0000 7f8af24af700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8aec0a8df0 0x7f8aec138a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.400+0000 7f8af1cae700 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f8aec0a6ee0 msgr2=0x7f8aec0a83f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:33:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.400+0000 7f8af24af700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8aec0a8df0 0x7f8aec138a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42016/0 (socket says 172.21.15.23:42016) 2020-11-20T11:33:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.400+0000 7f8af1cae700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f8aec0a6ee0 0x7f8aec0a83f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:33:56.408 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.400+0000 7f8af24af700 1 -- 172.21.15.23:0/1627876876 learned_addr learned my addr 172.21.15.23:0/1627876876 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:33:56.408 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.400+0000 7f8af2cb0700 1 --2- 172.21.15.23:0/1627876876 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8aec065d10 0x7f8aec0a37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:56.408 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.400+0000 7f8af24af700 1 -- 172.21.15.23:0/1627876876 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8aec065d10 msgr2=0x7f8aec0a37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:33:56.408 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.400+0000 7f8af24af700 1 --2- 172.21.15.23:0/1627876876 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8aec065d10 0x7f8aec0a37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:56.409 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.400+0000 7f8af24af700 1 -- 172.21.15.23:0/1627876876 >> v2:172.21.15.155:3300/0 conn(0x7f8aec0a6ee0 msgr2=0x7f8aec0a83f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:33:56.409 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.400+0000 7f8af24af700 1 --2- 172.21.15.23:0/1627876876 >> v2:172.21.15.155:3300/0 conn(0x7f8aec0a6ee0 0x7f8aec0a83f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:33:56.409 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.400+0000 7f8af24af700 1 -- 172.21.15.23:0/1627876876 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aec066420 con 0x7f8aec0a8df0 2020-11-20T11:33:56.409 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.400+0000 7f8af24af700 1 --2- 172.21.15.23:0/1627876876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec0a8df0 0x7f8aec138a40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f8aec061740 tx=0x7f8adc006ef0).ready entity=mon.0 client_cookie=ebbe859223701627 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:56.410 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.401+0000 7f8af14ad700 1 -- 172.21.15.23:0/1627876876 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8adc00b860 con 0x7f8aec0a8df0 2020-11-20T11:33:56.410 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.401+0000 7f8af14ad700 1 -- 172.21.15.23:0/1627876876 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8adc000e90 con 0x7f8aec0a8df0 2020-11-20T11:33:56.410 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.401+0000 7f8af14ad700 1 -- 172.21.15.23:0/1627876876 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8adc00bbd0 con 0x7f8aec0a8df0 2020-11-20T11:33:56.410 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.401+0000 7f8af4713700 1 -- 172.21.15.23:0/1627876876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec0a8df0 msgr2=0x7f8aec138a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:56.410 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.401+0000 7f8af4713700 1 --2- 172.21.15.23:0/1627876876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec0a8df0 0x7f8aec138a40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f8aec061740 tx=0x7f8adc006ef0).stop 2020-11-20T11:33:56.411 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.401+0000 7f8af4713700 1 -- 172.21.15.23:0/1627876876 shutdown_connections 2020-11-20T11:33:56.411 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.401+0000 7f8af4713700 1 --2- 172.21.15.23:0/1627876876 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8aec065d10 0x7f8aec0a37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:56.411 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.401+0000 7f8af4713700 1 --2- 172.21.15.23:0/1627876876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec0a8df0 0x7f8aec138a40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:56.411 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.401+0000 7f8af4713700 1 --2- 172.21.15.23:0/1627876876 >> v2:172.21.15.155:3300/0 conn(0x7f8aec0a6ee0 0x7f8aec0a83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:33:56.411 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.401+0000 7f8af4713700 1 -- 172.21.15.23:0/1627876876 >> 172.21.15.23:0/1627876876 conn(0x7f8aec0601b0 msgr2=0x7f8aec0605c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:33:56.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.401+0000 7f8af4713700 1 -- 172.21.15.23:0/1627876876 shutdown_connections 2020-11-20T11:33:56.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.401+0000 7f8af4713700 1 -- 172.21.15.23:0/1627876876 wait complete. 2020-11-20T11:33:56.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.402+0000 7f8af4713700 1 Processor -- start 2020-11-20T11:33:56.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.402+0000 7f8af4713700 1 -- start start 2020-11-20T11:33:56.417 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.403+0000 7f8af4713700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec065d10 0x7f8aec0a2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:56.418 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.403+0000 7f8af4713700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8aec0a6ee0 0x7f8aec0a1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:56.418 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.403+0000 7f8af4713700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8aec0a8df0 0x7f8aec14d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:56.418 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.403+0000 7f8af4713700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec0a2820 con 0x7f8aec065d10 2020-11-20T11:33:56.418 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.403+0000 7f8af4713700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec0a22e0 con 0x7f8aec0a6ee0 2020-11-20T11:33:56.418 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.403+0000 7f8af4713700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec0a2580 con 0x7f8aec0a8df0 2020-11-20T11:33:56.419 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.403+0000 7f8af24af700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec065d10 0x7f8aec0a2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:56.419 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.403+0000 7f8af1cae700 1 -- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8aec0a6ee0 msgr2=0x7f8aec0a1960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:33:56.419 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.403+0000 7f8af24af700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec065d10 0x7f8aec0a2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42022/0 (socket says 172.21.15.23:42022) 2020-11-20T11:33:56.419 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.403+0000 7f8af24af700 1 -- 172.21.15.23:0/457913831 learned_addr learned my addr 172.21.15.23:0/457913831 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:33:56.419 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.403+0000 7f8af1cae700 1 --2- 172.21.15.23:0/457913831 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8aec0a6ee0 0x7f8aec0a1960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:33:56.420 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.403+0000 7f8af2cb0700 1 --2- 172.21.15.23:0/457913831 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8aec0a8df0 0x7f8aec14d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:56.420 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.403+0000 7f8af24af700 1 -- 172.21.15.23:0/457913831 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8aec0a8df0 msgr2=0x7f8aec14d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:56.420 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.403+0000 7f8af24af700 1 --2- 172.21.15.23:0/457913831 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8aec0a8df0 0x7f8aec14d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:56.420 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.403+0000 7f8af24af700 1 -- 172.21.15.23:0/457913831 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8aec0a6ee0 msgr2=0x7f8aec0a1960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:33:56.420 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.403+0000 7f8af24af700 1 --2- 172.21.15.23:0/457913831 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8aec0a6ee0 0x7f8aec0a1960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:33:56.421 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.403+0000 7f8af24af700 1 -- 172.21.15.23:0/457913831 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aec0a34b0 con 0x7f8aec065d10 2020-11-20T11:33:56.421 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.403+0000 7f8af24af700 1 --2- 172.21.15.23:0/457913831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec065d10 0x7f8aec0a2dd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f8adc00b3c0 tx=0x7f8adc015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:56.421 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.403+0000 7f8ada7fc700 1 -- 172.21.15.23:0/457913831 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8adc0119b0 con 0x7f8aec065d10 2020-11-20T11:33:56.422 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.403+0000 7f8ada7fc700 1 -- 172.21.15.23:0/457913831 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8adc011b10 con 0x7f8aec065d10 2020-11-20T11:33:56.422 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.403+0000 7f8ada7fc700 1 -- 172.21.15.23:0/457913831 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8adc011e90 con 0x7f8aec065d10 2020-11-20T11:33:56.423 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.403+0000 7f8af4713700 1 -- 172.21.15.23:0/457913831 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aec0a1e50 con 0x7f8aec065d10 2020-11-20T11:33:56.423 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.403+0000 7f8af4713700 1 -- 172.21.15.23:0/457913831 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aec14dee0 con 0x7f8aec065d10 2020-11-20T11:33:56.423 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.404+0000 7f8af4713700 1 --2- 172.21.15.23:0/457913831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec13be70 0x7f8aec13c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:56.424 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.404+0000 7f8af4713700 1 -- 172.21.15.23:0/457913831 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec14d4f0 con 0x7f8aec13be70 2020-11-20T11:33:56.424 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.404+0000 7f8af1cae700 1 --2- 172.21.15.23:0/457913831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec13be70 0x7f8aec13c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:56.424 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.404+0000 7f8ada7fc700 1 -- 172.21.15.23:0/457913831 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8adc024030 con 0x7f8aec065d10 2020-11-20T11:33:56.425 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.405+0000 7f8af1cae700 1 -- 172.21.15.23:0/457913831 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8aec062c40 con 0x7f8aec13be70 2020-11-20T11:33:56.425 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.405+0000 7f8ada7fc700 1 --2- 172.21.15.23:0/457913831 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8ac4056d40 0x7f8ac4059200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:56.426 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.405+0000 7f8af1cae700 1 --2- 172.21.15.23:0/457913831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec13be70 0x7f8aec13c250 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f8aec0a9720 tx=0x7f8ae8006ea0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:56.426 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.405+0000 7f8ada7fc700 1 -- 172.21.15.23:0/457913831 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f8adc071040 con 0x7f8aec065d10 2020-11-20T11:33:56.426 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.405+0000 7f8ada7fc700 1 -- 172.21.15.23:0/457913831 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae8005320 con 0x7f8aec13be70 2020-11-20T11:33:56.427 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.405+0000 7f8af2cb0700 1 --2- 172.21.15.23:0/457913831 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8ac4056d40 0x7f8ac4059200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:56.427 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.405+0000 7f8ada7fc700 1 -- 172.21.15.23:0/457913831 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f8ae8005480 con 0x7f8aec13be70 2020-11-20T11:33:56.428 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.405+0000 7f8ada7fc700 1 -- 172.21.15.23:0/457913831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec13be70 msgr2=0x7f8aec13c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:56.428 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.405+0000 7f8ada7fc700 1 --2- 172.21.15.23:0/457913831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec13be70 0x7f8aec13c250 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f8aec0a9720 tx=0x7f8ae8006ea0).stop 2020-11-20T11:33:56.428 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.405+0000 7f8ada7fc700 1 -- 172.21.15.23:0/457913831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec13be70 msgr2=0x7f8aec13c250 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:33:56.429 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.405+0000 7f8ada7fc700 1 --2- 172.21.15.23:0/457913831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec13be70 0x7f8aec13c250 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:56.429 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.405+0000 7f8af2cb0700 1 --2- 172.21.15.23:0/457913831 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8ac4056d40 0x7f8ac4059200 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f8ae4009bc0 tx=0x7f8ae4008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:56.430 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.412+0000 7f8af4713700 1 --2- 172.21.15.23:0/457913831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec149f10 0x7f8aec14a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:56.430 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.412+0000 7f8af4713700 1 -- 172.21.15.23:0/457913831 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aec14e100 con 0x7f8aec149f10 2020-11-20T11:33:56.430 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.412+0000 7f8af1cae700 1 --2- 172.21.15.23:0/457913831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec149f10 0x7f8aec14a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:56.431 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.412+0000 7f8af1cae700 1 -- 172.21.15.23:0/457913831 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8aec0a20c0 con 0x7f8aec149f10 2020-11-20T11:33:56.431 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.412+0000 7f8af1cae700 1 --2- 172.21.15.23:0/457913831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec149f10 0x7f8aec14a320 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f8aec0a9720 tx=0x7f8ae8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:56.431 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.412+0000 7f8ada7fc700 1 -- 172.21.15.23:0/457913831 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae801b040 con 0x7f8aec149f10 2020-11-20T11:33:56.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.413+0000 7f8ada7fc700 1 -- 172.21.15.23:0/457913831 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4604 (secure 0 0 0) 0x7f8ae8012e00 con 0x7f8aec149f10 2020-11-20T11:33:56.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.413+0000 7f8ada7fc700 1 -- 172.21.15.23:0/457913831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec149f10 msgr2=0x7f8aec14a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:56.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.413+0000 7f8ada7fc700 1 --2- 172.21.15.23:0/457913831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec149f10 0x7f8aec14a320 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f8aec0a9720 tx=0x7f8ae8015000).stop 2020-11-20T11:33:56.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.413+0000 7f8ada7fc700 1 -- 172.21.15.23:0/457913831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec149f10 msgr2=0x7f8aec14a320 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:33:56.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.413+0000 7f8ada7fc700 1 --2- 172.21.15.23:0/457913831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec149f10 0x7f8aec14a320 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0x7f8aec0a9720 tx=0x7f8ae8015000).stop 2020-11-20T11:33:56.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.413+0000 7f8af4713700 1 -- 172.21.15.23:0/457913831 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8ac4056d40 msgr2=0x7f8ac4059200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:56.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.413+0000 7f8af4713700 1 --2- 172.21.15.23:0/457913831 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8ac4056d40 0x7f8ac4059200 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f8ae4009bc0 tx=0x7f8ae4008040).stop 2020-11-20T11:33:56.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.413+0000 7f8af4713700 1 -- 172.21.15.23:0/457913831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec065d10 msgr2=0x7f8aec0a2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:56.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.413+0000 7f8af4713700 1 --2- 172.21.15.23:0/457913831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec065d10 0x7f8aec0a2dd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f8adc00b3c0 tx=0x7f8adc015000).stop 2020-11-20T11:33:56.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.413+0000 7f8af2cb0700 1 -- 172.21.15.23:0/457913831 reap_dead start 2020-11-20T11:33:56.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.413+0000 7f8af4713700 1 -- 172.21.15.23:0/457913831 shutdown_connections 2020-11-20T11:33:56.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.413+0000 7f8af4713700 1 --2- 172.21.15.23:0/457913831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8aec065d10 0x7f8aec0a2dd0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:56.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.413+0000 7f8af4713700 1 -- 172.21.15.23:0/457913831 >> 172.21.15.23:0/457913831 conn(0x7f8aec0601b0 msgr2=0x7f8aec0aa620 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:33:56.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.413+0000 7f8af4713700 1 -- 172.21.15.23:0/457913831 shutdown_connections 2020-11-20T11:33:56.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.413+0000 7f8af4713700 1 -- 172.21.15.23:0/457913831 wait complete. 2020-11-20T11:33:56.436 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:33:56.437 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:33:56.437 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:33:56.437 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:33:56.437 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 14, 2020-11-20T11:33:56.437 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:33:56.438 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:33:56.438 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:33:56.438 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:56.438 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 0, 2020-11-20T11:33:56.438 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:33:56.439 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:33:56.439 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:33:56.439 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:33:56.439 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:33:56.439 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:33:56.440 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:33:56.440 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:33:56.440 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:33:56.440 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:33:56.440 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:33:56.440 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:56.441 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:33:56.441 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:33:56.441 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:33:56.441 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:33:56.441 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:33:56.442 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:33:56.442 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:33:56.442 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:33:56.442 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:33:56.442 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:33:56.443 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:56.443 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:56.443 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:33:56.443 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:33:56.443 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:33:56.444 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:33:56.444 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:33:56.444 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:33:56.444 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:33:56.444 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:33:56.445 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:33:56.445 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:33:56.445 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:33:56.445 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:33:56.445 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:33:56.446 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:33:56.446 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:33:56.446 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:33:56.446 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:33:56.446 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:33:56.447 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:33:56.447 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:33:56.447 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:33:56.447 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:33:56.447 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:33:56.447 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:56.448 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:33:56.448 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:56.448 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:33:56.448 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:56.448 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:33:56.449 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:33:56.449 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:33:56.449 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:33:56.449 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:56.449 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:33:56.450 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:33:56.450 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:56.450 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:56.450 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:56.450 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:33:56.451 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:33:56.451 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:56.451 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:56.451 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:56.451 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:56.452 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:33:56.452 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:33:56.452 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:33:56.452 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:33:56.452 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:33:56.453 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:56.453 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:56.453 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:33:56.453 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:33:56.453 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:33:56.454 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:33:56.454 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:56.454 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:33:56.454 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:33:56.455 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:56.455 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:56.455 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:56.455 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:33:56.455 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:33:56.456 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:56.456 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:56.456 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:56.456 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:56.456 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:33:56.457 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:33:56.457 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:33:56.457 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:33:56.457 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:33:56.457 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:56.458 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:56.458 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:33:56.458 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:33:56.458 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:33:56.458 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:33:56.459 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:56.459 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:33:56.459 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:33:56.459 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:56.459 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:56.460 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:56.460 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:33:56.460 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:33:56.460 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:56.460 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:56.461 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:56.461 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:56.461 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:33:56.461 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:33:56.461 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:33:56.462 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:33:56.462 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:33:56.462 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:56.462 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:56.462 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:56.463 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:33:56.463 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:33:56.463 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:56.463 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:56.463 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:56.464 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:33:56.464 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:56.464 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:56.464 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:33:56.464 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:56.465 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:56.465 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:56.465 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:33:56.465 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:56.465 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:56.466 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:33:56.466 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:56.466 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:56.466 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:56.466 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 4 2020-11-20T11:33:56.466 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:56.467 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:56.467 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:33:56.467 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:56.467 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:56.467 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:56.468 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:33:56.468 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:56.468 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:56.468 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:56.468 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:33:56.469 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:33:56.470 INFO:tasks.workunit.client.0.smithi023.stderr: 100K .......... .......... ..... 100% 362K=0.3s 2020-11-20T11:33:56.471 INFO:tasks.workunit.client.0.smithi023.stderr: 2020-11-20T11:33:56.471 INFO:tasks.workunit.client.0.smithi023.stderr:2020-11-20 11:33:56 (456 KB/s) - ‘ffsb.tar.bz2’ saved [128931/128931] 2020-11-20T11:33:56.471 INFO:tasks.workunit.client.0.smithi023.stderr: 2020-11-20T11:33:56.472 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:33:56.493 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/ 2020-11-20T11:33:56.493 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/ffsb_fs.c 2020-11-20T11:33:56.495 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/INSTALL 2020-11-20T11:33:56.497 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/fileops.c 2020-11-20T11:33:56.500 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/missing 2020-11-20T11:33:56.503 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/config.h.in 2020-11-20T11:33:56.507 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/config.guess 2020-11-20T11:33:56.510 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/fh.h 2020-11-20T11:33:56.512 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/COPYING 2020-11-20T11:33:56.515 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/util.h 2020-11-20T11:33:56.518 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/main.c 2020-11-20T11:33:56.520 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/ChangeLog 2020-11-20T11:33:56.523 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/rand.c 2020-11-20T11:33:56.525 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/ffsb_fs.h 2020-11-20T11:33:56.528 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/install-sh 2020-11-20T11:33:56.530 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/rwlock.h 2020-11-20T11:33:56.532 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/parser.h 2020-11-20T11:33:56.535 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/filelist.h 2020-11-20T11:33:56.538 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/LICENSE 2020-11-20T11:33:56.541 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/Makefile.in 2020-11-20T11:33:56.544 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/filelist.c 2020-11-20T11:33:56.546 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/configure 2020-11-20T11:33:56.553 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/README 2020-11-20T11:33:56.555 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/cirlist.c 2020-11-20T11:33:56.557 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/metaops.c 2020-11-20T11:33:56.561 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/registration-description 2020-11-20T11:33:56.563 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/list.c 2020-11-20T11:33:56.566 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/list.h 2020-11-20T11:33:56.569 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/rand.h 2020-11-20T11:33:56.572 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/parser.c 2020-11-20T11:33:56.575 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/ffsb_tg.c 2020-11-20T11:33:56.577 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/cirlist.h 2020-11-20T11:33:56.580 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/AUTHORS 2020-11-20T11:33:56.582 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/aclocal.m4 2020-11-20T11:33:56.584 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/configure.in 2020-11-20T11:33:56.586 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/examples/ 2020-11-20T11:33:56.587 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/examples/profile_everything 2020-11-20T11:33:56.590 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/depcomp 2020-11-20T11:33:56.593 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/ffsb_stats.c 2020-11-20T11:33:56.595 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/util.c 2020-11-20T11:33:56.600 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/Makefile.am 2020-11-20T11:33:56.603 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/ffsb_thread.c 2020-11-20T11:33:56.606 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/rbt.c 2020-11-20T11:33:56.609 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/rbt.h 2020-11-20T11:33:56.612 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/ffsb_tg.h 2020-11-20T11:33:56.614 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/stamp-h.in 2020-11-20T11:33:56.616 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/config.sub 2020-11-20T11:33:56.619 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/ffsb_thread.h 2020-11-20T11:33:56.621 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/fileops.h 2020-11-20T11:33:56.624 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/ffsb_op.h 2020-11-20T11:33:56.626 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/public-description 2020-11-20T11:33:56.629 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/mkinstalldirs 2020-11-20T11:33:56.630 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/ffsb.h 2020-11-20T11:33:56.633 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/metaops.h 2020-11-20T11:33:56.636 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/ffsb_op.c 2020-11-20T11:33:56.638 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/fh.c 2020-11-20T11:33:56.641 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/NEWS 2020-11-20T11:33:56.644 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/rwlock.c 2020-11-20T11:33:56.646 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/ffsb_stats.h 2020-11-20T11:33:56.647 INFO:tasks.workunit.client.0.smithi023.stdout:ffsb-6.0-rc2/ffsb_fc.c 2020-11-20T11:33:56.661 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.658+0000 7feee2656700 1 Processor -- start 2020-11-20T11:33:56.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.658+0000 7feee2656700 1 -- start start 2020-11-20T11:33:56.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.659+0000 7feee2656700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7feedc0ad600 0x7feedc13af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:56.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.659+0000 7feee2656700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feedc13be80 con 0x7feedc0ad9e0 2020-11-20T11:33:56.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.659+0000 7feee2656700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feedc0a6f10 con 0x7feedc0a6a70 2020-11-20T11:33:56.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.659+0000 7feee2656700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feedc0a71b0 con 0x7feedc0ad600 2020-11-20T11:33:56.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.659+0000 7feedbfff700 1 -- >> v2:172.21.15.155:3300/0 conn(0x7feedc0ad600 msgr2=0x7feedc13af90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:33:56.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.659+0000 7feee0bf3700 1 -- >> v1:172.21.15.155:6789/0 conn(0x7feedc0a6a70 legacy=0x7feedc13b770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:33:56.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.659+0000 7feedbfff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7feedc0ad600 0x7feedc13af90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:33:56.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.659+0000 7feedb7fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7feedc0ad9e0 0x7feedc0a90a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:56292/0 (socket says 172.21.15.23:56292) 2020-11-20T11:33:56.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.659+0000 7feedb7fe700 1 -- 172.21.15.23:0/413744292 learned_addr learned my addr 172.21.15.23:0/413744292 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:33:56.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.659+0000 7feedaffd700 1 -- 172.21.15.23:0/413744292 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087666199 0 0) 0x7feedc13be80 con 0x7feedc0ad9e0 2020-11-20T11:33:56.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.659+0000 7feedaffd700 1 -- 172.21.15.23:0/413744292 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feebc003280 con 0x7feedc0ad9e0 2020-11-20T11:33:56.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.660+0000 7feedaffd700 1 -- 172.21.15.23:0/413744292 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1359142136 0 0) 0x7feebc003280 con 0x7feedc0ad9e0 2020-11-20T11:33:56.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.660+0000 7feedaffd700 1 -- 172.21.15.23:0/413744292 >> v2:172.21.15.155:3300/0 conn(0x7feedc0ad600 msgr2=0x7feedc13af90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:33:56.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.660+0000 7feedaffd700 1 --2- 172.21.15.23:0/413744292 >> v2:172.21.15.155:3300/0 conn(0x7feedc0ad600 0x7feedc13af90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:33:56.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.660+0000 7feedaffd700 1 -- 172.21.15.23:0/413744292 >> v1:172.21.15.155:6789/0 conn(0x7feedc0a6a70 legacy=0x7feedc13b770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:33:56.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.660+0000 7feedaffd700 1 -- 172.21.15.23:0/413744292 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feedc0a4da0 con 0x7feedc0ad9e0 2020-11-20T11:33:56.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.660+0000 7feedaffd700 1 -- 172.21.15.23:0/413744292 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7feecc001520 con 0x7feedc0ad9e0 2020-11-20T11:33:56.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.660+0000 7feedaffd700 1 -- 172.21.15.23:0/413744292 >> v1:172.21.15.23:6789/0 conn(0x7feedc0ad9e0 legacy=0x7feedc0a90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.660+0000 7feedaffd700 1 --2- 172.21.15.23:0/413744292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feebc003770 0x7feebc003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.660+0000 7feedaffd700 1 --2- 172.21.15.23:0/413744292 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feebc0040d0 0x7feebc0045a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.660+0000 7feedaffd700 1 --2- 172.21.15.23:0/413744292 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feebc004ba0 0x7feebc00b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.660+0000 7feedaffd700 1 -- 172.21.15.23:0/413744292 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feedc0a71b0 con 0x7feebc0040d0 2020-11-20T11:33:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.660+0000 7feedaffd700 1 -- 172.21.15.23:0/413744292 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feebc0051a0 con 0x7feebc003770 2020-11-20T11:33:56.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.660+0000 7feedaffd700 1 -- 172.21.15.23:0/413744292 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feebc005360 con 0x7feebc004ba0 2020-11-20T11:33:56.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.660+0000 7feedb7fe700 1 --2- 172.21.15.23:0/413744292 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feebc0040d0 0x7feebc0045a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:56.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.660+0000 7feedbfff700 1 -- 172.21.15.23:0/413744292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feebc003770 msgr2=0x7feebc003b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:33:56.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.660+0000 7feedbfff700 1 --2- 172.21.15.23:0/413744292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feebc003770 0x7feebc003b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:33:56.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.660+0000 7feee0bf3700 1 --2- 172.21.15.23:0/413744292 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feebc004ba0 0x7feebc00b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:56.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.660+0000 7feedb7fe700 1 -- 172.21.15.23:0/413744292 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feebc004ba0 msgr2=0x7feebc00b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:33:56.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.660+0000 7feedb7fe700 1 --2- 172.21.15.23:0/413744292 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feebc004ba0 0x7feebc00b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:56.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.660+0000 7feedb7fe700 1 -- 172.21.15.23:0/413744292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feebc003770 msgr2=0x7feebc003b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:33:56.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.660+0000 7feedb7fe700 1 --2- 172.21.15.23:0/413744292 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feebc003770 0x7feebc003b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:33:56.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.660+0000 7feedb7fe700 1 -- 172.21.15.23:0/413744292 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feebc003280 con 0x7feebc0040d0 2020-11-20T11:33:56.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.660+0000 7feee0bf3700 1 --2- 172.21.15.23:0/413744292 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feebc004ba0 0x7feebc00b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:33:56.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.660+0000 7feee0bf3700 1 -- 172.21.15.23:0/413744292 reap_dead start 2020-11-20T11:33:56.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.660+0000 7feedb7fe700 1 --2- 172.21.15.23:0/413744292 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feebc0040d0 0x7feebc0045a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7feecc00c470 tx=0x7feedc0a61b0).ready entity=mon.0 client_cookie=853d5d84ea9a2a03 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:56.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.660+0000 7feedaffd700 1 -- 172.21.15.23:0/413744292 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feecc0072e0 con 0x7feebc0040d0 2020-11-20T11:33:56.675 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.661+0000 7feedaffd700 1 -- 172.21.15.23:0/413744292 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feecc007440 con 0x7feebc0040d0 2020-11-20T11:33:56.675 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.661+0000 7feedaffd700 1 -- 172.21.15.23:0/413744292 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feecc005880 con 0x7feebc0040d0 2020-11-20T11:33:56.675 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.661+0000 7feee2656700 1 -- 172.21.15.23:0/413744292 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feebc0040d0 msgr2=0x7feebc0045a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:56.675 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.661+0000 7feee2656700 1 --2- 172.21.15.23:0/413744292 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feebc0040d0 0x7feebc0045a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7feecc00c470 tx=0x7feedc0a61b0).stop 2020-11-20T11:33:56.675 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.661+0000 7feee2656700 1 -- 172.21.15.23:0/413744292 shutdown_connections 2020-11-20T11:33:56.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.661+0000 7feee2656700 1 --2- 172.21.15.23:0/413744292 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feebc0040d0 0x7feebc0045a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:56.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.661+0000 7feee2656700 1 -- 172.21.15.23:0/413744292 >> 172.21.15.23:0/413744292 conn(0x7feedc060c80 msgr2=0x7feedc0acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:33:56.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.661+0000 7feee2656700 1 -- 172.21.15.23:0/413744292 shutdown_connections 2020-11-20T11:33:56.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.661+0000 7feee2656700 1 -- 172.21.15.23:0/413744292 wait complete. 2020-11-20T11:33:56.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.662+0000 7feee2656700 1 Processor -- start 2020-11-20T11:33:56.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.662+0000 7feee2656700 1 -- start start 2020-11-20T11:33:56.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.663+0000 7feee2656700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feebc0040d0 0x7feedc09d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:56.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.663+0000 7feee2656700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feedc14aa20 0x7feedc064ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:56.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.663+0000 7feee2656700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feedc14ae80 0x7feedc0653e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:56.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.663+0000 7feee2656700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feedc09d4a0 con 0x7feedc14ae80 2020-11-20T11:33:56.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.663+0000 7feee2656700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feedc066dc0 con 0x7feebc0040d0 2020-11-20T11:33:56.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.663+0000 7feee2656700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feedc067060 con 0x7feedc14aa20 2020-11-20T11:33:56.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.663+0000 7feee0bf3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feedc14ae80 0x7feedc0653e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:56.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.663+0000 7feee0bf3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feedc14ae80 0x7feedc0653e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42058/0 (socket says 172.21.15.23:42058) 2020-11-20T11:33:56.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.663+0000 7feee0bf3700 1 -- 172.21.15.23:0/3188028378 learned_addr learned my addr 172.21.15.23:0/3188028378 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:33:56.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.663+0000 7feedbfff700 1 -- 172.21.15.23:0/3188028378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feebc0040d0 msgr2=0x7feedc09d700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:33:56.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.663+0000 7feedbfff700 1 --2- 172.21.15.23:0/3188028378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feebc0040d0 0x7feedc09d700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:33:56.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.663+0000 7feedb7fe700 1 --2- 172.21.15.23:0/3188028378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feedc14aa20 0x7feedc064ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:56.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.663+0000 7feee0bf3700 1 -- 172.21.15.23:0/3188028378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feedc14aa20 msgr2=0x7feedc064ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:56.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.663+0000 7feee0bf3700 1 --2- 172.21.15.23:0/3188028378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feedc14aa20 0x7feedc064ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:33:56.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.663+0000 7feee0bf3700 1 -- 172.21.15.23:0/3188028378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feebc0040d0 msgr2=0x7feedc09d700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:33:56.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.663+0000 7feee0bf3700 1 --2- 172.21.15.23:0/3188028378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feebc0040d0 0x7feedc09d700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:33:56.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.663+0000 7feee0bf3700 1 -- 172.21.15.23:0/3188028378 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feedc067300 con 0x7feedc14ae80 2020-11-20T11:33:56.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.663+0000 7feedb7fe700 1 --2- 172.21.15.23:0/3188028378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feedc14aa20 0x7feedc064ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:33:56.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.663+0000 7feee0bf3700 1 --2- 172.21.15.23:0/3188028378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feedc14ae80 0x7feedc0653e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7feedc0a8b50 tx=0x7feed4008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:56.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.663+0000 7feed3fff700 1 -- 172.21.15.23:0/3188028378 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed4003ca0 con 0x7feedc14ae80 2020-11-20T11:33:56.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.663+0000 7feed3fff700 1 -- 172.21.15.23:0/3188028378 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed4003e00 con 0x7feedc14ae80 2020-11-20T11:33:56.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.663+0000 7feed3fff700 1 -- 172.21.15.23:0/3188028378 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed4005280 con 0x7feedc14ae80 2020-11-20T11:33:56.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.664+0000 7feee2656700 1 -- 172.21.15.23:0/3188028378 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feedc0677a0 con 0x7feedc14ae80 2020-11-20T11:33:56.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.664+0000 7feee2656700 1 -- 172.21.15.23:0/3188028378 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feedc13a810 con 0x7feedc14ae80 2020-11-20T11:33:56.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.664+0000 7feee2656700 1 --2- 172.21.15.23:0/3188028378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feedc0658d0 0x7feedc130d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:56.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.664+0000 7feee2656700 1 -- 172.21.15.23:0/3188028378 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feedc135930 con 0x7feedc0658d0 2020-11-20T11:33:56.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.664+0000 7feedbfff700 1 --2- 172.21.15.23:0/3188028378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feedc0658d0 0x7feedc130d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:56.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.665+0000 7feed3fff700 1 -- 172.21.15.23:0/3188028378 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7feed4005670 con 0x7feedc14ae80 2020-11-20T11:33:56.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.665+0000 7feedbfff700 1 -- 172.21.15.23:0/3188028378 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feedc067fb0 con 0x7feedc0658d0 2020-11-20T11:33:56.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.665+0000 7feed3fff700 1 --2- 172.21.15.23:0/3188028378 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7feeb8056d40 0x7feeb8059200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:56.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.665+0000 7feedbfff700 1 --2- 172.21.15.23:0/3188028378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feedc0658d0 0x7feedc130d00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7feec400b660 tx=0x7feec4006ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:56.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.665+0000 7feed3fff700 1 -- 172.21.15.23:0/3188028378 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7feed406fb90 con 0x7feedc14ae80 2020-11-20T11:33:56.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.665+0000 7feed3fff700 1 -- 172.21.15.23:0/3188028378 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec4003a00 con 0x7feedc0658d0 2020-11-20T11:33:56.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.665+0000 7feedb7fe700 1 --2- 172.21.15.23:0/3188028378 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7feeb8056d40 0x7feeb8059200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:56.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.665+0000 7feed3fff700 1 -- 172.21.15.23:0/3188028378 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7feec4003b60 con 0x7feedc0658d0 2020-11-20T11:33:56.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.665+0000 7feed3fff700 1 -- 172.21.15.23:0/3188028378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feedc0658d0 msgr2=0x7feedc130d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:56.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.665+0000 7feed3fff700 1 --2- 172.21.15.23:0/3188028378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feedc0658d0 0x7feedc130d00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7feec400b660 tx=0x7feec4006ef0).stop 2020-11-20T11:33:56.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.665+0000 7feed3fff700 1 -- 172.21.15.23:0/3188028378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feedc0658d0 msgr2=0x7feedc130d00 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:33:56.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.665+0000 7feed3fff700 1 --2- 172.21.15.23:0/3188028378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feedc0658d0 0x7feedc130d00 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0x7feec400b660 tx=0x7feec4006ef0).stop 2020-11-20T11:33:56.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.666+0000 7feedb7fe700 1 --2- 172.21.15.23:0/3188028378 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7feeb8056d40 0x7feeb8059200 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7feecc001030 tx=0x7feecc013000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:56.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.672+0000 7feee2656700 1 --2- 172.21.15.23:0/3188028378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feedc1311f0 0x7feedc1315d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:33:56.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.672+0000 7feee2656700 1 -- 172.21.15.23:0/3188028378 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feedc067490 con 0x7feedc1311f0 2020-11-20T11:33:56.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.672+0000 7feedbfff700 1 --2- 172.21.15.23:0/3188028378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feedc1311f0 0x7feedc1315d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:33:56.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.673+0000 7feedbfff700 1 -- 172.21.15.23:0/3188028378 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7feedc05c520 con 0x7feedc1311f0 2020-11-20T11:33:56.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.673+0000 7feedbfff700 1 --2- 172.21.15.23:0/3188028378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feedc1311f0 0x7feedc1315d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7feec4012000 tx=0x7feec4015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:33:56.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.673+0000 7feed3fff700 1 -- 172.21.15.23:0/3188028378 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec4022410 con 0x7feedc1311f0 2020-11-20T11:33:56.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.673+0000 7feed3fff700 1 -- 172.21.15.23:0/3188028378 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4772 (secure 0 0 0) 0x7feec40037f0 con 0x7feedc1311f0 2020-11-20T11:33:56.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.673+0000 7feed3fff700 1 -- 172.21.15.23:0/3188028378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feedc1311f0 msgr2=0x7feedc1315d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:56.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.673+0000 7feed3fff700 1 --2- 172.21.15.23:0/3188028378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feedc1311f0 0x7feedc1315d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7feec4012000 tx=0x7feec4015000).stop 2020-11-20T11:33:56.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.673+0000 7feed3fff700 1 -- 172.21.15.23:0/3188028378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feedc1311f0 msgr2=0x7feedc1315d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:33:56.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.673+0000 7feed3fff700 1 --2- 172.21.15.23:0/3188028378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feedc1311f0 0x7feedc1315d0 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0x7feec4012000 tx=0x7feec4015000).stop 2020-11-20T11:33:56.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.673+0000 7feee2656700 1 -- 172.21.15.23:0/3188028378 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7feeb8056d40 msgr2=0x7feeb8059200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:56.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.673+0000 7feee2656700 1 --2- 172.21.15.23:0/3188028378 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7feeb8056d40 0x7feeb8059200 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7feecc001030 tx=0x7feecc013000).stop 2020-11-20T11:33:56.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.673+0000 7feee2656700 1 -- 172.21.15.23:0/3188028378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feedc14ae80 msgr2=0x7feedc0653e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:33:56.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.673+0000 7feee2656700 1 --2- 172.21.15.23:0/3188028378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feedc14ae80 0x7feedc0653e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7feedc0a8b50 tx=0x7feed4008040).stop 2020-11-20T11:33:56.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.673+0000 7feee0bf3700 1 -- 172.21.15.23:0/3188028378 reap_dead start 2020-11-20T11:33:56.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.673+0000 7feee0bf3700 1 -- 172.21.15.23:0/3188028378 reap_dead start 2020-11-20T11:33:56.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.673+0000 7feee2656700 1 -- 172.21.15.23:0/3188028378 shutdown_connections 2020-11-20T11:33:56.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.674+0000 7feee2656700 1 -- 172.21.15.23:0/3188028378 >> 172.21.15.23:0/3188028378 conn(0x7feedc060c80 msgr2=0x7feedc0a77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:33:56.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.674+0000 7feee2656700 1 -- 172.21.15.23:0/3188028378 shutdown_connections 2020-11-20T11:33:56.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:33:56.674+0000 7feee2656700 1 -- 172.21.15.23:0/3188028378 wait complete. 2020-11-20T11:33:56.693 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:33:56.693 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:33:56.693 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:33:56.694 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:33:56.694 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 14, 2020-11-20T11:33:56.694 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:33:56.694 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:33:56.694 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:33:56.695 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:56.695 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 0, 2020-11-20T11:33:56.695 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:33:56.695 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:33:56.695 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:33:56.696 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:33:56.696 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:33:56.696 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:33:56.696 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:33:56.696 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:33:56.697 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:33:56.697 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:33:56.697 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:33:56.697 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:56.697 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:33:56.698 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:33:56.698 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:33:56.698 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:33:56.698 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:33:56.698 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:33:56.699 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:33:56.699 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:33:56.699 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:33:56.699 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:33:56.700 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:56.700 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:56.700 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:33:56.700 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:33:56.700 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:33:56.701 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:33:56.701 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:33:56.701 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:33:56.701 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:33:56.702 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:33:56.702 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:33:56.702 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:33:56.702 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:33:56.702 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:33:56.703 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:33:56.703 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:33:56.703 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:33:56.703 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:33:56.703 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:33:56.704 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:33:56.704 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:33:56.704 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:33:56.704 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:33:56.704 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:33:56.705 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:33:56.705 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:56.705 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:33:56.705 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:56.705 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:33:56.706 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:56.706 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:33:56.706 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:33:56.706 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:33:56.706 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:33:56.707 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:56.707 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:33:56.707 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:33:56.707 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:56.708 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:56.708 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:56.708 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:33:56.708 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:33:56.708 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:56.709 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:56.709 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:56.709 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:56.709 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:33:56.709 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:33:56.710 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:33:56.710 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:33:56.710 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:33:56.710 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:56.711 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:56.711 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:33:56.711 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:33:56.711 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:33:56.711 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:33:56.712 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:56.712 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:33:56.712 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:33:56.712 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:56.712 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:56.713 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:56.713 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:33:56.713 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:33:56.713 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:56.713 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:56.714 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:56.714 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:56.714 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:33:56.714 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:33:56.715 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:33:56.715 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:33:56.715 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:33:56.715 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:56.715 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:56.716 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:33:56.716 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:33:56.716 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:33:56.716 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:33:56.717 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:56.717 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:33:56.717 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:33:56.717 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:56.718 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:56.718 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:56.718 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:33:56.718 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:33:56.718 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:33:56.719 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:56.719 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:56.719 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:56.719 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:33:56.719 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:33:56.720 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:33:56.720 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:33:56.720 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:33:56.720 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:56.720 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:56.721 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:56.721 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:33:56.721 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:33:56.721 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:56.722 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:56.722 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:56.722 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:33:56.722 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:56.722 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:56.723 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:33:56.723 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:56.723 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:56.723 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:56.723 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:33:56.724 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:56.724 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:56.724 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:33:56.724 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:56.724 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:56.725 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:56.725 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 6 2020-11-20T11:33:56.725 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:56.725 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:56.726 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:33:56.726 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:56.726 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:56.726 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:56.726 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:33:56.727 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:56.727 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:33:56.727 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:33:56.727 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:33:56.727 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:33:56.728 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:33:56.728 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:33:56.728 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:33:56.728 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:33:56.729 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:33:56.729 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:33:56.729 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:33:56.729 INFO:tasks.workunit.client.0.smithi023.stdout:patching file parser.c 2020-11-20T11:33:56.730 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2020-11-20T11:33:56.835 INFO:tasks.workunit.client.0.smithi023.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2020-11-20T11:33:57.846 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether build environment is sane... yes 2020-11-20T11:33:57.864 INFO:tasks.workunit.client.0.smithi023.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2020-11-20T11:33:57.865 INFO:tasks.workunit.client.0.smithi023.stdout:checking for gawk... gawk 2020-11-20T11:33:57.892 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether make sets $(MAKE)... yes 2020-11-20T11:33:57.942 INFO:tasks.workunit.client.0.smithi023.stdout:checking build system type... x86_64-unknown-linux-gnu 2020-11-20T11:33:57.942 INFO:tasks.workunit.client.0.smithi023.stdout:checking host system type... x86_64-unknown-linux-gnu 2020-11-20T11:33:57.963 INFO:tasks.workunit.client.0.smithi023.stdout:checking for style of include used by make... GNU 2020-11-20T11:33:57.968 INFO:tasks.workunit.client.0.smithi023.stdout:checking for gcc... gcc 2020-11-20T11:33:58.049 INFO:tasks.workunit.client.0.smithi023.stdout:checking for C compiler default output file name... a.out 2020-11-20T11:33:58.052 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether the C compiler works... yes 2020-11-20T11:33:58.056 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether we are cross compiling... no 2020-11-20T11:33:58.106 INFO:tasks.workunit.client.0.smithi023.stdout:checking for suffix of executables... 2020-11-20T11:33:58.159 INFO:tasks.workunit.client.0.smithi023.stdout:checking for suffix of object files... o 2020-11-20T11:33:58.201 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether we are using the GNU C compiler... yes 2020-11-20T11:33:58.243 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether gcc accepts -g... yes 2020-11-20T11:33:58.366 INFO:tasks.workunit.client.0.smithi023.stdout:checking for gcc option to accept ISO C89... none needed 2020-11-20T11:33:58.523 INFO:tasks.workunit.client.0.smithi023.stdout:checking dependency style of gcc... gcc3 2020-11-20T11:33:58.620 INFO:tasks.workunit.client.0.smithi023.stdout:checking how to run the C preprocessor... gcc -E 2020-11-20T11:33:58.678 INFO:tasks.workunit.client.0.smithi023.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2020-11-20T11:33:59.125 INFO:tasks.workunit.client.0.smithi023.stdout:checking for egrep... /usr/bin/grep -E 2020-11-20T11:33:59.125 INFO:tasks.workunit.client.0.smithi023.stdout:checking for AIX... no 2020-11-20T11:33:59.126 INFO:tasks.workunit.client.0.smithi023.stdout:checking for gcc... (cached) gcc 2020-11-20T11:33:59.126 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether we are using the GNU C compiler... (cached) yes 2020-11-20T11:33:59.126 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether gcc accepts -g... (cached) yes 2020-11-20T11:33:59.126 INFO:tasks.workunit.client.0.smithi023.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2020-11-20T11:33:59.126 INFO:tasks.workunit.client.0.smithi023.stdout:checking dependency style of gcc... (cached) gcc3 2020-11-20T11:33:59.127 INFO:tasks.workunit.client.0.smithi023.stdout:checking for main in -lm... yes 2020-11-20T11:33:59.127 INFO:tasks.workunit.client.0.smithi023.stdout:checking for main in -lpthread... yes 2020-11-20T11:33:59.127 INFO:tasks.workunit.client.0.smithi023.stdout:checking for ANSI C header files... yes 2020-11-20T11:33:59.182 INFO:tasks.workunit.client.0.smithi023.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2020-11-20T11:33:59.242 INFO:tasks.workunit.client.0.smithi023.stdout:checking for sys/types.h... yes 2020-11-20T11:33:59.307 INFO:tasks.workunit.client.0.smithi023.stdout:checking for sys/stat.h... yes 2020-11-20T11:33:59.376 INFO:tasks.workunit.client.0.smithi023.stdout:checking for stdlib.h... yes 2020-11-20T11:33:59.439 INFO:tasks.workunit.client.0.smithi023.stdout:checking for string.h... yes 2020-11-20T11:33:59.484 INFO:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:33:59.487 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:33:59.513 INFO:tasks.workunit.client.0.smithi023.stdout:checking for memory.h... yes 2020-11-20T11:33:59.580 INFO:tasks.workunit.client.0.smithi023.stdout:checking for strings.h... yes 2020-11-20T11:33:59.643 INFO:tasks.workunit.client.0.smithi023.stdout:checking for inttypes.h... yes 2020-11-20T11:33:59.713 INFO:tasks.workunit.client.0.smithi023.stdout:checking for stdint.h... yes 2020-11-20T11:33:59.781 INFO:tasks.workunit.client.0.smithi023.stdout:checking for unistd.h... yes 2020-11-20T11:33:59.880 INFO:tasks.workunit.client.0.smithi023.stdout:checking pthread.h usability... yes 2020-11-20T11:33:59.909 INFO:tasks.workunit.client.0.smithi023.stdout:checking pthread.h presence... yes 2020-11-20T11:33:59.909 INFO:tasks.workunit.client.0.smithi023.stdout:checking for pthread.h... yes 2020-11-20T11:33:59.978 INFO:tasks.workunit.client.0.smithi023.stdout:checking fcntl.h usability... yes 2020-11-20T11:34:00.005 INFO:tasks.workunit.client.0.smithi023.stdout:checking fcntl.h presence... yes 2020-11-20T11:34:00.005 INFO:tasks.workunit.client.0.smithi023.stdout:checking for fcntl.h... yes 2020-11-20T11:34:00.074 INFO:tasks.workunit.client.0.smithi023.stdout:checking limits.h usability... yes 2020-11-20T11:34:00.102 INFO:tasks.workunit.client.0.smithi023.stdout:checking limits.h presence... yes 2020-11-20T11:34:00.102 INFO:tasks.workunit.client.0.smithi023.stdout:checking for limits.h... yes 2020-11-20T11:34:00.123 INFO:tasks.workunit.client.0.smithi023.stdout:checking for stdint.h... (cached) yes 2020-11-20T11:34:00.182 INFO:tasks.workunit.client.0.smithi023.stdout:checking sys/time.h usability... yes 2020-11-20T11:34:00.211 INFO:tasks.workunit.client.0.smithi023.stdout:checking sys/time.h presence... yes 2020-11-20T11:34:00.212 INFO:tasks.workunit.client.0.smithi023.stdout:checking for sys/time.h... yes 2020-11-20T11:34:00.234 INFO:tasks.workunit.client.0.smithi023.stdout:checking for unistd.h... (cached) yes 2020-11-20T11:34:00.293 INFO:tasks.workunit.client.0.smithi023.stdout:checking sys/vfs.h usability... yes 2020-11-20T11:34:00.323 INFO:tasks.workunit.client.0.smithi023.stdout:checking sys/vfs.h presence... yes 2020-11-20T11:34:00.323 INFO:tasks.workunit.client.0.smithi023.stdout:checking for sys/vfs.h... yes 2020-11-20T11:34:00.388 INFO:tasks.workunit.client.0.smithi023.stdout:checking sys/limits.h usability... no 2020-11-20T11:34:00.420 INFO:tasks.workunit.client.0.smithi023.stdout:checking sys/limits.h presence... no 2020-11-20T11:34:00.421 INFO:tasks.workunit.client.0.smithi023.stdout:checking for sys/limits.h... no 2020-11-20T11:34:00.465 INFO:tasks.workunit.client.0.smithi023.stdout:checking for an ANSI C-conforming const... yes 2020-11-20T11:34:00.525 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether time.h and sys/time.h may both be included... yes 2020-11-20T11:34:00.534 INFO:tasks.workunit.client.0.smithi023.stdout:checking for function prototypes... yes 2020-11-20T11:34:00.610 INFO:tasks.workunit.client.0.smithi023.stdout:checking whether setvbuf arguments are reversed... no 2020-11-20T11:34:00.696 INFO:tasks.workunit.client.0.smithi023.stdout:checking for system... yes 2020-11-20T11:34:00.786 INFO:tasks.workunit.client.0.smithi023.stdout:checking for gettimeofday... yes 2020-11-20T11:34:00.878 INFO:tasks.workunit.client.0.smithi023.stdout:checking for mkdir... yes 2020-11-20T11:34:00.965 INFO:tasks.workunit.client.0.smithi023.stdout:checking for strerror... yes 2020-11-20T11:34:01.056 INFO:tasks.workunit.client.0.smithi023.stdout:checking for open64... yes 2020-11-20T11:34:01.157 INFO:tasks.workunit.client.0.smithi023.stdout:checking for stat64... yes 2020-11-20T11:34:01.246 INFO:tasks.workunit.client.0.smithi023.stdout:checking for fseeko64... yes 2020-11-20T11:34:01.333 INFO:tasks.workunit.client.0.smithi023.stdout:checking for lrand48_r... yes 2020-11-20T11:34:01.421 INFO:tasks.workunit.client.0.smithi023.stdout:checking for srand48_r... yes 2020-11-20T11:34:01.455 INFO:tasks.workunit.client.0.smithi023.stdout:configure: creating ./config.status 2020-11-20T11:34:01.628 INFO:tasks.workunit.client.0.smithi023.stdout:config.status: creating Makefile 2020-11-20T11:34:01.647 INFO:tasks.workunit.client.0.smithi023.stdout:config.status: creating config.h 2020-11-20T11:34:01.675 INFO:tasks.workunit.client.0.smithi023.stdout:config.status: executing depfiles commands 2020-11-20T11:34:01.756 INFO:tasks.workunit.client.0.smithi023.stdout:config.status: executing default commands 2020-11-20T11:34:01.855 INFO:tasks.workunit.client.0.smithi023.stdout:make all-am 2020-11-20T11:34:01.887 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb-6.0-rc2' 2020-11-20T11:34:01.887 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2020-11-20T11:34:01.948 INFO:tasks.workunit.client.0.smithi023.stderr:fileops.c: In function ‘ffsb_stat’: 2020-11-20T11:34:01.948 INFO:tasks.workunit.client.0.smithi023.stderr:fileops.c:542:2: warning: implicit declaration of function ‘fhstat’; did you mean ‘fstat’? [-Wimplicit-function-declaration] 2020-11-20T11:34:01.949 INFO:tasks.workunit.client.0.smithi023.stderr: fhstat(curfile->name, ft, fs); 2020-11-20T11:34:01.949 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~ 2020-11-20T11:34:01.949 INFO:tasks.workunit.client.0.smithi023.stderr: fstat 2020-11-20T11:34:02.089 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2020-11-20T11:34:02.095 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2020-11-20T11:34:02.179 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2020-11-20T11:34:02.185 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2020-11-20T11:34:02.298 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/main.Tpo .deps/main.Po 2020-11-20T11:34:02.304 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2020-11-20T11:34:02.419 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2020-11-20T11:34:02.424 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2020-11-20T11:34:02.579 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2020-11-20T11:34:02.585 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2020-11-20T11:34:02.657 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2020-11-20T11:34:02.662 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2020-11-20T11:34:02.715 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2020-11-20T11:34:02.721 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2020-11-20T11:34:02.781 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2020-11-20T11:34:02.787 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2020-11-20T11:34:02.959 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2020-11-20T11:34:02.966 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2020-11-20T11:34:03.099 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2020-11-20T11:34:03.105 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2020-11-20T11:34:03.149 INFO:tasks.workunit.client.0.smithi023.stderr:ffsb_fs.c: In function ‘fs_needs_stats’: 2020-11-20T11:34:03.150 INFO:tasks.workunit.client.0.smithi023.stderr:ffsb_fs.c:627:24: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2020-11-20T11:34:03.150 INFO:tasks.workunit.client.0.smithi023.stderr: return (fs != NULL) ? (int)fs->fsd.config : 0; 2020-11-20T11:34:03.150 INFO:tasks.workunit.client.0.smithi023.stderr: ^ 2020-11-20T11:34:03.268 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2020-11-20T11:34:03.275 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2020-11-20T11:34:03.358 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2020-11-20T11:34:03.365 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2020-11-20T11:34:03.453 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2020-11-20T11:34:03.460 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2020-11-20T11:34:03.589 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/util.Tpo .deps/util.Po 2020-11-20T11:34:03.596 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2020-11-20T11:34:03.637 INFO:tasks.workunit.client.0.smithi023.stderr:In file included from parser.h:22, 2020-11-20T11:34:03.637 INFO:tasks.workunit.client.0.smithi023.stderr: from parser.c:25: 2020-11-20T11:34:03.638 INFO:tasks.workunit.client.0.smithi023.stderr:parser.c: In function ‘init_filesys’: 2020-11-20T11:34:03.639 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2020-11-20T11:34:03.639 INFO:tasks.workunit.client.0.smithi023.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2020-11-20T11:34:03.639 INFO:tasks.workunit.client.0.smithi023.stderr: ^ 2020-11-20T11:34:03.639 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:33:28: note: in expansion of macro ‘offsetof’ 2020-11-20T11:34:03.640 INFO:tasks.workunit.client.0.smithi023.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2020-11-20T11:34:03.640 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~ 2020-11-20T11:34:03.640 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:36:2: note: in expansion of macro ‘container_of’ 2020-11-20T11:34:03.640 INFO:tasks.workunit.client.0.smithi023.stderr: container_of(ptr, type, member) 2020-11-20T11:34:03.641 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~~~~~ 2020-11-20T11:34:03.641 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:39:13: note: in expansion of macro ‘list_entry’ 2020-11-20T11:34:03.641 INFO:tasks.workunit.client.0.smithi023.stderr: for (pos = list_entry((head)->next, typeof(*pos), member); \ 2020-11-20T11:34:03.641 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~~~ 2020-11-20T11:34:03.641 INFO:tasks.workunit.client.0.smithi023.stderr:parser.c:934:3: note: in expansion of macro ‘list_for_each_entry’ 2020-11-20T11:34:03.642 INFO:tasks.workunit.client.0.smithi023.stderr: list_for_each_entry(tmp_list, &list_head->list, list) 2020-11-20T11:34:03.642 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~~~~~~~~~~~~ 2020-11-20T11:34:03.642 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2020-11-20T11:34:03.642 INFO:tasks.workunit.client.0.smithi023.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2020-11-20T11:34:03.642 INFO:tasks.workunit.client.0.smithi023.stderr: ^ 2020-11-20T11:34:03.643 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:33:28: note: in expansion of macro ‘offsetof’ 2020-11-20T11:34:03.643 INFO:tasks.workunit.client.0.smithi023.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2020-11-20T11:34:03.643 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~ 2020-11-20T11:34:03.643 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:36:2: note: in expansion of macro ‘container_of’ 2020-11-20T11:34:03.644 INFO:tasks.workunit.client.0.smithi023.stderr: container_of(ptr, type, member) 2020-11-20T11:34:03.644 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~~~~~ 2020-11-20T11:34:03.644 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:41:13: note: in expansion of macro ‘list_entry’ 2020-11-20T11:34:03.644 INFO:tasks.workunit.client.0.smithi023.stderr: pos = list_entry(pos->member.next, typeof(*pos), member)) 2020-11-20T11:34:03.644 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~~~ 2020-11-20T11:34:03.645 INFO:tasks.workunit.client.0.smithi023.stderr:parser.c:934:3: note: in expansion of macro ‘list_for_each_entry’ 2020-11-20T11:34:03.645 INFO:tasks.workunit.client.0.smithi023.stderr: list_for_each_entry(tmp_list, &list_head->list, list) 2020-11-20T11:34:03.645 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~~~~~~~~~~~~ 2020-11-20T11:34:03.645 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2020-11-20T11:34:03.645 INFO:tasks.workunit.client.0.smithi023.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2020-11-20T11:34:03.646 INFO:tasks.workunit.client.0.smithi023.stderr: ^ 2020-11-20T11:34:03.646 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:33:28: note: in expansion of macro ‘offsetof’ 2020-11-20T11:34:03.646 INFO:tasks.workunit.client.0.smithi023.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2020-11-20T11:34:03.646 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~ 2020-11-20T11:34:03.646 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:36:2: note: in expansion of macro ‘container_of’ 2020-11-20T11:34:03.647 INFO:tasks.workunit.client.0.smithi023.stderr: container_of(ptr, type, member) 2020-11-20T11:34:03.647 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~~~~~ 2020-11-20T11:34:03.647 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:39:13: note: in expansion of macro ‘list_entry’ 2020-11-20T11:34:03.647 INFO:tasks.workunit.client.0.smithi023.stderr: for (pos = list_entry((head)->next, typeof(*pos), member); \ 2020-11-20T11:34:03.647 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~~~ 2020-11-20T11:34:03.648 INFO:tasks.workunit.client.0.smithi023.stderr:parser.c:941:3: note: in expansion of macro ‘list_for_each_entry’ 2020-11-20T11:34:03.648 INFO:tasks.workunit.client.0.smithi023.stderr: list_for_each_entry(tmp_list, &list_head->list, list) { 2020-11-20T11:34:03.648 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~~~~~~~~~~~~ 2020-11-20T11:34:03.648 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2020-11-20T11:34:03.648 INFO:tasks.workunit.client.0.smithi023.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2020-11-20T11:34:03.649 INFO:tasks.workunit.client.0.smithi023.stderr: ^ 2020-11-20T11:34:03.649 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:33:28: note: in expansion of macro ‘offsetof’ 2020-11-20T11:34:03.649 INFO:tasks.workunit.client.0.smithi023.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2020-11-20T11:34:03.649 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~ 2020-11-20T11:34:03.649 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:36:2: note: in expansion of macro ‘container_of’ 2020-11-20T11:34:03.650 INFO:tasks.workunit.client.0.smithi023.stderr: container_of(ptr, type, member) 2020-11-20T11:34:03.650 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~~~~~ 2020-11-20T11:34:03.650 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:41:13: note: in expansion of macro ‘list_entry’ 2020-11-20T11:34:03.650 INFO:tasks.workunit.client.0.smithi023.stderr: pos = list_entry(pos->member.next, typeof(*pos), member)) 2020-11-20T11:34:03.650 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~~~ 2020-11-20T11:34:03.651 INFO:tasks.workunit.client.0.smithi023.stderr:parser.c:941:3: note: in expansion of macro ‘list_for_each_entry’ 2020-11-20T11:34:03.651 INFO:tasks.workunit.client.0.smithi023.stderr: list_for_each_entry(tmp_list, &list_head->list, list) { 2020-11-20T11:34:03.651 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~~~~~~~~~~~~ 2020-11-20T11:34:03.651 INFO:tasks.workunit.client.0.smithi023.stderr:parser.c: In function ‘init_tg_stats’: 2020-11-20T11:34:03.651 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2020-11-20T11:34:03.652 INFO:tasks.workunit.client.0.smithi023.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2020-11-20T11:34:03.652 INFO:tasks.workunit.client.0.smithi023.stderr: ^ 2020-11-20T11:34:03.652 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:33:28: note: in expansion of macro ‘offsetof’ 2020-11-20T11:34:03.652 INFO:tasks.workunit.client.0.smithi023.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2020-11-20T11:34:03.652 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~ 2020-11-20T11:34:03.653 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:36:2: note: in expansion of macro ‘container_of’ 2020-11-20T11:34:03.653 INFO:tasks.workunit.client.0.smithi023.stderr: container_of(ptr, type, member) 2020-11-20T11:34:03.653 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~~~~~ 2020-11-20T11:34:03.653 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:39:13: note: in expansion of macro ‘list_entry’ 2020-11-20T11:34:03.653 INFO:tasks.workunit.client.0.smithi023.stderr: for (pos = list_entry((head)->next, typeof(*pos), member); \ 2020-11-20T11:34:03.654 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~~~ 2020-11-20T11:34:03.654 INFO:tasks.workunit.client.0.smithi023.stderr:parser.c:971:6: note: in expansion of macro ‘list_for_each_entry’ 2020-11-20T11:34:03.654 INFO:tasks.workunit.client.0.smithi023.stderr: list_for_each_entry(tmp_list, 2020-11-20T11:34:03.654 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~~~~~~~~~~~~ 2020-11-20T11:34:03.654 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2020-11-20T11:34:03.655 INFO:tasks.workunit.client.0.smithi023.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2020-11-20T11:34:03.655 INFO:tasks.workunit.client.0.smithi023.stderr: ^ 2020-11-20T11:34:03.655 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:33:28: note: in expansion of macro ‘offsetof’ 2020-11-20T11:34:03.655 INFO:tasks.workunit.client.0.smithi023.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2020-11-20T11:34:03.656 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~ 2020-11-20T11:34:03.656 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:36:2: note: in expansion of macro ‘container_of’ 2020-11-20T11:34:03.656 INFO:tasks.workunit.client.0.smithi023.stderr: container_of(ptr, type, member) 2020-11-20T11:34:03.656 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~~~~~ 2020-11-20T11:34:03.657 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:41:13: note: in expansion of macro ‘list_entry’ 2020-11-20T11:34:03.657 INFO:tasks.workunit.client.0.smithi023.stderr: pos = list_entry(pos->member.next, typeof(*pos), member)) 2020-11-20T11:34:03.657 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~~~ 2020-11-20T11:34:03.657 INFO:tasks.workunit.client.0.smithi023.stderr:parser.c:971:6: note: in expansion of macro ‘list_for_each_entry’ 2020-11-20T11:34:03.657 INFO:tasks.workunit.client.0.smithi023.stderr: list_for_each_entry(tmp_list, 2020-11-20T11:34:03.658 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~~~~~~~~~~~~ 2020-11-20T11:34:03.658 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2020-11-20T11:34:03.658 INFO:tasks.workunit.client.0.smithi023.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2020-11-20T11:34:03.658 INFO:tasks.workunit.client.0.smithi023.stderr: ^ 2020-11-20T11:34:03.658 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:33:28: note: in expansion of macro ‘offsetof’ 2020-11-20T11:34:03.659 INFO:tasks.workunit.client.0.smithi023.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2020-11-20T11:34:03.659 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~ 2020-11-20T11:34:03.659 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:36:2: note: in expansion of macro ‘container_of’ 2020-11-20T11:34:03.659 INFO:tasks.workunit.client.0.smithi023.stderr: container_of(ptr, type, member) 2020-11-20T11:34:03.659 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~~~~~ 2020-11-20T11:34:03.660 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:39:13: note: in expansion of macro ‘list_entry’ 2020-11-20T11:34:03.660 INFO:tasks.workunit.client.0.smithi023.stderr: for (pos = list_entry((head)->next, typeof(*pos), member); \ 2020-11-20T11:34:03.660 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~~~ 2020-11-20T11:34:03.660 INFO:tasks.workunit.client.0.smithi023.stderr:parser.c:980:6: note: in expansion of macro ‘list_for_each_entry’ 2020-11-20T11:34:03.660 INFO:tasks.workunit.client.0.smithi023.stderr: list_for_each_entry(tmp_list, 2020-11-20T11:34:03.661 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~~~~~~~~~~~~ 2020-11-20T11:34:03.661 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2020-11-20T11:34:03.661 INFO:tasks.workunit.client.0.smithi023.stderr: #define offsetof(type, member) ((int) &((type *)0)->member) 2020-11-20T11:34:03.661 INFO:tasks.workunit.client.0.smithi023.stderr: ^ 2020-11-20T11:34:03.661 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:33:28: note: in expansion of macro ‘offsetof’ 2020-11-20T11:34:03.662 INFO:tasks.workunit.client.0.smithi023.stderr: (type *)((char *)__mptr - offsetof(type, member)); }) 2020-11-20T11:34:03.662 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~ 2020-11-20T11:34:03.662 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:36:2: note: in expansion of macro ‘container_of’ 2020-11-20T11:34:03.662 INFO:tasks.workunit.client.0.smithi023.stderr: container_of(ptr, type, member) 2020-11-20T11:34:03.662 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~~~~~ 2020-11-20T11:34:03.663 INFO:tasks.workunit.client.0.smithi023.stderr:list.h:41:13: note: in expansion of macro ‘list_entry’ 2020-11-20T11:34:03.663 INFO:tasks.workunit.client.0.smithi023.stderr: pos = list_entry(pos->member.next, typeof(*pos), member)) 2020-11-20T11:34:03.663 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~~~ 2020-11-20T11:34:03.663 INFO:tasks.workunit.client.0.smithi023.stderr:parser.c:980:6: note: in expansion of macro ‘list_for_each_entry’ 2020-11-20T11:34:03.663 INFO:tasks.workunit.client.0.smithi023.stderr: list_for_each_entry(tmp_list, 2020-11-20T11:34:03.664 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~~~~~~~~~~~~ 2020-11-20T11:34:03.749 INFO:tasks.workunit.client.0.smithi023.stderr:parser.c: In function ‘set_weight’: 2020-11-20T11:34:03.749 INFO:tasks.workunit.client.0.smithi023.stderr:parser.c:623:4: warning: ‘strncpy’ specified bound depends on the length of the source argument [-Wstringop-overflow=] 2020-11-20T11:34:03.749 INFO:tasks.workunit.client.0.smithi023.stderr: strncpy (op, tmp_config->name, len - 7); 2020-11-20T11:34:03.749 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2020-11-20T11:34:03.750 INFO:tasks.workunit.client.0.smithi023.stderr:parser.c:620:10: note: length computed here 2020-11-20T11:34:03.750 INFO:tasks.workunit.client.0.smithi023.stderr: len = strlen(tmp_config->name); 2020-11-20T11:34:03.750 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~~~~~~~~~~~~~~~~~ 2020-11-20T11:34:03.753 INFO:tasks.workunit.client.0.smithi023.stderr:parser.c: In function ‘get_weight_total’: 2020-11-20T11:34:03.753 INFO:tasks.workunit.client.0.smithi023.stderr:parser.c:645:4: warning: ‘strncpy’ specified bound depends on the length of the source argument [-Wstringop-overflow=] 2020-11-20T11:34:03.753 INFO:tasks.workunit.client.0.smithi023.stderr: strncpy (op, tmp_config->name, len - 7); 2020-11-20T11:34:03.754 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2020-11-20T11:34:03.754 INFO:tasks.workunit.client.0.smithi023.stderr:parser.c:642:10: note: length computed here 2020-11-20T11:34:03.754 INFO:tasks.workunit.client.0.smithi023.stderr: len = strlen(tmp_config->name); 2020-11-20T11:34:03.754 INFO:tasks.workunit.client.0.smithi023.stderr: ^~~~~~~~~~~~~~~~~~~~~~~~ 2020-11-20T11:34:03.885 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2020-11-20T11:34:03.891 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2020-11-20T11:34:03.967 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2020-11-20T11:34:03.973 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2020-11-20T11:34:04.114 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2020-11-20T11:34:04.119 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2020-11-20T11:34:04.158 INFO:tasks.workunit.client.0.smithi023.stdout:mv -f .deps/list.Tpo .deps/list.Po 2020-11-20T11:34:04.165 INFO:tasks.workunit.client.0.smithi023.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2020-11-20T11:34:04.267 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb-6.0-rc2' 2020-11-20T11:34:04.285 INFO:tasks.workunit.client.0.smithi023.stdout:FFSB version 6.0-RC2 started 2020-11-20T11:34:04.285 INFO:tasks.workunit.client.0.smithi023.stdout: 2020-11-20T11:34:04.288 INFO:tasks.workunit.client.0.smithi023.stdout:benchmark time = 300 2020-11-20T11:34:04.289 INFO:tasks.workunit.client.0.smithi023.stdout:ThreadGroup 0 2020-11-20T11:34:04.289 INFO:tasks.workunit.client.0.smithi023.stdout:================ 2020-11-20T11:34:04.289 INFO:tasks.workunit.client.0.smithi023.stdout: num_threads = 32 2020-11-20T11:34:04.289 INFO:tasks.workunit.client.0.smithi023.stdout: 2020-11-20T11:34:04.290 INFO:tasks.workunit.client.0.smithi023.stdout: read_random = off 2020-11-20T11:34:04.290 INFO:tasks.workunit.client.0.smithi023.stdout: read_size = 0 (0B) 2020-11-20T11:34:04.290 INFO:tasks.workunit.client.0.smithi023.stdout: read_blocksize = 0 (0B) 2020-11-20T11:34:04.290 INFO:tasks.workunit.client.0.smithi023.stdout: read_skip = off 2020-11-20T11:34:04.291 INFO:tasks.workunit.client.0.smithi023.stdout: read_skipsize = 0 (0B) 2020-11-20T11:34:04.291 INFO:tasks.workunit.client.0.smithi023.stdout: 2020-11-20T11:34:04.291 INFO:tasks.workunit.client.0.smithi023.stdout: write_random = on 2020-11-20T11:34:04.291 INFO:tasks.workunit.client.0.smithi023.stdout: write_size = 5242880 (5MB) 2020-11-20T11:34:04.292 INFO:tasks.workunit.client.0.smithi023.stdout: fsync_file = 0 2020-11-20T11:34:04.292 INFO:tasks.workunit.client.0.smithi023.stdout: write_blocksize = 4096 (4KB) 2020-11-20T11:34:04.292 INFO:tasks.workunit.client.0.smithi023.stdout: wait time = 0 2020-11-20T11:34:04.292 INFO:tasks.workunit.client.0.smithi023.stdout: 2020-11-20T11:34:04.293 INFO:tasks.workunit.client.0.smithi023.stdout: op weights 2020-11-20T11:34:04.293 INFO:tasks.workunit.client.0.smithi023.stdout: read = 0 (0.00%) 2020-11-20T11:34:04.293 INFO:tasks.workunit.client.0.smithi023.stdout: readall = 0 (0.00%) 2020-11-20T11:34:04.293 INFO:tasks.workunit.client.0.smithi023.stdout: write = 1 (100.00%) 2020-11-20T11:34:04.293 INFO:tasks.workunit.client.0.smithi023.stdout: create = 0 (0.00%) 2020-11-20T11:34:04.294 INFO:tasks.workunit.client.0.smithi023.stdout: append = 0 (0.00%) 2020-11-20T11:34:04.294 INFO:tasks.workunit.client.0.smithi023.stdout: delete = 0 (0.00%) 2020-11-20T11:34:04.294 INFO:tasks.workunit.client.0.smithi023.stdout: metaop = 0 (0.00%) 2020-11-20T11:34:04.294 INFO:tasks.workunit.client.0.smithi023.stdout: createdir = 0 (0.00%) 2020-11-20T11:34:04.295 INFO:tasks.workunit.client.0.smithi023.stdout: stat = 0 (0.00%) 2020-11-20T11:34:04.295 INFO:tasks.workunit.client.0.smithi023.stdout: writeall = 0 (0.00%) 2020-11-20T11:34:04.295 INFO:tasks.workunit.client.0.smithi023.stdout: writeall_fsync = 0 (0.00%) 2020-11-20T11:34:04.295 INFO:tasks.workunit.client.0.smithi023.stdout: open_close = 0 (0.00%) 2020-11-20T11:34:04.295 INFO:tasks.workunit.client.0.smithi023.stdout: write_fsync = 0 (0.00%) 2020-11-20T11:34:04.296 INFO:tasks.workunit.client.0.smithi023.stdout: create_fsync = 0 (0.00%) 2020-11-20T11:34:04.296 INFO:tasks.workunit.client.0.smithi023.stdout: append_fsync = 0 (0.00%) 2020-11-20T11:34:04.296 INFO:tasks.workunit.client.0.smithi023.stdout: 2020-11-20T11:34:04.296 INFO:tasks.workunit.client.0.smithi023.stdout:FileSystem . 2020-11-20T11:34:04.297 INFO:tasks.workunit.client.0.smithi023.stdout:========== 2020-11-20T11:34:04.297 INFO:tasks.workunit.client.0.smithi023.stdout: num_dirs = 0 2020-11-20T11:34:04.297 INFO:tasks.workunit.client.0.smithi023.stdout: starting files = 128 2020-11-20T11:34:04.298 INFO:tasks.workunit.client.0.smithi023.stdout: 2020-11-20T11:34:04.298 INFO:tasks.workunit.client.0.smithi023.stdout: min file size = 104857600 (100MB) 2020-11-20T11:34:04.298 INFO:tasks.workunit.client.0.smithi023.stdout: max file size = 104857600 (100MB) 2020-11-20T11:34:04.298 INFO:tasks.workunit.client.0.smithi023.stdout: directio = off 2020-11-20T11:34:04.299 INFO:tasks.workunit.client.0.smithi023.stdout: alignedio = on 2020-11-20T11:34:04.299 INFO:tasks.workunit.client.0.smithi023.stdout: bufferedio = off 2020-11-20T11:34:04.299 INFO:tasks.workunit.client.0.smithi023.stdout: 2020-11-20T11:34:04.299 INFO:tasks.workunit.client.0.smithi023.stdout: aging is off 2020-11-20T11:34:04.299 INFO:tasks.workunit.client.0.smithi023.stdout: current utilization = 0.00% 2020-11-20T11:34:04.300 INFO:tasks.workunit.client.0.smithi023.stdout: 2020-11-20T11:34:04.300 INFO:tasks.workunit.client.0.smithi023.stdout:checking existing fs: . 2020-11-20T11:34:04.300 INFO:tasks.workunit.client.0.smithi023.stdout:recreating new fileset 2020-11-20T11:34:04.300 INFO:tasks.workunit.client.0.smithi023.stderr:opendir: No such file or directory 2020-11-20T11:34:16.732 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2020-11-20T11:34:16.732 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2020-11-20T11:34:16.732 INFO:tasks.ceph.mon.b:Restarting daemon 2020-11-20T11:34:16.733 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2020-11-20T11:34:16.735 INFO:tasks.ceph.mon.b:Started 2020-11-20T11:34:16.736 INFO:tasks.mon_thrash:Sending STOP to mon a 2020-11-20T11:34:16.736 INFO:tasks.ceph.mon.a:Sent signal 19 2020-11-20T11:34:16.736 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2020-11-20T11:34:29.564 INFO:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:34:29.566 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:34:31.737 INFO:tasks.mon_thrash:Sending CONT to mon a 2020-11-20T11:34:31.738 INFO:tasks.ceph.mon.a:Sent signal 18 2020-11-20T11:34:31.739 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:34:31.739 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:34:31.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.873+0000 7fbfb8880700 1 Processor -- start 2020-11-20T11:34:31.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.873+0000 7fbfb8880700 1 -- start start 2020-11-20T11:34:31.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.874+0000 7fbfb8880700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fbfb40ad2f0 0x7fbfb413af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:31.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.874+0000 7fbfb8880700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb40ad860 con 0x7fbfb40ace70 2020-11-20T11:34:31.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.874+0000 7fbfb8880700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb40aaa90 con 0x7fbfb40a9040 2020-11-20T11:34:31.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.874+0000 7fbfb8880700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfb40aa8a0 con 0x7fbfb40ad2f0 2020-11-20T11:34:31.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.874+0000 7fbfb359e700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fbfb40ad2f0 0x7fbfb413af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:31.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.874+0000 7fbfb259c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fbfb40ace70 0x7fbfb413a830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:56328/0 (socket says 172.21.15.23:56328) 2020-11-20T11:34:31.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.874+0000 7fbfb259c700 1 -- 172.21.15.23:0/2432962009 learned_addr learned my addr 172.21.15.23:0/2432962009 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:34:31.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.874+0000 7fbfb359e700 1 -- 172.21.15.23:0/2432962009 >> v1:172.21.15.155:6789/0 conn(0x7fbfb40a9040 legacy=0x7fbfb40aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:31.882 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.875+0000 7fbfb1d9b700 1 -- 172.21.15.23:0/2432962009 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3449967075 0 0) 0x7fbfb40aaa90 con 0x7fbfb40a9040 2020-11-20T11:34:31.882 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.875+0000 7fbfb359e700 1 -- 172.21.15.23:0/2432962009 >> v1:172.21.15.23:6789/0 conn(0x7fbfb40ace70 legacy=0x7fbfb413a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:31.882 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.875+0000 7fbfb359e700 1 -- 172.21.15.23:0/2432962009 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb40a9420 con 0x7fbfb40ad2f0 2020-11-20T11:34:31.882 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.875+0000 7fbfb359e700 1 --2- 172.21.15.23:0/2432962009 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbfb40ad2f0 0x7fbfb413af40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fbfac009ce0 tx=0x7fbfac009220).ready entity=mon.0 client_cookie=4de5c04015cba900 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:31.883 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.875+0000 7fbfb1d9b700 1 -- 172.21.15.23:0/2432962009 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfac00baf0 con 0x7fbfb40ad2f0 2020-11-20T11:34:31.883 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.875+0000 7fbfb1d9b700 1 -- 172.21.15.23:0/2432962009 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfac00bc50 con 0x7fbfb40ad2f0 2020-11-20T11:34:31.883 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.875+0000 7fbfb1d9b700 1 -- 172.21.15.23:0/2432962009 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfac011400 con 0x7fbfb40ad2f0 2020-11-20T11:34:31.883 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.875+0000 7fbfb8880700 1 -- 172.21.15.23:0/2432962009 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbfb40ad2f0 msgr2=0x7fbfb413af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:31.883 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.875+0000 7fbfb8880700 1 --2- 172.21.15.23:0/2432962009 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbfb40ad2f0 0x7fbfb413af40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fbfac009ce0 tx=0x7fbfac009220).stop 2020-11-20T11:34:31.884 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.875+0000 7fbfb8880700 1 -- 172.21.15.23:0/2432962009 shutdown_connections 2020-11-20T11:34:31.884 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.875+0000 7fbfb8880700 1 --2- 172.21.15.23:0/2432962009 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbfb40ad2f0 0x7fbfb413af40 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:31.884 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.875+0000 7fbfb8880700 1 -- 172.21.15.23:0/2432962009 >> 172.21.15.23:0/2432962009 conn(0x7fbfb40683b0 msgr2=0x7fbfb40b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:34:31.884 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.875+0000 7fbfb8880700 1 -- 172.21.15.23:0/2432962009 shutdown_connections 2020-11-20T11:34:31.884 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.875+0000 7fbfb8880700 1 -- 172.21.15.23:0/2432962009 wait complete. 2020-11-20T11:34:31.885 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.877+0000 7fbfb8880700 1 Processor -- start 2020-11-20T11:34:31.885 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.877+0000 7fbfb8880700 1 -- start start 2020-11-20T11:34:31.885 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.877+0000 7fbfb8880700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbfb40a9040 0x7fbfb40a1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:31.885 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.877+0000 7fbfb8880700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbfb40ace70 0x7fbfb414a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:31.885 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.877+0000 7fbfb8880700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbfb40ad2f0 0x7fbfb40a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:31.886 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.877+0000 7fbfb8880700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb40a0250 con 0x7fbfb40ace70 2020-11-20T11:34:31.886 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.877+0000 7fbfb8880700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb409e470 con 0x7fbfb40a9040 2020-11-20T11:34:31.886 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.877+0000 7fbfb8880700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb40a1750 con 0x7fbfb40ad2f0 2020-11-20T11:34:31.886 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.877+0000 7fbfb259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbfb40ace70 0x7fbfb414a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:31.888 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.877+0000 7fbfb259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbfb40ace70 0x7fbfb414a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42084/0 (socket says 172.21.15.23:42084) 2020-11-20T11:34:31.888 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.877+0000 7fbfb259c700 1 -- 172.21.15.23:0/2502008283 learned_addr learned my addr 172.21.15.23:0/2502008283 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:34:31.888 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.878+0000 7fbfb2d9d700 1 --2- 172.21.15.23:0/2502008283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbfb40a9040 0x7fbfb40a1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:31.889 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.878+0000 7fbfb359e700 1 --2- 172.21.15.23:0/2502008283 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbfb40ad2f0 0x7fbfb40a0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:31.889 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.878+0000 7fbfb259c700 1 -- 172.21.15.23:0/2502008283 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbfb40ad2f0 msgr2=0x7fbfb40a0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:31.889 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.878+0000 7fbfb259c700 1 --2- 172.21.15.23:0/2502008283 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbfb40ad2f0 0x7fbfb40a0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:31.889 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.878+0000 7fbfb259c700 1 -- 172.21.15.23:0/2502008283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbfb40a9040 msgr2=0x7fbfb40a1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:31.889 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.878+0000 7fbfb259c700 1 --2- 172.21.15.23:0/2502008283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbfb40a9040 0x7fbfb40a1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:31.890 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.878+0000 7fbfb259c700 1 -- 172.21.15.23:0/2502008283 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb4149ed0 con 0x7fbfb40ace70 2020-11-20T11:34:31.890 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.878+0000 7fbfb259c700 1 --2- 172.21.15.23:0/2502008283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbfb40ace70 0x7fbfb414a370 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fbfa800c8b0 tx=0x7fbfa8007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:31.890 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.878+0000 7fbf9affd700 1 -- 172.21.15.23:0/2502008283 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa8007320 con 0x7fbfb40ace70 2020-11-20T11:34:31.890 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.878+0000 7fbf9affd700 1 -- 172.21.15.23:0/2502008283 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa8007480 con 0x7fbfb40ace70 2020-11-20T11:34:31.890 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.878+0000 7fbf9affd700 1 -- 172.21.15.23:0/2502008283 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa8005860 con 0x7fbfb40ace70 2020-11-20T11:34:31.891 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.878+0000 7fbfb8880700 1 -- 172.21.15.23:0/2502008283 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfb4149bc0 con 0x7fbfb40ace70 2020-11-20T11:34:31.891 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.878+0000 7fbfb8880700 1 -- 172.21.15.23:0/2502008283 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfb40a1130 con 0x7fbfb40ace70 2020-11-20T11:34:31.891 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.879+0000 7fbfb8880700 1 -- 172.21.15.23:0/2502008283 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfb41387a0 con 0x7fbfb40ace70 2020-11-20T11:34:31.891 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.880+0000 7fbf9affd700 1 -- 172.21.15.23:0/2502008283 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbfa8010030 con 0x7fbfb40ace70 2020-11-20T11:34:31.891 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.880+0000 7fbf9affd700 1 --2- 172.21.15.23:0/2502008283 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbf90056d60 0x7fbf90059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:31.892 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.880+0000 7fbf9affd700 1 -- 172.21.15.23:0/2502008283 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fbfa806fe20 con 0x7fbfb40ace70 2020-11-20T11:34:31.892 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.880+0000 7fbfb2d9d700 1 --2- 172.21.15.23:0/2502008283 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbf90056d60 0x7fbf90059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:31.892 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.881+0000 7fbfb2d9d700 1 --2- 172.21.15.23:0/2502008283 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbf90056d60 0x7fbf90059220 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fbfa4007fd0 tx=0x7fbfa4007a20).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:31.892 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:31.884+0000 7fbf9affd700 1 -- 172.21.15.23:0/2502008283 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fbfa8045090 con 0x7fbfb40ace70 2020-11-20T11:34:32.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.128+0000 7fbfb8880700 1 -- 172.21.15.23:0/2502008283 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbfb40a1910 con 0x7fbfb40ace70 2020-11-20T11:34:32.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.128+0000 7fbf9affd700 1 -- 172.21.15.23:0/2502008283 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1477 (secure 0 0 0) 0x7fbfa80486b0 con 0x7fbfb40ace70 2020-11-20T11:34:32.133 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:34:32.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.130+0000 7fbfb8880700 1 -- 172.21.15.23:0/2502008283 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbf90056d60 msgr2=0x7fbf90059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:32.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.130+0000 7fbfb8880700 1 --2- 172.21.15.23:0/2502008283 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbf90056d60 0x7fbf90059220 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fbfa4007fd0 tx=0x7fbfa4007a20).stop 2020-11-20T11:34:32.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.130+0000 7fbfb8880700 1 -- 172.21.15.23:0/2502008283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbfb40ace70 msgr2=0x7fbfb414a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:32.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.130+0000 7fbfb8880700 1 --2- 172.21.15.23:0/2502008283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbfb40ace70 0x7fbfb414a370 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fbfa800c8b0 tx=0x7fbfa8007c00).stop 2020-11-20T11:34:32.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.130+0000 7fbfb8880700 1 -- 172.21.15.23:0/2502008283 shutdown_connections 2020-11-20T11:34:32.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.130+0000 7fbfb8880700 1 --2- 172.21.15.23:0/2502008283 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbf90056d60 0x7fbf90059220 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:32.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.130+0000 7fbfb8880700 1 --2- 172.21.15.23:0/2502008283 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbfb40ad2f0 0x7fbfb40a0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:32.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.130+0000 7fbfb8880700 1 --2- 172.21.15.23:0/2502008283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbfb40a9040 0x7fbfb40a1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:32.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.130+0000 7fbfb8880700 1 --2- 172.21.15.23:0/2502008283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbfb40ace70 0x7fbfb414a370 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:32.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.130+0000 7fbfb8880700 1 -- 172.21.15.23:0/2502008283 >> 172.21.15.23:0/2502008283 conn(0x7fbfb40683b0 msgr2=0x7fbfb4137810 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:34:32.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.130+0000 7fbfb8880700 1 -- 172.21.15.23:0/2502008283 shutdown_connections 2020-11-20T11:34:32.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.130+0000 7fbfb8880700 1 -- 172.21.15.23:0/2502008283 wait complete. 2020-11-20T11:34:32.148 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:34:32.148 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:34:32.278 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.274+0000 7f7e18ea3700 1 Processor -- start 2020-11-20T11:34:32.278 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.274+0000 7f7e18ea3700 1 -- start start 2020-11-20T11:34:32.281 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.275+0000 7f7e18ea3700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e140ace70 0x7f7e1413a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:32.281 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.275+0000 7f7e18ea3700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e140ad860 con 0x7f7e140ad2f0 2020-11-20T11:34:32.281 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.275+0000 7f7e18ea3700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e140aaa90 con 0x7f7e140a9040 2020-11-20T11:34:32.281 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.275+0000 7f7e18ea3700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e140aa8a0 con 0x7f7e140ace70 2020-11-20T11:34:32.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.275+0000 7f7e13fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f7e140ad2f0 0x7f7e1413ad20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:56344/0 (socket says 172.21.15.23:56344) 2020-11-20T11:34:32.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.275+0000 7f7e13fff700 1 -- 172.21.15.23:0/2775751415 learned_addr learned my addr 172.21.15.23:0/2775751415 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:34:32.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.275+0000 7f7e12ffd700 1 --2- 172.21.15.23:0/2775751415 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e140ace70 0x7f7e1413a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:32.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.275+0000 7f7e127fc700 1 -- 172.21.15.23:0/2775751415 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 758499421 0 0) 0x7f7e140ad860 con 0x7f7e140ad2f0 2020-11-20T11:34:32.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.275+0000 7f7e127fc700 1 -- 172.21.15.23:0/2775751415 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e00003280 con 0x7f7e140ad2f0 2020-11-20T11:34:32.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.276+0000 7f7e127fc700 1 -- 172.21.15.23:0/2775751415 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2757744447 0 0) 0x7f7e140aaa90 con 0x7f7e140a9040 2020-11-20T11:34:32.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.276+0000 7f7e127fc700 1 -- 172.21.15.23:0/2775751415 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e140ad860 con 0x7f7e140a9040 2020-11-20T11:34:32.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.276+0000 7f7e12ffd700 1 -- 172.21.15.23:0/2775751415 >> v1:172.21.15.155:6789/0 conn(0x7f7e140a9040 legacy=0x7f7e140aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:32.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.276+0000 7f7e12ffd700 1 -- 172.21.15.23:0/2775751415 >> v1:172.21.15.23:6789/0 conn(0x7f7e140ad2f0 legacy=0x7f7e1413ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:32.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.276+0000 7f7e12ffd700 1 -- 172.21.15.23:0/2775751415 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e140a9420 con 0x7f7e140ace70 2020-11-20T11:34:32.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.276+0000 7f7e12ffd700 1 --2- 172.21.15.23:0/2775751415 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e140ace70 0x7f7e1413a830 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f7e0800ba00 tx=0x7f7e08006e30).ready entity=mon.2 client_cookie=2870675bb5625fba server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:32.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.276+0000 7f7e127fc700 1 -- 172.21.15.23:0/2775751415 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e08009630 con 0x7f7e140ace70 2020-11-20T11:34:32.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.276+0000 7f7e127fc700 1 -- 172.21.15.23:0/2775751415 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e08000e90 con 0x7f7e140ace70 2020-11-20T11:34:32.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.276+0000 7f7e127fc700 1 -- 172.21.15.23:0/2775751415 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e080099a0 con 0x7f7e140ace70 2020-11-20T11:34:32.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.276+0000 7f7e18ea3700 1 -- 172.21.15.23:0/2775751415 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e140ace70 msgr2=0x7f7e1413a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:32.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.276+0000 7f7e18ea3700 1 --2- 172.21.15.23:0/2775751415 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e140ace70 0x7f7e1413a830 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f7e0800ba00 tx=0x7f7e08006e30).stop 2020-11-20T11:34:32.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.276+0000 7f7e18ea3700 1 -- 172.21.15.23:0/2775751415 shutdown_connections 2020-11-20T11:34:32.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.276+0000 7f7e18ea3700 1 --2- 172.21.15.23:0/2775751415 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e140ace70 0x7f7e1413a830 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:32.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.276+0000 7f7e18ea3700 1 -- 172.21.15.23:0/2775751415 >> 172.21.15.23:0/2775751415 conn(0x7f7e140683b0 msgr2=0x7f7e140b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:34:32.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.276+0000 7f7e18ea3700 1 -- 172.21.15.23:0/2775751415 shutdown_connections 2020-11-20T11:34:32.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.276+0000 7f7e18ea3700 1 -- 172.21.15.23:0/2775751415 wait complete. 2020-11-20T11:34:32.289 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.278+0000 7f7e18ea3700 1 Processor -- start 2020-11-20T11:34:32.290 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.278+0000 7f7e18ea3700 1 -- start start 2020-11-20T11:34:32.290 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.279+0000 7f7e18ea3700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e140a9040 0x7f7e1413fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:32.290 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.279+0000 7f7e18ea3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e140ace70 0x7f7e1409eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:32.290 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.279+0000 7f7e18ea3700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e140ad2f0 0x7f7e1409f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:32.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.279+0000 7f7e18ea3700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1409f540 con 0x7f7e140ace70 2020-11-20T11:34:32.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.279+0000 7f7e18ea3700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e140a1bb0 con 0x7f7e140a9040 2020-11-20T11:34:32.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.279+0000 7f7e18ea3700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e140a1e50 con 0x7f7e140ad2f0 2020-11-20T11:34:32.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.279+0000 7f7e12ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e140ace70 0x7f7e1409eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:32.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.279+0000 7f7e12ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e140ace70 0x7f7e1409eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42098/0 (socket says 172.21.15.23:42098) 2020-11-20T11:34:32.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.279+0000 7f7e12ffd700 1 -- 172.21.15.23:0/471483929 learned_addr learned my addr 172.21.15.23:0/471483929 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:34:32.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.279+0000 7f7e137fe700 1 --2- 172.21.15.23:0/471483929 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e140a9040 0x7f7e1413fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:32.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.279+0000 7f7e13fff700 1 --2- 172.21.15.23:0/471483929 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e140ad2f0 0x7f7e1409f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:32.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.279+0000 7f7e12ffd700 1 -- 172.21.15.23:0/471483929 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e140ad2f0 msgr2=0x7f7e1409f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:32.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.279+0000 7f7e12ffd700 1 --2- 172.21.15.23:0/471483929 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7e140ad2f0 0x7f7e1409f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:32.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.279+0000 7f7e12ffd700 1 -- 172.21.15.23:0/471483929 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e140a9040 msgr2=0x7f7e1413fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:32.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.279+0000 7f7e12ffd700 1 --2- 172.21.15.23:0/471483929 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7e140a9040 0x7f7e1413fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:32.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.279+0000 7f7e12ffd700 1 -- 172.21.15.23:0/471483929 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e140a20f0 con 0x7f7e140ace70 2020-11-20T11:34:32.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.279+0000 7f7e12ffd700 1 --2- 172.21.15.23:0/471483929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e140ace70 0x7f7e1409eb60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f7e14069780 tx=0x7f7e08015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:32.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.279+0000 7f7df37fe700 1 -- 172.21.15.23:0/471483929 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0801d450 con 0x7f7e140ace70 2020-11-20T11:34:32.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.279+0000 7f7df37fe700 1 -- 172.21.15.23:0/471483929 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e0801d5b0 con 0x7f7e140ace70 2020-11-20T11:34:32.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.279+0000 7f7e18ea3700 1 -- 172.21.15.23:0/471483929 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e140a7980 con 0x7f7e140ace70 2020-11-20T11:34:32.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.279+0000 7f7e18ea3700 1 -- 172.21.15.23:0/471483929 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e14140860 con 0x7f7e140ace70 2020-11-20T11:34:32.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.280+0000 7f7df37fe700 1 -- 172.21.15.23:0/471483929 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0801d950 con 0x7f7e140ace70 2020-11-20T11:34:32.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.280+0000 7f7e18ea3700 1 --2- 172.21.15.23:0/471483929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e14136c20 0x7f7e14137000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:32.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.280+0000 7f7e18ea3700 1 -- 172.21.15.23:0/471483929 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e14069c00 con 0x7f7e14136c20 2020-11-20T11:34:32.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.280+0000 7f7e137fe700 1 --2- 172.21.15.23:0/471483929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e14136c20 0x7f7e14137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:32.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.281+0000 7f7df37fe700 1 -- 172.21.15.23:0/471483929 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7e0800f070 con 0x7f7e140ace70 2020-11-20T11:34:32.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.281+0000 7f7e137fe700 1 -- 172.21.15.23:0/471483929 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7e14057dc0 con 0x7f7e14136c20 2020-11-20T11:34:32.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.281+0000 7f7e137fe700 1 --2- 172.21.15.23:0/471483929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e14136c20 0x7f7e14137000 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f7e0400e660 tx=0x7f7e04009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:32.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.281+0000 7f7df37fe700 1 --2- 172.21.15.23:0/471483929 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7dec056d90 0x7f7dec059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:32.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.281+0000 7f7df37fe700 1 -- 172.21.15.23:0/471483929 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f7e08072010 con 0x7f7e140ace70 2020-11-20T11:34:32.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.281+0000 7f7df37fe700 1 -- 172.21.15.23:0/471483929 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e04010520 con 0x7f7e14136c20 2020-11-20T11:34:32.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.281+0000 7f7e13fff700 1 --2- 172.21.15.23:0/471483929 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7dec056d90 0x7f7dec059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:32.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.282+0000 7f7df37fe700 1 -- 172.21.15.23:0/471483929 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f7e04010680 con 0x7f7e14136c20 2020-11-20T11:34:32.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.282+0000 7f7df37fe700 1 -- 172.21.15.23:0/471483929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e14136c20 msgr2=0x7f7e14137000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:32.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.282+0000 7f7df37fe700 1 --2- 172.21.15.23:0/471483929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e14136c20 0x7f7e14137000 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f7e0400e660 tx=0x7f7e04009040).stop 2020-11-20T11:34:32.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.282+0000 7f7df37fe700 1 -- 172.21.15.23:0/471483929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e14136c20 msgr2=0x7f7e14137000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:34:32.298 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.282+0000 7f7df37fe700 1 --2- 172.21.15.23:0/471483929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e14136c20 0x7f7e14137000 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:32.298 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.282+0000 7f7e13fff700 1 --2- 172.21.15.23:0/471483929 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7dec056d90 0x7f7dec059250 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f7e0c001560 tx=0x7f7e0c009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:32.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.288+0000 7f7e18ea3700 1 --2- 172.21.15.23:0/471483929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e141374f0 0x7f7e1409d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:32.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.288+0000 7f7e18ea3700 1 -- 172.21.15.23:0/471483929 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1405be40 con 0x7f7e141374f0 2020-11-20T11:34:32.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.288+0000 7f7e137fe700 1 --2- 172.21.15.23:0/471483929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e141374f0 0x7f7e1409d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:32.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.289+0000 7f7e137fe700 1 -- 172.21.15.23:0/471483929 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7e14044710 con 0x7f7e141374f0 2020-11-20T11:34:32.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.289+0000 7f7e137fe700 1 --2- 172.21.15.23:0/471483929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e141374f0 0x7f7e1409d4a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f7e0400e660 tx=0x7f7e04009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:32.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.289+0000 7f7df37fe700 1 -- 172.21.15.23:0/471483929 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e04014e10 con 0x7f7e141374f0 2020-11-20T11:34:32.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.289+0000 7f7df37fe700 1 -- 172.21.15.23:0/471483929 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4615 (secure 0 0 0) 0x7f7e04010310 con 0x7f7e141374f0 2020-11-20T11:34:32.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.289+0000 7f7df37fe700 1 -- 172.21.15.23:0/471483929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e141374f0 msgr2=0x7f7e1409d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:32.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.289+0000 7f7df37fe700 1 --2- 172.21.15.23:0/471483929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e141374f0 0x7f7e1409d4a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f7e0400e660 tx=0x7f7e04009040).stop 2020-11-20T11:34:32.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.289+0000 7f7df37fe700 1 -- 172.21.15.23:0/471483929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e141374f0 msgr2=0x7f7e1409d4a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:34:32.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.289+0000 7f7df37fe700 1 --2- 172.21.15.23:0/471483929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e141374f0 0x7f7e1409d4a0 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0x7f7e0400e660 tx=0x7f7e04009040).stop 2020-11-20T11:34:32.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.290+0000 7f7e18ea3700 1 -- 172.21.15.23:0/471483929 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7dec056d90 msgr2=0x7f7dec059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:32.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.290+0000 7f7e18ea3700 1 --2- 172.21.15.23:0/471483929 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7dec056d90 0x7f7dec059250 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f7e0c001560 tx=0x7f7e0c009040).stop 2020-11-20T11:34:32.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.290+0000 7f7e18ea3700 1 -- 172.21.15.23:0/471483929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e140ace70 msgr2=0x7f7e1409eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:32.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.290+0000 7f7e18ea3700 1 --2- 172.21.15.23:0/471483929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e140ace70 0x7f7e1409eb60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f7e14069780 tx=0x7f7e08015000).stop 2020-11-20T11:34:32.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.290+0000 7f7e13fff700 1 -- 172.21.15.23:0/471483929 reap_dead start 2020-11-20T11:34:32.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.290+0000 7f7e18ea3700 1 -- 172.21.15.23:0/471483929 shutdown_connections 2020-11-20T11:34:32.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.290+0000 7f7e18ea3700 1 --2- 172.21.15.23:0/471483929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e140ace70 0x7f7e1409eb60 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:32.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.290+0000 7f7e18ea3700 1 -- 172.21.15.23:0/471483929 >> 172.21.15.23:0/471483929 conn(0x7f7e140683b0 msgr2=0x7f7e140aae50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:34:32.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.290+0000 7f7e18ea3700 1 -- 172.21.15.23:0/471483929 shutdown_connections 2020-11-20T11:34:32.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.290+0000 7f7e18ea3700 1 -- 172.21.15.23:0/471483929 wait complete. 2020-11-20T11:34:32.304 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:34:32.304 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:34:32.304 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:34:32.305 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:34:32.305 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 22, 2020-11-20T11:34:32.305 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:34:32.305 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:34:32.306 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:34:32.306 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:34:32.306 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:32.306 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 0, 2020-11-20T11:34:32.306 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:34:32.307 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:34:32.307 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:34:32.307 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:34:32.307 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:34:32.307 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:34:32.308 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:34:32.308 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:34:32.308 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:34:32.308 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:34:32.308 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:34:32.308 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:32.309 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:34:32.309 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:34:32.309 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:34:32.309 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:34:32.310 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:34:32.310 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:34:32.310 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:34:32.310 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:34:32.310 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:34:32.310 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:34:32.311 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:32.311 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.311 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:34:32.311 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:34:32.311 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:34:32.312 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:34:32.312 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:34:32.312 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:34:32.312 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:34:32.312 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:34:32.313 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:34:32.313 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:34:32.313 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:34:32.313 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:34:32.313 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:34:32.314 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:34:32.314 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:34:32.314 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:34:32.314 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:34:32.314 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:34:32.315 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:34:32.315 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:34:32.315 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:34:32.315 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:34:32.315 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:34:32.316 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:32.316 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:34:32.316 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.316 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:34:32.316 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.317 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:34:32.317 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:34:32.317 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:34:32.317 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:34:32.317 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.318 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:34:32.318 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:34:32.318 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:32.318 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.318 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.319 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:34:32.319 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:34:32.319 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:32.319 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:32.319 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:32.320 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.320 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:34:32.320 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:34:32.320 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:34:32.320 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:34:32.321 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:34:32.321 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.321 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.321 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:34:32.321 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:34:32.322 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:34:32.322 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:34:32.322 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.322 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:34:32.322 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:34:32.322 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:32.323 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.323 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.323 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:34:32.323 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:34:32.323 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:32.324 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:32.324 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:32.324 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.324 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:34:32.324 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:34:32.325 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:34:32.325 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:34:32.325 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:34:32.325 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.325 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.326 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:34:32.326 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:34:32.326 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:34:32.326 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:34:32.326 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.327 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:34:32.327 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:34:32.327 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:32.327 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.327 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.328 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:34:32.328 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:34:32.328 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:32.328 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:32.328 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:32.329 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.329 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:34:32.329 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:34:32.329 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:34:32.329 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:34:32.330 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:34:32.330 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:32.330 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:32.330 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.330 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:34:32.331 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:34:32.331 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.331 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:34:32.331 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:34:32.331 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:34:32.332 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:32.332 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:32.332 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:34:32.332 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.332 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:34:32.333 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:34:32.333 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:34:32.333 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:32.333 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:32.333 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:34:32.334 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.334 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:34:32.334 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:34:32.334 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 4 2020-11-20T11:34:32.334 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:32.335 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:32.335 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:34:32.335 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.335 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:34:32.335 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:34:32.336 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:34:32.336 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:32.336 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:32.336 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.336 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:34:32.337 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:34:32.338 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:34:32.466 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.461+0000 7f27d29ae700 1 Processor -- start 2020-11-20T11:34:32.467 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.461+0000 7f27d29ae700 1 -- start start 2020-11-20T11:34:32.467 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.462+0000 7f27d29ae700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f27cc0aae90 0x7f27cc1381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:32.467 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.462+0000 7f27d29ae700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27cc13aeb0 0x7f27cc0a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:32.467 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.462+0000 7f27d29ae700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27cc13e600 con 0x7f27cc0aaab0 2020-11-20T11:34:32.468 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.462+0000 7f27d29ae700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27cc13e8a0 con 0x7f27cc0aae90 2020-11-20T11:34:32.468 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.462+0000 7f27d29ae700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc0ab6d0 con 0x7f27cc13aeb0 2020-11-20T11:34:32.468 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.462+0000 7f27cb7fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f27cc0aae90 0x7f27cc1381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:32.468 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.462+0000 7f27d0f4b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27cc13aeb0 0x7f27cc0a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:32.468 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.462+0000 7f27d0f4b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27cc13aeb0 0x7f27cc0a8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:52352/0 (socket says 172.21.15.23:52352) 2020-11-20T11:34:32.469 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.462+0000 7f27cb7fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f27cc0aae90 0x7f27cc1381a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:35006/0 (socket says 172.21.15.23:35006) 2020-11-20T11:34:32.469 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.462+0000 7f27d0f4b700 1 -- 172.21.15.23:0/1328602872 learned_addr learned my addr 172.21.15.23:0/1328602872 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:34:32.469 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.462+0000 7f27cbfff700 1 --1- 172.21.15.23:0/1328602872 >> v1:172.21.15.155:6789/0 conn(0x7f27cc0aaab0 0x7f27cc13ed40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:34332/0 (socket says 172.21.15.23:34332) 2020-11-20T11:34:32.469 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.462+0000 7f27d0f4b700 1 -- 172.21.15.23:0/1328602872 >> v2:172.21.15.155:3300/0 conn(0x7f27cc0aae90 msgr2=0x7f27cc1381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:34:32.469 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.462+0000 7f27d0f4b700 1 --2- 172.21.15.23:0/1328602872 >> v2:172.21.15.155:3300/0 conn(0x7f27cc0aae90 0x7f27cc1381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:32.470 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.462+0000 7f27d0f4b700 1 -- 172.21.15.23:0/1328602872 >> v1:172.21.15.155:6789/0 conn(0x7f27cc0aaab0 legacy=0x7f27cc13ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:32.470 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.462+0000 7f27d0f4b700 1 -- 172.21.15.23:0/1328602872 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27cc138690 con 0x7f27cc13aeb0 2020-11-20T11:34:32.470 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.462+0000 7f27cb7fe700 1 --2- 172.21.15.23:0/1328602872 >> v2:172.21.15.155:3300/0 conn(0x7f27cc0aae90 0x7f27cc1381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:34:32.470 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.463+0000 7f27d0f4b700 1 --2- 172.21.15.23:0/1328602872 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27cc13aeb0 0x7f27cc0a8c60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f27c4009d90 tx=0x7f27c40092d0).ready entity=mon.2 client_cookie=52efa06f985eca66 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:32.471 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.463+0000 7f27caffd700 1 -- 172.21.15.23:0/1328602872 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c400bbf0 con 0x7f27cc13aeb0 2020-11-20T11:34:32.471 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.463+0000 7f27caffd700 1 -- 172.21.15.23:0/1328602872 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27c400bd50 con 0x7f27cc13aeb0 2020-11-20T11:34:32.471 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.463+0000 7f27caffd700 1 -- 172.21.15.23:0/1328602872 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c4011610 con 0x7f27cc13aeb0 2020-11-20T11:34:32.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.463+0000 7f27d29ae700 1 -- 172.21.15.23:0/1328602872 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27cc13aeb0 msgr2=0x7f27cc0a8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:32.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.463+0000 7f27d29ae700 1 --2- 172.21.15.23:0/1328602872 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27cc13aeb0 0x7f27cc0a8c60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f27c4009d90 tx=0x7f27c40092d0).stop 2020-11-20T11:34:32.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.463+0000 7f27d29ae700 1 -- 172.21.15.23:0/1328602872 shutdown_connections 2020-11-20T11:34:32.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.463+0000 7f27d29ae700 1 --2- 172.21.15.23:0/1328602872 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27cc13aeb0 0x7f27cc0a8c60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:32.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.463+0000 7f27d29ae700 1 --2- 172.21.15.23:0/1328602872 >> v2:172.21.15.155:3300/0 conn(0x7f27cc0aae90 0x7f27cc1381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:32.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.463+0000 7f27d29ae700 1 -- 172.21.15.23:0/1328602872 >> 172.21.15.23:0/1328602872 conn(0x7f27cc0678e0 msgr2=0x7f27cc067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:34:32.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.463+0000 7f27d29ae700 1 -- 172.21.15.23:0/1328602872 shutdown_connections 2020-11-20T11:34:32.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.463+0000 7f27d29ae700 1 -- 172.21.15.23:0/1328602872 wait complete. 2020-11-20T11:34:32.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.464+0000 7f27d29ae700 1 Processor -- start 2020-11-20T11:34:32.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.465+0000 7f27d29ae700 1 -- start start 2020-11-20T11:34:32.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.465+0000 7f27d29ae700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27cc0aaab0 0x7f27cc14d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:32.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.465+0000 7f27d29ae700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27cc0aae90 0x7f27cc0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:32.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.465+0000 7f27d29ae700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27cc0b0a30 0x7f27cc0af760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:32.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.465+0000 7f27d29ae700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc0b0eb0 con 0x7f27cc0b0a30 2020-11-20T11:34:32.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.465+0000 7f27d29ae700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc0b0050 con 0x7f27cc0aae90 2020-11-20T11:34:32.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.465+0000 7f27d29ae700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc0b02f0 con 0x7f27cc0aaab0 2020-11-20T11:34:32.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.465+0000 7f27d0f4b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27cc0b0a30 0x7f27cc0af760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:32.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.465+0000 7f27d0f4b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27cc0b0a30 0x7f27cc0af760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42118/0 (socket says 172.21.15.23:42118) 2020-11-20T11:34:32.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.465+0000 7f27d0f4b700 1 -- 172.21.15.23:0/3459066314 learned_addr learned my addr 172.21.15.23:0/3459066314 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:34:32.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.465+0000 7f27cb7fe700 1 --2- 172.21.15.23:0/3459066314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27cc0aae90 0x7f27cc0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:32.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.465+0000 7f27cbfff700 1 --2- 172.21.15.23:0/3459066314 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27cc0aaab0 0x7f27cc14d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:32.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.466+0000 7f27d0f4b700 1 -- 172.21.15.23:0/3459066314 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27cc0aaab0 msgr2=0x7f27cc14d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:32.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.466+0000 7f27d0f4b700 1 --2- 172.21.15.23:0/3459066314 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27cc0aaab0 0x7f27cc14d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:32.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.466+0000 7f27d0f4b700 1 -- 172.21.15.23:0/3459066314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27cc0aae90 msgr2=0x7f27cc0a62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:32.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.466+0000 7f27d0f4b700 1 --2- 172.21.15.23:0/3459066314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27cc0aae90 0x7f27cc0a62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:32.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.466+0000 7f27d0f4b700 1 -- 172.21.15.23:0/3459066314 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27cc0b0590 con 0x7f27cc0b0a30 2020-11-20T11:34:32.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.466+0000 7f27d0f4b700 1 --2- 172.21.15.23:0/3459066314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27cc0b0a30 0x7f27cc0af760 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f27c4009310 tx=0x7f27c4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:32.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.466+0000 7f27abfff700 1 -- 172.21.15.23:0/3459066314 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c400bec0 con 0x7f27cc0b0a30 2020-11-20T11:34:32.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.466+0000 7f27abfff700 1 -- 172.21.15.23:0/3459066314 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27c4011e70 con 0x7f27cc0b0a30 2020-11-20T11:34:32.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.466+0000 7f27d29ae700 1 -- 172.21.15.23:0/3459066314 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27cc1362a0 con 0x7f27cc0b0a30 2020-11-20T11:34:32.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.466+0000 7f27abfff700 1 -- 172.21.15.23:0/3459066314 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c4010cf0 con 0x7f27cc0b0a30 2020-11-20T11:34:32.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.466+0000 7f27d29ae700 1 -- 172.21.15.23:0/3459066314 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27cc132620 con 0x7f27cc0b0a30 2020-11-20T11:34:32.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.467+0000 7f27d29ae700 1 --2- 172.21.15.23:0/3459066314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27cc13aeb0 0x7f27cc132810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:32.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.467+0000 7f27d29ae700 1 -- 172.21.15.23:0/3459066314 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc0afe10 con 0x7f27cc13aeb0 2020-11-20T11:34:32.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.467+0000 7f27cbfff700 1 --2- 172.21.15.23:0/3459066314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27cc13aeb0 0x7f27cc132810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:32.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.468+0000 7f27abfff700 1 -- 172.21.15.23:0/3459066314 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f27c4017070 con 0x7f27cc0b0a30 2020-11-20T11:34:32.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.468+0000 7f27cbfff700 1 -- 172.21.15.23:0/3459066314 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f27cc0630f0 con 0x7f27cc13aeb0 2020-11-20T11:34:32.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.468+0000 7f27cbfff700 1 --2- 172.21.15.23:0/3459066314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27cc13aeb0 0x7f27cc132810 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f27bc001620 tx=0x7f27bc007bc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:32.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.468+0000 7f27abfff700 1 --2- 172.21.15.23:0/3459066314 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f27a4056d60 0x7f27a4059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:32.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.468+0000 7f27abfff700 1 -- 172.21.15.23:0/3459066314 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f27c40719b0 con 0x7f27cc0b0a30 2020-11-20T11:34:32.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.468+0000 7f27abfff700 1 -- 172.21.15.23:0/3459066314 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc011590 con 0x7f27cc13aeb0 2020-11-20T11:34:32.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.468+0000 7f27cb7fe700 1 --2- 172.21.15.23:0/3459066314 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f27a4056d60 0x7f27a4059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:32.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.468+0000 7f27abfff700 1 -- 172.21.15.23:0/3459066314 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f27bc0116f0 con 0x7f27cc13aeb0 2020-11-20T11:34:32.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.468+0000 7f27abfff700 1 -- 172.21.15.23:0/3459066314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27cc13aeb0 msgr2=0x7f27cc132810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:32.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.468+0000 7f27abfff700 1 --2- 172.21.15.23:0/3459066314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27cc13aeb0 0x7f27cc132810 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f27bc001620 tx=0x7f27bc007bc0).stop 2020-11-20T11:34:32.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.468+0000 7f27abfff700 1 -- 172.21.15.23:0/3459066314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27cc13aeb0 msgr2=0x7f27cc132810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:34:32.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.468+0000 7f27abfff700 1 --2- 172.21.15.23:0/3459066314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27cc13aeb0 0x7f27cc132810 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:32.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.469+0000 7f27cb7fe700 1 --2- 172.21.15.23:0/3459066314 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f27a4056d60 0x7f27a4059220 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f27cc135df0 tx=0x7f27c0006e30).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:32.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.479+0000 7f27d29ae700 1 --2- 172.21.15.23:0/3459066314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27cc132d00 0x7f27cc14a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:32.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.479+0000 7f27d29ae700 1 -- 172.21.15.23:0/3459066314 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc05c7d0 con 0x7f27cc132d00 2020-11-20T11:34:32.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.480+0000 7f27cbfff700 1 --2- 172.21.15.23:0/3459066314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27cc132d00 0x7f27cc14a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:32.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.481+0000 7f27cbfff700 1 -- 172.21.15.23:0/3459066314 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f27cc044710 con 0x7f27cc132d00 2020-11-20T11:34:32.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.481+0000 7f27cbfff700 1 --2- 172.21.15.23:0/3459066314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27cc132d00 0x7f27cc14a230 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f27bc001620 tx=0x7f27bc017000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:32.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.481+0000 7f27abfff700 1 -- 172.21.15.23:0/3459066314 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc015a30 con 0x7f27cc132d00 2020-11-20T11:34:32.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.482+0000 7f27abfff700 1 -- 172.21.15.23:0/3459066314 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4273 (secure 0 0 0) 0x7f27bc011380 con 0x7f27cc132d00 2020-11-20T11:34:32.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.482+0000 7f27abfff700 1 -- 172.21.15.23:0/3459066314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27cc132d00 msgr2=0x7f27cc14a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:32.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.482+0000 7f27abfff700 1 --2- 172.21.15.23:0/3459066314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27cc132d00 0x7f27cc14a230 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f27bc001620 tx=0x7f27bc017000).stop 2020-11-20T11:34:32.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.482+0000 7f27abfff700 1 -- 172.21.15.23:0/3459066314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27cc132d00 msgr2=0x7f27cc14a230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:34:32.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.482+0000 7f27abfff700 1 --2- 172.21.15.23:0/3459066314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f27cc132d00 0x7f27cc14a230 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:32.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.483+0000 7f27d29ae700 1 -- 172.21.15.23:0/3459066314 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f27a4056d60 msgr2=0x7f27a4059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:32.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.483+0000 7f27d29ae700 1 --2- 172.21.15.23:0/3459066314 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f27a4056d60 0x7f27a4059220 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f27cc135df0 tx=0x7f27c0006e30).stop 2020-11-20T11:34:32.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.483+0000 7f27d29ae700 1 -- 172.21.15.23:0/3459066314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27cc0b0a30 msgr2=0x7f27cc0af760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:32.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.483+0000 7f27d29ae700 1 --2- 172.21.15.23:0/3459066314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27cc0b0a30 0x7f27cc0af760 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f27c4009310 tx=0x7f27c4016000).stop 2020-11-20T11:34:32.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.483+0000 7f27d0f4b700 1 -- 172.21.15.23:0/3459066314 reap_dead start 2020-11-20T11:34:32.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.483+0000 7f27d0f4b700 1 -- 172.21.15.23:0/3459066314 reap_dead start 2020-11-20T11:34:32.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.483+0000 7f27d29ae700 1 -- 172.21.15.23:0/3459066314 shutdown_connections 2020-11-20T11:34:32.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.483+0000 7f27d29ae700 1 -- 172.21.15.23:0/3459066314 >> 172.21.15.23:0/3459066314 conn(0x7f27cc0678e0 msgr2=0x7f27cc068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:34:32.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.483+0000 7f27d29ae700 1 -- 172.21.15.23:0/3459066314 shutdown_connections 2020-11-20T11:34:32.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.483+0000 7f27d29ae700 1 -- 172.21.15.23:0/3459066314 wait complete. 2020-11-20T11:34:32.491 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:34:32.491 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:34:32.491 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:34:32.491 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:34:32.491 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 22, 2020-11-20T11:34:32.492 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:34:32.492 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:34:32.492 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:34:32.492 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:34:32.493 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:32.493 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 0, 2020-11-20T11:34:32.493 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:34:32.493 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:34:32.493 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:34:32.494 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:34:32.494 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:34:32.494 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:34:32.494 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:34:32.494 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:34:32.495 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:34:32.495 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:34:32.495 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:34:32.495 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:32.495 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:34:32.496 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:34:32.496 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:34:32.496 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:34:32.496 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:34:32.496 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:34:32.497 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:34:32.497 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:34:32.497 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:34:32.497 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:34:32.497 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:32.498 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.498 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:34:32.498 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:34:32.498 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:34:32.498 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:34:32.498 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:34:32.499 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:34:32.499 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:34:32.499 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:34:32.499 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:34:32.499 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:34:32.500 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:34:32.500 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:34:32.500 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:34:32.500 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:34:32.500 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:34:32.501 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:34:32.501 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:34:32.501 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:34:32.501 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:34:32.501 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:34:32.502 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:34:32.502 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:34:32.502 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:34:32.502 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:32.502 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:34:32.503 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.503 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:34:32.503 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.503 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:34:32.503 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:34:32.504 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:34:32.504 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:34:32.504 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.504 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:34:32.504 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:34:32.504 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:32.505 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.505 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.505 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:34:32.505 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:34:32.505 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:32.506 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:32.506 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:32.506 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.506 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:34:32.506 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:34:32.507 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:34:32.507 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:34:32.507 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:34:32.507 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.507 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.508 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:34:32.508 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:34:32.508 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:34:32.508 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:34:32.508 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.509 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:34:32.509 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:34:32.509 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:32.509 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.510 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.510 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:34:32.510 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:34:32.510 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:32.511 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:32.511 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:32.511 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.511 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:34:32.511 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:34:32.511 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:34:32.512 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:34:32.512 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:34:32.512 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.512 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.512 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:34:32.513 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:34:32.513 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:34:32.513 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:34:32.513 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.513 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:34:32.514 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:34:32.514 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:32.514 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.514 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.514 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:34:32.515 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:34:32.515 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:32.515 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:32.515 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:32.515 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.516 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:34:32.516 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:34:32.516 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:34:32.516 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:34:32.516 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:34:32.516 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:32.517 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:32.517 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.517 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:34:32.517 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:34:32.517 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.518 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:34:32.518 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:34:32.518 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:34:32.518 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:32.518 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:32.519 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:34:32.519 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.519 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:34:32.519 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:34:32.519 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:34:32.520 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:32.520 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:32.520 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.520 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:34:32.520 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:34:32.522 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:34:32.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.640+0000 7ff4a9f9e700 1 Processor -- start 2020-11-20T11:34:32.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.640+0000 7ff4a9f9e700 1 -- start start 2020-11-20T11:34:32.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.640+0000 7ff4a9f9e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff4a40a4e00 0x7ff4a4136480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:32.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.640+0000 7ff4a9f9e700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff4a4136970 0x7ff4a4136e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:32.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.640+0000 7ff4a9f9e700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4a4137310 con 0x7ff4a4061380 2020-11-20T11:34:32.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.640+0000 7ff4a9f9e700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4a40a75a0 con 0x7ff4a4136970 2020-11-20T11:34:32.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.640+0000 7ff4a9f9e700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4a40a6e60 con 0x7ff4a40a4e00 2020-11-20T11:34:32.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.640+0000 7ff4a8f9c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7ff4a4061380 0x7ff4a40a9210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:56376/0 (socket says 172.21.15.23:56376) 2020-11-20T11:34:32.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.641+0000 7ff4a8f9c700 1 -- 172.21.15.23:0/2585722472 learned_addr learned my addr 172.21.15.23:0/2585722472 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:34:32.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.641+0000 7ff4a979d700 1 --2- 172.21.15.23:0/2585722472 >> v2:172.21.15.23:3300/0 conn(0x7ff4a4136970 0x7ff4a4136e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:32.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.641+0000 7ff4a3fff700 1 --2- 172.21.15.23:0/2585722472 >> v2:172.21.15.155:3300/0 conn(0x7ff4a40a4e00 0x7ff4a4136480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:32.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.641+0000 7ff4a37fe700 1 -- 172.21.15.23:0/2585722472 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197536001 0 0) 0x7ff4a4137310 con 0x7ff4a4061380 2020-11-20T11:34:32.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.641+0000 7ff4a37fe700 1 -- 172.21.15.23:0/2585722472 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff488003280 con 0x7ff4a4061380 2020-11-20T11:34:32.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.641+0000 7ff4a979d700 1 -- 172.21.15.23:0/2585722472 >> v2:172.21.15.155:3300/0 conn(0x7ff4a40a4e00 msgr2=0x7ff4a4136480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:34:32.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.641+0000 7ff4a979d700 1 --2- 172.21.15.23:0/2585722472 >> v2:172.21.15.155:3300/0 conn(0x7ff4a40a4e00 0x7ff4a4136480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:32.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.641+0000 7ff4a979d700 1 -- 172.21.15.23:0/2585722472 >> v1:172.21.15.23:6789/0 conn(0x7ff4a4061380 legacy=0x7ff4a40a9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:32.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.641+0000 7ff4a979d700 1 -- 172.21.15.23:0/2585722472 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a40a7100 con 0x7ff4a4136970 2020-11-20T11:34:32.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.641+0000 7ff4a3fff700 1 --2- 172.21.15.23:0/2585722472 >> v2:172.21.15.155:3300/0 conn(0x7ff4a40a4e00 0x7ff4a4136480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:34:32.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.641+0000 7ff4a979d700 1 --2- 172.21.15.23:0/2585722472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4a4136970 0x7ff4a4136e20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7ff4a40623a0 tx=0x7ff49c009220).ready entity=mon.0 client_cookie=40f60870ad741d7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:32.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.641+0000 7ff4a37fe700 1 -- 172.21.15.23:0/2585722472 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49c00ba00 con 0x7ff4a4136970 2020-11-20T11:34:32.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.641+0000 7ff4a37fe700 1 -- 172.21.15.23:0/2585722472 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff49c00bb60 con 0x7ff4a4136970 2020-11-20T11:34:32.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.641+0000 7ff4a37fe700 1 -- 172.21.15.23:0/2585722472 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49c011400 con 0x7ff4a4136970 2020-11-20T11:34:32.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.642+0000 7ff4a9f9e700 1 -- 172.21.15.23:0/2585722472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4a4136970 msgr2=0x7ff4a4136e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:32.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.642+0000 7ff4a9f9e700 1 --2- 172.21.15.23:0/2585722472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4a4136970 0x7ff4a4136e20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7ff4a40623a0 tx=0x7ff49c009220).stop 2020-11-20T11:34:32.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.642+0000 7ff4a9f9e700 1 -- 172.21.15.23:0/2585722472 shutdown_connections 2020-11-20T11:34:32.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.642+0000 7ff4a9f9e700 1 --2- 172.21.15.23:0/2585722472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4a4136970 0x7ff4a4136e20 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:32.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.642+0000 7ff4a9f9e700 1 --2- 172.21.15.23:0/2585722472 >> v2:172.21.15.155:3300/0 conn(0x7ff4a40a4e00 0x7ff4a4136480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:32.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.642+0000 7ff4a9f9e700 1 -- 172.21.15.23:0/2585722472 >> 172.21.15.23:0/2585722472 conn(0x7ff4a40ae0a0 msgr2=0x7ff4a40aa0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:34:32.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.642+0000 7ff4a9f9e700 1 -- 172.21.15.23:0/2585722472 shutdown_connections 2020-11-20T11:34:32.652 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.642+0000 7ff4a9f9e700 1 -- 172.21.15.23:0/2585722472 wait complete. 2020-11-20T11:34:32.654 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.643+0000 7ff4a9f9e700 1 Processor -- start 2020-11-20T11:34:32.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.643+0000 7ff4a9f9e700 1 -- start start 2020-11-20T11:34:32.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.644+0000 7ff4a9f9e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff4a4061380 0x7ff4a40a1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:32.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.644+0000 7ff4a9f9e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4a40a4e00 0x7ff4a40655d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:32.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.644+0000 7ff4a9f9e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff4a4136970 0x7ff4a4135030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:32.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.644+0000 7ff4a9f9e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a4138a70 con 0x7ff4a40a4e00 2020-11-20T11:34:32.656 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.644+0000 7ff4a9f9e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a40a4480 con 0x7ff4a4136970 2020-11-20T11:34:32.656 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.644+0000 7ff4a9f9e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a4138ca0 con 0x7ff4a4061380 2020-11-20T11:34:32.656 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.644+0000 7ff4a3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4a40a4e00 0x7ff4a40655d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:32.656 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.644+0000 7ff4a3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4a40a4e00 0x7ff4a40655d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42134/0 (socket says 172.21.15.23:42134) 2020-11-20T11:34:32.657 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.644+0000 7ff4a3fff700 1 -- 172.21.15.23:0/61988948 learned_addr learned my addr 172.21.15.23:0/61988948 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:34:32.657 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.644+0000 7ff4a979d700 1 --2- 172.21.15.23:0/61988948 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff4a4136970 0x7ff4a4135030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:32.657 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.644+0000 7ff4a8f9c700 1 --2- 172.21.15.23:0/61988948 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff4a4061380 0x7ff4a40a1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:32.657 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.644+0000 7ff4a3fff700 1 -- 172.21.15.23:0/61988948 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff4a4061380 msgr2=0x7ff4a40a1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:32.657 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.644+0000 7ff4a3fff700 1 --2- 172.21.15.23:0/61988948 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff4a4061380 0x7ff4a40a1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:32.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.644+0000 7ff4a3fff700 1 -- 172.21.15.23:0/61988948 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff4a4136970 msgr2=0x7ff4a4135030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:32.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.644+0000 7ff4a3fff700 1 --2- 172.21.15.23:0/61988948 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff4a4136970 0x7ff4a4135030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:32.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.644+0000 7ff4a3fff700 1 -- 172.21.15.23:0/61988948 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a4138f40 con 0x7ff4a40a4e00 2020-11-20T11:34:32.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.644+0000 7ff4a8f9c700 1 --2- 172.21.15.23:0/61988948 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff4a4061380 0x7ff4a40a1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:34:32.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.644+0000 7ff4a3fff700 1 --2- 172.21.15.23:0/61988948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4a40a4e00 0x7ff4a40655d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7ff49000db30 tx=0x7ff490008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:32.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.645+0000 7ff4a0ff9700 1 -- 172.21.15.23:0/61988948 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4900097f0 con 0x7ff4a40a4e00 2020-11-20T11:34:32.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.645+0000 7ff4a0ff9700 1 -- 172.21.15.23:0/61988948 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff490009950 con 0x7ff4a40a4e00 2020-11-20T11:34:32.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.645+0000 7ff4a9f9e700 1 -- 172.21.15.23:0/61988948 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4a4065ac0 con 0x7ff4a40a4e00 2020-11-20T11:34:32.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.645+0000 7ff4a0ff9700 1 -- 172.21.15.23:0/61988948 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff490009cd0 con 0x7ff4a40a4e00 2020-11-20T11:34:32.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.645+0000 7ff4a9f9e700 1 -- 172.21.15.23:0/61988948 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4a4139910 con 0x7ff4a40a4e00 2020-11-20T11:34:32.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.646+0000 7ff4a0ff9700 1 -- 172.21.15.23:0/61988948 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff490013690 con 0x7ff4a40a4e00 2020-11-20T11:34:32.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.646+0000 7ff4a9f9e700 1 --2- 172.21.15.23:0/61988948 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff4a4061980 0x7ff4a4139140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:32.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.646+0000 7ff4a9f9e700 1 -- 172.21.15.23:0/61988948 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a4069c70 con 0x7ff4a4061980 2020-11-20T11:34:32.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.647+0000 7ff4a0ff9700 1 --2- 172.21.15.23:0/61988948 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff47c056d60 0x7ff47c059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:32.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.647+0000 7ff4a0ff9700 1 -- 172.21.15.23:0/61988948 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7ff49006fea0 con 0x7ff4a40a4e00 2020-11-20T11:34:32.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.647+0000 7ff4a979d700 1 --2- 172.21.15.23:0/61988948 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff47c056d60 0x7ff47c059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:32.661 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.647+0000 7ff4a8f9c700 1 --2- 172.21.15.23:0/61988948 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff4a4061980 0x7ff4a4139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:32.661 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.647+0000 7ff4a8f9c700 1 -- 172.21.15.23:0/61988948 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff4a4139630 con 0x7ff4a4061980 2020-11-20T11:34:32.661 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.647+0000 7ff4a979d700 1 --2- 172.21.15.23:0/61988948 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff47c056d60 0x7ff47c059220 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7ff49c0090b0 tx=0x7ff49c016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:32.661 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.647+0000 7ff4a8f9c700 1 --2- 172.21.15.23:0/61988948 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff4a4061980 0x7ff4a4139140 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7ff49800d7a0 tx=0x7ff498009040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:32.661 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.648+0000 7ff4a0ff9700 1 -- 172.21.15.23:0/61988948 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff498007320 con 0x7ff4a4061980 2020-11-20T11:34:32.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.648+0000 7ff4a0ff9700 1 -- 172.21.15.23:0/61988948 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7ff498007480 con 0x7ff4a4061980 2020-11-20T11:34:32.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.648+0000 7ff4a0ff9700 1 -- 172.21.15.23:0/61988948 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff4a4061980 msgr2=0x7ff4a4139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:32.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.648+0000 7ff4a0ff9700 1 --2- 172.21.15.23:0/61988948 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff4a4061980 0x7ff4a4139140 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7ff49800d7a0 tx=0x7ff498009040).stop 2020-11-20T11:34:32.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.648+0000 7ff4a0ff9700 1 -- 172.21.15.23:0/61988948 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff4a4061980 msgr2=0x7ff4a4139140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:34:32.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.648+0000 7ff4a0ff9700 1 --2- 172.21.15.23:0/61988948 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff4a4061980 0x7ff4a4139140 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:32.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.656+0000 7ff4a9f9e700 1 --2- 172.21.15.23:0/61988948 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff4a413a360 0x7ff4a4135520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:32.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.656+0000 7ff4a9f9e700 1 -- 172.21.15.23:0/61988948 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a4057dc0 con 0x7ff4a413a360 2020-11-20T11:34:32.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.656+0000 7ff4a8f9c700 1 --2- 172.21.15.23:0/61988948 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff4a413a360 0x7ff4a4135520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:32.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.657+0000 7ff4a8f9c700 1 -- 172.21.15.23:0/61988948 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff4a4044710 con 0x7ff4a413a360 2020-11-20T11:34:32.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.657+0000 7ff4a8f9c700 1 --2- 172.21.15.23:0/61988948 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff4a413a360 0x7ff4a4135520 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7ff498014000 tx=0x7ff498009040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:32.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.657+0000 7ff4a0ff9700 1 -- 172.21.15.23:0/61988948 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff498013c70 con 0x7ff4a413a360 2020-11-20T11:34:32.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.657+0000 7ff4a0ff9700 1 -- 172.21.15.23:0/61988948 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4783 (secure 0 0 0) 0x7ff498007110 con 0x7ff4a413a360 2020-11-20T11:34:32.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.657+0000 7ff4a0ff9700 1 -- 172.21.15.23:0/61988948 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff4a413a360 msgr2=0x7ff4a4135520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:32.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.657+0000 7ff4a0ff9700 1 --2- 172.21.15.23:0/61988948 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff4a413a360 0x7ff4a4135520 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7ff498014000 tx=0x7ff498009040).stop 2020-11-20T11:34:32.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.657+0000 7ff4a0ff9700 1 -- 172.21.15.23:0/61988948 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff4a413a360 msgr2=0x7ff4a4135520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:34:32.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.657+0000 7ff4a0ff9700 1 --2- 172.21.15.23:0/61988948 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff4a413a360 0x7ff4a4135520 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:32.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.658+0000 7ff4a9f9e700 1 -- 172.21.15.23:0/61988948 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff47c056d60 msgr2=0x7ff47c059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:32.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.658+0000 7ff4a9f9e700 1 --2- 172.21.15.23:0/61988948 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff47c056d60 0x7ff47c059220 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7ff49c0090b0 tx=0x7ff49c016000).stop 2020-11-20T11:34:32.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.658+0000 7ff4a9f9e700 1 -- 172.21.15.23:0/61988948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4a40a4e00 msgr2=0x7ff4a40655d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:32.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.658+0000 7ff4a9f9e700 1 --2- 172.21.15.23:0/61988948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4a40a4e00 0x7ff4a40655d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7ff49000db30 tx=0x7ff490008040).stop 2020-11-20T11:34:32.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.658+0000 7ff4a979d700 1 -- 172.21.15.23:0/61988948 reap_dead start 2020-11-20T11:34:32.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.658+0000 7ff4a979d700 1 -- 172.21.15.23:0/61988948 reap_dead start 2020-11-20T11:34:32.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.658+0000 7ff4a9f9e700 1 -- 172.21.15.23:0/61988948 shutdown_connections 2020-11-20T11:34:32.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.658+0000 7ff4a9f9e700 1 -- 172.21.15.23:0/61988948 >> 172.21.15.23:0/61988948 conn(0x7ff4a40ae0a0 msgr2=0x7ff4a40677e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:34:32.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.658+0000 7ff4a9f9e700 1 -- 172.21.15.23:0/61988948 shutdown_connections 2020-11-20T11:34:32.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.658+0000 7ff4a9f9e700 1 -- 172.21.15.23:0/61988948 wait complete. 2020-11-20T11:34:32.668 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:34:32.669 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:34:32.669 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:34:32.669 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:34:32.669 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 22, 2020-11-20T11:34:32.669 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:34:32.670 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:34:32.670 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:34:32.670 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:34:32.670 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:32.670 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 0, 2020-11-20T11:34:32.670 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:34:32.671 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:34:32.671 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:34:32.671 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:34:32.671 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:34:32.671 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:34:32.672 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:34:32.672 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:34:32.672 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:34:32.672 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:34:32.672 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:34:32.672 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:32.673 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:34:32.673 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:34:32.673 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:34:32.673 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:34:32.673 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:34:32.674 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:34:32.674 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:34:32.674 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:34:32.674 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:34:32.674 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:34:32.675 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:32.675 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.675 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:34:32.675 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:34:32.675 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:34:32.676 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:34:32.676 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:34:32.676 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:34:32.676 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:34:32.676 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:34:32.677 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:34:32.677 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:34:32.677 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:34:32.677 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:34:32.677 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:34:32.678 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:34:32.678 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:34:32.678 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:34:32.678 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:34:32.678 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:34:32.679 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:34:32.679 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:34:32.679 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:34:32.679 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:34:32.679 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:34:32.680 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:32.680 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:34:32.680 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.680 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:34:32.680 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.681 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:34:32.681 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:34:32.681 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:34:32.681 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:34:32.681 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.682 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:34:32.682 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:34:32.682 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:32.682 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.682 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.682 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:34:32.683 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:34:32.683 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:32.683 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:32.683 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:32.683 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.684 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:34:32.684 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:34:32.684 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:34:32.684 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:34:32.684 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:34:32.685 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.685 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.685 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:34:32.685 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:34:32.685 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:34:32.686 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:34:32.686 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.686 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:34:32.686 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:34:32.686 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:32.687 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.687 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.687 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:34:32.687 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:34:32.687 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:32.688 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:32.688 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:32.688 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.688 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:34:32.688 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:34:32.688 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:34:32.689 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:34:32.689 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:34:32.689 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.689 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.690 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:34:32.690 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:34:32.690 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:34:32.690 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:34:32.690 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.690 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:34:32.691 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:34:32.691 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:32.691 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.691 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.691 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:34:32.692 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:34:32.692 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:32.692 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:32.692 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:32.692 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.693 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:34:32.693 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:34:32.693 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:34:32.693 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:34:32.693 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:34:32.694 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:32.694 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:32.694 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.694 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:34:32.694 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:34:32.695 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.695 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:34:32.695 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:34:32.695 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:34:32.695 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:32.695 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:32.696 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:34:32.696 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.696 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:34:32.696 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:34:32.696 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:34:32.697 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:32.697 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:32.697 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:34:32.697 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.698 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:34:32.698 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:34:32.698 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 6 2020-11-20T11:34:32.698 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:32.698 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:32.699 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:34:32.699 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.699 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:34:32.699 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:34:32.699 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:34:32.700 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:32.700 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:32.700 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:34:32.700 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:32.700 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:34:32.701 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:34:32.701 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:34:32.701 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:32.701 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:32.701 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:32.702 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:34:32.702 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:34:32.703 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2020-11-20T11:34:32.703 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2020-11-20T11:34:32.834 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.830+0000 7fb805ae1700 1 Processor -- start 2020-11-20T11:34:32.834 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.830+0000 7fb805ae1700 1 -- start start 2020-11-20T11:34:32.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.831+0000 7fb805ae1700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb8000aae90 0x7fb8001381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:32.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.831+0000 7fb805ae1700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb80013aeb0 0x7fb8000a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:32.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.831+0000 7fb805ae1700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb80013e600 con 0x7fb8000aaab0 2020-11-20T11:34:32.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.831+0000 7fb805ae1700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb80013e8a0 con 0x7fb8000aae90 2020-11-20T11:34:32.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.831+0000 7fb805ae1700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8000ab6d0 con 0x7fb80013aeb0 2020-11-20T11:34:32.837 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.831+0000 7fb7fffff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb80013aeb0 0x7fb8000a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:32.837 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.831+0000 7fb7f7fff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb8000aae90 0x7fb8001381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:32.837 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.831+0000 7fb7fffff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb80013aeb0 0x7fb8000a8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:52388/0 (socket says 172.21.15.23:52388) 2020-11-20T11:34:32.837 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.831+0000 7fb7fffff700 1 -- 172.21.15.23:0/1213527758 learned_addr learned my addr 172.21.15.23:0/1213527758 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:34:32.838 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.831+0000 7fb7fffff700 1 -- 172.21.15.23:0/1213527758 >> v2:172.21.15.155:3300/0 conn(0x7fb8000aae90 msgr2=0x7fb8001381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:34:32.838 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.831+0000 7fb7fffff700 1 --2- 172.21.15.23:0/1213527758 >> v2:172.21.15.155:3300/0 conn(0x7fb8000aae90 0x7fb8001381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:32.838 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.831+0000 7fb7fffff700 1 -- 172.21.15.23:0/1213527758 >> v1:172.21.15.155:6789/0 conn(0x7fb8000aaab0 legacy=0x7fb80013ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:32.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.831+0000 7fb7fffff700 1 -- 172.21.15.23:0/1213527758 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb800138690 con 0x7fb80013aeb0 2020-11-20T11:34:32.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.831+0000 7fb7f7fff700 1 --2- 172.21.15.23:0/1213527758 >> v2:172.21.15.155:3300/0 conn(0x7fb8000aae90 0x7fb8001381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:34:32.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.832+0000 7fb7fffff700 1 --2- 172.21.15.23:0/1213527758 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb80013aeb0 0x7fb8000a8c60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fb7f8009d90 tx=0x7fb7f80092d0).ready entity=mon.2 client_cookie=5e4c4368488b5d08 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:32.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.832+0000 7fb7feffd700 1 -- 172.21.15.23:0/1213527758 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f800baf0 con 0x7fb80013aeb0 2020-11-20T11:34:32.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.832+0000 7fb7feffd700 1 -- 172.21.15.23:0/1213527758 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f800bc50 con 0x7fb80013aeb0 2020-11-20T11:34:32.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.832+0000 7fb7feffd700 1 -- 172.21.15.23:0/1213527758 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f8011400 con 0x7fb80013aeb0 2020-11-20T11:34:32.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.832+0000 7fb805ae1700 1 -- 172.21.15.23:0/1213527758 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb80013aeb0 msgr2=0x7fb8000a8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:32.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.832+0000 7fb805ae1700 1 --2- 172.21.15.23:0/1213527758 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb80013aeb0 0x7fb8000a8c60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fb7f8009d90 tx=0x7fb7f80092d0).stop 2020-11-20T11:34:32.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.832+0000 7fb805ae1700 1 -- 172.21.15.23:0/1213527758 shutdown_connections 2020-11-20T11:34:32.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.832+0000 7fb805ae1700 1 --2- 172.21.15.23:0/1213527758 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb80013aeb0 0x7fb8000a8c60 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:32.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.832+0000 7fb805ae1700 1 --2- 172.21.15.23:0/1213527758 >> v2:172.21.15.155:3300/0 conn(0x7fb8000aae90 0x7fb8001381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:32.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.832+0000 7fb805ae1700 1 -- 172.21.15.23:0/1213527758 >> 172.21.15.23:0/1213527758 conn(0x7fb8000678e0 msgr2=0x7fb800067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:34:32.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.832+0000 7fb805ae1700 1 -- 172.21.15.23:0/1213527758 shutdown_connections 2020-11-20T11:34:32.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.832+0000 7fb805ae1700 1 -- 172.21.15.23:0/1213527758 wait complete. 2020-11-20T11:34:32.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.834+0000 7fb805ae1700 1 Processor -- start 2020-11-20T11:34:32.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.834+0000 7fb805ae1700 1 -- start start 2020-11-20T11:34:32.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.834+0000 7fb805ae1700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8000aaab0 0x7fb8000b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:32.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.834+0000 7fb805ae1700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8000aae90 0x7fb8000a68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:32.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.834+0000 7fb805ae1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8000a6390 0x7fb800131660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:32.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.834+0000 7fb805ae1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80013d100 con 0x7fb8000a6390 2020-11-20T11:34:32.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.834+0000 7fb805ae1700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80013d3f0 con 0x7fb8000aae90 2020-11-20T11:34:32.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.834+0000 7fb805ae1700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb80013d690 con 0x7fb8000aaab0 2020-11-20T11:34:32.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.834+0000 7fb7fffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8000a6390 0x7fb800131660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:32.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.834+0000 7fb7fffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8000a6390 0x7fb800131660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42154/0 (socket says 172.21.15.23:42154) 2020-11-20T11:34:32.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.834+0000 7fb7fffff700 1 -- 172.21.15.23:0/2287060785 learned_addr learned my addr 172.21.15.23:0/2287060785 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:34:32.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.834+0000 7fb7ff7fe700 1 --2- 172.21.15.23:0/2287060785 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8000aaab0 0x7fb8000b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:32.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.835+0000 7fb7f7fff700 1 --2- 172.21.15.23:0/2287060785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8000aae90 0x7fb8000a68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:32.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.835+0000 7fb7fffff700 1 -- 172.21.15.23:0/2287060785 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8000aaab0 msgr2=0x7fb8000b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:32.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.835+0000 7fb7fffff700 1 --2- 172.21.15.23:0/2287060785 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8000aaab0 0x7fb8000b0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:32.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.835+0000 7fb7fffff700 1 -- 172.21.15.23:0/2287060785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8000aae90 msgr2=0x7fb8000a68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:32.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.835+0000 7fb7fffff700 1 --2- 172.21.15.23:0/2287060785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8000aae90 0x7fb8000a68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:32.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.835+0000 7fb7fffff700 1 -- 172.21.15.23:0/2287060785 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb80013d930 con 0x7fb8000a6390 2020-11-20T11:34:32.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.835+0000 7fb7fffff700 1 --2- 172.21.15.23:0/2287060785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8000a6390 0x7fb800131660 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fb8000ab810 tx=0x7fb7f8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:32.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.835+0000 7fb7f77fe700 1 -- 172.21.15.23:0/2287060785 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f801d480 con 0x7fb8000a6390 2020-11-20T11:34:32.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.835+0000 7fb7f77fe700 1 -- 172.21.15.23:0/2287060785 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f801d5e0 con 0x7fb8000a6390 2020-11-20T11:34:32.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.835+0000 7fb7f77fe700 1 -- 172.21.15.23:0/2287060785 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f801d980 con 0x7fb8000a6390 2020-11-20T11:34:32.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.835+0000 7fb805ae1700 1 -- 172.21.15.23:0/2287060785 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb80013c2b0 con 0x7fb8000a6390 2020-11-20T11:34:32.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.835+0000 7fb805ae1700 1 -- 172.21.15.23:0/2287060785 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb800132a40 con 0x7fb8000a6390 2020-11-20T11:34:32.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.836+0000 7fb805ae1700 1 -- 172.21.15.23:0/2287060785 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb80005c7d0 con 0x7fb8000a6390 2020-11-20T11:34:32.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.837+0000 7fb7f77fe700 1 -- 172.21.15.23:0/2287060785 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb7f8017070 con 0x7fb8000a6390 2020-11-20T11:34:32.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.837+0000 7fb7f77fe700 1 --2- 172.21.15.23:0/2287060785 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb7dc056d60 0x7fb7dc059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:32.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.837+0000 7fb7f77fe700 1 -- 172.21.15.23:0/2287060785 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fb7f80721d0 con 0x7fb8000a6390 2020-11-20T11:34:32.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.837+0000 7fb7ff7fe700 1 --2- 172.21.15.23:0/2287060785 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb7dc056d60 0x7fb7dc059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:32.850 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.838+0000 7fb7ff7fe700 1 --2- 172.21.15.23:0/2287060785 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb7dc056d60 0x7fb7dc059220 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fb7f0000f60 tx=0x7fb7f0007bb0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:32.850 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:32.841+0000 7fb7f77fe700 1 -- 172.21.15.23:0/2287060785 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb7f8047500 con 0x7fb8000a6390 2020-11-20T11:34:33.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:33.074+0000 7fb805ae1700 1 -- 172.21.15.23:0/2287060785 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fb80013dac0 con 0x7fb8000a6390 2020-11-20T11:34:33.425 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:33.075+0000 7fb7f77fe700 1 -- 172.21.15.23:0/2287060785 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fb7f8015020 con 0x7fb8000a6390 2020-11-20T11:34:33.426 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:33.077+0000 7fb805ae1700 1 -- 172.21.15.23:0/2287060785 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb7dc056d60 msgr2=0x7fb7dc059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:33.426 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:33.077+0000 7fb805ae1700 1 --2- 172.21.15.23:0/2287060785 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb7dc056d60 0x7fb7dc059220 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fb7f0000f60 tx=0x7fb7f0007bb0).stop 2020-11-20T11:34:33.426 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:33.077+0000 7fb805ae1700 1 -- 172.21.15.23:0/2287060785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8000a6390 msgr2=0x7fb800131660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:33.426 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:33.077+0000 7fb805ae1700 1 --2- 172.21.15.23:0/2287060785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8000a6390 0x7fb800131660 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fb8000ab810 tx=0x7fb7f8016000).stop 2020-11-20T11:34:33.426 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:33.077+0000 7fb805ae1700 1 -- 172.21.15.23:0/2287060785 shutdown_connections 2020-11-20T11:34:33.427 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:33.077+0000 7fb805ae1700 1 --2- 172.21.15.23:0/2287060785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8000a6390 0x7fb800131660 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:33.427 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:33.077+0000 7fb805ae1700 1 --2- 172.21.15.23:0/2287060785 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb7dc056d60 0x7fb7dc059220 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:33.427 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:33.077+0000 7fb805ae1700 1 --2- 172.21.15.23:0/2287060785 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8000aaab0 0x7fb8000b0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:33.427 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:33.077+0000 7fb805ae1700 1 --2- 172.21.15.23:0/2287060785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8000aae90 0x7fb8000a68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:33.428 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:33.077+0000 7fb805ae1700 1 -- 172.21.15.23:0/2287060785 >> 172.21.15.23:0/2287060785 conn(0x7fb8000678e0 msgr2=0x7fb800138e20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:34:33.428 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:33.077+0000 7fb805ae1700 1 -- 172.21.15.23:0/2287060785 shutdown_connections 2020-11-20T11:34:33.428 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:33.077+0000 7fb805ae1700 1 -- 172.21.15.23:0/2287060785 wait complete. 2020-11-20T11:34:33.428 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2020-11-20T11:34:43.430 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:34:43.430 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:34:43.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.563+0000 7fd96359e700 1 Processor -- start 2020-11-20T11:34:43.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.563+0000 7fd96359e700 1 -- start start 2020-11-20T11:34:43.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.563+0000 7fd96359e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd964067690 0x7fd964067b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:43.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.563+0000 7fd96359e700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd964135820 0x7fd964135c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:43.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.563+0000 7fd96359e700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9640a9fd0 con 0x7fd9640af060 2020-11-20T11:34:43.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.563+0000 7fd96359e700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd964068000 con 0x7fd964135820 2020-11-20T11:34:43.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.563+0000 7fd96359e700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9640aa230 con 0x7fd964067690 2020-11-20T11:34:43.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.564+0000 7fd962d9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd964135820 0x7fd964135c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:43.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.564+0000 7fd962d9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd964135820 0x7fd964135c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42162/0 (socket says 172.21.15.23:42162) 2020-11-20T11:34:43.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.564+0000 7fd962d9d700 1 -- 172.21.15.23:0/1371854910 learned_addr learned my addr 172.21.15.23:0/1371854910 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:34:43.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.564+0000 7fd95bfff700 1 --2- 172.21.15.23:0/1371854910 >> v2:172.21.15.155:3300/0 conn(0x7fd964067690 0x7fd964067b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:43.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.564+0000 7fd962d9d700 1 -- 172.21.15.23:0/1371854910 >> v2:172.21.15.155:3300/0 conn(0x7fd964067690 msgr2=0x7fd964067b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:34:43.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.564+0000 7fd962d9d700 1 --2- 172.21.15.23:0/1371854910 >> v2:172.21.15.155:3300/0 conn(0x7fd964067690 0x7fd964067b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:43.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.564+0000 7fd962d9d700 1 -- 172.21.15.23:0/1371854910 >> v1:172.21.15.155:6789/0 conn(0x7fd9640af060 legacy=0x7fd9640a6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:34:43.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.564+0000 7fd962d9d700 1 -- 172.21.15.23:0/1371854910 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9640aa4d0 con 0x7fd964135820 2020-11-20T11:34:43.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.564+0000 7fd95bfff700 1 --2- 172.21.15.23:0/1371854910 >> v2:172.21.15.155:3300/0 conn(0x7fd964067690 0x7fd964067b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:34:43.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.564+0000 7fd962d9d700 1 --2- 172.21.15.23:0/1371854910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd964135820 0x7fd964135c90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fd9640ab3c0 tx=0x7fd95c0092d0).ready entity=mon.0 client_cookie=a7100e90821adfd3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:43.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.564+0000 7fd961d9b700 1 -- 172.21.15.23:0/1371854910 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95c00bb80 con 0x7fd964135820 2020-11-20T11:34:43.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.564+0000 7fd961d9b700 1 -- 172.21.15.23:0/1371854910 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd95c00bce0 con 0x7fd964135820 2020-11-20T11:34:43.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.564+0000 7fd961d9b700 1 -- 172.21.15.23:0/1371854910 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95c011610 con 0x7fd964135820 2020-11-20T11:34:43.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.565+0000 7fd96359e700 1 -- 172.21.15.23:0/1371854910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd964135820 msgr2=0x7fd964135c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:43.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.565+0000 7fd96359e700 1 --2- 172.21.15.23:0/1371854910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd964135820 0x7fd964135c90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fd9640ab3c0 tx=0x7fd95c0092d0).stop 2020-11-20T11:34:43.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.565+0000 7fd96359e700 1 -- 172.21.15.23:0/1371854910 shutdown_connections 2020-11-20T11:34:43.573 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.565+0000 7fd96359e700 1 --2- 172.21.15.23:0/1371854910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd964135820 0x7fd964135c90 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:43.573 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.565+0000 7fd96359e700 1 --2- 172.21.15.23:0/1371854910 >> v2:172.21.15.155:3300/0 conn(0x7fd964067690 0x7fd964067b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:43.573 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.565+0000 7fd96359e700 1 -- 172.21.15.23:0/1371854910 >> 172.21.15.23:0/1371854910 conn(0x7fd964063980 msgr2=0x7fd964063d90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:34:43.573 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.565+0000 7fd96359e700 1 -- 172.21.15.23:0/1371854910 shutdown_connections 2020-11-20T11:34:43.573 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.565+0000 7fd96359e700 1 -- 172.21.15.23:0/1371854910 wait complete. 2020-11-20T11:34:43.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.566+0000 7fd96359e700 1 Processor -- start 2020-11-20T11:34:43.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.566+0000 7fd96359e700 1 -- start start 2020-11-20T11:34:43.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.567+0000 7fd96359e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd964067690 0x7fd964137c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:43.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.567+0000 7fd96359e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9640af060 0x7fd9640a7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:43.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.567+0000 7fd96359e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd964135820 0x7fd96413b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:43.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.567+0000 7fd96359e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9640a8910 con 0x7fd9640af060 2020-11-20T11:34:43.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.567+0000 7fd96359e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96413c3f0 con 0x7fd964067690 2020-11-20T11:34:43.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.567+0000 7fd96359e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96413c690 con 0x7fd964135820 2020-11-20T11:34:43.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.567+0000 7fd95bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9640af060 0x7fd9640a7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:43.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.567+0000 7fd96259c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd964067690 0x7fd964137c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:43.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.567+0000 7fd95bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9640af060 0x7fd9640a7e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42168/0 (socket says 172.21.15.23:42168) 2020-11-20T11:34:43.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.567+0000 7fd96259c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd964067690 0x7fd964137c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:35060/0 (socket says 172.21.15.23:35060) 2020-11-20T11:34:43.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.567+0000 7fd95bfff700 1 -- 172.21.15.23:0/4168928225 learned_addr learned my addr 172.21.15.23:0/4168928225 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:34:43.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.567+0000 7fd962d9d700 1 --2- 172.21.15.23:0/4168928225 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd964135820 0x7fd96413b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:43.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.567+0000 7fd95bfff700 1 -- 172.21.15.23:0/4168928225 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd964135820 msgr2=0x7fd96413b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:43.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.567+0000 7fd95bfff700 1 --2- 172.21.15.23:0/4168928225 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd964135820 0x7fd96413b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:43.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.567+0000 7fd95bfff700 1 -- 172.21.15.23:0/4168928225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd964067690 msgr2=0x7fd964137c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:43.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.567+0000 7fd95bfff700 1 --2- 172.21.15.23:0/4168928225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd964067690 0x7fd964137c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:43.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.567+0000 7fd95bfff700 1 -- 172.21.15.23:0/4168928225 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96413c930 con 0x7fd9640af060 2020-11-20T11:34:43.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.567+0000 7fd95bfff700 1 --2- 172.21.15.23:0/4168928225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9640af060 0x7fd9640a7e30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fd94c00cb50 tx=0x7fd94c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:43.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.568+0000 7fd95a7fc700 1 -- 172.21.15.23:0/4168928225 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94c00ce60 con 0x7fd9640af060 2020-11-20T11:34:43.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.568+0000 7fd95a7fc700 1 -- 172.21.15.23:0/4168928225 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd94c007740 con 0x7fd9640af060 2020-11-20T11:34:43.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.568+0000 7fd95a7fc700 1 -- 172.21.15.23:0/4168928225 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94c007ac0 con 0x7fd9640af060 2020-11-20T11:34:43.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.568+0000 7fd96359e700 1 -- 172.21.15.23:0/4168928225 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd96413be70 con 0x7fd9640af060 2020-11-20T11:34:43.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.568+0000 7fd96359e700 1 -- 172.21.15.23:0/4168928225 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd96413b8b0 con 0x7fd9640af060 2020-11-20T11:34:43.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.569+0000 7fd96359e700 1 -- 172.21.15.23:0/4168928225 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd964057dc0 con 0x7fd9640af060 2020-11-20T11:34:43.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.569+0000 7fd95a7fc700 1 -- 172.21.15.23:0/4168928225 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd94c01b460 con 0x7fd9640af060 2020-11-20T11:34:43.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.570+0000 7fd95a7fc700 1 --2- 172.21.15.23:0/4168928225 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd940056d60 0x7fd940059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:43.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.570+0000 7fd95a7fc700 1 -- 172.21.15.23:0/4168928225 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fd94c070150 con 0x7fd9640af060 2020-11-20T11:34:43.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.570+0000 7fd96259c700 1 --2- 172.21.15.23:0/4168928225 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd940056d60 0x7fd940059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:43.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.570+0000 7fd96259c700 1 --2- 172.21.15.23:0/4168928225 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd940056d60 0x7fd940059220 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fd95400c920 tx=0x7fd954008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:43.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.574+0000 7fd95a7fc700 1 -- 172.21.15.23:0/4168928225 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd94c0453e0 con 0x7fd9640af060 2020-11-20T11:34:43.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.823+0000 7fd96359e700 1 -- 172.21.15.23:0/4168928225 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd9640af440 con 0x7fd9640af060 2020-11-20T11:34:43.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.823+0000 7fd95a7fc700 1 -- 172.21.15.23:0/4168928225 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7fd94c048a00 con 0x7fd9640af060 2020-11-20T11:34:43.827 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:34:43.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.825+0000 7fd96359e700 1 -- 172.21.15.23:0/4168928225 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd940056d60 msgr2=0x7fd940059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:43.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.825+0000 7fd96359e700 1 --2- 172.21.15.23:0/4168928225 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd940056d60 0x7fd940059220 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fd95400c920 tx=0x7fd954008040).stop 2020-11-20T11:34:43.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.825+0000 7fd96359e700 1 -- 172.21.15.23:0/4168928225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9640af060 msgr2=0x7fd9640a7e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:43.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.825+0000 7fd96359e700 1 --2- 172.21.15.23:0/4168928225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9640af060 0x7fd9640a7e30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fd94c00cb50 tx=0x7fd94c008040).stop 2020-11-20T11:34:43.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.825+0000 7fd96359e700 1 -- 172.21.15.23:0/4168928225 shutdown_connections 2020-11-20T11:34:43.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.825+0000 7fd96359e700 1 --2- 172.21.15.23:0/4168928225 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd940056d60 0x7fd940059220 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:43.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.825+0000 7fd96359e700 1 --2- 172.21.15.23:0/4168928225 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd964135820 0x7fd96413b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:43.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.825+0000 7fd96359e700 1 --2- 172.21.15.23:0/4168928225 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd964067690 0x7fd964137c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:43.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.825+0000 7fd96359e700 1 --2- 172.21.15.23:0/4168928225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9640af060 0x7fd9640a7e30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:43.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.825+0000 7fd96359e700 1 -- 172.21.15.23:0/4168928225 >> 172.21.15.23:0/4168928225 conn(0x7fd964063980 msgr2=0x7fd96413ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:34:43.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.825+0000 7fd96359e700 1 -- 172.21.15.23:0/4168928225 shutdown_connections 2020-11-20T11:34:43.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.825+0000 7fd96359e700 1 -- 172.21.15.23:0/4168928225 wait complete. 2020-11-20T11:34:43.840 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:34:43.840 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2020-11-20T11:34:43.841 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:34:43.978 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.975+0000 7f23b1587700 1 Processor -- start 2020-11-20T11:34:43.978 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.975+0000 7f23b1587700 1 -- start start 2020-11-20T11:34:43.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.976+0000 7f23b1587700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f23ac068260 0x7f23ac065780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:43.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.976+0000 7f23b1587700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f23ac065c70 0x7f23ac137870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:43.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.976+0000 7f23b1587700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f23ac137d60 0x7f23ac13a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:43.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.976+0000 7f23b1587700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23ac134c40 con 0x7f23ac137d60 2020-11-20T11:34:43.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.976+0000 7f23b1587700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23ac0663a0 con 0x7f23ac065c70 2020-11-20T11:34:43.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.976+0000 7f23b1587700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac0a9000 con 0x7f23ac068260 2020-11-20T11:34:43.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.976+0000 7f23ab7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f23ac137d60 0x7f23ac13a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:43.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.976+0000 7f23ab7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f23ac137d60 0x7f23ac13a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42176/0 (socket says 172.21.15.23:42176) 2020-11-20T11:34:43.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.976+0000 7f23ab7fe700 1 -- 172.21.15.23:0/3770464436 learned_addr learned my addr 172.21.15.23:0/3770464436 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:34:43.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.976+0000 7f23a3fff700 1 --2- 172.21.15.23:0/3770464436 >> v2:172.21.15.155:3300/0 conn(0x7f23ac065c70 0x7f23ac137870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:43.983 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.976+0000 7f23aaffd700 1 --2- 172.21.15.23:0/3770464436 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f23ac068260 0x7f23ac065780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:43.983 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.976+0000 7f23ab7fe700 1 -- 172.21.15.23:0/3770464436 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f23ac068260 msgr2=0x7f23ac065780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:34:43.984 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.976+0000 7f23ab7fe700 1 --2- 172.21.15.23:0/3770464436 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f23ac068260 0x7f23ac065780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:43.985 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.976+0000 7f23ab7fe700 1 -- 172.21.15.23:0/3770464436 >> v2:172.21.15.155:3300/0 conn(0x7f23ac065c70 msgr2=0x7f23ac137870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:34:43.986 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.976+0000 7f23ab7fe700 1 --2- 172.21.15.23:0/3770464436 >> v2:172.21.15.155:3300/0 conn(0x7f23ac065c70 0x7f23ac137870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:43.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.976+0000 7f23ab7fe700 1 -- 172.21.15.23:0/3770464436 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23ac137170 con 0x7f23ac137d60 2020-11-20T11:34:43.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.977+0000 7f23ab7fe700 1 --2- 172.21.15.23:0/3770464436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ac137d60 0x7f23ac13a6c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f23ac0ae3f0 tx=0x7f23a400a470).ready entity=mon.0 client_cookie=ebee141c872029e1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:43.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.977+0000 7f23aa7fc700 1 -- 172.21.15.23:0/3770464436 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a400ce40 con 0x7f23ac137d60 2020-11-20T11:34:43.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.977+0000 7f23aa7fc700 1 -- 172.21.15.23:0/3770464436 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a4010410 con 0x7f23ac137d60 2020-11-20T11:34:43.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.977+0000 7f23aa7fc700 1 -- 172.21.15.23:0/3770464436 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a4010790 con 0x7f23ac137d60 2020-11-20T11:34:43.992 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.977+0000 7f23b1587700 1 -- 172.21.15.23:0/3770464436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ac137d60 msgr2=0x7f23ac13a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:43.992 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.977+0000 7f23b1587700 1 --2- 172.21.15.23:0/3770464436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ac137d60 0x7f23ac13a6c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f23ac0ae3f0 tx=0x7f23a400a470).stop 2020-11-20T11:34:43.993 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.977+0000 7f23b1587700 1 -- 172.21.15.23:0/3770464436 shutdown_connections 2020-11-20T11:34:43.995 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.977+0000 7f23b1587700 1 --2- 172.21.15.23:0/3770464436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ac137d60 0x7f23ac13a6c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:43.995 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.977+0000 7f23b1587700 1 --2- 172.21.15.23:0/3770464436 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f23ac068260 0x7f23ac065780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:43.997 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.977+0000 7f23b1587700 1 --2- 172.21.15.23:0/3770464436 >> v2:172.21.15.155:3300/0 conn(0x7f23ac065c70 0x7f23ac137870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:43.998 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.977+0000 7f23b1587700 1 -- 172.21.15.23:0/3770464436 >> 172.21.15.23:0/3770464436 conn(0x7f23ac0ad7a0 msgr2=0x7f23ac0adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:34:44.001 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.977+0000 7f23b1587700 1 -- 172.21.15.23:0/3770464436 shutdown_connections 2020-11-20T11:34:44.001 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.978+0000 7f23b1587700 1 -- 172.21.15.23:0/3770464436 wait complete. 2020-11-20T11:34:44.006 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.978+0000 7f23b1587700 1 Processor -- start 2020-11-20T11:34:44.006 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.979+0000 7f23b1587700 1 -- start start 2020-11-20T11:34:44.006 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.979+0000 7f23b1587700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f23ac068260 0x7f23ac060370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:44.006 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.979+0000 7f23b1587700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ac137d60 0x7f23ac1322c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:44.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.979+0000 7f23b1587700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f23ac133590 0x7f23ac1327b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:44.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.979+0000 7f23b1587700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac132ca0 con 0x7f23ac137d60 2020-11-20T11:34:44.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.979+0000 7f23b1587700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac134ed0 con 0x7f23ac068260 2020-11-20T11:34:44.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.979+0000 7f23b1587700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac0ad300 con 0x7f23ac133590 2020-11-20T11:34:44.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.979+0000 7f23a3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ac137d60 0x7f23ac1322c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:44.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.979+0000 7f23a3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ac137d60 0x7f23ac1322c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42178/0 (socket says 172.21.15.23:42178) 2020-11-20T11:34:44.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.979+0000 7f23a3fff700 1 -- 172.21.15.23:0/4126240308 learned_addr learned my addr 172.21.15.23:0/4126240308 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:34:44.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.979+0000 7f23aaffd700 1 --2- 172.21.15.23:0/4126240308 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f23ac068260 0x7f23ac060370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:44.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.980+0000 7f23ab7fe700 1 --2- 172.21.15.23:0/4126240308 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f23ac133590 0x7f23ac1327b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:44.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.980+0000 7f23a3fff700 1 -- 172.21.15.23:0/4126240308 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f23ac133590 msgr2=0x7f23ac1327b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:44.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.980+0000 7f23a3fff700 1 --2- 172.21.15.23:0/4126240308 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f23ac133590 0x7f23ac1327b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:44.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.980+0000 7f23a3fff700 1 -- 172.21.15.23:0/4126240308 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f23ac068260 msgr2=0x7f23ac060370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:44.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.980+0000 7f23a3fff700 1 --2- 172.21.15.23:0/4126240308 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f23ac068260 0x7f23ac060370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:44.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.980+0000 7f23a3fff700 1 -- 172.21.15.23:0/4126240308 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23ac0acdc0 con 0x7f23ac137d60 2020-11-20T11:34:44.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.980+0000 7f23ab7fe700 1 --2- 172.21.15.23:0/4126240308 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f23ac133590 0x7f23ac1327b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:34:44.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.980+0000 7f23a3fff700 1 --2- 172.21.15.23:0/4126240308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ac137d60 0x7f23ac1322c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f239400db60 tx=0x7f2394008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:44.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.980+0000 7f23a2ffd700 1 -- 172.21.15.23:0/4126240308 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2394009790 con 0x7f23ac137d60 2020-11-20T11:34:44.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.980+0000 7f23a2ffd700 1 -- 172.21.15.23:0/4126240308 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23940098f0 con 0x7f23ac137d60 2020-11-20T11:34:44.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.980+0000 7f23a2ffd700 1 -- 172.21.15.23:0/4126240308 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2394009c70 con 0x7f23ac137d60 2020-11-20T11:34:44.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.980+0000 7f23b1587700 1 -- 172.21.15.23:0/4126240308 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23ac0ad060 con 0x7f23ac137d60 2020-11-20T11:34:44.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.980+0000 7f23b1587700 1 -- 172.21.15.23:0/4126240308 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23ac060aa0 con 0x7f23ac137d60 2020-11-20T11:34:44.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.981+0000 7f23b1587700 1 --2- 172.21.15.23:0/4126240308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ac065c70 0x7f23ac12da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:44.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.981+0000 7f23b1587700 1 -- 172.21.15.23:0/4126240308 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac0acc70 con 0x7f23ac065c70 2020-11-20T11:34:44.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.981+0000 7f23aaffd700 1 --2- 172.21.15.23:0/4126240308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ac065c70 0x7f23ac12da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:44.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.981+0000 7f23aaffd700 1 -- 172.21.15.23:0/4126240308 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f23ac131fe0 con 0x7f23ac065c70 2020-11-20T11:34:44.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.981+0000 7f23aaffd700 1 --2- 172.21.15.23:0/4126240308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ac065c70 0x7f23ac12da30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f239c00f790 tx=0x7f239c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:44.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.982+0000 7f23a2ffd700 1 -- 172.21.15.23:0/4126240308 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239c013610 con 0x7f23ac065c70 2020-11-20T11:34:44.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.982+0000 7f23a2ffd700 1 -- 172.21.15.23:0/4126240308 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2394013480 con 0x7f23ac137d60 2020-11-20T11:34:44.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.982+0000 7f23a2ffd700 1 --2- 172.21.15.23:0/4126240308 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2388056e30 0x7f23880592f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:44.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.982+0000 7f23a2ffd700 1 -- 172.21.15.23:0/4126240308 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f2394070390 con 0x7f23ac137d60 2020-11-20T11:34:44.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.982+0000 7f23a2ffd700 1 -- 172.21.15.23:0/4126240308 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f239c000e90 con 0x7f23ac065c70 2020-11-20T11:34:44.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.982+0000 7f23a2ffd700 1 -- 172.21.15.23:0/4126240308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ac065c70 msgr2=0x7f23ac12da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:44.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.982+0000 7f23a2ffd700 1 --2- 172.21.15.23:0/4126240308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ac065c70 0x7f23ac12da30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f239c00f790 tx=0x7f239c008040).stop 2020-11-20T11:34:44.014 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.982+0000 7f23a2ffd700 1 -- 172.21.15.23:0/4126240308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ac065c70 msgr2=0x7f23ac12da30 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:34:44.014 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.982+0000 7f23a2ffd700 1 --2- 172.21.15.23:0/4126240308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ac065c70 0x7f23ac12da30 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0x7f239c00f790 tx=0x7f239c008040).stop 2020-11-20T11:34:44.014 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.982+0000 7f23ab7fe700 1 --2- 172.21.15.23:0/4126240308 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2388056e30 0x7f23880592f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:44.014 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.983+0000 7f23ab7fe700 1 --2- 172.21.15.23:0/4126240308 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2388056e30 0x7f23880592f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f23ac0ae3f0 tx=0x7f23a400a470).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:44.014 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.989+0000 7f23b1587700 1 --2- 172.21.15.23:0/4126240308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ac12df20 0x7f23ac0aa110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:44.015 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.989+0000 7f23b1587700 1 -- 172.21.15.23:0/4126240308 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ac062fe0 con 0x7f23ac12df20 2020-11-20T11:34:44.015 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.989+0000 7f23aaffd700 1 --2- 172.21.15.23:0/4126240308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ac12df20 0x7f23ac0aa110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:44.015 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.989+0000 7f23aaffd700 1 -- 172.21.15.23:0/4126240308 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f23ac044710 con 0x7f23ac12df20 2020-11-20T11:34:44.015 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.990+0000 7f23aaffd700 1 --2- 172.21.15.23:0/4126240308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ac12df20 0x7f23ac0aa110 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f239c00f790 tx=0x7f239c01a000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:44.015 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.990+0000 7f23a2ffd700 1 -- 172.21.15.23:0/4126240308 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239c01ce10 con 0x7f23ac12df20 2020-11-20T11:34:44.016 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.990+0000 7f23a2ffd700 1 -- 172.21.15.23:0/4126240308 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4616 (secure 0 0 0) 0x7f239c025070 con 0x7f23ac12df20 2020-11-20T11:34:44.016 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.990+0000 7f23a2ffd700 1 -- 172.21.15.23:0/4126240308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ac12df20 msgr2=0x7f23ac0aa110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:44.016 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.990+0000 7f23a2ffd700 1 --2- 172.21.15.23:0/4126240308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ac12df20 0x7f23ac0aa110 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f239c00f790 tx=0x7f239c01a000).stop 2020-11-20T11:34:44.016 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.990+0000 7f23a2ffd700 1 -- 172.21.15.23:0/4126240308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ac12df20 msgr2=0x7f23ac0aa110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:34:44.016 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.990+0000 7f23a2ffd700 1 --2- 172.21.15.23:0/4126240308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ac12df20 0x7f23ac0aa110 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:44.017 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.990+0000 7f23b1587700 1 -- 172.21.15.23:0/4126240308 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2388056e30 msgr2=0x7f23880592f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:44.017 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.990+0000 7f23b1587700 1 --2- 172.21.15.23:0/4126240308 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2388056e30 0x7f23880592f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f23ac0ae3f0 tx=0x7f23a400a470).stop 2020-11-20T11:34:44.017 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.990+0000 7f23b1587700 1 -- 172.21.15.23:0/4126240308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ac137d60 msgr2=0x7f23ac1322c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:44.017 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.990+0000 7f23b1587700 1 --2- 172.21.15.23:0/4126240308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23ac137d60 0x7f23ac1322c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f239400db60 tx=0x7f2394008040).stop 2020-11-20T11:34:44.017 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.990+0000 7f23ab7fe700 1 -- 172.21.15.23:0/4126240308 reap_dead start 2020-11-20T11:34:44.018 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.990+0000 7f23ab7fe700 1 -- 172.21.15.23:0/4126240308 reap_dead start 2020-11-20T11:34:44.018 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.990+0000 7f23b1587700 1 -- 172.21.15.23:0/4126240308 shutdown_connections 2020-11-20T11:34:44.018 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.990+0000 7f23b1587700 1 -- 172.21.15.23:0/4126240308 >> 172.21.15.23:0/4126240308 conn(0x7f23ac0ad7a0 msgr2=0x7f23ac068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:34:44.018 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.990+0000 7f23b1587700 1 -- 172.21.15.23:0/4126240308 shutdown_connections 2020-11-20T11:34:44.018 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:43.990+0000 7f23b1587700 1 -- 172.21.15.23:0/4126240308 wait complete. 2020-11-20T11:34:44.019 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:34:44.019 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:34:44.019 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:34:44.019 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:34:44.019 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 22, 2020-11-20T11:34:44.020 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:34:44.020 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:34:44.020 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:34:44.020 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:34:44.020 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:44.021 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 12, 2020-11-20T11:34:44.021 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:34:44.021 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:34:44.021 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:34:44.021 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:34:44.022 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:34:44.022 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:34:44.022 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:34:44.022 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:34:44.022 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:34:44.023 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:34:44.023 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:34:44.023 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:44.023 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:34:44.023 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:34:44.024 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:34:44.024 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:34:44.024 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:34:44.024 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:34:44.024 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:34:44.025 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:34:44.025 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:34:44.025 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:34:44.025 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:44.025 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.026 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:34:44.026 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:34:44.026 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:34:44.026 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:34:44.026 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:34:44.027 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:34:44.027 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:34:44.027 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:34:44.027 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:34:44.027 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:34:44.028 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:34:44.028 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:34:44.028 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:34:44.028 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:34:44.029 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:34:44.029 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:34:44.029 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:34:44.029 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:34:44.029 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:34:44.030 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:34:44.030 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:34:44.030 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:34:44.030 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:34:44.030 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:44.031 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:34:44.031 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.031 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:34:44.031 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.031 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:34:44.032 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:34:44.032 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:34:44.032 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:34:44.032 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.032 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:34:44.032 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:34:44.033 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:44.033 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.033 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.033 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:34:44.033 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:34:44.034 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:44.034 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:44.034 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:44.034 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.034 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:34:44.035 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:34:44.035 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:34:44.035 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:34:44.035 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:34:44.035 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.036 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.036 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:34:44.036 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:34:44.036 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:34:44.036 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:34:44.037 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.037 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:34:44.037 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:34:44.037 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:44.037 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.038 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.038 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:34:44.038 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:34:44.038 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:44.038 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:44.039 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:44.039 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.039 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:34:44.039 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:34:44.039 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:34:44.040 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:34:44.040 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:34:44.040 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.040 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.040 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:34:44.041 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:34:44.041 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:34:44.041 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:34:44.041 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.042 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:34:44.042 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:34:44.042 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:44.042 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.042 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.042 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:34:44.043 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:34:44.043 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:44.043 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:44.043 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:44.043 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.044 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:34:44.044 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:34:44.044 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:34:44.044 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:34:44.044 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:34:44.045 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:44.045 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:44.045 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.045 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:34:44.045 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:34:44.046 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.046 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:34:44.046 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:34:44.046 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:34:44.046 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:44.047 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:44.047 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:34:44.047 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.047 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:34:44.047 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:34:44.048 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:34:44.048 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:44.048 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:44.048 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:34:44.048 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.049 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:34:44.049 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:34:44.049 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 4 2020-11-20T11:34:44.049 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:44.049 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:44.050 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:34:44.050 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.050 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:34:44.050 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:34:44.050 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:34:44.051 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:44.051 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:44.051 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.051 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:34:44.051 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:34:44.052 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:34:44.189 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.185+0000 7f5fdd601700 1 Processor -- start 2020-11-20T11:34:44.189 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.186+0000 7f5fdd601700 1 -- start start 2020-11-20T11:34:44.191 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.186+0000 7f5fdd601700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5fd8061380 0x7f5fd813c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:44.191 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.186+0000 7f5fdd601700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5fd80aad10 0x7f5fd813c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:44.191 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.186+0000 7f5fdd601700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5fd80a8890 0x7f5fd8132470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:44.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.186+0000 7f5fdd601700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fd813cbf0 con 0x7f5fd80aad10 2020-11-20T11:34:44.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.186+0000 7f5fdd601700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fd8068810 con 0x7f5fd8061380 2020-11-20T11:34:44.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.187+0000 7f5fdd601700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd80a8e10 con 0x7f5fd80a8890 2020-11-20T11:34:44.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.187+0000 7f5fd67fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5fd80aad10 0x7f5fd813c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:44.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.187+0000 7f5fd67fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5fd80aad10 0x7f5fd813c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42192/0 (socket says 172.21.15.23:42192) 2020-11-20T11:34:44.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.187+0000 7f5fd67fc700 1 -- 172.21.15.23:0/3383153264 learned_addr learned my addr 172.21.15.23:0/3383153264 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:34:44.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.187+0000 7f5fd6ffd700 1 --2- 172.21.15.23:0/3383153264 >> v2:172.21.15.155:3300/0 conn(0x7f5fd8061380 0x7f5fd813c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:44.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.187+0000 7f5fd77fe700 1 --2- 172.21.15.23:0/3383153264 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5fd80a8890 0x7f5fd8132470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:44.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.187+0000 7f5fd67fc700 1 -- 172.21.15.23:0/3383153264 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5fd80a8890 msgr2=0x7f5fd8132470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:34:44.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.187+0000 7f5fd67fc700 1 --2- 172.21.15.23:0/3383153264 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5fd80a8890 0x7f5fd8132470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:44.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.187+0000 7f5fd67fc700 1 -- 172.21.15.23:0/3383153264 >> v2:172.21.15.155:3300/0 conn(0x7f5fd8061380 msgr2=0x7f5fd813c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:34:44.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.187+0000 7f5fd67fc700 1 --2- 172.21.15.23:0/3383153264 >> v2:172.21.15.155:3300/0 conn(0x7f5fd8061380 0x7f5fd813c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:44.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.187+0000 7f5fd67fc700 1 -- 172.21.15.23:0/3383153264 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd80a90b0 con 0x7f5fd80aad10 2020-11-20T11:34:44.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.187+0000 7f5fd67fc700 1 --2- 172.21.15.23:0/3383153264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5fd80aad10 0x7f5fd813c700 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f5fd805cbf0 tx=0x7f5fcc006ef0).ready entity=mon.0 client_cookie=ec32199ce99a82cc server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:44.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.187+0000 7f5fd5ffb700 1 -- 172.21.15.23:0/3383153264 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc00b860 con 0x7f5fd80aad10 2020-11-20T11:34:44.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.187+0000 7f5fd5ffb700 1 -- 172.21.15.23:0/3383153264 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fcc000e90 con 0x7f5fd80aad10 2020-11-20T11:34:44.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.187+0000 7f5fd5ffb700 1 -- 172.21.15.23:0/3383153264 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc00bbd0 con 0x7f5fd80aad10 2020-11-20T11:34:44.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.188+0000 7f5fdd601700 1 -- 172.21.15.23:0/3383153264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5fd80aad10 msgr2=0x7f5fd813c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:44.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.188+0000 7f5fdd601700 1 --2- 172.21.15.23:0/3383153264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5fd80aad10 0x7f5fd813c700 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f5fd805cbf0 tx=0x7f5fcc006ef0).stop 2020-11-20T11:34:44.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.188+0000 7f5fdd601700 1 -- 172.21.15.23:0/3383153264 shutdown_connections 2020-11-20T11:34:44.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.188+0000 7f5fdd601700 1 --2- 172.21.15.23:0/3383153264 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5fd80a8890 0x7f5fd8132470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:44.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.188+0000 7f5fdd601700 1 --2- 172.21.15.23:0/3383153264 >> v2:172.21.15.155:3300/0 conn(0x7f5fd8061380 0x7f5fd813c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:44.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.188+0000 7f5fdd601700 1 --2- 172.21.15.23:0/3383153264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5fd80aad10 0x7f5fd813c700 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:44.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.188+0000 7f5fdd601700 1 -- 172.21.15.23:0/3383153264 >> 172.21.15.23:0/3383153264 conn(0x7f5fd80ae120 msgr2=0x7f5fd80aa0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:34:44.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.188+0000 7f5fdd601700 1 -- 172.21.15.23:0/3383153264 shutdown_connections 2020-11-20T11:34:44.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.188+0000 7f5fdd601700 1 -- 172.21.15.23:0/3383153264 wait complete. 2020-11-20T11:34:44.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.189+0000 7f5fdd601700 1 Processor -- start 2020-11-20T11:34:44.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.189+0000 7f5fdd601700 1 -- start start 2020-11-20T11:34:44.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.190+0000 7f5fdd601700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5fd8061380 0x7f5fd809fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:44.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.190+0000 7f5fdd601700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fd80a8890 0x7f5fd81394a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:44.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.190+0000 7f5fdd601700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5fd80aad10 0x7f5fd809d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:44.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.190+0000 7f5fdd601700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd809d990 con 0x7f5fd80aad10 2020-11-20T11:34:44.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.190+0000 7f5fdd601700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd8138da0 con 0x7f5fd80a8890 2020-11-20T11:34:44.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.190+0000 7f5fdd601700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd809dbb0 con 0x7f5fd8061380 2020-11-20T11:34:44.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.190+0000 7f5fd77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5fd80aad10 0x7f5fd809d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:44.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.190+0000 7f5fd77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5fd80aad10 0x7f5fd809d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42200/0 (socket says 172.21.15.23:42200) 2020-11-20T11:34:44.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.190+0000 7f5fd77fe700 1 -- 172.21.15.23:0/1173098120 learned_addr learned my addr 172.21.15.23:0/1173098120 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:34:44.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.190+0000 7f5fd6ffd700 1 --2- 172.21.15.23:0/1173098120 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5fd8061380 0x7f5fd809fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:44.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.190+0000 7f5fd67fc700 1 --2- 172.21.15.23:0/1173098120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fd80a8890 0x7f5fd81394a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:44.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.190+0000 7f5fd77fe700 1 -- 172.21.15.23:0/1173098120 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5fd8061380 msgr2=0x7f5fd809fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:44.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.190+0000 7f5fd77fe700 1 --2- 172.21.15.23:0/1173098120 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5fd8061380 0x7f5fd809fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:44.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.190+0000 7f5fd77fe700 1 -- 172.21.15.23:0/1173098120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fd80a8890 msgr2=0x7f5fd81394a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:44.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.190+0000 7f5fd77fe700 1 --2- 172.21.15.23:0/1173098120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fd80a8890 0x7f5fd81394a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:44.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.190+0000 7f5fd77fe700 1 -- 172.21.15.23:0/1173098120 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd8139000 con 0x7f5fd80aad10 2020-11-20T11:34:44.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.190+0000 7f5fd77fe700 1 --2- 172.21.15.23:0/1173098120 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5fd80aad10 0x7f5fd809d4a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f5fd000ebc0 tx=0x7f5fd0008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:44.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.191+0000 7f5fbeffd700 1 -- 172.21.15.23:0/1173098120 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd000c5e0 con 0x7f5fd80aad10 2020-11-20T11:34:44.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.191+0000 7f5fbeffd700 1 -- 172.21.15.23:0/1173098120 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fd000c740 con 0x7f5fd80aad10 2020-11-20T11:34:44.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.191+0000 7f5fbeffd700 1 -- 172.21.15.23:0/1173098120 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd000cae0 con 0x7f5fd80aad10 2020-11-20T11:34:44.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.191+0000 7f5fdd601700 1 -- 172.21.15.23:0/1173098120 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fd809e370 con 0x7f5fd80aad10 2020-11-20T11:34:44.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.191+0000 7f5fdd601700 1 -- 172.21.15.23:0/1173098120 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fd814b150 con 0x7f5fd80aad10 2020-11-20T11:34:44.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.191+0000 7f5fdd601700 1 --2- 172.21.15.23:0/1173098120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fd814b3d0 0x7f5fd814b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:44.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.191+0000 7f5fdd601700 1 -- 172.21.15.23:0/1173098120 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd805cfc0 con 0x7f5fd814b3d0 2020-11-20T11:34:44.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.192+0000 7f5fd6ffd700 1 --2- 172.21.15.23:0/1173098120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fd814b3d0 0x7f5fd814b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:44.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.192+0000 7f5fbeffd700 1 -- 172.21.15.23:0/1173098120 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5fd001d460 con 0x7f5fd80aad10 2020-11-20T11:34:44.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.192+0000 7f5fd6ffd700 1 -- 172.21.15.23:0/1173098120 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5fd8139200 con 0x7f5fd814b3d0 2020-11-20T11:34:44.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.192+0000 7f5fbeffd700 1 --2- 172.21.15.23:0/1173098120 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5fb4056d60 0x7f5fb4059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:44.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.192+0000 7f5fbeffd700 1 -- 172.21.15.23:0/1173098120 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f5fd006ff60 con 0x7f5fd80aad10 2020-11-20T11:34:44.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.192+0000 7f5fd6ffd700 1 --2- 172.21.15.23:0/1173098120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fd814b3d0 0x7f5fd814b7b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f5fc8010b00 tx=0x7f5fc8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:44.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.192+0000 7f5fd67fc700 1 --2- 172.21.15.23:0/1173098120 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5fb4056d60 0x7f5fb4059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:44.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.192+0000 7f5fbeffd700 1 -- 172.21.15.23:0/1173098120 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc800ea00 con 0x7f5fd814b3d0 2020-11-20T11:34:44.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.193+0000 7f5fbeffd700 1 -- 172.21.15.23:0/1173098120 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f5fc800eb60 con 0x7f5fd814b3d0 2020-11-20T11:34:44.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.193+0000 7f5fbeffd700 1 -- 172.21.15.23:0/1173098120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fd814b3d0 msgr2=0x7f5fd814b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:44.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.193+0000 7f5fbeffd700 1 --2- 172.21.15.23:0/1173098120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fd814b3d0 0x7f5fd814b7b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f5fc8010b00 tx=0x7f5fc8008040).stop 2020-11-20T11:34:44.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.193+0000 7f5fd67fc700 1 --2- 172.21.15.23:0/1173098120 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5fb4056d60 0x7f5fb4059220 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f5fcc006f30 tx=0x7f5fcc016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:44.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.193+0000 7f5fbeffd700 1 -- 172.21.15.23:0/1173098120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fd814b3d0 msgr2=0x7f5fd814b7b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:34:44.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.193+0000 7f5fbeffd700 1 --2- 172.21.15.23:0/1173098120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fd814b3d0 0x7f5fd814b7b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:44.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.203+0000 7f5fdd601700 1 --2- 172.21.15.23:0/1173098120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fd814bca0 0x7f5fd8147b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:44.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.203+0000 7f5fdd601700 1 -- 172.21.15.23:0/1173098120 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd8057dc0 con 0x7f5fd814bca0 2020-11-20T11:34:44.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.204+0000 7f5fd6ffd700 1 --2- 172.21.15.23:0/1173098120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fd814bca0 0x7f5fd8147b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:44.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.204+0000 7f5fd6ffd700 1 -- 172.21.15.23:0/1173098120 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5fd8044710 con 0x7f5fd814bca0 2020-11-20T11:34:44.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.204+0000 7f5fd6ffd700 1 --2- 172.21.15.23:0/1173098120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fd814bca0 0x7f5fd8147b80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f5fc800b240 tx=0x7f5fc8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:44.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.204+0000 7f5fbeffd700 1 -- 172.21.15.23:0/1173098120 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc801d8a0 con 0x7f5fd814bca0 2020-11-20T11:34:44.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.205+0000 7f5fbeffd700 1 -- 172.21.15.23:0/1173098120 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4274 (secure 0 0 0) 0x7f5fc8000e90 con 0x7f5fd814bca0 2020-11-20T11:34:44.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.205+0000 7f5fbeffd700 1 -- 172.21.15.23:0/1173098120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fd814bca0 msgr2=0x7f5fd8147b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:44.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.205+0000 7f5fbeffd700 1 --2- 172.21.15.23:0/1173098120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fd814bca0 0x7f5fd8147b80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f5fc800b240 tx=0x7f5fc8008040).stop 2020-11-20T11:34:44.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.205+0000 7f5fbeffd700 1 -- 172.21.15.23:0/1173098120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fd814bca0 msgr2=0x7f5fd8147b80 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:34:44.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.205+0000 7f5fbeffd700 1 --2- 172.21.15.23:0/1173098120 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5fd814bca0 0x7f5fd8147b80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:44.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.205+0000 7f5fdd601700 1 -- 172.21.15.23:0/1173098120 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5fb4056d60 msgr2=0x7f5fb4059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:44.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.205+0000 7f5fdd601700 1 --2- 172.21.15.23:0/1173098120 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5fb4056d60 0x7f5fb4059220 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f5fcc006f30 tx=0x7f5fcc016000).stop 2020-11-20T11:34:44.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.205+0000 7f5fdd601700 1 -- 172.21.15.23:0/1173098120 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5fd80aad10 msgr2=0x7f5fd809d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:44.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.205+0000 7f5fdd601700 1 --2- 172.21.15.23:0/1173098120 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5fd80aad10 0x7f5fd809d4a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f5fd000ebc0 tx=0x7f5fd0008040).stop 2020-11-20T11:34:44.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.205+0000 7f5fd77fe700 1 -- 172.21.15.23:0/1173098120 reap_dead start 2020-11-20T11:34:44.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.205+0000 7f5fdd601700 1 -- 172.21.15.23:0/1173098120 shutdown_connections 2020-11-20T11:34:44.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.205+0000 7f5fdd601700 1 --2- 172.21.15.23:0/1173098120 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5fd80aad10 0x7f5fd809d4a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:44.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.205+0000 7f5fdd601700 1 -- 172.21.15.23:0/1173098120 >> 172.21.15.23:0/1173098120 conn(0x7f5fd80ae120 msgr2=0x7f5fd80aa010 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:34:44.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.205+0000 7f5fdd601700 1 -- 172.21.15.23:0/1173098120 shutdown_connections 2020-11-20T11:34:44.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.205+0000 7f5fdd601700 1 -- 172.21.15.23:0/1173098120 wait complete. 2020-11-20T11:34:44.216 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:34:44.217 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:34:44.217 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:34:44.217 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:34:44.217 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 22, 2020-11-20T11:34:44.217 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:34:44.218 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:34:44.218 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:34:44.218 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:34:44.218 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:44.218 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 12, 2020-11-20T11:34:44.219 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:34:44.219 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:34:44.219 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:34:44.219 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:34:44.219 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:34:44.219 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:34:44.220 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:34:44.220 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:34:44.220 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:34:44.220 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:34:44.220 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:34:44.221 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:44.221 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:34:44.221 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:34:44.221 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:34:44.221 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:34:44.222 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:34:44.222 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:34:44.222 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:34:44.222 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:34:44.222 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:34:44.223 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:34:44.223 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:44.223 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.223 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:34:44.223 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:34:44.224 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:34:44.224 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:34:44.224 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:34:44.224 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:34:44.224 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:34:44.224 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:34:44.225 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:34:44.225 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:34:44.226 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:34:44.226 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:34:44.226 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:34:44.226 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:34:44.226 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:34:44.227 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:34:44.227 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:34:44.227 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:34:44.227 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:34:44.227 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:34:44.227 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:34:44.228 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:34:44.228 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:34:44.228 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:44.228 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:34:44.228 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.229 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:34:44.229 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.229 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:34:44.229 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:34:44.229 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:34:44.230 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:34:44.230 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.230 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:34:44.230 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:34:44.230 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:44.231 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.231 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.231 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:34:44.231 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:34:44.231 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:44.232 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:44.232 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:44.232 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.232 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:34:44.232 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:34:44.232 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:34:44.233 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:34:44.233 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:34:44.233 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.233 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.233 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:34:44.234 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:34:44.234 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:34:44.234 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:34:44.234 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.234 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:34:44.235 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:34:44.235 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:44.235 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.236 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.236 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:34:44.236 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:34:44.237 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:44.237 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:44.237 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:44.237 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.237 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:34:44.238 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:34:44.238 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:34:44.238 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:34:44.238 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:34:44.238 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.239 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.239 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:34:44.239 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:34:44.239 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:34:44.239 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:34:44.240 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.240 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:34:44.240 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:34:44.240 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:44.240 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.240 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.241 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:34:44.241 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:34:44.241 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:44.241 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:44.241 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:44.242 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.242 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:34:44.242 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:34:44.242 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:34:44.242 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:34:44.243 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:34:44.243 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:44.243 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:44.243 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.243 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:34:44.244 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:34:44.244 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.244 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:34:44.244 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:34:44.244 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:34:44.244 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:44.245 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:44.245 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:34:44.245 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.245 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:34:44.245 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:34:44.246 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:34:44.246 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:44.247 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:44.247 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.247 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:34:44.247 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:34:44.248 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:34:44.397 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.393+0000 7f8684dcd700 1 Processor -- start 2020-11-20T11:34:44.397 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.393+0000 7f8684dcd700 1 -- start start 2020-11-20T11:34:44.402 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.394+0000 7f8684dcd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86800a8df0 0x7f8680138a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:44.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.394+0000 7f8684dcd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f86800a6ee0 0x7f86800a83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:44.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.394+0000 7f8684dcd700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86800a7780 con 0x7f8680065d10 2020-11-20T11:34:44.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.394+0000 7f8684dcd700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86800661c0 con 0x7f86800a6ee0 2020-11-20T11:34:44.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.394+0000 7f8684dcd700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8680066420 con 0x7f86800a8df0 2020-11-20T11:34:44.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.394+0000 7f867dd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f86800a6ee0 0x7f86800a83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:44.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.394+0000 7f867dd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f86800a6ee0 0x7f86800a83f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42208/0 (socket says 172.21.15.23:42208) 2020-11-20T11:34:44.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.394+0000 7f867dd9b700 1 -- 172.21.15.23:0/3084979135 learned_addr learned my addr 172.21.15.23:0/3084979135 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:34:44.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.395+0000 7f867e59c700 1 --2- 172.21.15.23:0/3084979135 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86800a8df0 0x7f8680138a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:44.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.395+0000 7f867dd9b700 1 -- 172.21.15.23:0/3084979135 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86800a8df0 msgr2=0x7f8680138a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:34:44.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.395+0000 7f867dd9b700 1 --2- 172.21.15.23:0/3084979135 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86800a8df0 0x7f8680138a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:44.407 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.395+0000 7f867dd9b700 1 -- 172.21.15.23:0/3084979135 >> v1:172.21.15.155:6789/0 conn(0x7f8680065d10 legacy=0x7f86800a37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:44.407 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.395+0000 7f867dd9b700 1 -- 172.21.15.23:0/3084979135 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86800a7d20 con 0x7f86800a6ee0 2020-11-20T11:34:44.408 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.395+0000 7f867e59c700 1 --2- 172.21.15.23:0/3084979135 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86800a8df0 0x7f8680138a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:34:44.408 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.395+0000 7f867dd9b700 1 --2- 172.21.15.23:0/3084979135 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86800a6ee0 0x7f86800a83f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f8674009a10 tx=0x7f8674006ef0).ready entity=mon.0 client_cookie=fae27565678cb352 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:44.408 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.395+0000 7f867d59a700 1 -- 172.21.15.23:0/3084979135 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867400b860 con 0x7f86800a6ee0 2020-11-20T11:34:44.409 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.395+0000 7f867d59a700 1 -- 172.21.15.23:0/3084979135 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8674000e90 con 0x7f86800a6ee0 2020-11-20T11:34:44.409 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.395+0000 7f867d59a700 1 -- 172.21.15.23:0/3084979135 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867400bbd0 con 0x7f86800a6ee0 2020-11-20T11:34:44.409 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.396+0000 7f8684dcd700 1 -- 172.21.15.23:0/3084979135 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86800a6ee0 msgr2=0x7f86800a83f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:44.410 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.396+0000 7f8684dcd700 1 --2- 172.21.15.23:0/3084979135 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86800a6ee0 0x7f86800a83f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f8674009a10 tx=0x7f8674006ef0).stop 2020-11-20T11:34:44.410 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.396+0000 7f8684dcd700 1 -- 172.21.15.23:0/3084979135 shutdown_connections 2020-11-20T11:34:44.411 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.396+0000 7f8684dcd700 1 --2- 172.21.15.23:0/3084979135 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86800a8df0 0x7f8680138a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:44.411 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.396+0000 7f8684dcd700 1 --2- 172.21.15.23:0/3084979135 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86800a6ee0 0x7f86800a83f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:44.411 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.396+0000 7f8684dcd700 1 -- 172.21.15.23:0/3084979135 >> 172.21.15.23:0/3084979135 conn(0x7f86800601b0 msgr2=0x7f86800605c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:34:44.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.396+0000 7f8684dcd700 1 -- 172.21.15.23:0/3084979135 shutdown_connections 2020-11-20T11:34:44.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.396+0000 7f8684dcd700 1 -- 172.21.15.23:0/3084979135 wait complete. 2020-11-20T11:34:44.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.397+0000 7f8684dcd700 1 Processor -- start 2020-11-20T11:34:44.413 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.397+0000 7f8684dcd700 1 -- start start 2020-11-20T11:34:44.413 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.398+0000 7f8684dcd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8680065d10 0x7f868009f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:44.414 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.398+0000 7f8684dcd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86800a6ee0 0x7f868009de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:44.414 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.398+0000 7f8684dcd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86800a8df0 0x7f8680147ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:44.415 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.398+0000 7f8684dcd700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868009e320 con 0x7f86800a6ee0 2020-11-20T11:34:44.415 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.398+0000 7f8684dcd700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868013c770 con 0x7f8680065d10 2020-11-20T11:34:44.415 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.398+0000 7f8684dcd700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f868013ca10 con 0x7f86800a8df0 2020-11-20T11:34:44.416 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.398+0000 7f867dd9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86800a6ee0 0x7f868009de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:44.416 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.398+0000 7f867e59c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8680065d10 0x7f868009f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:44.416 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.398+0000 7f867dd9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86800a6ee0 0x7f868009de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42216/0 (socket says 172.21.15.23:42216) 2020-11-20T11:34:44.416 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.398+0000 7f867dd9b700 1 -- 172.21.15.23:0/4150119695 learned_addr learned my addr 172.21.15.23:0/4150119695 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:34:44.417 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.398+0000 7f867ed9d700 1 --2- 172.21.15.23:0/4150119695 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86800a8df0 0x7f8680147ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:44.417 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.398+0000 7f867dd9b700 1 -- 172.21.15.23:0/4150119695 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86800a8df0 msgr2=0x7f8680147ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:44.417 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.398+0000 7f867dd9b700 1 --2- 172.21.15.23:0/4150119695 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f86800a8df0 0x7f8680147ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:44.417 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.398+0000 7f867dd9b700 1 -- 172.21.15.23:0/4150119695 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8680065d10 msgr2=0x7f868009f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:44.418 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.398+0000 7f867dd9b700 1 --2- 172.21.15.23:0/4150119695 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8680065d10 0x7f868009f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:44.418 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.398+0000 7f867dd9b700 1 -- 172.21.15.23:0/4150119695 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f868013ccb0 con 0x7f86800a6ee0 2020-11-20T11:34:44.418 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.398+0000 7f867dd9b700 1 --2- 172.21.15.23:0/4150119695 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86800a6ee0 0x7f868009de30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f8674009000 tx=0x7f8674015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:44.418 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.398+0000 7f866e7fc700 1 -- 172.21.15.23:0/4150119695 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8674011c30 con 0x7f86800a6ee0 2020-11-20T11:34:44.418 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.399+0000 7f866e7fc700 1 -- 172.21.15.23:0/4150119695 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8674011d90 con 0x7f86800a6ee0 2020-11-20T11:34:44.419 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.399+0000 7f866e7fc700 1 -- 172.21.15.23:0/4150119695 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867401c680 con 0x7f86800a6ee0 2020-11-20T11:34:44.419 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.399+0000 7f8684dcd700 1 -- 172.21.15.23:0/4150119695 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f868013c1f0 con 0x7f86800a6ee0 2020-11-20T11:34:44.419 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.399+0000 7f8684dcd700 1 -- 172.21.15.23:0/4150119695 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86801483b0 con 0x7f86800a6ee0 2020-11-20T11:34:44.419 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.400+0000 7f8684dcd700 1 --2- 172.21.15.23:0/4150119695 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8680135530 0x7f8680135910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:44.419 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.400+0000 7f8684dcd700 1 -- 172.21.15.23:0/4150119695 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8680062c40 con 0x7f8680135530 2020-11-20T11:34:44.422 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.400+0000 7f867e59c700 1 --2- 172.21.15.23:0/4150119695 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8680135530 0x7f8680135910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:44.423 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.400+0000 7f867e59c700 1 -- 172.21.15.23:0/4150119695 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f868013ce40 con 0x7f8680135530 2020-11-20T11:34:44.423 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.400+0000 7f867e59c700 1 --2- 172.21.15.23:0/4150119695 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8680135530 0x7f8680135910 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f8670010b00 tx=0x7f8670008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:44.423 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.401+0000 7f866e7fc700 1 -- 172.21.15.23:0/4150119695 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8674023030 con 0x7f86800a6ee0 2020-11-20T11:34:44.423 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.401+0000 7f866e7fc700 1 --2- 172.21.15.23:0/4150119695 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8654056d60 0x7f8654059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:44.423 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.401+0000 7f866e7fc700 1 -- 172.21.15.23:0/4150119695 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f8674070f30 con 0x7f86800a6ee0 2020-11-20T11:34:44.424 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.401+0000 7f866e7fc700 1 -- 172.21.15.23:0/4150119695 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867000e9a0 con 0x7f8680135530 2020-11-20T11:34:44.424 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.401+0000 7f866e7fc700 1 -- 172.21.15.23:0/4150119695 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f867000eb00 con 0x7f8680135530 2020-11-20T11:34:44.424 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.401+0000 7f866e7fc700 1 -- 172.21.15.23:0/4150119695 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8680135530 msgr2=0x7f8680135910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:44.424 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.401+0000 7f866e7fc700 1 --2- 172.21.15.23:0/4150119695 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8680135530 0x7f8680135910 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f8670010b00 tx=0x7f8670008040).stop 2020-11-20T11:34:44.424 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.401+0000 7f866e7fc700 1 -- 172.21.15.23:0/4150119695 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8680135530 msgr2=0x7f8680135910 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:34:44.425 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.401+0000 7f867ed9d700 1 --2- 172.21.15.23:0/4150119695 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8654056d60 0x7f8654059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:44.425 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.401+0000 7f866e7fc700 1 --2- 172.21.15.23:0/4150119695 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8680135530 0x7f8680135910 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:44.425 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.402+0000 7f867ed9d700 1 --2- 172.21.15.23:0/4150119695 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8654056d60 0x7f8654059220 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f8678004290 tx=0x7f8678007ef0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:44.425 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.408+0000 7f8684dcd700 1 --2- 172.21.15.23:0/4150119695 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8680135e00 0x7f868014b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:44.426 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.408+0000 7f8684dcd700 1 -- 172.21.15.23:0/4150119695 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8680044710 con 0x7f8680135e00 2020-11-20T11:34:44.426 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.408+0000 7f867e59c700 1 --2- 172.21.15.23:0/4150119695 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8680135e00 0x7f868014b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:44.426 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.409+0000 7f867e59c700 1 -- 172.21.15.23:0/4150119695 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f868005c6f0 con 0x7f8680135e00 2020-11-20T11:34:44.426 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.409+0000 7f867e59c700 1 --2- 172.21.15.23:0/4150119695 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8680135e00 0x7f868014b340 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f8670018000 tx=0x7f8670008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:44.426 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.409+0000 7f866e7fc700 1 -- 172.21.15.23:0/4150119695 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8670006020 con 0x7f8680135e00 2020-11-20T11:34:44.427 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.409+0000 7f866e7fc700 1 -- 172.21.15.23:0/4150119695 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4784 (secure 0 0 0) 0x7f867000e790 con 0x7f8680135e00 2020-11-20T11:34:44.427 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.409+0000 7f866e7fc700 1 -- 172.21.15.23:0/4150119695 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8680135e00 msgr2=0x7f868014b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:44.427 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.409+0000 7f866e7fc700 1 --2- 172.21.15.23:0/4150119695 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8680135e00 0x7f868014b340 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f8670018000 tx=0x7f8670008040).stop 2020-11-20T11:34:44.427 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.409+0000 7f866e7fc700 1 -- 172.21.15.23:0/4150119695 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8680135e00 msgr2=0x7f868014b340 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:34:44.427 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.409+0000 7f866e7fc700 1 --2- 172.21.15.23:0/4150119695 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8680135e00 0x7f868014b340 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:44.428 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.409+0000 7f8684dcd700 1 -- 172.21.15.23:0/4150119695 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8654056d60 msgr2=0x7f8654059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:44.428 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.409+0000 7f8684dcd700 1 --2- 172.21.15.23:0/4150119695 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8654056d60 0x7f8654059220 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f8678004290 tx=0x7f8678007ef0).stop 2020-11-20T11:34:44.428 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.409+0000 7f8684dcd700 1 -- 172.21.15.23:0/4150119695 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86800a6ee0 msgr2=0x7f868009de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:44.428 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.409+0000 7f8684dcd700 1 --2- 172.21.15.23:0/4150119695 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86800a6ee0 0x7f868009de30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f8674009000 tx=0x7f8674015000).stop 2020-11-20T11:34:44.428 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.409+0000 7f867ed9d700 1 -- 172.21.15.23:0/4150119695 reap_dead start 2020-11-20T11:34:44.429 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.409+0000 7f867ed9d700 1 -- 172.21.15.23:0/4150119695 reap_dead start 2020-11-20T11:34:44.429 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.410+0000 7f8684dcd700 1 -- 172.21.15.23:0/4150119695 shutdown_connections 2020-11-20T11:34:44.429 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.410+0000 7f8684dcd700 1 -- 172.21.15.23:0/4150119695 >> 172.21.15.23:0/4150119695 conn(0x7f86800601b0 msgr2=0x7f8680068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:34:44.429 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.410+0000 7f8684dcd700 1 -- 172.21.15.23:0/4150119695 shutdown_connections 2020-11-20T11:34:44.429 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.410+0000 7f8684dcd700 1 -- 172.21.15.23:0/4150119695 wait complete. 2020-11-20T11:34:44.430 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:34:44.430 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:34:44.430 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:34:44.430 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:34:44.431 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 22, 2020-11-20T11:34:44.431 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:34:44.431 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:34:44.431 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:34:44.431 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:34:44.432 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:44.432 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 12, 2020-11-20T11:34:44.432 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:34:44.432 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:34:44.432 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:34:44.433 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:34:44.433 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:34:44.433 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:34:44.433 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:34:44.433 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:34:44.434 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:34:44.434 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:34:44.434 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:34:44.434 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:44.434 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:34:44.435 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:34:44.435 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:34:44.435 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:34:44.435 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:34:44.435 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:34:44.436 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:34:44.436 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:34:44.436 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:34:44.436 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:34:44.436 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:44.437 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.437 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:34:44.437 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:34:44.437 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:34:44.437 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:34:44.438 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:34:44.438 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:34:44.438 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:34:44.438 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:34:44.438 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:34:44.439 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:34:44.439 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:34:44.439 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:34:44.439 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:34:44.439 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:34:44.440 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:34:44.440 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:34:44.440 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:34:44.440 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:34:44.440 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:34:44.441 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:34:44.441 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:34:44.441 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:34:44.441 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:34:44.441 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:44.442 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:34:44.442 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.442 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:34:44.442 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.442 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:34:44.443 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:34:44.443 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:34:44.443 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:34:44.443 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.443 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:34:44.444 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:34:44.444 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:44.444 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.444 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.444 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:34:44.445 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:34:44.445 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:44.445 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:44.445 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:44.445 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.446 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:34:44.446 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:34:44.446 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:34:44.446 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:34:44.446 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:34:44.447 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.447 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.447 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:34:44.447 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:34:44.447 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:34:44.448 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:34:44.448 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.448 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:34:44.448 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:34:44.448 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:44.449 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.449 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.449 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:34:44.449 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:34:44.449 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:44.450 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:44.450 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:44.450 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.450 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:34:44.450 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:34:44.451 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:34:44.451 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:34:44.451 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:34:44.451 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.451 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.452 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:34:44.452 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:34:44.452 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:34:44.452 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:34:44.452 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.453 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:34:44.453 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:34:44.453 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:44.453 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.453 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.454 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:34:44.454 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:34:44.454 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:34:44.454 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:44.454 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:44.455 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.455 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:34:44.455 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:34:44.455 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:34:44.455 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:34:44.456 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:34:44.456 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:44.456 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:44.456 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.456 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:34:44.457 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:34:44.457 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.457 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:34:44.457 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:34:44.457 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:34:44.458 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:44.458 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:44.458 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:34:44.458 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.458 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:34:44.459 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:34:44.459 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:34:44.459 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:44.459 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:44.459 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:34:44.460 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.460 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:34:44.460 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:34:44.460 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 6 2020-11-20T11:34:44.460 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:44.461 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:44.461 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:34:44.461 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.461 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:34:44.461 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:34:44.462 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:34:44.462 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:44.462 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:44.462 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:34:44.462 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:34:44.463 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:34:44.463 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:34:44.463 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:34:44.463 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:34:44.463 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:44.464 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:44.464 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:34:44.464 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:34:44.465 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2020-11-20T11:34:44.465 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2020-11-20T11:34:44.466 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2020-11-20T11:34:44.466 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2020-11-20T11:34:44.466 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2020-11-20T11:34:44.467 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T11:34:44.477 INFO:tasks.ceph.mon.b:Stopped 2020-11-20T11:34:44.478 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2020-11-20T11:34:44.478 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:34:44.618 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.615+0000 7f77b0712700 1 Processor -- start 2020-11-20T11:34:44.619 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.616+0000 7f77b0712700 1 -- start start 2020-11-20T11:34:44.620 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.616+0000 7f77b0712700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f77a80ab0f0 0x7f77a80ab5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:44.621 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.616+0000 7f77b0712700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77a80a5690 con 0x7f77a80619d0 2020-11-20T11:34:44.621 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.616+0000 7f77b0712700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77a80a5eb0 con 0x7f77a8061db0 2020-11-20T11:34:44.621 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.616+0000 7f77b0712700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77a80a6150 con 0x7f77a80ab0f0 2020-11-20T11:34:44.621 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.616+0000 7f77adcad700 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f77a8061db0 legacy=0x7f77a80aa9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:34:44.622 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.616+0000 7f77aecaf700 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f77a80ab0f0 msgr2=0x7f77a80ab5a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:34:44.622 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.617+0000 7f77ae4ae700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f77a80619d0 0x7f77a80a9740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:56476/0 (socket says 172.21.15.23:56476) 2020-11-20T11:34:44.622 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.617+0000 7f77ae4ae700 1 -- 172.21.15.23:0/2360789495 learned_addr learned my addr 172.21.15.23:0/2360789495 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:34:44.622 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.617+0000 7f77aecaf700 1 --2- 172.21.15.23:0/2360789495 >> v2:172.21.15.155:3300/0 conn(0x7f77a80ab0f0 0x7f77a80ab5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:34:44.622 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.617+0000 7f77ad4ac700 1 -- 172.21.15.23:0/2360789495 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2016223159 0 0) 0x7f77a80a5690 con 0x7f77a80619d0 2020-11-20T11:34:44.623 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.617+0000 7f77ad4ac700 1 -- 172.21.15.23:0/2360789495 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7790003280 con 0x7f77a80619d0 2020-11-20T11:34:44.623 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.617+0000 7f77ad4ac700 1 -- 172.21.15.23:0/2360789495 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1753933638 0 0) 0x7f7790003280 con 0x7f77a80619d0 2020-11-20T11:34:44.623 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.617+0000 7f77ad4ac700 1 -- 172.21.15.23:0/2360789495 >> v2:172.21.15.155:3300/0 conn(0x7f77a80ab0f0 msgr2=0x7f77a80ab5a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:34:44.623 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.617+0000 7f77ad4ac700 1 --2- 172.21.15.23:0/2360789495 >> v2:172.21.15.155:3300/0 conn(0x7f77a80ab0f0 0x7f77a80ab5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:34:44.623 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.617+0000 7f77ad4ac700 1 -- 172.21.15.23:0/2360789495 >> v1:172.21.15.155:6789/0 conn(0x7f77a8061db0 legacy=0x7f77a80aa9e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:34:44.624 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.617+0000 7f77ad4ac700 1 -- 172.21.15.23:0/2360789495 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a80650b0 con 0x7f77a80619d0 2020-11-20T11:34:44.624 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.617+0000 7f77ad4ac700 1 -- 172.21.15.23:0/2360789495 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f77a4001520 con 0x7f77a80619d0 2020-11-20T11:34:44.624 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.617+0000 7f77ad4ac700 1 -- 172.21.15.23:0/2360789495 >> v1:172.21.15.23:6789/0 conn(0x7f77a80619d0 legacy=0x7f77a80a9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:44.624 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.617+0000 7f77ad4ac700 1 --2- 172.21.15.23:0/2360789495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7790003770 0x7f7790003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:44.625 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.617+0000 7f77ad4ac700 1 --2- 172.21.15.23:0/2360789495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77900040d0 0x7f77900045a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:44.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.617+0000 7f77ad4ac700 1 --2- 172.21.15.23:0/2360789495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7790004ba0 0x7f779000b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:44.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.617+0000 7f77ad4ac700 1 -- 172.21.15.23:0/2360789495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a80a6150 con 0x7f77900040d0 2020-11-20T11:34:44.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.617+0000 7f77ad4ac700 1 -- 172.21.15.23:0/2360789495 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7790005240 con 0x7f7790003770 2020-11-20T11:34:44.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.617+0000 7f77ad4ac700 1 -- 172.21.15.23:0/2360789495 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f779000bc60 con 0x7f7790004ba0 2020-11-20T11:34:44.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.617+0000 7f77ae4ae700 1 -- 172.21.15.23:0/2360789495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7790003770 msgr2=0x7f7790003b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:34:44.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.617+0000 7f77ae4ae700 1 --2- 172.21.15.23:0/2360789495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7790003770 0x7f7790003b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:34:44.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.617+0000 7f77adcad700 1 --2- 172.21.15.23:0/2360789495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77900040d0 0x7f77900045a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:44.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.618+0000 7f77aecaf700 1 --2- 172.21.15.23:0/2360789495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7790004ba0 0x7f779000b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:44.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.618+0000 7f77adcad700 1 -- 172.21.15.23:0/2360789495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7790004ba0 msgr2=0x7f779000b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:34:44.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.618+0000 7f77adcad700 1 --2- 172.21.15.23:0/2360789495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7790004ba0 0x7f779000b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:44.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.618+0000 7f77adcad700 1 -- 172.21.15.23:0/2360789495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7790003770 msgr2=0x7f7790003b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:34:44.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.618+0000 7f77adcad700 1 --2- 172.21.15.23:0/2360789495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7790003770 0x7f7790003b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:34:44.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.618+0000 7f77adcad700 1 -- 172.21.15.23:0/2360789495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7790003280 con 0x7f77900040d0 2020-11-20T11:34:44.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.618+0000 7f77aecaf700 1 --2- 172.21.15.23:0/2360789495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7790004ba0 0x7f779000b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:34:44.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.618+0000 7f77aecaf700 1 -- 172.21.15.23:0/2360789495 reap_dead start 2020-11-20T11:34:44.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.618+0000 7f77adcad700 1 --2- 172.21.15.23:0/2360789495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77900040d0 0x7f77900045a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f7798009990 tx=0x7f7798006ed0).ready entity=mon.0 client_cookie=ebe798d9342cfdd7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:44.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.618+0000 7f77ad4ac700 1 -- 172.21.15.23:0/2360789495 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77980052e0 con 0x7f77900040d0 2020-11-20T11:34:44.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.618+0000 7f77ad4ac700 1 -- 172.21.15.23:0/2360789495 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7798005440 con 0x7f77900040d0 2020-11-20T11:34:44.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.618+0000 7f77ad4ac700 1 -- 172.21.15.23:0/2360789495 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7798003940 con 0x7f77900040d0 2020-11-20T11:34:44.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.619+0000 7f77b0712700 1 -- 172.21.15.23:0/2360789495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77900040d0 msgr2=0x7f77900045a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:44.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.619+0000 7f77b0712700 1 --2- 172.21.15.23:0/2360789495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77900040d0 0x7f77900045a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f7798009990 tx=0x7f7798006ed0).stop 2020-11-20T11:34:44.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.619+0000 7f77b0712700 1 -- 172.21.15.23:0/2360789495 shutdown_connections 2020-11-20T11:34:44.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.619+0000 7f77b0712700 1 --2- 172.21.15.23:0/2360789495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77900040d0 0x7f77900045a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:44.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.619+0000 7f77b0712700 1 -- 172.21.15.23:0/2360789495 >> 172.21.15.23:0/2360789495 conn(0x7f77a8063640 msgr2=0x7f77a80612c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:34:44.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.619+0000 7f77b0712700 1 -- 172.21.15.23:0/2360789495 shutdown_connections 2020-11-20T11:34:44.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.619+0000 7f77b0712700 1 -- 172.21.15.23:0/2360789495 wait complete. 2020-11-20T11:34:44.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.620+0000 7f77b0712700 1 Processor -- start 2020-11-20T11:34:44.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.620+0000 7f77b0712700 1 -- start start 2020-11-20T11:34:44.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.620+0000 7f77b0712700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77900040d0 0x7f77a80a0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:44.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.620+0000 7f77b0712700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77a8117150 0x7f77a80671d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:44.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.620+0000 7f77b0712700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77a8066600 0x7f77a8066a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:44.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.620+0000 7f77b0712700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a80661a0 con 0x7f77a8117150 2020-11-20T11:34:44.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.620+0000 7f77b0712700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a814d760 con 0x7f77a8066600 2020-11-20T11:34:44.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.620+0000 7f77b0712700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77a8067a70 con 0x7f77900040d0 2020-11-20T11:34:44.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.620+0000 7f77adcad700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77a8117150 0x7f77a80671d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:44.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.620+0000 7f77adcad700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77a8117150 0x7f77a80671d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42238/0 (socket says 172.21.15.23:42238) 2020-11-20T11:34:44.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.620+0000 7f77aecaf700 1 -- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77a8066600 msgr2=0x7f77a8066a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:34:44.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.620+0000 7f77adcad700 1 -- 172.21.15.23:0/638415993 learned_addr learned my addr 172.21.15.23:0/638415993 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:34:44.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.620+0000 7f77aecaf700 1 --2- 172.21.15.23:0/638415993 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77a8066600 0x7f77a8066a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:34:44.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.621+0000 7f77ae4ae700 1 --2- 172.21.15.23:0/638415993 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77900040d0 0x7f77a80a0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:44.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.621+0000 7f77adcad700 1 -- 172.21.15.23:0/638415993 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77900040d0 msgr2=0x7f77a80a0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:44.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.621+0000 7f77adcad700 1 --2- 172.21.15.23:0/638415993 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77900040d0 0x7f77a80a0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:44.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.621+0000 7f77adcad700 1 -- 172.21.15.23:0/638415993 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77a8066600 msgr2=0x7f77a8066a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:34:44.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.621+0000 7f77adcad700 1 --2- 172.21.15.23:0/638415993 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77a8066600 0x7f77a8066a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:34:44.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.621+0000 7f77adcad700 1 -- 172.21.15.23:0/638415993 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a8067d10 con 0x7f77a8117150 2020-11-20T11:34:44.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.621+0000 7f77adcad700 1 --2- 172.21.15.23:0/638415993 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77a8117150 0x7f77a80671d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f7790005210 tx=0x7f7798014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:44.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.621+0000 7f77967fc700 1 -- 172.21.15.23:0/638415993 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77980055b0 con 0x7f77a8117150 2020-11-20T11:34:44.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.621+0000 7f77967fc700 1 -- 172.21.15.23:0/638415993 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7798012660 con 0x7f77a8117150 2020-11-20T11:34:44.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.621+0000 7f77b0712700 1 -- 172.21.15.23:0/638415993 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77a80a13e0 con 0x7f77a8117150 2020-11-20T11:34:44.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.621+0000 7f77967fc700 1 -- 172.21.15.23:0/638415993 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7798012a20 con 0x7f77a8117150 2020-11-20T11:34:44.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.621+0000 7f77b0712700 1 -- 172.21.15.23:0/638415993 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77a814e4b0 con 0x7f77a8117150 2020-11-20T11:34:44.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.622+0000 7f77b0712700 1 -- 172.21.15.23:0/638415993 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77a80a3550 con 0x7f77a8117150 2020-11-20T11:34:44.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.623+0000 7f77967fc700 1 -- 172.21.15.23:0/638415993 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7798025020 con 0x7f77a8117150 2020-11-20T11:34:44.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.623+0000 7f77967fc700 1 --2- 172.21.15.23:0/638415993 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7780056d10 0x7f77800591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:44.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.623+0000 7f77967fc700 1 -- 172.21.15.23:0/638415993 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f7798071960 con 0x7f77a8117150 2020-11-20T11:34:44.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.623+0000 7f77ae4ae700 1 --2- 172.21.15.23:0/638415993 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7780056d10 0x7f77800591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:44.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.624+0000 7f77ae4ae700 1 --2- 172.21.15.23:0/638415993 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7780056d10 0x7f77800591d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f77a4001a80 tx=0x7f77a80a5820).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:44.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.627+0000 7f77967fc700 1 -- 172.21.15.23:0/638415993 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7798046c50 con 0x7f77a8117150 2020-11-20T11:34:44.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.871+0000 7f77b0712700 1 -- 172.21.15.23:0/638415993 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f77a8044710 con 0x7f77a8117150 2020-11-20T11:34:44.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.872+0000 7f77967fc700 1 -- 172.21.15.23:0/638415993 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7f779804a270 con 0x7f77a8117150 2020-11-20T11:34:44.876 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:34:44.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.874+0000 7f77b0712700 1 -- 172.21.15.23:0/638415993 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7780056d10 msgr2=0x7f77800591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:44.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.874+0000 7f77b0712700 1 --2- 172.21.15.23:0/638415993 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7780056d10 0x7f77800591d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f77a4001a80 tx=0x7f77a80a5820).stop 2020-11-20T11:34:44.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.874+0000 7f77b0712700 1 -- 172.21.15.23:0/638415993 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77a8117150 msgr2=0x7f77a80671d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:44.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.874+0000 7f77b0712700 1 --2- 172.21.15.23:0/638415993 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77a8117150 0x7f77a80671d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f7790005210 tx=0x7f7798014000).stop 2020-11-20T11:34:44.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.874+0000 7f77b0712700 1 -- 172.21.15.23:0/638415993 shutdown_connections 2020-11-20T11:34:44.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.874+0000 7f77b0712700 1 --2- 172.21.15.23:0/638415993 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f77a8066600 0x7f77a8066a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:34:44.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.874+0000 7f77b0712700 1 --2- 172.21.15.23:0/638415993 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7780056d10 0x7f77800591d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:44.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.874+0000 7f77b0712700 1 --2- 172.21.15.23:0/638415993 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f77900040d0 0x7f77a80a0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:44.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.874+0000 7f77b0712700 1 --2- 172.21.15.23:0/638415993 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77a8117150 0x7f77a80671d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:44.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.874+0000 7f77b0712700 1 -- 172.21.15.23:0/638415993 >> 172.21.15.23:0/638415993 conn(0x7f77a8063640 msgr2=0x7f77a80a1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:34:44.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.874+0000 7f77b0712700 1 -- 172.21.15.23:0/638415993 shutdown_connections 2020-11-20T11:34:44.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:44.874+0000 7f77b0712700 1 -- 172.21.15.23:0/638415993 wait complete. 2020-11-20T11:34:47.890 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:34:48.023 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.020+0000 7f11b9736700 1 Processor -- start 2020-11-20T11:34:48.023 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.020+0000 7f11b9736700 1 -- start start 2020-11-20T11:34:48.026 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.021+0000 7f11b9736700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f11b40aaf00 0x7f11b413e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:48.026 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.021+0000 7f11b9736700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11b40a85c0 0x7f11b413a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:48.026 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.021+0000 7f11b9736700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f11b40ab2e0 0x7f11b40a78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:48.026 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.021+0000 7f11b9736700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11b413adb0 con 0x7f11b40ab2e0 2020-11-20T11:34:48.027 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.021+0000 7f11b9736700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11b413e010 con 0x7f11b40aaf00 2020-11-20T11:34:48.027 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.021+0000 7f11b9736700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11b413e2b0 con 0x7f11b40a85c0 2020-11-20T11:34:48.027 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.021+0000 7f11b2ffd700 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f11b40aaf00 msgr2=0x7f11b413e9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:34:48.027 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.021+0000 7f11b2ffd700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f11b40aaf00 0x7f11b413e9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:34:48.027 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.021+0000 7f11b37fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f11b40ab2e0 0x7f11b40a78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:48.028 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.021+0000 7f11b37fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f11b40ab2e0 0x7f11b40a78b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42258/0 (socket says 172.21.15.23:42258) 2020-11-20T11:34:48.028 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.021+0000 7f11b37fe700 1 -- 172.21.15.23:0/3213585612 learned_addr learned my addr 172.21.15.23:0/3213585612 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:34:48.028 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.021+0000 7f11b27fc700 1 --2- 172.21.15.23:0/3213585612 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11b40a85c0 0x7f11b413a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:48.028 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.021+0000 7f11b37fe700 1 -- 172.21.15.23:0/3213585612 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11b40a85c0 msgr2=0x7f11b413a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:34:48.029 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.021+0000 7f11b37fe700 1 --2- 172.21.15.23:0/3213585612 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11b40a85c0 0x7f11b413a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:48.029 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.021+0000 7f11b37fe700 1 -- 172.21.15.23:0/3213585612 >> v2:172.21.15.155:3300/0 conn(0x7f11b40aaf00 msgr2=0x7f11b413e9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:34:48.030 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.021+0000 7f11b37fe700 1 --2- 172.21.15.23:0/3213585612 >> v2:172.21.15.155:3300/0 conn(0x7f11b40aaf00 0x7f11b413e9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:34:48.030 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.021+0000 7f11b37fe700 1 -- 172.21.15.23:0/3213585612 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b413e550 con 0x7f11b40ab2e0 2020-11-20T11:34:48.030 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.022+0000 7f11b37fe700 1 --2- 172.21.15.23:0/3213585612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11b40ab2e0 0x7f11b40a78b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f11b40b10c0 tx=0x7f11ac0092d0).ready entity=mon.0 client_cookie=26b99bebac713f98 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:48.030 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.022+0000 7f11b1ffb700 1 -- 172.21.15.23:0/3213585612 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac00bbf0 con 0x7f11b40ab2e0 2020-11-20T11:34:48.030 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.022+0000 7f11b1ffb700 1 -- 172.21.15.23:0/3213585612 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ac00bd50 con 0x7f11b40ab2e0 2020-11-20T11:34:48.031 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.022+0000 7f11b1ffb700 1 -- 172.21.15.23:0/3213585612 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac011610 con 0x7f11b40ab2e0 2020-11-20T11:34:48.031 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.022+0000 7f11b9736700 1 -- 172.21.15.23:0/3213585612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11b40ab2e0 msgr2=0x7f11b40a78b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:48.031 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.022+0000 7f11b9736700 1 --2- 172.21.15.23:0/3213585612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11b40ab2e0 0x7f11b40a78b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f11b40b10c0 tx=0x7f11ac0092d0).stop 2020-11-20T11:34:48.031 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.022+0000 7f11b9736700 1 -- 172.21.15.23:0/3213585612 shutdown_connections 2020-11-20T11:34:48.031 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.022+0000 7f11b9736700 1 --2- 172.21.15.23:0/3213585612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11b40ab2e0 0x7f11b40a78b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:48.032 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.022+0000 7f11b9736700 1 --2- 172.21.15.23:0/3213585612 >> v2:172.21.15.155:3300/0 conn(0x7f11b40aaf00 0x7f11b413e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:34:48.032 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.022+0000 7f11b9736700 1 --2- 172.21.15.23:0/3213585612 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11b40a85c0 0x7f11b413a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:48.032 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.022+0000 7f11b9736700 1 -- 172.21.15.23:0/3213585612 >> 172.21.15.23:0/3213585612 conn(0x7f11b40678e0 msgr2=0x7f11b4067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:34:48.032 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.022+0000 7f11b9736700 1 -- 172.21.15.23:0/3213585612 shutdown_connections 2020-11-20T11:34:48.032 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.022+0000 7f11b9736700 1 -- 172.21.15.23:0/3213585612 wait complete. 2020-11-20T11:34:48.035 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.024+0000 7f11b9736700 1 Processor -- start 2020-11-20T11:34:48.035 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.024+0000 7f11b9736700 1 -- start start 2020-11-20T11:34:48.035 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.024+0000 7f11b9736700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11b40a85c0 0x7f11b40a1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:48.036 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.024+0000 7f11b9736700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11b40aaf00 0x7f11b414a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:48.036 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.024+0000 7f11b9736700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11b40ab2e0 0x7f11b409e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:48.036 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.024+0000 7f11b9736700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b4149b60 con 0x7f11b40aaf00 2020-11-20T11:34:48.036 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.024+0000 7f11b9736700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b4149d50 con 0x7f11b40ab2e0 2020-11-20T11:34:48.036 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.024+0000 7f11b9736700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11b4149ff0 con 0x7f11b40a85c0 2020-11-20T11:34:48.037 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.024+0000 7f11b27fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11b40aaf00 0x7f11b414a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:48.037 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.024+0000 7f11b27fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11b40aaf00 0x7f11b414a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42262/0 (socket says 172.21.15.23:42262) 2020-11-20T11:34:48.037 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.024+0000 7f11b27fc700 1 -- 172.21.15.23:0/955468086 learned_addr learned my addr 172.21.15.23:0/955468086 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:34:48.037 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.024+0000 7f11b37fe700 1 -- 172.21.15.23:0/955468086 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11b40ab2e0 msgr2=0x7f11b409e9e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:34:48.038 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.024+0000 7f11b37fe700 1 --2- 172.21.15.23:0/955468086 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11b40ab2e0 0x7f11b409e9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:34:48.038 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.025+0000 7f11b2ffd700 1 --2- 172.21.15.23:0/955468086 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11b40a85c0 0x7f11b40a1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:48.038 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.025+0000 7f11b27fc700 1 -- 172.21.15.23:0/955468086 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11b40a85c0 msgr2=0x7f11b40a1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:48.038 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.025+0000 7f11b27fc700 1 --2- 172.21.15.23:0/955468086 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11b40a85c0 0x7f11b40a1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:48.038 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.025+0000 7f11b27fc700 1 -- 172.21.15.23:0/955468086 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11b40ab2e0 msgr2=0x7f11b409e9e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:34:48.039 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.025+0000 7f11b27fc700 1 --2- 172.21.15.23:0/955468086 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11b40ab2e0 0x7f11b409e9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:34:48.039 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.025+0000 7f11b27fc700 1 -- 172.21.15.23:0/955468086 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b409d780 con 0x7f11b40aaf00 2020-11-20T11:34:48.039 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.025+0000 7f11b2ffd700 1 --2- 172.21.15.23:0/955468086 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11b40a85c0 0x7f11b40a1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:34:48.039 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.025+0000 7f11b27fc700 1 --2- 172.21.15.23:0/955468086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11b40aaf00 0x7f11b414a490 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f11a800eba0 tx=0x7f11a8006e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:48.039 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.025+0000 7f119affd700 1 -- 172.21.15.23:0/955468086 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a800c9d0 con 0x7f11b40aaf00 2020-11-20T11:34:48.040 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.025+0000 7f119affd700 1 -- 172.21.15.23:0/955468086 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11a800cb30 con 0x7f11b40aaf00 2020-11-20T11:34:48.040 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.025+0000 7f11b9736700 1 -- 172.21.15.23:0/955468086 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11b414a980 con 0x7f11b40aaf00 2020-11-20T11:34:48.040 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.025+0000 7f119affd700 1 -- 172.21.15.23:0/955468086 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a80123f0 con 0x7f11b40aaf00 2020-11-20T11:34:48.041 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.025+0000 7f11b9736700 1 -- 172.21.15.23:0/955468086 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11b40a72c0 con 0x7f11b40aaf00 2020-11-20T11:34:48.041 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.026+0000 7f11b9736700 1 -- 172.21.15.23:0/955468086 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11b4057dc0 con 0x7f11b40aaf00 2020-11-20T11:34:48.041 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.026+0000 7f119affd700 1 -- 172.21.15.23:0/955468086 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f11a80127e0 con 0x7f11b40aaf00 2020-11-20T11:34:48.041 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.027+0000 7f119affd700 1 --2- 172.21.15.23:0/955468086 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1190056d10 0x7f11900591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:48.042 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.027+0000 7f119affd700 1 -- 172.21.15.23:0/955468086 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f11a80704e0 con 0x7f11b40aaf00 2020-11-20T11:34:48.042 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.027+0000 7f11b2ffd700 1 --2- 172.21.15.23:0/955468086 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1190056d10 0x7f11900591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:48.042 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.028+0000 7f11b2ffd700 1 --2- 172.21.15.23:0/955468086 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1190056d10 0x7f11900591d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f11b40a7140 tx=0x7f11a4008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:48.043 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.031+0000 7f119affd700 1 -- 172.21.15.23:0/955468086 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f11a80457d0 con 0x7f11b40aaf00 2020-11-20T11:34:48.281 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.275+0000 7f11b9736700 1 -- 172.21.15.23:0/955468086 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f11b40a9000 con 0x7f11b40aaf00 2020-11-20T11:34:48.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.276+0000 7f119affd700 1 -- 172.21.15.23:0/955468086 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7f11a8048df0 con 0x7f11b40aaf00 2020-11-20T11:34:48.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.278+0000 7f11b9736700 1 -- 172.21.15.23:0/955468086 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1190056d10 msgr2=0x7f11900591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:48.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.278+0000 7f11b9736700 1 --2- 172.21.15.23:0/955468086 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1190056d10 0x7f11900591d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f11b40a7140 tx=0x7f11a4008040).stop 2020-11-20T11:34:48.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.278+0000 7f11b9736700 1 -- 172.21.15.23:0/955468086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11b40aaf00 msgr2=0x7f11b414a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:48.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.278+0000 7f11b9736700 1 --2- 172.21.15.23:0/955468086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11b40aaf00 0x7f11b414a490 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f11a800eba0 tx=0x7f11a8006e30).stop 2020-11-20T11:34:48.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.278+0000 7f11b9736700 1 -- 172.21.15.23:0/955468086 shutdown_connections 2020-11-20T11:34:48.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.278+0000 7f11b9736700 1 --2- 172.21.15.23:0/955468086 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11b40ab2e0 0x7f11b409e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:34:48.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.278+0000 7f11b9736700 1 --2- 172.21.15.23:0/955468086 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1190056d10 0x7f11900591d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:48.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.278+0000 7f11b9736700 1 --2- 172.21.15.23:0/955468086 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11b40a85c0 0x7f11b40a1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:48.287 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.278+0000 7f11b9736700 1 --2- 172.21.15.23:0/955468086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11b40aaf00 0x7f11b414a490 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:48.287 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.278+0000 7f11b9736700 1 -- 172.21.15.23:0/955468086 >> 172.21.15.23:0/955468086 conn(0x7f11b40678e0 msgr2=0x7f11b40a9340 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:34:48.288 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.278+0000 7f11b9736700 1 -- 172.21.15.23:0/955468086 shutdown_connections 2020-11-20T11:34:48.288 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:48.278+0000 7f11b9736700 1 -- 172.21.15.23:0/955468086 wait complete. 2020-11-20T11:34:48.289 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:34:51.297 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:34:51.429 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.426+0000 7f49e2e57700 1 Processor -- start 2020-11-20T11:34:51.429 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.426+0000 7f49e2e57700 1 -- start start 2020-11-20T11:34:51.431 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.427+0000 7f49e2e57700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f49dc0ab2a0 0x7f49dc13e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:51.431 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.427+0000 7f49e2e57700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f49dc0a4c20 0x7f49dc0a8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:51.431 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.427+0000 7f49e2e57700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49dc0b1a20 con 0x7f49dc0ad8d0 2020-11-20T11:34:51.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.427+0000 7f49e2e57700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49dc13f4a0 con 0x7f49dc0ab2a0 2020-11-20T11:34:51.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.427+0000 7f49e2e57700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49dc0aa0e0 con 0x7f49dc0a4c20 2020-11-20T11:34:51.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.427+0000 7f49dbfff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f49dc0ad8d0 0x7f49dc0a4510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:56522/0 (socket says 172.21.15.23:56522) 2020-11-20T11:34:51.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.427+0000 7f49dbfff700 1 -- 172.21.15.23:0/3548555679 learned_addr learned my addr 172.21.15.23:0/3548555679 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:34:51.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.427+0000 7f49e0bf3700 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f49dc0ab2a0 msgr2=0x7f49dc13e570 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:34:51.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.427+0000 7f49e0bf3700 1 --2- 172.21.15.23:0/3548555679 >> v2:172.21.15.155:3300/0 conn(0x7f49dc0ab2a0 0x7f49dc13e570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:34:51.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.427+0000 7f49e13f4700 1 --2- 172.21.15.23:0/3548555679 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f49dc0a4c20 0x7f49dc0a8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:51.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.428+0000 7f49e13f4700 1 -- 172.21.15.23:0/3548555679 >> v2:172.21.15.155:3300/0 conn(0x7f49dc0ab2a0 msgr2=0x7f49dc13e570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:34:51.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.428+0000 7f49e13f4700 1 --2- 172.21.15.23:0/3548555679 >> v2:172.21.15.155:3300/0 conn(0x7f49dc0ab2a0 0x7f49dc13e570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:34:51.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.428+0000 7f49e13f4700 1 -- 172.21.15.23:0/3548555679 >> v1:172.21.15.23:6789/0 conn(0x7f49dc0ad8d0 legacy=0x7f49dc0a4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:51.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.428+0000 7f49e13f4700 1 -- 172.21.15.23:0/3548555679 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49dc0a8b90 con 0x7f49dc0a4c20 2020-11-20T11:34:51.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.428+0000 7f49e13f4700 1 --2- 172.21.15.23:0/3548555679 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f49dc0a4c20 0x7f49dc0a8fc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f49d4009ce0 tx=0x7f49d4009220).ready entity=mon.2 client_cookie=587b57a0317fd741 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:51.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.428+0000 7f49db7fe700 1 -- 172.21.15.23:0/3548555679 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d400bbf0 con 0x7f49dc0a4c20 2020-11-20T11:34:51.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.428+0000 7f49db7fe700 1 -- 172.21.15.23:0/3548555679 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49d400bd50 con 0x7f49dc0a4c20 2020-11-20T11:34:51.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.428+0000 7f49db7fe700 1 -- 172.21.15.23:0/3548555679 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d4011610 con 0x7f49dc0a4c20 2020-11-20T11:34:51.438 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.429+0000 7f49e2e57700 1 -- 172.21.15.23:0/3548555679 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f49dc0a4c20 msgr2=0x7f49dc0a8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:51.438 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.429+0000 7f49e2e57700 1 --2- 172.21.15.23:0/3548555679 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f49dc0a4c20 0x7f49dc0a8fc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f49d4009ce0 tx=0x7f49d4009220).stop 2020-11-20T11:34:51.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.429+0000 7f49e2e57700 1 -- 172.21.15.23:0/3548555679 shutdown_connections 2020-11-20T11:34:51.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.429+0000 7f49e2e57700 1 --2- 172.21.15.23:0/3548555679 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f49dc0a4c20 0x7f49dc0a8fc0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:51.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.429+0000 7f49e2e57700 1 --2- 172.21.15.23:0/3548555679 >> v2:172.21.15.155:3300/0 conn(0x7f49dc0ab2a0 0x7f49dc13e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:34:51.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.429+0000 7f49e2e57700 1 -- 172.21.15.23:0/3548555679 >> 172.21.15.23:0/3548555679 conn(0x7f49dc0683b0 msgr2=0x7f49dc0b1310 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:34:51.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.429+0000 7f49e2e57700 1 -- 172.21.15.23:0/3548555679 shutdown_connections 2020-11-20T11:34:51.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.429+0000 7f49e2e57700 1 -- 172.21.15.23:0/3548555679 wait complete. 2020-11-20T11:34:51.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.430+0000 7f49e2e57700 1 Processor -- start 2020-11-20T11:34:51.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.430+0000 7f49e2e57700 1 -- start start 2020-11-20T11:34:51.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.430+0000 7f49e2e57700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49dc0a4c20 0x7f49dc0a2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:51.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.431+0000 7f49e2e57700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49dc0ab2a0 0x7f49dc0a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:51.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.431+0000 7f49e2e57700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f49dc0ad8d0 0x7f49dc14d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:51.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.431+0000 7f49e2e57700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49dc13e8a0 con 0x7f49dc0ab2a0 2020-11-20T11:34:51.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.431+0000 7f49e2e57700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49dc09fa30 con 0x7f49dc0a4c20 2020-11-20T11:34:51.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.431+0000 7f49e2e57700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49dc09fcd0 con 0x7f49dc0ad8d0 2020-11-20T11:34:51.442 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.431+0000 7f49e0bf3700 1 -- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49dc0a4c20 msgr2=0x7f49dc0a2850 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:34:51.442 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.431+0000 7f49e0bf3700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49dc0a4c20 0x7f49dc0a2850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:34:51.442 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.431+0000 7f49dbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49dc0ab2a0 0x7f49dc0a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:51.442 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.431+0000 7f49dbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49dc0ab2a0 0x7f49dc0a0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42280/0 (socket says 172.21.15.23:42280) 2020-11-20T11:34:51.442 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.431+0000 7f49dbfff700 1 -- 172.21.15.23:0/146296794 learned_addr learned my addr 172.21.15.23:0/146296794 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:34:51.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.431+0000 7f49e13f4700 1 --2- 172.21.15.23:0/146296794 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f49dc0ad8d0 0x7f49dc14d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:51.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.431+0000 7f49dbfff700 1 -- 172.21.15.23:0/146296794 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f49dc0ad8d0 msgr2=0x7f49dc14d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:51.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.431+0000 7f49dbfff700 1 --2- 172.21.15.23:0/146296794 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f49dc0ad8d0 0x7f49dc14d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:51.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.431+0000 7f49dbfff700 1 -- 172.21.15.23:0/146296794 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49dc0a4c20 msgr2=0x7f49dc0a2850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:34:51.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.431+0000 7f49dbfff700 1 --2- 172.21.15.23:0/146296794 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49dc0a4c20 0x7f49dc0a2850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:34:51.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.431+0000 7f49dbfff700 1 -- 172.21.15.23:0/146296794 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49dc09ff70 con 0x7f49dc0ab2a0 2020-11-20T11:34:51.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.431+0000 7f49e13f4700 1 --2- 172.21.15.23:0/146296794 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f49dc0ad8d0 0x7f49dc14d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:34:51.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.431+0000 7f49dbfff700 1 --2- 172.21.15.23:0/146296794 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49dc0ab2a0 0x7f49dc0a0dc0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f49c800c710 tx=0x7f49c8007ce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:51.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.431+0000 7f49d8ff9700 1 -- 172.21.15.23:0/146296794 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c80072e0 con 0x7f49dc0ab2a0 2020-11-20T11:34:51.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.432+0000 7f49d8ff9700 1 -- 172.21.15.23:0/146296794 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c8007440 con 0x7f49dc0ab2a0 2020-11-20T11:34:51.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.432+0000 7f49d8ff9700 1 -- 172.21.15.23:0/146296794 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c8005840 con 0x7f49dc0ab2a0 2020-11-20T11:34:51.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.432+0000 7f49e2e57700 1 -- 172.21.15.23:0/146296794 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49dc0a12b0 con 0x7f49dc0ab2a0 2020-11-20T11:34:51.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.432+0000 7f49e2e57700 1 -- 172.21.15.23:0/146296794 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49dc09f600 con 0x7f49dc0ab2a0 2020-11-20T11:34:51.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.432+0000 7f49e2e57700 1 -- 172.21.15.23:0/146296794 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49dc0af330 con 0x7f49dc0ab2a0 2020-11-20T11:34:51.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.433+0000 7f49d8ff9700 1 -- 172.21.15.23:0/146296794 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f49c8010070 con 0x7f49dc0ab2a0 2020-11-20T11:34:51.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.433+0000 7f49d8ff9700 1 --2- 172.21.15.23:0/146296794 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f49b4056d10 0x7f49b40591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:51.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.433+0000 7f49d8ff9700 1 -- 172.21.15.23:0/146296794 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f49c806f5d0 con 0x7f49dc0ab2a0 2020-11-20T11:34:51.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.434+0000 7f49e0bf3700 1 --2- 172.21.15.23:0/146296794 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f49b4056d10 0x7f49b40591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:51.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.434+0000 7f49e0bf3700 1 --2- 172.21.15.23:0/146296794 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f49b4056d10 0x7f49b40591d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f49d0006fd0 tx=0x7f49d0006b20).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:51.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.438+0000 7f49d8ff9700 1 -- 172.21.15.23:0/146296794 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f49c8044970 con 0x7f49dc0ab2a0 2020-11-20T11:34:51.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.665+0000 7f49e2e57700 1 -- 172.21.15.23:0/146296794 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f49dc060640 con 0x7f49dc0ab2a0 2020-11-20T11:34:51.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.665+0000 7f49d8ff9700 1 -- 172.21.15.23:0/146296794 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7f49c8047f90 con 0x7f49dc0ab2a0 2020-11-20T11:34:51.669 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:34:51.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.667+0000 7f49e2e57700 1 -- 172.21.15.23:0/146296794 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f49b4056d10 msgr2=0x7f49b40591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:51.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.667+0000 7f49e2e57700 1 --2- 172.21.15.23:0/146296794 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f49b4056d10 0x7f49b40591d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f49d0006fd0 tx=0x7f49d0006b20).stop 2020-11-20T11:34:51.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.667+0000 7f49e2e57700 1 -- 172.21.15.23:0/146296794 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49dc0ab2a0 msgr2=0x7f49dc0a0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:51.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.667+0000 7f49e2e57700 1 --2- 172.21.15.23:0/146296794 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49dc0ab2a0 0x7f49dc0a0dc0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f49c800c710 tx=0x7f49c8007ce0).stop 2020-11-20T11:34:51.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.667+0000 7f49e2e57700 1 -- 172.21.15.23:0/146296794 shutdown_connections 2020-11-20T11:34:51.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.667+0000 7f49e2e57700 1 --2- 172.21.15.23:0/146296794 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f49b4056d10 0x7f49b40591d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:51.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.667+0000 7f49e2e57700 1 --2- 172.21.15.23:0/146296794 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f49dc0ad8d0 0x7f49dc14d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:51.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.667+0000 7f49e2e57700 1 --2- 172.21.15.23:0/146296794 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f49dc0a4c20 0x7f49dc0a2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:34:51.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.667+0000 7f49e2e57700 1 --2- 172.21.15.23:0/146296794 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f49dc0ab2a0 0x7f49dc0a0dc0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:51.675 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.667+0000 7f49e2e57700 1 -- 172.21.15.23:0/146296794 >> 172.21.15.23:0/146296794 conn(0x7f49dc0683b0 msgr2=0x7f49dc1419d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:34:51.675 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.667+0000 7f49e2e57700 1 -- 172.21.15.23:0/146296794 shutdown_connections 2020-11-20T11:34:51.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:51.667+0000 7f49e2e57700 1 -- 172.21.15.23:0/146296794 wait complete. 2020-11-20T11:34:54.695 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:34:55.149 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:55.146+0000 7fd8e359e700 1 Processor -- start 2020-11-20T11:34:55.149 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:55.146+0000 7fd8e359e700 1 -- start start 2020-11-20T11:34:55.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:55.147+0000 7fd8e359e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd8e40ace70 0x7fd8e413a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:55.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:55.147+0000 7fd8e359e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8e40ad2f0 0x7fd8e413ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:55.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:55.147+0000 7fd8e359e700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8e40ad860 con 0x7fd8e40a9040 2020-11-20T11:34:55.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:55.147+0000 7fd8e359e700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8e40aa8a0 con 0x7fd8e40ace70 2020-11-20T11:34:55.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:55.147+0000 7fd8e359e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e40aaa90 con 0x7fd8e40ad2f0 2020-11-20T11:34:55.152 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:55.147+0000 7fd8e259c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fd8e40a9040 0x7fd8e40aaf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:56534/0 (socket says 172.21.15.23:56534) 2020-11-20T11:34:55.152 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:55.147+0000 7fd8e1d9b700 1 -- >> v2:172.21.15.155:3300/0 conn(0x7fd8e40ace70 msgr2=0x7fd8e413a830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:34:55.152 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:55.147+0000 7fd8e259c700 1 -- 172.21.15.23:0/416344106 learned_addr learned my addr 172.21.15.23:0/416344106 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:34:55.152 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:55.147+0000 7fd8e1d9b700 1 --2- 172.21.15.23:0/416344106 >> v2:172.21.15.155:3300/0 conn(0x7fd8e40ace70 0x7fd8e413a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:34:55.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:55.147+0000 7fd8e2d9d700 1 --2- 172.21.15.23:0/416344106 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8e40ad2f0 0x7fd8e413ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:55.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:55.147+0000 7fd8e2d9d700 1 -- 172.21.15.23:0/416344106 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8e40ad2f0 msgr2=0x7fd8e413ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:34:55.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:55.147+0000 7fd8e2d9d700 1 -- 172.21.15.23:0/416344106 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8e40ad2f0 msgr2=0x7fd8e413ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:34:55.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:55.147+0000 7fd8e2d9d700 1 --2- 172.21.15.23:0/416344106 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8e40ad2f0 0x7fd8e413ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:34:55.155 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:55.147+0000 7fd8e2d9d700 1 --2- 172.21.15.23:0/416344106 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8e40ad2f0 0x7fd8e413ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:34:55.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:55.348+0000 7fd8e1d9b700 1 -- 172.21.15.23:0/416344106 >> v2:172.21.15.155:3300/0 conn(0x7fd8e40ace70 msgr2=0x7fd8e413a830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:34:55.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:55.348+0000 7fd8e1d9b700 1 --2- 172.21.15.23:0/416344106 >> v2:172.21.15.155:3300/0 conn(0x7fd8e40ace70 0x7fd8e413a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:34:55.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:55.348+0000 7fd8e2d9d700 1 --2- 172.21.15.23:0/416344106 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8e40ad2f0 0x7fd8e413ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:55.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:55.348+0000 7fd8e2d9d700 1 -- 172.21.15.23:0/416344106 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8e40ad2f0 msgr2=0x7fd8e413ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:34:55.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:55.348+0000 7fd8e2d9d700 1 -- 172.21.15.23:0/416344106 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8e40ad2f0 msgr2=0x7fd8e413ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:34:55.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:55.348+0000 7fd8e2d9d700 1 --2- 172.21.15.23:0/416344106 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8e40ad2f0 0x7fd8e413ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:34:55.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:55.348+0000 7fd8e2d9d700 1 --2- 172.21.15.23:0/416344106 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8e40ad2f0 0x7fd8e413ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:34:55.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:55.749+0000 7fd8e1d9b700 1 -- 172.21.15.23:0/416344106 >> v2:172.21.15.155:3300/0 conn(0x7fd8e40ace70 msgr2=0x7fd8e413a830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:34:55.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:55.749+0000 7fd8e1d9b700 1 --2- 172.21.15.23:0/416344106 >> v2:172.21.15.155:3300/0 conn(0x7fd8e40ace70 0x7fd8e413a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:34:55.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:55.749+0000 7fd8e2d9d700 1 --2- 172.21.15.23:0/416344106 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8e40ad2f0 0x7fd8e413ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:55.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:55.749+0000 7fd8e2d9d700 1 -- 172.21.15.23:0/416344106 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8e40ad2f0 msgr2=0x7fd8e413ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:34:55.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:55.749+0000 7fd8e2d9d700 1 -- 172.21.15.23:0/416344106 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8e40ad2f0 msgr2=0x7fd8e413ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:34:55.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:55.749+0000 7fd8e2d9d700 1 --2- 172.21.15.23:0/416344106 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8e40ad2f0 0x7fd8e413ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:34:55.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:55.749+0000 7fd8e2d9d700 1 --2- 172.21.15.23:0/416344106 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8e40ad2f0 0x7fd8e413ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:34:56.553 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:56.550+0000 7fd8e1d9b700 1 -- 172.21.15.23:0/416344106 >> v2:172.21.15.155:3300/0 conn(0x7fd8e40ace70 msgr2=0x7fd8e413a830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:34:56.554 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:56.550+0000 7fd8e1d9b700 1 --2- 172.21.15.23:0/416344106 >> v2:172.21.15.155:3300/0 conn(0x7fd8e40ace70 0x7fd8e413a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:34:56.554 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:56.550+0000 7fd8e2d9d700 1 --2- 172.21.15.23:0/416344106 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8e40ad2f0 0x7fd8e413ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:56.554 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:56.550+0000 7fd8e2d9d700 1 -- 172.21.15.23:0/416344106 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8e40ad2f0 msgr2=0x7fd8e413ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:34:56.555 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:56.550+0000 7fd8e2d9d700 1 -- 172.21.15.23:0/416344106 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8e40ad2f0 msgr2=0x7fd8e413ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:34:56.555 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:56.550+0000 7fd8e2d9d700 1 --2- 172.21.15.23:0/416344106 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8e40ad2f0 0x7fd8e413ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:34:56.555 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:56.550+0000 7fd8e2d9d700 1 --2- 172.21.15.23:0/416344106 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8e40ad2f0 0x7fd8e413ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:34:58.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:58.147+0000 7fd8d3fff700 1 -- 172.21.15.23:0/416344106 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8e40ad2f0 msgr2=0x7fd8e413ad20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:34:58.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:58.147+0000 7fd8d3fff700 1 --2- 172.21.15.23:0/416344106 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8e40ad2f0 0x7fd8e413ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:58.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:58.147+0000 7fd8d3fff700 1 -- 172.21.15.23:0/416344106 >> v2:172.21.15.155:3300/0 conn(0x7fd8e40ace70 msgr2=0x7fd8e413a830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:34:58.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:58.147+0000 7fd8d3fff700 1 --2- 172.21.15.23:0/416344106 >> v2:172.21.15.155:3300/0 conn(0x7fd8e40ace70 0x7fd8e413a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:34:58.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:58.147+0000 7fd8d3fff700 1 -- 172.21.15.23:0/416344106 >> v1:172.21.15.23:6789/0 conn(0x7fd8e40a9040 legacy=0x7fd8e40aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:58.155 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:58.147+0000 7fd8d3fff700 1 --2- 172.21.15.23:0/416344106 >> v2:172.21.15.23:3300/0 conn(0x7fd8c0003410 0x7fd8c0005870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:58.155 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:58.147+0000 7fd8d3fff700 1 -- 172.21.15.23:0/416344106 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8e40a9420 con 0x7fd8c0005e70 2020-11-20T11:34:58.155 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:58.147+0000 7fd8d3fff700 1 -- 172.21.15.23:0/416344106 --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8e40aaa90 con 0x7fd8c0003410 2020-11-20T11:34:58.155 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:58.147+0000 7fd8e1d9b700 1 -- 172.21.15.23:0/416344106 >> v1:172.21.15.155:6789/0 conn(0x7fd8c0005e70 legacy=0x7fd8c0008330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:34:58.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:58.147+0000 7fd8e259c700 1 --2- 172.21.15.23:0/416344106 >> v2:172.21.15.23:3300/0 conn(0x7fd8c0003410 0x7fd8c0005870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:58.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:58.147+0000 7fd8e259c700 1 -- 172.21.15.23:0/416344106 >> v2:172.21.15.23:3300/0 conn(0x7fd8c0003410 msgr2=0x7fd8c0005870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:34:58.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:58.147+0000 7fd8e259c700 1 -- 172.21.15.23:0/416344106 >> v2:172.21.15.23:3300/0 conn(0x7fd8c0003410 msgr2=0x7fd8c0005870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:34:58.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:58.147+0000 7fd8e259c700 1 --2- 172.21.15.23:0/416344106 >> v2:172.21.15.23:3300/0 conn(0x7fd8c0003410 0x7fd8c0005870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:34:58.157 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:58.147+0000 7fd8e259c700 1 --2- 172.21.15.23:0/416344106 >> v2:172.21.15.23:3300/0 conn(0x7fd8c0003410 0x7fd8c0005870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:34:58.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:58.348+0000 7fd8e1d9b700 1 -- 172.21.15.23:0/416344106 >> v1:172.21.15.155:6789/0 conn(0x7fd8c0005e70 legacy=0x7fd8c0008330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:34:58.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:58.348+0000 7fd8e259c700 1 --2- 172.21.15.23:0/416344106 >> v2:172.21.15.23:3300/0 conn(0x7fd8c0003410 0x7fd8c0005870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:58.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:58.348+0000 7fd8e259c700 1 -- 172.21.15.23:0/416344106 >> v2:172.21.15.23:3300/0 conn(0x7fd8c0003410 msgr2=0x7fd8c0005870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:34:58.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:58.348+0000 7fd8e259c700 1 -- 172.21.15.23:0/416344106 >> v2:172.21.15.23:3300/0 conn(0x7fd8c0003410 msgr2=0x7fd8c0005870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:34:58.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:58.348+0000 7fd8e259c700 1 --2- 172.21.15.23:0/416344106 >> v2:172.21.15.23:3300/0 conn(0x7fd8c0003410 0x7fd8c0005870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:34:58.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:58.348+0000 7fd8e259c700 1 --2- 172.21.15.23:0/416344106 >> v2:172.21.15.23:3300/0 conn(0x7fd8c0003410 0x7fd8c0005870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:34:58.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:58.749+0000 7fd8e259c700 1 --2- 172.21.15.23:0/416344106 >> v2:172.21.15.23:3300/0 conn(0x7fd8c0003410 0x7fd8c0005870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:58.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:58.749+0000 7fd8e1d9b700 1 -- 172.21.15.23:0/416344106 >> v1:172.21.15.155:6789/0 conn(0x7fd8c0005e70 legacy=0x7fd8c0008330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:34:58.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:58.749+0000 7fd8e259c700 1 -- 172.21.15.23:0/416344106 >> v2:172.21.15.23:3300/0 conn(0x7fd8c0003410 msgr2=0x7fd8c0005870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:34:58.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:58.749+0000 7fd8e259c700 1 -- 172.21.15.23:0/416344106 >> v2:172.21.15.23:3300/0 conn(0x7fd8c0003410 msgr2=0x7fd8c0005870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:34:58.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:58.749+0000 7fd8e259c700 1 --2- 172.21.15.23:0/416344106 >> v2:172.21.15.23:3300/0 conn(0x7fd8c0003410 0x7fd8c0005870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:34:58.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:58.749+0000 7fd8e259c700 1 --2- 172.21.15.23:0/416344106 >> v2:172.21.15.23:3300/0 conn(0x7fd8c0003410 0x7fd8c0005870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:34:59.557 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.550+0000 7fd8e1d9b700 1 -- 172.21.15.23:0/416344106 >> v1:172.21.15.155:6789/0 conn(0x7fd8c0005e70 legacy=0x7fd8c0008330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:34:59.558 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.551+0000 7fd8e259c700 1 --2- 172.21.15.23:0/416344106 >> v2:172.21.15.23:3300/0 conn(0x7fd8c0003410 0x7fd8c0005870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:59.559 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.551+0000 7fd8e259c700 1 -- 172.21.15.23:0/416344106 >> v1:172.21.15.155:6789/0 conn(0x7fd8c0005e70 legacy=0x7fd8c0008330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:34:59.559 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.552+0000 7fd8e259c700 1 -- 172.21.15.23:0/416344106 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c000ac10 con 0x7fd8c0003410 2020-11-20T11:34:59.560 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.552+0000 7fd8e259c700 1 --2- 172.21.15.23:0/416344106 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8c0003410 0x7fd8c0005870 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fd8d4007960 tx=0x7fd8d4009040).ready entity=mon.0 client_cookie=907bab0e0c49fbcb server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:59.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.552+0000 7fd8e159a700 1 -- 172.21.15.23:0/416344106 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d4007110 con 0x7fd8c0003410 2020-11-20T11:34:59.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.552+0000 7fd8e159a700 1 -- 172.21.15.23:0/416344106 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8d4007270 con 0x7fd8c0003410 2020-11-20T11:34:59.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.552+0000 7fd8e159a700 1 -- 172.21.15.23:0/416344106 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d4011b60 con 0x7fd8c0003410 2020-11-20T11:34:59.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.553+0000 7fd8e359e700 1 -- 172.21.15.23:0/416344106 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8c0003410 msgr2=0x7fd8c0005870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:59.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.553+0000 7fd8e359e700 1 --2- 172.21.15.23:0/416344106 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8c0003410 0x7fd8c0005870 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fd8d4007960 tx=0x7fd8d4009040).stop 2020-11-20T11:34:59.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.553+0000 7fd8e2d9d700 1 -- 172.21.15.23:0/416344106 reap_dead start 2020-11-20T11:34:59.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.553+0000 7fd8e359e700 1 -- 172.21.15.23:0/416344106 shutdown_connections 2020-11-20T11:34:59.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.553+0000 7fd8e359e700 1 --2- 172.21.15.23:0/416344106 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8c0003410 0x7fd8c0005870 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:59.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.553+0000 7fd8e359e700 1 -- 172.21.15.23:0/416344106 >> 172.21.15.23:0/416344106 conn(0x7fd8e40683b0 msgr2=0x7fd8e40b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:34:59.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.553+0000 7fd8e359e700 1 -- 172.21.15.23:0/416344106 shutdown_connections 2020-11-20T11:34:59.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.553+0000 7fd8e359e700 1 -- 172.21.15.23:0/416344106 wait complete. 2020-11-20T11:34:59.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.554+0000 7fd8e359e700 1 Processor -- start 2020-11-20T11:34:59.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.554+0000 7fd8e359e700 1 -- start start 2020-11-20T11:34:59.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.555+0000 7fd8e359e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8e40a1850 0x7fd8e409fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:59.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.555+0000 7fd8e359e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8e409e600 0x7fd8e40a70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:59.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.555+0000 7fd8e359e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8e40a75c0 0x7fd8e414d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:59.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.555+0000 7fd8e359e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e40a7a40 con 0x7fd8e409e600 2020-11-20T11:34:59.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.555+0000 7fd8e359e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e414a130 con 0x7fd8e40a1850 2020-11-20T11:34:59.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.555+0000 7fd8e359e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e414a3d0 con 0x7fd8e40a75c0 2020-11-20T11:34:59.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.555+0000 7fd8e259c700 1 -- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8e40a1850 msgr2=0x7fd8e409fe00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:34:59.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.555+0000 7fd8e1d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8e409e600 0x7fd8e40a70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:59.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.555+0000 7fd8e259c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8e40a1850 0x7fd8e409fe00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:34:59.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.555+0000 7fd8e1d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8e409e600 0x7fd8e40a70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42320/0 (socket says 172.21.15.23:42320) 2020-11-20T11:34:59.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.555+0000 7fd8e1d9b700 1 -- 172.21.15.23:0/1674304013 learned_addr learned my addr 172.21.15.23:0/1674304013 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:34:59.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.555+0000 7fd8e2d9d700 1 --2- 172.21.15.23:0/1674304013 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8e40a75c0 0x7fd8e414d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:59.573 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.556+0000 7fd8e1d9b700 1 -- 172.21.15.23:0/1674304013 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8e40a75c0 msgr2=0x7fd8e414d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:59.573 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.556+0000 7fd8e1d9b700 1 --2- 172.21.15.23:0/1674304013 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8e40a75c0 0x7fd8e414d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:59.573 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.556+0000 7fd8e1d9b700 1 -- 172.21.15.23:0/1674304013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8e40a1850 msgr2=0x7fd8e409fe00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:34:59.573 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.556+0000 7fd8e1d9b700 1 --2- 172.21.15.23:0/1674304013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8e40a1850 0x7fd8e409fe00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:34:59.574 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.556+0000 7fd8e1d9b700 1 -- 172.21.15.23:0/1674304013 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e414a670 con 0x7fd8e409e600 2020-11-20T11:34:59.574 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.556+0000 7fd8e2d9d700 1 --2- 172.21.15.23:0/1674304013 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8e40a75c0 0x7fd8e414d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:34:59.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.556+0000 7fd8e1d9b700 1 --2- 172.21.15.23:0/1674304013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8e409e600 0x7fd8e40a70d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fd8d8009f50 tx=0x7fd8d80095a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:59.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.556+0000 7fd8d27fc700 1 -- 172.21.15.23:0/1674304013 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d8006170 con 0x7fd8e409e600 2020-11-20T11:34:59.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.556+0000 7fd8d27fc700 1 -- 172.21.15.23:0/1674304013 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8d800f070 con 0x7fd8e409e600 2020-11-20T11:34:59.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.556+0000 7fd8e359e700 1 -- 172.21.15.23:0/1674304013 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8e40a1c30 con 0x7fd8e409e600 2020-11-20T11:34:59.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.556+0000 7fd8e359e700 1 -- 172.21.15.23:0/1674304013 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8e414ab10 con 0x7fd8e409e600 2020-11-20T11:34:59.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.556+0000 7fd8d27fc700 1 -- 172.21.15.23:0/1674304013 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d8006540 con 0x7fd8e409e600 2020-11-20T11:34:59.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.557+0000 7fd8e359e700 1 -- 172.21.15.23:0/1674304013 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8e414a800 con 0x7fd8e409e600 2020-11-20T11:34:59.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.558+0000 7fd8d27fc700 1 -- 172.21.15.23:0/1674304013 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd8d801c070 con 0x7fd8e409e600 2020-11-20T11:34:59.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.558+0000 7fd8d27fc700 1 --2- 172.21.15.23:0/1674304013 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd8b8056d10 0x7fd8b80591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:59.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.558+0000 7fd8d27fc700 1 -- 172.21.15.23:0/1674304013 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fd8d8070760 con 0x7fd8e409e600 2020-11-20T11:34:59.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.558+0000 7fd8e259c700 1 --2- 172.21.15.23:0/1674304013 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd8b8056d10 0x7fd8b80591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:59.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.559+0000 7fd8e259c700 1 --2- 172.21.15.23:0/1674304013 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd8b8056d10 0x7fd8b80591d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fd8c00037f0 tx=0x7fd8d4009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:59.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.563+0000 7fd8d27fc700 1 -- 172.21.15.23:0/1674304013 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd8d8045a50 con 0x7fd8e409e600 2020-11-20T11:34:59.686 INFO:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:34:59.692 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:34:59.796 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.794+0000 7fd8e359e700 1 -- 172.21.15.23:0/1674304013 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd8e4057dc0 con 0x7fd8e409e600 2020-11-20T11:34:59.797 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.794+0000 7fd8d27fc700 1 -- 172.21.15.23:0/1674304013 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1471 (secure 0 0 0) 0x7fd8d80096e0 con 0x7fd8e409e600 2020-11-20T11:34:59.797 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":26,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:34:59.799 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.796+0000 7fd8e359e700 1 -- 172.21.15.23:0/1674304013 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd8b8056d10 msgr2=0x7fd8b80591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:59.799 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.796+0000 7fd8e359e700 1 --2- 172.21.15.23:0/1674304013 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd8b8056d10 0x7fd8b80591d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fd8c00037f0 tx=0x7fd8d4009040).stop 2020-11-20T11:34:59.799 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.796+0000 7fd8e359e700 1 -- 172.21.15.23:0/1674304013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8e409e600 msgr2=0x7fd8e40a70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:59.800 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.796+0000 7fd8e359e700 1 --2- 172.21.15.23:0/1674304013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8e409e600 0x7fd8e40a70d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fd8d8009f50 tx=0x7fd8d80095a0).stop 2020-11-20T11:34:59.800 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.796+0000 7fd8e359e700 1 -- 172.21.15.23:0/1674304013 shutdown_connections 2020-11-20T11:34:59.800 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.796+0000 7fd8e359e700 1 --2- 172.21.15.23:0/1674304013 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd8b8056d10 0x7fd8b80591d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:59.800 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.796+0000 7fd8e359e700 1 --2- 172.21.15.23:0/1674304013 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd8e40a75c0 0x7fd8e414d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:59.800 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.796+0000 7fd8e359e700 1 --2- 172.21.15.23:0/1674304013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd8e40a1850 0x7fd8e409fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:34:59.801 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.796+0000 7fd8e359e700 1 --2- 172.21.15.23:0/1674304013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd8e409e600 0x7fd8e40a70d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:59.801 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.796+0000 7fd8e359e700 1 -- 172.21.15.23:0/1674304013 >> 172.21.15.23:0/1674304013 conn(0x7fd8e40683b0 msgr2=0x7fd8e40ae2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:34:59.801 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.796+0000 7fd8e359e700 1 -- 172.21.15.23:0/1674304013 shutdown_connections 2020-11-20T11:34:59.801 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.796+0000 7fd8e359e700 1 -- 172.21.15.23:0/1674304013 wait complete. 2020-11-20T11:34:59.813 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2020-11-20T11:34:59.814 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:34:59.963 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.960+0000 7f75563b5700 1 Processor -- start 2020-11-20T11:34:59.963 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.960+0000 7f75563b5700 1 -- start start 2020-11-20T11:34:59.965 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.961+0000 7f75563b5700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75500a99f0 0x7f7550137e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:59.966 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.961+0000 7f75563b5700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f7550138380 0x7f75501387f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:59.966 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.961+0000 7f75563b5700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7550138ce0 con 0x7f7550061b90 2020-11-20T11:34:59.966 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.961+0000 7f75563b5700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7550067190 con 0x7f7550138380 2020-11-20T11:34:59.966 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.961+0000 7f75563b5700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7550067430 con 0x7f75500a99f0 2020-11-20T11:34:59.966 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.961+0000 7f7554952700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f7550138380 0x7f75501387f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:59.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.961+0000 7f754ffff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f7550061b90 0x7f75500a92e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:56576/0 (socket says 172.21.15.23:56576) 2020-11-20T11:34:59.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.961+0000 7f7554952700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f7550138380 0x7f75501387f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42328/0 (socket says 172.21.15.23:42328) 2020-11-20T11:34:59.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.961+0000 7f754ffff700 1 -- 172.21.15.23:0/1663010471 learned_addr learned my addr 172.21.15.23:0/1663010471 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:34:59.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.961+0000 7f7547fff700 1 --2- 172.21.15.23:0/1663010471 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75500a99f0 0x7f7550137e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:59.968 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.961+0000 7f7554952700 1 -- 172.21.15.23:0/1663010471 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75500a99f0 msgr2=0x7f7550137e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:34:59.968 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.961+0000 7f7554952700 1 --2- 172.21.15.23:0/1663010471 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75500a99f0 0x7f7550137e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:59.969 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.961+0000 7f7554952700 1 -- 172.21.15.23:0/1663010471 >> v1:172.21.15.23:6789/0 conn(0x7f7550061b90 legacy=0x7f75500a92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:59.969 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.961+0000 7f7554952700 1 -- 172.21.15.23:0/1663010471 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7550067860 con 0x7f7550138380 2020-11-20T11:34:59.969 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.961+0000 7f754f7fe700 1 -- 172.21.15.23:0/1663010471 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660550018 0 0) 0x7f7550138ce0 con 0x7f7550061b90 2020-11-20T11:34:59.969 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.962+0000 7f7554952700 1 --2- 172.21.15.23:0/1663010471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7550138380 0x7f75501387f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f7548009ce0 tx=0x7f7548009220).ready entity=mon.0 client_cookie=d8a32f15a0474d11 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:59.970 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.962+0000 7f754f7fe700 1 -- 172.21.15.23:0/1663010471 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754800ba40 con 0x7f7550138380 2020-11-20T11:34:59.970 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.962+0000 7f754f7fe700 1 -- 172.21.15.23:0/1663010471 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f754800bba0 con 0x7f7550138380 2020-11-20T11:34:59.970 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.962+0000 7f754f7fe700 1 -- 172.21.15.23:0/1663010471 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7548011400 con 0x7f7550138380 2020-11-20T11:34:59.970 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.962+0000 7f75563b5700 1 -- 172.21.15.23:0/1663010471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7550138380 msgr2=0x7f75501387f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:59.970 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.962+0000 7f75563b5700 1 --2- 172.21.15.23:0/1663010471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7550138380 0x7f75501387f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f7548009ce0 tx=0x7f7548009220).stop 2020-11-20T11:34:59.971 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.963+0000 7f75563b5700 1 -- 172.21.15.23:0/1663010471 shutdown_connections 2020-11-20T11:34:59.971 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.963+0000 7f75563b5700 1 --2- 172.21.15.23:0/1663010471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7550138380 0x7f75501387f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:59.971 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.963+0000 7f75563b5700 1 --2- 172.21.15.23:0/1663010471 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75500a99f0 0x7f7550137e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:59.971 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.963+0000 7f75563b5700 1 -- 172.21.15.23:0/1663010471 >> 172.21.15.23:0/1663010471 conn(0x7f7550063640 msgr2=0x7f75500612c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:34:59.971 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.963+0000 7f75563b5700 1 -- 172.21.15.23:0/1663010471 shutdown_connections 2020-11-20T11:34:59.972 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.963+0000 7f75563b5700 1 -- 172.21.15.23:0/1663010471 wait complete. 2020-11-20T11:34:59.975 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.964+0000 7f75563b5700 1 Processor -- start 2020-11-20T11:34:59.975 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.964+0000 7f75563b5700 1 -- start start 2020-11-20T11:34:59.975 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.965+0000 7f75563b5700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7550061b90 0x7f7550146b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:59.975 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.965+0000 7f75563b5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75500a99f0 0x7f755009f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:59.976 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.965+0000 7f75563b5700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7550138380 0x7f755009f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:59.976 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.965+0000 7f75563b5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75500a13f0 con 0x7f75500a99f0 2020-11-20T11:34:59.976 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.965+0000 7f75563b5700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755009dfb0 con 0x7f7550061b90 2020-11-20T11:34:59.976 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.965+0000 7f75563b5700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f755009e250 con 0x7f7550138380 2020-11-20T11:34:59.977 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.965+0000 7f7547fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75500a99f0 0x7f755009f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:59.977 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.965+0000 7f7547fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75500a99f0 0x7f755009f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42332/0 (socket says 172.21.15.23:42332) 2020-11-20T11:34:59.977 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.965+0000 7f754ffff700 1 -- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7550061b90 msgr2=0x7f7550146b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:34:59.977 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.965+0000 7f7547fff700 1 -- 172.21.15.23:0/4242659643 learned_addr learned my addr 172.21.15.23:0/4242659643 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:34:59.977 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.965+0000 7f754ffff700 1 --2- 172.21.15.23:0/4242659643 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7550061b90 0x7f7550146b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:34:59.978 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.965+0000 7f7554952700 1 --2- 172.21.15.23:0/4242659643 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7550138380 0x7f755009f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:59.978 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.966+0000 7f7547fff700 1 -- 172.21.15.23:0/4242659643 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7550138380 msgr2=0x7f755009f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:59.978 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.966+0000 7f7547fff700 1 --2- 172.21.15.23:0/4242659643 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7550138380 0x7f755009f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:59.978 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.966+0000 7f7547fff700 1 -- 172.21.15.23:0/4242659643 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7550061b90 msgr2=0x7f7550146b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:34:59.979 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.966+0000 7f7547fff700 1 --2- 172.21.15.23:0/4242659643 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7550061b90 0x7f7550146b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:34:59.979 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.966+0000 7f7547fff700 1 -- 172.21.15.23:0/4242659643 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f755009e4f0 con 0x7f75500a99f0 2020-11-20T11:34:59.979 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.966+0000 7f7554952700 1 --2- 172.21.15.23:0/4242659643 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7550138380 0x7f755009f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:34:59.980 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.966+0000 7f7547fff700 1 --2- 172.21.15.23:0/4242659643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75500a99f0 0x7f755009f250 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f753800ebd0 tx=0x7f7538006e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:59.980 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.966+0000 7f754cff9700 1 -- 172.21.15.23:0/4242659643 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753800c920 con 0x7f75500a99f0 2020-11-20T11:34:59.980 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.966+0000 7f754cff9700 1 -- 172.21.15.23:0/4242659643 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f753800ca80 con 0x7f75500a99f0 2020-11-20T11:34:59.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.966+0000 7f75563b5700 1 -- 172.21.15.23:0/4242659643 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75500a1830 con 0x7f75500a99f0 2020-11-20T11:34:59.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.967+0000 7f754cff9700 1 -- 172.21.15.23:0/4242659643 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753800ce00 con 0x7f75500a99f0 2020-11-20T11:34:59.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.967+0000 7f75563b5700 1 -- 172.21.15.23:0/4242659643 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7550115c70 con 0x7f75500a99f0 2020-11-20T11:34:59.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.967+0000 7f754cff9700 1 -- 172.21.15.23:0/4242659643 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7538012600 con 0x7f75500a99f0 2020-11-20T11:34:59.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.967+0000 7f75563b5700 1 --2- 172.21.15.23:0/4242659643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f755005c070 0x7f755014a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:59.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.967+0000 7f75563b5700 1 -- 172.21.15.23:0/4242659643 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75500a5000 con 0x7f755005c070 2020-11-20T11:34:59.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.967+0000 7f754cff9700 1 --2- 172.21.15.23:0/4242659643 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f752c056d10 0x7f752c0591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:59.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.968+0000 7f754ffff700 1 --2- 172.21.15.23:0/4242659643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f755005c070 0x7f755014a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:59.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.968+0000 7f754cff9700 1 -- 172.21.15.23:0/4242659643 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f7538072510 con 0x7f75500a99f0 2020-11-20T11:34:59.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.968+0000 7f7554952700 1 --2- 172.21.15.23:0/4242659643 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f752c056d10 0x7f752c0591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:59.983 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.968+0000 7f754ffff700 1 -- 172.21.15.23:0/4242659643 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f755009e680 con 0x7f755005c070 2020-11-20T11:34:59.983 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.968+0000 7f754ffff700 1 --2- 172.21.15.23:0/4242659643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f755005c070 0x7f755014a120 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f75400045d0 tx=0x7f7540007bc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:59.983 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.968+0000 7f754cff9700 1 -- 172.21.15.23:0/4242659643 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7540007320 con 0x7f755005c070 2020-11-20T11:34:59.983 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.968+0000 7f7554952700 1 --2- 172.21.15.23:0/4242659643 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f752c056d10 0x7f752c0591d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f7548009260 tx=0x7f7548016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:59.984 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.968+0000 7f754cff9700 1 -- 172.21.15.23:0/4242659643 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f7540007480 con 0x7f755005c070 2020-11-20T11:34:59.984 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.968+0000 7f754cff9700 1 -- 172.21.15.23:0/4242659643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f755005c070 msgr2=0x7f755014a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:59.984 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.968+0000 7f754cff9700 1 --2- 172.21.15.23:0/4242659643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f755005c070 0x7f755014a120 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f75400045d0 tx=0x7f7540007bc0).stop 2020-11-20T11:34:59.984 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.968+0000 7f754cff9700 1 -- 172.21.15.23:0/4242659643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f755005c070 msgr2=0x7f755014a120 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:34:59.984 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.968+0000 7f754cff9700 1 --2- 172.21.15.23:0/4242659643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f755005c070 0x7f755014a120 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:59.986 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.976+0000 7f75563b5700 1 --2- 172.21.15.23:0/4242659643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7550147040 0x7f755014a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:34:59.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.976+0000 7f75563b5700 1 -- 172.21.15.23:0/4242659643 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7550044710 con 0x7f7550147040 2020-11-20T11:34:59.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.976+0000 7f754ffff700 1 --2- 172.21.15.23:0/4242659643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7550147040 0x7f755014a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:34:59.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.976+0000 7f754ffff700 1 -- 172.21.15.23:0/4242659643 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7550137b60 con 0x7f7550147040 2020-11-20T11:34:59.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.976+0000 7f754ffff700 1 --2- 172.21.15.23:0/4242659643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7550147040 0x7f755014a610 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f7540013000 tx=0x7f7540016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:34:59.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.976+0000 7f754cff9700 1 -- 172.21.15.23:0/4242659643 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7540013c40 con 0x7f7550147040 2020-11-20T11:34:59.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.977+0000 7f754cff9700 1 -- 172.21.15.23:0/4242659643 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4604 (secure 0 0 0) 0x7f754000ced0 con 0x7f7550147040 2020-11-20T11:34:59.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.977+0000 7f754cff9700 1 -- 172.21.15.23:0/4242659643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7550147040 msgr2=0x7f755014a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:59.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.977+0000 7f754cff9700 1 --2- 172.21.15.23:0/4242659643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7550147040 0x7f755014a610 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f7540013000 tx=0x7f7540016000).stop 2020-11-20T11:34:59.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.977+0000 7f754cff9700 1 -- 172.21.15.23:0/4242659643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7550147040 msgr2=0x7f755014a610 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:34:59.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.977+0000 7f754cff9700 1 --2- 172.21.15.23:0/4242659643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7550147040 0x7f755014a610 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:59.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.977+0000 7f75563b5700 1 -- 172.21.15.23:0/4242659643 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f752c056d10 msgr2=0x7f752c0591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:59.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.977+0000 7f75563b5700 1 --2- 172.21.15.23:0/4242659643 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f752c056d10 0x7f752c0591d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f7548009260 tx=0x7f7548016000).stop 2020-11-20T11:34:59.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.977+0000 7f75563b5700 1 -- 172.21.15.23:0/4242659643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75500a99f0 msgr2=0x7f755009f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:34:59.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.977+0000 7f75563b5700 1 --2- 172.21.15.23:0/4242659643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75500a99f0 0x7f755009f250 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f753800ebd0 tx=0x7f7538006e30).stop 2020-11-20T11:34:59.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.977+0000 7f7554952700 1 -- 172.21.15.23:0/4242659643 reap_dead start 2020-11-20T11:34:59.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.977+0000 7f75563b5700 1 -- 172.21.15.23:0/4242659643 shutdown_connections 2020-11-20T11:34:59.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.977+0000 7f75563b5700 1 --2- 172.21.15.23:0/4242659643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75500a99f0 0x7f755009f250 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:34:59.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.977+0000 7f75563b5700 1 -- 172.21.15.23:0/4242659643 >> 172.21.15.23:0/4242659643 conn(0x7f7550063640 msgr2=0x7f755013b1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:34:59.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.977+0000 7f75563b5700 1 -- 172.21.15.23:0/4242659643 shutdown_connections 2020-11-20T11:34:59.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:34:59.977+0000 7f75563b5700 1 -- 172.21.15.23:0/4242659643 wait complete. 2020-11-20T11:34:59.991 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:34:59.992 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:34:59.992 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:34:59.992 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:34:59.992 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 26, 2020-11-20T11:34:59.992 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:34:59.993 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:34:59.993 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:34:59.993 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:59.993 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 1, 2020-11-20T11:34:59.994 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:34:59.994 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:34:59.994 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:34:59.994 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:34:59.994 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:34:59.995 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:34:59.995 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:34:59.995 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:34:59.995 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:34:59.995 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:34:59.996 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:34:59.996 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:34:59.996 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:34:59.996 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:34:59.996 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:34:59.997 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:34:59.997 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:34:59.997 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:34:59.997 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:34:59.997 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:34:59.998 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:34:59.998 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:34:59.998 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:34:59.998 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:34:59.998 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:34:59.999 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:34:59.999 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:34:59.999 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:34:59.999 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:35:00.000 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:35:00.000 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:35:00.000 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:35:00.001 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:35:00.001 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:35:00.001 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:35:00.001 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:35:00.002 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:35:00.002 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:35:00.002 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:35:00.002 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:35:00.002 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:35:00.003 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:35:00.003 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:35:00.003 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:35:00.003 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:35:00.003 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:35:00.004 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:35:00.004 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:00.004 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:35:00.004 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:00.004 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:35:00.005 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:00.005 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:35:00.005 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:35:00.005 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:35:00.005 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:35:00.006 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:00.006 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:35:00.006 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:35:00.006 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:00.006 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:00.007 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:00.007 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:35:00.007 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:35:00.007 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:00.007 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:00.008 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:00.008 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:00.008 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:35:00.008 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:35:00.008 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:35:00.009 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:35:00.009 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:35:00.009 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:00.009 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:00.010 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:35:00.010 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:35:00.010 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:35:00.011 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:35:00.011 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:00.011 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:35:00.011 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:35:00.011 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:00.012 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:00.012 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:00.012 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:35:00.012 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:35:00.012 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:00.013 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:00.013 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:00.013 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:00.013 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:35:00.014 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:35:00.014 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:35:00.014 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:35:00.014 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:35:00.014 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:00.015 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:00.015 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:35:00.015 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:35:00.015 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:35:00.015 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:35:00.016 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:00.016 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:35:00.016 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:35:00.016 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:00.016 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:00.017 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:00.017 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:35:00.017 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:35:00.017 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:00.017 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:00.018 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:00.018 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:00.018 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:35:00.018 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:35:00.018 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:35:00.019 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:35:00.019 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:35:00.019 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:00.019 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:00.019 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:00.020 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:35:00.020 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:35:00.020 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:00.020 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:00.021 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:00.021 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:35:00.021 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:00.022 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:00.022 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:35:00.022 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:00.022 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:00.022 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:00.023 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:35:00.023 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:00.023 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:00.023 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:35:00.024 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:00.024 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:00.024 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:00.024 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 4 2020-11-20T11:35:00.024 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:00.025 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:00.025 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:35:00.025 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:00.025 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:00.025 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:00.026 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:35:00.026 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:00.026 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:00.026 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:00.026 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:35:00.027 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:35:00.028 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:35:00.181 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.165+0000 7f6f7b5b8700 1 Processor -- start 2020-11-20T11:35:00.183 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.165+0000 7f6f7b5b8700 1 -- start start 2020-11-20T11:35:00.183 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.165+0000 7f6f7b5b8700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f740aaab0 0x7f6f7413ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:00.184 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.166+0000 7f6f7b5b8700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6f740aae90 0x7f6f741381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:00.185 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.166+0000 7f6f7b5b8700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f7413e600 con 0x7f6f7413aeb0 2020-11-20T11:35:00.185 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.166+0000 7f6f7b5b8700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f7413e8a0 con 0x7f6f740aae90 2020-11-20T11:35:00.185 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.166+0000 7f6f7b5b8700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f740ab6d0 con 0x7f6f740aaab0 2020-11-20T11:35:00.186 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.166+0000 7f6f78b53700 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f6f740aae90 msgr2=0x7f6f741381a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:35:00.186 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.166+0000 7f6f79b55700 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f6f7413aeb0 legacy=0x7f6f740a8c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:35:00.186 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.166+0000 7f6f78b53700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6f740aae90 0x7f6f741381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:35:00.187 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.166+0000 7f6f79354700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f740aaab0 0x7f6f7413ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:00.187 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.166+0000 7f6f79354700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f740aaab0 0x7f6f7413ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:52582/0 (socket says 172.21.15.23:52582) 2020-11-20T11:35:00.188 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.166+0000 7f6f79354700 1 -- 172.21.15.23:0/3969926199 learned_addr learned my addr 172.21.15.23:0/3969926199 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:35:00.188 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.166+0000 7f6f79354700 1 -- 172.21.15.23:0/3969926199 >> v2:172.21.15.155:3300/0 conn(0x7f6f740aae90 msgr2=0x7f6f741381a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:35:00.188 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.166+0000 7f6f79354700 1 --2- 172.21.15.23:0/3969926199 >> v2:172.21.15.155:3300/0 conn(0x7f6f740aae90 0x7f6f741381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:35:00.189 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.166+0000 7f6f79354700 1 -- 172.21.15.23:0/3969926199 >> v1:172.21.15.155:6789/0 conn(0x7f6f7413aeb0 legacy=0x7f6f740a8c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:35:00.189 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.166+0000 7f6f79354700 1 -- 172.21.15.23:0/3969926199 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f740a9370 con 0x7f6f740aaab0 2020-11-20T11:35:00.189 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.167+0000 7f6f79354700 1 --2- 172.21.15.23:0/3969926199 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f740aaab0 0x7f6f7413ed40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f6f70009a10 tx=0x7f6f70006ef0).ready entity=mon.2 client_cookie=77a89f78605a8551 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:00.191 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.167+0000 7f6f63fff700 1 -- 172.21.15.23:0/3969926199 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7000b820 con 0x7f6f740aaab0 2020-11-20T11:35:00.191 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.167+0000 7f6f63fff700 1 -- 172.21.15.23:0/3969926199 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f70000e90 con 0x7f6f740aaab0 2020-11-20T11:35:00.191 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.167+0000 7f6f63fff700 1 -- 172.21.15.23:0/3969926199 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7000bb90 con 0x7f6f740aaab0 2020-11-20T11:35:00.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.167+0000 7f6f7b5b8700 1 -- 172.21.15.23:0/3969926199 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f740aaab0 msgr2=0x7f6f7413ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:00.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.167+0000 7f6f7b5b8700 1 --2- 172.21.15.23:0/3969926199 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f740aaab0 0x7f6f7413ed40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f6f70009a10 tx=0x7f6f70006ef0).stop 2020-11-20T11:35:00.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.167+0000 7f6f7b5b8700 1 -- 172.21.15.23:0/3969926199 shutdown_connections 2020-11-20T11:35:00.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.167+0000 7f6f7b5b8700 1 --2- 172.21.15.23:0/3969926199 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f740aaab0 0x7f6f7413ed40 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:00.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.167+0000 7f6f7b5b8700 1 --2- 172.21.15.23:0/3969926199 >> v2:172.21.15.155:3300/0 conn(0x7f6f740aae90 0x7f6f741381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:35:00.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.167+0000 7f6f7b5b8700 1 -- 172.21.15.23:0/3969926199 >> 172.21.15.23:0/3969926199 conn(0x7f6f740678e0 msgr2=0x7f6f74067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:00.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.167+0000 7f6f7b5b8700 1 -- 172.21.15.23:0/3969926199 shutdown_connections 2020-11-20T11:35:00.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.167+0000 7f6f7b5b8700 1 -- 172.21.15.23:0/3969926199 wait complete. 2020-11-20T11:35:00.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.168+0000 7f6f7b5b8700 1 Processor -- start 2020-11-20T11:35:00.197 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.168+0000 7f6f7b5b8700 1 -- start start 2020-11-20T11:35:00.197 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.169+0000 7f6f7b5b8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f740aaab0 0x7f6f740b0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:00.197 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.169+0000 7f6f7b5b8700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f740aae90 0x7f6f74137b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:00.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.169+0000 7f6f7b5b8700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f7414dac0 0x7f6f7414df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:00.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.169+0000 7f6f7b5b8700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7414e430 con 0x7f6f740aaab0 2020-11-20T11:35:00.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.169+0000 7f6f7b5b8700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7413da40 con 0x7f6f7414dac0 2020-11-20T11:35:00.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.169+0000 7f6f7b5b8700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7413dce0 con 0x7f6f740aae90 2020-11-20T11:35:00.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.170+0000 7f6f79354700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f740aaab0 0x7f6f740b0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:00.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.170+0000 7f6f79b55700 1 -- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f7414dac0 msgr2=0x7f6f7414df40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:35:00.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.170+0000 7f6f79354700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f740aaab0 0x7f6f740b0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42348/0 (socket says 172.21.15.23:42348) 2020-11-20T11:35:00.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.170+0000 7f6f79354700 1 -- 172.21.15.23:0/1280358617 learned_addr learned my addr 172.21.15.23:0/1280358617 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:35:00.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.170+0000 7f6f79b55700 1 --2- 172.21.15.23:0/1280358617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f7414dac0 0x7f6f7414df40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:35:00.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.170+0000 7f6f78b53700 1 --2- 172.21.15.23:0/1280358617 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f740aae90 0x7f6f74137b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:00.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.170+0000 7f6f79354700 1 -- 172.21.15.23:0/1280358617 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f740aae90 msgr2=0x7f6f74137b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:00.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.170+0000 7f6f79354700 1 --2- 172.21.15.23:0/1280358617 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f740aae90 0x7f6f74137b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:00.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.170+0000 7f6f79354700 1 -- 172.21.15.23:0/1280358617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f7414dac0 msgr2=0x7f6f7414df40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:35:00.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.170+0000 7f6f79354700 1 --2- 172.21.15.23:0/1280358617 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f7414dac0 0x7f6f7414df40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:35:00.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.170+0000 7f6f79354700 1 -- 172.21.15.23:0/1280358617 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f7413df80 con 0x7f6f740aaab0 2020-11-20T11:35:00.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.170+0000 7f6f79354700 1 --2- 172.21.15.23:0/1280358617 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f740aaab0 0x7f6f740b0d80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f6f70006f30 tx=0x7f6f70016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:00.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.170+0000 7f6f617fa700 1 -- 172.21.15.23:0/1280358617 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f70011d70 con 0x7f6f740aaab0 2020-11-20T11:35:00.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.170+0000 7f6f617fa700 1 -- 172.21.15.23:0/1280358617 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f7001c440 con 0x7f6f740aaab0 2020-11-20T11:35:00.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.170+0000 7f6f7b5b8700 1 -- 172.21.15.23:0/1280358617 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f741382b0 con 0x7f6f740aaab0 2020-11-20T11:35:00.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.170+0000 7f6f617fa700 1 -- 172.21.15.23:0/1280358617 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7001c7c0 con 0x7f6f740aaab0 2020-11-20T11:35:00.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.170+0000 7f6f7b5b8700 1 -- 172.21.15.23:0/1280358617 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f7413d810 con 0x7f6f740aaab0 2020-11-20T11:35:00.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.171+0000 7f6f52ffd700 1 --2- 172.21.15.23:0/1280358617 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f44002a50 0x7f6f44004f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:00.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.171+0000 7f6f52ffd700 1 -- 172.21.15.23:0/1280358617 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f44007bb0 con 0x7f6f44002a50 2020-11-20T11:35:00.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.171+0000 7f6f78b53700 1 --2- 172.21.15.23:0/1280358617 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f44002a50 0x7f6f44004f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:00.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.172+0000 7f6f617fa700 1 -- 172.21.15.23:0/1280358617 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6f7002c070 con 0x7f6f740aaab0 2020-11-20T11:35:00.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.172+0000 7f6f78b53700 1 -- 172.21.15.23:0/1280358617 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6f44007d70 con 0x7f6f44002a50 2020-11-20T11:35:00.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.172+0000 7f6f78b53700 1 --2- 172.21.15.23:0/1280358617 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f44002a50 0x7f6f44004f10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f6f740a5050 tx=0x7f6f64008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:00.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.172+0000 7f6f617fa700 1 --2- 172.21.15.23:0/1280358617 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6f4c056d10 0x7f6f4c0591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:00.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.172+0000 7f6f617fa700 1 -- 172.21.15.23:0/1280358617 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f6f70071db0 con 0x7f6f740aaab0 2020-11-20T11:35:00.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.172+0000 7f6f617fa700 1 -- 172.21.15.23:0/1280358617 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f640052e0 con 0x7f6f44002a50 2020-11-20T11:35:00.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.172+0000 7f6f79b55700 1 --2- 172.21.15.23:0/1280358617 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6f4c056d10 0x7f6f4c0591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:00.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.172+0000 7f6f617fa700 1 -- 172.21.15.23:0/1280358617 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f6f64005440 con 0x7f6f44002a50 2020-11-20T11:35:00.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.172+0000 7f6f617fa700 1 -- 172.21.15.23:0/1280358617 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f44002a50 msgr2=0x7f6f44004f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:00.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.172+0000 7f6f617fa700 1 --2- 172.21.15.23:0/1280358617 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f44002a50 0x7f6f44004f10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f6f740a5050 tx=0x7f6f64008040).stop 2020-11-20T11:35:00.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.172+0000 7f6f617fa700 1 -- 172.21.15.23:0/1280358617 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f44002a50 msgr2=0x7f6f44004f10 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:35:00.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.172+0000 7f6f617fa700 1 --2- 172.21.15.23:0/1280358617 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f44002a50 0x7f6f44004f10 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0x7f6f740a5050 tx=0x7f6f64008040).stop 2020-11-20T11:35:00.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.173+0000 7f6f79b55700 1 --2- 172.21.15.23:0/1280358617 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6f4c056d10 0x7f6f4c0591d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f6f6c003990 tx=0x7f6f6c006d80).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:00.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.180+0000 7f6f52ffd700 1 --2- 172.21.15.23:0/1280358617 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f44006c80 0x7f6f440070f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:00.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.180+0000 7f6f52ffd700 1 -- 172.21.15.23:0/1280358617 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f44007700 con 0x7f6f44006c80 2020-11-20T11:35:00.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.180+0000 7f6f78b53700 1 --2- 172.21.15.23:0/1280358617 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f44006c80 0x7f6f440070f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:00.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.181+0000 7f6f78b53700 1 -- 172.21.15.23:0/1280358617 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6f440078c0 con 0x7f6f44006c80 2020-11-20T11:35:00.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.181+0000 7f6f78b53700 1 --2- 172.21.15.23:0/1280358617 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f44006c80 0x7f6f440070f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f6f44007d40 tx=0x7f6f64008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:00.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.181+0000 7f6f617fa700 1 -- 172.21.15.23:0/1280358617 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f64013c70 con 0x7f6f44006c80 2020-11-20T11:35:00.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.181+0000 7f6f617fa700 1 -- 172.21.15.23:0/1280358617 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4772 (secure 0 0 0) 0x7f6f640050d0 con 0x7f6f44006c80 2020-11-20T11:35:00.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.181+0000 7f6f617fa700 1 -- 172.21.15.23:0/1280358617 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f44006c80 msgr2=0x7f6f440070f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:00.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.181+0000 7f6f617fa700 1 --2- 172.21.15.23:0/1280358617 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f44006c80 0x7f6f440070f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f6f44007d40 tx=0x7f6f64008040).stop 2020-11-20T11:35:00.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.181+0000 7f6f617fa700 1 -- 172.21.15.23:0/1280358617 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f44006c80 msgr2=0x7f6f440070f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:35:00.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.181+0000 7f6f617fa700 1 --2- 172.21.15.23:0/1280358617 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f44006c80 0x7f6f440070f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:00.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.182+0000 7f6f52ffd700 1 -- 172.21.15.23:0/1280358617 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6f4c056d10 msgr2=0x7f6f4c0591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:00.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.182+0000 7f6f52ffd700 1 --2- 172.21.15.23:0/1280358617 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6f4c056d10 0x7f6f4c0591d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f6f6c003990 tx=0x7f6f6c006d80).stop 2020-11-20T11:35:00.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.182+0000 7f6f52ffd700 1 -- 172.21.15.23:0/1280358617 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f740aaab0 msgr2=0x7f6f740b0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:00.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.182+0000 7f6f52ffd700 1 --2- 172.21.15.23:0/1280358617 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f740aaab0 0x7f6f740b0d80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f6f70006f30 tx=0x7f6f70016000).stop 2020-11-20T11:35:00.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.182+0000 7f6f79b55700 1 -- 172.21.15.23:0/1280358617 reap_dead start 2020-11-20T11:35:00.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.182+0000 7f6f52ffd700 1 -- 172.21.15.23:0/1280358617 shutdown_connections 2020-11-20T11:35:00.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.182+0000 7f6f52ffd700 1 --2- 172.21.15.23:0/1280358617 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f740aaab0 0x7f6f740b0d80 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:00.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.182+0000 7f6f52ffd700 1 -- 172.21.15.23:0/1280358617 >> 172.21.15.23:0/1280358617 conn(0x7f6f740678e0 msgr2=0x7f6f74140550 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:00.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.182+0000 7f6f52ffd700 1 -- 172.21.15.23:0/1280358617 shutdown_connections 2020-11-20T11:35:00.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:00.182+0000 7f6f52ffd700 1 -- 172.21.15.23:0/1280358617 wait complete. 2020-11-20T11:35:00.222 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:35:00.223 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:35:00.223 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:35:00.223 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:35:00.224 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 26, 2020-11-20T11:35:00.224 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:35:00.224 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:35:00.224 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:35:00.225 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:00.225 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 1, 2020-11-20T11:35:00.225 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:35:00.226 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:35:00.226 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:35:00.226 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:35:00.227 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:35:00.227 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:35:00.227 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:35:00.228 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:35:00.228 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:35:00.228 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:35:00.229 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:35:00.229 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:00.229 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:35:00.230 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:35:00.230 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:35:00.230 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:35:00.230 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:35:00.231 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:35:00.231 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:35:00.231 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:35:00.232 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:35:00.232 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:35:00.232 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:00.233 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:00.233 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:35:00.233 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:35:00.234 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:35:00.234 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:35:00.234 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:35:00.235 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:35:00.235 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:35:00.235 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:35:00.236 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:35:00.236 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:35:00.236 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:35:00.237 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:35:00.237 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:35:00.237 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:35:00.237 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:35:00.238 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:35:00.238 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:35:00.238 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:35:00.239 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:35:00.239 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:35:00.239 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:35:00.239 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:35:00.240 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:35:00.240 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:00.240 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:35:00.241 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:00.241 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:35:00.241 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:00.242 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:35:00.242 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:35:00.242 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:35:00.243 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:35:00.243 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:00.243 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:35:00.243 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:35:00.244 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:00.244 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:00.244 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:00.244 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:35:00.245 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:35:00.245 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:00.246 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:00.246 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:00.246 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:00.247 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:35:00.247 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:35:00.247 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:35:00.247 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:35:00.248 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:35:00.248 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:00.248 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:00.249 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:35:00.249 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:35:00.249 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:35:00.250 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:35:00.250 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:00.250 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:35:00.250 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:35:00.251 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:00.251 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:00.251 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:00.252 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:35:00.252 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:35:00.252 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:00.253 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:00.253 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:00.253 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:00.254 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:35:00.254 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:35:00.254 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:35:00.254 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:35:00.255 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:35:00.255 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:00.255 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:00.256 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:35:00.256 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:35:00.256 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:35:00.256 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:35:00.257 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:00.257 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:35:00.257 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:35:00.257 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:00.258 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:00.258 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:00.258 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:35:00.258 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:35:00.259 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:00.259 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:00.259 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:00.259 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:00.260 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:35:00.260 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:35:00.260 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:35:00.261 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:35:00.261 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:35:00.261 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:00.261 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:00.262 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:00.262 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:35:00.262 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:35:00.263 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:00.263 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:00.263 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:00.263 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:35:00.264 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:00.264 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:00.264 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:35:00.265 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:00.265 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:00.265 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:00.266 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:35:00.266 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:00.266 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:00.266 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:35:00.267 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:00.267 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:00.267 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:00.268 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 6 2020-11-20T11:35:00.268 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:00.268 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:00.268 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:35:00.269 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:00.269 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:00.269 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:00.270 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:35:00.270 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:00.270 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:00.271 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:35:00.271 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:00.271 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:00.272 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:00.272 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:35:00.272 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:00.272 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:00.273 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:00.273 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:35:00.273 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:35:00.274 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2020-11-20T11:35:20.275 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2020-11-20T11:35:20.276 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2020-11-20T11:35:20.276 INFO:tasks.ceph.mon.b:Restarting daemon 2020-11-20T11:35:20.276 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2020-11-20T11:35:20.282 INFO:tasks.ceph.mon.b:Started 2020-11-20T11:35:20.283 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:35:20.284 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:35:20.413 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.409+0000 7f11f8f98700 1 Processor -- start 2020-11-20T11:35:20.415 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.410+0000 7f11f8f98700 1 -- start start 2020-11-20T11:35:20.427 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.411+0000 7f11f8f98700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11f40aaab0 0x7f11f413ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:20.428 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.411+0000 7f11f8f98700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f11f40aae90 0x7f11f41381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:20.428 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.411+0000 7f11f8f98700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f413e600 con 0x7f11f413aeb0 2020-11-20T11:35:20.429 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.411+0000 7f11f8f98700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11f413e8a0 con 0x7f11f40aae90 2020-11-20T11:35:20.429 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.411+0000 7f11f8f98700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11f40ab6d0 con 0x7f11f40aaab0 2020-11-20T11:35:20.429 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.411+0000 7f11f1d9b700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f11f40aae90 0x7f11f41381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:20.430 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.411+0000 7f11f259c700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11f40aaab0 0x7f11f413ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:20.430 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.411+0000 7f11f2d9d700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f11f413aeb0 0x7f11f40a8c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:34590/0 (socket says 172.21.15.23:34590) 2020-11-20T11:35:20.430 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.411+0000 7f11f2d9d700 1 -- 172.21.15.23:0/479993366 learned_addr learned my addr 172.21.15.23:0/479993366 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:35:20.431 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.411+0000 7f11f1d9b700 1 -- 172.21.15.23:0/479993366 >> v2:172.21.15.155:3300/0 conn(0x7f11f40aae90 msgr2=0x7f11f41381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:35:20.431 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.411+0000 7f11f1d9b700 1 -- 172.21.15.23:0/479993366 >> v2:172.21.15.155:3300/0 conn(0x7f11f40aae90 msgr2=0x7f11f41381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:35:20.431 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.411+0000 7f11f1d9b700 1 --2- 172.21.15.23:0/479993366 >> v2:172.21.15.155:3300/0 conn(0x7f11f40aae90 0x7f11f41381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:35:20.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.411+0000 7f11f1d9b700 1 --2- 172.21.15.23:0/479993366 >> v2:172.21.15.155:3300/0 conn(0x7f11f40aae90 0x7f11f41381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:35:20.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.411+0000 7f11f259c700 1 -- 172.21.15.23:0/479993366 >> v2:172.21.15.155:3300/0 conn(0x7f11f40aae90 msgr2=0x7f11f41381a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:35:20.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.411+0000 7f11f259c700 1 --2- 172.21.15.23:0/479993366 >> v2:172.21.15.155:3300/0 conn(0x7f11f40aae90 0x7f11f41381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:20.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.411+0000 7f11f259c700 1 -- 172.21.15.23:0/479993366 >> v1:172.21.15.155:6789/0 conn(0x7f11f413aeb0 legacy=0x7f11f40a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:20.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.411+0000 7f11f259c700 1 -- 172.21.15.23:0/479993366 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f40a9370 con 0x7f11f40aaab0 2020-11-20T11:35:20.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.411+0000 7f11f259c700 1 --2- 172.21.15.23:0/479993366 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11f40aaab0 0x7f11f413ed40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f11e4006fd0 tx=0x7f11e4006e30).ready entity=mon.2 client_cookie=12510ca6c4e771b7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:20.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.412+0000 7f11f159a700 1 -- 172.21.15.23:0/479993366 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e40097e0 con 0x7f11f40aaab0 2020-11-20T11:35:20.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.412+0000 7f11f159a700 1 -- 172.21.15.23:0/479993366 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11e4000e90 con 0x7f11f40aaab0 2020-11-20T11:35:20.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.412+0000 7f11f159a700 1 -- 172.21.15.23:0/479993366 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e4009b50 con 0x7f11f40aaab0 2020-11-20T11:35:20.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.412+0000 7f11f8f98700 1 -- 172.21.15.23:0/479993366 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11f40aaab0 msgr2=0x7f11f413ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:20.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.412+0000 7f11f8f98700 1 --2- 172.21.15.23:0/479993366 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11f40aaab0 0x7f11f413ed40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f11e4006fd0 tx=0x7f11e4006e30).stop 2020-11-20T11:35:20.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.412+0000 7f11f8f98700 1 -- 172.21.15.23:0/479993366 shutdown_connections 2020-11-20T11:35:20.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.412+0000 7f11f8f98700 1 --2- 172.21.15.23:0/479993366 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11f40aaab0 0x7f11f413ed40 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:20.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.412+0000 7f11f8f98700 1 --2- 172.21.15.23:0/479993366 >> v2:172.21.15.155:3300/0 conn(0x7f11f40aae90 0x7f11f41381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:20.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.412+0000 7f11f8f98700 1 -- 172.21.15.23:0/479993366 >> 172.21.15.23:0/479993366 conn(0x7f11f40678e0 msgr2=0x7f11f4067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:20.437 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.412+0000 7f11f8f98700 1 -- 172.21.15.23:0/479993366 shutdown_connections 2020-11-20T11:35:20.437 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.412+0000 7f11f8f98700 1 -- 172.21.15.23:0/479993366 wait complete. 2020-11-20T11:35:20.437 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.413+0000 7f11f8f98700 1 Processor -- start 2020-11-20T11:35:20.438 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.413+0000 7f11f8f98700 1 -- start start 2020-11-20T11:35:20.438 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.414+0000 7f11f8f98700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11f40aaab0 0x7f11f4140cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:20.438 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.414+0000 7f11f8f98700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11f40aae90 0x7f11f4141a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:20.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.414+0000 7f11f8f98700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11f40a7fe0 0x7f11f40af840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:20.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.414+0000 7f11f8f98700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f4141300 con 0x7f11f40aae90 2020-11-20T11:35:20.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.414+0000 7f11f8f98700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f40af3a0 con 0x7f11f40a7fe0 2020-11-20T11:35:20.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.414+0000 7f11f8f98700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11f414a370 con 0x7f11f40aaab0 2020-11-20T11:35:20.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.414+0000 7f11f1d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11f40aae90 0x7f11f4141a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:20.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.414+0000 7f11f1d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11f40aae90 0x7f11f4141a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42370/0 (socket says 172.21.15.23:42370) 2020-11-20T11:35:20.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.414+0000 7f11f1d9b700 1 -- 172.21.15.23:0/980751202 learned_addr learned my addr 172.21.15.23:0/980751202 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:35:20.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.414+0000 7f11f259c700 1 --2- 172.21.15.23:0/980751202 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11f40aaab0 0x7f11f4140cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:20.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.414+0000 7f11f2d9d700 1 --2- 172.21.15.23:0/980751202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11f40a7fe0 0x7f11f40af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:20.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.414+0000 7f11f2d9d700 1 -- 172.21.15.23:0/980751202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11f40a7fe0 msgr2=0x7f11f40af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2020-11-20T11:35:20.442 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.414+0000 7f11f2d9d700 1 -- 172.21.15.23:0/980751202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11f40a7fe0 msgr2=0x7f11f40af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:35:20.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.414+0000 7f11f2d9d700 1 --2- 172.21.15.23:0/980751202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11f40a7fe0 0x7f11f40af840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:35:20.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.414+0000 7f11f2d9d700 1 --2- 172.21.15.23:0/980751202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11f40a7fe0 0x7f11f40af840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:35:20.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.414+0000 7f11f1d9b700 1 -- 172.21.15.23:0/980751202 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11f40aaab0 msgr2=0x7f11f4140cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:20.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.414+0000 7f11f1d9b700 1 --2- 172.21.15.23:0/980751202 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11f40aaab0 0x7f11f4140cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:20.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.414+0000 7f11f1d9b700 1 -- 172.21.15.23:0/980751202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11f40a7fe0 msgr2=0x7f11f40af840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:35:20.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.414+0000 7f11f1d9b700 1 --2- 172.21.15.23:0/980751202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11f40a7fe0 0x7f11f40af840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:20.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.414+0000 7f11f1d9b700 1 -- 172.21.15.23:0/980751202 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f40afd30 con 0x7f11f40aae90 2020-11-20T11:35:20.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.415+0000 7f11f1d9b700 1 --2- 172.21.15.23:0/980751202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11f40aae90 0x7f11f4141a90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f11e800db30 tx=0x7f11e8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:20.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.415+0000 7f11e27fc700 1 -- 172.21.15.23:0/980751202 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e800be10 con 0x7f11f40aae90 2020-11-20T11:35:20.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.415+0000 7f11e27fc700 1 -- 172.21.15.23:0/980751202 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11e80097c0 con 0x7f11f40aae90 2020-11-20T11:35:20.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.415+0000 7f11f8f98700 1 -- 172.21.15.23:0/980751202 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11f414a8b0 con 0x7f11f40aae90 2020-11-20T11:35:20.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.415+0000 7f11e27fc700 1 -- 172.21.15.23:0/980751202 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e8009b80 con 0x7f11f40aae90 2020-11-20T11:35:20.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.415+0000 7f11f8f98700 1 -- 172.21.15.23:0/980751202 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11f414adf0 con 0x7f11f40aae90 2020-11-20T11:35:20.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.416+0000 7f11f8f98700 1 -- 172.21.15.23:0/980751202 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11f405c7d0 con 0x7f11f40aae90 2020-11-20T11:35:20.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.416+0000 7f11e27fc700 1 -- 172.21.15.23:0/980751202 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f11e8013480 con 0x7f11f40aae90 2020-11-20T11:35:20.448 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.416+0000 7f11e27fc700 1 --2- 172.21.15.23:0/980751202 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f11c8056d60 0x7f11c8059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:20.448 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.416+0000 7f11e27fc700 1 -- 172.21.15.23:0/980751202 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f11e806ffc0 con 0x7f11f40aae90 2020-11-20T11:35:20.448 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.417+0000 7f11f259c700 1 --2- 172.21.15.23:0/980751202 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f11c8056d60 0x7f11c8059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:20.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.417+0000 7f11f259c700 1 --2- 172.21.15.23:0/980751202 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f11c8056d60 0x7f11c8059220 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f11e4006e70 tx=0x7f11e4015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:20.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.420+0000 7f11e27fc700 1 -- 172.21.15.23:0/980751202 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f11e8045250 con 0x7f11f40aae90 2020-11-20T11:35:20.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.675+0000 7f11f8f98700 1 -- 172.21.15.23:0/980751202 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f11f40af560 con 0x7f11f40aae90 2020-11-20T11:35:20.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.676+0000 7f11e27fc700 1 -- 172.21.15.23:0/980751202 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1472 (secure 0 0 0) 0x7f11e8048870 con 0x7f11f40aae90 2020-11-20T11:35:20.679 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":26,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:35:20.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.677+0000 7f11f8f98700 1 -- 172.21.15.23:0/980751202 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f11c8056d60 msgr2=0x7f11c8059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:20.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.677+0000 7f11f8f98700 1 --2- 172.21.15.23:0/980751202 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f11c8056d60 0x7f11c8059220 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f11e4006e70 tx=0x7f11e4015000).stop 2020-11-20T11:35:20.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.677+0000 7f11f8f98700 1 -- 172.21.15.23:0/980751202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11f40aae90 msgr2=0x7f11f4141a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:20.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.677+0000 7f11f8f98700 1 --2- 172.21.15.23:0/980751202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11f40aae90 0x7f11f4141a90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f11e800db30 tx=0x7f11e8008040).stop 2020-11-20T11:35:20.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.677+0000 7f11f8f98700 1 -- 172.21.15.23:0/980751202 shutdown_connections 2020-11-20T11:35:20.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.677+0000 7f11f8f98700 1 --2- 172.21.15.23:0/980751202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f11f40a7fe0 0x7f11f40af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:20.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.677+0000 7f11f8f98700 1 --2- 172.21.15.23:0/980751202 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f11c8056d60 0x7f11c8059220 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:20.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.677+0000 7f11f8f98700 1 --2- 172.21.15.23:0/980751202 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11f40aaab0 0x7f11f4140cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:20.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.677+0000 7f11f8f98700 1 --2- 172.21.15.23:0/980751202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11f40aae90 0x7f11f4141a90 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:20.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.677+0000 7f11f8f98700 1 -- 172.21.15.23:0/980751202 >> 172.21.15.23:0/980751202 conn(0x7f11f40678e0 msgr2=0x7f11f413a710 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:20.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.677+0000 7f11f8f98700 1 -- 172.21.15.23:0/980751202 shutdown_connections 2020-11-20T11:35:20.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:20.677+0000 7f11f8f98700 1 -- 172.21.15.23:0/980751202 wait complete. 2020-11-20T11:35:23.693 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:35:23.837 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.834+0000 7fbb80a6e700 1 Processor -- start 2020-11-20T11:35:23.838 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.834+0000 7fbb80a6e700 1 -- start start 2020-11-20T11:35:23.840 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.835+0000 7fbb80a6e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb7c0a2080 0x7fbb7c0aef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:23.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.835+0000 7fbb80a6e700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb7c0aaab0 con 0x7fbb7c067a10 2020-11-20T11:35:23.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.835+0000 7fbb80a6e700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb7c0aad50 con 0x7fbb7c0a2460 2020-11-20T11:35:23.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.835+0000 7fbb80a6e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7c13aa50 con 0x7fbb7c0a2080 2020-11-20T11:35:23.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.835+0000 7fbb7bfff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fbb7c067a10 0x7fbb7c1453c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:56628/0 (socket says 172.21.15.23:56628) 2020-11-20T11:35:23.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.835+0000 7fbb7bfff700 1 -- 172.21.15.23:0/1957489233 learned_addr learned my addr 172.21.15.23:0/1957489233 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:35:23.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.835+0000 7fbb7b7fe700 1 --2- 172.21.15.23:0/1957489233 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb7c0a2080 0x7fbb7c0aef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:23.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.835+0000 7fbb7a7fc700 1 -- 172.21.15.23:0/1957489233 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1726554558 0 0) 0x7fbb7c0aaab0 con 0x7fbb7c067a10 2020-11-20T11:35:23.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.835+0000 7fbb7a7fc700 1 -- 172.21.15.23:0/1957489233 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb68003280 con 0x7fbb7c067a10 2020-11-20T11:35:23.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.836+0000 7fbb7a7fc700 1 -- 172.21.15.23:0/1957489233 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 629548157 0 0) 0x7fbb68003280 con 0x7fbb7c067a10 2020-11-20T11:35:23.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.836+0000 7fbb7a7fc700 1 -- 172.21.15.23:0/1957489233 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb7c0a2080 msgr2=0x7fbb7c0aef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:35:23.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.836+0000 7fbb7a7fc700 1 --2- 172.21.15.23:0/1957489233 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb7c0a2080 0x7fbb7c0aef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:23.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.836+0000 7fbb7a7fc700 1 -- 172.21.15.23:0/1957489233 >> v1:172.21.15.155:6789/0 conn(0x7fbb7c0a2460 legacy=0x7fbb7c137df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:23.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.836+0000 7fbb7a7fc700 1 -- 172.21.15.23:0/1957489233 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb7c13b920 con 0x7fbb7c067a10 2020-11-20T11:35:23.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.836+0000 7fbb7a7fc700 1 -- 172.21.15.23:0/1957489233 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7fbb740018a0 con 0x7fbb7c067a10 2020-11-20T11:35:23.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.836+0000 7fbb7a7fc700 1 -- 172.21.15.23:0/1957489233 >> v1:172.21.15.23:6789/0 conn(0x7fbb7c067a10 legacy=0x7fbb7c1453c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:23.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.836+0000 7fbb7a7fc700 1 --2- 172.21.15.23:0/1957489233 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb680037a0 0x7fbb68003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:23.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.836+0000 7fbb7a7fc700 1 --2- 172.21.15.23:0/1957489233 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb68004520 0x7fbb680049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:23.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.836+0000 7fbb7a7fc700 1 --2- 172.21.15.23:0/1957489233 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb7c0a2080 0x7fbb68004e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:23.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.836+0000 7fbb7a7fc700 1 -- 172.21.15.23:0/1957489233 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb680053a0 con 0x7fbb68004520 2020-11-20T11:35:23.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.836+0000 7fbb7a7fc700 1 -- 172.21.15.23:0/1957489233 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb680042e0 con 0x7fbb680037a0 2020-11-20T11:35:23.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.836+0000 7fbb7a7fc700 1 -- 172.21.15.23:0/1957489233 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb68013f80 con 0x7fbb7c0a2080 2020-11-20T11:35:23.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.836+0000 7fbb7affd700 1 --2- 172.21.15.23:0/1957489233 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb68004520 0x7fbb680049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:23.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.836+0000 7fbb7b7fe700 1 --2- 172.21.15.23:0/1957489233 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb680037a0 0x7fbb68003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:23.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.836+0000 7fbb7bfff700 1 --2- 172.21.15.23:0/1957489233 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb7c0a2080 0x7fbb68004e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:23.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.836+0000 7fbb7affd700 1 -- 172.21.15.23:0/1957489233 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb7c0a2080 msgr2=0x7fbb68004e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:35:23.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.836+0000 7fbb7affd700 1 --2- 172.21.15.23:0/1957489233 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb7c0a2080 0x7fbb68004e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:23.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.836+0000 7fbb7affd700 1 -- 172.21.15.23:0/1957489233 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb680037a0 msgr2=0x7fbb68003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:35:23.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.836+0000 7fbb7affd700 1 --2- 172.21.15.23:0/1957489233 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb680037a0 0x7fbb68003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:23.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.836+0000 7fbb7affd700 1 -- 172.21.15.23:0/1957489233 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb68003280 con 0x7fbb68004520 2020-11-20T11:35:23.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.836+0000 7fbb7affd700 1 --2- 172.21.15.23:0/1957489233 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb68004520 0x7fbb680049a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fbb7c0a2d70 tx=0x7fbb70007bd0).ready entity=mon.0 client_cookie=e93757e53bfd588d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:23.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.837+0000 7fbb7a7fc700 1 -- 172.21.15.23:0/1957489233 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb70007020 con 0x7fbb68004520 2020-11-20T11:35:23.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.837+0000 7fbb7a7fc700 1 -- 172.21.15.23:0/1957489233 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb70007180 con 0x7fbb68004520 2020-11-20T11:35:23.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.837+0000 7fbb7a7fc700 1 -- 172.21.15.23:0/1957489233 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb7000c830 con 0x7fbb68004520 2020-11-20T11:35:23.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.837+0000 7fbb80a6e700 1 -- 172.21.15.23:0/1957489233 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb68004520 msgr2=0x7fbb680049a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:23.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.837+0000 7fbb80a6e700 1 --2- 172.21.15.23:0/1957489233 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb68004520 0x7fbb680049a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fbb7c0a2d70 tx=0x7fbb70007bd0).stop 2020-11-20T11:35:23.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.837+0000 7fbb7bfff700 1 -- 172.21.15.23:0/1957489233 reap_dead start 2020-11-20T11:35:23.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.837+0000 7fbb80a6e700 1 -- 172.21.15.23:0/1957489233 shutdown_connections 2020-11-20T11:35:23.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.837+0000 7fbb80a6e700 1 -- 172.21.15.23:0/1957489233 >> 172.21.15.23:0/1957489233 conn(0x7fbb7c063980 msgr2=0x7fbb7c063d90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:23.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.837+0000 7fbb80a6e700 1 -- 172.21.15.23:0/1957489233 shutdown_connections 2020-11-20T11:35:23.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.838+0000 7fbb80a6e700 1 -- 172.21.15.23:0/1957489233 wait complete. 2020-11-20T11:35:23.852 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.839+0000 7fbb80a6e700 1 Processor -- start 2020-11-20T11:35:23.852 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.839+0000 7fbb80a6e700 1 -- start start 2020-11-20T11:35:23.852 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.839+0000 7fbb80a6e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb7c0aee80 0x7fbb7c0a7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:23.852 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.839+0000 7fbb80a6e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb7c0a6300 0x7fbb7c0a7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:23.853 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.839+0000 7fbb80a6e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb7c0a3d70 0x7fbb7c0a41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:23.853 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.839+0000 7fbb80a6e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7c0a7c60 con 0x7fbb7c0a6300 2020-11-20T11:35:23.853 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.839+0000 7fbb80a6e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7c0a6dc0 con 0x7fbb7c0a3d70 2020-11-20T11:35:23.853 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.839+0000 7fbb80a6e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7c0a6880 con 0x7fbb7c0aee80 2020-11-20T11:35:23.853 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.840+0000 7fbb7affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb7c0a6300 0x7fbb7c0a7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:23.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.840+0000 7fbb7affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb7c0a6300 0x7fbb7c0a7770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42390/0 (socket says 172.21.15.23:42390) 2020-11-20T11:35:23.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.840+0000 7fbb7affd700 1 -- 172.21.15.23:0/606910933 learned_addr learned my addr 172.21.15.23:0/606910933 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:35:23.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.840+0000 7fbb7b7fe700 1 --2- 172.21.15.23:0/606910933 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb7c0aee80 0x7fbb7c0a7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:23.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.840+0000 7fbb7bfff700 1 --2- 172.21.15.23:0/606910933 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb7c0a3d70 0x7fbb7c0a41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:23.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.840+0000 7fbb7affd700 1 -- 172.21.15.23:0/606910933 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb7c0aee80 msgr2=0x7fbb7c0a7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:23.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.840+0000 7fbb7affd700 1 --2- 172.21.15.23:0/606910933 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb7c0aee80 0x7fbb7c0a7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:23.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.840+0000 7fbb7affd700 1 -- 172.21.15.23:0/606910933 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb7c0a3d70 msgr2=0x7fbb7c0a41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:23.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.840+0000 7fbb7affd700 1 --2- 172.21.15.23:0/606910933 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb7c0a3d70 0x7fbb7c0a41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:23.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.840+0000 7fbb7affd700 1 -- 172.21.15.23:0/606910933 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb7c0a6b20 con 0x7fbb7c0a6300 2020-11-20T11:35:23.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.840+0000 7fbb7affd700 1 --2- 172.21.15.23:0/606910933 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb7c0a6300 0x7fbb7c0a7770 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fbb7c0a5fe0 tx=0x7fbb70015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:23.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.840+0000 7fbb5b7fe700 1 -- 172.21.15.23:0/606910933 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb700138f0 con 0x7fbb7c0a6300 2020-11-20T11:35:23.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.840+0000 7fbb5b7fe700 1 -- 172.21.15.23:0/606910933 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb70013a50 con 0x7fbb7c0a6300 2020-11-20T11:35:23.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.840+0000 7fbb80a6e700 1 -- 172.21.15.23:0/606910933 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb7c0af260 con 0x7fbb7c0a6300 2020-11-20T11:35:23.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.840+0000 7fbb5b7fe700 1 -- 172.21.15.23:0/606910933 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb70013df0 con 0x7fbb7c0a6300 2020-11-20T11:35:23.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.840+0000 7fbb80a6e700 1 -- 172.21.15.23:0/606910933 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb7c0a9140 con 0x7fbb7c0a6300 2020-11-20T11:35:23.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.841+0000 7fbb80a6e700 1 -- 172.21.15.23:0/606910933 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb7c0a6f70 con 0x7fbb7c0a6300 2020-11-20T11:35:23.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.842+0000 7fbb5b7fe700 1 -- 172.21.15.23:0/606910933 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbb70026020 con 0x7fbb7c0a6300 2020-11-20T11:35:23.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.842+0000 7fbb5b7fe700 1 --2- 172.21.15.23:0/606910933 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbb54056d90 0x7fbb54059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:23.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.842+0000 7fbb5b7fe700 1 -- 172.21.15.23:0/606910933 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fbb70071480 con 0x7fbb7c0a6300 2020-11-20T11:35:23.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.843+0000 7fbb7b7fe700 1 --2- 172.21.15.23:0/606910933 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbb54056d90 0x7fbb54059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:23.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.843+0000 7fbb7b7fe700 1 --2- 172.21.15.23:0/606910933 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbb54056d90 0x7fbb54059250 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fbb6c006fd0 tx=0x7fbb6c008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:23.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:23.846+0000 7fbb5b7fe700 1 -- 172.21.15.23:0/606910933 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fbb70046770 con 0x7fbb7c0a6300 2020-11-20T11:35:24.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.071+0000 7fbb80a6e700 1 -- 172.21.15.23:0/606910933 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbb7c057dc0 con 0x7fbb7c0a6300 2020-11-20T11:35:24.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.072+0000 7fbb5b7fe700 1 -- 172.21.15.23:0/606910933 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1477 (secure 0 0 0) 0x7fbb70049d90 con 0x7fbb7c0a6300 2020-11-20T11:35:24.075 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:35:24.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.074+0000 7fbb80a6e700 1 -- 172.21.15.23:0/606910933 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbb54056d90 msgr2=0x7fbb54059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:24.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.074+0000 7fbb80a6e700 1 --2- 172.21.15.23:0/606910933 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbb54056d90 0x7fbb54059250 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fbb6c006fd0 tx=0x7fbb6c008040).stop 2020-11-20T11:35:24.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.074+0000 7fbb80a6e700 1 -- 172.21.15.23:0/606910933 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb7c0a6300 msgr2=0x7fbb7c0a7770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:24.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.074+0000 7fbb80a6e700 1 --2- 172.21.15.23:0/606910933 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb7c0a6300 0x7fbb7c0a7770 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fbb7c0a5fe0 tx=0x7fbb70015000).stop 2020-11-20T11:35:24.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.074+0000 7fbb80a6e700 1 -- 172.21.15.23:0/606910933 shutdown_connections 2020-11-20T11:35:24.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.074+0000 7fbb80a6e700 1 --2- 172.21.15.23:0/606910933 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb7c0a3d70 0x7fbb7c0a41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:24.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.074+0000 7fbb80a6e700 1 --2- 172.21.15.23:0/606910933 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbb54056d90 0x7fbb54059250 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:24.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.074+0000 7fbb80a6e700 1 --2- 172.21.15.23:0/606910933 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb7c0aee80 0x7fbb7c0a7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:24.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.074+0000 7fbb80a6e700 1 --2- 172.21.15.23:0/606910933 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb7c0a6300 0x7fbb7c0a7770 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:24.079 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.074+0000 7fbb80a6e700 1 -- 172.21.15.23:0/606910933 >> 172.21.15.23:0/606910933 conn(0x7fbb7c063980 msgr2=0x7fbb7c068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:24.079 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.074+0000 7fbb80a6e700 1 -- 172.21.15.23:0/606910933 shutdown_connections 2020-11-20T11:35:24.079 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.074+0000 7fbb80a6e700 1 -- 172.21.15.23:0/606910933 wait complete. 2020-11-20T11:35:24.089 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:35:24.090 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:35:24.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.210+0000 7f6b894c8700 1 Processor -- start 2020-11-20T11:35:24.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.210+0000 7f6b894c8700 1 -- start start 2020-11-20T11:35:24.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.211+0000 7f6b894c8700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6b840a2080 0x7f6b840aef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:24.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.211+0000 7f6b894c8700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6b84067a10 0x7f6b84067e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:24.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.211+0000 7f6b894c8700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b840aaab0 con 0x7f6b840a2460 2020-11-20T11:35:24.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.211+0000 7f6b894c8700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b8413aa50 con 0x7f6b84067a10 2020-11-20T11:35:24.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.211+0000 7f6b894c8700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b840aad50 con 0x7f6b840a2080 2020-11-20T11:35:24.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.211+0000 7f6b88cc7700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6b84067a10 0x7f6b84067e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:24.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.211+0000 7f6b837fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f6b840a2460 0x7f6b84137df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:56646/0 (socket says 172.21.15.23:56646) 2020-11-20T11:35:24.225 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.211+0000 7f6b837fe700 1 -- 172.21.15.23:0/3113786533 learned_addr learned my addr 172.21.15.23:0/3113786533 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:35:24.225 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.211+0000 7f6b83fff700 1 --2- 172.21.15.23:0/3113786533 >> v2:172.21.15.155:3300/0 conn(0x7f6b840a2080 0x7f6b840aef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:24.225 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.212+0000 7f6b88cc7700 1 -- 172.21.15.23:0/3113786533 >> v2:172.21.15.155:3300/0 conn(0x7f6b840a2080 msgr2=0x7f6b840aef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:35:24.226 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.212+0000 7f6b88cc7700 1 --2- 172.21.15.23:0/3113786533 >> v2:172.21.15.155:3300/0 conn(0x7f6b840a2080 0x7f6b840aef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:24.226 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.212+0000 7f6b88cc7700 1 -- 172.21.15.23:0/3113786533 >> v1:172.21.15.23:6789/0 conn(0x7f6b840a2460 legacy=0x7f6b84137df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:24.226 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.212+0000 7f6b88cc7700 1 -- 172.21.15.23:0/3113786533 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b8413b920 con 0x7f6b84067a10 2020-11-20T11:35:24.227 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.212+0000 7f6b82ffd700 1 -- 172.21.15.23:0/3113786533 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2769380393 0 0) 0x7f6b840aaab0 con 0x7f6b840a2460 2020-11-20T11:35:24.227 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.212+0000 7f6b88cc7700 1 --2- 172.21.15.23:0/3113786533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b84067a10 0x7f6b84067e40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f6b840635a0 tx=0x7f6b7c009220).ready entity=mon.0 client_cookie=e214bf8d78088171 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:24.227 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.212+0000 7f6b82ffd700 1 -- 172.21.15.23:0/3113786533 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c00bbf0 con 0x7f6b84067a10 2020-11-20T11:35:24.228 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.212+0000 7f6b82ffd700 1 -- 172.21.15.23:0/3113786533 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b7c00bd50 con 0x7f6b84067a10 2020-11-20T11:35:24.228 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.212+0000 7f6b82ffd700 1 -- 172.21.15.23:0/3113786533 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c011610 con 0x7f6b84067a10 2020-11-20T11:35:24.228 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.212+0000 7f6b894c8700 1 -- 172.21.15.23:0/3113786533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b84067a10 msgr2=0x7f6b84067e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:24.229 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.212+0000 7f6b894c8700 1 --2- 172.21.15.23:0/3113786533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b84067a10 0x7f6b84067e40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f6b840635a0 tx=0x7f6b7c009220).stop 2020-11-20T11:35:24.229 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.212+0000 7f6b894c8700 1 -- 172.21.15.23:0/3113786533 shutdown_connections 2020-11-20T11:35:24.229 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.212+0000 7f6b894c8700 1 --2- 172.21.15.23:0/3113786533 >> v2:172.21.15.155:3300/0 conn(0x7f6b840a2080 0x7f6b840aef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:24.230 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.212+0000 7f6b894c8700 1 --2- 172.21.15.23:0/3113786533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b84067a10 0x7f6b84067e40 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:24.230 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.212+0000 7f6b894c8700 1 -- 172.21.15.23:0/3113786533 >> 172.21.15.23:0/3113786533 conn(0x7f6b84063980 msgr2=0x7f6b84063d90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:24.230 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.213+0000 7f6b894c8700 1 -- 172.21.15.23:0/3113786533 shutdown_connections 2020-11-20T11:35:24.231 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.213+0000 7f6b894c8700 1 -- 172.21.15.23:0/3113786533 wait complete. 2020-11-20T11:35:24.231 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.214+0000 7f6b894c8700 1 Processor -- start 2020-11-20T11:35:24.231 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.214+0000 7f6b894c8700 1 -- start start 2020-11-20T11:35:24.232 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.215+0000 7f6b894c8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b84067a10 0x7f6b840ac1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:24.232 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.215+0000 7f6b894c8700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6b840a2080 0x7f6b8413d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:24.232 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.215+0000 7f6b894c8700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b840a2460 0x7f6b8413d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:24.233 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.215+0000 7f6b894c8700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8413dac0 con 0x7f6b84067a10 2020-11-20T11:35:24.233 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.215+0000 7f6b894c8700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b840a4380 con 0x7f6b840a2460 2020-11-20T11:35:24.233 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.215+0000 7f6b894c8700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8413c360 con 0x7f6b840a2080 2020-11-20T11:35:24.234 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.215+0000 7f6b83fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b84067a10 0x7f6b840ac1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:24.234 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.215+0000 7f6b83fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b84067a10 0x7f6b840ac1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42402/0 (socket says 172.21.15.23:42402) 2020-11-20T11:35:24.234 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.215+0000 7f6b83fff700 1 -- 172.21.15.23:0/530867626 learned_addr learned my addr 172.21.15.23:0/530867626 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:35:24.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.215+0000 7f6b837fe700 1 --2- 172.21.15.23:0/530867626 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6b840a2080 0x7f6b8413d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:24.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.215+0000 7f6b88cc7700 1 --2- 172.21.15.23:0/530867626 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b840a2460 0x7f6b8413d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:24.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.215+0000 7f6b83fff700 1 -- 172.21.15.23:0/530867626 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6b840a2080 msgr2=0x7f6b8413d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:24.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.215+0000 7f6b83fff700 1 --2- 172.21.15.23:0/530867626 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6b840a2080 0x7f6b8413d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:24.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.215+0000 7f6b83fff700 1 -- 172.21.15.23:0/530867626 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b840a2460 msgr2=0x7f6b8413d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:24.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.215+0000 7f6b83fff700 1 --2- 172.21.15.23:0/530867626 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b840a2460 0x7f6b8413d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:24.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.215+0000 7f6b83fff700 1 -- 172.21.15.23:0/530867626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b840a3e40 con 0x7f6b84067a10 2020-11-20T11:35:24.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.215+0000 7f6b83fff700 1 --2- 172.21.15.23:0/530867626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b84067a10 0x7f6b840ac1f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f6b7400db30 tx=0x7f6b74008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:24.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.216+0000 7f6b63fff700 1 -- 172.21.15.23:0/530867626 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7400de40 con 0x7f6b84067a10 2020-11-20T11:35:24.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.216+0000 7f6b63fff700 1 -- 172.21.15.23:0/530867626 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b74009850 con 0x7f6b84067a10 2020-11-20T11:35:24.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.216+0000 7f6b894c8700 1 -- 172.21.15.23:0/530867626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b840a40e0 con 0x7f6b84067a10 2020-11-20T11:35:24.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.216+0000 7f6b63fff700 1 -- 172.21.15.23:0/530867626 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b74009bd0 con 0x7f6b84067a10 2020-11-20T11:35:24.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.216+0000 7f6b894c8700 1 -- 172.21.15.23:0/530867626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b8413e870 con 0x7f6b84067a10 2020-11-20T11:35:24.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.216+0000 7f6b894c8700 1 --2- 172.21.15.23:0/530867626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b8413def0 0x7f6b84136230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:24.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.216+0000 7f6b894c8700 1 -- 172.21.15.23:0/530867626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b840671c0 con 0x7f6b8413def0 2020-11-20T11:35:24.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.217+0000 7f6b837fe700 1 --2- 172.21.15.23:0/530867626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b8413def0 0x7f6b84136230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:24.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.217+0000 7f6b837fe700 1 -- 172.21.15.23:0/530867626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6b840aa6d0 con 0x7f6b8413def0 2020-11-20T11:35:24.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.217+0000 7f6b837fe700 1 --2- 172.21.15.23:0/530867626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b8413def0 0x7f6b84136230 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f6b8413f060 tx=0x7f6b78007c60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:24.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.217+0000 7f6b63fff700 1 -- 172.21.15.23:0/530867626 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b78011590 con 0x7f6b8413def0 2020-11-20T11:35:24.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.217+0000 7f6b63fff700 1 -- 172.21.15.23:0/530867626 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6b74013480 con 0x7f6b84067a10 2020-11-20T11:35:24.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.218+0000 7f6b63fff700 1 --2- 172.21.15.23:0/530867626 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6b5c056e30 0x7f6b5c0592f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:24.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.218+0000 7f6b63fff700 1 -- 172.21.15.23:0/530867626 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f6b740705d0 con 0x7f6b84067a10 2020-11-20T11:35:24.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.218+0000 7f6b63fff700 1 -- 172.21.15.23:0/530867626 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f6b780116f0 con 0x7f6b8413def0 2020-11-20T11:35:24.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.218+0000 7f6b63fff700 1 -- 172.21.15.23:0/530867626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b8413def0 msgr2=0x7f6b84136230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:24.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.218+0000 7f6b63fff700 1 --2- 172.21.15.23:0/530867626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b8413def0 0x7f6b84136230 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f6b8413f060 tx=0x7f6b78007c60).stop 2020-11-20T11:35:24.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.218+0000 7f6b63fff700 1 -- 172.21.15.23:0/530867626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b8413def0 msgr2=0x7f6b84136230 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:35:24.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.218+0000 7f6b63fff700 1 --2- 172.21.15.23:0/530867626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b8413def0 0x7f6b84136230 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:24.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.218+0000 7f6b88cc7700 1 --2- 172.21.15.23:0/530867626 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6b5c056e30 0x7f6b5c0592f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:24.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.218+0000 7f6b88cc7700 1 --2- 172.21.15.23:0/530867626 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6b5c056e30 0x7f6b5c0592f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f6b7c0090b0 tx=0x7f6b7c016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:24.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.226+0000 7f6b894c8700 1 --2- 172.21.15.23:0/530867626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b84136720 0x7f6b840ade60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:24.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.226+0000 7f6b894c8700 1 -- 172.21.15.23:0/530867626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b84057dc0 con 0x7f6b84136720 2020-11-20T11:35:24.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.226+0000 7f6b837fe700 1 --2- 172.21.15.23:0/530867626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b84136720 0x7f6b840ade60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:24.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.226+0000 7f6b837fe700 1 -- 172.21.15.23:0/530867626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6b84044710 con 0x7f6b84136720 2020-11-20T11:35:24.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.226+0000 7f6b837fe700 1 --2- 172.21.15.23:0/530867626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b84136720 0x7f6b840ade60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f6b78016000 tx=0x7f6b78019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:24.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.226+0000 7f6b63fff700 1 -- 172.21.15.23:0/530867626 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b780135c0 con 0x7f6b84136720 2020-11-20T11:35:24.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.226+0000 7f6b63fff700 1 -- 172.21.15.23:0/530867626 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4615 (secure 0 0 0) 0x7f6b78011380 con 0x7f6b84136720 2020-11-20T11:35:24.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.226+0000 7f6b63fff700 1 -- 172.21.15.23:0/530867626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b84136720 msgr2=0x7f6b840ade60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:24.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.226+0000 7f6b63fff700 1 --2- 172.21.15.23:0/530867626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b84136720 0x7f6b840ade60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f6b78016000 tx=0x7f6b78019000).stop 2020-11-20T11:35:24.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.227+0000 7f6b63fff700 1 -- 172.21.15.23:0/530867626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b84136720 msgr2=0x7f6b840ade60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:35:24.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.227+0000 7f6b63fff700 1 --2- 172.21.15.23:0/530867626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b84136720 0x7f6b840ade60 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:24.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.227+0000 7f6b894c8700 1 -- 172.21.15.23:0/530867626 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6b5c056e30 msgr2=0x7f6b5c0592f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:24.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.227+0000 7f6b894c8700 1 --2- 172.21.15.23:0/530867626 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6b5c056e30 0x7f6b5c0592f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f6b7c0090b0 tx=0x7f6b7c016000).stop 2020-11-20T11:35:24.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.227+0000 7f6b894c8700 1 -- 172.21.15.23:0/530867626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b84067a10 msgr2=0x7f6b840ac1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:24.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.227+0000 7f6b894c8700 1 --2- 172.21.15.23:0/530867626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b84067a10 0x7f6b840ac1f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f6b7400db30 tx=0x7f6b74008040).stop 2020-11-20T11:35:24.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.227+0000 7f6b88cc7700 1 -- 172.21.15.23:0/530867626 reap_dead start 2020-11-20T11:35:24.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.227+0000 7f6b88cc7700 1 -- 172.21.15.23:0/530867626 reap_dead start 2020-11-20T11:35:24.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.227+0000 7f6b894c8700 1 -- 172.21.15.23:0/530867626 shutdown_connections 2020-11-20T11:35:24.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.227+0000 7f6b894c8700 1 -- 172.21.15.23:0/530867626 >> 172.21.15.23:0/530867626 conn(0x7f6b84063980 msgr2=0x7f6b840a18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:24.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.227+0000 7f6b894c8700 1 -- 172.21.15.23:0/530867626 shutdown_connections 2020-11-20T11:35:24.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.227+0000 7f6b894c8700 1 -- 172.21.15.23:0/530867626 wait complete. 2020-11-20T11:35:24.247 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:35:24.248 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:35:24.248 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:35:24.248 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:35:24.248 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 28, 2020-11-20T11:35:24.248 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:35:24.249 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:35:24.249 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:35:24.249 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:35:24.249 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:24.249 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 3, 2020-11-20T11:35:24.250 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:35:24.250 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:35:24.250 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:35:24.250 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:35:24.250 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:35:24.251 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:35:24.251 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:35:24.251 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:35:24.251 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:35:24.251 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:35:24.252 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:35:24.252 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:24.252 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:35:24.252 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:35:24.252 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:35:24.253 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:35:24.253 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:35:24.253 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:35:24.253 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:35:24.253 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:35:24.254 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:35:24.254 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:35:24.254 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:24.254 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:24.254 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:35:24.255 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:35:24.255 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:35:24.255 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:35:24.255 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:35:24.255 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:35:24.256 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:35:24.256 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:35:24.256 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:35:24.256 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:35:24.256 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:35:24.257 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:35:24.257 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:35:24.257 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:35:24.257 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:35:24.257 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:35:24.258 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:35:24.258 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:35:24.258 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:35:24.258 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:35:24.258 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:35:24.259 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:35:24.259 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:35:24.259 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:24.259 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:35:24.259 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:24.260 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:35:24.260 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:24.260 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:35:24.260 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:35:24.260 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:35:24.260 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:35:24.261 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:24.261 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:35:24.261 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:35:24.261 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:24.261 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:24.262 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:24.262 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:35:24.262 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:35:24.262 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:24.262 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:24.263 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:24.263 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:24.263 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:35:24.263 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:35:24.263 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:35:24.264 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:35:24.264 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:35:24.264 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:24.264 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:24.264 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:35:24.265 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:35:24.265 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:35:24.265 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:35:24.265 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:24.265 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:35:24.266 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:35:24.266 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:24.266 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:24.266 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:24.266 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:35:24.267 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:35:24.267 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:24.267 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:24.267 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:24.267 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:24.267 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:35:24.268 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:35:24.268 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:35:24.268 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:35:24.268 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:35:24.268 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:24.269 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:24.269 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:35:24.269 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:35:24.269 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:35:24.269 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:35:24.270 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:24.270 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:35:24.270 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:35:24.270 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:24.270 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:24.271 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:24.271 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:35:24.271 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:35:24.271 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:24.271 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:24.272 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:24.272 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:24.272 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:35:24.272 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:35:24.272 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:35:24.272 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:35:24.273 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:35:24.273 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:24.273 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:24.273 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:24.274 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:35:24.274 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:35:24.274 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:24.274 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:24.275 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:24.275 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:35:24.275 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:24.275 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:24.275 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:35:24.275 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:24.276 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:24.276 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:24.276 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:35:24.276 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:24.276 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:24.277 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:35:24.277 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:24.277 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:24.277 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:24.277 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 4 2020-11-20T11:35:24.278 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:24.278 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:24.278 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:35:24.278 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:24.278 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:24.279 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:24.279 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:35:24.279 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:24.279 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:24.279 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:24.280 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:35:24.280 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:35:24.281 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:35:24.422 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.419+0000 7fbb79909700 1 Processor -- start 2020-11-20T11:35:24.422 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.419+0000 7fbb79909700 1 -- start start 2020-11-20T11:35:24.425 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.420+0000 7fbb79909700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fbb740a2080 0x7fbb740aef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:24.425 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.420+0000 7fbb79909700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb740a2460 0x7fbb74137df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:24.425 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.420+0000 7fbb79909700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb740aaab0 con 0x7fbb74067a10 2020-11-20T11:35:24.425 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.420+0000 7fbb79909700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb7413aa50 con 0x7fbb740a2080 2020-11-20T11:35:24.426 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.420+0000 7fbb79909700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb740aad50 con 0x7fbb740a2460 2020-11-20T11:35:24.426 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.420+0000 7fbb79108700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fbb74067a10 0x7fbb7413e020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:56668/0 (socket says 172.21.15.23:56668) 2020-11-20T11:35:24.426 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.420+0000 7fbb79108700 1 -- 172.21.15.23:0/543523433 learned_addr learned my addr 172.21.15.23:0/543523433 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:35:24.426 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.420+0000 7fbb78907700 1 --2- 172.21.15.23:0/543523433 >> v2:172.21.15.155:3300/0 conn(0x7fbb740a2080 0x7fbb740aef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:24.426 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.420+0000 7fbb73fff700 1 --2- 172.21.15.23:0/543523433 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb740a2460 0x7fbb74137df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:24.427 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.420+0000 7fbb737fe700 1 -- 172.21.15.23:0/543523433 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3789496443 0 0) 0x7fbb740aaab0 con 0x7fbb74067a10 2020-11-20T11:35:24.427 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.420+0000 7fbb737fe700 1 -- 172.21.15.23:0/543523433 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb58003280 con 0x7fbb74067a10 2020-11-20T11:35:24.427 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.420+0000 7fbb78907700 1 -- 172.21.15.23:0/543523433 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb740a2460 msgr2=0x7fbb74137df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:35:24.427 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.420+0000 7fbb78907700 1 --2- 172.21.15.23:0/543523433 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb740a2460 0x7fbb74137df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:24.427 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.420+0000 7fbb78907700 1 -- 172.21.15.23:0/543523433 >> v1:172.21.15.23:6789/0 conn(0x7fbb74067a10 legacy=0x7fbb7413e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:24.428 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.420+0000 7fbb78907700 1 -- 172.21.15.23:0/543523433 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb7413b920 con 0x7fbb740a2080 2020-11-20T11:35:24.428 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.421+0000 7fbb73fff700 1 --2- 172.21.15.23:0/543523433 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb740a2460 0x7fbb74137df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:35:24.428 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.421+0000 7fbb78907700 1 --2- 172.21.15.23:0/543523433 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb740a2080 0x7fbb740aef10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fbb6800b9d0 tx=0x7fbb68006dd0).ready entity=mon.1 client_cookie=8bd28aaca7d4b654 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:24.428 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.421+0000 7fbb737fe700 1 -- 172.21.15.23:0/543523433 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb68006030 con 0x7fbb740a2080 2020-11-20T11:35:24.429 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.421+0000 7fbb737fe700 1 -- 172.21.15.23:0/543523433 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb68006190 con 0x7fbb740a2080 2020-11-20T11:35:24.429 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.421+0000 7fbb737fe700 1 -- 172.21.15.23:0/543523433 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb68006510 con 0x7fbb740a2080 2020-11-20T11:35:24.430 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.421+0000 7fbb79909700 1 -- 172.21.15.23:0/543523433 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb740a2080 msgr2=0x7fbb740aef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:24.430 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.421+0000 7fbb79909700 1 --2- 172.21.15.23:0/543523433 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb740a2080 0x7fbb740aef10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fbb6800b9d0 tx=0x7fbb68006dd0).stop 2020-11-20T11:35:24.430 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.421+0000 7fbb79909700 1 -- 172.21.15.23:0/543523433 shutdown_connections 2020-11-20T11:35:24.431 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.421+0000 7fbb79909700 1 --2- 172.21.15.23:0/543523433 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb740a2080 0x7fbb740aef10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:24.431 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.421+0000 7fbb79909700 1 --2- 172.21.15.23:0/543523433 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb740a2460 0x7fbb74137df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:24.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.421+0000 7fbb79909700 1 -- 172.21.15.23:0/543523433 >> 172.21.15.23:0/543523433 conn(0x7fbb74063980 msgr2=0x7fbb74063d90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:24.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.421+0000 7fbb79909700 1 -- 172.21.15.23:0/543523433 shutdown_connections 2020-11-20T11:35:24.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.421+0000 7fbb79909700 1 -- 172.21.15.23:0/543523433 wait complete. 2020-11-20T11:35:24.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.422+0000 7fbb79909700 1 Processor -- start 2020-11-20T11:35:24.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.423+0000 7fbb79909700 1 -- start start 2020-11-20T11:35:24.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.423+0000 7fbb79909700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb74067a10 0x7fbb740a8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:25.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.423+0000 7fbb79909700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb740a2080 0x7fbb740a8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:25.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.423+0000 7fbb79909700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb740a2460 0x7fbb740a65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:25.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.423+0000 7fbb79909700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb740a6a90 con 0x7fbb74067a10 2020-11-20T11:35:25.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.423+0000 7fbb79909700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb740aec10 con 0x7fbb740a2460 2020-11-20T11:35:25.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.423+0000 7fbb79909700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb740aeeb0 con 0x7fbb740a2080 2020-11-20T11:35:25.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.423+0000 7fbb78907700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb74067a10 0x7fbb740a8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:25.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.423+0000 7fbb78907700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb74067a10 0x7fbb740a8780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42420/0 (socket says 172.21.15.23:42420) 2020-11-20T11:35:25.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.423+0000 7fbb78907700 1 -- 172.21.15.23:0/336000311 learned_addr learned my addr 172.21.15.23:0/336000311 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:35:25.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.424+0000 7fbb73fff700 1 --2- 172.21.15.23:0/336000311 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb740a2080 0x7fbb740a8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:25.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.424+0000 7fbb78907700 1 -- 172.21.15.23:0/336000311 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb740a2080 msgr2=0x7fbb740a8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:25.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.424+0000 7fbb79108700 1 --2- 172.21.15.23:0/336000311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb740a2460 0x7fbb740a65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:25.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.424+0000 7fbb78907700 1 --2- 172.21.15.23:0/336000311 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb740a2080 0x7fbb740a8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:25.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.424+0000 7fbb78907700 1 -- 172.21.15.23:0/336000311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb740a2460 msgr2=0x7fbb740a65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:25.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.424+0000 7fbb78907700 1 --2- 172.21.15.23:0/336000311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb740a2460 0x7fbb740a65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:25.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.424+0000 7fbb78907700 1 -- 172.21.15.23:0/336000311 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb740a1610 con 0x7fbb74067a10 2020-11-20T11:35:25.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.424+0000 7fbb73fff700 1 --2- 172.21.15.23:0/336000311 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbb740a2080 0x7fbb740a8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:35:25.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.424+0000 7fbb79108700 1 --2- 172.21.15.23:0/336000311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb740a2460 0x7fbb740a65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:35:25.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.424+0000 7fbb78907700 1 --2- 172.21.15.23:0/336000311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb74067a10 0x7fbb740a8780 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fbb74064520 tx=0x7fbb68015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:25.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.424+0000 7fbb70ff9700 1 -- 172.21.15.23:0/336000311 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb68009910 con 0x7fbb74067a10 2020-11-20T11:35:25.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.424+0000 7fbb70ff9700 1 -- 172.21.15.23:0/336000311 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb68009a70 con 0x7fbb74067a10 2020-11-20T11:35:25.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.424+0000 7fbb79909700 1 -- 172.21.15.23:0/336000311 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb740a6300 con 0x7fbb74067a10 2020-11-20T11:35:25.197 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.424+0000 7fbb70ff9700 1 -- 172.21.15.23:0/336000311 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb68009df0 con 0x7fbb74067a10 2020-11-20T11:35:25.197 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.424+0000 7fbb79909700 1 -- 172.21.15.23:0/336000311 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb740a8270 con 0x7fbb74067a10 2020-11-20T11:35:25.197 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.425+0000 7fbb79909700 1 --2- 172.21.15.23:0/336000311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb741371f0 0x7fbb741375d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:25.197 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.425+0000 7fbb79909700 1 -- 172.21.15.23:0/336000311 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb740aeac0 con 0x7fbb741371f0 2020-11-20T11:35:25.197 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.425+0000 7fbb73fff700 1 --2- 172.21.15.23:0/336000311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb741371f0 0x7fbb741375d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:25.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.426+0000 7fbb73fff700 1 -- 172.21.15.23:0/336000311 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb740a8460 con 0x7fbb741371f0 2020-11-20T11:35:25.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.426+0000 7fbb73fff700 1 --2- 172.21.15.23:0/336000311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb741371f0 0x7fbb741375d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fbb740af290 tx=0x7fbb60008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:25.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.426+0000 7fbb70ff9700 1 -- 172.21.15.23:0/336000311 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6000d8c0 con 0x7fbb741371f0 2020-11-20T11:35:25.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.426+0000 7fbb70ff9700 1 -- 172.21.15.23:0/336000311 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbb6801a040 con 0x7fbb74067a10 2020-11-20T11:35:25.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.426+0000 7fbb70ff9700 1 --2- 172.21.15.23:0/336000311 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbb4c056e00 0x7fbb4c0592c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:25.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.427+0000 7fbb70ff9700 1 -- 172.21.15.23:0/336000311 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fbb68071030 con 0x7fbb74067a10 2020-11-20T11:35:25.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.427+0000 7fbb70ff9700 1 -- 172.21.15.23:0/336000311 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fbb60000e90 con 0x7fbb741371f0 2020-11-20T11:35:25.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.427+0000 7fbb70ff9700 1 -- 172.21.15.23:0/336000311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb741371f0 msgr2=0x7fbb741375d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:25.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.427+0000 7fbb70ff9700 1 --2- 172.21.15.23:0/336000311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb741371f0 0x7fbb741375d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fbb740af290 tx=0x7fbb60008040).stop 2020-11-20T11:35:25.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.427+0000 7fbb70ff9700 1 -- 172.21.15.23:0/336000311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb741371f0 msgr2=0x7fbb741375d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:35:25.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.427+0000 7fbb70ff9700 1 --2- 172.21.15.23:0/336000311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb741371f0 0x7fbb741375d0 secure :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0x7fbb740af290 tx=0x7fbb60008040).stop 2020-11-20T11:35:25.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.427+0000 7fbb79108700 1 --2- 172.21.15.23:0/336000311 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbb4c056e00 0x7fbb4c0592c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:25.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.427+0000 7fbb79108700 1 --2- 172.21.15.23:0/336000311 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbb4c056e00 0x7fbb4c0592c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fbb6c007fd0 tx=0x7fbb6c009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:25.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.436+0000 7fbb79909700 1 --2- 172.21.15.23:0/336000311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb741516c0 0x7fbb74151ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:25.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.436+0000 7fbb79909700 1 -- 172.21.15.23:0/336000311 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb740a8630 con 0x7fbb741516c0 2020-11-20T11:35:25.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.437+0000 7fbb73fff700 1 --2- 172.21.15.23:0/336000311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb741516c0 0x7fbb74151ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:25.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.437+0000 7fbb73fff700 1 -- 172.21.15.23:0/336000311 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbb74057dc0 con 0x7fbb741516c0 2020-11-20T11:35:25.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.437+0000 7fbb73fff700 1 --2- 172.21.15.23:0/336000311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb741516c0 0x7fbb74151ad0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fbb60017000 tx=0x7fbb6001a000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:25.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.437+0000 7fbb70ff9700 1 -- 172.21.15.23:0/336000311 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb60020610 con 0x7fbb741516c0 2020-11-20T11:35:25.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.438+0000 7fbb70ff9700 1 -- 172.21.15.23:0/336000311 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4273 (secure 0 0 0) 0x7fbb6000d6b0 con 0x7fbb741516c0 2020-11-20T11:35:25.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.438+0000 7fbb70ff9700 1 -- 172.21.15.23:0/336000311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb741516c0 msgr2=0x7fbb74151ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:25.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.438+0000 7fbb70ff9700 1 --2- 172.21.15.23:0/336000311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb741516c0 0x7fbb74151ad0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fbb60017000 tx=0x7fbb6001a000).stop 2020-11-20T11:35:25.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.438+0000 7fbb70ff9700 1 -- 172.21.15.23:0/336000311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb741516c0 msgr2=0x7fbb74151ad0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:35:25.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.438+0000 7fbb70ff9700 1 --2- 172.21.15.23:0/336000311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbb741516c0 0x7fbb74151ad0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:25.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.438+0000 7fbb79909700 1 -- 172.21.15.23:0/336000311 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbb4c056e00 msgr2=0x7fbb4c0592c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:25.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.438+0000 7fbb79909700 1 --2- 172.21.15.23:0/336000311 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbb4c056e00 0x7fbb4c0592c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fbb6c007fd0 tx=0x7fbb6c009040).stop 2020-11-20T11:35:25.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.438+0000 7fbb79909700 1 -- 172.21.15.23:0/336000311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb74067a10 msgr2=0x7fbb740a8780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:25.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.438+0000 7fbb79909700 1 --2- 172.21.15.23:0/336000311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb74067a10 0x7fbb740a8780 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fbb74064520 tx=0x7fbb68015000).stop 2020-11-20T11:35:25.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.438+0000 7fbb79108700 1 -- 172.21.15.23:0/336000311 reap_dead start 2020-11-20T11:35:25.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.438+0000 7fbb79909700 1 -- 172.21.15.23:0/336000311 shutdown_connections 2020-11-20T11:35:25.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.438+0000 7fbb79909700 1 --2- 172.21.15.23:0/336000311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb74067a10 0x7fbb740a8780 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:25.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.438+0000 7fbb79909700 1 -- 172.21.15.23:0/336000311 >> 172.21.15.23:0/336000311 conn(0x7fbb74063980 msgr2=0x7fbb74068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:25.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.438+0000 7fbb79909700 1 -- 172.21.15.23:0/336000311 shutdown_connections 2020-11-20T11:35:25.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:24.438+0000 7fbb79909700 1 -- 172.21.15.23:0/336000311 wait complete. 2020-11-20T11:35:25.207 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:35:25.207 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:35:25.207 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:35:25.208 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:35:25.208 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 28, 2020-11-20T11:35:25.208 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:35:25.208 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:35:25.208 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:35:25.209 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:35:25.209 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:25.209 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 3, 2020-11-20T11:35:25.209 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:35:25.209 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:35:25.210 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:35:25.210 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:35:25.210 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:35:25.210 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:35:25.210 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:35:25.211 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:35:25.211 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:35:25.211 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:35:25.211 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:35:25.211 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:25.212 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:35:25.212 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:35:25.212 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:35:25.212 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:35:25.212 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:35:25.213 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:35:25.213 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:35:25.213 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:35:25.213 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:35:25.213 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:35:25.214 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:25.214 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:25.214 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:35:25.214 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:35:25.214 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:35:25.214 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:35:25.215 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:35:25.215 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:35:25.215 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:35:25.215 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:35:25.215 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:35:25.216 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:35:25.216 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:35:25.216 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:35:25.216 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:35:25.216 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:35:25.217 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:35:25.217 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:35:25.217 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:35:25.217 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:35:25.217 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:35:25.218 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:35:25.218 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:35:25.218 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:35:25.218 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:35:25.218 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:25.219 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:35:25.219 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:25.219 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:35:25.219 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:25.219 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:35:25.220 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:35:25.220 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:35:25.220 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:35:25.220 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:25.220 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:35:25.220 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:35:25.221 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:25.221 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:25.221 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:25.221 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:35:25.221 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:35:25.222 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:25.222 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:25.222 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:25.222 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:25.222 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:35:25.223 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:35:25.223 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:35:25.223 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:35:25.223 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:35:25.223 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:25.224 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:25.224 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:35:25.224 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:35:25.224 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:35:25.224 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:35:25.225 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:25.225 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:35:25.225 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:35:25.225 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:25.225 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:25.226 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:25.226 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:35:25.226 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:35:25.226 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:25.226 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:25.226 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:25.227 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:25.227 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:35:25.227 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:35:25.227 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:35:25.227 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:35:25.228 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:35:25.228 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:25.228 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:25.228 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:35:25.228 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:35:25.229 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:35:25.229 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:35:25.229 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:25.229 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:35:25.229 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:35:25.230 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:25.230 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:25.230 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:25.230 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:35:25.230 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:35:25.231 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:25.231 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:25.231 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:25.231 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:25.231 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:35:25.232 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:35:25.232 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:35:25.232 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:35:25.232 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:35:25.232 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:25.233 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:25.233 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:25.233 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:35:25.233 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:35:25.233 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:25.234 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:25.234 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:25.234 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:35:25.234 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:25.234 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:25.235 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:35:25.235 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:25.235 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:25.235 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:25.235 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:35:25.236 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:25.236 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:25.236 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:25.236 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:35:25.236 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:35:25.237 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:35:25.369 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.366+0000 7f0b841cc700 1 Processor -- start 2020-11-20T11:35:25.369 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.366+0000 7f0b841cc700 1 -- start start 2020-11-20T11:35:25.371 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.367+0000 7f0b841cc700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0b7c065780 0x7f0b7c0a9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:25.371 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.367+0000 7f0b841cc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0b7c0a9cf0 0x7f0b7c0601b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:25.371 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.367+0000 7f0b841cc700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b7c067430 con 0x7f0b7c05bc80 2020-11-20T11:35:25.372 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.367+0000 7f0b841cc700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b7c0a8ca0 con 0x7f0b7c0a9cf0 2020-11-20T11:35:25.372 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.367+0000 7f0b841cc700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b7c05c060 con 0x7f0b7c065780 2020-11-20T11:35:25.372 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.367+0000 7f0b82769700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0b7c0a9cf0 0x7f0b7c0601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:25.372 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.367+0000 7f0b82769700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0b7c0a9cf0 0x7f0b7c0601b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42436/0 (socket says 172.21.15.23:42436) 2020-11-20T11:35:25.372 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.367+0000 7f0b82769700 1 -- 172.21.15.23:0/1512718406 learned_addr learned my addr 172.21.15.23:0/1512718406 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:35:25.373 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.367+0000 7f0b81f68700 1 --1- 172.21.15.23:0/1512718406 >> v1:172.21.15.23:6789/0 conn(0x7f0b7c05bc80 0x7f0b7c1379e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:56682/0 (socket says 172.21.15.23:56682) 2020-11-20T11:35:25.373 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.367+0000 7f0b81767700 1 --2- 172.21.15.23:0/1512718406 >> v2:172.21.15.155:3300/0 conn(0x7f0b7c065780 0x7f0b7c0a9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:25.373 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.368+0000 7f0b82769700 1 -- 172.21.15.23:0/1512718406 >> v2:172.21.15.155:3300/0 conn(0x7f0b7c065780 msgr2=0x7f0b7c0a9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:35:25.373 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.368+0000 7f0b82769700 1 --2- 172.21.15.23:0/1512718406 >> v2:172.21.15.155:3300/0 conn(0x7f0b7c065780 0x7f0b7c0a9140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:25.373 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.368+0000 7f0b82769700 1 -- 172.21.15.23:0/1512718406 >> v1:172.21.15.23:6789/0 conn(0x7f0b7c05bc80 legacy=0x7f0b7c1379e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:25.374 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.368+0000 7f0b80f66700 1 -- 172.21.15.23:0/1512718406 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 813279276 0 0) 0x7f0b7c067430 con 0x7f0b7c05bc80 2020-11-20T11:35:25.374 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.368+0000 7f0b82769700 1 -- 172.21.15.23:0/1512718406 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7c0a8810 con 0x7f0b7c0a9cf0 2020-11-20T11:35:25.374 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.368+0000 7f0b82769700 1 --2- 172.21.15.23:0/1512718406 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b7c0a9cf0 0x7f0b7c0601b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f0b7c064c40 tx=0x7f0b740092d0).ready entity=mon.0 client_cookie=ce91f6eb678f46f4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:25.374 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.368+0000 7f0b80f66700 1 -- 172.21.15.23:0/1512718406 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7400bbf0 con 0x7f0b7c0a9cf0 2020-11-20T11:35:25.374 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.368+0000 7f0b80f66700 1 -- 172.21.15.23:0/1512718406 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7400bd50 con 0x7f0b7c0a9cf0 2020-11-20T11:35:25.375 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.368+0000 7f0b80f66700 1 -- 172.21.15.23:0/1512718406 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b74011610 con 0x7f0b7c0a9cf0 2020-11-20T11:35:25.377 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.368+0000 7f0b841cc700 1 -- 172.21.15.23:0/1512718406 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b7c0a9cf0 msgr2=0x7f0b7c0601b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:25.377 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.368+0000 7f0b841cc700 1 --2- 172.21.15.23:0/1512718406 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b7c0a9cf0 0x7f0b7c0601b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f0b7c064c40 tx=0x7f0b740092d0).stop 2020-11-20T11:35:25.377 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.368+0000 7f0b841cc700 1 -- 172.21.15.23:0/1512718406 shutdown_connections 2020-11-20T11:35:25.378 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.368+0000 7f0b841cc700 1 --2- 172.21.15.23:0/1512718406 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b7c0a9cf0 0x7f0b7c0601b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:25.378 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.368+0000 7f0b841cc700 1 --2- 172.21.15.23:0/1512718406 >> v2:172.21.15.155:3300/0 conn(0x7f0b7c065780 0x7f0b7c0a9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:25.378 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.368+0000 7f0b841cc700 1 -- 172.21.15.23:0/1512718406 >> 172.21.15.23:0/1512718406 conn(0x7f0b7c0ad7a0 msgr2=0x7f0b7c0adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:25.378 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.369+0000 7f0b841cc700 1 -- 172.21.15.23:0/1512718406 shutdown_connections 2020-11-20T11:35:25.378 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.369+0000 7f0b841cc700 1 -- 172.21.15.23:0/1512718406 wait complete. 2020-11-20T11:35:25.379 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.369+0000 7f0b841cc700 1 Processor -- start 2020-11-20T11:35:25.379 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.370+0000 7f0b841cc700 1 -- start start 2020-11-20T11:35:25.379 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.370+0000 7f0b841cc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0b7c05bc80 0x7f0b7c0a75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:25.379 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.370+0000 7f0b841cc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b7c065780 0x7f0b7c0a1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:25.379 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.370+0000 7f0b841cc700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b7c0a9cf0 0x7f0b7c0a1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:25.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.370+0000 7f0b841cc700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c0a13e0 con 0x7f0b7c065780 2020-11-20T11:35:25.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.370+0000 7f0b841cc700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c0a2c80 con 0x7f0b7c05bc80 2020-11-20T11:35:25.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.370+0000 7f0b841cc700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c0a2f20 con 0x7f0b7c0a9cf0 2020-11-20T11:35:25.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.370+0000 7f0b81767700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b7c065780 0x7f0b7c0a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:25.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.370+0000 7f0b81767700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b7c065780 0x7f0b7c0a1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42440/0 (socket says 172.21.15.23:42440) 2020-11-20T11:35:25.381 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.370+0000 7f0b81767700 1 -- 172.21.15.23:0/476542200 learned_addr learned my addr 172.21.15.23:0/476542200 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:35:25.381 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.370+0000 7f0b81f68700 1 --2- 172.21.15.23:0/476542200 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0b7c05bc80 0x7f0b7c0a75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:25.381 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.370+0000 7f0b82769700 1 --2- 172.21.15.23:0/476542200 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b7c0a9cf0 0x7f0b7c0a1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:25.381 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.371+0000 7f0b81767700 1 -- 172.21.15.23:0/476542200 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b7c0a9cf0 msgr2=0x7f0b7c0a1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:25.381 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.371+0000 7f0b81767700 1 --2- 172.21.15.23:0/476542200 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b7c0a9cf0 0x7f0b7c0a1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:25.382 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.371+0000 7f0b81767700 1 -- 172.21.15.23:0/476542200 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0b7c05bc80 msgr2=0x7f0b7c0a75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:25.382 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.371+0000 7f0b81767700 1 --2- 172.21.15.23:0/476542200 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0b7c05bc80 0x7f0b7c0a75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:25.382 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.371+0000 7f0b81767700 1 -- 172.21.15.23:0/476542200 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7c0a0920 con 0x7f0b7c065780 2020-11-20T11:35:25.382 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.371+0000 7f0b81767700 1 --2- 172.21.15.23:0/476542200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b7c065780 0x7f0b7c0a1810 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f0b6c00eba0 tx=0x7f0b6c006e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:25.382 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.371+0000 7f0b69ffb700 1 -- 172.21.15.23:0/476542200 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c00c850 con 0x7f0b7c065780 2020-11-20T11:35:25.383 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.371+0000 7f0b69ffb700 1 -- 172.21.15.23:0/476542200 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b6c00c9b0 con 0x7f0b7c065780 2020-11-20T11:35:25.383 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.371+0000 7f0b69ffb700 1 -- 172.21.15.23:0/476542200 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c00cd30 con 0x7f0b7c065780 2020-11-20T11:35:25.386 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.371+0000 7f0b841cc700 1 -- 172.21.15.23:0/476542200 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b7c0a49f0 con 0x7f0b7c065780 2020-11-20T11:35:25.386 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.371+0000 7f0b841cc700 1 -- 172.21.15.23:0/476542200 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b7c0a4d20 con 0x7f0b7c065780 2020-11-20T11:35:25.386 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.372+0000 7f0b841cc700 1 --2- 172.21.15.23:0/476542200 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b7c14d680 0x7f0b7c14da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:25.386 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.372+0000 7f0b841cc700 1 -- 172.21.15.23:0/476542200 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c138b90 con 0x7f0b7c14d680 2020-11-20T11:35:25.387 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.372+0000 7f0b69ffb700 1 -- 172.21.15.23:0/476542200 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0b6c012600 con 0x7f0b7c065780 2020-11-20T11:35:25.387 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.372+0000 7f0b81f68700 1 --2- 172.21.15.23:0/476542200 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b7c14d680 0x7f0b7c14da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:25.387 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.372+0000 7f0b69ffb700 1 --2- 172.21.15.23:0/476542200 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0b54056d90 0x7f0b54059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:25.387 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.373+0000 7f0b69ffb700 1 -- 172.21.15.23:0/476542200 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f0b6c070350 con 0x7f0b7c065780 2020-11-20T11:35:25.387 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.373+0000 7f0b81f68700 1 -- 172.21.15.23:0/476542200 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0b7c0a1550 con 0x7f0b7c14d680 2020-11-20T11:35:25.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.373+0000 7f0b81f68700 1 --2- 172.21.15.23:0/476542200 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b7c14d680 0x7f0b7c14da60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f0b78004fd0 tx=0x7f0b78007a60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:25.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.373+0000 7f0b82769700 1 --2- 172.21.15.23:0/476542200 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0b54056d90 0x7f0b54059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:25.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.373+0000 7f0b69ffb700 1 -- 172.21.15.23:0/476542200 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b78011590 con 0x7f0b7c14d680 2020-11-20T11:35:25.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.373+0000 7f0b69ffb700 1 -- 172.21.15.23:0/476542200 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f0b780116f0 con 0x7f0b7c14d680 2020-11-20T11:35:25.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.373+0000 7f0b69ffb700 1 -- 172.21.15.23:0/476542200 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b7c14d680 msgr2=0x7f0b7c14da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:25.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.373+0000 7f0b69ffb700 1 --2- 172.21.15.23:0/476542200 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b7c14d680 0x7f0b7c14da60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f0b78004fd0 tx=0x7f0b78007a60).stop 2020-11-20T11:35:25.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.373+0000 7f0b69ffb700 1 -- 172.21.15.23:0/476542200 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b7c14d680 msgr2=0x7f0b7c14da60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:35:25.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.373+0000 7f0b69ffb700 1 --2- 172.21.15.23:0/476542200 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b7c14d680 0x7f0b7c14da60 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:25.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.374+0000 7f0b82769700 1 --2- 172.21.15.23:0/476542200 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0b54056d90 0x7f0b54059250 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f0b74009130 tx=0x7f0b74016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:25.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.381+0000 7f0b841cc700 1 --2- 172.21.15.23:0/476542200 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b7c149b20 0x7f0b7c149f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:25.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.381+0000 7f0b841cc700 1 -- 172.21.15.23:0/476542200 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c0a30e0 con 0x7f0b7c149b20 2020-11-20T11:35:25.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.381+0000 7f0b81f68700 1 --2- 172.21.15.23:0/476542200 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b7c149b20 0x7f0b7c149f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:25.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.381+0000 7f0b81f68700 1 -- 172.21.15.23:0/476542200 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0b7c062fe0 con 0x7f0b7c149b20 2020-11-20T11:35:25.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.381+0000 7f0b81f68700 1 --2- 172.21.15.23:0/476542200 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b7c149b20 0x7f0b7c149f30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f0b78004fd0 tx=0x7f0b78017000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:25.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.382+0000 7f0b69ffb700 1 -- 172.21.15.23:0/476542200 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b78014e30 con 0x7f0b7c149b20 2020-11-20T11:35:25.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.382+0000 7f0b69ffb700 1 -- 172.21.15.23:0/476542200 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4783 (secure 0 0 0) 0x7f0b78011380 con 0x7f0b7c149b20 2020-11-20T11:35:25.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.382+0000 7f0b69ffb700 1 -- 172.21.15.23:0/476542200 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b7c149b20 msgr2=0x7f0b7c149f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:25.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.382+0000 7f0b69ffb700 1 --2- 172.21.15.23:0/476542200 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b7c149b20 0x7f0b7c149f30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f0b78004fd0 tx=0x7f0b78017000).stop 2020-11-20T11:35:25.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.382+0000 7f0b69ffb700 1 -- 172.21.15.23:0/476542200 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b7c149b20 msgr2=0x7f0b7c149f30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:35:25.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.382+0000 7f0b69ffb700 1 --2- 172.21.15.23:0/476542200 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0b7c149b20 0x7f0b7c149f30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:25.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.382+0000 7f0b841cc700 1 -- 172.21.15.23:0/476542200 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0b54056d90 msgr2=0x7f0b54059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:25.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.382+0000 7f0b841cc700 1 --2- 172.21.15.23:0/476542200 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0b54056d90 0x7f0b54059250 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f0b74009130 tx=0x7f0b74016000).stop 2020-11-20T11:35:25.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.382+0000 7f0b841cc700 1 -- 172.21.15.23:0/476542200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b7c065780 msgr2=0x7f0b7c0a1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:25.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.382+0000 7f0b841cc700 1 --2- 172.21.15.23:0/476542200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b7c065780 0x7f0b7c0a1810 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f0b6c00eba0 tx=0x7f0b6c006e30).stop 2020-11-20T11:35:25.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.382+0000 7f0b82769700 1 -- 172.21.15.23:0/476542200 reap_dead start 2020-11-20T11:35:25.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.382+0000 7f0b82769700 1 -- 172.21.15.23:0/476542200 reap_dead start 2020-11-20T11:35:25.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.382+0000 7f0b841cc700 1 -- 172.21.15.23:0/476542200 shutdown_connections 2020-11-20T11:35:25.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.382+0000 7f0b841cc700 1 -- 172.21.15.23:0/476542200 >> 172.21.15.23:0/476542200 conn(0x7f0b7c0ad7a0 msgr2=0x7f0b7c0aa220 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:25.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.382+0000 7f0b841cc700 1 -- 172.21.15.23:0/476542200 shutdown_connections 2020-11-20T11:35:25.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.382+0000 7f0b841cc700 1 -- 172.21.15.23:0/476542200 wait complete. 2020-11-20T11:35:25.394 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:35:25.394 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:35:25.394 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:35:25.394 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:35:25.394 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 28, 2020-11-20T11:35:25.395 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:35:25.395 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:35:25.395 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:35:25.395 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:35:25.395 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:25.396 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 4, 2020-11-20T11:35:25.396 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:35:25.396 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:35:25.396 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:35:25.396 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:35:25.396 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:35:25.397 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:35:25.397 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:35:25.397 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:35:25.397 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:35:25.398 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:35:25.398 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:35:25.398 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:25.398 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:35:25.398 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:35:25.399 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:35:25.399 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:35:25.399 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:35:25.399 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:35:25.399 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:35:25.400 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:35:25.400 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:35:25.400 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:35:25.400 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:25.400 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:25.401 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:35:25.401 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:35:25.401 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:35:25.401 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:35:25.401 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:35:25.402 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:35:25.402 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:35:25.402 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:35:25.402 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:35:25.402 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:35:25.403 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:35:25.403 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:35:25.403 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:35:25.403 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:35:25.403 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:35:25.403 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:35:25.404 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:35:25.404 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:35:25.404 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:35:25.404 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:35:25.404 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:35:25.405 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:35:25.405 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:35:25.405 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:25.405 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:35:25.405 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:25.406 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:35:25.406 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:25.406 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:35:25.406 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:35:25.406 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:35:25.407 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:35:25.407 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:25.407 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:35:25.407 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:35:25.407 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:25.408 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:25.408 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:25.408 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:35:25.408 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:35:25.408 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:25.408 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:25.409 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:25.409 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:25.409 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:35:25.409 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:35:25.409 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:35:25.410 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:35:25.410 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:35:25.410 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:25.410 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:25.410 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:35:25.411 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:35:25.411 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:35:25.411 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:35:25.411 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:25.411 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:35:25.412 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:35:25.412 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:25.412 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:25.412 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:25.412 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:35:25.413 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:35:25.413 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:25.413 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:25.413 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:25.413 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:25.414 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:35:25.414 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:35:25.414 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:35:25.414 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:35:25.414 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:35:25.414 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:25.415 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:25.415 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:35:25.415 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:35:25.415 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:35:25.415 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:35:25.416 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:25.416 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:35:25.416 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:35:25.416 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:25.416 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:25.417 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:25.417 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:35:25.417 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:35:25.417 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:25.417 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:25.418 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:25.418 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:25.418 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:35:25.418 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:35:25.418 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:35:25.419 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:35:25.419 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:35:25.419 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:25.419 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:25.419 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:25.419 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:35:25.420 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:35:25.420 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:25.420 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:25.420 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:25.420 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:35:25.421 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:25.421 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:25.421 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:35:25.421 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:25.422 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:25.422 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:25.422 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:35:25.422 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:25.422 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:25.422 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:35:25.423 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:25.423 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:25.423 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:25.423 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 6 2020-11-20T11:35:25.423 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:25.424 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:25.424 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:35:25.424 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:25.424 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:25.424 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:25.425 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:35:25.425 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:25.425 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:25.425 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:35:25.425 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:25.426 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:25.426 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:25.426 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:35:25.426 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:25.426 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:25.427 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:25.427 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:35:25.427 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:35:25.428 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2020-11-20T11:35:25.429 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2020-11-20T11:35:25.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.561+0000 7f0ce7948700 1 Processor -- start 2020-11-20T11:35:25.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.561+0000 7f0ce7948700 1 -- start start 2020-11-20T11:35:25.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.562+0000 7f0ce7948700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0ce00ad8d0 0x7f0ce00a4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:25.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.562+0000 7f0ce7948700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ce013e570 0x7f0ce00a8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:25.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.562+0000 7f0ce7948700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce00b1a20 con 0x7f0ce00ab2a0 2020-11-20T11:35:25.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.562+0000 7f0ce7948700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ce013f4a0 con 0x7f0ce00ad8d0 2020-11-20T11:35:25.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.562+0000 7f0ce7948700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ce00aa0e0 con 0x7f0ce013e570 2020-11-20T11:35:25.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.562+0000 7f0ce56e4700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f0ce00ab2a0 0x7f0ce00a4510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:56700/0 (socket says 172.21.15.23:56700) 2020-11-20T11:35:25.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.562+0000 7f0ce56e4700 1 -- 172.21.15.23:0/253058735 learned_addr learned my addr 172.21.15.23:0/253058735 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:35:25.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.562+0000 7f0ce4ee3700 1 --2- 172.21.15.23:0/253058735 >> v2:172.21.15.23:3300/0 conn(0x7f0ce00ad8d0 0x7f0ce00a4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:25.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.563+0000 7f0ce4ee3700 1 -- 172.21.15.23:0/253058735 >> v2:172.21.15.155:3300/0 conn(0x7f0ce013e570 msgr2=0x7f0ce00a8fc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:35:25.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.563+0000 7f0ce4ee3700 1 --2- 172.21.15.23:0/253058735 >> v2:172.21.15.155:3300/0 conn(0x7f0ce013e570 0x7f0ce00a8fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:35:25.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.563+0000 7f0ce4ee3700 1 -- 172.21.15.23:0/253058735 >> v1:172.21.15.23:6789/0 conn(0x7f0ce00ab2a0 legacy=0x7f0ce00a4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:25.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.563+0000 7f0ce4ee3700 1 -- 172.21.15.23:0/253058735 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce00a8b90 con 0x7f0ce00ad8d0 2020-11-20T11:35:25.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.563+0000 7f0ce4ee3700 1 --2- 172.21.15.23:0/253058735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ce00ad8d0 0x7f0ce00a4c20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f0ce00ad400 tx=0x7f0cd0006e30).ready entity=mon.0 client_cookie=3a935f5cd866e744 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:25.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.563+0000 7f0ccffff700 1 -- 172.21.15.23:0/253058735 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd00097b0 con 0x7f0ce00ad8d0 2020-11-20T11:35:25.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.563+0000 7f0ccffff700 1 -- 172.21.15.23:0/253058735 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd0000e90 con 0x7f0ce00ad8d0 2020-11-20T11:35:25.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.563+0000 7f0ccffff700 1 -- 172.21.15.23:0/253058735 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd0009b20 con 0x7f0ce00ad8d0 2020-11-20T11:35:25.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.564+0000 7f0ce7948700 1 -- 172.21.15.23:0/253058735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ce00ad8d0 msgr2=0x7f0ce00a4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:25.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.564+0000 7f0ce7948700 1 --2- 172.21.15.23:0/253058735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ce00ad8d0 0x7f0ce00a4c20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f0ce00ad400 tx=0x7f0cd0006e30).stop 2020-11-20T11:35:25.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.567+0000 7f0ce7948700 1 -- 172.21.15.23:0/253058735 shutdown_connections 2020-11-20T11:35:25.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.567+0000 7f0ce7948700 1 --2- 172.21.15.23:0/253058735 >> v2:172.21.15.155:3300/0 conn(0x7f0ce013e570 0x7f0ce00a8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:35:25.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.567+0000 7f0ce7948700 1 --2- 172.21.15.23:0/253058735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ce00ad8d0 0x7f0ce00a4c20 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:25.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.567+0000 7f0ce7948700 1 -- 172.21.15.23:0/253058735 >> 172.21.15.23:0/253058735 conn(0x7f0ce00683b0 msgr2=0x7f0ce00b1310 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:25.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.568+0000 7f0ce7948700 1 -- 172.21.15.23:0/253058735 shutdown_connections 2020-11-20T11:35:25.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.568+0000 7f0ce7948700 1 -- 172.21.15.23:0/253058735 wait complete. 2020-11-20T11:35:25.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.569+0000 7f0ce7948700 1 Processor -- start 2020-11-20T11:35:25.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.569+0000 7f0ce7948700 1 -- start start 2020-11-20T11:35:25.574 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.570+0000 7f0ce7948700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ce00ab2a0 0x7f0ce009f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:25.574 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.570+0000 7f0ce7948700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ce00ad8d0 0x7f0ce00a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:25.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.570+0000 7f0ce7948700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ce013e570 0x7f0ce014ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:25.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.570+0000 7f0ce7948700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce009dcd0 con 0x7f0ce013e570 2020-11-20T11:35:25.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.570+0000 7f0ce7948700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce009df00 con 0x7f0ce00ad8d0 2020-11-20T11:35:25.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.570+0000 7f0ce7948700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce009e1a0 con 0x7f0ce00ab2a0 2020-11-20T11:35:25.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.570+0000 7f0ce5ee5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ce013e570 0x7f0ce014ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:25.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.570+0000 7f0ce5ee5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ce013e570 0x7f0ce014ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42458/0 (socket says 172.21.15.23:42458) 2020-11-20T11:35:25.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.570+0000 7f0ce5ee5700 1 -- 172.21.15.23:0/2326656250 learned_addr learned my addr 172.21.15.23:0/2326656250 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:35:25.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.570+0000 7f0ce56e4700 1 --2- 172.21.15.23:0/2326656250 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ce00ab2a0 0x7f0ce009f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:25.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.570+0000 7f0ce4ee3700 1 --2- 172.21.15.23:0/2326656250 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ce00ad8d0 0x7f0ce00a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:25.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.571+0000 7f0ce5ee5700 1 -- 172.21.15.23:0/2326656250 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ce00ab2a0 msgr2=0x7f0ce009f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.571+0000 7f0ce5ee5700 1 --2- 172.21.15.23:0/2326656250 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ce00ab2a0 0x7f0ce009f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.571+0000 7f0ce5ee5700 1 -- 172.21.15.23:0/2326656250 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ce00ad8d0 msgr2=0x7f0ce00a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.571+0000 7f0ce5ee5700 1 --2- 172.21.15.23:0/2326656250 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ce00ad8d0 0x7f0ce00a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.571+0000 7f0ce5ee5700 1 -- 172.21.15.23:0/2326656250 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce009ed50 con 0x7f0ce013e570 2020-11-20T11:35:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.571+0000 7f0ce5ee5700 1 --2- 172.21.15.23:0/2326656250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ce013e570 0x7f0ce014ea30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f0cd800bae0 tx=0x7f0cd8009120).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:25.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.571+0000 7f0ccd7fa700 1 -- 172.21.15.23:0/2326656250 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd800c710 con 0x7f0ce013e570 2020-11-20T11:35:25.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.571+0000 7f0ccd7fa700 1 -- 172.21.15.23:0/2326656250 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd800c890 con 0x7f0ce013e570 2020-11-20T11:35:25.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.571+0000 7f0ce7948700 1 -- 172.21.15.23:0/2326656250 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ce009e640 con 0x7f0ce013e570 2020-11-20T11:35:25.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.571+0000 7f0ccd7fa700 1 -- 172.21.15.23:0/2326656250 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd800ca40 con 0x7f0ce013e570 2020-11-20T11:35:25.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.571+0000 7f0ce7948700 1 -- 172.21.15.23:0/2326656250 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce0067450 con 0x7f0ce013e570 2020-11-20T11:35:25.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.572+0000 7f0ce7948700 1 -- 172.21.15.23:0/2326656250 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ce0060640 con 0x7f0ce013e570 2020-11-20T11:35:25.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.572+0000 7f0ccd7fa700 1 -- 172.21.15.23:0/2326656250 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0cd801c070 con 0x7f0ce013e570 2020-11-20T11:35:25.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.573+0000 7f0ccd7fa700 1 --2- 172.21.15.23:0/2326656250 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0cb8056d60 0x7f0cb8059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:25.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.573+0000 7f0ccd7fa700 1 -- 172.21.15.23:0/2326656250 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f0cd806f690 con 0x7f0ce013e570 2020-11-20T11:35:25.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.573+0000 7f0ce56e4700 1 --2- 172.21.15.23:0/2326656250 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0cb8056d60 0x7f0cb8059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:25.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.573+0000 7f0ce56e4700 1 --2- 172.21.15.23:0/2326656250 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0cb8056d60 0x7f0cb8059220 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f0cdc004610 tx=0x7f0cdc007bc0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:25.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.577+0000 7f0ccd7fa700 1 -- 172.21.15.23:0/2326656250 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0cd8044980 con 0x7f0ce013e570 2020-11-20T11:35:25.802 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.799+0000 7f0ce7948700 1 -- 172.21.15.23:0/2326656250 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f0ce009e360 con 0x7f0ce013e570 2020-11-20T11:35:25.802 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.800+0000 7f0ccd7fa700 1 -- 172.21.15.23:0/2326656250 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f0cd8047fa0 con 0x7f0ce013e570 2020-11-20T11:35:25.804 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.801+0000 7f0ce7948700 1 -- 172.21.15.23:0/2326656250 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0cb8056d60 msgr2=0x7f0cb8059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:25.805 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.801+0000 7f0ce7948700 1 --2- 172.21.15.23:0/2326656250 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0cb8056d60 0x7f0cb8059220 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f0cdc004610 tx=0x7f0cdc007bc0).stop 2020-11-20T11:35:25.805 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.801+0000 7f0ce7948700 1 -- 172.21.15.23:0/2326656250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ce013e570 msgr2=0x7f0ce014ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:25.805 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.801+0000 7f0ce7948700 1 --2- 172.21.15.23:0/2326656250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ce013e570 0x7f0ce014ea30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f0cd800bae0 tx=0x7f0cd8009120).stop 2020-11-20T11:35:25.805 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.802+0000 7f0ce7948700 1 -- 172.21.15.23:0/2326656250 shutdown_connections 2020-11-20T11:35:25.806 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.802+0000 7f0ce7948700 1 --2- 172.21.15.23:0/2326656250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ce013e570 0x7f0ce014ea30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:25.806 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.802+0000 7f0ce7948700 1 --2- 172.21.15.23:0/2326656250 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0cb8056d60 0x7f0cb8059220 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:25.806 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.802+0000 7f0ce7948700 1 --2- 172.21.15.23:0/2326656250 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ce00ab2a0 0x7f0ce009f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:25.806 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.802+0000 7f0ce7948700 1 --2- 172.21.15.23:0/2326656250 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0ce00ad8d0 0x7f0ce00a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:25.806 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.802+0000 7f0ce7948700 1 -- 172.21.15.23:0/2326656250 >> 172.21.15.23:0/2326656250 conn(0x7f0ce00683b0 msgr2=0x7f0ce00a9760 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:25.807 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.802+0000 7f0ce7948700 1 -- 172.21.15.23:0/2326656250 shutdown_connections 2020-11-20T11:35:25.807 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:25.802+0000 7f0ce7948700 1 -- 172.21.15.23:0/2326656250 wait complete. 2020-11-20T11:35:25.817 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2020-11-20T11:35:29.783 INFO:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:35:29.785 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:35:35.818 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:35:35.819 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:35:35.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.938+0000 7f01ed6f2700 1 Processor -- start 2020-11-20T11:35:35.942 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.938+0000 7f01ed6f2700 1 -- start start 2020-11-20T11:35:35.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.939+0000 7f01ed6f2700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f01e8061380 0x7f01e80a8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:35.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.939+0000 7f01ed6f2700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f01e80a8d70 0x7f01e813e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:35.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.939+0000 7f01ed6f2700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01e80a5000 con 0x7f01e813edc0 2020-11-20T11:35:35.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.939+0000 7f01ed6f2700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01e80a5d30 con 0x7f01e8061380 2020-11-20T11:35:35.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.939+0000 7f01ed6f2700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01e80a5fd0 con 0x7f01e80a8d70 2020-11-20T11:35:35.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.939+0000 7f01e7fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f01e8061380 0x7f01e80a8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:35.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.939+0000 7f01e7fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f01e8061380 0x7f01e80a8880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42462/0 (socket says 172.21.15.23:42462) 2020-11-20T11:35:35.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.939+0000 7f01e7fff700 1 -- 172.21.15.23:0/2914251297 learned_addr learned my addr 172.21.15.23:0/2914251297 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:35:35.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.939+0000 7f01e77fe700 1 --2- 172.21.15.23:0/2914251297 >> v2:172.21.15.155:3300/0 conn(0x7f01e80a8d70 0x7f01e813e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:35.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.939+0000 7f01e7fff700 1 -- 172.21.15.23:0/2914251297 >> v2:172.21.15.155:3300/0 conn(0x7f01e80a8d70 msgr2=0x7f01e813e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:35:35.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.939+0000 7f01e7fff700 1 --2- 172.21.15.23:0/2914251297 >> v2:172.21.15.155:3300/0 conn(0x7f01e80a8d70 0x7f01e813e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:35.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.939+0000 7f01e7fff700 1 -- 172.21.15.23:0/2914251297 >> v1:172.21.15.155:6789/0 conn(0x7f01e813edc0 legacy=0x7f01e80a6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:35:35.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.939+0000 7f01e7fff700 1 -- 172.21.15.23:0/2914251297 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e80a8140 con 0x7f01e8061380 2020-11-20T11:35:35.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.940+0000 7f01e7fff700 1 --2- 172.21.15.23:0/2914251297 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01e8061380 0x7f01e80a8880 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f01e80623a0 tx=0x7f01d8006ef0).ready entity=mon.0 client_cookie=561e7429fa407113 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:35.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.940+0000 7f01e6ffd700 1 -- 172.21.15.23:0/2914251297 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d800b820 con 0x7f01e8061380 2020-11-20T11:35:35.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.940+0000 7f01e6ffd700 1 -- 172.21.15.23:0/2914251297 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01d8000e90 con 0x7f01e8061380 2020-11-20T11:35:35.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.940+0000 7f01e6ffd700 1 -- 172.21.15.23:0/2914251297 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d800bb90 con 0x7f01e8061380 2020-11-20T11:35:35.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.940+0000 7f01ed6f2700 1 -- 172.21.15.23:0/2914251297 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01e8061380 msgr2=0x7f01e80a8880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:35.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.940+0000 7f01ed6f2700 1 --2- 172.21.15.23:0/2914251297 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01e8061380 0x7f01e80a8880 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f01e80623a0 tx=0x7f01d8006ef0).stop 2020-11-20T11:35:35.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.941+0000 7f01ed6f2700 1 -- 172.21.15.23:0/2914251297 shutdown_connections 2020-11-20T11:35:35.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.941+0000 7f01ed6f2700 1 --2- 172.21.15.23:0/2914251297 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01e8061380 0x7f01e80a8880 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:35.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.941+0000 7f01ed6f2700 1 --2- 172.21.15.23:0/2914251297 >> v2:172.21.15.155:3300/0 conn(0x7f01e80a8d70 0x7f01e813e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:35.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.941+0000 7f01ed6f2700 1 -- 172.21.15.23:0/2914251297 >> 172.21.15.23:0/2914251297 conn(0x7f01e80ae0a0 msgr2=0x7f01e80aa0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:35.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.941+0000 7f01ed6f2700 1 -- 172.21.15.23:0/2914251297 shutdown_connections 2020-11-20T11:35:35.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.941+0000 7f01ed6f2700 1 -- 172.21.15.23:0/2914251297 wait complete. 2020-11-20T11:35:35.951 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.942+0000 7f01ed6f2700 1 Processor -- start 2020-11-20T11:35:35.951 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.942+0000 7f01ed6f2700 1 -- start start 2020-11-20T11:35:35.951 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.942+0000 7f01ed6f2700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f01e8061380 0x7f01e80a3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:35.951 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.942+0000 7f01ed6f2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01e80a8d70 0x7f01e80670f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:35.952 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.942+0000 7f01ed6f2700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f01e813edc0 0x7f01e80675e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:35.952 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.942+0000 7f01ed6f2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e809d4a0 con 0x7f01e80a8d70 2020-11-20T11:35:35.952 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.942+0000 7f01ed6f2700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e809f150 con 0x7f01e8061380 2020-11-20T11:35:35.952 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.942+0000 7f01ed6f2700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01e809f380 con 0x7f01e813edc0 2020-11-20T11:35:35.952 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.943+0000 7f01e77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01e80a8d70 0x7f01e80670f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:35.953 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.943+0000 7f01e77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01e80a8d70 0x7f01e80670f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42470/0 (socket says 172.21.15.23:42470) 2020-11-20T11:35:35.953 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.943+0000 7f01e77fe700 1 -- 172.21.15.23:0/439283202 learned_addr learned my addr 172.21.15.23:0/439283202 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:35:35.953 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.943+0000 7f01e7fff700 1 --2- 172.21.15.23:0/439283202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f01e8061380 0x7f01e80a3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:35.953 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.943+0000 7f01ecef1700 1 --2- 172.21.15.23:0/439283202 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f01e813edc0 0x7f01e80675e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:35.953 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.943+0000 7f01e77fe700 1 -- 172.21.15.23:0/439283202 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f01e813edc0 msgr2=0x7f01e80675e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:35.954 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.943+0000 7f01e77fe700 1 --2- 172.21.15.23:0/439283202 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f01e813edc0 0x7f01e80675e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:35.954 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.943+0000 7f01e77fe700 1 -- 172.21.15.23:0/439283202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f01e8061380 msgr2=0x7f01e80a3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:35.954 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.943+0000 7f01e77fe700 1 --2- 172.21.15.23:0/439283202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f01e8061380 0x7f01e80a3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:35.954 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.943+0000 7f01e77fe700 1 -- 172.21.15.23:0/439283202 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e8068090 con 0x7f01e80a8d70 2020-11-20T11:35:35.954 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.943+0000 7f01e77fe700 1 --2- 172.21.15.23:0/439283202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01e80a8d70 0x7f01e80670f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f01dc00c900 tx=0x7f01dc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:35.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.943+0000 7f01c7fff700 1 -- 172.21.15.23:0/439283202 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01dc010580 con 0x7f01e80a8d70 2020-11-20T11:35:35.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.943+0000 7f01c7fff700 1 -- 172.21.15.23:0/439283202 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01dc0106e0 con 0x7f01e80a8d70 2020-11-20T11:35:35.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.943+0000 7f01ed6f2700 1 -- 172.21.15.23:0/439283202 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01e809d6d0 con 0x7f01e80a8d70 2020-11-20T11:35:35.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.943+0000 7f01c7fff700 1 -- 172.21.15.23:0/439283202 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01dc010a80 con 0x7f01e80a8d70 2020-11-20T11:35:35.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.943+0000 7f01ed6f2700 1 -- 172.21.15.23:0/439283202 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01e80a4520 con 0x7f01e80a8d70 2020-11-20T11:35:35.956 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.944+0000 7f01ed6f2700 1 -- 172.21.15.23:0/439283202 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01e8069c70 con 0x7f01e80a8d70 2020-11-20T11:35:35.956 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.945+0000 7f01c7fff700 1 -- 172.21.15.23:0/439283202 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f01dc01b460 con 0x7f01e80a8d70 2020-11-20T11:35:35.956 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.945+0000 7f01c7fff700 1 --2- 172.21.15.23:0/439283202 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f01c0056d60 0x7f01c0059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:35.956 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.945+0000 7f01c7fff700 1 -- 172.21.15.23:0/439283202 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f01dc070100 con 0x7f01e80a8d70 2020-11-20T11:35:35.956 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.945+0000 7f01e7fff700 1 --2- 172.21.15.23:0/439283202 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f01c0056d60 0x7f01c0059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:35.957 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.946+0000 7f01e7fff700 1 --2- 172.21.15.23:0/439283202 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f01c0056d60 0x7f01c0059220 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f01d8006f30 tx=0x7f01d8016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:35.957 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:35.951+0000 7f01c7fff700 1 -- 172.21.15.23:0/439283202 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f01dc045390 con 0x7f01e80a8d70 2020-11-20T11:35:36.174 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.171+0000 7f01ed6f2700 1 -- 172.21.15.23:0/439283202 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f01e8057dc0 con 0x7f01e80a8d70 2020-11-20T11:35:36.174 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.171+0000 7f01c7fff700 1 -- 172.21.15.23:0/439283202 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7f01dc0489b0 con 0x7f01e80a8d70 2020-11-20T11:35:36.175 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:35:36.177 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.173+0000 7f01ed6f2700 1 -- 172.21.15.23:0/439283202 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f01c0056d60 msgr2=0x7f01c0059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:36.177 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.173+0000 7f01ed6f2700 1 --2- 172.21.15.23:0/439283202 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f01c0056d60 0x7f01c0059220 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f01d8006f30 tx=0x7f01d8016000).stop 2020-11-20T11:35:36.178 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.173+0000 7f01ed6f2700 1 -- 172.21.15.23:0/439283202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01e80a8d70 msgr2=0x7f01e80670f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:36.178 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.173+0000 7f01ed6f2700 1 --2- 172.21.15.23:0/439283202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01e80a8d70 0x7f01e80670f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f01dc00c900 tx=0x7f01dc008040).stop 2020-11-20T11:35:36.178 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.173+0000 7f01ed6f2700 1 -- 172.21.15.23:0/439283202 shutdown_connections 2020-11-20T11:35:36.179 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.173+0000 7f01ed6f2700 1 --2- 172.21.15.23:0/439283202 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f01c0056d60 0x7f01c0059220 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:36.179 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.173+0000 7f01ed6f2700 1 --2- 172.21.15.23:0/439283202 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f01e813edc0 0x7f01e80675e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:36.179 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.173+0000 7f01ed6f2700 1 --2- 172.21.15.23:0/439283202 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f01e8061380 0x7f01e80a3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:36.179 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.173+0000 7f01ed6f2700 1 --2- 172.21.15.23:0/439283202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01e80a8d70 0x7f01e80670f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:36.179 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.173+0000 7f01ed6f2700 1 -- 172.21.15.23:0/439283202 >> 172.21.15.23:0/439283202 conn(0x7f01e80ae0a0 msgr2=0x7f01e813e210 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:36.180 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.173+0000 7f01ed6f2700 1 -- 172.21.15.23:0/439283202 shutdown_connections 2020-11-20T11:35:36.180 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.173+0000 7f01ed6f2700 1 -- 172.21.15.23:0/439283202 wait complete. 2020-11-20T11:35:36.188 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:35:36.188 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2020-11-20T11:35:36.188 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:35:36.360 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.357+0000 7effefff4700 1 Processor -- start 2020-11-20T11:35:36.361 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.358+0000 7effefff4700 1 -- start start 2020-11-20T11:35:36.361 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.359+0000 7effefff4700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7effe8137cd0 0x7effe81342a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:36.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.359+0000 7effefff4700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe80a9000 con 0x7effe8137870 2020-11-20T11:35:36.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.359+0000 7effefff4700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe8137170 con 0x7effe8068260 2020-11-20T11:35:36.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.359+0000 7effefff4700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effe8134c40 con 0x7effe8137cd0 2020-11-20T11:35:36.364 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.359+0000 7effee591700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7effe8137cd0 0x7effe81342a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:36.364 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.359+0000 7effee591700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7effe8137cd0 0x7effe81342a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42480/0 (socket says 172.21.15.23:42480) 2020-11-20T11:35:36.364 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.359+0000 7effee591700 1 -- 172.21.15.23:0/1136009536 learned_addr learned my addr 172.21.15.23:0/1136009536 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:35:36.364 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.359+0000 7effee591700 1 -- 172.21.15.23:0/1136009536 >> v1:172.21.15.155:6789/0 conn(0x7effe8068260 legacy=0x7effe8065780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:36.364 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.359+0000 7effee591700 1 -- 172.21.15.23:0/1136009536 >> v1:172.21.15.23:6789/0 conn(0x7effe8137870 legacy=0x7effe8133b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:36.365 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.359+0000 7effee591700 1 -- 172.21.15.23:0/1136009536 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe8065e90 con 0x7effe8137cd0 2020-11-20T11:35:36.365 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.360+0000 7effee591700 1 --2- 172.21.15.23:0/1136009536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effe8137cd0 0x7effe81342a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7effd8009a10 tx=0x7effd8006ef0).ready entity=mon.0 client_cookie=e3e851ac42f65ba1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:36.365 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.360+0000 7effecd8e700 1 -- 172.21.15.23:0/1136009536 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd800b860 con 0x7effe8137cd0 2020-11-20T11:35:36.365 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.360+0000 7effecd8e700 1 -- 172.21.15.23:0/1136009536 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd8000e90 con 0x7effe8137cd0 2020-11-20T11:35:36.365 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.360+0000 7effecd8e700 1 -- 172.21.15.23:0/1136009536 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd800bbd0 con 0x7effe8137cd0 2020-11-20T11:35:36.366 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.361+0000 7effefff4700 1 -- 172.21.15.23:0/1136009536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effe8137cd0 msgr2=0x7effe81342a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:36.366 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.361+0000 7effefff4700 1 --2- 172.21.15.23:0/1136009536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effe8137cd0 0x7effe81342a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7effd8009a10 tx=0x7effd8006ef0).stop 2020-11-20T11:35:36.366 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.361+0000 7effefff4700 1 -- 172.21.15.23:0/1136009536 shutdown_connections 2020-11-20T11:35:36.366 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.361+0000 7effefff4700 1 --2- 172.21.15.23:0/1136009536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effe8137cd0 0x7effe81342a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:36.367 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.361+0000 7effefff4700 1 -- 172.21.15.23:0/1136009536 >> 172.21.15.23:0/1136009536 conn(0x7effe80ad7a0 msgr2=0x7effe80adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:36.367 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.361+0000 7effefff4700 1 -- 172.21.15.23:0/1136009536 shutdown_connections 2020-11-20T11:35:36.367 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.361+0000 7effefff4700 1 -- 172.21.15.23:0/1136009536 wait complete. 2020-11-20T11:35:36.369 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.362+0000 7effefff4700 1 Processor -- start 2020-11-20T11:35:36.369 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.362+0000 7effefff4700 1 -- start start 2020-11-20T11:35:36.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.363+0000 7effefff4700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7effe8068260 0x7effe813bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:36.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.363+0000 7effefff4700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effe8137870 0x7effe80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:36.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.363+0000 7effefff4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effe8137cd0 0x7effe8130cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:36.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.363+0000 7effefff4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe80a5610 con 0x7effe8137cd0 2020-11-20T11:35:36.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.363+0000 7effefff4700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe80a9aa0 con 0x7effe8137870 2020-11-20T11:35:36.371 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.363+0000 7effefff4700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effe80a9d40 con 0x7effe8068260 2020-11-20T11:35:36.371 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.363+0000 7effee591700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effe8137cd0 0x7effe8130cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:36.371 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.363+0000 7effee591700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effe8137cd0 0x7effe8130cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42486/0 (socket says 172.21.15.23:42486) 2020-11-20T11:35:36.371 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.363+0000 7effee591700 1 -- 172.21.15.23:0/2002052339 learned_addr learned my addr 172.21.15.23:0/2002052339 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:35:36.372 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.363+0000 7effedd90700 1 --2- 172.21.15.23:0/2002052339 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7effe8068260 0x7effe813bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:36.372 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.363+0000 7effed58f700 1 --2- 172.21.15.23:0/2002052339 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effe8137870 0x7effe80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:36.372 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.363+0000 7effee591700 1 -- 172.21.15.23:0/2002052339 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7effe8068260 msgr2=0x7effe813bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:36.372 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.363+0000 7effee591700 1 --2- 172.21.15.23:0/2002052339 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7effe8068260 0x7effe813bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:36.372 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.363+0000 7effee591700 1 -- 172.21.15.23:0/2002052339 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effe8137870 msgr2=0x7effe80a4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:36.373 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.363+0000 7effee591700 1 --2- 172.21.15.23:0/2002052339 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effe8137870 0x7effe80a4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:36.373 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.363+0000 7effee591700 1 -- 172.21.15.23:0/2002052339 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe80aa1e0 con 0x7effe8137cd0 2020-11-20T11:35:36.373 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.363+0000 7effed58f700 1 --2- 172.21.15.23:0/2002052339 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7effe8137870 0x7effe80a4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:35:36.373 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.364+0000 7effee591700 1 --2- 172.21.15.23:0/2002052339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effe8137cd0 0x7effe8130cd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7effe8137300 tx=0x7effd8017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:36.373 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.364+0000 7effe5ffb700 1 -- 172.21.15.23:0/2002052339 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd8010e70 con 0x7effe8137cd0 2020-11-20T11:35:36.374 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.364+0000 7effe5ffb700 1 -- 172.21.15.23:0/2002052339 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd8018400 con 0x7effe8137cd0 2020-11-20T11:35:36.374 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.364+0000 7effefff4700 1 -- 172.21.15.23:0/2002052339 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe80a52d0 con 0x7effe8137cd0 2020-11-20T11:35:36.374 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.364+0000 7effe5ffb700 1 -- 172.21.15.23:0/2002052339 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd8018780 con 0x7effe8137cd0 2020-11-20T11:35:36.374 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.364+0000 7effefff4700 1 -- 172.21.15.23:0/2002052339 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe813c4f0 con 0x7effe8137cd0 2020-11-20T11:35:36.376 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.365+0000 7effefff4700 1 --2- 172.21.15.23:0/2002052339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effe81311c0 0x7effe812e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:36.376 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.365+0000 7effefff4700 1 -- 172.21.15.23:0/2002052339 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe80a54b0 con 0x7effe81311c0 2020-11-20T11:35:36.377 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.365+0000 7effedd90700 1 --2- 172.21.15.23:0/2002052339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effe81311c0 0x7effe812e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:36.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.365+0000 7effedd90700 1 -- 172.21.15.23:0/2002052339 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7effe80a9f40 con 0x7effe81311c0 2020-11-20T11:35:36.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.365+0000 7effedd90700 1 --2- 172.21.15.23:0/2002052339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effe81311c0 0x7effe812e9e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7effdc00f850 tx=0x7effdc007bd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:36.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.366+0000 7effe5ffb700 1 -- 172.21.15.23:0/2002052339 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7effd802b070 con 0x7effe8137cd0 2020-11-20T11:35:36.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.366+0000 7effe5ffb700 1 --2- 172.21.15.23:0/2002052339 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7effc0056d00 0x7effc00591c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:36.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.366+0000 7effe5ffb700 1 -- 172.21.15.23:0/2002052339 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effdc003040 con 0x7effe81311c0 2020-11-20T11:35:36.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.366+0000 7effe5ffb700 1 -- 172.21.15.23:0/2002052339 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7effd8071670 con 0x7effe8137cd0 2020-11-20T11:35:36.395 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.366+0000 7effed58f700 1 --2- 172.21.15.23:0/2002052339 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7effc0056d00 0x7effc00591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:36.396 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.366+0000 7effe5ffb700 1 -- 172.21.15.23:0/2002052339 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7effdc007030 con 0x7effe81311c0 2020-11-20T11:35:36.397 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.366+0000 7effe5ffb700 1 -- 172.21.15.23:0/2002052339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effe81311c0 msgr2=0x7effe812e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:36.397 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.366+0000 7effe5ffb700 1 --2- 172.21.15.23:0/2002052339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effe81311c0 0x7effe812e9e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7effdc00f850 tx=0x7effdc007bd0).stop 2020-11-20T11:35:36.398 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.366+0000 7effe5ffb700 1 -- 172.21.15.23:0/2002052339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effe81311c0 msgr2=0x7effe812e9e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:35:36.399 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.366+0000 7effe5ffb700 1 --2- 172.21.15.23:0/2002052339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effe81311c0 0x7effe812e9e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:36.399 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.367+0000 7effed58f700 1 --2- 172.21.15.23:0/2002052339 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7effc0056d00 0x7effc00591c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7effe80ab150 tx=0x7effe0009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:36.400 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.374+0000 7effefff4700 1 --2- 172.21.15.23:0/2002052339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effe812eed0 0x7effe8145930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:36.401 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.374+0000 7effefff4700 1 -- 172.21.15.23:0/2002052339 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe8062fe0 con 0x7effe812eed0 2020-11-20T11:35:36.401 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.374+0000 7effedd90700 1 --2- 172.21.15.23:0/2002052339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effe812eed0 0x7effe8145930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:36.403 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.375+0000 7effedd90700 1 -- 172.21.15.23:0/2002052339 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7effe8044710 con 0x7effe812eed0 2020-11-20T11:35:36.403 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.375+0000 7effedd90700 1 --2- 172.21.15.23:0/2002052339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effe812eed0 0x7effe8145930 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7effdc00f850 tx=0x7effdc018000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:36.403 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.375+0000 7effe5ffb700 1 -- 172.21.15.23:0/2002052339 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effdc006a80 con 0x7effe812eed0 2020-11-20T11:35:36.403 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.375+0000 7effe5ffb700 1 -- 172.21.15.23:0/2002052339 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4616 (secure 0 0 0) 0x7effdc00d4f0 con 0x7effe812eed0 2020-11-20T11:35:36.403 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.375+0000 7effe5ffb700 1 -- 172.21.15.23:0/2002052339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effe812eed0 msgr2=0x7effe8145930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:36.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.375+0000 7effe5ffb700 1 --2- 172.21.15.23:0/2002052339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effe812eed0 0x7effe8145930 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7effdc00f850 tx=0x7effdc018000).stop 2020-11-20T11:35:36.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.375+0000 7effe5ffb700 1 -- 172.21.15.23:0/2002052339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effe812eed0 msgr2=0x7effe8145930 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:35:36.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.375+0000 7effe5ffb700 1 --2- 172.21.15.23:0/2002052339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effe812eed0 0x7effe8145930 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:36.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.376+0000 7effefff4700 1 -- 172.21.15.23:0/2002052339 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7effc0056d00 msgr2=0x7effc00591c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:36.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.376+0000 7effefff4700 1 --2- 172.21.15.23:0/2002052339 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7effc0056d00 0x7effc00591c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7effe80ab150 tx=0x7effe0009040).stop 2020-11-20T11:35:36.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.376+0000 7effefff4700 1 -- 172.21.15.23:0/2002052339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effe8137cd0 msgr2=0x7effe8130cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:36.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.376+0000 7effefff4700 1 --2- 172.21.15.23:0/2002052339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effe8137cd0 0x7effe8130cd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7effe8137300 tx=0x7effd8017000).stop 2020-11-20T11:35:36.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.376+0000 7effee591700 1 -- 172.21.15.23:0/2002052339 reap_dead start 2020-11-20T11:35:36.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.376+0000 7effee591700 1 -- 172.21.15.23:0/2002052339 reap_dead start 2020-11-20T11:35:36.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.376+0000 7effefff4700 1 -- 172.21.15.23:0/2002052339 shutdown_connections 2020-11-20T11:35:36.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.376+0000 7effefff4700 1 -- 172.21.15.23:0/2002052339 >> 172.21.15.23:0/2002052339 conn(0x7effe80ad7a0 msgr2=0x7effe8138f30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:36.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.376+0000 7effefff4700 1 -- 172.21.15.23:0/2002052339 shutdown_connections 2020-11-20T11:35:36.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.376+0000 7effefff4700 1 -- 172.21.15.23:0/2002052339 wait complete. 2020-11-20T11:35:36.406 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:35:36.407 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:35:36.407 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:35:36.407 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:35:36.407 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 28, 2020-11-20T11:35:36.407 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:35:36.408 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:35:36.408 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:35:36.408 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:35:36.408 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:36.408 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 15, 2020-11-20T11:35:36.409 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:35:36.409 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:35:36.409 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:35:36.409 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:35:36.410 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:35:36.410 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:35:36.410 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:35:36.410 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:35:36.410 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:35:36.411 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:35:36.411 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:35:36.411 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:36.411 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:35:36.411 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:35:36.412 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:35:36.412 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:35:36.412 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:35:36.412 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:35:36.413 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:35:36.413 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:35:36.413 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:35:36.413 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:35:36.413 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:36.414 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.414 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:35:36.414 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:35:36.414 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:35:36.414 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:35:36.415 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:35:36.415 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:35:36.415 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:35:36.415 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:35:36.415 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:35:36.416 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:35:36.416 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:35:36.416 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:35:36.416 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:35:36.416 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:35:36.417 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:35:36.417 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:35:36.417 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:35:36.417 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:35:36.417 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:35:36.418 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:35:36.418 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:35:36.418 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:35:36.418 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:35:36.418 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:36.419 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:35:36.419 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.419 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:35:36.419 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.420 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:35:36.420 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:35:36.420 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:35:36.420 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:35:36.420 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.420 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:35:36.421 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:35:36.421 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:36.421 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.421 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.422 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:35:36.422 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:35:36.422 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:36.422 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:36.422 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:36.423 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.423 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:35:36.423 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:35:36.423 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:35:36.423 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:35:36.424 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:35:36.424 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.424 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.424 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:35:36.424 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:35:36.425 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:35:36.425 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:35:36.425 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.425 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:35:36.425 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:35:36.426 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:36.426 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.426 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.426 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:35:36.426 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:35:36.427 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:36.427 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:36.427 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:36.427 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.427 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:35:36.428 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:35:36.428 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:35:36.428 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:35:36.428 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:35:36.428 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.429 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.429 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:35:36.429 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:35:36.429 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:35:36.429 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:35:36.430 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.430 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:35:36.430 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:35:36.430 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:36.430 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.431 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.431 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:35:36.431 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:35:36.431 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:36.431 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:36.432 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:36.432 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.432 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:35:36.432 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:35:36.432 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:35:36.433 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:35:36.433 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:35:36.433 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:36.433 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:36.433 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.434 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:35:36.434 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:35:36.434 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.434 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:36.435 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:36.435 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:35:36.435 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:36.435 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:36.436 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:35:36.436 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.436 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:36.436 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:36.436 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:35:36.437 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:36.437 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:36.437 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:35:36.437 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.437 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:36.438 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:36.438 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 4 2020-11-20T11:35:36.438 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:36.438 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:36.438 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:35:36.439 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.439 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:36.439 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:36.439 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:35:36.439 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:36.440 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:36.440 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.440 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:35:36.440 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:35:36.441 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:35:36.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.579+0000 7f5351c45700 1 Processor -- start 2020-11-20T11:35:36.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.579+0000 7f5351c45700 1 -- start start 2020-11-20T11:35:36.585 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.580+0000 7f5351c45700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f534c0a9040 0x7f534c0aaf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:36.585 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.580+0000 7f5351c45700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f534c0ad2f0 0x7f534c13e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:36.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.580+0000 7f5351c45700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f534c0ad860 con 0x7f534c0ace70 2020-11-20T11:35:36.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.580+0000 7f5351c45700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f534c0aa8a0 con 0x7f534c0a9040 2020-11-20T11:35:36.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.580+0000 7f5351c45700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f534c0aaa90 con 0x7f534c0ad2f0 2020-11-20T11:35:36.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.580+0000 7f5350c43700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f534c0a9040 0x7f534c0aaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:36.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.580+0000 7f5351444700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f534c0ad2f0 0x7f534c13e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:36.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.580+0000 7f534bfff700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f534c0ace70 0x7f534c13de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:34720/0 (socket says 172.21.15.23:34720) 2020-11-20T11:35:36.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.580+0000 7f534bfff700 1 -- 172.21.15.23:0/1538005309 learned_addr learned my addr 172.21.15.23:0/1538005309 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:35:36.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.581+0000 7f5350c43700 1 -- 172.21.15.23:0/1538005309 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f534c0ad2f0 msgr2=0x7f534c13e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:35:36.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.581+0000 7f5350c43700 1 --2- 172.21.15.23:0/1538005309 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f534c0ad2f0 0x7f534c13e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:36.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.581+0000 7f5350c43700 1 -- 172.21.15.23:0/1538005309 >> v1:172.21.15.155:6789/0 conn(0x7f534c0ace70 legacy=0x7f534c13de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:36.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.581+0000 7f534b7fe700 1 -- 172.21.15.23:0/1538005309 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1683437791 0 0) 0x7f534c0ad860 con 0x7f534c0ace70 2020-11-20T11:35:36.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.581+0000 7f5350c43700 1 -- 172.21.15.23:0/1538005309 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f534c0a9420 con 0x7f534c0a9040 2020-11-20T11:35:36.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.581+0000 7f5351444700 1 --2- 172.21.15.23:0/1538005309 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f534c0ad2f0 0x7f534c13e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:35:36.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.581+0000 7f5350c43700 1 --2- 172.21.15.23:0/1538005309 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f534c0a9040 0x7f534c0aaf30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f534000b9d0 tx=0x7f5340006dd0).ready entity=mon.1 client_cookie=f953d044f5d154d4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:36.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.581+0000 7f534b7fe700 1 -- 172.21.15.23:0/1538005309 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5340006030 con 0x7f534c0a9040 2020-11-20T11:35:36.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.581+0000 7f534b7fe700 1 -- 172.21.15.23:0/1538005309 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5340006190 con 0x7f534c0a9040 2020-11-20T11:35:36.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.581+0000 7f534b7fe700 1 -- 172.21.15.23:0/1538005309 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5340006510 con 0x7f534c0a9040 2020-11-20T11:35:36.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.582+0000 7f5351c45700 1 -- 172.21.15.23:0/1538005309 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f534c0a9040 msgr2=0x7f534c0aaf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:36.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.582+0000 7f5351c45700 1 --2- 172.21.15.23:0/1538005309 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f534c0a9040 0x7f534c0aaf30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f534000b9d0 tx=0x7f5340006dd0).stop 2020-11-20T11:35:36.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.582+0000 7f5351c45700 1 -- 172.21.15.23:0/1538005309 shutdown_connections 2020-11-20T11:35:36.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.582+0000 7f5351c45700 1 --2- 172.21.15.23:0/1538005309 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f534c0a9040 0x7f534c0aaf30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:36.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.582+0000 7f5351c45700 1 --2- 172.21.15.23:0/1538005309 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f534c0ad2f0 0x7f534c13e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:36.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.582+0000 7f5351c45700 1 -- 172.21.15.23:0/1538005309 >> 172.21.15.23:0/1538005309 conn(0x7f534c0683b0 msgr2=0x7f534c0b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:36.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.582+0000 7f5351c45700 1 -- 172.21.15.23:0/1538005309 shutdown_connections 2020-11-20T11:35:36.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.582+0000 7f5351c45700 1 -- 172.21.15.23:0/1538005309 wait complete. 2020-11-20T11:35:36.594 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.583+0000 7f5351c45700 1 Processor -- start 2020-11-20T11:35:36.594 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.583+0000 7f5351c45700 1 -- start start 2020-11-20T11:35:36.595 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.584+0000 7f5351c45700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f534c0a9040 0x7f534c09fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:36.595 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.584+0000 7f5351c45700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f534c0ace70 0x7f534c13f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:36.595 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.584+0000 7f5351c45700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f534c0ad2f0 0x7f534c136fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:36.595 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.584+0000 7f5351c45700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534c0a0530 con 0x7f534c0ad2f0 2020-11-20T11:35:36.595 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.584+0000 7f5351c45700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534c09e790 con 0x7f534c0a9040 2020-11-20T11:35:36.596 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.584+0000 7f5351c45700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f534c0a00a0 con 0x7f534c0ace70 2020-11-20T11:35:36.596 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.584+0000 7f5351444700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f534c0ad2f0 0x7f534c136fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:36.596 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.584+0000 7f5351444700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f534c0ad2f0 0x7f534c136fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42504/0 (socket says 172.21.15.23:42504) 2020-11-20T11:35:36.596 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.584+0000 7f5351444700 1 -- 172.21.15.23:0/1001032599 learned_addr learned my addr 172.21.15.23:0/1001032599 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:35:36.596 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.584+0000 7f5350c43700 1 --2- 172.21.15.23:0/1001032599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f534c0a9040 0x7f534c09fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:36.597 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.584+0000 7f534bfff700 1 --2- 172.21.15.23:0/1001032599 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f534c0ace70 0x7f534c13f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:36.597 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.584+0000 7f5351444700 1 -- 172.21.15.23:0/1001032599 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f534c0ace70 msgr2=0x7f534c13f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:36.597 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.584+0000 7f5351444700 1 --2- 172.21.15.23:0/1001032599 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f534c0ace70 0x7f534c13f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:36.597 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.584+0000 7f5351444700 1 -- 172.21.15.23:0/1001032599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f534c0a9040 msgr2=0x7f534c09fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:36.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.584+0000 7f5351444700 1 --2- 172.21.15.23:0/1001032599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f534c0a9040 0x7f534c09fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:36.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.584+0000 7f5351444700 1 -- 172.21.15.23:0/1001032599 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f534c09e980 con 0x7f534c0ad2f0 2020-11-20T11:35:36.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.584+0000 7f5351444700 1 --2- 172.21.15.23:0/1001032599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f534c0ad2f0 0x7f534c136fa0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f534400ef50 tx=0x7f5344009220).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:36.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.584+0000 7f5348ff9700 1 -- 172.21.15.23:0/1001032599 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534400cb80 con 0x7f534c0ad2f0 2020-11-20T11:35:36.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.584+0000 7f5348ff9700 1 -- 172.21.15.23:0/1001032599 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f534400cce0 con 0x7f534c0ad2f0 2020-11-20T11:35:36.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.584+0000 7f5348ff9700 1 -- 172.21.15.23:0/1001032599 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5344012620 con 0x7f534c0ad2f0 2020-11-20T11:35:36.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.585+0000 7f5351c45700 1 -- 172.21.15.23:0/1001032599 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f534c09e4c0 con 0x7f534c0ad2f0 2020-11-20T11:35:36.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.585+0000 7f5351c45700 1 -- 172.21.15.23:0/1001032599 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f534c09d4a0 con 0x7f534c0ad2f0 2020-11-20T11:35:36.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.585+0000 7f532e7fc700 1 --2- 172.21.15.23:0/1001032599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f531c002a50 0x7f531c004f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:36.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.585+0000 7f532e7fc700 1 -- 172.21.15.23:0/1001032599 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531c007bb0 con 0x7f531c002a50 2020-11-20T11:35:36.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.586+0000 7f5350c43700 1 --2- 172.21.15.23:0/1001032599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f531c002a50 0x7f531c004f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:36.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.586+0000 7f5348ff9700 1 -- 172.21.15.23:0/1001032599 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5344022070 con 0x7f534c0ad2f0 2020-11-20T11:35:36.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.586+0000 7f5350c43700 1 -- 172.21.15.23:0/1001032599 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f531c007d70 con 0x7f531c002a50 2020-11-20T11:35:36.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.586+0000 7f5350c43700 1 --2- 172.21.15.23:0/1001032599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f531c002a50 0x7f531c004f10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f5340012000 tx=0x7f5340015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:36.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.586+0000 7f5348ff9700 1 --2- 172.21.15.23:0/1001032599 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5324056d60 0x7f5324059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:36.601 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.586+0000 7f5348ff9700 1 -- 172.21.15.23:0/1001032599 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f5344070940 con 0x7f534c0ad2f0 2020-11-20T11:35:36.601 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.586+0000 7f5348ff9700 1 -- 172.21.15.23:0/1001032599 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5340005710 con 0x7f531c002a50 2020-11-20T11:35:36.601 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.586+0000 7f534bfff700 1 --2- 172.21.15.23:0/1001032599 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5324056d60 0x7f5324059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:36.601 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.587+0000 7f5348ff9700 1 -- 172.21.15.23:0/1001032599 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f5340000e90 con 0x7f531c002a50 2020-11-20T11:35:36.601 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.587+0000 7f5348ff9700 1 -- 172.21.15.23:0/1001032599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f531c002a50 msgr2=0x7f531c004f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:36.602 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.587+0000 7f5348ff9700 1 --2- 172.21.15.23:0/1001032599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f531c002a50 0x7f531c004f10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f5340012000 tx=0x7f5340015000).stop 2020-11-20T11:35:36.602 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.587+0000 7f5348ff9700 1 -- 172.21.15.23:0/1001032599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f531c002a50 msgr2=0x7f531c004f10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:35:36.602 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.587+0000 7f5348ff9700 1 --2- 172.21.15.23:0/1001032599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f531c002a50 0x7f531c004f10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:36.602 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.587+0000 7f534bfff700 1 --2- 172.21.15.23:0/1001032599 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5324056d60 0x7f5324059220 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f534c0a7a00 tx=0x7f5338007a40).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:36.604 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.594+0000 7f532e7fc700 1 --2- 172.21.15.23:0/1001032599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f531c006c30 0x7f531c0070a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:36.604 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.594+0000 7f532e7fc700 1 -- 172.21.15.23:0/1001032599 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531c0076b0 con 0x7f531c006c30 2020-11-20T11:35:36.605 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.594+0000 7f5350c43700 1 --2- 172.21.15.23:0/1001032599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f531c006c30 0x7f531c0070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:36.605 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.594+0000 7f5350c43700 1 -- 172.21.15.23:0/1001032599 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f531c007870 con 0x7f531c006c30 2020-11-20T11:35:36.605 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.594+0000 7f5350c43700 1 --2- 172.21.15.23:0/1001032599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f531c006c30 0x7f531c0070a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f5340006e10 tx=0x7f5340021070).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:36.605 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.595+0000 7f5348ff9700 1 -- 172.21.15.23:0/1001032599 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5340032060 con 0x7f531c006c30 2020-11-20T11:35:36.606 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.595+0000 7f5348ff9700 1 -- 172.21.15.23:0/1001032599 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4274 (secure 0 0 0) 0x7f5340028de0 con 0x7f531c006c30 2020-11-20T11:35:36.606 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.595+0000 7f5348ff9700 1 -- 172.21.15.23:0/1001032599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f531c006c30 msgr2=0x7f531c0070a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:36.606 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.595+0000 7f5348ff9700 1 --2- 172.21.15.23:0/1001032599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f531c006c30 0x7f531c0070a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f5340006e10 tx=0x7f5340021070).stop 2020-11-20T11:35:36.606 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.595+0000 7f5348ff9700 1 -- 172.21.15.23:0/1001032599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f531c006c30 msgr2=0x7f531c0070a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:35:36.606 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.595+0000 7f5348ff9700 1 --2- 172.21.15.23:0/1001032599 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f531c006c30 0x7f531c0070a0 secure :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0x7f5340006e10 tx=0x7f5340021070).stop 2020-11-20T11:35:36.607 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.595+0000 7f532e7fc700 1 -- 172.21.15.23:0/1001032599 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5324056d60 msgr2=0x7f5324059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:36.607 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.595+0000 7f532e7fc700 1 --2- 172.21.15.23:0/1001032599 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5324056d60 0x7f5324059220 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f534c0a7a00 tx=0x7f5338007a40).stop 2020-11-20T11:35:36.607 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.595+0000 7f532e7fc700 1 -- 172.21.15.23:0/1001032599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f534c0ad2f0 msgr2=0x7f534c136fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:36.607 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.595+0000 7f532e7fc700 1 --2- 172.21.15.23:0/1001032599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f534c0ad2f0 0x7f534c136fa0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f534400ef50 tx=0x7f5344009220).stop 2020-11-20T11:35:36.607 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.595+0000 7f5351444700 1 -- 172.21.15.23:0/1001032599 reap_dead start 2020-11-20T11:35:36.608 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.595+0000 7f5351444700 1 -- 172.21.15.23:0/1001032599 reap_dead start 2020-11-20T11:35:36.608 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.595+0000 7f532e7fc700 1 -- 172.21.15.23:0/1001032599 shutdown_connections 2020-11-20T11:35:36.608 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.595+0000 7f532e7fc700 1 -- 172.21.15.23:0/1001032599 >> 172.21.15.23:0/1001032599 conn(0x7f534c0683b0 msgr2=0x7f534c0af4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:36.608 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.595+0000 7f532e7fc700 1 -- 172.21.15.23:0/1001032599 shutdown_connections 2020-11-20T11:35:36.608 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.596+0000 7f532e7fc700 1 -- 172.21.15.23:0/1001032599 wait complete. 2020-11-20T11:35:36.609 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:35:36.609 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:35:36.609 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:35:36.609 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:35:36.610 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 28, 2020-11-20T11:35:36.610 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:35:36.610 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:35:36.610 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:35:36.611 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:35:36.611 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:36.611 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 15, 2020-11-20T11:35:36.611 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:35:36.611 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:35:36.612 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:35:36.612 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:35:36.612 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:35:36.612 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:35:36.612 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:35:36.613 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:35:36.613 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:35:36.613 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:35:36.613 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:35:36.613 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:36.614 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:35:36.614 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:35:36.614 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:35:36.614 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:35:36.614 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:35:36.615 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:35:36.615 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:35:36.615 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:35:36.615 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:35:36.615 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:35:36.616 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:36.616 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.616 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:35:36.616 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:35:36.616 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:35:36.617 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:35:36.617 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:35:36.617 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:35:36.617 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:35:36.617 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:35:36.618 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:35:36.618 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:35:36.618 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:35:36.618 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:35:36.618 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:35:36.619 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:35:36.619 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:35:36.619 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:35:36.619 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:35:36.620 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:35:36.620 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:35:36.620 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:35:36.620 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:35:36.620 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:35:36.621 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:35:36.621 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:36.621 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:35:36.621 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.621 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:35:36.622 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.622 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:35:36.622 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:35:36.622 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:35:36.622 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:35:36.623 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.623 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:35:36.623 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:35:36.623 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:36.623 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.624 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.624 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:35:36.624 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:35:36.624 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:36.624 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:36.625 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:36.625 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.625 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:35:36.625 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:35:36.625 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:35:36.626 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:35:36.626 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:35:36.626 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.626 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.626 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:35:36.627 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:35:36.627 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:35:36.627 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:35:36.627 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.628 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:35:36.628 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:35:36.628 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:36.628 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.628 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.629 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:35:36.629 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:35:36.629 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:36.629 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:36.629 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:36.630 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.630 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:35:36.630 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:35:36.630 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:35:36.630 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:35:36.631 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:35:36.631 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.631 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.631 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:35:36.631 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:35:36.632 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:35:36.632 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:35:36.632 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.632 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:35:36.632 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:35:36.633 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:36.633 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.633 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.633 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:35:36.633 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:35:36.634 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:36.634 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:36.634 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:36.634 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.634 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:35:36.635 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:35:36.635 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:35:36.635 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:35:36.635 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:35:36.635 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:36.636 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:36.636 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.636 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:35:36.636 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:35:36.636 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.637 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:36.637 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:36.637 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:35:36.637 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:36.637 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:36.638 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:35:36.638 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.638 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:36.638 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:36.638 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:35:36.639 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:36.639 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:36.639 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.639 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:35:36.639 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:35:36.641 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:35:36.777 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.774+0000 7f9bc30f7700 1 Processor -- start 2020-11-20T11:35:36.777 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.774+0000 7f9bc30f7700 1 -- start start 2020-11-20T11:35:36.779 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.774+0000 7f9bc30f7700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f9bbc0aaab0 0x7f9bbc13ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:36.780 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.774+0000 7f9bc30f7700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f9bbc0aae90 0x7f9bbc1381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:36.780 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.774+0000 7f9bc30f7700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bbc13e600 con 0x7f9bbc13aeb0 2020-11-20T11:35:36.780 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.774+0000 7f9bc30f7700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bbc13e8a0 con 0x7f9bbc0aae90 2020-11-20T11:35:36.780 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.774+0000 7f9bc30f7700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bbc0ab6d0 con 0x7f9bbc0aaab0 2020-11-20T11:35:36.781 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.774+0000 7f9bbbfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f9bbc0aae90 0x7f9bbc1381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:36.781 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.774+0000 7f9bbbfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f9bbc0aae90 0x7f9bbc1381a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42514/0 (socket says 172.21.15.23:42514) 2020-11-20T11:35:36.781 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.775+0000 7f9bbbfff700 1 -- 172.21.15.23:0/1145782081 learned_addr learned my addr 172.21.15.23:0/1145782081 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:35:36.781 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.775+0000 7f9bc0e93700 1 --2- 172.21.15.23:0/1145782081 >> v2:172.21.15.155:3300/0 conn(0x7f9bbc0aaab0 0x7f9bbc13ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:36.782 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.775+0000 7f9bbbfff700 1 -- 172.21.15.23:0/1145782081 >> v2:172.21.15.155:3300/0 conn(0x7f9bbc0aaab0 msgr2=0x7f9bbc13ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:35:36.782 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.775+0000 7f9bbbfff700 1 --2- 172.21.15.23:0/1145782081 >> v2:172.21.15.155:3300/0 conn(0x7f9bbc0aaab0 0x7f9bbc13ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:36.782 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.775+0000 7f9bbbfff700 1 -- 172.21.15.23:0/1145782081 >> v1:172.21.15.155:6789/0 conn(0x7f9bbc13aeb0 legacy=0x7f9bbc0a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:36.782 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.775+0000 7f9bbbfff700 1 -- 172.21.15.23:0/1145782081 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bbc0a9370 con 0x7f9bbc0aae90 2020-11-20T11:35:36.782 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.775+0000 7f9bbbfff700 1 --2- 172.21.15.23:0/1145782081 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9bbc0aae90 0x7f9bbc1381a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f9bbc0b23d0 tx=0x7f9ba8006ef0).ready entity=mon.0 client_cookie=9cc2482e3b1b1548 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:36.783 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.775+0000 7f9bbb7fe700 1 -- 172.21.15.23:0/1145782081 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba800b860 con 0x7f9bbc0aae90 2020-11-20T11:35:36.783 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.775+0000 7f9bbb7fe700 1 -- 172.21.15.23:0/1145782081 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ba8000e90 con 0x7f9bbc0aae90 2020-11-20T11:35:36.783 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.775+0000 7f9bbb7fe700 1 -- 172.21.15.23:0/1145782081 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba800bbd0 con 0x7f9bbc0aae90 2020-11-20T11:35:36.783 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.776+0000 7f9bc30f7700 1 -- 172.21.15.23:0/1145782081 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9bbc0aae90 msgr2=0x7f9bbc1381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:36.783 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.776+0000 7f9bc30f7700 1 --2- 172.21.15.23:0/1145782081 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9bbc0aae90 0x7f9bbc1381a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f9bbc0b23d0 tx=0x7f9ba8006ef0).stop 2020-11-20T11:35:36.784 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.776+0000 7f9bc30f7700 1 -- 172.21.15.23:0/1145782081 shutdown_connections 2020-11-20T11:35:36.784 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.776+0000 7f9bc30f7700 1 --2- 172.21.15.23:0/1145782081 >> v2:172.21.15.155:3300/0 conn(0x7f9bbc0aaab0 0x7f9bbc13ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:36.784 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.776+0000 7f9bc30f7700 1 --2- 172.21.15.23:0/1145782081 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9bbc0aae90 0x7f9bbc1381a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:36.784 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.776+0000 7f9bc30f7700 1 -- 172.21.15.23:0/1145782081 >> 172.21.15.23:0/1145782081 conn(0x7f9bbc0678e0 msgr2=0x7f9bbc067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:36.784 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.776+0000 7f9bc30f7700 1 -- 172.21.15.23:0/1145782081 shutdown_connections 2020-11-20T11:35:36.785 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.776+0000 7f9bc30f7700 1 -- 172.21.15.23:0/1145782081 wait complete. 2020-11-20T11:35:36.785 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.777+0000 7f9bc30f7700 1 Processor -- start 2020-11-20T11:35:36.788 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.777+0000 7f9bc30f7700 1 -- start start 2020-11-20T11:35:36.789 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.778+0000 7f9bc30f7700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9bbc0aaab0 0x7f9bbc13c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:36.789 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.778+0000 7f9bc30f7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9bbc0aae90 0x7f9bbc14a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:36.789 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.778+0000 7f9bc30f7700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9bbc0b03e0 0x7f9bbc0b0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:36.789 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.778+0000 7f9bc30f7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bbc0b0d30 con 0x7f9bbc0aae90 2020-11-20T11:35:36.789 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.778+0000 7f9bc30f7700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bbc140910 con 0x7f9bbc0aaab0 2020-11-20T11:35:36.790 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.778+0000 7f9bc30f7700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bbc140bb0 con 0x7f9bbc0b03e0 2020-11-20T11:35:36.790 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.778+0000 7f9bbbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9bbc0aae90 0x7f9bbc14a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:36.790 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.778+0000 7f9bbbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9bbc0aae90 0x7f9bbc14a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42520/0 (socket says 172.21.15.23:42520) 2020-11-20T11:35:36.790 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.778+0000 7f9bbbfff700 1 -- 172.21.15.23:0/2030808138 learned_addr learned my addr 172.21.15.23:0/2030808138 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:35:36.790 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.778+0000 7f9bc0e93700 1 --2- 172.21.15.23:0/2030808138 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9bbc0aaab0 0x7f9bbc13c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:36.791 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.778+0000 7f9bc1694700 1 --2- 172.21.15.23:0/2030808138 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9bbc0b03e0 0x7f9bbc0b0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:36.791 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.778+0000 7f9bbbfff700 1 -- 172.21.15.23:0/2030808138 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9bbc0b03e0 msgr2=0x7f9bbc0b0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:36.791 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.778+0000 7f9bbbfff700 1 --2- 172.21.15.23:0/2030808138 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9bbc0b03e0 0x7f9bbc0b0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:36.791 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.778+0000 7f9bbbfff700 1 -- 172.21.15.23:0/2030808138 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9bbc0aaab0 msgr2=0x7f9bbc13c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:36.792 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.778+0000 7f9bbbfff700 1 --2- 172.21.15.23:0/2030808138 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9bbc0aaab0 0x7f9bbc13c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:36.792 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.778+0000 7f9bbbfff700 1 -- 172.21.15.23:0/2030808138 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bbc140e50 con 0x7f9bbc0aae90 2020-11-20T11:35:36.792 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.778+0000 7f9bc1694700 1 --2- 172.21.15.23:0/2030808138 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9bbc0b03e0 0x7f9bbc0b0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:35:36.792 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.778+0000 7f9bbbfff700 1 --2- 172.21.15.23:0/2030808138 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9bbc0aae90 0x7f9bbc14a2c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f9ba800c010 tx=0x7f9ba8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:36.792 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.778+0000 7f9bb8ff9700 1 -- 172.21.15.23:0/2030808138 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba8011be0 con 0x7f9bbc0aae90 2020-11-20T11:35:36.793 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.778+0000 7f9bb8ff9700 1 -- 172.21.15.23:0/2030808138 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ba8011d40 con 0x7f9bbc0aae90 2020-11-20T11:35:36.793 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.779+0000 7f9bc30f7700 1 -- 172.21.15.23:0/2030808138 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bbc1412f0 con 0x7f9bbc0aae90 2020-11-20T11:35:36.793 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.779+0000 7f9bb8ff9700 1 -- 172.21.15.23:0/2030808138 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba801d6d0 con 0x7f9bbc0aae90 2020-11-20T11:35:36.793 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.779+0000 7f9bc30f7700 1 -- 172.21.15.23:0/2030808138 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bbc14a970 con 0x7f9bbc0aae90 2020-11-20T11:35:36.793 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.779+0000 7f9bc30f7700 1 --2- 172.21.15.23:0/2030808138 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9bbc13aeb0 0x7f9bbc133680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:36.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.779+0000 7f9bc30f7700 1 -- 172.21.15.23:0/2030808138 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bbc141050 con 0x7f9bbc13aeb0 2020-11-20T11:35:36.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.780+0000 7f9bc0e93700 1 --2- 172.21.15.23:0/2030808138 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9bbc13aeb0 0x7f9bbc133680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:36.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.780+0000 7f9bb8ff9700 1 -- 172.21.15.23:0/2030808138 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9ba8017070 con 0x7f9bbc0aae90 2020-11-20T11:35:36.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.780+0000 7f9bc0e93700 1 -- 172.21.15.23:0/2030808138 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9bbc05c7d0 con 0x7f9bbc13aeb0 2020-11-20T11:35:36.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.780+0000 7f9bc0e93700 1 --2- 172.21.15.23:0/2030808138 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9bbc13aeb0 0x7f9bbc133680 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f9bb0010b00 tx=0x7f9bb0008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:36.795 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.780+0000 7f9bb8ff9700 1 --2- 172.21.15.23:0/2030808138 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9b94056db0 0x7f9b94059270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:36.795 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.781+0000 7f9bb8ff9700 1 -- 172.21.15.23:0/2030808138 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f9ba8071d30 con 0x7f9bbc0aae90 2020-11-20T11:35:36.795 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.781+0000 7f9bb8ff9700 1 -- 172.21.15.23:0/2030808138 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb000ea00 con 0x7f9bbc13aeb0 2020-11-20T11:35:36.795 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.781+0000 7f9bc1694700 1 --2- 172.21.15.23:0/2030808138 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9b94056db0 0x7f9b94059270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:36.795 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.781+0000 7f9bb8ff9700 1 -- 172.21.15.23:0/2030808138 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f9bb000eb60 con 0x7f9bbc13aeb0 2020-11-20T11:35:36.796 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.781+0000 7f9bb8ff9700 1 -- 172.21.15.23:0/2030808138 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9bbc13aeb0 msgr2=0x7f9bbc133680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:36.796 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.781+0000 7f9bb8ff9700 1 --2- 172.21.15.23:0/2030808138 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9bbc13aeb0 0x7f9bbc133680 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f9bb0010b00 tx=0x7f9bb0008040).stop 2020-11-20T11:35:36.796 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.781+0000 7f9bb8ff9700 1 -- 172.21.15.23:0/2030808138 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9bbc13aeb0 msgr2=0x7f9bbc133680 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:35:36.796 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.781+0000 7f9bb8ff9700 1 --2- 172.21.15.23:0/2030808138 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9bbc13aeb0 0x7f9bbc133680 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:36.796 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.781+0000 7f9bc1694700 1 --2- 172.21.15.23:0/2030808138 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9b94056db0 0x7f9b94059270 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f9bb40012e0 tx=0x7f9bb4009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:36.798 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.788+0000 7f9bc30f7700 1 --2- 172.21.15.23:0/2030808138 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9bbc0601b0 0x7f9bbc133b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:36.799 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.788+0000 7f9bc30f7700 1 -- 172.21.15.23:0/2030808138 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bbc044710 con 0x7f9bbc0601b0 2020-11-20T11:35:36.799 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.788+0000 7f9bc0e93700 1 --2- 172.21.15.23:0/2030808138 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9bbc0601b0 0x7f9bbc133b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:36.799 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.789+0000 7f9bc0e93700 1 -- 172.21.15.23:0/2030808138 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9bbc05c280 con 0x7f9bbc0601b0 2020-11-20T11:35:36.800 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.789+0000 7f9bc0e93700 1 --2- 172.21.15.23:0/2030808138 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9bbc0601b0 0x7f9bbc133b70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f9bb000b240 tx=0x7f9bb0008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:36.800 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.789+0000 7f9bb8ff9700 1 -- 172.21.15.23:0/2030808138 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb001d770 con 0x7f9bbc0601b0 2020-11-20T11:35:36.800 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.789+0000 7f9bb8ff9700 1 -- 172.21.15.23:0/2030808138 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4784 (secure 0 0 0) 0x7f9bb0000e90 con 0x7f9bbc0601b0 2020-11-20T11:35:36.800 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.789+0000 7f9bb8ff9700 1 -- 172.21.15.23:0/2030808138 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9bbc0601b0 msgr2=0x7f9bbc133b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:36.801 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.789+0000 7f9bb8ff9700 1 --2- 172.21.15.23:0/2030808138 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9bbc0601b0 0x7f9bbc133b70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f9bb000b240 tx=0x7f9bb0008040).stop 2020-11-20T11:35:36.802 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.789+0000 7f9bb8ff9700 1 -- 172.21.15.23:0/2030808138 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9bbc0601b0 msgr2=0x7f9bbc133b70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:35:36.802 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.789+0000 7f9bb8ff9700 1 --2- 172.21.15.23:0/2030808138 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9bbc0601b0 0x7f9bbc133b70 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:36.802 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.789+0000 7f9bc30f7700 1 -- 172.21.15.23:0/2030808138 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9b94056db0 msgr2=0x7f9b94059270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:36.802 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.789+0000 7f9bc30f7700 1 --2- 172.21.15.23:0/2030808138 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9b94056db0 0x7f9b94059270 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f9bb40012e0 tx=0x7f9bb4009040).stop 2020-11-20T11:35:36.803 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.789+0000 7f9bc30f7700 1 -- 172.21.15.23:0/2030808138 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9bbc0aae90 msgr2=0x7f9bbc14a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:36.803 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.789+0000 7f9bc30f7700 1 --2- 172.21.15.23:0/2030808138 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9bbc0aae90 0x7f9bbc14a2c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f9ba800c010 tx=0x7f9ba8016000).stop 2020-11-20T11:35:36.803 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.789+0000 7f9bc1694700 1 -- 172.21.15.23:0/2030808138 reap_dead start 2020-11-20T11:35:36.803 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.790+0000 7f9bc1694700 1 -- 172.21.15.23:0/2030808138 reap_dead start 2020-11-20T11:35:36.804 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.790+0000 7f9bc30f7700 1 -- 172.21.15.23:0/2030808138 shutdown_connections 2020-11-20T11:35:36.804 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.790+0000 7f9bc30f7700 1 -- 172.21.15.23:0/2030808138 >> 172.21.15.23:0/2030808138 conn(0x7f9bbc0678e0 msgr2=0x7f9bbc139f70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:36.804 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.790+0000 7f9bc30f7700 1 -- 172.21.15.23:0/2030808138 shutdown_connections 2020-11-20T11:35:36.804 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.790+0000 7f9bc30f7700 1 -- 172.21.15.23:0/2030808138 wait complete. 2020-11-20T11:35:36.805 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:35:36.805 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:35:36.805 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:35:36.805 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:35:36.805 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 28, 2020-11-20T11:35:36.806 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:35:36.806 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:35:36.806 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:35:36.806 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:35:36.806 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:36.807 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 15, 2020-11-20T11:35:36.808 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:35:36.808 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:35:36.808 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:35:36.808 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:35:36.809 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:35:36.810 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:35:36.810 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:35:36.810 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:35:36.810 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:35:36.810 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:35:36.811 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:35:36.811 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:36.811 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:35:36.811 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:35:36.811 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:35:36.812 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:35:36.812 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:35:36.812 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:35:36.812 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:35:36.812 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:35:36.813 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:35:36.813 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:35:36.813 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:36.813 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.813 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:35:36.814 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:35:36.814 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:35:36.814 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:35:36.814 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:35:36.814 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:35:36.815 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:35:36.815 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:35:36.815 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:35:36.815 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:35:36.816 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:35:36.816 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:35:36.816 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:35:36.816 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:35:36.816 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:35:36.816 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:35:36.817 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:35:36.817 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:35:36.817 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:35:36.817 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:35:36.818 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:35:36.818 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:35:36.818 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:35:36.818 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:36.819 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:35:36.819 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.819 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:35:36.819 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.819 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:35:36.820 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:35:36.820 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:35:36.820 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:35:36.820 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.820 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:35:36.821 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:35:36.821 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:36.821 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.821 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.821 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:35:36.822 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:35:36.822 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:36.822 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:36.822 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:36.823 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.823 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:35:36.823 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:35:36.823 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:35:36.823 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:35:36.824 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:35:36.824 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.824 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.824 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:35:36.824 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:35:36.825 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:35:36.825 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:35:36.825 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.825 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:35:36.825 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:35:36.826 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:36.826 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.826 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.826 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:35:36.826 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:35:36.827 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:36.827 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:36.827 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:36.827 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.827 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:35:36.828 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:35:36.828 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:35:36.828 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:35:36.828 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:35:36.828 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.829 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.829 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:35:36.829 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:35:36.829 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:35:36.829 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:35:36.830 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.830 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:35:36.830 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:35:36.830 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:36.830 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.831 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.831 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:35:36.831 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:35:36.831 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:36.832 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:36.832 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:36.832 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.832 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:35:36.832 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:35:36.833 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:35:36.833 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:35:36.833 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:35:36.833 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:36.833 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:36.834 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.834 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:35:36.834 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:35:36.834 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.834 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:36.835 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:36.835 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:35:36.835 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:36.835 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:36.835 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:35:36.836 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.836 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:36.836 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:36.836 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:35:36.836 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:36.837 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:36.837 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:35:36.837 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.837 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:36.837 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:36.838 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 6 2020-11-20T11:35:36.838 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:36.838 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:36.838 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:35:36.838 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.839 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:36.839 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:36.839 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:35:36.839 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:36.839 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:36.840 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:35:36.840 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:36.840 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:36.840 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:36.840 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:35:36.841 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:36.841 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:36.841 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:36.841 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:35:36.841 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:35:36.843 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2020-11-20T11:35:36.843 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2020-11-20T11:35:36.843 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2020-11-20T11:35:36.843 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2020-11-20T11:35:36.844 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2020-11-20T11:35:36.844 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T11:35:36.860 INFO:tasks.ceph.mon.c:Stopped 2020-11-20T11:35:36.861 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2020-11-20T11:35:36.862 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:35:36.994 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.991+0000 7f4c7adf5700 1 Processor -- start 2020-11-20T11:35:36.995 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.991+0000 7f4c7adf5700 1 -- start start 2020-11-20T11:35:37.001 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.992+0000 7f4c7adf5700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c74137870 0x7f4c74137cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:37.001 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.992+0000 7f4c7adf5700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c740a9000 con 0x7f4c74133b90 2020-11-20T11:35:37.001 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.992+0000 7f4c7adf5700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c74137170 con 0x7f4c74068260 2020-11-20T11:35:37.002 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.992+0000 7f4c7adf5700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c74134c40 con 0x7f4c74137870 2020-11-20T11:35:37.002 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.992+0000 7f4c73fff700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c74137870 msgr2=0x7f4c74137cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:35:37.002 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.992+0000 7f4c73fff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c74137870 0x7f4c74137cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:35:37.002 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.992+0000 7f4c79392700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f4c74133b90 0x7f4c74134020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:56802/0 (socket says 172.21.15.23:56802) 2020-11-20T11:35:37.002 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.992+0000 7f4c79392700 1 -- 172.21.15.23:0/3865409374 learned_addr learned my addr 172.21.15.23:0/3865409374 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:35:37.003 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.992+0000 7f4c737fe700 1 -- 172.21.15.23:0/3865409374 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212337570 0 0) 0x7f4c740a9000 con 0x7f4c74133b90 2020-11-20T11:35:37.003 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.992+0000 7f4c737fe700 1 -- 172.21.15.23:0/3865409374 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c58003280 con 0x7f4c74133b90 2020-11-20T11:35:37.003 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.992+0000 7f4c737fe700 1 -- 172.21.15.23:0/3865409374 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 132800276 0 0) 0x7f4c74137170 con 0x7f4c74068260 2020-11-20T11:35:37.004 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.992+0000 7f4c737fe700 1 -- 172.21.15.23:0/3865409374 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c740a9000 con 0x7f4c74068260 2020-11-20T11:35:37.004 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.993+0000 7f4c737fe700 1 -- 172.21.15.23:0/3865409374 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 407532015 0 0) 0x7f4c58003280 con 0x7f4c74133b90 2020-11-20T11:35:37.004 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.993+0000 7f4c737fe700 1 -- 172.21.15.23:0/3865409374 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c74137870 msgr2=0x7f4c74137cd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:35:37.004 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.993+0000 7f4c737fe700 1 --2- 172.21.15.23:0/3865409374 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c74137870 0x7f4c74137cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:35:37.004 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.993+0000 7f4c737fe700 1 -- 172.21.15.23:0/3865409374 >> v1:172.21.15.155:6789/0 conn(0x7f4c74068260 legacy=0x7f4c74065780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:37.005 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.993+0000 7f4c737fe700 1 -- 172.21.15.23:0/3865409374 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c74065e90 con 0x7f4c74133b90 2020-11-20T11:35:37.005 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.993+0000 7f4c737fe700 1 -- 172.21.15.23:0/3865409374 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f4c6c0018a0 con 0x7f4c74133b90 2020-11-20T11:35:37.005 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.993+0000 7f4c737fe700 1 -- 172.21.15.23:0/3865409374 >> v1:172.21.15.23:6789/0 conn(0x7f4c74133b90 legacy=0x7f4c74134020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:37.005 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.993+0000 7f4c737fe700 1 --2- 172.21.15.23:0/3865409374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c58003930 0x7f4c58003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:37.006 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.993+0000 7f4c737fe700 1 --2- 172.21.15.23:0/3865409374 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c74137870 0x7f4c58004520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:37.006 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.993+0000 7f4c737fe700 1 --2- 172.21.15.23:0/3865409374 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4c58004a10 0x7f4c58004e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:37.006 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.993+0000 7f4c737fe700 1 -- 172.21.15.23:0/3865409374 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c74134c40 con 0x7f4c58003930 2020-11-20T11:35:37.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.993+0000 7f4c737fe700 1 -- 172.21.15.23:0/3865409374 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c58005380 con 0x7f4c58004a10 2020-11-20T11:35:37.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.993+0000 7f4c737fe700 1 -- 172.21.15.23:0/3865409374 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5800c880 con 0x7f4c74137870 2020-11-20T11:35:37.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.993+0000 7f4c73fff700 1 -- 172.21.15.23:0/3865409374 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c74137870 msgr2=0x7f4c58004520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:35:37.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.993+0000 7f4c73fff700 1 --2- 172.21.15.23:0/3865409374 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c74137870 0x7f4c58004520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:35:37.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.993+0000 7f4c78b91700 1 --2- 172.21.15.23:0/3865409374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c58003930 0x7f4c58003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:37.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.993+0000 7f4c79392700 1 --2- 172.21.15.23:0/3865409374 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4c58004a10 0x7f4c58004e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:37.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.993+0000 7f4c78b91700 1 -- 172.21.15.23:0/3865409374 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c74137870 msgr2=0x7f4c58004520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:35:37.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.993+0000 7f4c78b91700 1 --2- 172.21.15.23:0/3865409374 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c74137870 0x7f4c58004520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:35:37.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.993+0000 7f4c78b91700 1 -- 172.21.15.23:0/3865409374 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4c58004a10 msgr2=0x7f4c58004e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:35:37.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.993+0000 7f4c78b91700 1 --2- 172.21.15.23:0/3865409374 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4c58004a10 0x7f4c58004e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:37.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.993+0000 7f4c78b91700 1 -- 172.21.15.23:0/3865409374 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c58003280 con 0x7f4c58003930 2020-11-20T11:35:37.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.993+0000 7f4c78b91700 1 --2- 172.21.15.23:0/3865409374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c58003930 0x7f4c58003d30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f4c68007e80 tx=0x7f4c68007b90).ready entity=mon.0 client_cookie=fde5186d1fbceeb9 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:37.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.993+0000 7f4c737fe700 1 -- 172.21.15.23:0/3865409374 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6800c7c0 con 0x7f4c58003930 2020-11-20T11:35:37.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.993+0000 7f4c737fe700 1 -- 172.21.15.23:0/3865409374 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c6800c920 con 0x7f4c58003930 2020-11-20T11:35:37.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.993+0000 7f4c737fe700 1 -- 172.21.15.23:0/3865409374 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6800cce0 con 0x7f4c58003930 2020-11-20T11:35:37.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.994+0000 7f4c7adf5700 1 -- 172.21.15.23:0/3865409374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c58003930 msgr2=0x7f4c58003d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:37.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.994+0000 7f4c7adf5700 1 --2- 172.21.15.23:0/3865409374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c58003930 0x7f4c58003d30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f4c68007e80 tx=0x7f4c68007b90).stop 2020-11-20T11:35:37.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.994+0000 7f4c79392700 1 -- 172.21.15.23:0/3865409374 reap_dead start 2020-11-20T11:35:37.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.994+0000 7f4c7adf5700 1 -- 172.21.15.23:0/3865409374 shutdown_connections 2020-11-20T11:35:37.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.994+0000 7f4c7adf5700 1 -- 172.21.15.23:0/3865409374 >> 172.21.15.23:0/3865409374 conn(0x7f4c740ad7a0 msgr2=0x7f4c740adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:37.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.994+0000 7f4c7adf5700 1 -- 172.21.15.23:0/3865409374 shutdown_connections 2020-11-20T11:35:37.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.994+0000 7f4c7adf5700 1 -- 172.21.15.23:0/3865409374 wait complete. 2020-11-20T11:35:37.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.995+0000 7f4c7adf5700 1 Processor -- start 2020-11-20T11:35:37.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.995+0000 7f4c7adf5700 1 -- start start 2020-11-20T11:35:37.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.996+0000 7f4c7adf5700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4c74130bf0 0x7f4c740a9190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:37.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.996+0000 7f4c7adf5700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c74136110 0x7f4c741398b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:37.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.996+0000 7f4c7adf5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c74139da0 0x7f4c7413a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:37.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.996+0000 7f4c7adf5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c740a4470 con 0x7f4c74139da0 2020-11-20T11:35:37.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.996+0000 7f4c7adf5700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7413a6f0 con 0x7f4c74130bf0 2020-11-20T11:35:37.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.996+0000 7f4c7adf5700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c740a9680 con 0x7f4c74136110 2020-11-20T11:35:37.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.996+0000 7f4c73fff700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c74136110 msgr2=0x7f4c741398b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:35:37.014 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.996+0000 7f4c73fff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c74136110 0x7f4c741398b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:35:37.014 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.996+0000 7f4c79392700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c74139da0 0x7f4c7413a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:37.014 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.996+0000 7f4c79392700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c74139da0 0x7f4c7413a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42564/0 (socket says 172.21.15.23:42564) 2020-11-20T11:35:37.014 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.996+0000 7f4c79392700 1 -- 172.21.15.23:0/1702187456 learned_addr learned my addr 172.21.15.23:0/1702187456 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:35:37.014 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.996+0000 7f4c78b91700 1 --2- 172.21.15.23:0/1702187456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4c74130bf0 0x7f4c740a9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:37.015 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.996+0000 7f4c79392700 1 -- 172.21.15.23:0/1702187456 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c74136110 msgr2=0x7f4c741398b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:35:37.015 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.996+0000 7f4c79392700 1 --2- 172.21.15.23:0/1702187456 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c74136110 0x7f4c741398b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:35:37.015 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.996+0000 7f4c79392700 1 -- 172.21.15.23:0/1702187456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4c74130bf0 msgr2=0x7f4c740a9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:37.015 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.996+0000 7f4c79392700 1 --2- 172.21.15.23:0/1702187456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4c74130bf0 0x7f4c740a9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:37.015 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.996+0000 7f4c79392700 1 -- 172.21.15.23:0/1702187456 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c740a46a0 con 0x7f4c74139da0 2020-11-20T11:35:37.016 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.997+0000 7f4c79392700 1 --2- 172.21.15.23:0/1702187456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c74139da0 0x7f4c7413a200 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f4c6c001350 tx=0x7f4c6c007b10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:37.016 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.997+0000 7f4c70ff9700 1 -- 172.21.15.23:0/1702187456 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c00fab0 con 0x7f4c74139da0 2020-11-20T11:35:37.016 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.997+0000 7f4c70ff9700 1 -- 172.21.15.23:0/1702187456 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c6c00fc10 con 0x7f4c74139da0 2020-11-20T11:35:37.017 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.997+0000 7f4c70ff9700 1 -- 172.21.15.23:0/1702187456 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c012350 con 0x7f4c74139da0 2020-11-20T11:35:37.017 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.997+0000 7f4c7adf5700 1 -- 172.21.15.23:0/1702187456 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c74135c70 con 0x7f4c74139da0 2020-11-20T11:35:37.017 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.997+0000 7f4c7adf5700 1 -- 172.21.15.23:0/1702187456 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c74130fd0 con 0x7f4c74139da0 2020-11-20T11:35:37.017 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.998+0000 7f4c7adf5700 1 -- 172.21.15.23:0/1702187456 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c741372f0 con 0x7f4c74139da0 2020-11-20T11:35:37.018 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.998+0000 7f4c70ff9700 1 -- 172.21.15.23:0/1702187456 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4c6c0126c0 con 0x7f4c74139da0 2020-11-20T11:35:37.019 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.998+0000 7f4c70ff9700 1 --2- 172.21.15.23:0/1702187456 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4c4c056d10 0x7f4c4c0591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:37.019 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.998+0000 7f4c70ff9700 1 -- 172.21.15.23:0/1702187456 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f4c6c06f940 con 0x7f4c74139da0 2020-11-20T11:35:37.019 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.999+0000 7f4c78b91700 1 --2- 172.21.15.23:0/1702187456 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4c4c056d10 0x7f4c4c0591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:37.019 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:36.999+0000 7f4c78b91700 1 --2- 172.21.15.23:0/1702187456 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4c4c056d10 0x7f4c4c0591d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f4c74068990 tx=0x7f4c68019000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:37.019 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:37.003+0000 7f4c70ff9700 1 -- 172.21.15.23:0/1702187456 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4c6c044c30 con 0x7f4c74139da0 2020-11-20T11:35:37.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:37.245+0000 7f4c7adf5700 1 -- 172.21.15.23:0/1702187456 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4c74135e00 con 0x7f4c74139da0 2020-11-20T11:35:37.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:37.246+0000 7f4c70ff9700 1 -- 172.21.15.23:0/1702187456 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7f4c6c019080 con 0x7f4c74139da0 2020-11-20T11:35:37.250 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:35:37.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:37.247+0000 7f4c7adf5700 1 -- 172.21.15.23:0/1702187456 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4c4c056d10 msgr2=0x7f4c4c0591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:37.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:37.247+0000 7f4c7adf5700 1 --2- 172.21.15.23:0/1702187456 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4c4c056d10 0x7f4c4c0591d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f4c74068990 tx=0x7f4c68019000).stop 2020-11-20T11:35:37.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:37.247+0000 7f4c7adf5700 1 -- 172.21.15.23:0/1702187456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c74139da0 msgr2=0x7f4c7413a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:37.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:37.247+0000 7f4c7adf5700 1 --2- 172.21.15.23:0/1702187456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c74139da0 0x7f4c7413a200 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f4c6c001350 tx=0x7f4c6c007b10).stop 2020-11-20T11:35:37.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:37.247+0000 7f4c7adf5700 1 -- 172.21.15.23:0/1702187456 shutdown_connections 2020-11-20T11:35:37.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:37.247+0000 7f4c7adf5700 1 --2- 172.21.15.23:0/1702187456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c74139da0 0x7f4c7413a200 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:37.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:37.247+0000 7f4c7adf5700 1 --2- 172.21.15.23:0/1702187456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4c74130bf0 0x7f4c740a9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:37.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:37.247+0000 7f4c7adf5700 1 --2- 172.21.15.23:0/1702187456 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4c4c056d10 0x7f4c4c0591d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:37.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:37.247+0000 7f4c7adf5700 1 --2- 172.21.15.23:0/1702187456 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c74136110 0x7f4c741398b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:35:37.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:37.247+0000 7f4c7adf5700 1 -- 172.21.15.23:0/1702187456 >> 172.21.15.23:0/1702187456 conn(0x7f4c740ad7a0 msgr2=0x7f4c74065760 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:37.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:37.247+0000 7f4c7adf5700 1 -- 172.21.15.23:0/1702187456 shutdown_connections 2020-11-20T11:35:37.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:37.247+0000 7f4c7adf5700 1 -- 172.21.15.23:0/1702187456 wait complete. 2020-11-20T11:35:40.265 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:35:40.399 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.396+0000 7f796bc49700 1 Processor -- start 2020-11-20T11:35:40.399 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.396+0000 7f796bc49700 1 -- start start 2020-11-20T11:35:40.401 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.397+0000 7f796bc49700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f79640aaab0 0x7f796413ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:40.401 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.397+0000 7f796bc49700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f796413aeb0 0x7f79640a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:40.402 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.397+0000 7f796bc49700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79641388b0 con 0x7f79640aae90 2020-11-20T11:35:40.402 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.397+0000 7f796bc49700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f796413e600 con 0x7f796413aeb0 2020-11-20T11:35:40.402 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.397+0000 7f796bc49700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f796413e8a0 con 0x7f79640aaab0 2020-11-20T11:35:40.402 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.397+0000 7f796a1e6700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f796413aeb0 0x7f79640a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:40.403 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.397+0000 7f796a1e6700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f796413aeb0 0x7f79640a8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42582/0 (socket says 172.21.15.23:42582) 2020-11-20T11:35:40.403 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.397+0000 7f796a1e6700 1 -- 172.21.15.23:0/2152196993 learned_addr learned my addr 172.21.15.23:0/2152196993 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:35:40.403 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.397+0000 7f79691e4700 1 --1- 172.21.15.23:0/2152196993 >> v1:172.21.15.23:6789/0 conn(0x7f79640aae90 0x7f79641381a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:56828/0 (socket says 172.21.15.23:56828) 2020-11-20T11:35:40.403 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.397+0000 7f79699e5700 1 --2- 172.21.15.23:0/2152196993 >> v2:172.21.15.155:3300/0 conn(0x7f79640aaab0 0x7f796413ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:40.403 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.397+0000 7f796a1e6700 1 -- 172.21.15.23:0/2152196993 >> v2:172.21.15.155:3300/0 conn(0x7f79640aaab0 msgr2=0x7f796413ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:35:40.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.397+0000 7f796a1e6700 1 --2- 172.21.15.23:0/2152196993 >> v2:172.21.15.155:3300/0 conn(0x7f79640aaab0 0x7f796413ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:40.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.397+0000 7f796a1e6700 1 -- 172.21.15.23:0/2152196993 >> v1:172.21.15.23:6789/0 conn(0x7f79640aae90 legacy=0x7f79641381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:40.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.397+0000 7f796a1e6700 1 -- 172.21.15.23:0/2152196993 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79640ab6d0 con 0x7f796413aeb0 2020-11-20T11:35:40.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.397+0000 7f796a1e6700 1 --2- 172.21.15.23:0/2152196993 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f796413aeb0 0x7f79640a8c60 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f79640b23d0 tx=0x7f795c0092d0).ready entity=mon.0 client_cookie=3ff6b09eb9b12fdf server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:40.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.398+0000 7f79689e3700 1 -- 172.21.15.23:0/2152196993 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795c00bbf0 con 0x7f796413aeb0 2020-11-20T11:35:40.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.398+0000 7f79689e3700 1 -- 172.21.15.23:0/2152196993 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f795c00bd50 con 0x7f796413aeb0 2020-11-20T11:35:40.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.398+0000 7f79689e3700 1 -- 172.21.15.23:0/2152196993 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795c011610 con 0x7f796413aeb0 2020-11-20T11:35:40.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.398+0000 7f796bc49700 1 -- 172.21.15.23:0/2152196993 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f796413aeb0 msgr2=0x7f79640a8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:40.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.398+0000 7f796bc49700 1 --2- 172.21.15.23:0/2152196993 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f796413aeb0 0x7f79640a8c60 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f79640b23d0 tx=0x7f795c0092d0).stop 2020-11-20T11:35:40.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.398+0000 7f796bc49700 1 -- 172.21.15.23:0/2152196993 shutdown_connections 2020-11-20T11:35:40.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.398+0000 7f796bc49700 1 --2- 172.21.15.23:0/2152196993 >> v2:172.21.15.155:3300/0 conn(0x7f79640aaab0 0x7f796413ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:40.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.398+0000 7f796bc49700 1 --2- 172.21.15.23:0/2152196993 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f796413aeb0 0x7f79640a8c60 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:40.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.398+0000 7f796bc49700 1 -- 172.21.15.23:0/2152196993 >> 172.21.15.23:0/2152196993 conn(0x7f79640678e0 msgr2=0x7f7964067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:40.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.398+0000 7f796bc49700 1 -- 172.21.15.23:0/2152196993 shutdown_connections 2020-11-20T11:35:40.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.398+0000 7f796bc49700 1 -- 172.21.15.23:0/2152196993 wait complete. 2020-11-20T11:35:40.409 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.399+0000 7f796bc49700 1 Processor -- start 2020-11-20T11:35:40.409 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.399+0000 7f796bc49700 1 -- start start 2020-11-20T11:35:40.409 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.400+0000 7f796bc49700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79640aaab0 0x7f796413ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:40.410 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.400+0000 7f796bc49700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79640aae90 0x7f7964140600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:40.410 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.400+0000 7f796bc49700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79640a53f0 0x7f79641354d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:40.410 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.400+0000 7f796bc49700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79640a7830 con 0x7f79640aae90 2020-11-20T11:35:40.410 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.400+0000 7f796bc49700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79640a7ad0 con 0x7f79640a53f0 2020-11-20T11:35:40.410 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.400+0000 7f796bc49700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79640a5da0 con 0x7f79640aaab0 2020-11-20T11:35:40.411 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.400+0000 7f79699e5700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79640aaab0 msgr2=0x7f796413ce90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:35:40.411 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.400+0000 7f79699e5700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79640aaab0 0x7f796413ce90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:35:40.411 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.400+0000 7f79691e4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79640aae90 0x7f7964140600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:40.411 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.400+0000 7f79691e4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79640aae90 0x7f7964140600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42586/0 (socket says 172.21.15.23:42586) 2020-11-20T11:35:40.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.400+0000 7f79691e4700 1 -- 172.21.15.23:0/1695040494 learned_addr learned my addr 172.21.15.23:0/1695040494 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:35:40.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.400+0000 7f796a1e6700 1 --2- 172.21.15.23:0/1695040494 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79640a53f0 0x7f79641354d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:40.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.400+0000 7f79691e4700 1 -- 172.21.15.23:0/1695040494 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79640aaab0 msgr2=0x7f796413ce90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:35:40.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.400+0000 7f79691e4700 1 --2- 172.21.15.23:0/1695040494 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79640aaab0 0x7f796413ce90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:35:40.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.400+0000 7f79691e4700 1 -- 172.21.15.23:0/1695040494 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79640a53f0 msgr2=0x7f79641354d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:40.413 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.400+0000 7f79691e4700 1 --2- 172.21.15.23:0/1695040494 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79640a53f0 0x7f79641354d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:40.413 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.400+0000 7f79691e4700 1 -- 172.21.15.23:0/1695040494 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79640a8290 con 0x7f79640aae90 2020-11-20T11:35:40.413 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.401+0000 7f79691e4700 1 --2- 172.21.15.23:0/1695040494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79640aae90 0x7f7964140600 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f795400c710 tx=0x7f7954007c80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:40.413 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.401+0000 7f7951ffb700 1 -- 172.21.15.23:0/1695040494 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79540072e0 con 0x7f79640aae90 2020-11-20T11:35:40.413 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.401+0000 7f7951ffb700 1 -- 172.21.15.23:0/1695040494 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7954007440 con 0x7f79640aae90 2020-11-20T11:35:40.414 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.401+0000 7f7951ffb700 1 -- 172.21.15.23:0/1695040494 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7954005780 con 0x7f79640aae90 2020-11-20T11:35:40.414 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.401+0000 7f796bc49700 1 -- 172.21.15.23:0/1695040494 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79640a8540 con 0x7f79640aae90 2020-11-20T11:35:40.414 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.401+0000 7f796bc49700 1 -- 172.21.15.23:0/1695040494 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79640a74b0 con 0x7f79640aae90 2020-11-20T11:35:40.414 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.402+0000 7f796bc49700 1 -- 172.21.15.23:0/1695040494 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79640a76a0 con 0x7f79640aae90 2020-11-20T11:35:40.414 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.402+0000 7f7951ffb700 1 -- 172.21.15.23:0/1695040494 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7954010070 con 0x7f79640aae90 2020-11-20T11:35:40.415 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.402+0000 7f7951ffb700 1 --2- 172.21.15.23:0/1695040494 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f793c056d10 0x7f793c0591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:40.415 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.403+0000 7f7951ffb700 1 -- 172.21.15.23:0/1695040494 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f795406f750 con 0x7f79640aae90 2020-11-20T11:35:40.415 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.403+0000 7f79699e5700 1 --2- 172.21.15.23:0/1695040494 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f793c056d10 0x7f793c0591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:40.415 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.403+0000 7f79699e5700 1 --2- 172.21.15.23:0/1695040494 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f793c056d10 0x7f793c0591d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f7960006fd0 tx=0x7f7960006e30).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:40.416 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.406+0000 7f7951ffb700 1 -- 172.21.15.23:0/1695040494 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7954044a40 con 0x7f79640aae90 2020-11-20T11:35:40.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.652+0000 7f796bc49700 1 -- 172.21.15.23:0/1695040494 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f79640a5f60 con 0x7f79640aae90 2020-11-20T11:35:40.656 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.653+0000 7f7951ffb700 1 -- 172.21.15.23:0/1695040494 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7f79640a5f60 con 0x7f79640aae90 2020-11-20T11:35:40.656 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:35:40.657 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.654+0000 7f796bc49700 1 -- 172.21.15.23:0/1695040494 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f793c056d10 msgr2=0x7f793c0591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:40.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.654+0000 7f796bc49700 1 --2- 172.21.15.23:0/1695040494 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f793c056d10 0x7f793c0591d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f7960006fd0 tx=0x7f7960006e30).stop 2020-11-20T11:35:40.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.654+0000 7f796bc49700 1 -- 172.21.15.23:0/1695040494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79640aae90 msgr2=0x7f7964140600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:40.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.654+0000 7f796bc49700 1 --2- 172.21.15.23:0/1695040494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79640aae90 0x7f7964140600 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f795400c710 tx=0x7f7954007c80).stop 2020-11-20T11:35:40.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.654+0000 7f796bc49700 1 -- 172.21.15.23:0/1695040494 shutdown_connections 2020-11-20T11:35:40.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.654+0000 7f796bc49700 1 --2- 172.21.15.23:0/1695040494 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79640a53f0 0x7f79641354d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:40.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.654+0000 7f796bc49700 1 --2- 172.21.15.23:0/1695040494 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f793c056d10 0x7f793c0591d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:40.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.654+0000 7f796bc49700 1 --2- 172.21.15.23:0/1695040494 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79640aaab0 0x7f796413ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:35:40.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.654+0000 7f796bc49700 1 --2- 172.21.15.23:0/1695040494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79640aae90 0x7f7964140600 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:40.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.654+0000 7f796bc49700 1 -- 172.21.15.23:0/1695040494 >> 172.21.15.23:0/1695040494 conn(0x7f79640678e0 msgr2=0x7f79640b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:40.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.655+0000 7f796bc49700 1 -- 172.21.15.23:0/1695040494 shutdown_connections 2020-11-20T11:35:40.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:40.655+0000 7f796bc49700 1 -- 172.21.15.23:0/1695040494 wait complete. 2020-11-20T11:35:43.670 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:35:43.801 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.798+0000 7f8d66460700 1 Processor -- start 2020-11-20T11:35:43.802 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.799+0000 7f8d66460700 1 -- start start 2020-11-20T11:35:43.803 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.799+0000 7f8d66460700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f8d600665f0 0x7f8d6013adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:43.803 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.799+0000 7f8d66460700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d6013bd10 con 0x7f8d600ab2a0 2020-11-20T11:35:43.803 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.799+0000 7f8d66460700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d600aa930 con 0x7f8d60066210 2020-11-20T11:35:43.803 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.799+0000 7f8d66460700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d600ab700 con 0x7f8d600665f0 2020-11-20T11:35:43.804 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.799+0000 7f8d65c5f700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f8d600ab2a0 0x7f8d60137f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:56848/0 (socket says 172.21.15.23:56848) 2020-11-20T11:35:43.804 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.800+0000 7f8d65c5f700 1 -- 172.21.15.23:0/4028920016 learned_addr learned my addr 172.21.15.23:0/4028920016 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:35:43.804 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.800+0000 7f8d64c5d700 1 --2- 172.21.15.23:0/4028920016 >> v2:172.21.15.155:3300/0 conn(0x7f8d600665f0 0x7f8d6013adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:43.804 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.800+0000 7f8d64c5d700 1 -- 172.21.15.23:0/4028920016 >> v1:172.21.15.155:6789/0 conn(0x7f8d60066210 legacy=0x7f8d600a8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:43.805 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.800+0000 7f8d64c5d700 1 -- 172.21.15.23:0/4028920016 >> v1:172.21.15.23:6789/0 conn(0x7f8d600ab2a0 legacy=0x7f8d60137f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:43.805 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.800+0000 7f8d64c5d700 1 -- 172.21.15.23:0/4028920016 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d600a9700 con 0x7f8d600665f0 2020-11-20T11:35:43.808 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.800+0000 7f8d64c5d700 1 --2- 172.21.15.23:0/4028920016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d600665f0 0x7f8d6013adb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f8d50006fd0 tx=0x7f8d50006e30).ready entity=mon.1 client_cookie=4fef9478b43becec server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:43.808 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.800+0000 7f8d4ffff700 1 -- 172.21.15.23:0/4028920016 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d500097e0 con 0x7f8d600665f0 2020-11-20T11:35:43.808 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.801+0000 7f8d4ffff700 1 -- 172.21.15.23:0/4028920016 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d50000e90 con 0x7f8d600665f0 2020-11-20T11:35:43.808 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.801+0000 7f8d4ffff700 1 -- 172.21.15.23:0/4028920016 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d50009b50 con 0x7f8d600665f0 2020-11-20T11:35:43.809 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.801+0000 7f8d66460700 1 -- 172.21.15.23:0/4028920016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d600665f0 msgr2=0x7f8d6013adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:43.809 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.801+0000 7f8d66460700 1 --2- 172.21.15.23:0/4028920016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d600665f0 0x7f8d6013adb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f8d50006fd0 tx=0x7f8d50006e30).stop 2020-11-20T11:35:43.809 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.801+0000 7f8d66460700 1 -- 172.21.15.23:0/4028920016 shutdown_connections 2020-11-20T11:35:43.810 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.801+0000 7f8d66460700 1 --2- 172.21.15.23:0/4028920016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d600665f0 0x7f8d6013adb0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:43.810 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.801+0000 7f8d66460700 1 -- 172.21.15.23:0/4028920016 >> 172.21.15.23:0/4028920016 conn(0x7f8d600612c0 msgr2=0x7f8d600616d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:43.810 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.801+0000 7f8d66460700 1 -- 172.21.15.23:0/4028920016 shutdown_connections 2020-11-20T11:35:43.810 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.801+0000 7f8d66460700 1 -- 172.21.15.23:0/4028920016 wait complete. 2020-11-20T11:35:43.810 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.802+0000 7f8d66460700 1 Processor -- start 2020-11-20T11:35:43.811 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.802+0000 7f8d66460700 1 -- start start 2020-11-20T11:35:43.811 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.803+0000 7f8d66460700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60066210 0x7f8d60131f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:43.811 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.803+0000 7f8d66460700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d600665f0 0x7f8d600a7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:43.811 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.803+0000 7f8d66460700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8d600ab2a0 0x7f8d6014e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:43.811 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.803+0000 7f8d66460700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d600a7fa0 con 0x7f8d60066210 2020-11-20T11:35:43.812 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.803+0000 7f8d66460700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6014dae0 con 0x7f8d600665f0 2020-11-20T11:35:43.812 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.803+0000 7f8d66460700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6014dd80 con 0x7f8d600ab2a0 2020-11-20T11:35:43.812 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.803+0000 7f8d6545e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60066210 0x7f8d60131f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:43.812 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.803+0000 7f8d6545e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60066210 0x7f8d60131f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42600/0 (socket says 172.21.15.23:42600) 2020-11-20T11:35:43.812 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.803+0000 7f8d6545e700 1 -- 172.21.15.23:0/1176485313 learned_addr learned my addr 172.21.15.23:0/1176485313 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:35:43.813 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.803+0000 7f8d65c5f700 1 -- 172.21.15.23:0/1176485313 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8d600ab2a0 msgr2=0x7f8d6014e220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:35:43.813 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.803+0000 7f8d65c5f700 1 --2- 172.21.15.23:0/1176485313 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8d600ab2a0 0x7f8d6014e220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:35:43.813 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.804+0000 7f8d64c5d700 1 --2- 172.21.15.23:0/1176485313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d600665f0 0x7f8d600a7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:43.813 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.804+0000 7f8d6545e700 1 -- 172.21.15.23:0/1176485313 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8d600ab2a0 msgr2=0x7f8d6014e220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:35:43.814 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.804+0000 7f8d6545e700 1 --2- 172.21.15.23:0/1176485313 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8d600ab2a0 0x7f8d6014e220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:35:43.814 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.804+0000 7f8d6545e700 1 -- 172.21.15.23:0/1176485313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d600665f0 msgr2=0x7f8d600a7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:43.814 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.804+0000 7f8d6545e700 1 --2- 172.21.15.23:0/1176485313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d600665f0 0x7f8d600a7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:43.814 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.804+0000 7f8d6545e700 1 -- 172.21.15.23:0/1176485313 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d600a7c60 con 0x7f8d60066210 2020-11-20T11:35:43.814 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.804+0000 7f8d64c5d700 1 --2- 172.21.15.23:0/1176485313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d600665f0 0x7f8d600a7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:35:43.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.804+0000 7f8d6545e700 1 --2- 172.21.15.23:0/1176485313 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60066210 0x7f8d60131f30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f8d5c00c710 tx=0x7f8d5c007c90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:43.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.804+0000 7f8d4d7fa700 1 -- 172.21.15.23:0/1176485313 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c007020 con 0x7f8d60066210 2020-11-20T11:35:43.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.804+0000 7f8d4d7fa700 1 -- 172.21.15.23:0/1176485313 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d5c001e90 con 0x7f8d60066210 2020-11-20T11:35:43.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.804+0000 7f8d4d7fa700 1 -- 172.21.15.23:0/1176485313 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c0073b0 con 0x7f8d60066210 2020-11-20T11:35:43.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.804+0000 7f8d66460700 1 -- 172.21.15.23:0/1176485313 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d6014a440 con 0x7f8d60066210 2020-11-20T11:35:43.816 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.804+0000 7f8d66460700 1 -- 172.21.15.23:0/1176485313 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d60132870 con 0x7f8d60066210 2020-11-20T11:35:43.816 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.805+0000 7f8d66460700 1 -- 172.21.15.23:0/1176485313 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d60062760 con 0x7f8d60066210 2020-11-20T11:35:43.817 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.806+0000 7f8d4d7fa700 1 -- 172.21.15.23:0/1176485313 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8d5c010070 con 0x7f8d60066210 2020-11-20T11:35:43.817 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.806+0000 7f8d4d7fa700 1 --2- 172.21.15.23:0/1176485313 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8d38056d10 0x7f8d380591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:43.817 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.806+0000 7f8d4d7fa700 1 -- 172.21.15.23:0/1176485313 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f8d5c06f410 con 0x7f8d60066210 2020-11-20T11:35:43.817 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.806+0000 7f8d64c5d700 1 --2- 172.21.15.23:0/1176485313 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8d38056d10 0x7f8d380591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:43.818 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.807+0000 7f8d64c5d700 1 --2- 172.21.15.23:0/1176485313 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8d38056d10 0x7f8d380591d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f8d50006e70 tx=0x7f8d50015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:43.818 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:43.810+0000 7f8d4d7fa700 1 -- 172.21.15.23:0/1176485313 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8d5c044700 con 0x7f8d60066210 2020-11-20T11:35:44.030 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:44.027+0000 7f8d66460700 1 -- 172.21.15.23:0/1176485313 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8d60057dc0 con 0x7f8d60066210 2020-11-20T11:35:44.031 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:44.028+0000 7f8d4d7fa700 1 -- 172.21.15.23:0/1176485313 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7f8d5c047d20 con 0x7f8d60066210 2020-11-20T11:35:44.031 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":23,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:35:44.034 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:44.030+0000 7f8d66460700 1 -- 172.21.15.23:0/1176485313 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8d38056d10 msgr2=0x7f8d380591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:44.034 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:44.030+0000 7f8d66460700 1 --2- 172.21.15.23:0/1176485313 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8d38056d10 0x7f8d380591d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f8d50006e70 tx=0x7f8d50015000).stop 2020-11-20T11:35:44.034 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:44.030+0000 7f8d66460700 1 -- 172.21.15.23:0/1176485313 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60066210 msgr2=0x7f8d60131f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:44.035 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:44.030+0000 7f8d66460700 1 --2- 172.21.15.23:0/1176485313 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60066210 0x7f8d60131f30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f8d5c00c710 tx=0x7f8d5c007c90).stop 2020-11-20T11:35:44.035 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:44.031+0000 7f8d66460700 1 -- 172.21.15.23:0/1176485313 shutdown_connections 2020-11-20T11:35:44.036 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:44.031+0000 7f8d66460700 1 --2- 172.21.15.23:0/1176485313 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8d38056d10 0x7f8d380591d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:44.036 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:44.031+0000 7f8d66460700 1 --2- 172.21.15.23:0/1176485313 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8d600ab2a0 0x7f8d6014e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:35:44.036 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:44.031+0000 7f8d66460700 1 --2- 172.21.15.23:0/1176485313 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60066210 0x7f8d60131f30 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:44.037 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:44.031+0000 7f8d66460700 1 --2- 172.21.15.23:0/1176485313 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8d600665f0 0x7f8d600a7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:44.037 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:44.031+0000 7f8d66460700 1 -- 172.21.15.23:0/1176485313 >> 172.21.15.23:0/1176485313 conn(0x7f8d600612c0 msgr2=0x7f8d600a8e90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:44.037 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:44.031+0000 7f8d66460700 1 -- 172.21.15.23:0/1176485313 shutdown_connections 2020-11-20T11:35:44.038 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:44.031+0000 7f8d66460700 1 -- 172.21.15.23:0/1176485313 wait complete. 2020-11-20T11:35:47.051 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:35:47.184 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:47.180+0000 7fa9e114a700 1 Processor -- start 2020-11-20T11:35:47.184 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:47.180+0000 7fa9e114a700 1 -- start start 2020-11-20T11:35:47.185 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:47.180+0000 7fa9e114a700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa9dc0aaab0 0x7fa9dc13ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:47.185 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:47.181+0000 7fa9e114a700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9dc1388b0 con 0x7fa9dc13aeb0 2020-11-20T11:35:47.185 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:47.181+0000 7fa9e114a700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9dc13e600 con 0x7fa9dc0aae90 2020-11-20T11:35:47.185 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:47.181+0000 7fa9e114a700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9dc13e8a0 con 0x7fa9dc0aaab0 2020-11-20T11:35:47.186 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:47.181+0000 7fa9db59e700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fa9dc13aeb0 0x7fa9dc0a8c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:56862/0 (socket says 172.21.15.23:56862) 2020-11-20T11:35:47.186 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:47.181+0000 7fa9db59e700 1 -- 172.21.15.23:0/3730962180 learned_addr learned my addr 172.21.15.23:0/3730962180 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:35:47.186 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:47.181+0000 7fa9dad9d700 1 --2- 172.21.15.23:0/3730962180 >> v2:172.21.15.155:3300/0 conn(0x7fa9dc0aaab0 0x7fa9dc13ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:47.187 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:47.181+0000 7fa9dad9d700 1 -- 172.21.15.23:0/3730962180 >> v2:172.21.15.155:3300/0 conn(0x7fa9dc0aaab0 msgr2=0x7fa9dc13ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:35:47.187 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:47.181+0000 7fa9dad9d700 1 -- 172.21.15.23:0/3730962180 >> v2:172.21.15.155:3300/0 conn(0x7fa9dc0aaab0 msgr2=0x7fa9dc13ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:35:47.187 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:47.181+0000 7fa9dad9d700 1 --2- 172.21.15.23:0/3730962180 >> v2:172.21.15.155:3300/0 conn(0x7fa9dc0aaab0 0x7fa9dc13ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:35:47.187 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:47.181+0000 7fa9dad9d700 1 --2- 172.21.15.23:0/3730962180 >> v2:172.21.15.155:3300/0 conn(0x7fa9dc0aaab0 0x7fa9dc13ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:35:47.385 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:47.382+0000 7fa9dad9d700 1 --2- 172.21.15.23:0/3730962180 >> v2:172.21.15.155:3300/0 conn(0x7fa9dc0aaab0 0x7fa9dc13ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:47.386 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:47.382+0000 7fa9dad9d700 1 -- 172.21.15.23:0/3730962180 >> v2:172.21.15.155:3300/0 conn(0x7fa9dc0aaab0 msgr2=0x7fa9dc13ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:35:47.386 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:47.382+0000 7fa9dad9d700 1 -- 172.21.15.23:0/3730962180 >> v2:172.21.15.155:3300/0 conn(0x7fa9dc0aaab0 msgr2=0x7fa9dc13ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:35:47.386 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:47.382+0000 7fa9dad9d700 1 --2- 172.21.15.23:0/3730962180 >> v2:172.21.15.155:3300/0 conn(0x7fa9dc0aaab0 0x7fa9dc13ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:35:47.387 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:47.382+0000 7fa9dad9d700 1 --2- 172.21.15.23:0/3730962180 >> v2:172.21.15.155:3300/0 conn(0x7fa9dc0aaab0 0x7fa9dc13ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:35:47.786 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:47.783+0000 7fa9dad9d700 1 --2- 172.21.15.23:0/3730962180 >> v2:172.21.15.155:3300/0 conn(0x7fa9dc0aaab0 0x7fa9dc13ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:47.786 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:47.783+0000 7fa9dad9d700 1 -- 172.21.15.23:0/3730962180 >> v2:172.21.15.155:3300/0 conn(0x7fa9dc0aaab0 msgr2=0x7fa9dc13ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:35:47.787 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:47.783+0000 7fa9dad9d700 1 -- 172.21.15.23:0/3730962180 >> v2:172.21.15.155:3300/0 conn(0x7fa9dc0aaab0 msgr2=0x7fa9dc13ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:35:47.787 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:47.783+0000 7fa9dad9d700 1 --2- 172.21.15.23:0/3730962180 >> v2:172.21.15.155:3300/0 conn(0x7fa9dc0aaab0 0x7fa9dc13ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:35:47.787 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:47.783+0000 7fa9dad9d700 1 --2- 172.21.15.23:0/3730962180 >> v2:172.21.15.155:3300/0 conn(0x7fa9dc0aaab0 0x7fa9dc13ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:35:48.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:48.584+0000 7fa9dad9d700 1 --2- 172.21.15.23:0/3730962180 >> v2:172.21.15.155:3300/0 conn(0x7fa9dc0aaab0 0x7fa9dc13ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:49.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:48.584+0000 7fa9dad9d700 1 -- 172.21.15.23:0/3730962180 >> v2:172.21.15.155:3300/0 conn(0x7fa9dc0aaab0 msgr2=0x7fa9dc13ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:35:49.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:48.584+0000 7fa9dad9d700 1 -- 172.21.15.23:0/3730962180 >> v2:172.21.15.155:3300/0 conn(0x7fa9dc0aaab0 msgr2=0x7fa9dc13ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:35:49.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:48.584+0000 7fa9dad9d700 1 --2- 172.21.15.23:0/3730962180 >> v2:172.21.15.155:3300/0 conn(0x7fa9dc0aaab0 0x7fa9dc13ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:35:49.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:48.584+0000 7fa9dad9d700 1 --2- 172.21.15.23:0/3730962180 >> v2:172.21.15.155:3300/0 conn(0x7fa9dc0aaab0 0x7fa9dc13ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:35:50.184 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:50.181+0000 7fa9d8d99700 1 -- 172.21.15.23:0/3730962180 >> v2:172.21.15.155:3300/0 conn(0x7fa9dc0aaab0 msgr2=0x7fa9dc13ed40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:35:50.185 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:50.181+0000 7fa9d8d99700 1 --2- 172.21.15.23:0/3730962180 >> v2:172.21.15.155:3300/0 conn(0x7fa9dc0aaab0 0x7fa9dc13ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:50.185 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:50.181+0000 7fa9d8d99700 1 -- 172.21.15.23:0/3730962180 >> v1:172.21.15.155:6789/0 conn(0x7fa9dc0aae90 legacy=0x7fa9dc1381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:50.185 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:50.181+0000 7fa9d8d99700 1 -- 172.21.15.23:0/3730962180 >> v1:172.21.15.23:6789/0 conn(0x7fa9dc13aeb0 legacy=0x7fa9dc0a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:50.185 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:50.181+0000 7fa9d8d99700 1 --2- 172.21.15.23:0/3730962180 >> v2:172.21.15.23:3300/0 conn(0x7fa9bc0033f0 0x7fa9bc005870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:50.185 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:50.181+0000 7fa9d8d99700 1 --2- 172.21.15.23:0/3730962180 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9bc005e60 0x7fa9bc008300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:50.186 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:50.181+0000 7fa9d8d99700 1 -- 172.21.15.23:0/3730962180 --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9dc13e8a0 con 0x7fa9bc0033f0 2020-11-20T11:35:50.186 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:50.181+0000 7fa9d8d99700 1 -- 172.21.15.23:0/3730962180 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9bc0089e0 con 0x7fa9bc005e60 2020-11-20T11:35:50.186 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:50.181+0000 7fa9da59c700 1 -- 172.21.15.23:0/3730962180 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9bc005e60 msgr2=0x7fa9bc008300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:35:50.186 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:50.181+0000 7fa9da59c700 1 --2- 172.21.15.23:0/3730962180 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9bc005e60 0x7fa9bc008300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:35:50.187 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:50.181+0000 7fa9dad9d700 1 --2- 172.21.15.23:0/3730962180 >> v2:172.21.15.23:3300/0 conn(0x7fa9bc0033f0 0x7fa9bc005870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:50.187 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:50.181+0000 7fa9dad9d700 1 -- 172.21.15.23:0/3730962180 >> v2:172.21.15.23:3300/0 conn(0x7fa9bc0033f0 msgr2=0x7fa9bc005870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:35:50.187 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:50.181+0000 7fa9dad9d700 1 -- 172.21.15.23:0/3730962180 >> v2:172.21.15.23:3300/0 conn(0x7fa9bc0033f0 msgr2=0x7fa9bc005870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:35:50.187 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:50.181+0000 7fa9dad9d700 1 --2- 172.21.15.23:0/3730962180 >> v2:172.21.15.23:3300/0 conn(0x7fa9bc0033f0 0x7fa9bc005870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:35:50.187 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:50.181+0000 7fa9dad9d700 1 --2- 172.21.15.23:0/3730962180 >> v2:172.21.15.23:3300/0 conn(0x7fa9bc0033f0 0x7fa9bc005870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:35:50.385 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:50.382+0000 7fa9dad9d700 1 --2- 172.21.15.23:0/3730962180 >> v2:172.21.15.23:3300/0 conn(0x7fa9bc0033f0 0x7fa9bc005870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:50.385 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:50.382+0000 7fa9da59c700 1 -- 172.21.15.23:0/3730962180 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9bc005e60 msgr2=0x7fa9bc008300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:35:50.386 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:50.382+0000 7fa9da59c700 1 --2- 172.21.15.23:0/3730962180 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9bc005e60 0x7fa9bc008300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:35:50.386 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:50.382+0000 7fa9dad9d700 1 -- 172.21.15.23:0/3730962180 >> v2:172.21.15.23:3300/0 conn(0x7fa9bc0033f0 msgr2=0x7fa9bc005870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:35:50.386 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:50.382+0000 7fa9dad9d700 1 -- 172.21.15.23:0/3730962180 >> v2:172.21.15.23:3300/0 conn(0x7fa9bc0033f0 msgr2=0x7fa9bc005870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:35:50.386 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:50.382+0000 7fa9dad9d700 1 --2- 172.21.15.23:0/3730962180 >> v2:172.21.15.23:3300/0 conn(0x7fa9bc0033f0 0x7fa9bc005870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:35:50.387 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:50.382+0000 7fa9dad9d700 1 --2- 172.21.15.23:0/3730962180 >> v2:172.21.15.23:3300/0 conn(0x7fa9bc0033f0 0x7fa9bc005870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:35:50.786 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:50.783+0000 7fa9da59c700 1 -- 172.21.15.23:0/3730962180 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9bc005e60 msgr2=0x7fa9bc008300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:35:50.786 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:50.783+0000 7fa9dad9d700 1 --2- 172.21.15.23:0/3730962180 >> v2:172.21.15.23:3300/0 conn(0x7fa9bc0033f0 0x7fa9bc005870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:50.786 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:50.783+0000 7fa9da59c700 1 --2- 172.21.15.23:0/3730962180 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9bc005e60 0x7fa9bc008300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:35:50.787 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:50.783+0000 7fa9dad9d700 1 -- 172.21.15.23:0/3730962180 >> v2:172.21.15.23:3300/0 conn(0x7fa9bc0033f0 msgr2=0x7fa9bc005870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:35:50.787 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:50.783+0000 7fa9dad9d700 1 -- 172.21.15.23:0/3730962180 >> v2:172.21.15.23:3300/0 conn(0x7fa9bc0033f0 msgr2=0x7fa9bc005870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:35:50.787 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:50.783+0000 7fa9dad9d700 1 --2- 172.21.15.23:0/3730962180 >> v2:172.21.15.23:3300/0 conn(0x7fa9bc0033f0 0x7fa9bc005870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:35:50.787 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:50.783+0000 7fa9dad9d700 1 --2- 172.21.15.23:0/3730962180 >> v2:172.21.15.23:3300/0 conn(0x7fa9bc0033f0 0x7fa9bc005870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:35:51.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:51.584+0000 7fa9da59c700 1 -- 172.21.15.23:0/3730962180 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9bc005e60 msgr2=0x7fa9bc008300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:35:51.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:51.584+0000 7fa9da59c700 1 --2- 172.21.15.23:0/3730962180 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9bc005e60 0x7fa9bc008300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:35:51.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:51.584+0000 7fa9dad9d700 1 --2- 172.21.15.23:0/3730962180 >> v2:172.21.15.23:3300/0 conn(0x7fa9bc0033f0 0x7fa9bc005870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:51.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:51.584+0000 7fa9dad9d700 1 -- 172.21.15.23:0/3730962180 >> v2:172.21.15.23:3300/0 conn(0x7fa9bc0033f0 msgr2=0x7fa9bc005870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:35:51.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:51.584+0000 7fa9dad9d700 1 -- 172.21.15.23:0/3730962180 >> v2:172.21.15.23:3300/0 conn(0x7fa9bc0033f0 msgr2=0x7fa9bc005870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:35:51.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:51.584+0000 7fa9dad9d700 1 --2- 172.21.15.23:0/3730962180 >> v2:172.21.15.23:3300/0 conn(0x7fa9bc0033f0 0x7fa9bc005870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:35:51.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:51.584+0000 7fa9dad9d700 1 --2- 172.21.15.23:0/3730962180 >> v2:172.21.15.23:3300/0 conn(0x7fa9bc0033f0 0x7fa9bc005870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:35:53.185 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.181+0000 7fa9d8d99700 1 -- 172.21.15.23:0/3730962180 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9bc005e60 msgr2=0x7fa9bc008300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:35:53.185 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.181+0000 7fa9d8d99700 1 --2- 172.21.15.23:0/3730962180 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9bc005e60 0x7fa9bc008300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:35:53.185 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.181+0000 7fa9d8d99700 1 -- 172.21.15.23:0/3730962180 >> v2:172.21.15.23:3300/0 conn(0x7fa9bc0033f0 msgr2=0x7fa9bc005870 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:35:53.186 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.181+0000 7fa9d8d99700 1 --2- 172.21.15.23:0/3730962180 >> v2:172.21.15.23:3300/0 conn(0x7fa9bc0033f0 0x7fa9bc005870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:53.186 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.181+0000 7fa9db59e700 1 -- 172.21.15.23:0/3730962180 reap_dead start 2020-11-20T11:35:53.186 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.181+0000 7fa9d8d99700 1 --2- 172.21.15.23:0/3730962180 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9bc007a10 0x7fa9bc007e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:53.186 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.181+0000 7fa9d8d99700 1 --2- 172.21.15.23:0/3730962180 >> v2:172.21.15.23:3300/0 conn(0x7fa9bc00adf0 0x7fa9bc00d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:53.186 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.181+0000 7fa9d8d99700 1 -- 172.21.15.23:0/3730962180 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9dc0ab6d0 con 0x7fa9bc003860 2020-11-20T11:35:53.187 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.181+0000 7fa9d8d99700 1 -- 172.21.15.23:0/3730962180 --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9dc13e8a0 con 0x7fa9bc00adf0 2020-11-20T11:35:53.187 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.181+0000 7fa9d8d99700 1 -- 172.21.15.23:0/3730962180 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9bc0089e0 con 0x7fa9bc007a10 2020-11-20T11:35:53.187 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.181+0000 7fa9da59c700 1 -- 172.21.15.23:0/3730962180 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9bc007a10 msgr2=0x7fa9bc007e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:35:53.187 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.181+0000 7fa9da59c700 1 --2- 172.21.15.23:0/3730962180 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9bc007a10 0x7fa9bc007e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:35:53.187 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.181+0000 7fa9db59e700 1 --2- 172.21.15.23:0/3730962180 >> v2:172.21.15.23:3300/0 conn(0x7fa9bc00adf0 0x7fa9bc00d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:53.188 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.181+0000 7fa9db59e700 1 -- 172.21.15.23:0/3730962180 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9bc007a10 msgr2=0x7fa9bc007e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:35:53.188 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.181+0000 7fa9db59e700 1 --2- 172.21.15.23:0/3730962180 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9bc007a10 0x7fa9bc007e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:35:53.188 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.181+0000 7fa9db59e700 1 -- 172.21.15.23:0/3730962180 >> v1:172.21.15.155:6789/0 conn(0x7fa9bc003860 legacy=0x7fa9bc0062f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:53.188 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.181+0000 7fa9db59e700 1 -- 172.21.15.23:0/3730962180 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9bc00e6e0 con 0x7fa9bc00adf0 2020-11-20T11:35:53.188 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.182+0000 7fa9db59e700 1 --2- 172.21.15.23:0/3730962180 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9bc00adf0 0x7fa9bc00d1e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fa9dc0af200 tx=0x7fa9d4007ee0).ready entity=mon.0 client_cookie=dad434b0d8513f1e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:53.189 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.182+0000 7fa9d9d9b700 1 -- 172.21.15.23:0/3730962180 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4001580 con 0x7fa9bc00adf0 2020-11-20T11:35:53.189 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.182+0000 7fa9d9d9b700 1 -- 172.21.15.23:0/3730962180 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d4005bc0 con 0x7fa9bc00adf0 2020-11-20T11:35:53.189 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.182+0000 7fa9d9d9b700 1 -- 172.21.15.23:0/3730962180 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d400c780 con 0x7fa9bc00adf0 2020-11-20T11:35:53.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.182+0000 7fa9e114a700 1 -- 172.21.15.23:0/3730962180 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9bc00adf0 msgr2=0x7fa9bc00d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:53.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.182+0000 7fa9e114a700 1 --2- 172.21.15.23:0/3730962180 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9bc00adf0 0x7fa9bc00d1e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fa9dc0af200 tx=0x7fa9d4007ee0).stop 2020-11-20T11:35:53.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.182+0000 7fa9e114a700 1 -- 172.21.15.23:0/3730962180 shutdown_connections 2020-11-20T11:35:53.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.182+0000 7fa9e114a700 1 --2- 172.21.15.23:0/3730962180 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9bc00adf0 0x7fa9bc00d1e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:53.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.182+0000 7fa9e114a700 1 --2- 172.21.15.23:0/3730962180 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9bc007a10 0x7fa9bc007e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:35:53.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.182+0000 7fa9e114a700 1 -- 172.21.15.23:0/3730962180 >> 172.21.15.23:0/3730962180 conn(0x7fa9dc0678e0 msgr2=0x7fa9dc067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:53.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.183+0000 7fa9e114a700 1 -- 172.21.15.23:0/3730962180 shutdown_connections 2020-11-20T11:35:53.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.183+0000 7fa9e114a700 1 -- 172.21.15.23:0/3730962180 wait complete. 2020-11-20T11:35:53.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.184+0000 7fa9e114a700 1 Processor -- start 2020-11-20T11:35:53.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.184+0000 7fa9e114a700 1 -- start start 2020-11-20T11:35:53.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.184+0000 7fa9e114a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9bc003860 0x7fa9dc135b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:53.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.184+0000 7fa9e114a700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9bc007a10 0x7fa9dc14e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:53.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.184+0000 7fa9e114a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9bc00adf0 0x7fa9dc0a6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:53.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.184+0000 7fa9e114a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc0a6960 con 0x7fa9bc00adf0 2020-11-20T11:35:53.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.184+0000 7fa9e114a700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc14d9a0 con 0x7fa9bc003860 2020-11-20T11:35:53.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.184+0000 7fa9e114a700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc14dc40 con 0x7fa9bc007a10 2020-11-20T11:35:53.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.184+0000 7fa9db59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9bc00adf0 0x7fa9dc0a6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:53.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.184+0000 7fa9da59c700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9bc007a10 msgr2=0x7fa9dc14e0e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:35:53.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.184+0000 7fa9da59c700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9bc007a10 0x7fa9dc14e0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:35:53.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.184+0000 7fa9db59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9bc00adf0 0x7fa9dc0a6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42648/0 (socket says 172.21.15.23:42648) 2020-11-20T11:35:53.197 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.184+0000 7fa9db59e700 1 -- 172.21.15.23:0/1679984455 learned_addr learned my addr 172.21.15.23:0/1679984455 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:35:53.197 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.185+0000 7fa9dad9d700 1 --2- 172.21.15.23:0/1679984455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9bc003860 0x7fa9dc135b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:53.197 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.185+0000 7fa9db59e700 1 -- 172.21.15.23:0/1679984455 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9bc007a10 msgr2=0x7fa9dc14e0e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:35:53.197 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.185+0000 7fa9db59e700 1 --2- 172.21.15.23:0/1679984455 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9bc007a10 0x7fa9dc14e0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:35:53.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.185+0000 7fa9db59e700 1 -- 172.21.15.23:0/1679984455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9bc003860 msgr2=0x7fa9dc135b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:53.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.185+0000 7fa9db59e700 1 --2- 172.21.15.23:0/1679984455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9bc003860 0x7fa9dc135b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:53.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.185+0000 7fa9db59e700 1 -- 172.21.15.23:0/1679984455 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9dc14e5d0 con 0x7fa9bc00adf0 2020-11-20T11:35:53.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.185+0000 7fa9db59e700 1 --2- 172.21.15.23:0/1679984455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9bc00adf0 0x7fa9dc0a6050 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fa9bc005de0 tx=0x7fa9d4013000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:53.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.185+0000 7fa9c2ffd700 1 -- 172.21.15.23:0/1679984455 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4017070 con 0x7fa9bc00adf0 2020-11-20T11:35:53.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.185+0000 7fa9c2ffd700 1 -- 172.21.15.23:0/1679984455 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d40074b0 con 0x7fa9bc00adf0 2020-11-20T11:35:53.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.185+0000 7fa9c2ffd700 1 -- 172.21.15.23:0/1679984455 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4007870 con 0x7fa9bc00adf0 2020-11-20T11:35:53.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.185+0000 7fa9e114a700 1 -- 172.21.15.23:0/1679984455 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9dc132b10 con 0x7fa9bc00adf0 2020-11-20T11:35:53.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.185+0000 7fa9e114a700 1 -- 172.21.15.23:0/1679984455 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9dc14a250 con 0x7fa9bc00adf0 2020-11-20T11:35:53.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.186+0000 7fa9e114a700 1 -- 172.21.15.23:0/1679984455 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9dc14a4a0 con 0x7fa9bc00adf0 2020-11-20T11:35:53.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.186+0000 7fa9c2ffd700 1 -- 172.21.15.23:0/1679984455 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa9d4010070 con 0x7fa9bc00adf0 2020-11-20T11:35:53.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.187+0000 7fa9c2ffd700 1 --2- 172.21.15.23:0/1679984455 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa9b0056d10 0x7fa9b00591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:53.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.187+0000 7fa9c2ffd700 1 -- 172.21.15.23:0/1679984455 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fa9d401a040 con 0x7fa9bc00adf0 2020-11-20T11:35:53.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.187+0000 7fa9dad9d700 1 --2- 172.21.15.23:0/1679984455 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa9b0056d10 0x7fa9b00591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:53.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.188+0000 7fa9dad9d700 1 --2- 172.21.15.23:0/1679984455 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa9b0056d10 0x7fa9b00591d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fa9dc0a5df0 tx=0x7fa9cc008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:53.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.191+0000 7fa9c2ffd700 1 -- 172.21.15.23:0/1679984455 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa9dc14a4a0 con 0x7fa9bc00adf0 2020-11-20T11:35:53.417 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.414+0000 7fa9e114a700 1 -- 172.21.15.23:0/1679984455 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa9dc14a4a0 con 0x7fa9bc00adf0 2020-11-20T11:35:53.418 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.415+0000 7fa9c2ffd700 1 -- 172.21.15.23:0/1679984455 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1471 (secure 0 0 0) 0x7fa9dc14a4a0 con 0x7fa9bc00adf0 2020-11-20T11:35:53.418 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":32,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:35:53.420 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.417+0000 7fa9e114a700 1 -- 172.21.15.23:0/1679984455 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa9b0056d10 msgr2=0x7fa9b00591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:53.420 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.417+0000 7fa9e114a700 1 --2- 172.21.15.23:0/1679984455 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa9b0056d10 0x7fa9b00591d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fa9dc0a5df0 tx=0x7fa9cc008040).stop 2020-11-20T11:35:53.420 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.417+0000 7fa9e114a700 1 -- 172.21.15.23:0/1679984455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9bc00adf0 msgr2=0x7fa9dc0a6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:53.420 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.417+0000 7fa9e114a700 1 --2- 172.21.15.23:0/1679984455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9bc00adf0 0x7fa9dc0a6050 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fa9bc005de0 tx=0x7fa9d4013000).stop 2020-11-20T11:35:53.421 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.417+0000 7fa9e114a700 1 -- 172.21.15.23:0/1679984455 shutdown_connections 2020-11-20T11:35:53.421 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.417+0000 7fa9e114a700 1 --2- 172.21.15.23:0/1679984455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9bc00adf0 0x7fa9dc0a6050 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:53.421 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.417+0000 7fa9e114a700 1 --2- 172.21.15.23:0/1679984455 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9bc003860 0x7fa9dc135b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:53.421 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.417+0000 7fa9e114a700 1 --2- 172.21.15.23:0/1679984455 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa9b0056d10 0x7fa9b00591d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:53.421 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.417+0000 7fa9e114a700 1 --2- 172.21.15.23:0/1679984455 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9bc007a10 0x7fa9dc14e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:35:53.422 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.417+0000 7fa9e114a700 1 -- 172.21.15.23:0/1679984455 >> 172.21.15.23:0/1679984455 conn(0x7fa9dc0678e0 msgr2=0x7fa9dc136330 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:53.422 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.417+0000 7fa9e114a700 1 -- 172.21.15.23:0/1679984455 shutdown_connections 2020-11-20T11:35:53.422 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.417+0000 7fa9e114a700 1 -- 172.21.15.23:0/1679984455 wait complete. 2020-11-20T11:35:53.431 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2020-11-20T11:35:53.432 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:35:53.555 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.552+0000 7fbe44d1b700 1 Processor -- start 2020-11-20T11:35:53.556 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.552+0000 7fbe44d1b700 1 -- start start 2020-11-20T11:35:53.558 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.553+0000 7fbe44d1b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fbe400619d0 0x7fbe400a9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:53.558 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.553+0000 7fbe44d1b700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe400610b0 con 0x7fbe40061db0 2020-11-20T11:35:53.559 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.553+0000 7fbe44d1b700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe400a9750 con 0x7fbe40066170 2020-11-20T11:35:53.559 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.553+0000 7fbe44d1b700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe40066680 con 0x7fbe400619d0 2020-11-20T11:35:53.559 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.553+0000 7fbe3e59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fbe400619d0 0x7fbe400a9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:53.559 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.553+0000 7fbe3e59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fbe400619d0 0x7fbe400a9260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42652/0 (socket says 172.21.15.23:42652) 2020-11-20T11:35:53.560 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.553+0000 7fbe3e59c700 1 -- 172.21.15.23:0/3744256174 learned_addr learned my addr 172.21.15.23:0/3744256174 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:35:53.560 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.553+0000 7fbe3dd9b700 1 --1- 172.21.15.23:0/3744256174 >> v1:172.21.15.23:6789/0 conn(0x7fbe40061db0 0x7fbe400609a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:56904/0 (socket says 172.21.15.23:56904) 2020-11-20T11:35:53.560 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.554+0000 7fbe3e59c700 1 -- 172.21.15.23:0/3744256174 >> v1:172.21.15.155:6789/0 conn(0x7fbe40066170 legacy=0x7fbe40137e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:53.560 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.554+0000 7fbe3e59c700 1 -- 172.21.15.23:0/3744256174 >> v1:172.21.15.23:6789/0 conn(0x7fbe40061db0 legacy=0x7fbe400609a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:53.560 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.554+0000 7fbe3e59c700 1 -- 172.21.15.23:0/3744256174 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe40066920 con 0x7fbe400619d0 2020-11-20T11:35:53.561 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.554+0000 7fbe3e59c700 1 --2- 172.21.15.23:0/3744256174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe400619d0 0x7fbe400a9260 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fbe30009a10 tx=0x7fbe30006ef0).ready entity=mon.0 client_cookie=281daec3c9033b75 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:53.561 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.554+0000 7fbe3d59a700 1 -- 172.21.15.23:0/3744256174 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3000b860 con 0x7fbe400619d0 2020-11-20T11:35:53.561 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.554+0000 7fbe3d59a700 1 -- 172.21.15.23:0/3744256174 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe30000e90 con 0x7fbe400619d0 2020-11-20T11:35:53.561 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.554+0000 7fbe3d59a700 1 -- 172.21.15.23:0/3744256174 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3000bbd0 con 0x7fbe400619d0 2020-11-20T11:35:53.561 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.555+0000 7fbe44d1b700 1 -- 172.21.15.23:0/3744256174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe400619d0 msgr2=0x7fbe400a9260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:53.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.555+0000 7fbe44d1b700 1 --2- 172.21.15.23:0/3744256174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe400619d0 0x7fbe400a9260 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fbe30009a10 tx=0x7fbe30006ef0).stop 2020-11-20T11:35:53.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.555+0000 7fbe44d1b700 1 -- 172.21.15.23:0/3744256174 shutdown_connections 2020-11-20T11:35:53.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.555+0000 7fbe44d1b700 1 --2- 172.21.15.23:0/3744256174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe400619d0 0x7fbe400a9260 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:53.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.555+0000 7fbe44d1b700 1 -- 172.21.15.23:0/3744256174 >> 172.21.15.23:0/3744256174 conn(0x7fbe40063640 msgr2=0x7fbe400612c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:53.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.555+0000 7fbe44d1b700 1 -- 172.21.15.23:0/3744256174 shutdown_connections 2020-11-20T11:35:53.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.555+0000 7fbe44d1b700 1 -- 172.21.15.23:0/3744256174 wait complete. 2020-11-20T11:35:53.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.556+0000 7fbe44d1b700 1 Processor -- start 2020-11-20T11:35:53.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.556+0000 7fbe44d1b700 1 -- start start 2020-11-20T11:35:53.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.556+0000 7fbe44d1b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbe400619d0 0x7fbe400a3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:53.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.556+0000 7fbe44d1b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe40061db0 0x7fbe401160d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:53.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.556+0000 7fbe44d1b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe40066170 0x7fbe4014a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:53.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.556+0000 7fbe44d1b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4014de00 con 0x7fbe40066170 2020-11-20T11:35:53.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.556+0000 7fbe44d1b700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4014e0a0 con 0x7fbe40061db0 2020-11-20T11:35:53.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.556+0000 7fbe44d1b700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4014e340 con 0x7fbe400619d0 2020-11-20T11:35:53.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.557+0000 7fbe3e59c700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbe400619d0 msgr2=0x7fbe400a3b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:35:53.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.557+0000 7fbe3ed9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe40066170 0x7fbe4014a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:53.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.557+0000 7fbe3e59c700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbe400619d0 0x7fbe400a3b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:35:53.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.557+0000 7fbe3ed9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe40066170 0x7fbe4014a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42662/0 (socket says 172.21.15.23:42662) 2020-11-20T11:35:53.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.557+0000 7fbe3ed9d700 1 -- 172.21.15.23:0/338310039 learned_addr learned my addr 172.21.15.23:0/338310039 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:35:53.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.557+0000 7fbe3dd9b700 1 --2- 172.21.15.23:0/338310039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe40061db0 0x7fbe401160d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:53.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.557+0000 7fbe3ed9d700 1 -- 172.21.15.23:0/338310039 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbe400619d0 msgr2=0x7fbe400a3b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:35:53.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.557+0000 7fbe3ed9d700 1 --2- 172.21.15.23:0/338310039 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbe400619d0 0x7fbe400a3b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:35:53.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.557+0000 7fbe3ed9d700 1 -- 172.21.15.23:0/338310039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe40061db0 msgr2=0x7fbe401160d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:53.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.557+0000 7fbe3ed9d700 1 --2- 172.21.15.23:0/338310039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe40061db0 0x7fbe401160d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:53.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.557+0000 7fbe3ed9d700 1 -- 172.21.15.23:0/338310039 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe4014e7e0 con 0x7fbe40066170 2020-11-20T11:35:53.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.557+0000 7fbe3dd9b700 1 --2- 172.21.15.23:0/338310039 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbe40061db0 0x7fbe401160d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:35:53.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.557+0000 7fbe3ed9d700 1 --2- 172.21.15.23:0/338310039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe40066170 0x7fbe4014a530 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fbe3800ca90 tx=0x7fbe380013a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:53.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.557+0000 7fbe2e7fc700 1 -- 172.21.15.23:0/338310039 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3800a5e0 con 0x7fbe40066170 2020-11-20T11:35:53.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.557+0000 7fbe2e7fc700 1 -- 172.21.15.23:0/338310039 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe3800a740 con 0x7fbe40066170 2020-11-20T11:35:53.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.557+0000 7fbe44d1b700 1 -- 172.21.15.23:0/338310039 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe4014da80 con 0x7fbe40066170 2020-11-20T11:35:53.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.557+0000 7fbe2e7fc700 1 -- 172.21.15.23:0/338310039 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3800aac0 con 0x7fbe40066170 2020-11-20T11:35:53.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.557+0000 7fbe44d1b700 1 -- 172.21.15.23:0/338310039 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe400a8090 con 0x7fbe40066170 2020-11-20T11:35:53.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.558+0000 7fbe44d1b700 1 --2- 172.21.15.23:0/338310039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe4005c070 0x7fbe40136ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:53.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.558+0000 7fbe44d1b700 1 -- 172.21.15.23:0/338310039 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4014dca0 con 0x7fbe4005c070 2020-11-20T11:35:53.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.558+0000 7fbe3e59c700 1 --2- 172.21.15.23:0/338310039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe4005c070 0x7fbe40136ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:53.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.559+0000 7fbe3e59c700 1 -- 172.21.15.23:0/338310039 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbe4014e500 con 0x7fbe4005c070 2020-11-20T11:35:53.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.559+0000 7fbe2e7fc700 1 -- 172.21.15.23:0/338310039 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbe38019030 con 0x7fbe40066170 2020-11-20T11:35:53.573 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.559+0000 7fbe3e59c700 1 --2- 172.21.15.23:0/338310039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe4005c070 0x7fbe40136ef0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fbe30009a10 tx=0x7fbe30016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:53.573 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.559+0000 7fbe2e7fc700 1 --2- 172.21.15.23:0/338310039 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbe14056d10 0x7fbe140591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:53.574 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.559+0000 7fbe2e7fc700 1 -- 172.21.15.23:0/338310039 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fbe3806f6b0 con 0x7fbe40066170 2020-11-20T11:35:53.574 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.559+0000 7fbe2e7fc700 1 -- 172.21.15.23:0/338310039 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe30011d30 con 0x7fbe4005c070 2020-11-20T11:35:53.574 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.559+0000 7fbe2e7fc700 1 -- 172.21.15.23:0/338310039 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fbe30011e90 con 0x7fbe4005c070 2020-11-20T11:35:53.574 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.559+0000 7fbe2e7fc700 1 -- 172.21.15.23:0/338310039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe4005c070 msgr2=0x7fbe40136ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:53.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.559+0000 7fbe2e7fc700 1 --2- 172.21.15.23:0/338310039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe4005c070 0x7fbe40136ef0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fbe30009a10 tx=0x7fbe30016000).stop 2020-11-20T11:35:53.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.559+0000 7fbe2e7fc700 1 -- 172.21.15.23:0/338310039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe4005c070 msgr2=0x7fbe40136ef0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:35:53.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.559+0000 7fbe2e7fc700 1 --2- 172.21.15.23:0/338310039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe4005c070 0x7fbe40136ef0 secure :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0x7fbe30009a10 tx=0x7fbe30016000).stop 2020-11-20T11:35:53.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.559+0000 7fbe3dd9b700 1 --2- 172.21.15.23:0/338310039 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbe14056d10 0x7fbe140591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:53.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.560+0000 7fbe3dd9b700 1 --2- 172.21.15.23:0/338310039 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbe14056d10 0x7fbe140591d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fbe400a4060 tx=0x7fbe34007120).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:53.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.566+0000 7fbe44d1b700 1 --2- 172.21.15.23:0/338310039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe400622d0 0x7fbe401373e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:53.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.566+0000 7fbe44d1b700 1 -- 172.21.15.23:0/338310039 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe401378d0 con 0x7fbe400622d0 2020-11-20T11:35:53.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.566+0000 7fbe3e59c700 1 --2- 172.21.15.23:0/338310039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe400622d0 0x7fbe401373e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:53.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.566+0000 7fbe3e59c700 1 -- 172.21.15.23:0/338310039 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbe400652b0 con 0x7fbe400622d0 2020-11-20T11:35:53.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.567+0000 7fbe3e59c700 1 --2- 172.21.15.23:0/338310039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe400622d0 0x7fbe401373e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fbe4005c480 tx=0x7fbe3001e000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:53.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.567+0000 7fbe2e7fc700 1 -- 172.21.15.23:0/338310039 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3001da10 con 0x7fbe400622d0 2020-11-20T11:35:53.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.567+0000 7fbe2e7fc700 1 -- 172.21.15.23:0/338310039 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4774 (secure 0 0 0) 0x7fbe30011b20 con 0x7fbe400622d0 2020-11-20T11:35:53.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.567+0000 7fbe2e7fc700 1 -- 172.21.15.23:0/338310039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe400622d0 msgr2=0x7fbe401373e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:53.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.567+0000 7fbe2e7fc700 1 --2- 172.21.15.23:0/338310039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe400622d0 0x7fbe401373e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fbe4005c480 tx=0x7fbe3001e000).stop 2020-11-20T11:35:53.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.567+0000 7fbe2e7fc700 1 -- 172.21.15.23:0/338310039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe400622d0 msgr2=0x7fbe401373e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:35:53.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.567+0000 7fbe2e7fc700 1 --2- 172.21.15.23:0/338310039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe400622d0 0x7fbe401373e0 secure :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0x7fbe4005c480 tx=0x7fbe3001e000).stop 2020-11-20T11:35:53.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.567+0000 7fbe44d1b700 1 -- 172.21.15.23:0/338310039 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbe14056d10 msgr2=0x7fbe140591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:53.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.567+0000 7fbe44d1b700 1 --2- 172.21.15.23:0/338310039 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbe14056d10 0x7fbe140591d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fbe400a4060 tx=0x7fbe34007120).stop 2020-11-20T11:35:53.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.567+0000 7fbe44d1b700 1 -- 172.21.15.23:0/338310039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe40066170 msgr2=0x7fbe4014a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:53.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.567+0000 7fbe44d1b700 1 --2- 172.21.15.23:0/338310039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe40066170 0x7fbe4014a530 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fbe3800ca90 tx=0x7fbe380013a0).stop 2020-11-20T11:35:53.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.567+0000 7fbe3ed9d700 1 -- 172.21.15.23:0/338310039 reap_dead start 2020-11-20T11:35:53.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.567+0000 7fbe44d1b700 1 -- 172.21.15.23:0/338310039 shutdown_connections 2020-11-20T11:35:53.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.567+0000 7fbe44d1b700 1 --2- 172.21.15.23:0/338310039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe40066170 0x7fbe4014a530 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:53.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.567+0000 7fbe44d1b700 1 -- 172.21.15.23:0/338310039 >> 172.21.15.23:0/338310039 conn(0x7fbe40063640 msgr2=0x7fbe40066b60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:53.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.567+0000 7fbe44d1b700 1 -- 172.21.15.23:0/338310039 shutdown_connections 2020-11-20T11:35:53.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.567+0000 7fbe44d1b700 1 -- 172.21.15.23:0/338310039 wait complete. 2020-11-20T11:35:53.582 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:35:53.582 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:35:53.582 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:35:53.582 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:35:53.583 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 32, 2020-11-20T11:35:53.583 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:35:53.583 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:35:53.584 INFO:teuthology.orchestra.run.smithi023.stdout: 1 2020-11-20T11:35:53.584 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:53.584 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 1, 2020-11-20T11:35:53.584 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:35:53.584 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:35:53.585 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:35:53.585 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:35:53.585 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:35:53.585 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:35:53.585 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:35:53.586 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:35:53.586 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:35:53.586 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:35:53.586 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:35:53.586 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:53.587 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:35:53.587 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:35:53.587 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:35:53.587 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:35:53.587 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:35:53.588 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:35:53.588 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:35:53.588 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:35:53.588 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:35:53.588 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:35:53.589 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:53.589 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:53.589 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:35:53.589 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:35:53.589 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:35:53.590 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:35:53.590 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:35:53.590 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:35:53.590 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:35:53.590 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:35:53.591 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:35:53.592 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:35:53.593 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:35:53.593 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:35:53.593 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:35:53.593 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:35:53.593 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:35:53.594 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:35:53.594 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:35:53.594 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:35:53.594 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:35:53.594 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:35:53.595 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:35:53.595 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:35:53.595 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:35:53.595 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:53.595 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:35:53.596 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:53.596 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:35:53.596 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:53.596 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:35:53.596 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:35:53.597 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:35:53.597 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:35:53.597 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:53.597 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:35:53.597 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:35:53.598 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:53.598 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:53.598 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:53.598 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:35:53.598 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:35:53.599 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:53.599 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:53.599 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:53.599 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:53.599 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:35:53.600 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:35:53.600 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:35:53.600 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:35:53.600 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:35:53.600 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:53.601 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:53.601 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:35:53.601 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:35:53.601 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:35:53.602 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:35:53.602 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:53.602 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:35:53.602 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:35:53.602 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:53.603 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:53.603 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:53.603 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:35:53.603 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:35:53.603 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:53.604 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:53.604 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:53.604 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:53.604 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:35:53.604 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:35:53.605 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:35:53.605 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:35:53.605 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:35:53.605 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:53.605 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:53.606 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:35:53.606 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:35:53.606 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:35:53.606 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:35:53.606 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:53.607 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:35:53.607 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:35:53.607 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:53.607 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:53.607 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:53.608 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:35:53.608 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:35:53.608 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:53.608 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:53.608 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:53.609 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:53.609 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:35:53.609 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:35:53.609 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:35:53.609 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:35:53.610 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:35:53.610 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:53.610 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:53.610 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:53.610 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:35:53.611 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:35:53.611 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:53.611 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:53.611 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:53.611 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:35:53.612 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:53.612 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:53.612 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:35:53.612 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:53.612 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:53.612 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:53.613 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:35:53.613 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:53.613 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:53.613 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:35:53.614 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:53.614 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:53.614 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:53.614 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 6 2020-11-20T11:35:53.614 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:53.615 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:53.615 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:35:53.615 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:53.615 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:53.615 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:53.616 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:35:53.616 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:53.616 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:53.616 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:35:53.616 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:53.616 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:53.617 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:53.617 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:35:53.617 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:53.617 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:53.618 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:53.618 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:35:53.618 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:35:53.619 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:35:53.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.746+0000 7fa2c8462700 1 Processor -- start 2020-11-20T11:35:53.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.746+0000 7fa2c8462700 1 -- start start 2020-11-20T11:35:53.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.746+0000 7fa2c8462700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa2c00aaf00 0x7fa2c013e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:53.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.746+0000 7fa2c8462700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2c00a85c0 0x7fa2c013a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:53.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.747+0000 7fa2c8462700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa2c00ab2e0 0x7fa2c00a78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:53.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.747+0000 7fa2c8462700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2c013adb0 con 0x7fa2c00ab2e0 2020-11-20T11:35:53.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.747+0000 7fa2c8462700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2c013e010 con 0x7fa2c00aaf00 2020-11-20T11:35:53.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.747+0000 7fa2c8462700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c013e2b0 con 0x7fa2c00a85c0 2020-11-20T11:35:53.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.747+0000 7fa2c69ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa2c00ab2e0 0x7fa2c00a78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:53.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.747+0000 7fa2c59fd700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2c00a85c0 msgr2=0x7fa2c013a8c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:35:53.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.747+0000 7fa2c59fd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2c00a85c0 0x7fa2c013a8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:35:53.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.747+0000 7fa2c69ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa2c00ab2e0 0x7fa2c00a78b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42674/0 (socket says 172.21.15.23:42674) 2020-11-20T11:35:53.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.747+0000 7fa2c69ff700 1 -- 172.21.15.23:0/3564093563 learned_addr learned my addr 172.21.15.23:0/3564093563 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:35:53.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.747+0000 7fa2c61fe700 1 --2- 172.21.15.23:0/3564093563 >> v2:172.21.15.155:3300/0 conn(0x7fa2c00aaf00 0x7fa2c013e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:53.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.747+0000 7fa2c69ff700 1 -- 172.21.15.23:0/3564093563 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2c00a85c0 msgr2=0x7fa2c013a8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:35:53.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.747+0000 7fa2c69ff700 1 --2- 172.21.15.23:0/3564093563 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2c00a85c0 0x7fa2c013a8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:35:53.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.747+0000 7fa2c69ff700 1 -- 172.21.15.23:0/3564093563 >> v2:172.21.15.155:3300/0 conn(0x7fa2c00aaf00 msgr2=0x7fa2c013e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:35:53.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.747+0000 7fa2c69ff700 1 --2- 172.21.15.23:0/3564093563 >> v2:172.21.15.155:3300/0 conn(0x7fa2c00aaf00 0x7fa2c013e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:53.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.747+0000 7fa2c69ff700 1 -- 172.21.15.23:0/3564093563 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c013e550 con 0x7fa2c00ab2e0 2020-11-20T11:35:53.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.747+0000 7fa2c69ff700 1 --2- 172.21.15.23:0/3564093563 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2c00ab2e0 0x7fa2c00a78b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fa2c00b10c0 tx=0x7fa2b80092d0).ready entity=mon.0 client_cookie=bb8de1ba51887cfc server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:53.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.747+0000 7fa2c51fc700 1 -- 172.21.15.23:0/3564093563 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b800bbf0 con 0x7fa2c00ab2e0 2020-11-20T11:35:53.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.747+0000 7fa2c51fc700 1 -- 172.21.15.23:0/3564093563 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b800bd50 con 0x7fa2c00ab2e0 2020-11-20T11:35:53.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.747+0000 7fa2c51fc700 1 -- 172.21.15.23:0/3564093563 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b8011610 con 0x7fa2c00ab2e0 2020-11-20T11:35:53.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.748+0000 7fa2c8462700 1 -- 172.21.15.23:0/3564093563 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2c00ab2e0 msgr2=0x7fa2c00a78b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:53.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.748+0000 7fa2c8462700 1 --2- 172.21.15.23:0/3564093563 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2c00ab2e0 0x7fa2c00a78b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fa2c00b10c0 tx=0x7fa2b80092d0).stop 2020-11-20T11:35:53.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.748+0000 7fa2c8462700 1 -- 172.21.15.23:0/3564093563 shutdown_connections 2020-11-20T11:35:53.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.748+0000 7fa2c8462700 1 --2- 172.21.15.23:0/3564093563 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2c00ab2e0 0x7fa2c00a78b0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:53.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.748+0000 7fa2c8462700 1 --2- 172.21.15.23:0/3564093563 >> v2:172.21.15.155:3300/0 conn(0x7fa2c00aaf00 0x7fa2c013e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:53.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.748+0000 7fa2c8462700 1 --2- 172.21.15.23:0/3564093563 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2c00a85c0 0x7fa2c013a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:35:53.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.748+0000 7fa2c8462700 1 -- 172.21.15.23:0/3564093563 >> 172.21.15.23:0/3564093563 conn(0x7fa2c00678e0 msgr2=0x7fa2c0067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:53.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.748+0000 7fa2c8462700 1 -- 172.21.15.23:0/3564093563 shutdown_connections 2020-11-20T11:35:53.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.748+0000 7fa2c8462700 1 -- 172.21.15.23:0/3564093563 wait complete. 2020-11-20T11:35:53.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.749+0000 7fa2c8462700 1 Processor -- start 2020-11-20T11:35:53.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.749+0000 7fa2c8462700 1 -- start start 2020-11-20T11:35:53.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.750+0000 7fa2c8462700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2c00a85c0 0x7fa2c00a70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:53.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.750+0000 7fa2c8462700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2c00aaf00 0x7fa2c009f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:53.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.750+0000 7fa2c8462700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2c00ab2e0 0x7fa2c009df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:53.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.750+0000 7fa2c8462700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c009e9e0 con 0x7fa2c00ab2e0 2020-11-20T11:35:53.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.750+0000 7fa2c8462700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c009e740 con 0x7fa2c00a85c0 2020-11-20T11:35:53.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.750+0000 7fa2c8462700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c009f140 con 0x7fa2c00aaf00 2020-11-20T11:35:53.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.750+0000 7fa2c59fd700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2c00aaf00 msgr2=0x7fa2c009f5e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:35:53.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.750+0000 7fa2c59fd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2c00aaf00 0x7fa2c009f5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:35:53.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.750+0000 7fa2c69ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2c00ab2e0 0x7fa2c009df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:53.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.750+0000 7fa2c61fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2c00a85c0 0x7fa2c00a70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:53.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.750+0000 7fa2c69ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2c00ab2e0 0x7fa2c009df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42680/0 (socket says 172.21.15.23:42680) 2020-11-20T11:35:53.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.750+0000 7fa2c69ff700 1 -- 172.21.15.23:0/1582787776 learned_addr learned my addr 172.21.15.23:0/1582787776 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:35:53.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.750+0000 7fa2c61fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2c00a85c0 0x7fa2c00a70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:35572/0 (socket says 172.21.15.23:35572) 2020-11-20T11:35:53.769 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.750+0000 7fa2c69ff700 1 -- 172.21.15.23:0/1582787776 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2c00aaf00 msgr2=0x7fa2c009f5e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:35:53.769 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.750+0000 7fa2c69ff700 1 --2- 172.21.15.23:0/1582787776 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa2c00aaf00 0x7fa2c009f5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:35:53.769 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.750+0000 7fa2c69ff700 1 -- 172.21.15.23:0/1582787776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2c00a85c0 msgr2=0x7fa2c00a70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:53.769 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.750+0000 7fa2c69ff700 1 --2- 172.21.15.23:0/1582787776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2c00a85c0 0x7fa2c00a70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:53.770 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.750+0000 7fa2c69ff700 1 -- 172.21.15.23:0/1582787776 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c009fad0 con 0x7fa2c00ab2e0 2020-11-20T11:35:53.770 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.751+0000 7fa2c69ff700 1 --2- 172.21.15.23:0/1582787776 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2c00ab2e0 0x7fa2c009df10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fa2b8007fd0 tx=0x7fa2b8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:53.770 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.751+0000 7fa2ae7fc700 1 -- 172.21.15.23:0/1582787776 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b8009ea0 con 0x7fa2c00ab2e0 2020-11-20T11:35:53.770 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.751+0000 7fa2ae7fc700 1 -- 172.21.15.23:0/1582787776 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b800bec0 con 0x7fa2c00ab2e0 2020-11-20T11:35:53.770 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.751+0000 7fa2c8462700 1 -- 172.21.15.23:0/1582787776 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2c009e400 con 0x7fa2c00ab2e0 2020-11-20T11:35:53.771 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.751+0000 7fa2ae7fc700 1 -- 172.21.15.23:0/1582787776 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b8010da0 con 0x7fa2c00ab2e0 2020-11-20T11:35:53.771 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.751+0000 7fa2c8462700 1 -- 172.21.15.23:0/1582787776 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2c00a1780 con 0x7fa2c00ab2e0 2020-11-20T11:35:53.771 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.752+0000 7fa2c8462700 1 --2- 172.21.15.23:0/1582787776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2c0149a80 0x7fa2c014a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:53.771 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.752+0000 7fa2c8462700 1 -- 172.21.15.23:0/1582787776 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c005cf20 con 0x7fa2c0149a80 2020-11-20T11:35:53.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.752+0000 7fa2c61fe700 1 --2- 172.21.15.23:0/1582787776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2c0149a80 0x7fa2c014a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:53.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.752+0000 7fa2ae7fc700 1 -- 172.21.15.23:0/1582787776 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa2b8017070 con 0x7fa2c00ab2e0 2020-11-20T11:35:53.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.752+0000 7fa2c61fe700 1 -- 172.21.15.23:0/1582787776 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa2c009e590 con 0x7fa2c0149a80 2020-11-20T11:35:53.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.752+0000 7fa2c61fe700 1 --2- 172.21.15.23:0/1582787776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2c0149a80 0x7fa2c014a150 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fa2c00a1960 tx=0x7fa2bc008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:53.773 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.752+0000 7fa2ae7fc700 1 --2- 172.21.15.23:0/1582787776 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa298056d10 0x7fa2980591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:53.774 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.753+0000 7fa2ae7fc700 1 -- 172.21.15.23:0/1582787776 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fa2b80721e0 con 0x7fa2c00ab2e0 2020-11-20T11:35:53.774 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.753+0000 7fa2ae7fc700 1 -- 172.21.15.23:0/1582787776 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc00e960 con 0x7fa2c0149a80 2020-11-20T11:35:53.774 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.753+0000 7fa2c59fd700 1 --2- 172.21.15.23:0/1582787776 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa298056d10 0x7fa2980591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:53.774 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.753+0000 7fa2ae7fc700 1 -- 172.21.15.23:0/1582787776 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fa2bc00eac0 con 0x7fa2c0149a80 2020-11-20T11:35:53.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.753+0000 7fa2ae7fc700 1 -- 172.21.15.23:0/1582787776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2c0149a80 msgr2=0x7fa2c014a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:53.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.753+0000 7fa2ae7fc700 1 --2- 172.21.15.23:0/1582787776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2c0149a80 0x7fa2c014a150 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fa2c00a1960 tx=0x7fa2bc008040).stop 2020-11-20T11:35:53.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.753+0000 7fa2ae7fc700 1 -- 172.21.15.23:0/1582787776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2c0149a80 msgr2=0x7fa2c014a150 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:35:53.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.753+0000 7fa2ae7fc700 1 --2- 172.21.15.23:0/1582787776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2c0149a80 0x7fa2c014a150 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:53.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.753+0000 7fa2c59fd700 1 --2- 172.21.15.23:0/1582787776 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa298056d10 0x7fa2980591d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fa2b0006fd0 tx=0x7fa2b0006b20).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:53.777 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.767+0000 7fa2c8462700 1 --2- 172.21.15.23:0/1582787776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2c014a640 0x7fa2c014d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:35:53.777 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.767+0000 7fa2c8462700 1 -- 172.21.15.23:0/1582787776 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c014aa20 con 0x7fa2c014a640 2020-11-20T11:35:53.778 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.768+0000 7fa2c61fe700 1 --2- 172.21.15.23:0/1582787776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2c014a640 0x7fa2c014d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:35:53.778 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.768+0000 7fa2c61fe700 1 -- 172.21.15.23:0/1582787776 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa2c009f300 con 0x7fa2c014a640 2020-11-20T11:35:53.778 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.768+0000 7fa2c61fe700 1 --2- 172.21.15.23:0/1582787776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2c014a640 0x7fa2c014d030 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fa2c00a1960 tx=0x7fa2bc008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:35:53.778 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.768+0000 7fa2ae7fc700 1 -- 172.21.15.23:0/1582787776 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc01d6f0 con 0x7fa2c014a640 2020-11-20T11:35:53.779 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.769+0000 7fa2ae7fc700 1 -- 172.21.15.23:0/1582787776 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4602 (secure 0 0 0) 0x7fa2bc000e90 con 0x7fa2c014a640 2020-11-20T11:35:53.779 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.769+0000 7fa2ae7fc700 1 -- 172.21.15.23:0/1582787776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2c014a640 msgr2=0x7fa2c014d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:53.779 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.769+0000 7fa2ae7fc700 1 --2- 172.21.15.23:0/1582787776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2c014a640 0x7fa2c014d030 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fa2c00a1960 tx=0x7fa2bc008040).stop 2020-11-20T11:35:53.779 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.769+0000 7fa2ae7fc700 1 -- 172.21.15.23:0/1582787776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2c014a640 msgr2=0x7fa2c014d030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:35:53.779 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.769+0000 7fa2ae7fc700 1 --2- 172.21.15.23:0/1582787776 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa2c014a640 0x7fa2c014d030 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:35:53.780 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.769+0000 7fa2c8462700 1 -- 172.21.15.23:0/1582787776 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa298056d10 msgr2=0x7fa2980591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:53.780 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.769+0000 7fa2c8462700 1 --2- 172.21.15.23:0/1582787776 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa298056d10 0x7fa2980591d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fa2b0006fd0 tx=0x7fa2b0006b20).stop 2020-11-20T11:35:53.780 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.769+0000 7fa2c8462700 1 -- 172.21.15.23:0/1582787776 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2c00ab2e0 msgr2=0x7fa2c009df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:35:53.780 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.769+0000 7fa2c8462700 1 --2- 172.21.15.23:0/1582787776 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2c00ab2e0 0x7fa2c009df10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fa2b8007fd0 tx=0x7fa2b8016000).stop 2020-11-20T11:35:53.780 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.769+0000 7fa2c69ff700 1 -- 172.21.15.23:0/1582787776 reap_dead start 2020-11-20T11:35:53.781 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.769+0000 7fa2c69ff700 1 -- 172.21.15.23:0/1582787776 reap_dead start 2020-11-20T11:35:53.781 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.769+0000 7fa2c8462700 1 -- 172.21.15.23:0/1582787776 shutdown_connections 2020-11-20T11:35:53.781 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.769+0000 7fa2c8462700 1 -- 172.21.15.23:0/1582787776 >> 172.21.15.23:0/1582787776 conn(0x7fa2c00678e0 msgr2=0x7fa2c00a7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:35:53.781 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.769+0000 7fa2c8462700 1 -- 172.21.15.23:0/1582787776 shutdown_connections 2020-11-20T11:35:53.781 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:35:53.769+0000 7fa2c8462700 1 -- 172.21.15.23:0/1582787776 wait complete. 2020-11-20T11:35:53.782 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:35:53.782 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:35:53.782 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:35:53.783 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:35:53.783 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 32, 2020-11-20T11:35:53.783 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:35:53.783 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:35:53.784 INFO:teuthology.orchestra.run.smithi023.stdout: 1 2020-11-20T11:35:53.784 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:53.784 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 2, 2020-11-20T11:35:53.784 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:35:53.785 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:35:53.785 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:35:53.785 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:35:53.785 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:35:53.785 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:35:53.786 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:35:53.786 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:35:53.786 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:35:53.786 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:35:53.786 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:35:53.787 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:53.787 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:35:53.787 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:35:53.787 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:35:53.787 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:35:53.788 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:35:53.788 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:35:53.788 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:35:53.788 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:35:53.788 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:35:53.789 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:35:53.789 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:53.789 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:53.789 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:35:53.790 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:35:53.790 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:35:53.790 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:35:53.790 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:35:53.790 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:35:53.791 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:35:53.791 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:35:53.791 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:35:53.791 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:35:53.791 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:35:53.792 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:35:53.792 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:35:53.792 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:35:53.792 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:35:53.792 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:35:53.793 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:35:53.793 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:35:53.793 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:35:53.793 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:35:53.794 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:35:53.794 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:35:53.794 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:35:53.794 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:53.794 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:35:53.795 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:53.795 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:35:53.795 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:53.795 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:35:53.795 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:35:53.796 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:35:53.796 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:35:53.796 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:53.796 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:35:53.796 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:35:53.797 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:53.797 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:53.797 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:53.797 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:35:53.797 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:35:53.798 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:53.798 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:53.798 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:53.798 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:53.798 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:35:53.799 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:35:53.799 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:35:53.799 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:35:53.799 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:35:53.799 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:53.799 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:53.800 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:35:53.800 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:35:53.800 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:35:53.800 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:35:53.801 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:53.801 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:35:53.801 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:35:53.801 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:53.801 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:53.802 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:53.802 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:35:53.802 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:35:53.802 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:53.802 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:53.803 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:53.803 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:53.803 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:35:53.803 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:35:53.803 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:35:53.804 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:35:53.804 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:35:53.804 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:53.804 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:53.804 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:35:53.805 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:35:53.805 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:35:53.805 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:35:53.805 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:53.805 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:35:53.806 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:35:53.806 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:53.806 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:53.806 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:53.806 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:35:53.807 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:35:53.807 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:35:53.807 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:53.807 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:53.808 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:53.808 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:35:53.808 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:35:53.808 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:35:53.808 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:35:53.808 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:35:53.809 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:53.809 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:53.809 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:53.809 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:35:53.810 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:35:53.810 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:53.810 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:53.810 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:53.810 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:35:53.810 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:53.811 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:53.811 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:35:53.811 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:53.811 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:53.812 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:53.812 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:35:53.812 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:53.812 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:53.812 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:35:53.813 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:53.813 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:53.813 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:53.813 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:35:53.813 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:53.814 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:35:53.814 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:35:53.814 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:35:53.814 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:35:53.814 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:35:53.815 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:35:53.815 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:35:53.815 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:35:53.815 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:35:53.815 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:35:53.816 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:35:53.817 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2020-11-20T11:35:59.881 INFO:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:35:59.895 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:36:13.817 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2020-11-20T11:36:13.818 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2020-11-20T11:36:13.818 INFO:tasks.ceph.mon.c:Restarting daemon 2020-11-20T11:36:13.818 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2020-11-20T11:36:13.821 INFO:tasks.ceph.mon.c:Started 2020-11-20T11:36:13.822 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:36:13.822 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:36:13.964 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.961+0000 7f06bc58e700 1 Processor -- start 2020-11-20T11:36:13.964 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.961+0000 7f06bc58e700 1 -- start start 2020-11-20T11:36:13.966 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.961+0000 7f06bc58e700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f06b40aaab0 0x7f06b413ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:13.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.961+0000 7f06bc58e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f06b40aae90 0x7f06b41381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:13.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.961+0000 7f06bc58e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06b413aeb0 0x7f06b40a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:13.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.961+0000 7f06bc58e700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06b413e600 con 0x7f06b40aaab0 2020-11-20T11:36:13.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.961+0000 7f06bc58e700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06b413e8a0 con 0x7f06b40aae90 2020-11-20T11:36:13.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.961+0000 7f06bc58e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06b40ab6d0 con 0x7f06b413aeb0 2020-11-20T11:36:13.968 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.962+0000 7f06ba32a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f06b40aaab0 0x7f06b413ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:13.968 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.962+0000 7f06ba32a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f06b40aaab0 0x7f06b413ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42690/0 (socket says 172.21.15.23:42690) 2020-11-20T11:36:13.968 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.962+0000 7f06ba32a700 1 -- 172.21.15.23:0/1898027470 learned_addr learned my addr 172.21.15.23:0/1898027470 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:36:13.968 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.962+0000 7f06bab2b700 1 --2- 172.21.15.23:0/1898027470 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06b413aeb0 0x7f06b40a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:13.968 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.962+0000 7f06b9b29700 1 --2- 172.21.15.23:0/1898027470 >> v2:172.21.15.155:3300/0 conn(0x7f06b40aae90 0x7f06b41381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:13.969 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.962+0000 7f06bab2b700 1 -- 172.21.15.23:0/1898027470 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06b413aeb0 msgr2=0x7f06b40a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:36:13.969 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.962+0000 7f06bab2b700 1 -- 172.21.15.23:0/1898027470 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06b413aeb0 msgr2=0x7f06b40a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:36:13.969 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.962+0000 7f06bab2b700 1 --2- 172.21.15.23:0/1898027470 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06b413aeb0 0x7f06b40a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:36:13.969 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.962+0000 7f06bab2b700 1 --2- 172.21.15.23:0/1898027470 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06b413aeb0 0x7f06b40a8c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:36:13.969 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.962+0000 7f06ba32a700 1 -- 172.21.15.23:0/1898027470 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06b413aeb0 msgr2=0x7f06b40a8c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:36:13.970 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.962+0000 7f06ba32a700 1 --2- 172.21.15.23:0/1898027470 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06b413aeb0 0x7f06b40a8c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:13.970 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.962+0000 7f06ba32a700 1 -- 172.21.15.23:0/1898027470 >> v2:172.21.15.155:3300/0 conn(0x7f06b40aae90 msgr2=0x7f06b41381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:36:13.970 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.962+0000 7f06ba32a700 1 --2- 172.21.15.23:0/1898027470 >> v2:172.21.15.155:3300/0 conn(0x7f06b40aae90 0x7f06b41381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:13.970 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.962+0000 7f06ba32a700 1 -- 172.21.15.23:0/1898027470 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b413f230 con 0x7f06b40aaab0 2020-11-20T11:36:13.971 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.962+0000 7f06ba32a700 1 --2- 172.21.15.23:0/1898027470 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06b40aaab0 0x7f06b413ed40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f06b40ab930 tx=0x7f06b0006ef0).ready entity=mon.0 client_cookie=fcc010efd8338ac5 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:13.971 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.962+0000 7f06b9328700 1 -- 172.21.15.23:0/1898027470 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b000b860 con 0x7f06b40aaab0 2020-11-20T11:36:13.971 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.962+0000 7f06b9328700 1 -- 172.21.15.23:0/1898027470 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b0000e90 con 0x7f06b40aaab0 2020-11-20T11:36:13.971 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.962+0000 7f06b9328700 1 -- 172.21.15.23:0/1898027470 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b000bbd0 con 0x7f06b40aaab0 2020-11-20T11:36:13.971 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.963+0000 7f06bc58e700 1 -- 172.21.15.23:0/1898027470 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06b40aaab0 msgr2=0x7f06b413ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:13.972 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.963+0000 7f06bc58e700 1 --2- 172.21.15.23:0/1898027470 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06b40aaab0 0x7f06b413ed40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f06b40ab930 tx=0x7f06b0006ef0).stop 2020-11-20T11:36:13.972 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.963+0000 7f06bc58e700 1 -- 172.21.15.23:0/1898027470 shutdown_connections 2020-11-20T11:36:13.972 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.963+0000 7f06bc58e700 1 --2- 172.21.15.23:0/1898027470 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06b413aeb0 0x7f06b40a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:13.972 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.963+0000 7f06bc58e700 1 --2- 172.21.15.23:0/1898027470 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06b40aaab0 0x7f06b413ed40 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:13.972 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.963+0000 7f06bc58e700 1 --2- 172.21.15.23:0/1898027470 >> v2:172.21.15.155:3300/0 conn(0x7f06b40aae90 0x7f06b41381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:13.973 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.963+0000 7f06bc58e700 1 -- 172.21.15.23:0/1898027470 >> 172.21.15.23:0/1898027470 conn(0x7f06b40678e0 msgr2=0x7f06b4067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:13.973 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.963+0000 7f06bc58e700 1 -- 172.21.15.23:0/1898027470 shutdown_connections 2020-11-20T11:36:13.973 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.963+0000 7f06bc58e700 1 -- 172.21.15.23:0/1898027470 wait complete. 2020-11-20T11:36:13.976 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.964+0000 7f06bc58e700 1 Processor -- start 2020-11-20T11:36:13.976 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.964+0000 7f06bc58e700 1 -- start start 2020-11-20T11:36:13.976 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.965+0000 7f06bc58e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06b40aaab0 0x7f06b409ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:13.976 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.965+0000 7f06bc58e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06b40aae90 0x7f06b40a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:13.977 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.965+0000 7f06bc58e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06b40a8000 0x7f06b40a4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:13.977 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.965+0000 7f06bc58e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b40a5d80 con 0x7f06b40aae90 2020-11-20T11:36:13.977 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.965+0000 7f06bc58e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b40a5170 con 0x7f06b40aaab0 2020-11-20T11:36:13.977 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.965+0000 7f06bc58e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06b40a53a0 con 0x7f06b40a8000 2020-11-20T11:36:13.978 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.965+0000 7f06b9b29700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06b40aae90 0x7f06b40a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:13.978 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.965+0000 7f06b9b29700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06b40aae90 0x7f06b40a62c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42698/0 (socket says 172.21.15.23:42698) 2020-11-20T11:36:13.978 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.965+0000 7f06b9b29700 1 -- 172.21.15.23:0/3206543393 learned_addr learned my addr 172.21.15.23:0/3206543393 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:36:13.978 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.965+0000 7f06ba32a700 1 --2- 172.21.15.23:0/3206543393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06b40aaab0 0x7f06b409ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:13.978 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.965+0000 7f06bab2b700 1 --2- 172.21.15.23:0/3206543393 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06b40a8000 0x7f06b40a4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:13.979 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.965+0000 7f06bab2b700 1 -- 172.21.15.23:0/3206543393 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06b40a8000 msgr2=0x7f06b40a4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2020-11-20T11:36:13.979 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.965+0000 7f06bab2b700 1 -- 172.21.15.23:0/3206543393 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06b40a8000 msgr2=0x7f06b40a4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:36:13.979 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.965+0000 7f06b9b29700 1 -- 172.21.15.23:0/3206543393 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06b40a8000 msgr2=0x7f06b40a4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:13.979 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.965+0000 7f06bab2b700 1 --2- 172.21.15.23:0/3206543393 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06b40a8000 0x7f06b40a4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:36:13.979 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.965+0000 7f06bab2b700 1 --2- 172.21.15.23:0/3206543393 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06b40a8000 0x7f06b40a4680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:36:13.980 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.965+0000 7f06b9b29700 1 --2- 172.21.15.23:0/3206543393 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06b40a8000 0x7f06b40a4680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:13.980 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.965+0000 7f06b9b29700 1 -- 172.21.15.23:0/3206543393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06b40aaab0 msgr2=0x7f06b409ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:13.980 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.965+0000 7f06b9b29700 1 --2- 172.21.15.23:0/3206543393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06b40aaab0 0x7f06b409ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:13.980 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.965+0000 7f06b9b29700 1 -- 172.21.15.23:0/3206543393 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b40a5640 con 0x7f06b40aae90 2020-11-20T11:36:13.980 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.965+0000 7f06b9b29700 1 --2- 172.21.15.23:0/3206543393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06b40aae90 0x7f06b40a62c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f06a400eba0 tx=0x7f06a4006e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:13.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.966+0000 7f06a27fc700 1 -- 172.21.15.23:0/3206543393 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a400c920 con 0x7f06b40aae90 2020-11-20T11:36:13.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.966+0000 7f06a27fc700 1 -- 172.21.15.23:0/3206543393 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a400ca80 con 0x7f06b40aae90 2020-11-20T11:36:13.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.966+0000 7f06a27fc700 1 -- 172.21.15.23:0/3206543393 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a400ce00 con 0x7f06b40aae90 2020-11-20T11:36:13.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.966+0000 7f06bc58e700 1 -- 172.21.15.23:0/3206543393 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06b40a58e0 con 0x7f06b40aae90 2020-11-20T11:36:13.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.966+0000 7f06bc58e700 1 -- 172.21.15.23:0/3206543393 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06b40a4db0 con 0x7f06b40aae90 2020-11-20T11:36:13.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.967+0000 7f06bc58e700 1 -- 172.21.15.23:0/3206543393 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06b405c7d0 con 0x7f06b40aae90 2020-11-20T11:36:13.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.967+0000 7f06a27fc700 1 -- 172.21.15.23:0/3206543393 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f06a4012600 con 0x7f06b40aae90 2020-11-20T11:36:13.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.967+0000 7f06a27fc700 1 --2- 172.21.15.23:0/3206543393 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f068c056d60 0x7f068c059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:13.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.967+0000 7f06a27fc700 1 -- 172.21.15.23:0/3206543393 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f06a4070450 con 0x7f06b40aae90 2020-11-20T11:36:13.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.968+0000 7f06ba32a700 1 --2- 172.21.15.23:0/3206543393 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f068c056d60 0x7f068c059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:13.983 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.968+0000 7f06ba32a700 1 --2- 172.21.15.23:0/3206543393 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f068c056d60 0x7f068c059220 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f06b0006f30 tx=0x7f06b0016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:13.983 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:13.971+0000 7f06a27fc700 1 -- 172.21.15.23:0/3206543393 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f06a4045740 con 0x7f06b40aae90 2020-11-20T11:36:14.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:14.212+0000 7f06bc58e700 1 -- 172.21.15.23:0/3206543393 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f06b4044710 con 0x7f06b40aae90 2020-11-20T11:36:14.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:14.212+0000 7f06a27fc700 1 -- 172.21.15.23:0/3206543393 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1472 (secure 0 0 0) 0x7f06a4048d60 con 0x7f06b40aae90 2020-11-20T11:36:14.216 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":32,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:36:14.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:14.214+0000 7f06bc58e700 1 -- 172.21.15.23:0/3206543393 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f068c056d60 msgr2=0x7f068c059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:14.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:14.214+0000 7f06bc58e700 1 --2- 172.21.15.23:0/3206543393 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f068c056d60 0x7f068c059220 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f06b0006f30 tx=0x7f06b0016000).stop 2020-11-20T11:36:14.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:14.214+0000 7f06bc58e700 1 -- 172.21.15.23:0/3206543393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06b40aae90 msgr2=0x7f06b40a62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:14.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:14.214+0000 7f06bc58e700 1 --2- 172.21.15.23:0/3206543393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06b40aae90 0x7f06b40a62c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f06a400eba0 tx=0x7f06a4006e30).stop 2020-11-20T11:36:14.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:14.214+0000 7f06bc58e700 1 -- 172.21.15.23:0/3206543393 shutdown_connections 2020-11-20T11:36:14.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:14.214+0000 7f06bc58e700 1 --2- 172.21.15.23:0/3206543393 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f068c056d60 0x7f068c059220 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:14.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:14.214+0000 7f06bc58e700 1 --2- 172.21.15.23:0/3206543393 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f06b40a8000 0x7f06b40a4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:14.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:14.214+0000 7f06bc58e700 1 --2- 172.21.15.23:0/3206543393 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f06b40aaab0 0x7f06b409ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:14.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:14.214+0000 7f06bc58e700 1 --2- 172.21.15.23:0/3206543393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06b40aae90 0x7f06b40a62c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:14.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:14.214+0000 7f06bc58e700 1 -- 172.21.15.23:0/3206543393 >> 172.21.15.23:0/3206543393 conn(0x7f06b40678e0 msgr2=0x7f06b40aba00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:14.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:14.214+0000 7f06bc58e700 1 -- 172.21.15.23:0/3206543393 shutdown_connections 2020-11-20T11:36:14.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:14.214+0000 7f06bc58e700 1 -- 172.21.15.23:0/3206543393 wait complete. 2020-11-20T11:36:17.229 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:36:17.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.347+0000 7ffb639c4700 1 Processor -- start 2020-11-20T11:36:17.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.347+0000 7ffb639c4700 1 -- start start 2020-11-20T11:36:17.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.348+0000 7ffb639c4700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ffb5c067fd0 0x7ffb5c13bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:17.354 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.348+0000 7ffb639c4700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb5c0a8c20 con 0x7ffb5c05bc80 2020-11-20T11:36:17.354 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.348+0000 7ffb639c4700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb5c0684e0 con 0x7ffb5c0a9080 2020-11-20T11:36:17.354 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.348+0000 7ffb639c4700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb5c0a94f0 con 0x7ffb5c067fd0 2020-11-20T11:36:17.355 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.348+0000 7ffb60f5f700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ffb5c067fd0 0x7ffb5c13bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:17.355 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.348+0000 7ffb61760700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7ffb5c05bc80 0x7ffb5c0a6ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:56954/0 (socket says 172.21.15.23:56954) 2020-11-20T11:36:17.355 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.348+0000 7ffb60f5f700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ffb5c067fd0 0x7ffb5c13bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42706/0 (socket says 172.21.15.23:42706) 2020-11-20T11:36:17.355 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.348+0000 7ffb60f5f700 1 -- 172.21.15.23:0/3856960378 learned_addr learned my addr 172.21.15.23:0/3856960378 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:36:17.356 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.348+0000 7ffb60f5f700 1 -- 172.21.15.23:0/3856960378 >> v1:172.21.15.155:6789/0 conn(0x7ffb5c0a9080 legacy=0x7ffb5c143300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:17.356 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.348+0000 7ffb60f5f700 1 -- 172.21.15.23:0/3856960378 >> v1:172.21.15.23:6789/0 conn(0x7ffb5c05bc80 legacy=0x7ffb5c0a6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:17.356 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.348+0000 7ffb60f5f700 1 -- 172.21.15.23:0/3856960378 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb5c05c060 con 0x7ffb5c067fd0 2020-11-20T11:36:17.356 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.348+0000 7ffb60f5f700 1 --2- 172.21.15.23:0/3856960378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb5c067fd0 0x7ffb5c13bfa0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7ffb4c009a10 tx=0x7ffb4c006ef0).ready entity=mon.0 client_cookie=c6e3b38802b563c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:17.356 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.348+0000 7ffb4bfff700 1 -- 172.21.15.23:0/3856960378 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c00b860 con 0x7ffb5c067fd0 2020-11-20T11:36:17.357 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.349+0000 7ffb4bfff700 1 -- 172.21.15.23:0/3856960378 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb4c000e90 con 0x7ffb5c067fd0 2020-11-20T11:36:17.357 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.349+0000 7ffb4bfff700 1 -- 172.21.15.23:0/3856960378 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c00bbd0 con 0x7ffb5c067fd0 2020-11-20T11:36:17.357 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.349+0000 7ffb639c4700 1 -- 172.21.15.23:0/3856960378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb5c067fd0 msgr2=0x7ffb5c13bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:17.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.349+0000 7ffb639c4700 1 --2- 172.21.15.23:0/3856960378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb5c067fd0 0x7ffb5c13bfa0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7ffb4c009a10 tx=0x7ffb4c006ef0).stop 2020-11-20T11:36:17.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.349+0000 7ffb639c4700 1 -- 172.21.15.23:0/3856960378 shutdown_connections 2020-11-20T11:36:17.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.349+0000 7ffb639c4700 1 --2- 172.21.15.23:0/3856960378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb5c067fd0 0x7ffb5c13bfa0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:17.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.349+0000 7ffb639c4700 1 -- 172.21.15.23:0/3856960378 >> 172.21.15.23:0/3856960378 conn(0x7ffb5c0ad7a0 msgr2=0x7ffb5c0adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:17.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.349+0000 7ffb639c4700 1 -- 172.21.15.23:0/3856960378 shutdown_connections 2020-11-20T11:36:17.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.349+0000 7ffb639c4700 1 -- 172.21.15.23:0/3856960378 wait complete. 2020-11-20T11:36:17.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.350+0000 7ffb639c4700 1 Processor -- start 2020-11-20T11:36:17.360 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.350+0000 7ffb639c4700 1 -- start start 2020-11-20T11:36:17.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.351+0000 7ffb639c4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb5c05bc80 0x7ffb5c0acb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:17.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.351+0000 7ffb639c4700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb5c067fd0 0x7ffb5c134aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:17.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.351+0000 7ffb639c4700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb5c0a9080 0x7ffb5c0a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:17.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.351+0000 7ffb639c4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5c0a3070 con 0x7ffb5c05bc80 2020-11-20T11:36:17.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.351+0000 7ffb639c4700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5c0a3ac0 con 0x7ffb5c0a9080 2020-11-20T11:36:17.363 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.351+0000 7ffb639c4700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5c0a2020 con 0x7ffb5c067fd0 2020-11-20T11:36:17.363 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.351+0000 7ffb61760700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb5c05bc80 0x7ffb5c0acb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:17.363 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.351+0000 7ffb61760700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb5c05bc80 0x7ffb5c0acb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42710/0 (socket says 172.21.15.23:42710) 2020-11-20T11:36:17.364 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.352+0000 7ffb61760700 1 -- 172.21.15.23:0/1815377843 learned_addr learned my addr 172.21.15.23:0/1815377843 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:36:17.364 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.352+0000 7ffb61f61700 1 --2- 172.21.15.23:0/1815377843 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb5c0a9080 0x7ffb5c0a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:17.364 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.352+0000 7ffb60f5f700 1 --2- 172.21.15.23:0/1815377843 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb5c067fd0 0x7ffb5c134aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:17.365 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.352+0000 7ffb61760700 1 -- 172.21.15.23:0/1815377843 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb5c067fd0 msgr2=0x7ffb5c134aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:17.365 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.352+0000 7ffb61760700 1 --2- 172.21.15.23:0/1815377843 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb5c067fd0 0x7ffb5c134aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:17.365 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.352+0000 7ffb61760700 1 -- 172.21.15.23:0/1815377843 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb5c0a9080 msgr2=0x7ffb5c0a3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:17.365 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.352+0000 7ffb61760700 1 --2- 172.21.15.23:0/1815377843 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb5c0a9080 0x7ffb5c0a3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:17.366 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.352+0000 7ffb61760700 1 -- 172.21.15.23:0/1815377843 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb5c0a2250 con 0x7ffb5c05bc80 2020-11-20T11:36:17.366 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.352+0000 7ffb61f61700 1 --2- 172.21.15.23:0/1815377843 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb5c0a9080 0x7ffb5c0a3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:36:17.366 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.352+0000 7ffb61760700 1 --2- 172.21.15.23:0/1815377843 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb5c05bc80 0x7ffb5c0acb90 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7ffb5800c710 tx=0x7ffb58007c80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:17.366 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.352+0000 7ffb497fa700 1 -- 172.21.15.23:0/1815377843 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb58007320 con 0x7ffb5c05bc80 2020-11-20T11:36:17.366 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.352+0000 7ffb497fa700 1 -- 172.21.15.23:0/1815377843 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb58005570 con 0x7ffb5c05bc80 2020-11-20T11:36:17.367 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.352+0000 7ffb639c4700 1 -- 172.21.15.23:0/1815377843 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb5c0a24f0 con 0x7ffb5c05bc80 2020-11-20T11:36:17.367 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.352+0000 7ffb497fa700 1 -- 172.21.15.23:0/1815377843 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb580058f0 con 0x7ffb5c05bc80 2020-11-20T11:36:17.367 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.352+0000 7ffb639c4700 1 -- 172.21.15.23:0/1815377843 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb5c0a4500 con 0x7ffb5c05bc80 2020-11-20T11:36:17.367 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.354+0000 7ffb639c4700 1 -- 172.21.15.23:0/1815377843 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb5c062fe0 con 0x7ffb5c05bc80 2020-11-20T11:36:17.367 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.354+0000 7ffb497fa700 1 -- 172.21.15.23:0/1815377843 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ffb58010030 con 0x7ffb5c05bc80 2020-11-20T11:36:17.368 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.354+0000 7ffb497fa700 1 --2- 172.21.15.23:0/1815377843 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ffb34056d60 0x7ffb34059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:17.368 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.354+0000 7ffb497fa700 1 -- 172.21.15.23:0/1815377843 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7ffb5806feb0 con 0x7ffb5c05bc80 2020-11-20T11:36:17.368 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.354+0000 7ffb60f5f700 1 --2- 172.21.15.23:0/1815377843 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ffb34056d60 0x7ffb34059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:17.368 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.355+0000 7ffb60f5f700 1 --2- 172.21.15.23:0/1815377843 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ffb34056d60 0x7ffb34059220 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7ffb5c068410 tx=0x7ffb4c016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:17.368 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.359+0000 7ffb497fa700 1 -- 172.21.15.23:0/1815377843 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ffb58045090 con 0x7ffb5c05bc80 2020-11-20T11:36:17.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:17.587+0000 7ffb639c4700 1 -- 172.21.15.23:0/1815377843 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ffb5c044710 con 0x7ffb5c05bc80 2020-11-20T11:36:22.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:22.402+0000 7ffb497fa700 1 -- 172.21.15.23:0/1815377843 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1471 (secure 0 0 0) 0x7ffb580486b0 con 0x7ffb5c05bc80 2020-11-20T11:36:22.406 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":36,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:36:22.408 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:22.404+0000 7ffb639c4700 1 -- 172.21.15.23:0/1815377843 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ffb34056d60 msgr2=0x7ffb34059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:22.408 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:22.404+0000 7ffb639c4700 1 --2- 172.21.15.23:0/1815377843 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ffb34056d60 0x7ffb34059220 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7ffb5c068410 tx=0x7ffb4c016000).stop 2020-11-20T11:36:22.409 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:22.404+0000 7ffb639c4700 1 -- 172.21.15.23:0/1815377843 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb5c05bc80 msgr2=0x7ffb5c0acb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:22.409 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:22.404+0000 7ffb639c4700 1 --2- 172.21.15.23:0/1815377843 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb5c05bc80 0x7ffb5c0acb90 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7ffb5800c710 tx=0x7ffb58007c80).stop 2020-11-20T11:36:22.409 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:22.404+0000 7ffb639c4700 1 -- 172.21.15.23:0/1815377843 shutdown_connections 2020-11-20T11:36:22.409 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:22.404+0000 7ffb639c4700 1 --2- 172.21.15.23:0/1815377843 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffb5c0a9080 0x7ffb5c0a3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:22.410 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:22.404+0000 7ffb639c4700 1 --2- 172.21.15.23:0/1815377843 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb5c05bc80 0x7ffb5c0acb90 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:22.410 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:22.404+0000 7ffb639c4700 1 --2- 172.21.15.23:0/1815377843 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ffb34056d60 0x7ffb34059220 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:22.410 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:22.404+0000 7ffb639c4700 1 --2- 172.21.15.23:0/1815377843 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffb5c067fd0 0x7ffb5c134aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:22.410 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:22.404+0000 7ffb639c4700 1 -- 172.21.15.23:0/1815377843 >> 172.21.15.23:0/1815377843 conn(0x7ffb5c0ad7a0 msgr2=0x7ffb5c139600 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:22.410 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:22.404+0000 7ffb639c4700 1 -- 172.21.15.23:0/1815377843 shutdown_connections 2020-11-20T11:36:22.411 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:22.404+0000 7ffb639c4700 1 -- 172.21.15.23:0/1815377843 wait complete. 2020-11-20T11:36:25.422 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:36:25.555 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.551+0000 7f72fd4a6700 1 Processor -- start 2020-11-20T11:36:25.555 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.551+0000 7f72fd4a6700 1 -- start start 2020-11-20T11:36:25.556 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.551+0000 7f72fd4a6700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f72f80aaf00 0x7f72f813e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:25.557 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.552+0000 7f72fd4a6700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f72f80a85c0 0x7f72f813a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:25.557 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.552+0000 7f72fd4a6700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f813adb0 con 0x7f72f80ab2e0 2020-11-20T11:36:25.557 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.552+0000 7f72fd4a6700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72f813e010 con 0x7f72f80aaf00 2020-11-20T11:36:25.557 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.552+0000 7f72fd4a6700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f813e2b0 con 0x7f72f80a85c0 2020-11-20T11:36:25.558 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.552+0000 7f72f67fc700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f72f80a85c0 0x7f72f813a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:25.558 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.552+0000 7f72f6ffd700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f72f80aaf00 0x7f72f813e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:25.558 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.552+0000 7f72f6ffd700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f72f80aaf00 0x7f72f813e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:35616/0 (socket says 172.21.15.23:35616) 2020-11-20T11:36:25.558 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.552+0000 7f72f77fe700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f72f80ab2e0 0x7f72f80a78b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:34948/0 (socket says 172.21.15.23:34948) 2020-11-20T11:36:25.558 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.552+0000 7f72f6ffd700 1 -- 172.21.15.23:0/1402937214 learned_addr learned my addr 172.21.15.23:0/1402937214 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:36:25.559 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.552+0000 7f72f67fc700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f72f80a85c0 0x7f72f813a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:52962/0 (socket says 172.21.15.23:52962) 2020-11-20T11:36:25.559 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.552+0000 7f72f6ffd700 1 -- 172.21.15.23:0/1402937214 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f72f80a85c0 msgr2=0x7f72f813a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:36:25.559 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.552+0000 7f72f6ffd700 1 --2- 172.21.15.23:0/1402937214 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f72f80a85c0 0x7f72f813a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:25.559 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.552+0000 7f72f6ffd700 1 -- 172.21.15.23:0/1402937214 >> v1:172.21.15.155:6789/0 conn(0x7f72f80ab2e0 legacy=0x7f72f80a78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:25.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.552+0000 7f72f6ffd700 1 -- 172.21.15.23:0/1402937214 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f813e550 con 0x7f72f80aaf00 2020-11-20T11:36:25.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.552+0000 7f72f5ffb700 1 -- 172.21.15.23:0/1402937214 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3019320064 0 0) 0x7f72f0001490 con 0x7f72f80ab2e0 2020-11-20T11:36:25.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.552+0000 7f72f67fc700 1 --2- 172.21.15.23:0/1402937214 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f72f80a85c0 0x7f72f813a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:36:25.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.553+0000 7f72f6ffd700 1 --2- 172.21.15.23:0/1402937214 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72f80aaf00 0x7f72f813e9f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f72e8009a10 tx=0x7f72e8006ef0).ready entity=mon.1 client_cookie=62515240889a753d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:25.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.553+0000 7f72f5ffb700 1 -- 172.21.15.23:0/1402937214 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e800b860 con 0x7f72f80aaf00 2020-11-20T11:36:25.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.553+0000 7f72f5ffb700 1 -- 172.21.15.23:0/1402937214 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e8000e90 con 0x7f72f80aaf00 2020-11-20T11:36:25.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.553+0000 7f72f5ffb700 1 -- 172.21.15.23:0/1402937214 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e800bbd0 con 0x7f72f80aaf00 2020-11-20T11:36:25.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.553+0000 7f72fd4a6700 1 -- 172.21.15.23:0/1402937214 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72f80aaf00 msgr2=0x7f72f813e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:25.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.553+0000 7f72fd4a6700 1 --2- 172.21.15.23:0/1402937214 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72f80aaf00 0x7f72f813e9f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f72e8009a10 tx=0x7f72e8006ef0).stop 2020-11-20T11:36:25.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.553+0000 7f72fd4a6700 1 -- 172.21.15.23:0/1402937214 shutdown_connections 2020-11-20T11:36:25.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.553+0000 7f72fd4a6700 1 --2- 172.21.15.23:0/1402937214 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72f80aaf00 0x7f72f813e9f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:25.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.553+0000 7f72fd4a6700 1 --2- 172.21.15.23:0/1402937214 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f72f80a85c0 0x7f72f813a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:25.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.553+0000 7f72fd4a6700 1 -- 172.21.15.23:0/1402937214 >> 172.21.15.23:0/1402937214 conn(0x7f72f80678e0 msgr2=0x7f72f8067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:25.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.553+0000 7f72fd4a6700 1 -- 172.21.15.23:0/1402937214 shutdown_connections 2020-11-20T11:36:25.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.554+0000 7f72fd4a6700 1 -- 172.21.15.23:0/1402937214 wait complete. 2020-11-20T11:36:25.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.555+0000 7f72fd4a6700 1 Processor -- start 2020-11-20T11:36:25.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.555+0000 7f72fd4a6700 1 -- start start 2020-11-20T11:36:25.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.555+0000 7f72fd4a6700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f72f80a85c0 0x7f72f809f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:25.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.555+0000 7f72fd4a6700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72f80aaf00 0x7f72f80a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:25.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.555+0000 7f72fd4a6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72f80ab2e0 0x7f72f80a6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:25.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.555+0000 7f72fd4a6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f809e650 con 0x7f72f80ab2e0 2020-11-20T11:36:25.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.555+0000 7f72fd4a6700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f809e8f0 con 0x7f72f80aaf00 2020-11-20T11:36:25.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.555+0000 7f72fd4a6700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f809eb90 con 0x7f72f80a85c0 2020-11-20T11:36:25.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.556+0000 7f72f77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72f80ab2e0 0x7f72f80a6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:25.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.556+0000 7f72f77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72f80ab2e0 0x7f72f80a6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42730/0 (socket says 172.21.15.23:42730) 2020-11-20T11:36:25.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.556+0000 7f72f77fe700 1 -- 172.21.15.23:0/2346473001 learned_addr learned my addr 172.21.15.23:0/2346473001 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:36:25.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.556+0000 7f72f6ffd700 1 --2- 172.21.15.23:0/2346473001 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f72f80a85c0 0x7f72f809f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:25.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.556+0000 7f72f67fc700 1 --2- 172.21.15.23:0/2346473001 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72f80aaf00 0x7f72f80a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:25.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.556+0000 7f72f77fe700 1 -- 172.21.15.23:0/2346473001 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f72f80a85c0 msgr2=0x7f72f809f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:25.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.556+0000 7f72f77fe700 1 --2- 172.21.15.23:0/2346473001 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f72f80a85c0 0x7f72f809f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:25.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.556+0000 7f72f77fe700 1 -- 172.21.15.23:0/2346473001 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72f80aaf00 msgr2=0x7f72f80a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:25.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.556+0000 7f72f77fe700 1 --2- 172.21.15.23:0/2346473001 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72f80aaf00 0x7f72f80a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:25.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.556+0000 7f72f77fe700 1 -- 172.21.15.23:0/2346473001 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f809fd30 con 0x7f72f80ab2e0 2020-11-20T11:36:25.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.556+0000 7f72f77fe700 1 --2- 172.21.15.23:0/2346473001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72f80ab2e0 0x7f72f80a6d10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f72f000cae0 tx=0x7f72f0001360).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:25.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.556+0000 7f72deffd700 1 -- 172.21.15.23:0/2346473001 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f0005b00 con 0x7f72f80ab2e0 2020-11-20T11:36:25.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.556+0000 7f72deffd700 1 -- 172.21.15.23:0/2346473001 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f0005c60 con 0x7f72f80ab2e0 2020-11-20T11:36:25.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.556+0000 7f72fd4a6700 1 -- 172.21.15.23:0/2346473001 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f80a1750 con 0x7f72f80ab2e0 2020-11-20T11:36:25.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.556+0000 7f72deffd700 1 -- 172.21.15.23:0/2346473001 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f0007110 con 0x7f72f80ab2e0 2020-11-20T11:36:25.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.556+0000 7f72fd4a6700 1 -- 172.21.15.23:0/2346473001 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f80a7200 con 0x7f72f80ab2e0 2020-11-20T11:36:25.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.557+0000 7f72fd4a6700 1 -- 172.21.15.23:0/2346473001 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72f809ed50 con 0x7f72f80ab2e0 2020-11-20T11:36:25.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.558+0000 7f72deffd700 1 -- 172.21.15.23:0/2346473001 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f72f0010030 con 0x7f72f80ab2e0 2020-11-20T11:36:25.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.558+0000 7f72deffd700 1 --2- 172.21.15.23:0/2346473001 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f72d4056d60 0x7f72d4059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:25.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.558+0000 7f72deffd700 1 -- 172.21.15.23:0/2346473001 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f72f006fd60 con 0x7f72f80ab2e0 2020-11-20T11:36:25.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.558+0000 7f72f6ffd700 1 --2- 172.21.15.23:0/2346473001 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f72d4056d60 0x7f72d4059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:25.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.559+0000 7f72f6ffd700 1 --2- 172.21.15.23:0/2346473001 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f72d4056d60 0x7f72d4059220 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f72e8006f30 tx=0x7f72e8016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:25.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.562+0000 7f72deffd700 1 -- 172.21.15.23:0/2346473001 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f72f0045050 con 0x7f72f80ab2e0 2020-11-20T11:36:25.779 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.775+0000 7f72fd4a6700 1 -- 172.21.15.23:0/2346473001 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f72f813adb0 con 0x7f72f80ab2e0 2020-11-20T11:36:25.779 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.775+0000 7f72deffd700 1 -- 172.21.15.23:0/2346473001 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1477 (secure 0 0 0) 0x7f72f813adb0 con 0x7f72f80ab2e0 2020-11-20T11:36:25.780 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:36:25.781 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.777+0000 7f72fd4a6700 1 -- 172.21.15.23:0/2346473001 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f72d4056d60 msgr2=0x7f72d4059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:25.781 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.777+0000 7f72fd4a6700 1 --2- 172.21.15.23:0/2346473001 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f72d4056d60 0x7f72d4059220 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f72e8006f30 tx=0x7f72e8016000).stop 2020-11-20T11:36:25.782 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.777+0000 7f72fd4a6700 1 -- 172.21.15.23:0/2346473001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72f80ab2e0 msgr2=0x7f72f80a6d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:25.782 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.777+0000 7f72fd4a6700 1 --2- 172.21.15.23:0/2346473001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72f80ab2e0 0x7f72f80a6d10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f72f000cae0 tx=0x7f72f0001360).stop 2020-11-20T11:36:25.782 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.777+0000 7f72fd4a6700 1 -- 172.21.15.23:0/2346473001 shutdown_connections 2020-11-20T11:36:25.782 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.777+0000 7f72fd4a6700 1 --2- 172.21.15.23:0/2346473001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f72f80ab2e0 0x7f72f80a6d10 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:25.783 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.777+0000 7f72fd4a6700 1 --2- 172.21.15.23:0/2346473001 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f72d4056d60 0x7f72d4059220 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:25.783 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.777+0000 7f72fd4a6700 1 --2- 172.21.15.23:0/2346473001 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f72f80a85c0 0x7f72f809f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:25.783 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.777+0000 7f72fd4a6700 1 --2- 172.21.15.23:0/2346473001 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f72f80aaf00 0x7f72f80a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:25.783 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.777+0000 7f72fd4a6700 1 -- 172.21.15.23:0/2346473001 >> 172.21.15.23:0/2346473001 conn(0x7f72f80678e0 msgr2=0x7f72f813cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:25.783 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.777+0000 7f72fd4a6700 1 -- 172.21.15.23:0/2346473001 shutdown_connections 2020-11-20T11:36:25.784 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.777+0000 7f72fd4a6700 1 -- 172.21.15.23:0/2346473001 wait complete. 2020-11-20T11:36:25.794 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:36:25.794 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:36:25.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.915+0000 7fe5ffdb4700 1 Processor -- start 2020-11-20T11:36:25.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.915+0000 7fe5ffdb4700 1 -- start start 2020-11-20T11:36:25.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.916+0000 7fe5ffdb4700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe5f8065780 0x7fe5f80a9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:25.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.916+0000 7fe5ffdb4700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe5f80a9cf0 0x7fe5f80601b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:25.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.916+0000 7fe5ffdb4700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5f8067430 con 0x7fe5f805bc80 2020-11-20T11:36:25.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.916+0000 7fe5ffdb4700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5f80a8ca0 con 0x7fe5f8065780 2020-11-20T11:36:25.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.916+0000 7fe5ffdb4700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5f805c060 con 0x7fe5f80a9cf0 2020-11-20T11:36:25.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.916+0000 7fe5fd34f700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe5f8065780 0x7fe5f80a9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:25.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.916+0000 7fe5fd34f700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe5f8065780 0x7fe5f80a9140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42736/0 (socket says 172.21.15.23:42736) 2020-11-20T11:36:25.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.916+0000 7fe5fd34f700 1 -- 172.21.15.23:0/3881782849 learned_addr learned my addr 172.21.15.23:0/3881782849 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:36:25.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.916+0000 7fe5fe351700 1 --2- 172.21.15.23:0/3881782849 >> v2:172.21.15.155:3300/0 conn(0x7fe5f80a9cf0 0x7fe5f80601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:25.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.916+0000 7fe5fd34f700 1 -- 172.21.15.23:0/3881782849 >> v2:172.21.15.155:3300/0 conn(0x7fe5f80a9cf0 msgr2=0x7fe5f80601b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:36:25.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.916+0000 7fe5fd34f700 1 --2- 172.21.15.23:0/3881782849 >> v2:172.21.15.155:3300/0 conn(0x7fe5f80a9cf0 0x7fe5f80601b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:25.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.916+0000 7fe5fd34f700 1 -- 172.21.15.23:0/3881782849 >> v1:172.21.15.155:6789/0 conn(0x7fe5f805bc80 legacy=0x7fe5f81379e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:25.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.916+0000 7fe5fd34f700 1 -- 172.21.15.23:0/3881782849 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f80a8810 con 0x7fe5f8065780 2020-11-20T11:36:25.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.917+0000 7fe5fd34f700 1 --2- 172.21.15.23:0/3881782849 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5f8065780 0x7fe5f80a9140 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fe5f8061930 tx=0x7fe5e8006ef0).ready entity=mon.0 client_cookie=7c18f4eaa48cd478 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:25.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.917+0000 7fe5fcb4e700 1 -- 172.21.15.23:0/3881782849 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e800b860 con 0x7fe5f8065780 2020-11-20T11:36:25.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.917+0000 7fe5fcb4e700 1 -- 172.21.15.23:0/3881782849 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5e8000e90 con 0x7fe5f8065780 2020-11-20T11:36:25.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.917+0000 7fe5fcb4e700 1 -- 172.21.15.23:0/3881782849 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e800bbd0 con 0x7fe5f8065780 2020-11-20T11:36:25.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.917+0000 7fe5ffdb4700 1 -- 172.21.15.23:0/3881782849 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5f8065780 msgr2=0x7fe5f80a9140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:25.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.917+0000 7fe5ffdb4700 1 --2- 172.21.15.23:0/3881782849 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5f8065780 0x7fe5f80a9140 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fe5f8061930 tx=0x7fe5e8006ef0).stop 2020-11-20T11:36:25.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.917+0000 7fe5ffdb4700 1 -- 172.21.15.23:0/3881782849 shutdown_connections 2020-11-20T11:36:25.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.917+0000 7fe5ffdb4700 1 --2- 172.21.15.23:0/3881782849 >> v2:172.21.15.155:3300/0 conn(0x7fe5f80a9cf0 0x7fe5f80601b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:25.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.917+0000 7fe5ffdb4700 1 --2- 172.21.15.23:0/3881782849 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5f8065780 0x7fe5f80a9140 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:25.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.917+0000 7fe5ffdb4700 1 -- 172.21.15.23:0/3881782849 >> 172.21.15.23:0/3881782849 conn(0x7fe5f80ad7a0 msgr2=0x7fe5f80adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:25.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.917+0000 7fe5ffdb4700 1 -- 172.21.15.23:0/3881782849 shutdown_connections 2020-11-20T11:36:25.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.917+0000 7fe5ffdb4700 1 -- 172.21.15.23:0/3881782849 wait complete. 2020-11-20T11:36:25.929 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.919+0000 7fe5ffdb4700 1 Processor -- start 2020-11-20T11:36:25.929 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.919+0000 7fe5ffdb4700 1 -- start start 2020-11-20T11:36:25.930 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.919+0000 7fe5ffdb4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5f805bc80 0x7fe5f814d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:25.930 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.919+0000 7fe5ffdb4700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5f8065780 0x7fe5f80a1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:25.930 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.919+0000 7fe5ffdb4700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe5f80a9cf0 0x7fe5f809fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:25.930 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.919+0000 7fe5ffdb4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f80a0310 con 0x7fe5f805bc80 2020-11-20T11:36:25.930 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.919+0000 7fe5ffdb4700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f8138b20 con 0x7fe5f8065780 2020-11-20T11:36:25.931 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.919+0000 7fe5ffdb4700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f814dc30 con 0x7fe5f80a9cf0 2020-11-20T11:36:25.931 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.919+0000 7fe5fdb50700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5f805bc80 0x7fe5f814d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:25.931 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.919+0000 7fe5fdb50700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5f805bc80 0x7fe5f814d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42740/0 (socket says 172.21.15.23:42740) 2020-11-20T11:36:25.931 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.920+0000 7fe5fdb50700 1 -- 172.21.15.23:0/3078485952 learned_addr learned my addr 172.21.15.23:0/3078485952 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:36:25.931 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.920+0000 7fe5fd34f700 1 --2- 172.21.15.23:0/3078485952 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5f8065780 0x7fe5f80a1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:25.932 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.920+0000 7fe5fe351700 1 --2- 172.21.15.23:0/3078485952 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe5f80a9cf0 0x7fe5f809fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:25.932 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.920+0000 7fe5fdb50700 1 -- 172.21.15.23:0/3078485952 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe5f80a9cf0 msgr2=0x7fe5f809fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:25.932 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.920+0000 7fe5fdb50700 1 --2- 172.21.15.23:0/3078485952 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe5f80a9cf0 0x7fe5f809fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:25.932 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.920+0000 7fe5fdb50700 1 -- 172.21.15.23:0/3078485952 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5f8065780 msgr2=0x7fe5f80a1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:25.932 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.920+0000 7fe5fdb50700 1 --2- 172.21.15.23:0/3078485952 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe5f8065780 0x7fe5f80a1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:25.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.920+0000 7fe5fdb50700 1 -- 172.21.15.23:0/3078485952 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f814ded0 con 0x7fe5f805bc80 2020-11-20T11:36:25.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.920+0000 7fe5fdb50700 1 --2- 172.21.15.23:0/3078485952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5f805bc80 0x7fe5f814d590 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fe5f400c710 tx=0x7fe5f4007c90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:25.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.920+0000 7fe5e5ffb700 1 -- 172.21.15.23:0/3078485952 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f4007020 con 0x7fe5f805bc80 2020-11-20T11:36:25.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.920+0000 7fe5e5ffb700 1 -- 172.21.15.23:0/3078485952 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f4001e90 con 0x7fe5f805bc80 2020-11-20T11:36:25.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.920+0000 7fe5ffdb4700 1 -- 172.21.15.23:0/3078485952 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5f814e170 con 0x7fe5f805bc80 2020-11-20T11:36:25.934 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.920+0000 7fe5e5ffb700 1 -- 172.21.15.23:0/3078485952 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f40073b0 con 0x7fe5f805bc80 2020-11-20T11:36:25.934 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.920+0000 7fe5ffdb4700 1 -- 172.21.15.23:0/3078485952 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5f80a7570 con 0x7fe5f805bc80 2020-11-20T11:36:25.934 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.921+0000 7fe5ffdb4700 1 --2- 172.21.15.23:0/3078485952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5f80a7820 0x7fe5f8149fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:25.934 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.921+0000 7fe5ffdb4700 1 -- 172.21.15.23:0/3078485952 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f80a7c00 con 0x7fe5f80a7820 2020-11-20T11:36:25.934 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.921+0000 7fe5fd34f700 1 --2- 172.21.15.23:0/3078485952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5f80a7820 0x7fe5f8149fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:25.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.921+0000 7fe5e5ffb700 1 -- 172.21.15.23:0/3078485952 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe5f4010070 con 0x7fe5f805bc80 2020-11-20T11:36:25.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.922+0000 7fe5fd34f700 1 -- 172.21.15.23:0/3078485952 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe5f8062fe0 con 0x7fe5f80a7820 2020-11-20T11:36:25.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.922+0000 7fe5fd34f700 1 --2- 172.21.15.23:0/3078485952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5f80a7820 0x7fe5f8149fc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fe5f8061930 tx=0x7fe5e8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:25.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.922+0000 7fe5e5ffb700 1 --2- 172.21.15.23:0/3078485952 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe5d0056d60 0x7fe5d0059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:25.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.922+0000 7fe5e5ffb700 1 -- 172.21.15.23:0/3078485952 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fe5f406f3b0 con 0x7fe5f805bc80 2020-11-20T11:36:25.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.922+0000 7fe5fe351700 1 --2- 172.21.15.23:0/3078485952 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe5d0056d60 0x7fe5d0059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:25.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.922+0000 7fe5e5ffb700 1 -- 172.21.15.23:0/3078485952 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e8011b30 con 0x7fe5f80a7820 2020-11-20T11:36:25.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.923+0000 7fe5fe351700 1 --2- 172.21.15.23:0/3078485952 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe5d0056d60 0x7fe5d0059220 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fe5f0009d40 tx=0x7fe5f0008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:25.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.923+0000 7fe5e5ffb700 1 -- 172.21.15.23:0/3078485952 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fe5e8011c90 con 0x7fe5f80a7820 2020-11-20T11:36:25.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.923+0000 7fe5e5ffb700 1 -- 172.21.15.23:0/3078485952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5f80a7820 msgr2=0x7fe5f8149fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:25.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.923+0000 7fe5e5ffb700 1 --2- 172.21.15.23:0/3078485952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5f80a7820 0x7fe5f8149fc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fe5f8061930 tx=0x7fe5e8016000).stop 2020-11-20T11:36:25.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.923+0000 7fe5e5ffb700 1 -- 172.21.15.23:0/3078485952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5f80a7820 msgr2=0x7fe5f8149fc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:36:25.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.923+0000 7fe5e5ffb700 1 --2- 172.21.15.23:0/3078485952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5f80a7820 0x7fe5f8149fc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:25.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.930+0000 7fe5ffdb4700 1 --2- 172.21.15.23:0/3078485952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5f814a4b0 0x7fe5f809e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:25.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.930+0000 7fe5ffdb4700 1 -- 172.21.15.23:0/3078485952 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f8044710 con 0x7fe5f814a4b0 2020-11-20T11:36:25.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.930+0000 7fe5fd34f700 1 --2- 172.21.15.23:0/3078485952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5f814a4b0 0x7fe5f809e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:25.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.930+0000 7fe5fd34f700 1 -- 172.21.15.23:0/3078485952 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe5f8069770 con 0x7fe5f814a4b0 2020-11-20T11:36:25.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.931+0000 7fe5fd34f700 1 --2- 172.21.15.23:0/3078485952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5f814a4b0 0x7fe5f809e5d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fe5e801d000 tx=0x7fe5e801f040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:25.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.931+0000 7fe5e5ffb700 1 -- 172.21.15.23:0/3078485952 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e8026810 con 0x7fe5f814a4b0 2020-11-20T11:36:25.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.931+0000 7fe5e5ffb700 1 -- 172.21.15.23:0/3078485952 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4785 (secure 0 0 0) 0x7fe5e802c040 con 0x7fe5f814a4b0 2020-11-20T11:36:25.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.931+0000 7fe5e5ffb700 1 -- 172.21.15.23:0/3078485952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5f814a4b0 msgr2=0x7fe5f809e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:25.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.931+0000 7fe5e5ffb700 1 --2- 172.21.15.23:0/3078485952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5f814a4b0 0x7fe5f809e5d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fe5e801d000 tx=0x7fe5e801f040).stop 2020-11-20T11:36:25.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.931+0000 7fe5e5ffb700 1 -- 172.21.15.23:0/3078485952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5f814a4b0 msgr2=0x7fe5f809e5d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:36:25.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.931+0000 7fe5e5ffb700 1 --2- 172.21.15.23:0/3078485952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5f814a4b0 0x7fe5f809e5d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:25.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.931+0000 7fe5ffdb4700 1 -- 172.21.15.23:0/3078485952 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe5d0056d60 msgr2=0x7fe5d0059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:25.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.931+0000 7fe5ffdb4700 1 --2- 172.21.15.23:0/3078485952 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe5d0056d60 0x7fe5d0059220 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fe5f0009d40 tx=0x7fe5f0008040).stop 2020-11-20T11:36:25.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.931+0000 7fe5ffdb4700 1 -- 172.21.15.23:0/3078485952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5f805bc80 msgr2=0x7fe5f814d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:25.942 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.931+0000 7fe5ffdb4700 1 --2- 172.21.15.23:0/3078485952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe5f805bc80 0x7fe5f814d590 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fe5f400c710 tx=0x7fe5f4007c90).stop 2020-11-20T11:36:25.942 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.931+0000 7fe5fe351700 1 -- 172.21.15.23:0/3078485952 reap_dead start 2020-11-20T11:36:25.942 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.931+0000 7fe5fe351700 1 -- 172.21.15.23:0/3078485952 reap_dead start 2020-11-20T11:36:25.942 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.931+0000 7fe5ffdb4700 1 -- 172.21.15.23:0/3078485952 shutdown_connections 2020-11-20T11:36:25.942 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.931+0000 7fe5ffdb4700 1 -- 172.21.15.23:0/3078485952 >> 172.21.15.23:0/3078485952 conn(0x7fe5f80ad7a0 msgr2=0x7fe5f80a9180 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:25.943 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.932+0000 7fe5ffdb4700 1 -- 172.21.15.23:0/3078485952 shutdown_connections 2020-11-20T11:36:25.943 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:25.932+0000 7fe5ffdb4700 1 -- 172.21.15.23:0/3078485952 wait complete. 2020-11-20T11:36:25.943 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:36:25.943 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:36:25.944 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:36:25.944 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:36:25.944 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 38, 2020-11-20T11:36:25.944 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:36:25.944 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:36:25.945 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:36:25.945 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:36:25.945 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:25.945 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 2, 2020-11-20T11:36:25.945 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:36:25.946 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:36:25.946 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:36:25.946 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:36:25.946 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:36:25.946 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:36:25.947 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:36:25.947 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:36:25.947 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:36:25.947 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:36:25.947 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:36:25.947 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:25.948 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:36:25.948 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:36:25.948 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:36:25.948 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:36:25.948 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:36:25.949 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:36:25.949 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:36:25.949 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:36:25.949 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:36:25.949 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:36:25.950 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:25.950 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:25.950 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:36:25.950 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:36:25.950 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:36:25.951 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:36:25.951 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:36:25.951 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:36:25.951 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:36:25.951 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:36:25.952 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:36:25.952 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:36:25.952 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:36:25.952 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:36:25.952 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:36:25.953 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:36:25.953 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:36:25.953 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:36:25.953 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:36:25.953 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:36:25.954 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:36:25.954 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:36:25.954 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:36:25.954 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:36:25.954 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:36:25.954 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:25.955 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:36:25.955 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:25.955 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:36:25.955 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:25.955 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:36:25.956 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:36:25.956 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:36:25.956 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:36:25.956 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:25.956 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:36:25.957 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:36:25.957 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:25.957 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:25.957 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:25.957 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:36:25.958 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:36:25.958 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:25.958 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:25.958 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:25.958 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:25.959 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:36:25.959 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:36:25.959 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:36:25.959 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:36:25.959 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:36:25.959 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:25.960 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:25.960 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:36:25.960 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:36:25.960 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:36:25.960 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:36:25.961 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:25.961 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:36:25.961 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:36:25.961 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:25.961 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:25.962 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:25.962 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:36:25.962 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:36:25.962 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:25.962 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:25.963 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:25.963 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:25.963 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:36:25.963 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:36:25.963 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:36:25.964 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:36:25.964 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:36:25.964 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:25.964 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:25.964 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:36:25.965 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:36:25.965 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:36:25.965 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:36:25.965 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:25.965 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:36:25.966 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:36:25.966 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:25.966 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:25.966 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:25.966 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:36:25.966 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:36:25.967 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:25.967 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:25.967 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:25.967 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:25.967 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:36:25.968 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:36:25.968 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:36:25.968 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:36:25.968 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:36:25.968 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:25.969 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:25.969 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:25.969 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:36:25.969 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:36:25.969 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:25.970 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:36:25.970 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:36:25.970 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:36:25.970 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:25.970 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:25.971 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:36:25.971 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:25.971 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:36:25.971 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:36:25.971 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:36:25.972 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:25.972 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:25.972 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:36:25.972 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:25.972 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:36:25.973 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:36:25.973 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 6 2020-11-20T11:36:25.973 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:25.973 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:25.973 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:36:25.974 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:25.974 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:36:25.974 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:36:25.974 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:36:25.974 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:25.974 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:25.975 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:36:25.975 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:25.975 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:36:25.975 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:36:25.975 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:36:25.976 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:25.976 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:25.976 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:25.976 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:36:25.976 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:36:25.978 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:36:26.103 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.099+0000 7f687597b700 1 Processor -- start 2020-11-20T11:36:26.103 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.099+0000 7f687597b700 1 -- start start 2020-11-20T11:36:26.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.100+0000 7f687597b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68700aae90 0x7f68701381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:26.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.100+0000 7f687597b700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f687013e600 con 0x7f68700aaab0 2020-11-20T11:36:26.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.100+0000 7f687597b700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f687013e8a0 con 0x7f687013aeb0 2020-11-20T11:36:26.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.100+0000 7f687597b700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68700ab6d0 con 0x7f68700aae90 2020-11-20T11:36:26.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.100+0000 7f686effd700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f68700aaab0 0x7f687013ed40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57002/0 (socket says 172.21.15.23:57002) 2020-11-20T11:36:26.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.100+0000 7f686effd700 1 -- 172.21.15.23:0/236309450 learned_addr learned my addr 172.21.15.23:0/236309450 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:36:26.107 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.100+0000 7f686e7fc700 1 --2- 172.21.15.23:0/236309450 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68700aae90 0x7f68701381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:26.107 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.100+0000 7f686dffb700 1 -- 172.21.15.23:0/236309450 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3769565568 0 0) 0x7f687013e600 con 0x7f68700aaab0 2020-11-20T11:36:26.107 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.100+0000 7f686dffb700 1 -- 172.21.15.23:0/236309450 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f685c003280 con 0x7f68700aaab0 2020-11-20T11:36:26.107 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.100+0000 7f686dffb700 1 -- 172.21.15.23:0/236309450 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048511155 0 0) 0x7f687013e8a0 con 0x7f687013aeb0 2020-11-20T11:36:26.107 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.100+0000 7f686dffb700 1 -- 172.21.15.23:0/236309450 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f687013e600 con 0x7f687013aeb0 2020-11-20T11:36:26.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.101+0000 7f686e7fc700 1 -- 172.21.15.23:0/236309450 >> v1:172.21.15.155:6789/0 conn(0x7f687013aeb0 legacy=0x7f68700a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:26.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.101+0000 7f686e7fc700 1 -- 172.21.15.23:0/236309450 >> v1:172.21.15.23:6789/0 conn(0x7f68700aaab0 legacy=0x7f687013ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:26.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.101+0000 7f686dffb700 1 -- 172.21.15.23:0/236309450 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1428305642 0 0) 0x7f685c003280 con 0x7f68700aaab0 2020-11-20T11:36:26.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.101+0000 7f686e7fc700 1 -- 172.21.15.23:0/236309450 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68700a9370 con 0x7f68700aae90 2020-11-20T11:36:26.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.101+0000 7f686e7fc700 1 --2- 172.21.15.23:0/236309450 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68700aae90 0x7f68701381a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f686400ba00 tx=0x7f6864006e30).ready entity=mon.2 client_cookie=6761e38879bc513f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:26.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.101+0000 7f686dffb700 1 -- 172.21.15.23:0/236309450 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6860001520 con 0x7f68700aae90 2020-11-20T11:36:26.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.101+0000 7f686dffb700 1 -- 172.21.15.23:0/236309450 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6864000e90 con 0x7f68700aae90 2020-11-20T11:36:26.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.101+0000 7f686dffb700 1 -- 172.21.15.23:0/236309450 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6864011610 con 0x7f68700aae90 2020-11-20T11:36:26.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.101+0000 7f687597b700 1 -- 172.21.15.23:0/236309450 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68700aae90 msgr2=0x7f68701381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:26.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.101+0000 7f687597b700 1 --2- 172.21.15.23:0/236309450 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68700aae90 0x7f68701381a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f686400ba00 tx=0x7f6864006e30).stop 2020-11-20T11:36:26.110 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.101+0000 7f687597b700 1 -- 172.21.15.23:0/236309450 shutdown_connections 2020-11-20T11:36:26.110 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.101+0000 7f687597b700 1 --2- 172.21.15.23:0/236309450 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68700aae90 0x7f68701381a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:26.110 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.101+0000 7f687597b700 1 -- 172.21.15.23:0/236309450 >> 172.21.15.23:0/236309450 conn(0x7f68700678e0 msgr2=0x7f6870067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:26.110 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.102+0000 7f687597b700 1 -- 172.21.15.23:0/236309450 shutdown_connections 2020-11-20T11:36:26.110 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.102+0000 7f687597b700 1 -- 172.21.15.23:0/236309450 wait complete. 2020-11-20T11:36:26.113 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.103+0000 7f687597b700 1 Processor -- start 2020-11-20T11:36:26.113 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.103+0000 7f687597b700 1 -- start start 2020-11-20T11:36:26.114 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.103+0000 7f687597b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68700aaab0 0x7f687009ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:26.114 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.103+0000 7f687597b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68700aae90 0x7f68700a7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:26.114 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.104+0000 7f687597b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f687014daa0 0x7f68700a45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:26.114 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.104+0000 7f687597b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68700b0f90 con 0x7f68700aae90 2020-11-20T11:36:26.115 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.104+0000 7f687597b700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68700a6420 con 0x7f68700aaab0 2020-11-20T11:36:26.115 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.104+0000 7f687597b700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68700a66c0 con 0x7f687014daa0 2020-11-20T11:36:26.115 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.104+0000 7f686e7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68700aae90 0x7f68700a7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:26.115 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.104+0000 7f686e7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68700aae90 0x7f68700a7770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42760/0 (socket says 172.21.15.23:42760) 2020-11-20T11:36:26.115 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.104+0000 7f686e7fc700 1 -- 172.21.15.23:0/1275112338 learned_addr learned my addr 172.21.15.23:0/1275112338 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:36:26.116 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.104+0000 7f686f7fe700 1 --2- 172.21.15.23:0/1275112338 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f687014daa0 0x7f68700a45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:26.116 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.104+0000 7f686effd700 1 --2- 172.21.15.23:0/1275112338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68700aaab0 0x7f687009ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:26.116 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.104+0000 7f686e7fc700 1 -- 172.21.15.23:0/1275112338 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f687014daa0 msgr2=0x7f68700a45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:26.116 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.104+0000 7f686e7fc700 1 --2- 172.21.15.23:0/1275112338 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f687014daa0 0x7f68700a45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:26.116 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.104+0000 7f686e7fc700 1 -- 172.21.15.23:0/1275112338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68700aaab0 msgr2=0x7f687009ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:26.117 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.104+0000 7f686e7fc700 1 --2- 172.21.15.23:0/1275112338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68700aaab0 0x7f687009ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:26.117 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.104+0000 7f686e7fc700 1 -- 172.21.15.23:0/1275112338 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68700a6960 con 0x7f68700aae90 2020-11-20T11:36:26.117 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.104+0000 7f686e7fc700 1 --2- 172.21.15.23:0/1275112338 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68700aae90 0x7f68700a7770 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f6870062f90 tx=0x7f6864015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:26.117 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.104+0000 7f6856ffd700 1 -- 172.21.15.23:0/1275112338 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6864010b20 con 0x7f68700aae90 2020-11-20T11:36:26.117 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.104+0000 7f6856ffd700 1 -- 172.21.15.23:0/1275112338 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6864010c80 con 0x7f68700aae90 2020-11-20T11:36:26.118 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.104+0000 7f687597b700 1 -- 172.21.15.23:0/1275112338 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68700a6e00 con 0x7f68700aae90 2020-11-20T11:36:26.118 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.104+0000 7f6856ffd700 1 -- 172.21.15.23:0/1275112338 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686401e460 con 0x7f68700aae90 2020-11-20T11:36:26.118 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.104+0000 7f687597b700 1 -- 172.21.15.23:0/1275112338 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68700a7c60 con 0x7f68700aae90 2020-11-20T11:36:26.118 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.105+0000 7f687597b700 1 --2- 172.21.15.23:0/1275112338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f687013aeb0 0x7f687014de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:26.118 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.105+0000 7f687597b700 1 -- 172.21.15.23:0/1275112338 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68700a7e50 con 0x7f687013aeb0 2020-11-20T11:36:26.119 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.105+0000 7f686effd700 1 --2- 172.21.15.23:0/1275112338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f687013aeb0 0x7f687014de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:26.119 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.106+0000 7f6856ffd700 1 -- 172.21.15.23:0/1275112338 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6864025030 con 0x7f68700aae90 2020-11-20T11:36:26.119 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.106+0000 7f686effd700 1 -- 172.21.15.23:0/1275112338 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6870139aa0 con 0x7f687013aeb0 2020-11-20T11:36:26.119 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.106+0000 7f686effd700 1 --2- 172.21.15.23:0/1275112338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f687013aeb0 0x7f687014de80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f68600011e0 tx=0x7f6860001a80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:26.119 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.106+0000 7f6856ffd700 1 --2- 172.21.15.23:0/1275112338 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f684c056d60 0x7f684c059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:26.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.106+0000 7f6856ffd700 1 -- 172.21.15.23:0/1275112338 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f6864071450 con 0x7f68700aae90 2020-11-20T11:36:26.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.106+0000 7f6856ffd700 1 -- 172.21.15.23:0/1275112338 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686000d980 con 0x7f687013aeb0 2020-11-20T11:36:26.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.106+0000 7f686f7fe700 1 --2- 172.21.15.23:0/1275112338 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f684c056d60 0x7f684c059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:26.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.106+0000 7f6856ffd700 1 -- 172.21.15.23:0/1275112338 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f686000dae0 con 0x7f687013aeb0 2020-11-20T11:36:26.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.106+0000 7f6856ffd700 1 -- 172.21.15.23:0/1275112338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f687013aeb0 msgr2=0x7f687014de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:26.121 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.106+0000 7f6856ffd700 1 --2- 172.21.15.23:0/1275112338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f687013aeb0 0x7f687014de80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f68600011e0 tx=0x7f6860001a80).stop 2020-11-20T11:36:26.121 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.106+0000 7f6856ffd700 1 -- 172.21.15.23:0/1275112338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f687013aeb0 msgr2=0x7f687014de80 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:36:26.121 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.106+0000 7f6856ffd700 1 --2- 172.21.15.23:0/1275112338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f687013aeb0 0x7f687014de80 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0x7f68600011e0 tx=0x7f6860001a80).stop 2020-11-20T11:36:26.121 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.106+0000 7f686f7fe700 1 --2- 172.21.15.23:0/1275112338 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f684c056d60 0x7f684c059220 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f68680012e0 tx=0x7f6868007d80).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:26.123 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.113+0000 7f687597b700 1 --2- 172.21.15.23:0/1275112338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68700601b0 0x7f68700a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:26.123 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.113+0000 7f687597b700 1 -- 172.21.15.23:0/1275112338 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68700a4fd0 con 0x7f68700601b0 2020-11-20T11:36:26.124 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.114+0000 7f686effd700 1 --2- 172.21.15.23:0/1275112338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68700601b0 0x7f68700a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:26.124 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.114+0000 7f686effd700 1 -- 172.21.15.23:0/1275112338 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f687005c7d0 con 0x7f68700601b0 2020-11-20T11:36:26.124 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.114+0000 7f686effd700 1 --2- 172.21.15.23:0/1275112338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68700601b0 0x7f68700a4ae0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f68600011e0 tx=0x7f6860016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:26.124 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.114+0000 7f6856ffd700 1 -- 172.21.15.23:0/1275112338 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686001e040 con 0x7f68700601b0 2020-11-20T11:36:26.124 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.115+0000 7f6856ffd700 1 -- 172.21.15.23:0/1275112338 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4613 (secure 0 0 0) 0x7f686000d770 con 0x7f68700601b0 2020-11-20T11:36:26.125 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.115+0000 7f6856ffd700 1 -- 172.21.15.23:0/1275112338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68700601b0 msgr2=0x7f68700a4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:26.125 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.115+0000 7f6856ffd700 1 --2- 172.21.15.23:0/1275112338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68700601b0 0x7f68700a4ae0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f68600011e0 tx=0x7f6860016000).stop 2020-11-20T11:36:26.125 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.115+0000 7f6856ffd700 1 -- 172.21.15.23:0/1275112338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68700601b0 msgr2=0x7f68700a4ae0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:36:26.125 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.115+0000 7f6856ffd700 1 --2- 172.21.15.23:0/1275112338 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68700601b0 0x7f68700a4ae0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:26.125 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.115+0000 7f687597b700 1 -- 172.21.15.23:0/1275112338 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f684c056d60 msgr2=0x7f684c059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:26.126 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.115+0000 7f687597b700 1 --2- 172.21.15.23:0/1275112338 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f684c056d60 0x7f684c059220 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f68680012e0 tx=0x7f6868007d80).stop 2020-11-20T11:36:26.126 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.115+0000 7f687597b700 1 -- 172.21.15.23:0/1275112338 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68700aae90 msgr2=0x7f68700a7770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:26.126 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.115+0000 7f687597b700 1 --2- 172.21.15.23:0/1275112338 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68700aae90 0x7f68700a7770 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f6870062f90 tx=0x7f6864015000).stop 2020-11-20T11:36:26.126 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.115+0000 7f686f7fe700 1 -- 172.21.15.23:0/1275112338 reap_dead start 2020-11-20T11:36:26.126 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.115+0000 7f687597b700 1 -- 172.21.15.23:0/1275112338 shutdown_connections 2020-11-20T11:36:26.127 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.115+0000 7f687597b700 1 --2- 172.21.15.23:0/1275112338 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68700aae90 0x7f68700a7770 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:26.127 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.115+0000 7f687597b700 1 -- 172.21.15.23:0/1275112338 >> 172.21.15.23:0/1275112338 conn(0x7f68700678e0 msgr2=0x7f68700acf80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:26.127 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.115+0000 7f687597b700 1 -- 172.21.15.23:0/1275112338 shutdown_connections 2020-11-20T11:36:26.127 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.115+0000 7f687597b700 1 -- 172.21.15.23:0/1275112338 wait complete. 2020-11-20T11:36:26.127 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:36:26.128 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:36:26.128 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:36:26.128 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:36:26.128 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 38, 2020-11-20T11:36:26.128 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:36:26.129 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:36:26.129 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:36:26.129 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:36:26.129 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:26.129 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 2, 2020-11-20T11:36:26.130 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:36:26.130 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:36:26.130 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:36:26.130 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:36:26.130 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:36:26.131 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:36:26.131 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:36:26.131 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:36:26.131 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:36:26.131 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:36:26.132 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:36:26.132 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:26.132 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:36:26.132 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:36:26.132 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:36:26.133 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:36:26.133 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:36:26.133 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:36:26.133 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:36:26.133 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:36:26.134 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:36:26.134 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:36:26.134 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:26.134 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:26.134 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:36:26.134 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:36:26.135 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:36:26.135 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:36:26.135 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:36:26.135 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:36:26.135 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:36:26.136 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:36:26.136 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:36:26.136 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:36:26.136 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:36:26.136 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:36:26.137 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:36:26.137 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:36:26.137 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:36:26.137 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:36:26.137 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:36:26.138 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:36:26.138 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:36:26.138 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:36:26.138 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:36:26.138 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:36:26.139 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:36:26.139 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:26.139 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:36:26.139 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:26.139 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:36:26.140 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:26.140 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:36:26.140 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:36:26.140 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:36:26.140 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:36:26.140 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:26.141 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:36:26.141 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:36:26.141 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:26.141 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:26.141 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:26.142 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:36:26.142 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:36:26.142 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:26.142 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:26.142 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:26.143 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:26.144 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:36:26.144 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:36:26.144 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:36:26.144 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:36:26.145 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:36:26.145 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:26.145 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:26.145 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:36:26.146 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:36:26.146 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:36:26.146 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:36:26.146 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:26.146 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:36:26.146 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:36:26.147 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:26.147 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:26.147 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:26.147 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:36:26.148 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:36:26.148 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:26.148 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:26.149 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:26.149 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:26.149 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:36:26.149 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:36:26.149 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:36:26.150 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:36:26.150 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:36:26.150 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:26.150 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:26.150 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:36:26.151 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:36:26.151 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:36:26.151 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:36:26.151 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:26.152 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:36:26.152 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:36:26.152 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:26.152 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:26.153 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:26.153 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:36:26.153 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:36:26.153 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:26.153 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:26.154 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:26.154 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:26.154 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:36:26.154 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:36:26.154 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:36:26.155 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:36:26.155 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:36:26.155 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:26.155 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:26.155 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:26.156 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:36:26.156 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:36:26.156 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:26.157 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:36:26.157 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:36:26.157 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:36:26.157 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:26.157 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:26.158 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:36:26.158 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:26.158 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:36:26.158 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:36:26.159 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:36:26.159 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:26.159 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:26.159 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:36:26.159 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:26.160 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:36:26.160 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:36:26.160 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:36:26.160 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:26.160 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:26.161 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:36:26.161 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:26.161 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:36:26.161 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:36:26.162 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:36:26.162 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:26.162 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:26.162 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:26.162 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:36:26.163 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:36:26.164 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:36:26.288 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.284+0000 7fe138c1c700 1 Processor -- start 2020-11-20T11:36:26.288 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.284+0000 7fe138c1c700 1 -- start start 2020-11-20T11:36:26.290 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.285+0000 7fe138c1c700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe1340ace70 0x7fe13413a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:26.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.285+0000 7fe138c1c700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe1340ad2f0 0x7fe13413ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:26.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.285+0000 7fe138c1c700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1340ad860 con 0x7fe1340a9040 2020-11-20T11:36:26.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.285+0000 7fe138c1c700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1340aa8a0 con 0x7fe1340ace70 2020-11-20T11:36:26.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.285+0000 7fe138c1c700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1340aaa90 con 0x7fe1340ad2f0 2020-11-20T11:36:26.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.285+0000 7fe1337fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fe1340a9040 0x7fe1340aaf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57020/0 (socket says 172.21.15.23:57020) 2020-11-20T11:36:26.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.285+0000 7fe1337fe700 1 -- 172.21.15.23:0/4050702248 learned_addr learned my addr 172.21.15.23:0/4050702248 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:36:26.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.285+0000 7fe133fff700 1 --2- 172.21.15.23:0/4050702248 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe1340ad2f0 0x7fe13413ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:26.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.285+0000 7fe132ffd700 1 --2- 172.21.15.23:0/4050702248 >> v2:172.21.15.155:3300/0 conn(0x7fe1340ace70 0x7fe13413a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:26.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.285+0000 7fe1327fc700 1 -- 172.21.15.23:0/4050702248 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4113887208 0 0) 0x7fe1340ad860 con 0x7fe1340a9040 2020-11-20T11:36:26.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.285+0000 7fe1327fc700 1 -- 172.21.15.23:0/4050702248 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe120003280 con 0x7fe1340a9040 2020-11-20T11:36:26.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.285+0000 7fe133fff700 1 -- 172.21.15.23:0/4050702248 >> v2:172.21.15.155:3300/0 conn(0x7fe1340ace70 msgr2=0x7fe13413a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:36:26.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.285+0000 7fe1327fc700 1 -- 172.21.15.23:0/4050702248 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 906020480 0 0) 0x7fe120003280 con 0x7fe1340a9040 2020-11-20T11:36:26.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.285+0000 7fe133fff700 1 --2- 172.21.15.23:0/4050702248 >> v2:172.21.15.155:3300/0 conn(0x7fe1340ace70 0x7fe13413a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:26.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.285+0000 7fe133fff700 1 -- 172.21.15.23:0/4050702248 >> v1:172.21.15.23:6789/0 conn(0x7fe1340a9040 legacy=0x7fe1340aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:26.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.285+0000 7fe133fff700 1 -- 172.21.15.23:0/4050702248 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1340a9420 con 0x7fe1340ad2f0 2020-11-20T11:36:26.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.286+0000 7fe133fff700 1 --2- 172.21.15.23:0/4050702248 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe1340ad2f0 0x7fe13413ad20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fe12c009ce0 tx=0x7fe12c009220).ready entity=mon.2 client_cookie=6e37d679eee62691 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:26.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.286+0000 7fe1327fc700 1 -- 172.21.15.23:0/4050702248 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1240016a0 con 0x7fe1340ad2f0 2020-11-20T11:36:26.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.286+0000 7fe1327fc700 1 -- 172.21.15.23:0/4050702248 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe12c00b680 con 0x7fe1340ad2f0 2020-11-20T11:36:26.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.286+0000 7fe1327fc700 1 -- 172.21.15.23:0/4050702248 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12c00ba00 con 0x7fe1340ad2f0 2020-11-20T11:36:26.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.286+0000 7fe138c1c700 1 -- 172.21.15.23:0/4050702248 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe1340ad2f0 msgr2=0x7fe13413ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:26.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.286+0000 7fe138c1c700 1 --2- 172.21.15.23:0/4050702248 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe1340ad2f0 0x7fe13413ad20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fe12c009ce0 tx=0x7fe12c009220).stop 2020-11-20T11:36:26.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.286+0000 7fe138c1c700 1 -- 172.21.15.23:0/4050702248 shutdown_connections 2020-11-20T11:36:26.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.286+0000 7fe138c1c700 1 --2- 172.21.15.23:0/4050702248 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe1340ad2f0 0x7fe13413ad20 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:26.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.286+0000 7fe138c1c700 1 --2- 172.21.15.23:0/4050702248 >> v2:172.21.15.155:3300/0 conn(0x7fe1340ace70 0x7fe13413a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:26.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.286+0000 7fe138c1c700 1 -- 172.21.15.23:0/4050702248 >> 172.21.15.23:0/4050702248 conn(0x7fe1340683b0 msgr2=0x7fe1340b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:26.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.286+0000 7fe138c1c700 1 -- 172.21.15.23:0/4050702248 shutdown_connections 2020-11-20T11:36:26.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.286+0000 7fe138c1c700 1 -- 172.21.15.23:0/4050702248 wait complete. 2020-11-20T11:36:26.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.287+0000 7fe138c1c700 1 Processor -- start 2020-11-20T11:36:26.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.287+0000 7fe138c1c700 1 -- start start 2020-11-20T11:36:26.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.288+0000 7fe138c1c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe1340a9040 0x7fe1340a1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:26.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.288+0000 7fe138c1c700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe1340ace70 0x7fe13409e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:26.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.288+0000 7fe138c1c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe1340ad2f0 0x7fe1340a70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:26.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.288+0000 7fe138c1c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe13409d8d0 con 0x7fe1340ad2f0 2020-11-20T11:36:26.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.288+0000 7fe138c1c700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe13409f730 con 0x7fe1340a9040 2020-11-20T11:36:26.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.288+0000 7fe138c1c700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe13409f9d0 con 0x7fe1340ace70 2020-11-20T11:36:26.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.288+0000 7fe133fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe1340ad2f0 0x7fe1340a70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:26.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.288+0000 7fe133fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe1340ad2f0 0x7fe1340a70c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42780/0 (socket says 172.21.15.23:42780) 2020-11-20T11:36:26.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.288+0000 7fe133fff700 1 -- 172.21.15.23:0/416105237 learned_addr learned my addr 172.21.15.23:0/416105237 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:36:26.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.288+0000 7fe132ffd700 1 --2- 172.21.15.23:0/416105237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe1340ace70 0x7fe13409e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:26.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.288+0000 7fe1337fe700 1 --2- 172.21.15.23:0/416105237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe1340a9040 0x7fe1340a1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:26.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.289+0000 7fe133fff700 1 -- 172.21.15.23:0/416105237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe1340ace70 msgr2=0x7fe13409e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:26.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.289+0000 7fe133fff700 1 --2- 172.21.15.23:0/416105237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe1340ace70 0x7fe13409e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:26.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.289+0000 7fe133fff700 1 -- 172.21.15.23:0/416105237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe1340a9040 msgr2=0x7fe1340a1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:26.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.289+0000 7fe133fff700 1 --2- 172.21.15.23:0/416105237 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe1340a9040 0x7fe1340a1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:26.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.289+0000 7fe133fff700 1 -- 172.21.15.23:0/416105237 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe13409fc70 con 0x7fe1340ad2f0 2020-11-20T11:36:26.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.289+0000 7fe133fff700 1 --2- 172.21.15.23:0/416105237 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe1340ad2f0 0x7fe1340a70c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fe12c0090b0 tx=0x7fe12c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:26.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.289+0000 7fe1137fe700 1 -- 172.21.15.23:0/416105237 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12c01c650 con 0x7fe1340ad2f0 2020-11-20T11:36:26.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.289+0000 7fe1137fe700 1 -- 172.21.15.23:0/416105237 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe12c01c7b0 con 0x7fe1340ad2f0 2020-11-20T11:36:26.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.289+0000 7fe1137fe700 1 -- 172.21.15.23:0/416105237 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12c01cb30 con 0x7fe1340ad2f0 2020-11-20T11:36:26.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.289+0000 7fe138c1c700 1 -- 172.21.15.23:0/416105237 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe13409f420 con 0x7fe1340ad2f0 2020-11-20T11:36:26.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.289+0000 7fe138c1c700 1 -- 172.21.15.23:0/416105237 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1340a75b0 con 0x7fe1340ad2f0 2020-11-20T11:36:26.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.290+0000 7fe138c1c700 1 --2- 172.21.15.23:0/416105237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe134149800 0x7fe134149be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:26.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.290+0000 7fe138c1c700 1 -- 172.21.15.23:0/416105237 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe13405cf20 con 0x7fe134149800 2020-11-20T11:36:26.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.290+0000 7fe1337fe700 1 --2- 172.21.15.23:0/416105237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe134149800 0x7fe134149be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:26.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.290+0000 7fe1137fe700 1 -- 172.21.15.23:0/416105237 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe12c022030 con 0x7fe1340ad2f0 2020-11-20T11:36:26.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.291+0000 7fe1337fe700 1 -- 172.21.15.23:0/416105237 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe134057dc0 con 0x7fe134149800 2020-11-20T11:36:26.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.291+0000 7fe1337fe700 1 --2- 172.21.15.23:0/416105237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe134149800 0x7fe134149be0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fe12400faf0 tx=0x7fe124001440).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:26.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.291+0000 7fe1137fe700 1 --2- 172.21.15.23:0/416105237 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe10c056d60 0x7fe10c059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:26.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.291+0000 7fe1137fe700 1 -- 172.21.15.23:0/416105237 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fe12c072700 con 0x7fe1340ad2f0 2020-11-20T11:36:26.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.291+0000 7fe132ffd700 1 --2- 172.21.15.23:0/416105237 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe10c056d60 0x7fe10c059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:26.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.291+0000 7fe1137fe700 1 -- 172.21.15.23:0/416105237 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12400d9b0 con 0x7fe134149800 2020-11-20T11:36:26.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.291+0000 7fe1137fe700 1 -- 172.21.15.23:0/416105237 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fe12400db10 con 0x7fe134149800 2020-11-20T11:36:26.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.291+0000 7fe1137fe700 1 -- 172.21.15.23:0/416105237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe134149800 msgr2=0x7fe134149be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:26.308 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.291+0000 7fe1137fe700 1 --2- 172.21.15.23:0/416105237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe134149800 0x7fe134149be0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fe12400faf0 tx=0x7fe124001440).stop 2020-11-20T11:36:26.308 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.291+0000 7fe1137fe700 1 -- 172.21.15.23:0/416105237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe134149800 msgr2=0x7fe134149be0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:36:26.308 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.291+0000 7fe1137fe700 1 --2- 172.21.15.23:0/416105237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe134149800 0x7fe134149be0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:26.308 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.292+0000 7fe132ffd700 1 --2- 172.21.15.23:0/416105237 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe10c056d60 0x7fe10c059220 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fe1340a10e0 tx=0x7fe128006e30).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:26.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.299+0000 7fe138c1c700 1 --2- 172.21.15.23:0/416105237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe13414cdd0 0x7fe13414d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:26.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.299+0000 7fe138c1c700 1 -- 172.21.15.23:0/416105237 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe13409ef80 con 0x7fe13414cdd0 2020-11-20T11:36:26.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.300+0000 7fe1337fe700 1 --2- 172.21.15.23:0/416105237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe13414cdd0 0x7fe13414d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:26.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.300+0000 7fe1337fe700 1 -- 172.21.15.23:0/416105237 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe134044710 con 0x7fe13414cdd0 2020-11-20T11:36:26.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.300+0000 7fe1337fe700 1 --2- 172.21.15.23:0/416105237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe13414cdd0 0x7fe13414d1b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fe12400faf0 tx=0x7fe124019000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:26.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.300+0000 7fe1137fe700 1 -- 172.21.15.23:0/416105237 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1240136f0 con 0x7fe13414cdd0 2020-11-20T11:36:26.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.301+0000 7fe1137fe700 1 -- 172.21.15.23:0/416105237 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4273 (secure 0 0 0) 0x7fe12400d7a0 con 0x7fe13414cdd0 2020-11-20T11:36:26.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.301+0000 7fe1137fe700 1 -- 172.21.15.23:0/416105237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe13414cdd0 msgr2=0x7fe13414d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:26.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.301+0000 7fe1137fe700 1 --2- 172.21.15.23:0/416105237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe13414cdd0 0x7fe13414d1b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fe12400faf0 tx=0x7fe124019000).stop 2020-11-20T11:36:26.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.301+0000 7fe1137fe700 1 -- 172.21.15.23:0/416105237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe13414cdd0 msgr2=0x7fe13414d1b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:36:26.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.301+0000 7fe1137fe700 1 --2- 172.21.15.23:0/416105237 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe13414cdd0 0x7fe13414d1b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:26.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.301+0000 7fe138c1c700 1 -- 172.21.15.23:0/416105237 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe10c056d60 msgr2=0x7fe10c059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:26.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.301+0000 7fe138c1c700 1 --2- 172.21.15.23:0/416105237 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe10c056d60 0x7fe10c059220 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fe1340a10e0 tx=0x7fe128006e30).stop 2020-11-20T11:36:26.313 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.301+0000 7fe138c1c700 1 -- 172.21.15.23:0/416105237 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe1340ad2f0 msgr2=0x7fe1340a70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:26.313 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.301+0000 7fe138c1c700 1 --2- 172.21.15.23:0/416105237 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe1340ad2f0 0x7fe1340a70c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fe12c0090b0 tx=0x7fe12c016000).stop 2020-11-20T11:36:26.313 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.301+0000 7fe133fff700 1 -- 172.21.15.23:0/416105237 reap_dead start 2020-11-20T11:36:26.313 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.301+0000 7fe133fff700 1 -- 172.21.15.23:0/416105237 reap_dead start 2020-11-20T11:36:26.313 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.301+0000 7fe138c1c700 1 -- 172.21.15.23:0/416105237 shutdown_connections 2020-11-20T11:36:26.314 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.301+0000 7fe138c1c700 1 -- 172.21.15.23:0/416105237 >> 172.21.15.23:0/416105237 conn(0x7fe1340683b0 msgr2=0x7fe1340af7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:26.314 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.301+0000 7fe138c1c700 1 -- 172.21.15.23:0/416105237 shutdown_connections 2020-11-20T11:36:26.314 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.301+0000 7fe138c1c700 1 -- 172.21.15.23:0/416105237 wait complete. 2020-11-20T11:36:26.314 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:36:26.315 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:36:26.315 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:36:26.315 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:36:26.315 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 38, 2020-11-20T11:36:26.315 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:36:26.316 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:36:26.316 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:36:26.316 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:36:26.316 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:26.316 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 2, 2020-11-20T11:36:26.317 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:36:26.317 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:36:26.317 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:36:26.317 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:36:26.318 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:36:26.318 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:36:26.318 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:36:26.318 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:36:26.319 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:36:26.319 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:36:26.319 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:36:26.319 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:26.319 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:36:26.320 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:36:26.320 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:36:26.320 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:36:26.320 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:36:26.320 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:36:26.321 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:36:26.321 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:36:26.321 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:36:26.321 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:36:26.322 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:26.322 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:26.322 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:36:26.322 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:36:26.322 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:36:26.323 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:36:26.323 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:36:26.323 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:36:26.323 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:36:26.323 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:36:26.324 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:36:26.324 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:36:26.324 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:36:26.324 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:36:26.324 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:36:26.325 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:36:26.325 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:36:26.325 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:36:26.325 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:36:26.325 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:36:26.326 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:36:26.326 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:36:26.326 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:36:26.326 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:36:26.326 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:36:26.327 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:26.327 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:36:26.327 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:26.327 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:36:26.327 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:26.328 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:36:26.328 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:36:26.328 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:36:26.329 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:36:26.329 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:26.329 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:36:26.329 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:36:26.330 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:26.330 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:26.330 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:26.330 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:36:26.330 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:36:26.331 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:26.331 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:26.331 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:26.331 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:26.331 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:36:26.332 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:36:26.332 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:36:26.332 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:36:26.332 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:36:26.333 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:26.333 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:26.333 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:36:26.333 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:36:26.334 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:36:26.334 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:36:26.334 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:26.334 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:36:26.334 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:36:26.335 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:26.335 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:26.335 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:26.335 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:36:26.336 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:36:26.336 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:26.336 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:26.336 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:26.337 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:26.337 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:36:26.337 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:36:26.337 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:36:26.337 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:36:26.338 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:36:26.338 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:26.338 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:26.338 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:36:26.339 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:36:26.339 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:36:26.339 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:36:26.339 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:26.340 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:36:26.340 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:36:26.340 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:26.340 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:26.341 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:26.341 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:36:26.341 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:36:26.341 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:26.342 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:26.342 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:26.342 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:26.342 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:36:26.342 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:36:26.343 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:36:26.343 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:36:26.343 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:36:26.343 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:26.343 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:26.344 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:26.344 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:36:26.344 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:36:26.344 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:26.344 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:36:26.345 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:36:26.345 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:36:26.345 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:26.345 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:26.345 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:36:26.346 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:26.346 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:36:26.346 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:36:26.346 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:36:26.346 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:26.347 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:26.347 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:26.347 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:36:26.347 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:36:26.348 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2020-11-20T11:36:26.349 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2020-11-20T11:36:26.472 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.468+0000 7f97d4b9d700 1 Processor -- start 2020-11-20T11:36:26.472 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.468+0000 7f97d4b9d700 1 -- start start 2020-11-20T11:36:26.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.469+0000 7f97d4b9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f97d00665f0 0x7f97d013adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:26.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.469+0000 7f97d4b9d700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97d013bd10 con 0x7f97d0066210 2020-11-20T11:36:26.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.469+0000 7f97d4b9d700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97d00aa930 con 0x7f97d00ab2a0 2020-11-20T11:36:26.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.469+0000 7f97d4b9d700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97d00ab700 con 0x7f97d00665f0 2020-11-20T11:36:26.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.469+0000 7f97c7fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f97d00665f0 0x7f97d013adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:26.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.469+0000 7f97c7fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f97d00665f0 0x7f97d013adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42790/0 (socket says 172.21.15.23:42790) 2020-11-20T11:36:26.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.469+0000 7f97cf7fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f97d0066210 0x7f97d00a8ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57038/0 (socket says 172.21.15.23:57038) 2020-11-20T11:36:26.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.469+0000 7f97c7fff700 1 -- 172.21.15.23:0/1966356775 learned_addr learned my addr 172.21.15.23:0/1966356775 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:36:26.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.469+0000 7f97c7fff700 1 -- 172.21.15.23:0/1966356775 >> v1:172.21.15.155:6789/0 conn(0x7f97d00ab2a0 legacy=0x7f97d0137f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:36:26.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.469+0000 7f97c7fff700 1 -- 172.21.15.23:0/1966356775 >> v1:172.21.15.23:6789/0 conn(0x7f97d0066210 legacy=0x7f97d00a8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:26.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.469+0000 7f97ceffd700 1 -- 172.21.15.23:0/1966356775 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2245448983 0 0) 0x7f97d013bd10 con 0x7f97d0066210 2020-11-20T11:36:26.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.469+0000 7f97c7fff700 1 -- 172.21.15.23:0/1966356775 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d00a9700 con 0x7f97d00665f0 2020-11-20T11:36:26.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.469+0000 7f97c7fff700 1 --2- 172.21.15.23:0/1966356775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97d00665f0 0x7f97d013adb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f97b8009a10 tx=0x7f97b8006ef0).ready entity=mon.0 client_cookie=1509acaf0829ced1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:26.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.470+0000 7f97ceffd700 1 -- 172.21.15.23:0/1966356775 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b800b820 con 0x7f97d00665f0 2020-11-20T11:36:26.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.470+0000 7f97ceffd700 1 -- 172.21.15.23:0/1966356775 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b8000e90 con 0x7f97d00665f0 2020-11-20T11:36:26.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.470+0000 7f97ceffd700 1 -- 172.21.15.23:0/1966356775 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b800bb90 con 0x7f97d00665f0 2020-11-20T11:36:26.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.470+0000 7f97d4b9d700 1 -- 172.21.15.23:0/1966356775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97d00665f0 msgr2=0x7f97d013adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:26.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.470+0000 7f97d4b9d700 1 --2- 172.21.15.23:0/1966356775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97d00665f0 0x7f97d013adb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f97b8009a10 tx=0x7f97b8006ef0).stop 2020-11-20T11:36:26.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.470+0000 7f97d4b9d700 1 -- 172.21.15.23:0/1966356775 shutdown_connections 2020-11-20T11:36:26.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.470+0000 7f97d4b9d700 1 --2- 172.21.15.23:0/1966356775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97d00665f0 0x7f97d013adb0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:26.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.470+0000 7f97d4b9d700 1 -- 172.21.15.23:0/1966356775 >> 172.21.15.23:0/1966356775 conn(0x7f97d00612c0 msgr2=0x7f97d00616d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:26.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.470+0000 7f97d4b9d700 1 -- 172.21.15.23:0/1966356775 shutdown_connections 2020-11-20T11:36:26.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.470+0000 7f97d4b9d700 1 -- 172.21.15.23:0/1966356775 wait complete. 2020-11-20T11:36:26.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.471+0000 7f97d4b9d700 1 Processor -- start 2020-11-20T11:36:26.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.471+0000 7f97d4b9d700 1 -- start start 2020-11-20T11:36:26.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.472+0000 7f97d4b9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97d0066210 0x7f97d01368e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:26.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.472+0000 7f97d4b9d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97d00665f0 0x7f97d00a70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:26.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.472+0000 7f97d4b9d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97d00ab2a0 0x7f97d014d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:26.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.472+0000 7f97d4b9d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d00a79b0 con 0x7f97d0066210 2020-11-20T11:36:26.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.472+0000 7f97d4b9d700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d00a5590 con 0x7f97d00665f0 2020-11-20T11:36:26.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.472+0000 7f97d4b9d700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97d00a5830 con 0x7f97d00ab2a0 2020-11-20T11:36:26.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.472+0000 7f97cf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97d0066210 0x7f97d01368e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:26.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.472+0000 7f97cf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97d0066210 0x7f97d01368e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42794/0 (socket says 172.21.15.23:42794) 2020-11-20T11:36:26.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.472+0000 7f97cf7fe700 1 -- 172.21.15.23:0/3798160135 learned_addr learned my addr 172.21.15.23:0/3798160135 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:36:26.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.472+0000 7f97c7fff700 1 --2- 172.21.15.23:0/3798160135 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97d00665f0 0x7f97d00a70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:26.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.472+0000 7f97cffff700 1 --2- 172.21.15.23:0/3798160135 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97d00ab2a0 0x7f97d014d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:26.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.472+0000 7f97cf7fe700 1 -- 172.21.15.23:0/3798160135 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97d00ab2a0 msgr2=0x7f97d014d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:26.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.472+0000 7f97cf7fe700 1 --2- 172.21.15.23:0/3798160135 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97d00ab2a0 0x7f97d014d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:26.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.472+0000 7f97cf7fe700 1 -- 172.21.15.23:0/3798160135 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97d00665f0 msgr2=0x7f97d00a70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:26.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.472+0000 7f97cf7fe700 1 --2- 172.21.15.23:0/3798160135 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97d00665f0 0x7f97d00a70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:26.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.472+0000 7f97cf7fe700 1 -- 172.21.15.23:0/3798160135 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d00a5cd0 con 0x7f97d0066210 2020-11-20T11:36:26.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.473+0000 7f97cf7fe700 1 --2- 172.21.15.23:0/3798160135 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97d0066210 0x7f97d01368e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f97c000c790 tx=0x7f97c0007b20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:26.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.473+0000 7f97c77fe700 1 -- 172.21.15.23:0/3798160135 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c0007320 con 0x7f97d0066210 2020-11-20T11:36:26.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.473+0000 7f97c77fe700 1 -- 172.21.15.23:0/3798160135 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97c0005570 con 0x7f97d0066210 2020-11-20T11:36:26.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.473+0000 7f97d4b9d700 1 -- 172.21.15.23:0/3798160135 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97d00a75d0 con 0x7f97d0066210 2020-11-20T11:36:26.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.473+0000 7f97c77fe700 1 -- 172.21.15.23:0/3798160135 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c00058f0 con 0x7f97d0066210 2020-11-20T11:36:26.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.473+0000 7f97d4b9d700 1 -- 172.21.15.23:0/3798160135 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97d00a8230 con 0x7f97d0066210 2020-11-20T11:36:26.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.474+0000 7f97d4b9d700 1 -- 172.21.15.23:0/3798160135 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97d0062760 con 0x7f97d0066210 2020-11-20T11:36:26.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.474+0000 7f97c77fe700 1 -- 172.21.15.23:0/3798160135 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f97c0010030 con 0x7f97d0066210 2020-11-20T11:36:26.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.474+0000 7f97c77fe700 1 --2- 172.21.15.23:0/3798160135 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f97ac056d60 0x7f97ac059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:26.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.475+0000 7f97c77fe700 1 -- 172.21.15.23:0/3798160135 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f97c006fdd0 con 0x7f97d0066210 2020-11-20T11:36:26.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.475+0000 7f97c7fff700 1 --2- 172.21.15.23:0/3798160135 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f97ac056d60 0x7f97ac059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:26.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.475+0000 7f97c7fff700 1 --2- 172.21.15.23:0/3798160135 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f97ac056d60 0x7f97ac059220 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f97b8006c80 tx=0x7f97b8016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:26.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.478+0000 7f97c77fe700 1 -- 172.21.15.23:0/3798160135 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f97c00450c0 con 0x7f97d0066210 2020-11-20T11:36:26.730 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.726+0000 7f97d4b9d700 1 -- 172.21.15.23:0/3798160135 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f97d00a77c0 con 0x7f97d0066210 2020-11-20T11:36:27.325 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.727+0000 7f97c77fe700 1 -- 172.21.15.23:0/3798160135 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f97c00486e0 con 0x7f97d0066210 2020-11-20T11:36:27.326 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.729+0000 7f97d4b9d700 1 -- 172.21.15.23:0/3798160135 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f97ac056d60 msgr2=0x7f97ac059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:27.326 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.729+0000 7f97d4b9d700 1 --2- 172.21.15.23:0/3798160135 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f97ac056d60 0x7f97ac059220 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f97b8006c80 tx=0x7f97b8016000).stop 2020-11-20T11:36:27.326 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.729+0000 7f97d4b9d700 1 -- 172.21.15.23:0/3798160135 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97d0066210 msgr2=0x7f97d01368e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:27.327 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.729+0000 7f97d4b9d700 1 --2- 172.21.15.23:0/3798160135 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97d0066210 0x7f97d01368e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f97c000c790 tx=0x7f97c0007b20).stop 2020-11-20T11:36:27.327 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.730+0000 7f97d4b9d700 1 -- 172.21.15.23:0/3798160135 shutdown_connections 2020-11-20T11:36:27.328 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.730+0000 7f97d4b9d700 1 --2- 172.21.15.23:0/3798160135 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f97ac056d60 0x7f97ac059220 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:27.328 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.730+0000 7f97d4b9d700 1 --2- 172.21.15.23:0/3798160135 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97d00ab2a0 0x7f97d014d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:27.328 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.730+0000 7f97d4b9d700 1 --2- 172.21.15.23:0/3798160135 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97d0066210 0x7f97d01368e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:27.329 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.730+0000 7f97d4b9d700 1 --2- 172.21.15.23:0/3798160135 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97d00665f0 0x7f97d00a70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:27.329 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.730+0000 7f97d4b9d700 1 -- 172.21.15.23:0/3798160135 >> 172.21.15.23:0/3798160135 conn(0x7f97d00612c0 msgr2=0x7f97d0067850 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:27.330 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.730+0000 7f97d4b9d700 1 -- 172.21.15.23:0/3798160135 shutdown_connections 2020-11-20T11:36:27.330 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:26.730+0000 7f97d4b9d700 1 -- 172.21.15.23:0/3798160135 wait complete. 2020-11-20T11:36:27.331 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2020-11-20T11:36:29.987 INFO:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:36:29.989 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:36:37.332 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:36:37.333 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:36:37.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.451+0000 7f8ec9bf9700 1 Processor -- start 2020-11-20T11:36:37.456 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.451+0000 7f8ec9bf9700 1 -- start start 2020-11-20T11:36:37.458 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.452+0000 7f8ec9bf9700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f8ec40aaab0 0x7f8ec413ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:37.458 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.452+0000 7f8ec9bf9700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8ec40aae90 0x7f8ec41381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:37.459 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.452+0000 7f8ec9bf9700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ec413aeb0 0x7f8ec40a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:37.459 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.452+0000 7f8ec9bf9700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ec413e600 con 0x7f8ec40aae90 2020-11-20T11:36:37.459 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.452+0000 7f8ec9bf9700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ec413e8a0 con 0x7f8ec40aaab0 2020-11-20T11:36:37.459 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.452+0000 7f8ec9bf9700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec40ab6d0 con 0x7f8ec413aeb0 2020-11-20T11:36:37.459 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.452+0000 7f8ec2ffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8ec40aae90 0x7f8ec41381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:37.460 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.452+0000 7f8ec2ffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8ec40aae90 0x7f8ec41381a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42802/0 (socket says 172.21.15.23:42802) 2020-11-20T11:36:37.460 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.452+0000 7f8ec2ffd700 1 -- 172.21.15.23:0/813554184 learned_addr learned my addr 172.21.15.23:0/813554184 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:36:37.460 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.452+0000 7f8ec3fff700 1 --2- 172.21.15.23:0/813554184 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ec413aeb0 0x7f8ec40a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:37.460 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.452+0000 7f8ec37fe700 1 --2- 172.21.15.23:0/813554184 >> v2:172.21.15.155:3300/0 conn(0x7f8ec40aaab0 0x7f8ec413ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:37.460 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.452+0000 7f8ec2ffd700 1 -- 172.21.15.23:0/813554184 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ec413aeb0 msgr2=0x7f8ec40a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:36:37.461 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.452+0000 7f8ec2ffd700 1 --2- 172.21.15.23:0/813554184 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ec413aeb0 0x7f8ec40a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:37.461 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.452+0000 7f8ec2ffd700 1 -- 172.21.15.23:0/813554184 >> v2:172.21.15.155:3300/0 conn(0x7f8ec40aaab0 msgr2=0x7f8ec413ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:36:37.461 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.452+0000 7f8ec2ffd700 1 --2- 172.21.15.23:0/813554184 >> v2:172.21.15.155:3300/0 conn(0x7f8ec40aaab0 0x7f8ec413ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:37.461 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.452+0000 7f8ec2ffd700 1 -- 172.21.15.23:0/813554184 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec413f230 con 0x7f8ec40aae90 2020-11-20T11:36:37.461 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.452+0000 7f8ec3fff700 1 --2- 172.21.15.23:0/813554184 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ec413aeb0 0x7f8ec40a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:36:37.462 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.452+0000 7f8ec2ffd700 1 --2- 172.21.15.23:0/813554184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ec40aae90 0x7f8ec41381a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f8ec40b23d0 tx=0x7f8eb8006ef0).ready entity=mon.0 client_cookie=9699a253267c0ca9 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:37.462 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.453+0000 7f8ec27fc700 1 -- 172.21.15.23:0/813554184 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb800b860 con 0x7f8ec40aae90 2020-11-20T11:36:37.462 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.453+0000 7f8ec27fc700 1 -- 172.21.15.23:0/813554184 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb8000e90 con 0x7f8ec40aae90 2020-11-20T11:36:37.462 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.453+0000 7f8ec27fc700 1 -- 172.21.15.23:0/813554184 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb800bbd0 con 0x7f8ec40aae90 2020-11-20T11:36:37.463 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.453+0000 7f8ec9bf9700 1 -- 172.21.15.23:0/813554184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ec40aae90 msgr2=0x7f8ec41381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:37.463 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.453+0000 7f8ec9bf9700 1 --2- 172.21.15.23:0/813554184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ec40aae90 0x7f8ec41381a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f8ec40b23d0 tx=0x7f8eb8006ef0).stop 2020-11-20T11:36:37.463 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.453+0000 7f8ec9bf9700 1 -- 172.21.15.23:0/813554184 shutdown_connections 2020-11-20T11:36:37.463 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.453+0000 7f8ec9bf9700 1 --2- 172.21.15.23:0/813554184 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ec413aeb0 0x7f8ec40a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:37.463 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.453+0000 7f8ec9bf9700 1 --2- 172.21.15.23:0/813554184 >> v2:172.21.15.155:3300/0 conn(0x7f8ec40aaab0 0x7f8ec413ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:37.464 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.453+0000 7f8ec9bf9700 1 --2- 172.21.15.23:0/813554184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ec40aae90 0x7f8ec41381a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:37.464 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.453+0000 7f8ec9bf9700 1 -- 172.21.15.23:0/813554184 >> 172.21.15.23:0/813554184 conn(0x7f8ec40678e0 msgr2=0x7f8ec4067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:37.464 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.453+0000 7f8ec9bf9700 1 -- 172.21.15.23:0/813554184 shutdown_connections 2020-11-20T11:36:37.464 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.453+0000 7f8ec9bf9700 1 -- 172.21.15.23:0/813554184 wait complete. 2020-11-20T11:36:37.464 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.454+0000 7f8ec9bf9700 1 Processor -- start 2020-11-20T11:36:37.465 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.454+0000 7f8ec9bf9700 1 -- start start 2020-11-20T11:36:37.467 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.455+0000 7f8ec9bf9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ec40aaab0 0x7f8ec4140bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:37.467 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.455+0000 7f8ec9bf9700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ec40aae90 0x7f8ec4131f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:37.467 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.455+0000 7f8ec9bf9700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ec41323f0 0x7f8ec4132970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:37.468 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.455+0000 7f8ec9bf9700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec414a6f0 con 0x7f8ec40aaab0 2020-11-20T11:36:37.468 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.455+0000 7f8ec9bf9700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec414b060 con 0x7f8ec41323f0 2020-11-20T11:36:37.468 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.455+0000 7f8ec9bf9700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec4131cb0 con 0x7f8ec40aae90 2020-11-20T11:36:37.468 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.455+0000 7f8ec37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ec40aaab0 0x7f8ec4140bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:37.468 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.455+0000 7f8ec37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ec40aaab0 0x7f8ec4140bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42808/0 (socket says 172.21.15.23:42808) 2020-11-20T11:36:37.469 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.455+0000 7f8ec37fe700 1 -- 172.21.15.23:0/3746047753 learned_addr learned my addr 172.21.15.23:0/3746047753 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:36:37.469 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.455+0000 7f8ec2ffd700 1 --2- 172.21.15.23:0/3746047753 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ec40aae90 0x7f8ec4131f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:37.469 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.455+0000 7f8ec3fff700 1 --2- 172.21.15.23:0/3746047753 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ec41323f0 0x7f8ec4132970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:37.469 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.455+0000 7f8ec37fe700 1 -- 172.21.15.23:0/3746047753 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ec40aae90 msgr2=0x7f8ec4131f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:37.470 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.455+0000 7f8ec37fe700 1 --2- 172.21.15.23:0/3746047753 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ec40aae90 0x7f8ec4131f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:37.470 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.455+0000 7f8ec37fe700 1 -- 172.21.15.23:0/3746047753 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ec41323f0 msgr2=0x7f8ec4132970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:37.470 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.455+0000 7f8ec37fe700 1 --2- 172.21.15.23:0/3746047753 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ec41323f0 0x7f8ec4132970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:37.470 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.456+0000 7f8ec37fe700 1 -- 172.21.15.23:0/3746047753 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec414a920 con 0x7f8ec40aaab0 2020-11-20T11:36:37.471 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.456+0000 7f8ec37fe700 1 --2- 172.21.15.23:0/3746047753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ec40aaab0 0x7f8ec4140bd0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f8eb400eba0 tx=0x7f8eb4006e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:37.471 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.456+0000 7f8ea37fe700 1 -- 172.21.15.23:0/3746047753 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb400c850 con 0x7f8ec40aaab0 2020-11-20T11:36:37.471 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.456+0000 7f8ea37fe700 1 -- 172.21.15.23:0/3746047753 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb400c9b0 con 0x7f8ec40aaab0 2020-11-20T11:36:37.472 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.456+0000 7f8ec9bf9700 1 -- 172.21.15.23:0/3746047753 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ec414abc0 con 0x7f8ec40aaab0 2020-11-20T11:36:37.472 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.456+0000 7f8ea37fe700 1 -- 172.21.15.23:0/3746047753 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb400cd30 con 0x7f8ec40aaab0 2020-11-20T11:36:37.472 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.456+0000 7f8ec9bf9700 1 -- 172.21.15.23:0/3746047753 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ec413f4a0 con 0x7f8ec40aaab0 2020-11-20T11:36:37.473 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.457+0000 7f8ec9bf9700 1 -- 172.21.15.23:0/3746047753 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ec405c7d0 con 0x7f8ec40aaab0 2020-11-20T11:36:37.473 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.457+0000 7f8ea37fe700 1 -- 172.21.15.23:0/3746047753 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8eb4012600 con 0x7f8ec40aaab0 2020-11-20T11:36:37.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.458+0000 7f8ea37fe700 1 --2- 172.21.15.23:0/3746047753 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8e9c056d60 0x7f8e9c059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:37.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.458+0000 7f8ea37fe700 1 -- 172.21.15.23:0/3746047753 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f8eb4070390 con 0x7f8ec40aaab0 2020-11-20T11:36:37.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.458+0000 7f8ec2ffd700 1 --2- 172.21.15.23:0/3746047753 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8e9c056d60 0x7f8e9c059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:37.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.458+0000 7f8ec2ffd700 1 --2- 172.21.15.23:0/3746047753 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8e9c056d60 0x7f8e9c059220 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f8eb8006f30 tx=0x7f8eb8016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:37.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.461+0000 7f8ea37fe700 1 -- 172.21.15.23:0/3746047753 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8eb4045680 con 0x7f8ec40aaab0 2020-11-20T11:36:37.708 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.704+0000 7f8ec9bf9700 1 -- 172.21.15.23:0/3746047753 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8ec4133640 con 0x7f8ec40aaab0 2020-11-20T11:36:37.709 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.705+0000 7f8ea37fe700 1 -- 172.21.15.23:0/3746047753 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7f8eb4048ca0 con 0x7f8ec40aaab0 2020-11-20T11:36:37.709 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:36:37.710 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.706+0000 7f8ec9bf9700 1 -- 172.21.15.23:0/3746047753 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8e9c056d60 msgr2=0x7f8e9c059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:37.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.706+0000 7f8ec9bf9700 1 --2- 172.21.15.23:0/3746047753 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8e9c056d60 0x7f8e9c059220 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f8eb8006f30 tx=0x7f8eb8016000).stop 2020-11-20T11:36:37.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.706+0000 7f8ec9bf9700 1 -- 172.21.15.23:0/3746047753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ec40aaab0 msgr2=0x7f8ec4140bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:37.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.706+0000 7f8ec9bf9700 1 --2- 172.21.15.23:0/3746047753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ec40aaab0 0x7f8ec4140bd0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f8eb400eba0 tx=0x7f8eb4006e30).stop 2020-11-20T11:36:37.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.706+0000 7f8ec9bf9700 1 -- 172.21.15.23:0/3746047753 shutdown_connections 2020-11-20T11:36:37.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.706+0000 7f8ec9bf9700 1 --2- 172.21.15.23:0/3746047753 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ec41323f0 0x7f8ec4132970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:37.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.706+0000 7f8ec9bf9700 1 --2- 172.21.15.23:0/3746047753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ec40aaab0 0x7f8ec4140bd0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:37.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.706+0000 7f8ec9bf9700 1 --2- 172.21.15.23:0/3746047753 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8e9c056d60 0x7f8e9c059220 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:37.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.706+0000 7f8ec9bf9700 1 --2- 172.21.15.23:0/3746047753 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ec40aae90 0x7f8ec4131f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:37.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.706+0000 7f8ec9bf9700 1 -- 172.21.15.23:0/3746047753 >> 172.21.15.23:0/3746047753 conn(0x7f8ec40678e0 msgr2=0x7f8ec40ac1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:37.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.707+0000 7f8ec9bf9700 1 -- 172.21.15.23:0/3746047753 shutdown_connections 2020-11-20T11:36:37.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.707+0000 7f8ec9bf9700 1 -- 172.21.15.23:0/3746047753 wait complete. 2020-11-20T11:36:37.725 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:36:37.725 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2020-11-20T11:36:37.725 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:36:37.850 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.846+0000 7f6ee77a1700 1 Processor -- start 2020-11-20T11:36:37.851 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.846+0000 7f6ee77a1700 1 -- start start 2020-11-20T11:36:37.853 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.847+0000 7f6ee77a1700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6ee00aaab0 0x7f6ee013ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:37.853 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.847+0000 7f6ee77a1700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6ee00aae90 0x7f6ee01381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:37.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.847+0000 7f6ee77a1700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ee013e600 con 0x7f6ee013aeb0 2020-11-20T11:36:37.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.847+0000 7f6ee77a1700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ee013e8a0 con 0x7f6ee00aae90 2020-11-20T11:36:37.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.847+0000 7f6ee77a1700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee00ab6d0 con 0x7f6ee00aaab0 2020-11-20T11:36:37.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.848+0000 7f6ee553d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6ee00aaab0 0x7f6ee013ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:37.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.848+0000 7f6ee5d3e700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f6ee013aeb0 0x7f6ee00a8c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:35044/0 (socket says 172.21.15.23:35044) 2020-11-20T11:36:37.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.848+0000 7f6ee5d3e700 1 -- 172.21.15.23:0/3547079470 learned_addr learned my addr 172.21.15.23:0/3547079470 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:36:37.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.848+0000 7f6ee4d3c700 1 --2- 172.21.15.23:0/3547079470 >> v2:172.21.15.155:3300/0 conn(0x7f6ee00aae90 0x7f6ee01381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:37.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.848+0000 7f6ee4d3c700 1 -- 172.21.15.23:0/3547079470 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6ee00aaab0 msgr2=0x7f6ee013ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:36:37.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.848+0000 7f6ee4d3c700 1 --2- 172.21.15.23:0/3547079470 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6ee00aaab0 0x7f6ee013ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:37.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.848+0000 7f6ee4d3c700 1 -- 172.21.15.23:0/3547079470 >> v1:172.21.15.155:6789/0 conn(0x7f6ee013aeb0 legacy=0x7f6ee00a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:37.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.848+0000 7f6ecffff700 1 -- 172.21.15.23:0/3547079470 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3974118829 0 0) 0x7f6ee013e600 con 0x7f6ee013aeb0 2020-11-20T11:36:37.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.848+0000 7f6ee4d3c700 1 -- 172.21.15.23:0/3547079470 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee00a9370 con 0x7f6ee00aae90 2020-11-20T11:36:37.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.848+0000 7f6ee553d700 1 --2- 172.21.15.23:0/3547079470 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6ee00aaab0 0x7f6ee013ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:36:37.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.848+0000 7f6ee4d3c700 1 --2- 172.21.15.23:0/3547079470 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ee00aae90 0x7f6ee01381a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f6ed0006fd0 tx=0x7f6ed0006e30).ready entity=mon.1 client_cookie=829a39d3ac2e4325 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:37.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.848+0000 7f6ecffff700 1 -- 172.21.15.23:0/3547079470 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed00097e0 con 0x7f6ee00aae90 2020-11-20T11:36:37.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.848+0000 7f6ecffff700 1 -- 172.21.15.23:0/3547079470 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed0000e90 con 0x7f6ee00aae90 2020-11-20T11:36:37.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.848+0000 7f6ecffff700 1 -- 172.21.15.23:0/3547079470 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed0009b50 con 0x7f6ee00aae90 2020-11-20T11:36:37.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.849+0000 7f6ee77a1700 1 -- 172.21.15.23:0/3547079470 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ee00aae90 msgr2=0x7f6ee01381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:37.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.849+0000 7f6ee77a1700 1 --2- 172.21.15.23:0/3547079470 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ee00aae90 0x7f6ee01381a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f6ed0006fd0 tx=0x7f6ed0006e30).stop 2020-11-20T11:36:37.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.849+0000 7f6ee77a1700 1 -- 172.21.15.23:0/3547079470 shutdown_connections 2020-11-20T11:36:37.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.849+0000 7f6ee77a1700 1 --2- 172.21.15.23:0/3547079470 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6ee00aaab0 0x7f6ee013ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:37.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.849+0000 7f6ee77a1700 1 --2- 172.21.15.23:0/3547079470 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ee00aae90 0x7f6ee01381a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:37.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.849+0000 7f6ee77a1700 1 -- 172.21.15.23:0/3547079470 >> 172.21.15.23:0/3547079470 conn(0x7f6ee00678e0 msgr2=0x7f6ee0067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:37.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.849+0000 7f6ee77a1700 1 -- 172.21.15.23:0/3547079470 shutdown_connections 2020-11-20T11:36:37.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.849+0000 7f6ee77a1700 1 -- 172.21.15.23:0/3547079470 wait complete. 2020-11-20T11:36:37.861 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.850+0000 7f6ee77a1700 1 Processor -- start 2020-11-20T11:36:37.861 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.850+0000 7f6ee77a1700 1 -- start start 2020-11-20T11:36:37.862 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.851+0000 7f6ee77a1700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ee00aaab0 0x7f6ee00b0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:37.862 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.851+0000 7f6ee77a1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee00aae90 0x7f6ee00a7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:37.862 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.851+0000 7f6ee77a1700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6ee00a7fc0 0x7f6ee014d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:37.862 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.851+0000 7f6ee77a1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee00a5490 con 0x7f6ee00aae90 2020-11-20T11:36:37.863 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.851+0000 7f6ee77a1700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee00a5730 con 0x7f6ee00aaab0 2020-11-20T11:36:37.863 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.851+0000 7f6ee77a1700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee00a5bd0 con 0x7f6ee00a7fc0 2020-11-20T11:36:37.863 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.851+0000 7f6ee4d3c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee00aae90 0x7f6ee00a7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:37.863 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.851+0000 7f6ee4d3c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee00aae90 0x7f6ee00a7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42824/0 (socket says 172.21.15.23:42824) 2020-11-20T11:36:37.863 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.851+0000 7f6ee553d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ee00aaab0 0x7f6ee00b0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:37.864 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.851+0000 7f6ee4d3c700 1 -- 172.21.15.23:0/3679657040 learned_addr learned my addr 172.21.15.23:0/3679657040 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:36:37.864 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.851+0000 7f6ee5d3e700 1 --2- 172.21.15.23:0/3679657040 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6ee00a7fc0 0x7f6ee014d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:37.864 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.851+0000 7f6ee553d700 1 -- 172.21.15.23:0/3679657040 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6ee00a7fc0 msgr2=0x7f6ee014d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:37.864 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.851+0000 7f6ee553d700 1 --2- 172.21.15.23:0/3679657040 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6ee00a7fc0 0x7f6ee014d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:37.864 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.851+0000 7f6ee553d700 1 -- 172.21.15.23:0/3679657040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee00aae90 msgr2=0x7f6ee00a7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:37.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.851+0000 7f6ee553d700 1 --2- 172.21.15.23:0/3679657040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee00aae90 0x7f6ee00a7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:37.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.851+0000 7f6ee553d700 1 -- 172.21.15.23:0/3679657040 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee00a7520 con 0x7f6ee00aaab0 2020-11-20T11:36:37.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.851+0000 7f6ee4d3c700 1 --2- 172.21.15.23:0/3679657040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee00aae90 0x7f6ee00a7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:36:37.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.852+0000 7f6ee553d700 1 --2- 172.21.15.23:0/3679657040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ee00aaab0 0x7f6ee00b0bf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f6edc00ecf0 tx=0x7f6edc006e30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:37.866 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.852+0000 7f6ecd7fa700 1 -- 172.21.15.23:0/3679657040 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6edc00b550 con 0x7f6ee00aaab0 2020-11-20T11:36:37.866 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.852+0000 7f6ecd7fa700 1 -- 172.21.15.23:0/3679657040 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6edc00c980 con 0x7f6ee00aaab0 2020-11-20T11:36:37.866 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.852+0000 7f6ecd7fa700 1 -- 172.21.15.23:0/3679657040 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6edc00cd00 con 0x7f6ee00aaab0 2020-11-20T11:36:37.866 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.852+0000 7f6ee77a1700 1 -- 172.21.15.23:0/3679657040 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ee014db10 con 0x7f6ee00aaab0 2020-11-20T11:36:37.867 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.852+0000 7f6ee77a1700 1 -- 172.21.15.23:0/3679657040 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee0135860 con 0x7f6ee00aaab0 2020-11-20T11:36:37.867 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.853+0000 7f6ee77a1700 1 --2- 172.21.15.23:0/3679657040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee013aeb0 0x7f6ee0135a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:37.867 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.853+0000 7f6ee77a1700 1 -- 172.21.15.23:0/3679657040 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee00a5d90 con 0x7f6ee013aeb0 2020-11-20T11:36:37.867 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.853+0000 7f6ee4d3c700 1 --2- 172.21.15.23:0/3679657040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee013aeb0 0x7f6ee0135a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:37.867 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.853+0000 7f6ecd7fa700 1 -- 172.21.15.23:0/3679657040 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6edc012600 con 0x7f6ee00aaab0 2020-11-20T11:36:37.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.853+0000 7f6ee4d3c700 1 -- 172.21.15.23:0/3679657040 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6ee00a7830 con 0x7f6ee013aeb0 2020-11-20T11:36:37.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.853+0000 7f6ee4d3c700 1 --2- 172.21.15.23:0/3679657040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee013aeb0 0x7f6ee0135a80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f6ee014e670 tx=0x7f6ed0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:37.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.853+0000 7f6ecd7fa700 1 --2- 172.21.15.23:0/3679657040 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6eb8056d90 0x7f6eb8059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:37.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.854+0000 7f6ecd7fa700 1 -- 172.21.15.23:0/3679657040 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f6edc070450 con 0x7f6ee00aaab0 2020-11-20T11:36:37.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.854+0000 7f6ecd7fa700 1 -- 172.21.15.23:0/3679657040 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed0011c70 con 0x7f6ee013aeb0 2020-11-20T11:36:37.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.854+0000 7f6ee5d3e700 1 --2- 172.21.15.23:0/3679657040 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6eb8056d90 0x7f6eb8059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:37.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.854+0000 7f6ecd7fa700 1 -- 172.21.15.23:0/3679657040 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f6ed0011dd0 con 0x7f6ee013aeb0 2020-11-20T11:36:37.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.854+0000 7f6ecd7fa700 1 -- 172.21.15.23:0/3679657040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee013aeb0 msgr2=0x7f6ee0135a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:37.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.854+0000 7f6ecd7fa700 1 --2- 172.21.15.23:0/3679657040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee013aeb0 0x7f6ee0135a80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f6ee014e670 tx=0x7f6ed0015000).stop 2020-11-20T11:36:37.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.854+0000 7f6ecd7fa700 1 -- 172.21.15.23:0/3679657040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee013aeb0 msgr2=0x7f6ee0135a80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:36:37.870 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.854+0000 7f6ecd7fa700 1 --2- 172.21.15.23:0/3679657040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee013aeb0 0x7f6ee0135a80 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:37.870 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.854+0000 7f6ee5d3e700 1 --2- 172.21.15.23:0/3679657040 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6eb8056d90 0x7f6eb8059250 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f6ee00a83e0 tx=0x7f6ed8007f60).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:37.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.862+0000 7f6ee77a1700 1 --2- 172.21.15.23:0/3679657040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee00601b0 0x7f6ee0141700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:37.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.862+0000 7f6ee77a1700 1 -- 172.21.15.23:0/3679657040 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee014ddb0 con 0x7f6ee00601b0 2020-11-20T11:36:37.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.862+0000 7f6ee4d3c700 1 --2- 172.21.15.23:0/3679657040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee00601b0 0x7f6ee0141700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:37.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.862+0000 7f6ee4d3c700 1 -- 172.21.15.23:0/3679657040 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6ee005c7d0 con 0x7f6ee00601b0 2020-11-20T11:36:37.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.862+0000 7f6ee4d3c700 1 --2- 172.21.15.23:0/3679657040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee00601b0 0x7f6ee0141700 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f6ed0009c80 tx=0x7f6ed0021070).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:37.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.862+0000 7f6ecd7fa700 1 -- 172.21.15.23:0/3679657040 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed0029b60 con 0x7f6ee00601b0 2020-11-20T11:36:37.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.862+0000 7f6ecd7fa700 1 -- 172.21.15.23:0/3679657040 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4786 (secure 0 0 0) 0x7f6ed0011a60 con 0x7f6ee00601b0 2020-11-20T11:36:37.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.862+0000 7f6ecd7fa700 1 -- 172.21.15.23:0/3679657040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee00601b0 msgr2=0x7f6ee0141700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:37.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.862+0000 7f6ecd7fa700 1 --2- 172.21.15.23:0/3679657040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee00601b0 0x7f6ee0141700 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f6ed0009c80 tx=0x7f6ed0021070).stop 2020-11-20T11:36:37.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.863+0000 7f6ecd7fa700 1 -- 172.21.15.23:0/3679657040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee00601b0 msgr2=0x7f6ee0141700 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:36:37.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.863+0000 7f6ecd7fa700 1 --2- 172.21.15.23:0/3679657040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee00601b0 0x7f6ee0141700 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:37.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.863+0000 7f6ee77a1700 1 -- 172.21.15.23:0/3679657040 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6eb8056d90 msgr2=0x7f6eb8059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:37.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.863+0000 7f6ee77a1700 1 --2- 172.21.15.23:0/3679657040 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6eb8056d90 0x7f6eb8059250 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f6ee00a83e0 tx=0x7f6ed8007f60).stop 2020-11-20T11:36:37.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.863+0000 7f6ee77a1700 1 -- 172.21.15.23:0/3679657040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ee00aaab0 msgr2=0x7f6ee00b0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:37.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.863+0000 7f6ee77a1700 1 --2- 172.21.15.23:0/3679657040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ee00aaab0 0x7f6ee00b0bf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f6edc00ecf0 tx=0x7f6edc006e30).stop 2020-11-20T11:36:37.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.863+0000 7f6ee5d3e700 1 -- 172.21.15.23:0/3679657040 reap_dead start 2020-11-20T11:36:37.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.863+0000 7f6ee77a1700 1 -- 172.21.15.23:0/3679657040 shutdown_connections 2020-11-20T11:36:37.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.863+0000 7f6ee77a1700 1 --2- 172.21.15.23:0/3679657040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ee00aaab0 0x7f6ee00b0bf0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:37.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.863+0000 7f6ee77a1700 1 -- 172.21.15.23:0/3679657040 >> 172.21.15.23:0/3679657040 conn(0x7f6ee00678e0 msgr2=0x7f6ee00addc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:37.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.863+0000 7f6ee77a1700 1 -- 172.21.15.23:0/3679657040 shutdown_connections 2020-11-20T11:36:37.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:37.863+0000 7f6ee77a1700 1 -- 172.21.15.23:0/3679657040 wait complete. 2020-11-20T11:36:37.877 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:36:37.877 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:36:37.877 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:36:37.878 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:36:37.878 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 38, 2020-11-20T11:36:37.878 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:36:37.878 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:36:37.878 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:36:37.879 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:36:37.879 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:37.879 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 14, 2020-11-20T11:36:37.879 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:36:37.879 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:36:37.880 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:36:37.880 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:36:37.880 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:36:37.880 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:36:37.880 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:36:37.881 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:36:37.881 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:36:37.881 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:36:37.881 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:36:37.881 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:37.882 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:36:37.882 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:36:37.882 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:36:37.882 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:36:37.882 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:36:37.883 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:36:37.883 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:36:37.883 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:36:37.883 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:36:37.884 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:36:37.884 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:37.884 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:37.884 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:36:37.884 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:36:37.885 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:36:37.885 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:36:37.885 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:36:37.885 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:36:37.885 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:36:37.886 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:36:37.886 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:36:37.886 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:36:37.886 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:36:37.886 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:36:37.887 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:36:37.887 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:36:37.887 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:36:37.887 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:36:37.887 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:36:37.888 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:36:37.888 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:36:37.888 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:36:37.888 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:36:37.888 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:36:37.889 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:36:37.889 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:37.889 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:36:37.889 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:37.889 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:36:37.890 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:37.890 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:36:37.890 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:36:37.890 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:36:37.890 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:36:37.891 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:37.891 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:36:37.891 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:36:37.891 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:37.891 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:37.892 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:37.892 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:36:37.892 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:36:37.892 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:37.893 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:37.893 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:37.893 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:37.893 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:36:37.893 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:36:37.894 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:36:37.894 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:36:37.894 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:36:37.894 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:37.894 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:37.895 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:36:37.895 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:36:37.895 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:36:37.895 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:36:37.895 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:37.896 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:36:37.896 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:36:37.896 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:37.896 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:37.897 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:37.897 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:36:37.897 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:36:37.897 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:37.897 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:37.898 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:37.898 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:37.898 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:36:37.898 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:36:37.898 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:36:37.899 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:36:37.899 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:36:37.899 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:37.899 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:37.900 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:36:37.900 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:36:37.900 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:36:37.900 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:36:37.900 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:37.900 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:36:37.901 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:36:37.901 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:37.901 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:37.901 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:37.902 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:36:37.902 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:36:37.902 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:37.902 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:37.902 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:37.903 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:37.903 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:36:37.903 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:36:37.903 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:36:37.903 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:36:37.904 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:36:37.904 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:37.904 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:37.904 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:37.905 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:36:37.905 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:36:37.905 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:37.905 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:36:37.905 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:36:37.906 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:36:37.906 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:37.906 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:37.906 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:36:37.906 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:37.907 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:36:37.907 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:36:37.907 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:36:37.907 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:37.907 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:37.908 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:36:37.908 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:37.908 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:36:37.908 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:36:37.909 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 6 2020-11-20T11:36:37.909 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:37.909 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:37.909 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:36:37.910 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:37.910 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:36:37.910 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:36:37.910 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 4 2020-11-20T11:36:37.910 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:37.911 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:37.911 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:36:37.911 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:37.911 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:36:37.912 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:36:37.912 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:36:37.912 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:37.912 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:37.912 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:37.913 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:36:37.913 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:36:37.914 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:36:38.040 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.036+0000 7fd97bcc1700 1 Processor -- start 2020-11-20T11:36:38.041 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.036+0000 7fd97bcc1700 1 -- start start 2020-11-20T11:36:38.043 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.037+0000 7fd97bcc1700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd974061db0 0x7fd9740a9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:38.043 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.037+0000 7fd97bcc1700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9740610b0 con 0x7fd974066170 2020-11-20T11:36:38.043 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.037+0000 7fd97bcc1700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9740a9750 con 0x7fd9740619d0 2020-11-20T11:36:38.044 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.037+0000 7fd97bcc1700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd974066680 con 0x7fd974061db0 2020-11-20T11:36:38.044 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.037+0000 7fd97925c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd974061db0 0x7fd9740a9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:38.044 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.037+0000 7fd97925c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd974061db0 0x7fd9740a9260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42838/0 (socket says 172.21.15.23:42838) 2020-11-20T11:36:38.044 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.037+0000 7fd97925c700 1 -- 172.21.15.23:0/1093312543 learned_addr learned my addr 172.21.15.23:0/1093312543 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:36:38.045 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.037+0000 7fd97a25e700 1 --1- 172.21.15.23:0/1093312543 >> v1:172.21.15.23:6789/0 conn(0x7fd974066170 0x7fd974137e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57086/0 (socket says 172.21.15.23:57086) 2020-11-20T11:36:38.045 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.038+0000 7fd97925c700 1 -- 172.21.15.23:0/1093312543 >> v1:172.21.15.155:6789/0 conn(0x7fd9740619d0 legacy=0x7fd9740609a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:38.045 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.038+0000 7fd97925c700 1 -- 172.21.15.23:0/1093312543 >> v1:172.21.15.23:6789/0 conn(0x7fd974066170 legacy=0x7fd974137e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:38.045 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.038+0000 7fd978a5b700 1 -- 172.21.15.23:0/1093312543 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 472267829 0 0) 0x7fd9740610b0 con 0x7fd974066170 2020-11-20T11:36:38.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.038+0000 7fd97925c700 1 -- 172.21.15.23:0/1093312543 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd974066920 con 0x7fd974061db0 2020-11-20T11:36:38.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.038+0000 7fd97925c700 1 --2- 172.21.15.23:0/1093312543 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd974061db0 0x7fd9740a9260 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fd964009a10 tx=0x7fd964006ef0).ready entity=mon.0 client_cookie=c02f02641bd33287 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:38.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.038+0000 7fd978a5b700 1 -- 172.21.15.23:0/1093312543 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96400b860 con 0x7fd974061db0 2020-11-20T11:36:38.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.038+0000 7fd978a5b700 1 -- 172.21.15.23:0/1093312543 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd964000e90 con 0x7fd974061db0 2020-11-20T11:36:38.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.038+0000 7fd978a5b700 1 -- 172.21.15.23:0/1093312543 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96400bbd0 con 0x7fd974061db0 2020-11-20T11:36:38.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.039+0000 7fd97bcc1700 1 -- 172.21.15.23:0/1093312543 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd974061db0 msgr2=0x7fd9740a9260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:38.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.039+0000 7fd97bcc1700 1 --2- 172.21.15.23:0/1093312543 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd974061db0 0x7fd9740a9260 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fd964009a10 tx=0x7fd964006ef0).stop 2020-11-20T11:36:38.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.039+0000 7fd97bcc1700 1 -- 172.21.15.23:0/1093312543 shutdown_connections 2020-11-20T11:36:38.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.039+0000 7fd97bcc1700 1 --2- 172.21.15.23:0/1093312543 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd974061db0 0x7fd9740a9260 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:38.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.039+0000 7fd97bcc1700 1 -- 172.21.15.23:0/1093312543 >> 172.21.15.23:0/1093312543 conn(0x7fd974063640 msgr2=0x7fd9740612c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:38.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.039+0000 7fd97bcc1700 1 -- 172.21.15.23:0/1093312543 shutdown_connections 2020-11-20T11:36:38.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.039+0000 7fd97bcc1700 1 -- 172.21.15.23:0/1093312543 wait complete. 2020-11-20T11:36:38.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.040+0000 7fd97bcc1700 1 Processor -- start 2020-11-20T11:36:38.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.040+0000 7fd97bcc1700 1 -- start start 2020-11-20T11:36:38.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.040+0000 7fd97bcc1700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9740619d0 0x7fd97413d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:38.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.041+0000 7fd97bcc1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd974061db0 0x7fd97414dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:38.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.041+0000 7fd97bcc1700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd974066170 0x7fd97414e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:38.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.041+0000 7fd97bcc1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9740aca60 con 0x7fd974061db0 2020-11-20T11:36:38.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.041+0000 7fd97bcc1700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97414e5c0 con 0x7fd9740619d0 2020-11-20T11:36:38.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.041+0000 7fd97bcc1700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd974134ba0 con 0x7fd974066170 2020-11-20T11:36:38.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.041+0000 7fd97925c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd974061db0 0x7fd97414dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:38.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.041+0000 7fd97925c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd974061db0 0x7fd97414dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42842/0 (socket says 172.21.15.23:42842) 2020-11-20T11:36:38.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.041+0000 7fd97925c700 1 -- 172.21.15.23:0/3709946 learned_addr learned my addr 172.21.15.23:0/3709946 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:36:38.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.041+0000 7fd97a25e700 1 --2- 172.21.15.23:0/3709946 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd974066170 0x7fd97414e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:38.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.041+0000 7fd979a5d700 1 --2- 172.21.15.23:0/3709946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9740619d0 0x7fd97413d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:38.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.041+0000 7fd97925c700 1 -- 172.21.15.23:0/3709946 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd974066170 msgr2=0x7fd97414e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:38.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.041+0000 7fd97925c700 1 --2- 172.21.15.23:0/3709946 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd974066170 0x7fd97414e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:38.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.041+0000 7fd97925c700 1 -- 172.21.15.23:0/3709946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9740619d0 msgr2=0x7fd97413d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:38.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.041+0000 7fd97925c700 1 --2- 172.21.15.23:0/3709946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd9740619d0 0x7fd97413d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:38.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.041+0000 7fd97925c700 1 -- 172.21.15.23:0/3709946 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd974134dd0 con 0x7fd974061db0 2020-11-20T11:36:38.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.041+0000 7fd97925c700 1 --2- 172.21.15.23:0/3709946 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd974061db0 0x7fd97414dbe0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fd964009f80 tx=0x7fd964016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:38.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.041+0000 7fd961ffb700 1 -- 172.21.15.23:0/3709946 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd964011dc0 con 0x7fd974061db0 2020-11-20T11:36:38.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.041+0000 7fd961ffb700 1 -- 172.21.15.23:0/3709946 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd96401c440 con 0x7fd974061db0 2020-11-20T11:36:38.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.041+0000 7fd97bcc1700 1 -- 172.21.15.23:0/3709946 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd97413e0e0 con 0x7fd974061db0 2020-11-20T11:36:38.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.041+0000 7fd961ffb700 1 -- 172.21.15.23:0/3709946 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96401c7c0 con 0x7fd974061db0 2020-11-20T11:36:38.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.042+0000 7fd97bcc1700 1 -- 172.21.15.23:0/3709946 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd974135c00 con 0x7fd974061db0 2020-11-20T11:36:38.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.042+0000 7fd97bcc1700 1 --2- 172.21.15.23:0/3709946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd974135f60 0x7fd9741312c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:38.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.042+0000 7fd97bcc1700 1 -- 172.21.15.23:0/3709946 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd974135630 con 0x7fd974135f60 2020-11-20T11:36:38.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.043+0000 7fd979a5d700 1 --2- 172.21.15.23:0/3709946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd974135f60 0x7fd9741312c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:38.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.043+0000 7fd961ffb700 1 -- 172.21.15.23:0/3709946 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd964017040 con 0x7fd974061db0 2020-11-20T11:36:38.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.043+0000 7fd979a5d700 1 -- 172.21.15.23:0/3709946 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd9740a9380 con 0x7fd974135f60 2020-11-20T11:36:38.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.043+0000 7fd979a5d700 1 --2- 172.21.15.23:0/3709946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd974135f60 0x7fd9741312c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fd97000f850 tx=0x7fd970007bd0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:38.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.043+0000 7fd961ffb700 1 --2- 172.21.15.23:0/3709946 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd94c056d60 0x7fd94c059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:38.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.043+0000 7fd961ffb700 1 -- 172.21.15.23:0/3709946 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fd964071b40 con 0x7fd974061db0 2020-11-20T11:36:38.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.043+0000 7fd961ffb700 1 -- 172.21.15.23:0/3709946 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd970003040 con 0x7fd974135f60 2020-11-20T11:36:38.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.043+0000 7fd97a25e700 1 --2- 172.21.15.23:0/3709946 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd94c056d60 0x7fd94c059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:38.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.044+0000 7fd961ffb700 1 -- 172.21.15.23:0/3709946 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fd970007030 con 0x7fd974135f60 2020-11-20T11:36:38.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.044+0000 7fd961ffb700 1 -- 172.21.15.23:0/3709946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd974135f60 msgr2=0x7fd9741312c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:38.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.044+0000 7fd961ffb700 1 --2- 172.21.15.23:0/3709946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd974135f60 0x7fd9741312c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fd97000f850 tx=0x7fd970007bd0).stop 2020-11-20T11:36:38.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.044+0000 7fd961ffb700 1 -- 172.21.15.23:0/3709946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd974135f60 msgr2=0x7fd9741312c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:36:38.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.044+0000 7fd961ffb700 1 --2- 172.21.15.23:0/3709946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd974135f60 0x7fd9741312c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:38.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.044+0000 7fd97a25e700 1 --2- 172.21.15.23:0/3709946 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd94c056d60 0x7fd94c059220 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fd96c007fd0 tx=0x7fd96c007f60).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:38.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.052+0000 7fd97bcc1700 1 --2- 172.21.15.23:0/3709946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd97405c070 0x7fd9741317b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:38.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.052+0000 7fd97bcc1700 1 -- 172.21.15.23:0/3709946 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd974131ca0 con 0x7fd97405c070 2020-11-20T11:36:38.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.052+0000 7fd979a5d700 1 --2- 172.21.15.23:0/3709946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd97405c070 0x7fd9741317b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:38.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.052+0000 7fd979a5d700 1 -- 172.21.15.23:0/3709946 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd9741351b0 con 0x7fd97405c070 2020-11-20T11:36:38.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.052+0000 7fd979a5d700 1 --2- 172.21.15.23:0/3709946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd97405c070 0x7fd9741317b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fd97000f850 tx=0x7fd970018000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:38.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.053+0000 7fd961ffb700 1 -- 172.21.15.23:0/3709946 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97001e040 con 0x7fd97405c070 2020-11-20T11:36:38.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.053+0000 7fd961ffb700 1 -- 172.21.15.23:0/3709946 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4614 (secure 0 0 0) 0x7fd97000d4f0 con 0x7fd97405c070 2020-11-20T11:36:38.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.053+0000 7fd961ffb700 1 -- 172.21.15.23:0/3709946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd97405c070 msgr2=0x7fd9741317b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:38.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.053+0000 7fd961ffb700 1 --2- 172.21.15.23:0/3709946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd97405c070 0x7fd9741317b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fd97000f850 tx=0x7fd970018000).stop 2020-11-20T11:36:38.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.053+0000 7fd961ffb700 1 -- 172.21.15.23:0/3709946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd97405c070 msgr2=0x7fd9741317b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:36:38.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.053+0000 7fd961ffb700 1 --2- 172.21.15.23:0/3709946 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd97405c070 0x7fd9741317b0 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0x7fd97000f850 tx=0x7fd970018000).stop 2020-11-20T11:36:38.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.053+0000 7fd97bcc1700 1 -- 172.21.15.23:0/3709946 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd94c056d60 msgr2=0x7fd94c059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:38.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.053+0000 7fd97bcc1700 1 --2- 172.21.15.23:0/3709946 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd94c056d60 0x7fd94c059220 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fd96c007fd0 tx=0x7fd96c007f60).stop 2020-11-20T11:36:38.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.053+0000 7fd97bcc1700 1 -- 172.21.15.23:0/3709946 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd974061db0 msgr2=0x7fd97414dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:38.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.053+0000 7fd97bcc1700 1 --2- 172.21.15.23:0/3709946 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd974061db0 0x7fd97414dbe0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fd964009f80 tx=0x7fd964016000).stop 2020-11-20T11:36:38.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.053+0000 7fd97a25e700 1 -- 172.21.15.23:0/3709946 reap_dead start 2020-11-20T11:36:38.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.053+0000 7fd97a25e700 1 -- 172.21.15.23:0/3709946 reap_dead start 2020-11-20T11:36:38.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.053+0000 7fd97bcc1700 1 -- 172.21.15.23:0/3709946 shutdown_connections 2020-11-20T11:36:38.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.053+0000 7fd97bcc1700 1 -- 172.21.15.23:0/3709946 >> 172.21.15.23:0/3709946 conn(0x7fd974063640 msgr2=0x7fd9740aa7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:38.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.053+0000 7fd97bcc1700 1 -- 172.21.15.23:0/3709946 shutdown_connections 2020-11-20T11:36:38.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.053+0000 7fd97bcc1700 1 -- 172.21.15.23:0/3709946 wait complete. 2020-11-20T11:36:38.067 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:36:38.067 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:36:38.068 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:36:38.068 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:36:38.068 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 38, 2020-11-20T11:36:38.068 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:36:38.069 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:36:38.069 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:36:38.069 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:36:38.069 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:38.070 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 14, 2020-11-20T11:36:38.070 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:36:38.070 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:36:38.070 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:36:38.071 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:36:38.071 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:36:38.071 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:36:38.071 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:36:38.071 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:36:38.072 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:36:38.072 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:36:38.072 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:36:38.072 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:38.072 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:36:38.073 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:36:38.073 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:36:38.073 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:36:38.073 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:36:38.074 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:36:38.074 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:36:38.074 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:36:38.074 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:36:38.074 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:36:38.075 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:38.075 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:38.075 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:36:38.076 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:36:38.076 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:36:38.076 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:36:38.076 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:36:38.076 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:36:38.077 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:36:38.077 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:36:38.077 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:36:38.077 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:36:38.077 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:36:38.078 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:36:38.078 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:36:38.078 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:36:38.078 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:36:38.079 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:36:38.079 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:36:38.079 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:36:38.079 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:36:38.079 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:36:38.080 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:36:38.080 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:36:38.080 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:36:38.080 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:38.080 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:36:38.081 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:38.081 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:36:38.081 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:38.082 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:36:38.082 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:36:38.082 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:36:38.082 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:36:38.082 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:38.083 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:36:38.083 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:36:38.083 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:38.083 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:38.083 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:38.084 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:36:38.084 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:36:38.084 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:38.084 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:38.085 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:38.085 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:38.085 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:36:38.085 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:36:38.086 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:36:38.086 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:36:38.086 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:36:38.086 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:38.086 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:38.087 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:36:38.087 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:36:38.087 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:36:38.087 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:36:38.088 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:38.088 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:36:38.088 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:36:38.088 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:38.088 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:38.089 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:38.089 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:36:38.089 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:36:38.089 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:38.090 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:38.090 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:38.090 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:38.090 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:36:38.090 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:36:38.091 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:36:38.091 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:36:38.091 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:36:38.091 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:38.092 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:38.092 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:36:38.092 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:36:38.092 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:36:38.092 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:36:38.093 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:38.093 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:36:38.093 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:36:38.093 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:38.094 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:38.094 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:38.094 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:36:38.094 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:36:38.094 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:38.095 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:38.095 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:38.095 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:38.095 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:36:38.096 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:36:38.096 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:36:38.096 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:36:38.096 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:36:38.096 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:38.097 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:38.097 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:38.097 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:36:38.097 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:36:38.098 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:38.098 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:36:38.098 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:36:38.098 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:36:38.099 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:38.099 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:38.099 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:36:38.099 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:38.099 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:36:38.100 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:36:38.100 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:36:38.100 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:38.100 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:38.100 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:36:38.101 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:38.101 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:36:38.101 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:36:38.101 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:36:38.102 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:38.102 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:38.102 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:36:38.102 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:38.103 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:36:38.103 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:36:38.103 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:36:38.103 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:38.103 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:38.104 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:38.104 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:36:38.104 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:36:38.106 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:36:38.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.231+0000 7f6c42ab3700 1 Processor -- start 2020-11-20T11:36:38.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.231+0000 7f6c42ab3700 1 -- start start 2020-11-20T11:36:38.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.232+0000 7f6c42ab3700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6c3c13bfa0 0x7f6c3c0a9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:38.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.232+0000 7f6c42ab3700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c3c0a8c20 con 0x7f6c3c067fd0 2020-11-20T11:36:38.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.232+0000 7f6c42ab3700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c3c0684e0 con 0x7f6c3c05bc80 2020-11-20T11:36:38.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.232+0000 7f6c42ab3700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c3c13c410 con 0x7f6c3c13bfa0 2020-11-20T11:36:38.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.232+0000 7f6c41050700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6c3c13bfa0 0x7f6c3c0a9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:38.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.232+0000 7f6c41050700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6c3c13bfa0 0x7f6c3c0a9080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42856/0 (socket says 172.21.15.23:42856) 2020-11-20T11:36:38.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.232+0000 7f6c41050700 1 -- 172.21.15.23:0/43868876 learned_addr learned my addr 172.21.15.23:0/43868876 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:36:38.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.232+0000 7f6c3bfff700 1 --1- 172.21.15.23:0/43868876 >> v1:172.21.15.23:6789/0 conn(0x7f6c3c067fd0 0x7f6c3c1412f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57104/0 (socket says 172.21.15.23:57104) 2020-11-20T11:36:38.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.232+0000 7f6c41050700 1 -- 172.21.15.23:0/43868876 >> v1:172.21.15.155:6789/0 conn(0x7f6c3c05bc80 legacy=0x7f6c3c0a6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:36:38.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.232+0000 7f6c41050700 1 -- 172.21.15.23:0/43868876 >> v1:172.21.15.23:6789/0 conn(0x7f6c3c067fd0 legacy=0x7f6c3c1412f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:38.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.232+0000 7f6c41050700 1 -- 172.21.15.23:0/43868876 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3c05c060 con 0x7f6c3c13bfa0 2020-11-20T11:36:38.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.233+0000 7f6c41050700 1 --2- 172.21.15.23:0/43868876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c3c13bfa0 0x7f6c3c0a9080 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f6c34009d90 tx=0x7f6c340092d0).ready entity=mon.0 client_cookie=361a259504cad6dd server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:38.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.233+0000 7f6c3b7fe700 1 -- 172.21.15.23:0/43868876 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c3400bbb0 con 0x7f6c3c13bfa0 2020-11-20T11:36:38.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.233+0000 7f6c3b7fe700 1 -- 172.21.15.23:0/43868876 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c3400bd10 con 0x7f6c3c13bfa0 2020-11-20T11:36:38.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.233+0000 7f6c3b7fe700 1 -- 172.21.15.23:0/43868876 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c34011610 con 0x7f6c3c13bfa0 2020-11-20T11:36:38.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.233+0000 7f6c42ab3700 1 -- 172.21.15.23:0/43868876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c3c13bfa0 msgr2=0x7f6c3c0a9080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:38.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.233+0000 7f6c42ab3700 1 --2- 172.21.15.23:0/43868876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c3c13bfa0 0x7f6c3c0a9080 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f6c34009d90 tx=0x7f6c340092d0).stop 2020-11-20T11:36:38.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.233+0000 7f6c42ab3700 1 -- 172.21.15.23:0/43868876 shutdown_connections 2020-11-20T11:36:38.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.233+0000 7f6c42ab3700 1 --2- 172.21.15.23:0/43868876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c3c13bfa0 0x7f6c3c0a9080 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:38.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.233+0000 7f6c42ab3700 1 -- 172.21.15.23:0/43868876 >> 172.21.15.23:0/43868876 conn(0x7f6c3c0ad7a0 msgr2=0x7f6c3c0adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:38.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.233+0000 7f6c42ab3700 1 -- 172.21.15.23:0/43868876 shutdown_connections 2020-11-20T11:36:38.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.233+0000 7f6c42ab3700 1 -- 172.21.15.23:0/43868876 wait complete. 2020-11-20T11:36:38.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.234+0000 7f6c42ab3700 1 Processor -- start 2020-11-20T11:36:38.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.235+0000 7f6c42ab3700 1 -- start start 2020-11-20T11:36:38.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.235+0000 7f6c42ab3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c3c05bc80 0x7f6c3c0a2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:38.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.235+0000 7f6c42ab3700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6c3c067fd0 0x7f6c3c0670e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:38.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.235+0000 7f6c42ab3700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6c3c13bfa0 0x7f6c3c13f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:38.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.235+0000 7f6c42ab3700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c065f40 con 0x7f6c3c05bc80 2020-11-20T11:36:38.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.235+0000 7f6c42ab3700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c0661e0 con 0x7f6c3c067fd0 2020-11-20T11:36:38.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.235+0000 7f6c42ab3700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c066480 con 0x7f6c3c13bfa0 2020-11-20T11:36:38.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.235+0000 7f6c4084f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c3c05bc80 0x7f6c3c0a2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:38.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.235+0000 7f6c4084f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c3c05bc80 0x7f6c3c0a2bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42858/0 (socket says 172.21.15.23:42858) 2020-11-20T11:36:38.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.235+0000 7f6c4084f700 1 -- 172.21.15.23:0/3592962018 learned_addr learned my addr 172.21.15.23:0/3592962018 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:36:38.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.236+0000 7f6c3bfff700 1 --2- 172.21.15.23:0/3592962018 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6c3c067fd0 0x7f6c3c0670e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:38.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.236+0000 7f6c41050700 1 --2- 172.21.15.23:0/3592962018 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6c3c13bfa0 0x7f6c3c13f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:38.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.236+0000 7f6c4084f700 1 -- 172.21.15.23:0/3592962018 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6c3c13bfa0 msgr2=0x7f6c3c13f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:38.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.236+0000 7f6c4084f700 1 --2- 172.21.15.23:0/3592962018 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6c3c13bfa0 0x7f6c3c13f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:38.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.236+0000 7f6c4084f700 1 -- 172.21.15.23:0/3592962018 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6c3c067fd0 msgr2=0x7f6c3c0670e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:38.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.236+0000 7f6c4084f700 1 --2- 172.21.15.23:0/3592962018 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6c3c067fd0 0x7f6c3c0670e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:38.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.236+0000 7f6c4084f700 1 -- 172.21.15.23:0/3592962018 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3c0675d0 con 0x7f6c3c05bc80 2020-11-20T11:36:38.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.236+0000 7f6c4084f700 1 --2- 172.21.15.23:0/3592962018 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c3c05bc80 0x7f6c3c0a2bc0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f6c3000c730 tx=0x7f6c30007c70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:38.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.236+0000 7f6c38ff9700 1 -- 172.21.15.23:0/3592962018 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c3000e4a0 con 0x7f6c3c05bc80 2020-11-20T11:36:38.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.236+0000 7f6c38ff9700 1 -- 172.21.15.23:0/3592962018 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c3000e620 con 0x7f6c3c05bc80 2020-11-20T11:36:38.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.236+0000 7f6c42ab3700 1 -- 172.21.15.23:0/3592962018 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c3c065a30 con 0x7f6c3c05bc80 2020-11-20T11:36:38.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.236+0000 7f6c38ff9700 1 -- 172.21.15.23:0/3592962018 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c3000e9e0 con 0x7f6c3c05bc80 2020-11-20T11:36:38.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.236+0000 7f6c42ab3700 1 -- 172.21.15.23:0/3592962018 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c3c13fae0 con 0x7f6c3c05bc80 2020-11-20T11:36:38.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.237+0000 7f6c42ab3700 1 --2- 172.21.15.23:0/3592962018 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6c3c14a050 0x7f6c3c14a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:38.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.237+0000 7f6c42ab3700 1 -- 172.21.15.23:0/3592962018 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c060560 con 0x7f6c3c14a050 2020-11-20T11:36:38.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.237+0000 7f6c3bfff700 1 --2- 172.21.15.23:0/3592962018 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6c3c14a050 0x7f6c3c14a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:38.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.237+0000 7f6c3bfff700 1 -- 172.21.15.23:0/3592962018 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6c3c065d30 con 0x7f6c3c14a050 2020-11-20T11:36:38.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.238+0000 7f6c3bfff700 1 --2- 172.21.15.23:0/3592962018 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6c3c14a050 0x7f6c3c14a430 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f6c3c0659c0 tx=0x7f6c28007bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:38.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.238+0000 7f6c38ff9700 1 -- 172.21.15.23:0/3592962018 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6c30010030 con 0x7f6c3c05bc80 2020-11-20T11:36:38.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.238+0000 7f6c38ff9700 1 --2- 172.21.15.23:0/3592962018 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6c14056d60 0x7f6c14059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:38.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.238+0000 7f6c38ff9700 1 -- 172.21.15.23:0/3592962018 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f6c3006fe50 con 0x7f6c3c05bc80 2020-11-20T11:36:38.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.238+0000 7f6c38ff9700 1 -- 172.21.15.23:0/3592962018 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c28011590 con 0x7f6c3c14a050 2020-11-20T11:36:38.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.238+0000 7f6c38ff9700 1 -- 172.21.15.23:0/3592962018 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f6c280116f0 con 0x7f6c3c14a050 2020-11-20T11:36:38.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.238+0000 7f6c38ff9700 1 -- 172.21.15.23:0/3592962018 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6c3c14a050 msgr2=0x7f6c3c14a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:38.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.238+0000 7f6c41050700 1 --2- 172.21.15.23:0/3592962018 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6c14056d60 0x7f6c14059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:38.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.238+0000 7f6c38ff9700 1 --2- 172.21.15.23:0/3592962018 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6c3c14a050 0x7f6c3c14a430 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f6c3c0659c0 tx=0x7f6c28007bc0).stop 2020-11-20T11:36:38.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.238+0000 7f6c38ff9700 1 -- 172.21.15.23:0/3592962018 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6c3c14a050 msgr2=0x7f6c3c14a430 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:36:38.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.238+0000 7f6c38ff9700 1 --2- 172.21.15.23:0/3592962018 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6c3c14a050 0x7f6c3c14a430 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0x7f6c3c0659c0 tx=0x7f6c28007bc0).stop 2020-11-20T11:36:38.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.239+0000 7f6c41050700 1 --2- 172.21.15.23:0/3592962018 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6c14056d60 0x7f6c14059220 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f6c34009310 tx=0x7f6c34016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:38.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.249+0000 7f6c42ab3700 1 --2- 172.21.15.23:0/3592962018 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6c3c14d620 0x7f6c3c14da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:38.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.249+0000 7f6c42ab3700 1 -- 172.21.15.23:0/3592962018 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c062fe0 con 0x7f6c3c14d620 2020-11-20T11:36:38.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.249+0000 7f6c3bfff700 1 --2- 172.21.15.23:0/3592962018 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6c3c14d620 0x7f6c3c14da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:38.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.249+0000 7f6c3bfff700 1 -- 172.21.15.23:0/3592962018 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6c3c044710 con 0x7f6c3c14d620 2020-11-20T11:36:38.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.250+0000 7f6c3bfff700 1 --2- 172.21.15.23:0/3592962018 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6c3c14d620 0x7f6c3c14da00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f6c2800a000 tx=0x7f6c28017000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:38.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.250+0000 7f6c38ff9700 1 -- 172.21.15.23:0/3592962018 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c280159f0 con 0x7f6c3c14d620 2020-11-20T11:36:38.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.250+0000 7f6c38ff9700 1 -- 172.21.15.23:0/3592962018 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4274 (secure 0 0 0) 0x7f6c28011380 con 0x7f6c3c14d620 2020-11-20T11:36:38.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.250+0000 7f6c38ff9700 1 -- 172.21.15.23:0/3592962018 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6c3c14d620 msgr2=0x7f6c3c14da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:38.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.250+0000 7f6c38ff9700 1 --2- 172.21.15.23:0/3592962018 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6c3c14d620 0x7f6c3c14da00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f6c2800a000 tx=0x7f6c28017000).stop 2020-11-20T11:36:38.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.250+0000 7f6c38ff9700 1 -- 172.21.15.23:0/3592962018 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6c3c14d620 msgr2=0x7f6c3c14da00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:36:38.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.250+0000 7f6c38ff9700 1 --2- 172.21.15.23:0/3592962018 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6c3c14d620 0x7f6c3c14da00 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:38.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.250+0000 7f6c42ab3700 1 -- 172.21.15.23:0/3592962018 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6c14056d60 msgr2=0x7f6c14059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:38.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.250+0000 7f6c42ab3700 1 --2- 172.21.15.23:0/3592962018 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6c14056d60 0x7f6c14059220 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f6c34009310 tx=0x7f6c34016000).stop 2020-11-20T11:36:38.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.250+0000 7f6c42ab3700 1 -- 172.21.15.23:0/3592962018 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c3c05bc80 msgr2=0x7f6c3c0a2bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:38.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.250+0000 7f6c42ab3700 1 --2- 172.21.15.23:0/3592962018 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c3c05bc80 0x7f6c3c0a2bc0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f6c3000c730 tx=0x7f6c30007c70).stop 2020-11-20T11:36:38.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.250+0000 7f6c41050700 1 -- 172.21.15.23:0/3592962018 reap_dead start 2020-11-20T11:36:38.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.250+0000 7f6c41050700 1 -- 172.21.15.23:0/3592962018 reap_dead start 2020-11-20T11:36:38.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.250+0000 7f6c42ab3700 1 -- 172.21.15.23:0/3592962018 shutdown_connections 2020-11-20T11:36:38.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.250+0000 7f6c42ab3700 1 -- 172.21.15.23:0/3592962018 >> 172.21.15.23:0/3592962018 conn(0x7f6c3c0ad7a0 msgr2=0x7f6c3c0a7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:38.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.250+0000 7f6c42ab3700 1 -- 172.21.15.23:0/3592962018 shutdown_connections 2020-11-20T11:36:38.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.251+0000 7f6c42ab3700 1 -- 172.21.15.23:0/3592962018 wait complete. 2020-11-20T11:36:38.262 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:36:38.262 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:36:38.262 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:36:38.263 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:36:38.263 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 38, 2020-11-20T11:36:38.263 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:36:38.263 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:36:38.264 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:36:38.264 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:36:38.264 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:38.264 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 14, 2020-11-20T11:36:38.264 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:36:38.265 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:36:38.265 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:36:38.265 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:36:38.265 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:36:38.265 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:36:38.266 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:36:38.266 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:36:38.266 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:36:38.266 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:36:38.266 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:36:38.267 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:38.267 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:36:38.267 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:36:38.267 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:36:38.267 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:36:38.268 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:36:38.268 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:36:38.268 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:36:38.268 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:36:38.268 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:36:38.269 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:36:38.269 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:38.269 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:38.269 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:36:38.269 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:36:38.270 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:36:38.270 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:36:38.270 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:36:38.270 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:36:38.270 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:36:38.271 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:36:38.271 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:36:38.271 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:36:38.271 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:36:38.271 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:36:38.272 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:36:38.272 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:36:38.272 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:36:38.272 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:36:38.272 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:36:38.273 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:36:38.273 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:36:38.273 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:36:38.273 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:36:38.273 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:36:38.274 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:36:38.274 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:38.274 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:36:38.274 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:38.274 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:36:38.275 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:38.275 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:36:38.275 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:36:38.275 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:36:38.275 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:36:38.276 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:38.276 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:36:38.276 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:36:38.276 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:38.276 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:38.277 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:38.277 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:36:38.277 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:36:38.277 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:38.277 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:38.278 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:38.278 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:38.278 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:36:38.278 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:36:38.279 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:36:38.279 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:36:38.279 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:36:38.279 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:38.279 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:38.279 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:36:38.280 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:36:38.280 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:36:38.280 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:36:38.280 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:38.280 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:36:38.281 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:36:38.281 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:38.281 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:38.281 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:38.281 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:36:38.282 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:36:38.282 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:38.282 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:38.282 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:38.282 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:38.283 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:36:38.283 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:36:38.283 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:36:38.283 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:36:38.283 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:36:38.283 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:38.284 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:38.284 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:36:38.284 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:36:38.284 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:36:38.284 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:36:38.285 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:38.285 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:36:38.285 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:36:38.285 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:38.285 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:38.286 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:38.286 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:36:38.286 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:36:38.286 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:38.286 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:38.286 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:38.287 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:38.287 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:36:38.287 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:36:38.287 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:36:38.287 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:36:38.288 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:36:38.288 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:38.288 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:38.288 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:38.288 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:36:38.289 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:36:38.289 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:38.289 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:36:38.289 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:36:38.289 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:36:38.290 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:38.290 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:38.290 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:36:38.290 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:38.290 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:36:38.291 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:36:38.291 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:36:38.291 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:38.291 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:38.291 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:38.291 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:36:38.292 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:36:38.293 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2020-11-20T11:36:38.293 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2020-11-20T11:36:38.293 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2020-11-20T11:36:38.293 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2020-11-20T11:36:38.294 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2020-11-20T11:36:38.294 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T11:36:38.306 INFO:tasks.ceph.mon.a:Stopped 2020-11-20T11:36:38.307 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2020-11-20T11:36:38.307 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:36:38.470 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.466+0000 7efe40c09700 1 Processor -- start 2020-11-20T11:36:38.470 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.466+0000 7efe40c09700 1 -- start start 2020-11-20T11:36:38.473 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.467+0000 7efe40c09700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7efe3c0aaab0 0x7efe3c13ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:38.473 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.467+0000 7efe40c09700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe3c13aeb0 0x7efe3c0a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:38.473 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.467+0000 7efe40c09700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe3c1388b0 con 0x7efe3c0aae90 2020-11-20T11:36:38.473 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.467+0000 7efe40c09700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe3c13e600 con 0x7efe3c0aaab0 2020-11-20T11:36:38.473 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.467+0000 7efe40c09700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe3c13e8a0 con 0x7efe3c13aeb0 2020-11-20T11:36:38.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.467+0000 7efe3a59c700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7efe3c0aaab0 msgr2=0x7efe3c13ed40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:36:38.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.467+0000 7efe39d9b700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7efe3c0aae90 legacy=0x7efe3c1381a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2020-11-20T11:36:38.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.467+0000 7efe3a59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7efe3c0aaab0 0x7efe3c13ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:36:38.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.467+0000 7efe3ad9d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe3c13aeb0 0x7efe3c0a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:38.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.467+0000 7efe3ad9d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe3c13aeb0 0x7efe3c0a8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:53156/0 (socket says 172.21.15.23:53156) 2020-11-20T11:36:38.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.467+0000 7efe3ad9d700 1 -- 172.21.15.23:0/1135750399 learned_addr learned my addr 172.21.15.23:0/1135750399 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:36:38.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.468+0000 7efe3ad9d700 1 -- 172.21.15.23:0/1135750399 >> v2:172.21.15.23:3300/0 conn(0x7efe3c0aaab0 msgr2=0x7efe3c13ed40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:36:38.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.468+0000 7efe3ad9d700 1 --2- 172.21.15.23:0/1135750399 >> v2:172.21.15.23:3300/0 conn(0x7efe3c0aaab0 0x7efe3c13ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:36:38.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.468+0000 7efe3ad9d700 1 -- 172.21.15.23:0/1135750399 >> v1:172.21.15.23:6789/0 conn(0x7efe3c0aae90 legacy=0x7efe3c1381a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:36:38.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.468+0000 7efe3ad9d700 1 -- 172.21.15.23:0/1135750399 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe3c0ab6d0 con 0x7efe3c13aeb0 2020-11-20T11:36:38.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.468+0000 7efe3ad9d700 1 --2- 172.21.15.23:0/1135750399 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe3c13aeb0 0x7efe3c0a8c60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7efe34009d90 tx=0x7efe340092d0).ready entity=mon.2 client_cookie=3fce322d75488e66 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:38.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.468+0000 7efe3959a700 1 -- 172.21.15.23:0/1135750399 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3400bbb0 con 0x7efe3c13aeb0 2020-11-20T11:36:38.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.468+0000 7efe3959a700 1 -- 172.21.15.23:0/1135750399 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe3400bd10 con 0x7efe3c13aeb0 2020-11-20T11:36:38.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.468+0000 7efe3959a700 1 -- 172.21.15.23:0/1135750399 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe34011610 con 0x7efe3c13aeb0 2020-11-20T11:36:38.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.469+0000 7efe40c09700 1 -- 172.21.15.23:0/1135750399 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe3c13aeb0 msgr2=0x7efe3c0a8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:38.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.469+0000 7efe40c09700 1 --2- 172.21.15.23:0/1135750399 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe3c13aeb0 0x7efe3c0a8c60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7efe34009d90 tx=0x7efe340092d0).stop 2020-11-20T11:36:38.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.469+0000 7efe40c09700 1 -- 172.21.15.23:0/1135750399 shutdown_connections 2020-11-20T11:36:38.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.469+0000 7efe40c09700 1 --2- 172.21.15.23:0/1135750399 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe3c13aeb0 0x7efe3c0a8c60 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:38.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.469+0000 7efe40c09700 1 --2- 172.21.15.23:0/1135750399 >> v2:172.21.15.23:3300/0 conn(0x7efe3c0aaab0 0x7efe3c13ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:36:38.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.469+0000 7efe40c09700 1 -- 172.21.15.23:0/1135750399 >> 172.21.15.23:0/1135750399 conn(0x7efe3c0678e0 msgr2=0x7efe3c067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:38.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.469+0000 7efe40c09700 1 -- 172.21.15.23:0/1135750399 shutdown_connections 2020-11-20T11:36:38.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.469+0000 7efe40c09700 1 -- 172.21.15.23:0/1135750399 wait complete. 2020-11-20T11:36:38.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.470+0000 7efe40c09700 1 Processor -- start 2020-11-20T11:36:38.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.470+0000 7efe40c09700 1 -- start start 2020-11-20T11:36:38.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.470+0000 7efe40c09700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe3c0aaab0 0x7efe3c0b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:38.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.470+0000 7efe40c09700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe3c0aae90 0x7efe3c0a53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:38.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.470+0000 7efe40c09700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe3c0a7ff0 0x7efe3c0a5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:38.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.470+0000 7efe40c09700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3c0a6650 con 0x7efe3c0aae90 2020-11-20T11:36:38.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.470+0000 7efe40c09700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3c0a68f0 con 0x7efe3c0aaab0 2020-11-20T11:36:38.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.470+0000 7efe40c09700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe3c0a6b90 con 0x7efe3c0a7ff0 2020-11-20T11:36:38.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.471+0000 7efe39d9b700 1 -- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe3c0aae90 msgr2=0x7efe3c0a53a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:36:38.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.471+0000 7efe39d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe3c0aae90 0x7efe3c0a53a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:36:38.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.471+0000 7efe3ad9d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe3c0a7ff0 0x7efe3c0a5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:38.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.471+0000 7efe3ad9d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe3c0a7ff0 0x7efe3c0a5890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:53162/0 (socket says 172.21.15.23:53162) 2020-11-20T11:36:38.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.471+0000 7efe3ad9d700 1 -- 172.21.15.23:0/1938569088 learned_addr learned my addr 172.21.15.23:0/1938569088 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:36:38.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.471+0000 7efe3a59c700 1 --2- 172.21.15.23:0/1938569088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe3c0aaab0 0x7efe3c0b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:38.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.471+0000 7efe3ad9d700 1 -- 172.21.15.23:0/1938569088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe3c0aaab0 msgr2=0x7efe3c0b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:38.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.471+0000 7efe3ad9d700 1 --2- 172.21.15.23:0/1938569088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe3c0aaab0 0x7efe3c0b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:38.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.471+0000 7efe3ad9d700 1 -- 172.21.15.23:0/1938569088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe3c0aae90 msgr2=0x7efe3c0a53a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:36:38.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.471+0000 7efe3ad9d700 1 --2- 172.21.15.23:0/1938569088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe3c0aae90 0x7efe3c0a53a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:36:38.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.471+0000 7efe3ad9d700 1 -- 172.21.15.23:0/1938569088 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe3c13d960 con 0x7efe3c0a7ff0 2020-11-20T11:36:38.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.471+0000 7efe3ad9d700 1 --2- 172.21.15.23:0/1938569088 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe3c0a7ff0 0x7efe3c0a5890 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7efe34007fd0 tx=0x7efe34016000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:38.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.471+0000 7efe2a7fc700 1 -- 172.21.15.23:0/1938569088 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3400be80 con 0x7efe3c0a7ff0 2020-11-20T11:36:38.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.472+0000 7efe2a7fc700 1 -- 172.21.15.23:0/1938569088 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe34010b10 con 0x7efe3c0a7ff0 2020-11-20T11:36:38.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.472+0000 7efe2a7fc700 1 -- 172.21.15.23:0/1938569088 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3401c470 con 0x7efe3c0a7ff0 2020-11-20T11:36:38.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.472+0000 7efe40c09700 1 -- 172.21.15.23:0/1938569088 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe3c0a60d0 con 0x7efe3c0a7ff0 2020-11-20T11:36:38.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.472+0000 7efe40c09700 1 -- 172.21.15.23:0/1938569088 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe3c136640 con 0x7efe3c0a7ff0 2020-11-20T11:36:38.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.473+0000 7efe40c09700 1 -- 172.21.15.23:0/1938569088 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe3c05c7d0 con 0x7efe3c0a7ff0 2020-11-20T11:36:38.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.473+0000 7efe2a7fc700 1 -- 172.21.15.23:0/1938569088 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7efe34017070 con 0x7efe3c0a7ff0 2020-11-20T11:36:38.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.473+0000 7efe2a7fc700 1 --2- 172.21.15.23:0/1938569088 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efe10056d10 0x7efe100591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:38.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.473+0000 7efe2a7fc700 1 -- 172.21.15.23:0/1938569088 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7efe34071f90 con 0x7efe3c0a7ff0 2020-11-20T11:36:38.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.474+0000 7efe3a59c700 1 --2- 172.21.15.23:0/1938569088 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efe10056d10 0x7efe100591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:38.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.474+0000 7efe3a59c700 1 --2- 172.21.15.23:0/1938569088 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efe10056d10 0x7efe100591d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7efe2c006fd0 tx=0x7efe2c006e10).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:38.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.478+0000 7efe2a7fc700 1 -- 172.21.15.23:0/1938569088 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7efe3406efd0 con 0x7efe3c0a7ff0 2020-11-20T11:36:38.726 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.723+0000 7efe40c09700 1 -- 172.21.15.23:0/1938569088 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7efe3c044710 con 0x7efe3c0a7ff0 2020-11-20T11:36:38.727 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.723+0000 7efe2a7fc700 1 -- 172.21.15.23:0/1938569088 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7efe34015070 con 0x7efe3c0a7ff0 2020-11-20T11:36:38.727 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:36:38.730 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.725+0000 7efe40c09700 1 -- 172.21.15.23:0/1938569088 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efe10056d10 msgr2=0x7efe100591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:38.730 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.725+0000 7efe40c09700 1 --2- 172.21.15.23:0/1938569088 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efe10056d10 0x7efe100591d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7efe2c006fd0 tx=0x7efe2c006e10).stop 2020-11-20T11:36:38.731 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.725+0000 7efe40c09700 1 -- 172.21.15.23:0/1938569088 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe3c0a7ff0 msgr2=0x7efe3c0a5890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:38.731 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.725+0000 7efe40c09700 1 --2- 172.21.15.23:0/1938569088 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe3c0a7ff0 0x7efe3c0a5890 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7efe34007fd0 tx=0x7efe34016000).stop 2020-11-20T11:36:38.731 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.725+0000 7efe40c09700 1 -- 172.21.15.23:0/1938569088 shutdown_connections 2020-11-20T11:36:38.731 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.725+0000 7efe40c09700 1 --2- 172.21.15.23:0/1938569088 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efe10056d10 0x7efe100591d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:38.731 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.725+0000 7efe40c09700 1 --2- 172.21.15.23:0/1938569088 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe3c0a7ff0 0x7efe3c0a5890 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:38.732 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.725+0000 7efe40c09700 1 --2- 172.21.15.23:0/1938569088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe3c0aaab0 0x7efe3c0b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:38.732 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.725+0000 7efe40c09700 1 --2- 172.21.15.23:0/1938569088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe3c0aae90 0x7efe3c0a53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:36:38.732 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.725+0000 7efe40c09700 1 -- 172.21.15.23:0/1938569088 >> 172.21.15.23:0/1938569088 conn(0x7efe3c0678e0 msgr2=0x7efe3c139aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:38.732 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.725+0000 7efe40c09700 1 -- 172.21.15.23:0/1938569088 shutdown_connections 2020-11-20T11:36:38.733 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:38.725+0000 7efe40c09700 1 -- 172.21.15.23:0/1938569088 wait complete. 2020-11-20T11:36:41.743 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:36:41.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.864+0000 7f840759e700 1 Processor -- start 2020-11-20T11:36:41.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.864+0000 7f840759e700 1 -- start start 2020-11-20T11:36:41.870 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.865+0000 7f840759e700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f84080a9040 0x7f84080aaf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:41.871 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.865+0000 7f840759e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84080ace70 0x7f840813de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:41.871 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.865+0000 7f840759e700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84080ad860 con 0x7f84080ad2f0 2020-11-20T11:36:41.871 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.865+0000 7f840759e700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84080aa8a0 con 0x7f84080a9040 2020-11-20T11:36:41.871 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.865+0000 7f840759e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84080aaa90 con 0x7f84080ace70 2020-11-20T11:36:41.871 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.865+0000 7f8406d9d700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f84080ad2f0 legacy=0x7f840813e2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2020-11-20T11:36:41.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.865+0000 7f840659c700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f84080a9040 msgr2=0x7f84080aaf30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:36:41.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.865+0000 7f840659c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f84080a9040 0x7f84080aaf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:36:41.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.866+0000 7f83fffff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84080ace70 0x7f840813de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:41.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.866+0000 7f83fffff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84080ace70 0x7f840813de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:53168/0 (socket says 172.21.15.23:53168) 2020-11-20T11:36:41.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.866+0000 7f83fffff700 1 -- 172.21.15.23:0/453845168 learned_addr learned my addr 172.21.15.23:0/453845168 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:36:41.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.866+0000 7f83fffff700 1 -- 172.21.15.23:0/453845168 >> v2:172.21.15.23:3300/0 conn(0x7f84080a9040 msgr2=0x7f84080aaf30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:36:41.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.866+0000 7f83fffff700 1 --2- 172.21.15.23:0/453845168 >> v2:172.21.15.23:3300/0 conn(0x7f84080a9040 0x7f84080aaf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:36:41.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.866+0000 7f83fffff700 1 -- 172.21.15.23:0/453845168 >> v1:172.21.15.23:6789/0 conn(0x7f84080ad2f0 legacy=0x7f840813e2f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:36:41.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.866+0000 7f83fffff700 1 -- 172.21.15.23:0/453845168 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84080a9420 con 0x7f84080ace70 2020-11-20T11:36:41.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.866+0000 7f83fffff700 1 --2- 172.21.15.23:0/453845168 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84080ace70 0x7f840813de00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f83f0009a10 tx=0x7f83f0006ef0).ready entity=mon.2 client_cookie=8de6111c3f44a330 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:41.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.866+0000 7f8405d9b700 1 -- 172.21.15.23:0/453845168 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f000b820 con 0x7f84080ace70 2020-11-20T11:36:41.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.866+0000 7f8405d9b700 1 -- 172.21.15.23:0/453845168 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83f0000e90 con 0x7f84080ace70 2020-11-20T11:36:41.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.867+0000 7f8405d9b700 1 -- 172.21.15.23:0/453845168 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f000bb90 con 0x7f84080ace70 2020-11-20T11:36:41.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.867+0000 7f840759e700 1 -- 172.21.15.23:0/453845168 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84080ace70 msgr2=0x7f840813de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:41.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.867+0000 7f840759e700 1 --2- 172.21.15.23:0/453845168 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84080ace70 0x7f840813de00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f83f0009a10 tx=0x7f83f0006ef0).stop 2020-11-20T11:36:41.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.867+0000 7f840759e700 1 -- 172.21.15.23:0/453845168 shutdown_connections 2020-11-20T11:36:41.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.867+0000 7f840759e700 1 --2- 172.21.15.23:0/453845168 >> v2:172.21.15.23:3300/0 conn(0x7f84080a9040 0x7f84080aaf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:36:41.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.867+0000 7f840759e700 1 --2- 172.21.15.23:0/453845168 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84080ace70 0x7f840813de00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:41.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.867+0000 7f840759e700 1 -- 172.21.15.23:0/453845168 >> 172.21.15.23:0/453845168 conn(0x7f84080683b0 msgr2=0x7f84080b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:41.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.867+0000 7f840759e700 1 -- 172.21.15.23:0/453845168 shutdown_connections 2020-11-20T11:36:41.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.867+0000 7f840759e700 1 -- 172.21.15.23:0/453845168 wait complete. 2020-11-20T11:36:41.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.868+0000 7f840759e700 1 Processor -- start 2020-11-20T11:36:41.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.869+0000 7f840759e700 1 -- start start 2020-11-20T11:36:41.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.869+0000 7f840759e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84080a9040 0x7f84080a0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:41.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.869+0000 7f840759e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84080ace70 0x7f840814a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:41.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.869+0000 7f840759e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84080ad2f0 0x7f840813d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:41.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.869+0000 7f840759e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84080a7370 con 0x7f84080a9040 2020-11-20T11:36:41.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.869+0000 7f840759e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f840814ada0 con 0x7f84080ace70 2020-11-20T11:36:41.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.869+0000 7f840759e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f840813cd70 con 0x7f84080ad2f0 2020-11-20T11:36:41.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.869+0000 7f840659c700 1 -- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84080a9040 msgr2=0x7f84080a0740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:36:41.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.869+0000 7f840659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84080a9040 0x7f84080a0740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:36:41.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.870+0000 7f8406d9d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84080ad2f0 0x7f840813d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:41.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.870+0000 7f8406d9d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84080ad2f0 0x7f840813d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:53176/0 (socket says 172.21.15.23:53176) 2020-11-20T11:36:41.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.870+0000 7f8406d9d700 1 -- 172.21.15.23:0/3420621661 learned_addr learned my addr 172.21.15.23:0/3420621661 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:36:41.882 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.870+0000 7f83fffff700 1 --2- 172.21.15.23:0/3420621661 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84080ace70 0x7f840814a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:41.882 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.870+0000 7f8406d9d700 1 -- 172.21.15.23:0/3420621661 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84080ace70 msgr2=0x7f840814a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:41.882 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.870+0000 7f8406d9d700 1 --2- 172.21.15.23:0/3420621661 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84080ace70 0x7f840814a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:41.882 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.870+0000 7f8406d9d700 1 -- 172.21.15.23:0/3420621661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84080a9040 msgr2=0x7f84080a0740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:36:41.882 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.870+0000 7f8406d9d700 1 --2- 172.21.15.23:0/3420621661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84080a9040 0x7f84080a0740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:36:41.883 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.870+0000 7f8406d9d700 1 -- 172.21.15.23:0/3420621661 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f840813d010 con 0x7f84080ad2f0 2020-11-20T11:36:41.883 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.870+0000 7f8406d9d700 1 --2- 172.21.15.23:0/3420621661 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84080ad2f0 0x7f840813d4b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f840000bae0 tx=0x7f84000091a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:41.883 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.870+0000 7f83fe7fc700 1 -- 172.21.15.23:0/3420621661 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8400003ca0 con 0x7f84080ad2f0 2020-11-20T11:36:41.883 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.870+0000 7f83fe7fc700 1 -- 172.21.15.23:0/3420621661 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8400003e00 con 0x7f84080ad2f0 2020-11-20T11:36:41.883 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.870+0000 7f840759e700 1 -- 172.21.15.23:0/3420621661 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f840814a910 con 0x7f84080ad2f0 2020-11-20T11:36:41.884 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.870+0000 7f83fe7fc700 1 -- 172.21.15.23:0/3420621661 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8400006170 con 0x7f84080ad2f0 2020-11-20T11:36:41.884 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.870+0000 7f840759e700 1 -- 172.21.15.23:0/3420621661 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f840813db60 con 0x7f84080ad2f0 2020-11-20T11:36:41.884 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.871+0000 7f840759e700 1 -- 172.21.15.23:0/3420621661 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8408057dc0 con 0x7f84080ad2f0 2020-11-20T11:36:41.884 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.872+0000 7f83fe7fc700 1 -- 172.21.15.23:0/3420621661 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f84000064e0 con 0x7f84080ad2f0 2020-11-20T11:36:41.885 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.872+0000 7f83fe7fc700 1 --2- 172.21.15.23:0/3420621661 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f83e4056d10 0x7f83e40591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:41.885 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.872+0000 7f83fe7fc700 1 -- 172.21.15.23:0/3420621661 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f8400011020 con 0x7f84080ad2f0 2020-11-20T11:36:41.885 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.872+0000 7f840659c700 1 --2- 172.21.15.23:0/3420621661 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f83e4056d10 0x7f83e40591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:41.885 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.873+0000 7f840659c700 1 --2- 172.21.15.23:0/3420621661 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f83e4056d10 0x7f83e40591d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f83f8003610 tx=0x7f83f8006ac0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:41.886 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:41.876+0000 7f83fe7fc700 1 -- 172.21.15.23:0/3420621661 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8400044ce0 con 0x7f84080ad2f0 2020-11-20T11:36:42.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:42.097+0000 7f840759e700 1 -- 172.21.15.23:0/3420621661 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f840813d1a0 con 0x7f84080ad2f0 2020-11-20T11:36:42.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:42.098+0000 7f83fe7fc700 1 -- 172.21.15.23:0/3420621661 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7f8400006770 con 0x7f84080ad2f0 2020-11-20T11:36:42.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:42.100+0000 7f840759e700 1 -- 172.21.15.23:0/3420621661 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f83e4056d10 msgr2=0x7f83e40591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:42.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:42.100+0000 7f840759e700 1 --2- 172.21.15.23:0/3420621661 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f83e4056d10 0x7f83e40591d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f83f8003610 tx=0x7f83f8006ac0).stop 2020-11-20T11:36:42.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:42.100+0000 7f840759e700 1 -- 172.21.15.23:0/3420621661 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84080ad2f0 msgr2=0x7f840813d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:42.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:42.100+0000 7f840759e700 1 --2- 172.21.15.23:0/3420621661 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84080ad2f0 0x7f840813d4b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f840000bae0 tx=0x7f84000091a0).stop 2020-11-20T11:36:42.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:42.100+0000 7f840759e700 1 -- 172.21.15.23:0/3420621661 shutdown_connections 2020-11-20T11:36:42.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:42.100+0000 7f840759e700 1 --2- 172.21.15.23:0/3420621661 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f83e4056d10 0x7f83e40591d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:42.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:42.100+0000 7f840759e700 1 --2- 172.21.15.23:0/3420621661 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84080ad2f0 0x7f840813d4b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:42.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:42.100+0000 7f840759e700 1 --2- 172.21.15.23:0/3420621661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84080a9040 0x7f84080a0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:36:42.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:42.100+0000 7f840759e700 1 --2- 172.21.15.23:0/3420621661 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84080ace70 0x7f840814a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:42.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:42.100+0000 7f840759e700 1 -- 172.21.15.23:0/3420621661 >> 172.21.15.23:0/3420621661 conn(0x7f84080683b0 msgr2=0x7f84080aba40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:42.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:42.100+0000 7f840759e700 1 -- 172.21.15.23:0/3420621661 shutdown_connections 2020-11-20T11:36:42.137 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:42.100+0000 7f840759e700 1 -- 172.21.15.23:0/3420621661 wait complete. 2020-11-20T11:36:42.137 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":18,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:36:45.138 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:36:45.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.642+0000 7f3d973cd700 1 Processor -- start 2020-11-20T11:36:45.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.643+0000 7f3d973cd700 1 -- start start 2020-11-20T11:36:45.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.643+0000 7f3d973cd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3d900aaab0 0x7f3d9013ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:45.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.643+0000 7f3d973cd700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3d900aae90 0x7f3d901381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:45.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.643+0000 7f3d973cd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d9013aeb0 0x7f3d900a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:45.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.643+0000 7f3d973cd700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d9013e600 con 0x7f3d900aaab0 2020-11-20T11:36:45.652 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.643+0000 7f3d973cd700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d9013e8a0 con 0x7f3d900aae90 2020-11-20T11:36:45.652 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.643+0000 7f3d973cd700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d900ab6d0 con 0x7f3d9013aeb0 2020-11-20T11:36:45.652 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.643+0000 7f3d95169700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f3d900aaab0 msgr2=0x7f3d9013ed40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:36:45.652 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.643+0000 7f3d95169700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3d900aaab0 0x7f3d9013ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:36:45.652 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.644+0000 7f3d94968700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3d900aae90 0x7f3d901381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:45.653 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.644+0000 7f3d9596a700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d9013aeb0 0x7f3d900a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:45.653 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.644+0000 7f3d9596a700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d9013aeb0 0x7f3d900a8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:53184/0 (socket says 172.21.15.23:53184) 2020-11-20T11:36:45.653 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.644+0000 7f3d9596a700 1 -- 172.21.15.23:0/683668650 learned_addr learned my addr 172.21.15.23:0/683668650 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:36:45.653 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.644+0000 7f3d94968700 1 -- 172.21.15.23:0/683668650 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d9013aeb0 msgr2=0x7f3d900a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:36:45.653 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.644+0000 7f3d94968700 1 --2- 172.21.15.23:0/683668650 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d9013aeb0 0x7f3d900a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:45.654 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.644+0000 7f3d94968700 1 -- 172.21.15.23:0/683668650 >> v2:172.21.15.23:3300/0 conn(0x7f3d900aaab0 msgr2=0x7f3d9013ed40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:36:45.654 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.644+0000 7f3d94968700 1 --2- 172.21.15.23:0/683668650 >> v2:172.21.15.23:3300/0 conn(0x7f3d900aaab0 0x7f3d9013ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:36:45.654 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.644+0000 7f3d94968700 1 -- 172.21.15.23:0/683668650 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d9013f230 con 0x7f3d900aae90 2020-11-20T11:36:45.654 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.644+0000 7f3d9596a700 1 --2- 172.21.15.23:0/683668650 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d9013aeb0 0x7f3d900a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:36:45.654 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.644+0000 7f3d94968700 1 --2- 172.21.15.23:0/683668650 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d900aae90 0x7f3d901381a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f3d900ab930 tx=0x7f3d84006e30).ready entity=mon.1 client_cookie=7e73329e80358744 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:45.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.644+0000 7f3d8ffff700 1 -- 172.21.15.23:0/683668650 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d840097b0 con 0x7f3d900aae90 2020-11-20T11:36:45.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.644+0000 7f3d8ffff700 1 -- 172.21.15.23:0/683668650 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d84000e90 con 0x7f3d900aae90 2020-11-20T11:36:45.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.644+0000 7f3d8ffff700 1 -- 172.21.15.23:0/683668650 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d84009b20 con 0x7f3d900aae90 2020-11-20T11:36:45.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.645+0000 7f3d973cd700 1 -- 172.21.15.23:0/683668650 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d900aae90 msgr2=0x7f3d901381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:45.656 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.645+0000 7f3d973cd700 1 --2- 172.21.15.23:0/683668650 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d900aae90 0x7f3d901381a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f3d900ab930 tx=0x7f3d84006e30).stop 2020-11-20T11:36:45.656 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.645+0000 7f3d973cd700 1 -- 172.21.15.23:0/683668650 shutdown_connections 2020-11-20T11:36:45.656 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.645+0000 7f3d973cd700 1 --2- 172.21.15.23:0/683668650 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d9013aeb0 0x7f3d900a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:45.657 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.645+0000 7f3d973cd700 1 --2- 172.21.15.23:0/683668650 >> v2:172.21.15.23:3300/0 conn(0x7f3d900aaab0 0x7f3d9013ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:36:45.657 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.645+0000 7f3d973cd700 1 --2- 172.21.15.23:0/683668650 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d900aae90 0x7f3d901381a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:45.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.645+0000 7f3d973cd700 1 -- 172.21.15.23:0/683668650 >> 172.21.15.23:0/683668650 conn(0x7f3d900678e0 msgr2=0x7f3d90067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:45.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.645+0000 7f3d973cd700 1 -- 172.21.15.23:0/683668650 shutdown_connections 2020-11-20T11:36:45.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.645+0000 7f3d973cd700 1 -- 172.21.15.23:0/683668650 wait complete. 2020-11-20T11:36:45.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.646+0000 7f3d973cd700 1 Processor -- start 2020-11-20T11:36:45.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.646+0000 7f3d973cd700 1 -- start start 2020-11-20T11:36:45.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.647+0000 7f3d973cd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d900aaab0 0x7f3d900b1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:45.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.647+0000 7f3d973cd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d900aae90 0x7f3d900a4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:45.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.647+0000 7f3d973cd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d900a8020 0x7f3d9013c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:45.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.647+0000 7f3d973cd700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9014e200 con 0x7f3d900aae90 2020-11-20T11:36:45.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.647+0000 7f3d973cd700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9014e4a0 con 0x7f3d900a8020 2020-11-20T11:36:45.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.647+0000 7f3d973cd700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9014e740 con 0x7f3d900aaab0 2020-11-20T11:36:45.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.647+0000 7f3d94968700 1 -- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d900aae90 msgr2=0x7f3d900a4560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:36:45.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.647+0000 7f3d94968700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d900aae90 0x7f3d900a4560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:36:45.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.647+0000 7f3d95169700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d900aaab0 0x7f3d900b1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:45.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.647+0000 7f3d9596a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d900a8020 0x7f3d9013c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:45.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.647+0000 7f3d95169700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d900aaab0 0x7f3d900b1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:53186/0 (socket says 172.21.15.23:53186) 2020-11-20T11:36:45.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.647+0000 7f3d95169700 1 -- 172.21.15.23:0/1651852585 learned_addr learned my addr 172.21.15.23:0/1651852585 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:36:45.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.647+0000 7f3d9596a700 1 -- 172.21.15.23:0/1651852585 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d900aaab0 msgr2=0x7f3d900b1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:45.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.647+0000 7f3d9596a700 1 --2- 172.21.15.23:0/1651852585 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d900aaab0 0x7f3d900b1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:45.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.647+0000 7f3d9596a700 1 -- 172.21.15.23:0/1651852585 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d900aae90 msgr2=0x7f3d900a4560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:36:45.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.647+0000 7f3d9596a700 1 --2- 172.21.15.23:0/1651852585 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d900aae90 0x7f3d900a4560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:36:45.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.647+0000 7f3d9596a700 1 -- 172.21.15.23:0/1651852585 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d900b1d30 con 0x7f3d900a8020 2020-11-20T11:36:45.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.647+0000 7f3d95169700 1 --2- 172.21.15.23:0/1651852585 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d900aaab0 0x7f3d900b1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:36:45.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.648+0000 7f3d9596a700 1 --2- 172.21.15.23:0/1651852585 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d900a8020 0x7f3d9013c4b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f3d88009260 tx=0x7f3d88006ef0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:45.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.648+0000 7f3d8d7fa700 1 -- 172.21.15.23:0/1651852585 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8800c820 con 0x7f3d900a8020 2020-11-20T11:36:45.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.648+0000 7f3d8d7fa700 1 -- 172.21.15.23:0/1651852585 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d8800c980 con 0x7f3d900a8020 2020-11-20T11:36:45.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.648+0000 7f3d973cd700 1 -- 172.21.15.23:0/1651852585 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d9014de80 con 0x7f3d900a8020 2020-11-20T11:36:45.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.648+0000 7f3d8d7fa700 1 -- 172.21.15.23:0/1651852585 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8800cd00 con 0x7f3d900a8020 2020-11-20T11:36:45.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.648+0000 7f3d973cd700 1 -- 172.21.15.23:0/1651852585 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d9009dff0 con 0x7f3d900a8020 2020-11-20T11:36:45.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.649+0000 7f3d973cd700 1 -- 172.21.15.23:0/1651852585 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d9014e070 con 0x7f3d900a8020 2020-11-20T11:36:45.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.649+0000 7f3d8d7fa700 1 -- 172.21.15.23:0/1651852585 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3d88022070 con 0x7f3d900a8020 2020-11-20T11:36:45.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.650+0000 7f3d8d7fa700 1 --2- 172.21.15.23:0/1651852585 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3d68056d40 0x7f3d68059200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:45.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.650+0000 7f3d8d7fa700 1 -- 172.21.15.23:0/1651852585 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f3d88070580 con 0x7f3d900a8020 2020-11-20T11:36:45.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.650+0000 7f3d95169700 1 --2- 172.21.15.23:0/1651852585 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3d68056d40 0x7f3d68059200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:45.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.650+0000 7f3d95169700 1 --2- 172.21.15.23:0/1651852585 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3d68056d40 0x7f3d68059200 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f3d80006fd0 tx=0x7f3d80009290).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:45.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.654+0000 7f3d8d7fa700 1 -- 172.21.15.23:0/1651852585 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3d88045870 con 0x7f3d900a8020 2020-11-20T11:36:45.910 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.902+0000 7f3d973cd700 1 -- 172.21.15.23:0/1651852585 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3d900ab390 con 0x7f3d900a8020 2020-11-20T11:36:45.912 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.902+0000 7f3d8d7fa700 1 -- 172.21.15.23:0/1651852585 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7f3d88048e90 con 0x7f3d900a8020 2020-11-20T11:36:45.912 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.904+0000 7f3d973cd700 1 -- 172.21.15.23:0/1651852585 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3d68056d40 msgr2=0x7f3d68059200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:45.914 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.904+0000 7f3d973cd700 1 --2- 172.21.15.23:0/1651852585 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3d68056d40 0x7f3d68059200 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f3d80006fd0 tx=0x7f3d80009290).stop 2020-11-20T11:36:45.916 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.904+0000 7f3d973cd700 1 -- 172.21.15.23:0/1651852585 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d900a8020 msgr2=0x7f3d9013c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:45.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.904+0000 7f3d973cd700 1 --2- 172.21.15.23:0/1651852585 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d900a8020 0x7f3d9013c4b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f3d88009260 tx=0x7f3d88006ef0).stop 2020-11-20T11:36:45.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.904+0000 7f3d973cd700 1 -- 172.21.15.23:0/1651852585 shutdown_connections 2020-11-20T11:36:45.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.904+0000 7f3d973cd700 1 --2- 172.21.15.23:0/1651852585 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d900a8020 0x7f3d9013c4b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:45.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.904+0000 7f3d973cd700 1 --2- 172.21.15.23:0/1651852585 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3d68056d40 0x7f3d68059200 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:45.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.904+0000 7f3d973cd700 1 --2- 172.21.15.23:0/1651852585 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d900aaab0 0x7f3d900b1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:45.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.904+0000 7f3d973cd700 1 --2- 172.21.15.23:0/1651852585 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d900aae90 0x7f3d900a4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:36:45.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.904+0000 7f3d973cd700 1 -- 172.21.15.23:0/1651852585 >> 172.21.15.23:0/1651852585 conn(0x7f3d900678e0 msgr2=0x7f3d9013a280 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:45.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.904+0000 7f3d973cd700 1 -- 172.21.15.23:0/1651852585 shutdown_connections 2020-11-20T11:36:45.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:45.904+0000 7f3d973cd700 1 -- 172.21.15.23:0/1651852585 wait complete. 2020-11-20T11:36:45.922 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:36:48.924 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:36:49.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:49.052+0000 7f665dc6c700 1 Processor -- start 2020-11-20T11:36:49.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:49.052+0000 7f665dc6c700 1 -- start start 2020-11-20T11:36:49.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:49.053+0000 7f665dc6c700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66580aaab0 0x7f665813ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:49.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:49.053+0000 7f665dc6c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f66580aae90 0x7f66581381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:49.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:49.053+0000 7f665dc6c700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f665813e600 con 0x7f665813aeb0 2020-11-20T11:36:49.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:49.053+0000 7f665dc6c700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f665813e8a0 con 0x7f66580aae90 2020-11-20T11:36:49.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:49.053+0000 7f665dc6c700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66580ab6d0 con 0x7f66580aaab0 2020-11-20T11:36:49.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:49.053+0000 7f664ffff700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f66580aae90 msgr2=0x7f66581381a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:36:49.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:49.053+0000 7f664ffff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f66580aae90 0x7f66581381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:36:49.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:49.053+0000 7f66577fe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66580aaab0 0x7f665813ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:49.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:49.053+0000 7f6657fff700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f665813aeb0 0x7f66580a8c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:35182/0 (socket says 172.21.15.23:35182) 2020-11-20T11:36:49.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:49.053+0000 7f6657fff700 1 -- 172.21.15.23:0/1437881296 learned_addr learned my addr 172.21.15.23:0/1437881296 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:36:49.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:49.053+0000 7f66577fe700 1 -- 172.21.15.23:0/1437881296 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66580aaab0 msgr2=0x7f665813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:36:49.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:49.053+0000 7f66577fe700 1 -- 172.21.15.23:0/1437881296 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66580aaab0 msgr2=0x7f665813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:36:49.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:49.053+0000 7f66577fe700 1 --2- 172.21.15.23:0/1437881296 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66580aaab0 0x7f665813ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:36:49.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:49.053+0000 7f66577fe700 1 --2- 172.21.15.23:0/1437881296 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66580aaab0 0x7f665813ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:36:49.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:49.254+0000 7f664ffff700 1 -- 172.21.15.23:0/1437881296 >> v2:172.21.15.23:3300/0 conn(0x7f66580aae90 msgr2=0x7f66581381a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:36:49.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:49.254+0000 7f664ffff700 1 --2- 172.21.15.23:0/1437881296 >> v2:172.21.15.23:3300/0 conn(0x7f66580aae90 0x7f66581381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:36:49.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:49.254+0000 7f66577fe700 1 --2- 172.21.15.23:0/1437881296 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66580aaab0 0x7f665813ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:49.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:49.254+0000 7f66577fe700 1 -- 172.21.15.23:0/1437881296 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66580aaab0 msgr2=0x7f665813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:36:49.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:49.254+0000 7f66577fe700 1 -- 172.21.15.23:0/1437881296 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66580aaab0 msgr2=0x7f665813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:36:49.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:49.254+0000 7f66577fe700 1 --2- 172.21.15.23:0/1437881296 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66580aaab0 0x7f665813ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:36:49.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:49.254+0000 7f66577fe700 1 --2- 172.21.15.23:0/1437881296 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66580aaab0 0x7f665813ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:36:49.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:49.655+0000 7f664ffff700 1 -- 172.21.15.23:0/1437881296 >> v2:172.21.15.23:3300/0 conn(0x7f66580aae90 msgr2=0x7f66581381a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:36:49.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:49.655+0000 7f664ffff700 1 --2- 172.21.15.23:0/1437881296 >> v2:172.21.15.23:3300/0 conn(0x7f66580aae90 0x7f66581381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:36:49.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:49.655+0000 7f66577fe700 1 --2- 172.21.15.23:0/1437881296 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66580aaab0 0x7f665813ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:49.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:49.655+0000 7f66577fe700 1 -- 172.21.15.23:0/1437881296 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66580aaab0 msgr2=0x7f665813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:36:49.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:49.655+0000 7f66577fe700 1 -- 172.21.15.23:0/1437881296 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66580aaab0 msgr2=0x7f665813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:36:49.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:49.655+0000 7f66577fe700 1 --2- 172.21.15.23:0/1437881296 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66580aaab0 0x7f665813ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:36:49.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:49.655+0000 7f66577fe700 1 --2- 172.21.15.23:0/1437881296 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66580aaab0 0x7f665813ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:36:50.459 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:50.456+0000 7f664ffff700 1 -- 172.21.15.23:0/1437881296 >> v2:172.21.15.23:3300/0 conn(0x7f66580aae90 msgr2=0x7f66581381a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:36:50.459 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:50.456+0000 7f664ffff700 1 --2- 172.21.15.23:0/1437881296 >> v2:172.21.15.23:3300/0 conn(0x7f66580aae90 0x7f66581381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:36:50.460 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:50.456+0000 7f66577fe700 1 --2- 172.21.15.23:0/1437881296 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66580aaab0 0x7f665813ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:50.460 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:50.456+0000 7f66577fe700 1 -- 172.21.15.23:0/1437881296 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66580aaab0 msgr2=0x7f665813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:36:50.460 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:50.456+0000 7f66577fe700 1 -- 172.21.15.23:0/1437881296 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66580aaab0 msgr2=0x7f665813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:36:50.460 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:50.456+0000 7f66577fe700 1 --2- 172.21.15.23:0/1437881296 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66580aaab0 0x7f665813ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:36:50.460 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:50.456+0000 7f66577fe700 1 --2- 172.21.15.23:0/1437881296 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66580aaab0 0x7f665813ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:36:52.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:52.053+0000 7f6655ffb700 1 -- 172.21.15.23:0/1437881296 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66580aaab0 msgr2=0x7f665813ed40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:36:52.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:52.053+0000 7f6655ffb700 1 --2- 172.21.15.23:0/1437881296 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f66580aaab0 0x7f665813ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:52.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:52.053+0000 7f6655ffb700 1 -- 172.21.15.23:0/1437881296 >> v2:172.21.15.23:3300/0 conn(0x7f66580aae90 msgr2=0x7f66581381a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:36:52.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:52.053+0000 7f6655ffb700 1 --2- 172.21.15.23:0/1437881296 >> v2:172.21.15.23:3300/0 conn(0x7f66580aae90 0x7f66581381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:36:52.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:52.053+0000 7f6655ffb700 1 -- 172.21.15.23:0/1437881296 >> v1:172.21.15.155:6789/0 conn(0x7f665813aeb0 legacy=0x7f66580a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:52.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:52.053+0000 7f6655ffb700 1 --2- 172.21.15.23:0/1437881296 >> v2:172.21.15.155:3300/0 conn(0x7f663c0070a0 0x7f663c009560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:52.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:52.053+0000 7f6655ffb700 1 -- 172.21.15.23:0/1437881296 --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66580a9370 con 0x7f663c0033f0 2020-11-20T11:36:52.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:52.053+0000 7f6655ffb700 1 -- 172.21.15.23:0/1437881296 --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f665813e8a0 con 0x7f663c0070a0 2020-11-20T11:36:52.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:52.053+0000 7f66577fe700 1 -- 172.21.15.23:0/1437881296 >> v1:172.21.15.23:6789/0 conn(0x7f663c0033f0 legacy=0x7f663c005870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2020-11-20T11:36:52.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:52.053+0000 7f664ffff700 1 --2- 172.21.15.23:0/1437881296 >> v2:172.21.15.155:3300/0 conn(0x7f663c0070a0 0x7f663c009560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:52.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:52.053+0000 7f664ffff700 1 -- 172.21.15.23:0/1437881296 >> v2:172.21.15.155:3300/0 conn(0x7f663c0070a0 msgr2=0x7f663c009560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:36:52.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:52.053+0000 7f664ffff700 1 -- 172.21.15.23:0/1437881296 >> v2:172.21.15.155:3300/0 conn(0x7f663c0070a0 msgr2=0x7f663c009560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:36:52.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:52.053+0000 7f664ffff700 1 --2- 172.21.15.23:0/1437881296 >> v2:172.21.15.155:3300/0 conn(0x7f663c0070a0 0x7f663c009560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:36:52.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:52.053+0000 7f664ffff700 1 --2- 172.21.15.23:0/1437881296 >> v2:172.21.15.155:3300/0 conn(0x7f663c0070a0 0x7f663c009560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:36:52.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:52.254+0000 7f66577fe700 1 -- 172.21.15.23:0/1437881296 >> v1:172.21.15.23:6789/0 conn(0x7f663c0033f0 legacy=0x7f663c005870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2020-11-20T11:36:52.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:52.254+0000 7f664ffff700 1 --2- 172.21.15.23:0/1437881296 >> v2:172.21.15.155:3300/0 conn(0x7f663c0070a0 0x7f663c009560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:52.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:52.254+0000 7f664ffff700 1 -- 172.21.15.23:0/1437881296 >> v2:172.21.15.155:3300/0 conn(0x7f663c0070a0 msgr2=0x7f663c009560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:36:52.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:52.254+0000 7f664ffff700 1 -- 172.21.15.23:0/1437881296 >> v2:172.21.15.155:3300/0 conn(0x7f663c0070a0 msgr2=0x7f663c009560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:36:52.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:52.254+0000 7f664ffff700 1 --2- 172.21.15.23:0/1437881296 >> v2:172.21.15.155:3300/0 conn(0x7f663c0070a0 0x7f663c009560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:36:52.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:52.254+0000 7f664ffff700 1 --2- 172.21.15.23:0/1437881296 >> v2:172.21.15.155:3300/0 conn(0x7f663c0070a0 0x7f663c009560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:36:52.657 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:52.655+0000 7f66577fe700 1 -- 172.21.15.23:0/1437881296 >> v1:172.21.15.23:6789/0 conn(0x7f663c0033f0 legacy=0x7f663c005870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2020-11-20T11:36:52.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:52.655+0000 7f664ffff700 1 --2- 172.21.15.23:0/1437881296 >> v2:172.21.15.155:3300/0 conn(0x7f663c0070a0 0x7f663c009560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:52.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:52.655+0000 7f664ffff700 1 -- 172.21.15.23:0/1437881296 >> v2:172.21.15.155:3300/0 conn(0x7f663c0070a0 msgr2=0x7f663c009560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:36:52.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:52.655+0000 7f664ffff700 1 -- 172.21.15.23:0/1437881296 >> v2:172.21.15.155:3300/0 conn(0x7f663c0070a0 msgr2=0x7f663c009560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:36:52.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:52.655+0000 7f664ffff700 1 --2- 172.21.15.23:0/1437881296 >> v2:172.21.15.155:3300/0 conn(0x7f663c0070a0 0x7f663c009560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:36:52.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:52.655+0000 7f664ffff700 1 --2- 172.21.15.23:0/1437881296 >> v2:172.21.15.155:3300/0 conn(0x7f663c0070a0 0x7f663c009560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:36:53.459 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.456+0000 7f66577fe700 1 -- 172.21.15.23:0/1437881296 >> v1:172.21.15.23:6789/0 conn(0x7f663c0033f0 legacy=0x7f663c005870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2020-11-20T11:36:53.459 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.456+0000 7f664ffff700 1 --2- 172.21.15.23:0/1437881296 >> v2:172.21.15.155:3300/0 conn(0x7f663c0070a0 0x7f663c009560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:53.462 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.456+0000 7f664ffff700 1 -- 172.21.15.23:0/1437881296 >> v1:172.21.15.23:6789/0 conn(0x7f663c0033f0 legacy=0x7f663c005870 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:36:53.462 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.456+0000 7f664ffff700 1 -- 172.21.15.23:0/1437881296 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f663c00ac20 con 0x7f663c0070a0 2020-11-20T11:36:53.462 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.457+0000 7f664ffff700 1 --2- 172.21.15.23:0/1437881296 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f663c0070a0 0x7f663c009560 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f66580ab930 tx=0x7f6640003c90).ready entity=mon.1 client_cookie=25352054f0e78109 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:53.463 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.457+0000 7f6656ffd700 1 -- 172.21.15.23:0/1437881296 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6640005ac0 con 0x7f663c0070a0 2020-11-20T11:36:53.463 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.457+0000 7f6656ffd700 1 -- 172.21.15.23:0/1437881296 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f664000f070 con 0x7f663c0070a0 2020-11-20T11:36:53.463 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.457+0000 7f6656ffd700 1 -- 172.21.15.23:0/1437881296 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6640005e30 con 0x7f663c0070a0 2020-11-20T11:36:53.464 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.457+0000 7f665dc6c700 1 -- 172.21.15.23:0/1437881296 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f663c0070a0 msgr2=0x7f663c009560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:53.464 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.457+0000 7f665dc6c700 1 --2- 172.21.15.23:0/1437881296 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f663c0070a0 0x7f663c009560 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f66580ab930 tx=0x7f6640003c90).stop 2020-11-20T11:36:53.464 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.457+0000 7f6657fff700 1 -- 172.21.15.23:0/1437881296 reap_dead start 2020-11-20T11:36:53.466 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.457+0000 7f665dc6c700 1 -- 172.21.15.23:0/1437881296 shutdown_connections 2020-11-20T11:36:53.466 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.457+0000 7f665dc6c700 1 --2- 172.21.15.23:0/1437881296 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f663c0070a0 0x7f663c009560 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:53.467 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.457+0000 7f665dc6c700 1 -- 172.21.15.23:0/1437881296 >> 172.21.15.23:0/1437881296 conn(0x7f66580678e0 msgr2=0x7f6658067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:53.467 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.458+0000 7f665dc6c700 1 -- 172.21.15.23:0/1437881296 shutdown_connections 2020-11-20T11:36:53.467 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.458+0000 7f665dc6c700 1 -- 172.21.15.23:0/1437881296 wait complete. 2020-11-20T11:36:53.467 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.459+0000 7f665dc6c700 1 Processor -- start 2020-11-20T11:36:53.468 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.459+0000 7f665dc6c700 1 -- start start 2020-11-20T11:36:53.471 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.459+0000 7f665dc6c700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f663c0070a0 0x7f6658140850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:53.472 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.459+0000 7f665dc6c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f665813e020 0x7f6658131660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:53.472 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.459+0000 7f665dc6c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f665813db20 0x7f6658131b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:53.472 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.459+0000 7f665dc6c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6658136210 con 0x7f665813e020 2020-11-20T11:36:53.473 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.459+0000 7f665dc6c700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6658135a30 con 0x7f665813db20 2020-11-20T11:36:53.473 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.459+0000 7f665dc6c700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6658135cd0 con 0x7f663c0070a0 2020-11-20T11:36:53.473 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.459+0000 7f664ffff700 1 -- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f665813e020 msgr2=0x7f6658131660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:36:53.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.459+0000 7f664ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f665813e020 0x7f6658131660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:36:53.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.460+0000 7f66577fe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f663c0070a0 0x7f6658140850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:53.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.460+0000 7f6657fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f665813db20 0x7f6658131b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:53.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.460+0000 7f66577fe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f663c0070a0 0x7f6658140850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:53224/0 (socket says 172.21.15.23:53224) 2020-11-20T11:36:53.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.460+0000 7f66577fe700 1 -- 172.21.15.23:0/2713951888 learned_addr learned my addr 172.21.15.23:0/2713951888 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:36:53.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.460+0000 7f66577fe700 1 -- 172.21.15.23:0/2713951888 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f665813db20 msgr2=0x7f6658131b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:53.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.460+0000 7f66577fe700 1 --2- 172.21.15.23:0/2713951888 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f665813db20 0x7f6658131b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:53.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.460+0000 7f66577fe700 1 -- 172.21.15.23:0/2713951888 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f665813e020 msgr2=0x7f6658131660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:36:53.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.460+0000 7f66577fe700 1 --2- 172.21.15.23:0/2713951888 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f665813e020 0x7f6658131660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:36:53.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.460+0000 7f66577fe700 1 -- 172.21.15.23:0/2713951888 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6658135f70 con 0x7f663c0070a0 2020-11-20T11:36:53.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.460+0000 7f6657fff700 1 --2- 172.21.15.23:0/2713951888 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f665813db20 0x7f6658131b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:36:53.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.460+0000 7f66577fe700 1 --2- 172.21.15.23:0/2713951888 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f663c0070a0 0x7f6658140850 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f6648012cb0 tx=0x7f6648008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:53.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.460+0000 7f664f7fe700 1 -- 172.21.15.23:0/2713951888 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664800fa10 con 0x7f663c0070a0 2020-11-20T11:36:53.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.460+0000 7f664f7fe700 1 -- 172.21.15.23:0/2713951888 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f664800fb70 con 0x7f663c0070a0 2020-11-20T11:36:53.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.460+0000 7f665dc6c700 1 -- 172.21.15.23:0/2713951888 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f665814a650 con 0x7f663c0070a0 2020-11-20T11:36:53.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.460+0000 7f664f7fe700 1 -- 172.21.15.23:0/2713951888 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6648015450 con 0x7f663c0070a0 2020-11-20T11:36:53.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.460+0000 7f665dc6c700 1 -- 172.21.15.23:0/2713951888 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f665814ab60 con 0x7f663c0070a0 2020-11-20T11:36:53.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.461+0000 7f665dc6c700 1 -- 172.21.15.23:0/2713951888 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f665805c7d0 con 0x7f663c0070a0 2020-11-20T11:36:53.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.462+0000 7f664f7fe700 1 -- 172.21.15.23:0/2713951888 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6648025070 con 0x7f663c0070a0 2020-11-20T11:36:53.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.462+0000 7f664f7fe700 1 --2- 172.21.15.23:0/2713951888 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6634056d10 0x7f66340591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:53.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.462+0000 7f664f7fe700 1 -- 172.21.15.23:0/2713951888 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f6648000e40 con 0x7f663c0070a0 2020-11-20T11:36:53.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.462+0000 7f664ffff700 1 --2- 172.21.15.23:0/2713951888 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6634056d10 0x7f66340591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:53.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.463+0000 7f664ffff700 1 --2- 172.21.15.23:0/2713951888 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6634056d10 0x7f66340591d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f663c003360 tx=0x7f6640019000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:53.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.466+0000 7f664f7fe700 1 -- 172.21.15.23:0/2713951888 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6648045f20 con 0x7f663c0070a0 2020-11-20T11:36:53.717 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.714+0000 7f665dc6c700 1 -- 172.21.15.23:0/2713951888 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6658044710 con 0x7f663c0070a0 2020-11-20T11:36:53.717 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.714+0000 7f664f7fe700 1 -- 172.21.15.23:0/2713951888 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1471 (secure 0 0 0) 0x7f6648070e60 con 0x7f663c0070a0 2020-11-20T11:36:53.718 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":40,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:36:53.719 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.716+0000 7f665dc6c700 1 -- 172.21.15.23:0/2713951888 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6634056d10 msgr2=0x7f66340591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:53.720 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.716+0000 7f665dc6c700 1 --2- 172.21.15.23:0/2713951888 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6634056d10 0x7f66340591d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f663c003360 tx=0x7f6640019000).stop 2020-11-20T11:36:53.720 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.716+0000 7f665dc6c700 1 -- 172.21.15.23:0/2713951888 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f663c0070a0 msgr2=0x7f6658140850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:53.720 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.716+0000 7f665dc6c700 1 --2- 172.21.15.23:0/2713951888 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f663c0070a0 0x7f6658140850 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f6648012cb0 tx=0x7f6648008040).stop 2020-11-20T11:36:53.720 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.716+0000 7f665dc6c700 1 -- 172.21.15.23:0/2713951888 shutdown_connections 2020-11-20T11:36:53.721 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.716+0000 7f665dc6c700 1 --2- 172.21.15.23:0/2713951888 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f665813db20 0x7f6658131b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:53.721 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.716+0000 7f665dc6c700 1 --2- 172.21.15.23:0/2713951888 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6634056d10 0x7f66340591d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:53.721 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.716+0000 7f665dc6c700 1 --2- 172.21.15.23:0/2713951888 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f663c0070a0 0x7f6658140850 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:53.721 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.716+0000 7f665dc6c700 1 --2- 172.21.15.23:0/2713951888 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f665813e020 0x7f6658131660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:36:53.721 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.716+0000 7f665dc6c700 1 -- 172.21.15.23:0/2713951888 >> 172.21.15.23:0/2713951888 conn(0x7f66580678e0 msgr2=0x7f66580a86d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:53.722 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.717+0000 7f665dc6c700 1 -- 172.21.15.23:0/2713951888 shutdown_connections 2020-11-20T11:36:53.722 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.717+0000 7f665dc6c700 1 -- 172.21.15.23:0/2713951888 wait complete. 2020-11-20T11:36:53.731 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2020-11-20T11:36:53.732 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:36:53.863 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.857+0000 7fa9a15d7700 1 Processor -- start 2020-11-20T11:36:53.863 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.857+0000 7fa9a15d7700 1 -- start start 2020-11-20T11:36:53.863 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.858+0000 7fa9a15d7700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa99c0a27e0 0x7fa99c0a7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:53.863 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.858+0000 7fa9a15d7700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa99c0a84b0 0x7fa99c0a8940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:53.864 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.858+0000 7fa9a15d7700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa99c0a1ae0 con 0x7fa99c061380 2020-11-20T11:36:53.864 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.858+0000 7fa9a15d7700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa99c0a9270 con 0x7fa99c0a27e0 2020-11-20T11:36:53.864 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.858+0000 7fa9a15d7700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c0a9510 con 0x7fa99c0a84b0 2020-11-20T11:36:53.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.858+0000 7fa99bfff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7fa99c061380 legacy=0x7fa99c0a20d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2020-11-20T11:36:53.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.858+0000 7fa9a0dd6700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa99c0a84b0 0x7fa99c0a8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:53.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.858+0000 7fa99b7fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa99c0a27e0 0x7fa99c0a7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:53.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.858+0000 7fa99b7fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa99c0a27e0 0x7fa99c0a7fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:35890/0 (socket says 172.21.15.23:35890) 2020-11-20T11:36:53.866 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.858+0000 7fa9a0dd6700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa99c0a84b0 0x7fa99c0a8940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:53236/0 (socket says 172.21.15.23:53236) 2020-11-20T11:36:53.866 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.858+0000 7fa99b7fe700 1 -- 172.21.15.23:0/3240304614 learned_addr learned my addr 172.21.15.23:0/3240304614 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:36:53.866 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.859+0000 7fa99b7fe700 1 -- 172.21.15.23:0/3240304614 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa99c0a84b0 msgr2=0x7fa99c0a8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:36:53.866 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.859+0000 7fa99b7fe700 1 --2- 172.21.15.23:0/3240304614 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa99c0a84b0 0x7fa99c0a8940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:53.867 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.859+0000 7fa99b7fe700 1 -- 172.21.15.23:0/3240304614 >> v1:172.21.15.23:6789/0 conn(0x7fa99c061380 legacy=0x7fa99c0a20d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:36:53.867 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.859+0000 7fa99b7fe700 1 -- 172.21.15.23:0/3240304614 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99c061760 con 0x7fa99c0a27e0 2020-11-20T11:36:53.867 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.859+0000 7fa9a0dd6700 1 --2- 172.21.15.23:0/3240304614 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa99c0a84b0 0x7fa99c0a8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:36:53.867 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.859+0000 7fa99b7fe700 1 --2- 172.21.15.23:0/3240304614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa99c0a27e0 0x7fa99c0a7fc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fa98c009a40 tx=0x7fa98c006ef0).ready entity=mon.1 client_cookie=a947b31e1f64db80 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:53.867 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.859+0000 7fa99affd700 1 -- 172.21.15.23:0/3240304614 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c00b7e0 con 0x7fa99c0a27e0 2020-11-20T11:36:53.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.859+0000 7fa99affd700 1 -- 172.21.15.23:0/3240304614 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa98c000e90 con 0x7fa99c0a27e0 2020-11-20T11:36:53.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.859+0000 7fa99affd700 1 -- 172.21.15.23:0/3240304614 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c00bb50 con 0x7fa99c0a27e0 2020-11-20T11:36:53.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.859+0000 7fa9a15d7700 1 -- 172.21.15.23:0/3240304614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa99c0a27e0 msgr2=0x7fa99c0a7fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:53.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.859+0000 7fa9a15d7700 1 --2- 172.21.15.23:0/3240304614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa99c0a27e0 0x7fa99c0a7fc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fa98c009a40 tx=0x7fa98c006ef0).stop 2020-11-20T11:36:53.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.859+0000 7fa9a15d7700 1 -- 172.21.15.23:0/3240304614 shutdown_connections 2020-11-20T11:36:53.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.859+0000 7fa9a15d7700 1 --2- 172.21.15.23:0/3240304614 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa99c0a84b0 0x7fa99c0a8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:53.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.859+0000 7fa9a15d7700 1 --2- 172.21.15.23:0/3240304614 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa99c0a27e0 0x7fa99c0a7fc0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:53.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.859+0000 7fa9a15d7700 1 -- 172.21.15.23:0/3240304614 >> 172.21.15.23:0/3240304614 conn(0x7fa99c0ae0a0 msgr2=0x7fa99c0aa0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:53.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.859+0000 7fa9a15d7700 1 -- 172.21.15.23:0/3240304614 shutdown_connections 2020-11-20T11:36:53.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.860+0000 7fa9a15d7700 1 -- 172.21.15.23:0/3240304614 wait complete. 2020-11-20T11:36:53.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.861+0000 7fa9a15d7700 1 Processor -- start 2020-11-20T11:36:53.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.861+0000 7fa9a15d7700 1 -- start start 2020-11-20T11:36:53.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.861+0000 7fa9a15d7700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa99c061380 0x7fa99c14e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:53.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.861+0000 7fa9a15d7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa99c0a27e0 0x7fa99c09ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:53.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.861+0000 7fa9a15d7700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa99c0a84b0 0x7fa99c065870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:53.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.862+0000 7fa99b7fe700 1 -- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa99c0a27e0 msgr2=0x7fa99c09ec20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:36:53.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.862+0000 7fa9a15d7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c13b360 con 0x7fa99c0a27e0 2020-11-20T11:36:53.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.862+0000 7fa99b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa99c0a27e0 0x7fa99c09ec20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:36:53.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.862+0000 7fa9a15d7700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c14da40 con 0x7fa99c0a84b0 2020-11-20T11:36:53.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.862+0000 7fa9a15d7700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c0683f0 con 0x7fa99c061380 2020-11-20T11:36:53.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.862+0000 7fa99bfff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa99c061380 0x7fa99c14e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:53.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.862+0000 7fa9a0dd6700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa99c0a84b0 0x7fa99c065870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:53.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.862+0000 7fa99bfff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa99c061380 0x7fa99c14e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:53238/0 (socket says 172.21.15.23:53238) 2020-11-20T11:36:53.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.862+0000 7fa99bfff700 1 -- 172.21.15.23:0/3349782941 learned_addr learned my addr 172.21.15.23:0/3349782941 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:36:53.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.862+0000 7fa99bfff700 1 -- 172.21.15.23:0/3349782941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa99c0a84b0 msgr2=0x7fa99c065870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:53.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.862+0000 7fa99bfff700 1 --2- 172.21.15.23:0/3349782941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa99c0a84b0 0x7fa99c065870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:53.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.862+0000 7fa99bfff700 1 -- 172.21.15.23:0/3349782941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa99c0a27e0 msgr2=0x7fa99c09ec20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:36:53.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.862+0000 7fa99bfff700 1 --2- 172.21.15.23:0/3349782941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa99c0a27e0 0x7fa99c09ec20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:36:53.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.862+0000 7fa99bfff700 1 -- 172.21.15.23:0/3349782941 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99c068690 con 0x7fa99c061380 2020-11-20T11:36:53.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.862+0000 7fa9a0dd6700 1 --2- 172.21.15.23:0/3349782941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa99c0a84b0 0x7fa99c065870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:36:53.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.862+0000 7fa99bfff700 1 --2- 172.21.15.23:0/3349782941 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa99c061380 0x7fa99c14e330 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fa9900097c0 tx=0x7fa990006ea0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:53.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.863+0000 7fa97bfff700 1 -- 172.21.15.23:0/3349782941 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa990006170 con 0x7fa99c061380 2020-11-20T11:36:53.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.863+0000 7fa97bfff700 1 -- 172.21.15.23:0/3349782941 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9900062d0 con 0x7fa99c061380 2020-11-20T11:36:53.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.863+0000 7fa9a15d7700 1 -- 172.21.15.23:0/3349782941 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa99c119680 con 0x7fa99c061380 2020-11-20T11:36:53.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.863+0000 7fa97bfff700 1 -- 172.21.15.23:0/3349782941 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa990003920 con 0x7fa99c061380 2020-11-20T11:36:53.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.863+0000 7fa9a15d7700 1 -- 172.21.15.23:0/3349782941 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa99c119970 con 0x7fa99c061380 2020-11-20T11:36:53.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.863+0000 7fa9a15d7700 1 --2- 172.21.15.23:0/3349782941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa99c0664f0 0x7fa99c0a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:53.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.863+0000 7fa9a15d7700 1 -- 172.21.15.23:0/3349782941 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c0655d0 con 0x7fa99c0664f0 2020-11-20T11:36:53.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.864+0000 7fa99b7fe700 1 --2- 172.21.15.23:0/3349782941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa99c0664f0 0x7fa99c0a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:53.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.864+0000 7fa99b7fe700 1 -- 172.21.15.23:0/3349782941 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa99c069c70 con 0x7fa99c0664f0 2020-11-20T11:36:53.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.864+0000 7fa97bfff700 1 -- 172.21.15.23:0/3349782941 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa990017030 con 0x7fa99c061380 2020-11-20T11:36:53.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.864+0000 7fa99b7fe700 1 --2- 172.21.15.23:0/3349782941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa99c0664f0 0x7fa99c0a4a60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fa98c009a40 tx=0x7fa98c016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:53.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.864+0000 7fa97bfff700 1 --2- 172.21.15.23:0/3349782941 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa974056d10 0x7fa9740591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:53.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.864+0000 7fa97bfff700 1 -- 172.21.15.23:0/3349782941 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fa99006f370 con 0x7fa99c061380 2020-11-20T11:36:53.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.865+0000 7fa97bfff700 1 -- 172.21.15.23:0/3349782941 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c011890 con 0x7fa99c0664f0 2020-11-20T11:36:53.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.865+0000 7fa97bfff700 1 -- 172.21.15.23:0/3349782941 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fa98c0119f0 con 0x7fa99c0664f0 2020-11-20T11:36:53.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.865+0000 7fa97bfff700 1 -- 172.21.15.23:0/3349782941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa99c0664f0 msgr2=0x7fa99c0a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:53.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.865+0000 7fa97bfff700 1 --2- 172.21.15.23:0/3349782941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa99c0664f0 0x7fa99c0a4a60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fa98c009a40 tx=0x7fa98c016000).stop 2020-11-20T11:36:53.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.865+0000 7fa9a0dd6700 1 --2- 172.21.15.23:0/3349782941 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa974056d10 0x7fa9740591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:53.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.865+0000 7fa97bfff700 1 -- 172.21.15.23:0/3349782941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa99c0664f0 msgr2=0x7fa99c0a4a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:36:53.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.865+0000 7fa97bfff700 1 --2- 172.21.15.23:0/3349782941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa99c0664f0 0x7fa99c0a4a60 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0x7fa98c009a40 tx=0x7fa98c016000).stop 2020-11-20T11:36:53.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.865+0000 7fa9a0dd6700 1 --2- 172.21.15.23:0/3349782941 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa974056d10 0x7fa9740591d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fa99400f6e0 tx=0x7fa994009220).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:53.883 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.872+0000 7fa9a15d7700 1 --2- 172.21.15.23:0/3349782941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa99c061980 0x7fa99c0a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:53.883 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.872+0000 7fa9a15d7700 1 -- 172.21.15.23:0/3349782941 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c057dc0 con 0x7fa99c061980 2020-11-20T11:36:53.883 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.873+0000 7fa99b7fe700 1 --2- 172.21.15.23:0/3349782941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa99c061980 0x7fa99c0a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:53.883 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.873+0000 7fa99b7fe700 1 -- 172.21.15.23:0/3349782941 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa99c0a0880 con 0x7fa99c061980 2020-11-20T11:36:53.884 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.873+0000 7fa99b7fe700 1 --2- 172.21.15.23:0/3349782941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa99c061980 0x7fa99c0a4f50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fa98c009a10 tx=0x7fa98c01e000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:53.884 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.873+0000 7fa97bfff700 1 -- 172.21.15.23:0/3349782941 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c01d660 con 0x7fa99c061980 2020-11-20T11:36:53.884 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.874+0000 7fa97bfff700 1 -- 172.21.15.23:0/3349782941 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4774 (secure 0 0 0) 0x7fa98c011680 con 0x7fa99c061980 2020-11-20T11:36:53.884 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.874+0000 7fa97bfff700 1 -- 172.21.15.23:0/3349782941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa99c061980 msgr2=0x7fa99c0a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:53.884 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.874+0000 7fa97bfff700 1 --2- 172.21.15.23:0/3349782941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa99c061980 0x7fa99c0a4f50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fa98c009a10 tx=0x7fa98c01e000).stop 2020-11-20T11:36:53.885 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.874+0000 7fa97bfff700 1 -- 172.21.15.23:0/3349782941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa99c061980 msgr2=0x7fa99c0a4f50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:36:53.885 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.874+0000 7fa97bfff700 1 --2- 172.21.15.23:0/3349782941 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa99c061980 0x7fa99c0a4f50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:53.885 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.874+0000 7fa9a15d7700 1 -- 172.21.15.23:0/3349782941 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa974056d10 msgr2=0x7fa9740591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:53.885 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.874+0000 7fa9a15d7700 1 --2- 172.21.15.23:0/3349782941 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa974056d10 0x7fa9740591d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fa99400f6e0 tx=0x7fa994009220).stop 2020-11-20T11:36:53.886 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.874+0000 7fa9a15d7700 1 -- 172.21.15.23:0/3349782941 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa99c061380 msgr2=0x7fa99c14e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:53.886 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.874+0000 7fa9a15d7700 1 --2- 172.21.15.23:0/3349782941 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa99c061380 0x7fa99c14e330 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fa9900097c0 tx=0x7fa990006ea0).stop 2020-11-20T11:36:53.886 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.874+0000 7fa9a0dd6700 1 -- 172.21.15.23:0/3349782941 reap_dead start 2020-11-20T11:36:53.886 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.874+0000 7fa9a15d7700 1 -- 172.21.15.23:0/3349782941 shutdown_connections 2020-11-20T11:36:53.886 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.874+0000 7fa9a15d7700 1 --2- 172.21.15.23:0/3349782941 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa99c061380 0x7fa99c14e330 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:53.887 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.874+0000 7fa9a15d7700 1 -- 172.21.15.23:0/3349782941 >> 172.21.15.23:0/3349782941 conn(0x7fa99c0ae0a0 msgr2=0x7fa99c0ace80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:53.887 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.874+0000 7fa9a15d7700 1 -- 172.21.15.23:0/3349782941 shutdown_connections 2020-11-20T11:36:53.887 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:53.874+0000 7fa9a15d7700 1 -- 172.21.15.23:0/3349782941 wait complete. 2020-11-20T11:36:53.887 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:36:53.887 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:36:53.888 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:36:53.888 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:36:53.888 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 40, 2020-11-20T11:36:53.888 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:36:53.889 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:36:53.889 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:36:53.889 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:53.889 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 0, 2020-11-20T11:36:53.889 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:36:53.890 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:36:53.890 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:36:53.890 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:36:53.890 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:36:53.890 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:36:53.891 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:36:53.891 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:36:53.891 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:36:53.891 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:36:53.891 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:36:53.892 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:53.892 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:36:53.892 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:36:53.892 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:36:53.892 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:36:53.893 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:36:53.893 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:36:53.893 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:36:53.893 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:36:53.893 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:36:53.894 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:36:53.894 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:53.894 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:53.894 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:36:53.894 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:36:53.895 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:36:53.895 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:36:53.895 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:36:53.895 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:36:53.895 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:36:53.896 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:36:53.896 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:36:53.896 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:36:53.896 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:36:53.896 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:36:53.897 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:36:53.897 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:36:53.897 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:36:53.897 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:36:53.897 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:36:53.898 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:36:53.898 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:36:53.898 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:36:53.898 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:36:53.898 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:36:53.898 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:36:53.899 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:53.899 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:36:53.899 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:53.899 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:36:53.899 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:53.900 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:36:53.900 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:36:53.900 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:36:53.900 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:36:53.900 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:53.901 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:36:53.901 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:36:53.901 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:53.901 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:53.901 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:53.902 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:36:53.902 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:36:53.902 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:53.902 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:53.902 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:53.903 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:53.903 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:36:53.903 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:36:53.903 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:36:53.903 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:36:53.904 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:36:53.904 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:53.904 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:53.904 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:36:53.904 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:36:53.905 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:36:53.905 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:36:53.905 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:53.905 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:36:53.905 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:36:53.906 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:53.906 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:53.906 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:53.906 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:36:53.906 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:36:53.907 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:53.907 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:53.907 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:53.907 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:53.907 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:36:53.908 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:36:53.908 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:36:53.908 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:36:53.908 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:36:53.908 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:53.909 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:53.909 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:36:53.909 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:36:53.909 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:36:53.909 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:36:53.910 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:53.910 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:36:53.910 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:36:53.910 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:53.910 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:53.911 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:53.911 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:36:53.911 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:36:53.911 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:53.911 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:53.912 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:53.912 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:53.912 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:36:53.912 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:36:53.912 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:36:53.913 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:36:53.913 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:36:53.913 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:53.913 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:53.913 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:53.914 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:36:53.914 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:36:53.914 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:53.914 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:36:53.914 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:36:53.915 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:36:53.915 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:53.915 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:53.915 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:36:53.915 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:53.916 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:36:53.916 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:36:53.916 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:36:53.916 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:53.916 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:53.917 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:36:53.917 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:53.917 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:36:53.917 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:36:53.917 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:36:53.918 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:53.918 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:53.918 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:36:53.918 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:53.918 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:36:53.919 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:36:53.919 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 4 2020-11-20T11:36:53.919 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:53.919 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:53.919 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:36:53.920 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:53.920 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:36:53.920 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:36:53.920 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:36:53.920 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:53.921 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:53.921 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:53.921 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:36:53.921 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:36:53.923 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:36:54.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.059+0000 7f922b59e700 1 Processor -- start 2020-11-20T11:36:54.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.059+0000 7f922b59e700 1 -- start start 2020-11-20T11:36:54.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.060+0000 7f922b59e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f922c135f30 0x7f922c067b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:54.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.060+0000 7f922b59e700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f922c0a9fd0 con 0x7f922c067690 2020-11-20T11:36:54.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.060+0000 7f922b59e700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f922c068000 con 0x7f922c0af060 2020-11-20T11:36:54.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.060+0000 7f922b59e700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f922c1363a0 con 0x7f922c135f30 2020-11-20T11:36:54.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.060+0000 7f9229d9b700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f922c067690 legacy=0x7f922c135820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2020-11-20T11:36:54.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.061+0000 7f922ad9d700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f922c135f30 0x7f922c067b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:54.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.061+0000 7f922a59c700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f922c0af060 0x7f922c0a6590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:35234/0 (socket says 172.21.15.23:35234) 2020-11-20T11:36:54.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.061+0000 7f922a59c700 1 -- 172.21.15.23:0/2930958863 learned_addr learned my addr 172.21.15.23:0/2930958863 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:36:54.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.061+0000 7f922ad9d700 1 -- 172.21.15.23:0/2930958863 >> v1:172.21.15.155:6789/0 conn(0x7f922c0af060 legacy=0x7f922c0a6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:54.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.061+0000 7f922959a700 1 -- 172.21.15.23:0/2930958863 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 849268445 0 0) 0x7f922c068000 con 0x7f922c0af060 2020-11-20T11:36:54.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.061+0000 7f922ad9d700 1 -- 172.21.15.23:0/2930958863 >> v1:172.21.15.23:6789/0 conn(0x7f922c067690 legacy=0x7f922c135820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:36:54.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.061+0000 7f922ad9d700 1 -- 172.21.15.23:0/2930958863 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f922c0aa230 con 0x7f922c135f30 2020-11-20T11:36:54.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.061+0000 7f922ad9d700 1 --2- 172.21.15.23:0/2930958863 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f922c135f30 0x7f922c067b10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f922c0635a0 tx=0x7f9224009220).ready entity=mon.1 client_cookie=d941c847f6842c27 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:54.070 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.061+0000 7f922959a700 1 -- 172.21.15.23:0/2930958863 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922400bbb0 con 0x7f922c135f30 2020-11-20T11:36:54.071 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.062+0000 7f922959a700 1 -- 172.21.15.23:0/2930958863 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f922400bd10 con 0x7f922c135f30 2020-11-20T11:36:54.071 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.062+0000 7f922959a700 1 -- 172.21.15.23:0/2930958863 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9224011610 con 0x7f922c135f30 2020-11-20T11:36:54.071 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.062+0000 7f922b59e700 1 -- 172.21.15.23:0/2930958863 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f922c135f30 msgr2=0x7f922c067b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:54.071 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.062+0000 7f922b59e700 1 --2- 172.21.15.23:0/2930958863 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f922c135f30 0x7f922c067b10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f922c0635a0 tx=0x7f9224009220).stop 2020-11-20T11:36:54.072 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.062+0000 7f922b59e700 1 -- 172.21.15.23:0/2930958863 shutdown_connections 2020-11-20T11:36:54.072 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.062+0000 7f922b59e700 1 --2- 172.21.15.23:0/2930958863 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f922c135f30 0x7f922c067b10 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:54.072 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.062+0000 7f922b59e700 1 -- 172.21.15.23:0/2930958863 >> 172.21.15.23:0/2930958863 conn(0x7f922c063980 msgr2=0x7f922c063d90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:54.072 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.062+0000 7f922b59e700 1 -- 172.21.15.23:0/2930958863 shutdown_connections 2020-11-20T11:36:54.072 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.062+0000 7f922b59e700 1 -- 172.21.15.23:0/2930958863 wait complete. 2020-11-20T11:36:54.073 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.063+0000 7f922b59e700 1 Processor -- start 2020-11-20T11:36:54.073 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.063+0000 7f922b59e700 1 -- start start 2020-11-20T11:36:54.073 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.064+0000 7f922b59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f922c067690 0x7f922c1380a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:54.073 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.064+0000 7f922b59e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f922c0af060 0x7f922c138cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:54.073 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.064+0000 7f922b59e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f922c135f30 0x7f922c09e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:54.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.064+0000 7f922b59e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922c1376c0 con 0x7f922c067690 2020-11-20T11:36:54.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.064+0000 7f922b59e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922c137960 con 0x7f922c0af060 2020-11-20T11:36:54.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.064+0000 7f922b59e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f922c137c00 con 0x7f922c135f30 2020-11-20T11:36:54.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.064+0000 7f922a59c700 1 -- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f922c067690 msgr2=0x7f922c1380a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:36:54.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.064+0000 7f922a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f922c067690 0x7f922c1380a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:36:54.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.064+0000 7f9229d9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f922c0af060 0x7f922c138cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:54.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.064+0000 7f9229d9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f922c0af060 0x7f922c138cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:35912/0 (socket says 172.21.15.23:35912) 2020-11-20T11:36:54.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.064+0000 7f9229d9b700 1 -- 172.21.15.23:0/942807056 learned_addr learned my addr 172.21.15.23:0/942807056 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:36:54.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.064+0000 7f922ad9d700 1 --2- 172.21.15.23:0/942807056 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f922c135f30 0x7f922c09e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:54.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.065+0000 7f9229d9b700 1 -- 172.21.15.23:0/942807056 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f922c135f30 msgr2=0x7f922c09e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:54.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.065+0000 7f9229d9b700 1 --2- 172.21.15.23:0/942807056 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f922c135f30 0x7f922c09e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:54.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.065+0000 7f9229d9b700 1 -- 172.21.15.23:0/942807056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f922c067690 msgr2=0x7f922c1380a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:36:54.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.065+0000 7f9229d9b700 1 --2- 172.21.15.23:0/942807056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f922c067690 0x7f922c1380a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:36:54.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.065+0000 7f9229d9b700 1 -- 172.21.15.23:0/942807056 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f922c1391b0 con 0x7f922c0af060 2020-11-20T11:36:54.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.065+0000 7f9229d9b700 1 --2- 172.21.15.23:0/942807056 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f922c0af060 0x7f922c138cc0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f92200097c0 tx=0x7f9220006e60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:54.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.065+0000 7f921a7fc700 1 -- 172.21.15.23:0/942807056 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9220006170 con 0x7f922c0af060 2020-11-20T11:36:54.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.065+0000 7f921a7fc700 1 -- 172.21.15.23:0/942807056 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92200062d0 con 0x7f922c0af060 2020-11-20T11:36:54.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.065+0000 7f922b59e700 1 -- 172.21.15.23:0/942807056 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f922c09e6b0 con 0x7f922c0af060 2020-11-20T11:36:54.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.065+0000 7f921a7fc700 1 -- 172.21.15.23:0/942807056 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9220003920 con 0x7f922c0af060 2020-11-20T11:36:54.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.065+0000 7f922b59e700 1 -- 172.21.15.23:0/942807056 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f922c147970 con 0x7f922c0af060 2020-11-20T11:36:54.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.066+0000 7f922b59e700 1 --2- 172.21.15.23:0/942807056 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f922c147bf0 0x7f922c147fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:54.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.066+0000 7f922b59e700 1 -- 172.21.15.23:0/942807056 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f922c09e8a0 con 0x7f922c147bf0 2020-11-20T11:36:54.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.066+0000 7f922a59c700 1 --2- 172.21.15.23:0/942807056 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f922c147bf0 0x7f922c147fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:54.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.066+0000 7f921a7fc700 1 -- 172.21.15.23:0/942807056 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9220017030 con 0x7f922c0af060 2020-11-20T11:36:54.079 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.067+0000 7f921a7fc700 1 --2- 172.21.15.23:0/942807056 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9200056d10 0x7f92000591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:54.079 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.067+0000 7f922a59c700 1 -- 172.21.15.23:0/942807056 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f922c139340 con 0x7f922c147bf0 2020-11-20T11:36:54.079 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.067+0000 7f921a7fc700 1 -- 172.21.15.23:0/942807056 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f922006f3d0 con 0x7f922c0af060 2020-11-20T11:36:54.079 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.067+0000 7f922a59c700 1 --2- 172.21.15.23:0/942807056 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f922c147bf0 0x7f922c147fd0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f921c004620 tx=0x7f921c007bb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:54.079 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.067+0000 7f922ad9d700 1 --2- 172.21.15.23:0/942807056 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9200056d10 0x7f92000591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:54.080 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.067+0000 7f921a7fc700 1 -- 172.21.15.23:0/942807056 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921c007320 con 0x7f922c147bf0 2020-11-20T11:36:54.080 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.067+0000 7f922ad9d700 1 --2- 172.21.15.23:0/942807056 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9200056d10 0x7f92000591d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f92240090b0 tx=0x7f9224016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:54.080 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.067+0000 7f921a7fc700 1 -- 172.21.15.23:0/942807056 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f921c007480 con 0x7f922c147bf0 2020-11-20T11:36:54.080 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.067+0000 7f921a7fc700 1 -- 172.21.15.23:0/942807056 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f922c147bf0 msgr2=0x7f922c147fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:54.080 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.067+0000 7f921a7fc700 1 --2- 172.21.15.23:0/942807056 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f922c147bf0 0x7f922c147fd0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f921c004620 tx=0x7f921c007bb0).stop 2020-11-20T11:36:54.081 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.067+0000 7f921a7fc700 1 -- 172.21.15.23:0/942807056 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f922c147bf0 msgr2=0x7f922c147fd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:36:54.081 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.067+0000 7f921a7fc700 1 --2- 172.21.15.23:0/942807056 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f922c147bf0 0x7f922c147fd0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:54.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.074+0000 7f922b59e700 1 --2- 172.21.15.23:0/942807056 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f922c1484c0 0x7f922c14af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:36:54.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.074+0000 7f922b59e700 1 -- 172.21.15.23:0/942807056 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f922c137dc0 con 0x7f922c1484c0 2020-11-20T11:36:54.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.074+0000 7f922a59c700 1 --2- 172.21.15.23:0/942807056 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f922c1484c0 0x7f922c14af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:36:54.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.075+0000 7f922a59c700 1 -- 172.21.15.23:0/942807056 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f922c057dc0 con 0x7f922c1484c0 2020-11-20T11:36:54.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.075+0000 7f922a59c700 1 --2- 172.21.15.23:0/942807056 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f922c1484c0 0x7f922c14af00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f921c013000 tx=0x7f921c016000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:36:54.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.075+0000 7f921a7fc700 1 -- 172.21.15.23:0/942807056 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921c013bc0 con 0x7f922c1484c0 2020-11-20T11:36:54.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.075+0000 7f921a7fc700 1 -- 172.21.15.23:0/942807056 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4432 (secure 0 0 0) 0x7f921c007110 con 0x7f922c1484c0 2020-11-20T11:36:54.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.075+0000 7f921a7fc700 1 -- 172.21.15.23:0/942807056 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f922c1484c0 msgr2=0x7f922c14af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:54.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.075+0000 7f921a7fc700 1 --2- 172.21.15.23:0/942807056 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f922c1484c0 0x7f922c14af00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f921c013000 tx=0x7f921c016000).stop 2020-11-20T11:36:54.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.075+0000 7f921a7fc700 1 -- 172.21.15.23:0/942807056 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f922c1484c0 msgr2=0x7f922c14af00 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:36:54.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.075+0000 7f921a7fc700 1 --2- 172.21.15.23:0/942807056 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f922c1484c0 0x7f922c14af00 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0x7f921c013000 tx=0x7f921c016000).stop 2020-11-20T11:36:54.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.075+0000 7f922b59e700 1 -- 172.21.15.23:0/942807056 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9200056d10 msgr2=0x7f92000591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:54.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.075+0000 7f922b59e700 1 --2- 172.21.15.23:0/942807056 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9200056d10 0x7f92000591d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f92240090b0 tx=0x7f9224016000).stop 2020-11-20T11:36:54.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.075+0000 7f922b59e700 1 -- 172.21.15.23:0/942807056 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f922c0af060 msgr2=0x7f922c138cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:36:54.086 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.075+0000 7f922b59e700 1 --2- 172.21.15.23:0/942807056 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f922c0af060 0x7f922c138cc0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f92200097c0 tx=0x7f9220006e60).stop 2020-11-20T11:36:54.086 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.075+0000 7f922ad9d700 1 -- 172.21.15.23:0/942807056 reap_dead start 2020-11-20T11:36:54.086 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.076+0000 7f922b59e700 1 -- 172.21.15.23:0/942807056 shutdown_connections 2020-11-20T11:36:54.086 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.076+0000 7f922b59e700 1 --2- 172.21.15.23:0/942807056 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f922c0af060 0x7f922c138cc0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:36:54.086 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.076+0000 7f922b59e700 1 -- 172.21.15.23:0/942807056 >> 172.21.15.23:0/942807056 conn(0x7f922c063980 msgr2=0x7f922c1355f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:36:54.087 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.076+0000 7f922b59e700 1 -- 172.21.15.23:0/942807056 shutdown_connections 2020-11-20T11:36:54.087 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:36:54.076+0000 7f922b59e700 1 -- 172.21.15.23:0/942807056 wait complete. 2020-11-20T11:36:54.087 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:36:54.087 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:36:54.088 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:36:54.088 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:36:54.088 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 40, 2020-11-20T11:36:54.088 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:36:54.088 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:36:54.089 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:36:54.089 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:54.089 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 1, 2020-11-20T11:36:54.089 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:36:54.089 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:36:54.090 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:36:54.090 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:36:54.090 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:36:54.090 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:36:54.090 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:36:54.091 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:36:54.091 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:36:54.091 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:36:54.091 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:36:54.091 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:54.092 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:36:54.092 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:36:54.092 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:36:54.092 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:36:54.092 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:36:54.093 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:36:54.093 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:36:54.093 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:36:54.093 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:36:54.093 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:36:54.094 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:54.094 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:54.094 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:36:54.094 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:36:54.094 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:36:54.095 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:36:54.095 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:36:54.095 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:36:54.095 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:36:54.095 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:36:54.096 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:36:54.096 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:36:54.096 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:36:54.096 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:36:54.096 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:36:54.097 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:36:54.097 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:36:54.097 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:36:54.097 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:36:54.097 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:36:54.098 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:36:54.098 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:36:54.098 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:36:54.098 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:36:54.098 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:36:54.099 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:54.099 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:36:54.099 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:54.099 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:36:54.099 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:54.100 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:36:54.100 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:36:54.100 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:36:54.100 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:36:54.100 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:54.100 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:36:54.101 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:36:54.101 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:54.101 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:54.101 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:54.101 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:36:54.102 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:36:54.102 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:54.102 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:54.102 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:54.102 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:54.103 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:36:54.103 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:36:54.103 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:36:54.103 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:36:54.103 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:36:54.104 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:54.104 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:54.104 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:36:54.104 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:36:54.104 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:36:54.105 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:36:54.105 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:54.105 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:36:54.105 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:36:54.105 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:54.106 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:54.106 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:54.106 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:36:54.106 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:36:54.106 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:54.107 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:54.107 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:54.107 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:54.107 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:36:54.107 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:36:54.108 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:36:54.108 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:36:54.108 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:36:54.108 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:54.108 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:54.109 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:36:54.109 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:36:54.109 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:36:54.109 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:36:54.109 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:54.110 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:36:54.110 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:36:54.110 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:54.110 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:54.110 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:54.111 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:36:54.111 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:36:54.111 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:36:54.111 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:54.111 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:54.112 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:54.112 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:36:54.112 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:36:54.112 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:36:54.112 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:36:54.113 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:36:54.113 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:54.113 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:54.113 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:54.113 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:36:54.114 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:36:54.114 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:54.114 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:36:54.114 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:36:54.114 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:36:54.115 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:54.115 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:54.115 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:36:54.115 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:54.115 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:36:54.115 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:36:54.116 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:36:54.116 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:54.116 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:36:54.116 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:36:54.116 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:36:54.117 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:36:54.117 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:36:54.117 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:36:54.117 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:36:54.118 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:36:54.118 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:36:54.118 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:36:54.118 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:36:54.119 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2020-11-20T11:37:00.240 INFO:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:37:00.243 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:37:14.121 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2020-11-20T11:37:14.121 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2020-11-20T11:37:14.121 INFO:tasks.ceph.mon.a:Restarting daemon 2020-11-20T11:37:14.122 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2020-11-20T11:37:14.127 INFO:tasks.ceph.mon.a:Started 2020-11-20T11:37:14.127 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:37:14.127 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:37:14.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.253+0000 7f99f1612700 1 Processor -- start 2020-11-20T11:37:14.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.253+0000 7f99f1612700 1 -- start start 2020-11-20T11:37:14.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.253+0000 7f99f1612700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f99ec0aae90 0x7f99ec1381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:14.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.253+0000 7f99f1612700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f99ec13aeb0 0x7f99ec0a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:14.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.253+0000 7f99f1612700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99ec13e600 con 0x7f99ec0aaab0 2020-11-20T11:37:14.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.253+0000 7f99f1612700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99ec13e8a0 con 0x7f99ec13aeb0 2020-11-20T11:37:14.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.253+0000 7f99f1612700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99ec0ab6d0 con 0x7f99ec0aae90 2020-11-20T11:37:14.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.253+0000 7f99eb7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f99ec13aeb0 0x7f99ec0a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:14.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.253+0000 7f99eb7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f99ec13aeb0 0x7f99ec0a8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43036/0 (socket says 172.21.15.23:43036) 2020-11-20T11:37:14.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.254+0000 7f99eb7fe700 1 -- 172.21.15.23:0/3367636496 learned_addr learned my addr 172.21.15.23:0/3367636496 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:37:14.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.253+0000 7f99eaffd700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f99ec0aaab0 0x7f99ec13ed40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57282/0 (socket says 172.21.15.23:57282) 2020-11-20T11:37:14.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.254+0000 7f99eb7fe700 1 -- 172.21.15.23:0/3367636496 >> v2:172.21.15.23:3300/0 conn(0x7f99ec13aeb0 msgr2=0x7f99ec0a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:37:14.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.254+0000 7f99eb7fe700 1 -- 172.21.15.23:0/3367636496 >> v2:172.21.15.23:3300/0 conn(0x7f99ec13aeb0 msgr2=0x7f99ec0a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:37:14.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.254+0000 7f99eb7fe700 1 --2- 172.21.15.23:0/3367636496 >> v2:172.21.15.23:3300/0 conn(0x7f99ec13aeb0 0x7f99ec0a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:14.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.254+0000 7f99eb7fe700 1 --2- 172.21.15.23:0/3367636496 >> v2:172.21.15.23:3300/0 conn(0x7f99ec13aeb0 0x7f99ec0a8c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:37:14.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.254+0000 7f99e3fff700 1 --2- 172.21.15.23:0/3367636496 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f99ec0aae90 0x7f99ec1381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:14.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.254+0000 7f99e3fff700 1 -- 172.21.15.23:0/3367636496 >> v2:172.21.15.23:3300/0 conn(0x7f99ec13aeb0 msgr2=0x7f99ec0a8c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:37:14.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.254+0000 7f99e3fff700 1 --2- 172.21.15.23:0/3367636496 >> v2:172.21.15.23:3300/0 conn(0x7f99ec13aeb0 0x7f99ec0a8c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:14.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.254+0000 7f99e3fff700 1 -- 172.21.15.23:0/3367636496 >> v1:172.21.15.23:6789/0 conn(0x7f99ec0aaab0 legacy=0x7f99ec13ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:14.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.254+0000 7f99e3fff700 1 -- 172.21.15.23:0/3367636496 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99ec138690 con 0x7f99ec0aae90 2020-11-20T11:37:14.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.254+0000 7f99e3fff700 1 --2- 172.21.15.23:0/3367636496 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f99ec0aae90 0x7f99ec1381a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f99d4006fd0 tx=0x7f99d4006e30).ready entity=mon.2 client_cookie=bd0576cdef6290a0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:14.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.254+0000 7f99ea7fc700 1 -- 172.21.15.23:0/3367636496 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d40097e0 con 0x7f99ec0aae90 2020-11-20T11:37:14.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.255+0000 7f99ea7fc700 1 -- 172.21.15.23:0/3367636496 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d4000e90 con 0x7f99ec0aae90 2020-11-20T11:37:14.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.255+0000 7f99ea7fc700 1 -- 172.21.15.23:0/3367636496 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d4009b50 con 0x7f99ec0aae90 2020-11-20T11:37:14.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.255+0000 7f99f1612700 1 -- 172.21.15.23:0/3367636496 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f99ec0aae90 msgr2=0x7f99ec1381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:14.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.255+0000 7f99f1612700 1 --2- 172.21.15.23:0/3367636496 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f99ec0aae90 0x7f99ec1381a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f99d4006fd0 tx=0x7f99d4006e30).stop 2020-11-20T11:37:14.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.255+0000 7f99f1612700 1 -- 172.21.15.23:0/3367636496 shutdown_connections 2020-11-20T11:37:14.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.255+0000 7f99f1612700 1 --2- 172.21.15.23:0/3367636496 >> v2:172.21.15.23:3300/0 conn(0x7f99ec13aeb0 0x7f99ec0a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:14.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.255+0000 7f99f1612700 1 --2- 172.21.15.23:0/3367636496 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f99ec0aae90 0x7f99ec1381a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:14.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.255+0000 7f99f1612700 1 -- 172.21.15.23:0/3367636496 >> 172.21.15.23:0/3367636496 conn(0x7f99ec0678e0 msgr2=0x7f99ec067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:14.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.255+0000 7f99f1612700 1 -- 172.21.15.23:0/3367636496 shutdown_connections 2020-11-20T11:37:14.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.255+0000 7f99f1612700 1 -- 172.21.15.23:0/3367636496 wait complete. 2020-11-20T11:37:14.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.256+0000 7f99f1612700 1 Processor -- start 2020-11-20T11:37:14.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.256+0000 7f99f1612700 1 -- start start 2020-11-20T11:37:14.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.257+0000 7f99f1612700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99ec0aaab0 0x7f99ec0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:14.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.257+0000 7f99f1612700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f99ec0aae90 0x7f99ec13d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:14.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.257+0000 7f99f1612700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99ec14e3e0 0x7f99ec1338b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:14.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.257+0000 7f99f1612700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99ec136680 con 0x7f99ec14e3e0 2020-11-20T11:37:14.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.257+0000 7f99f1612700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99ec14e860 con 0x7f99ec0aaab0 2020-11-20T11:37:14.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.257+0000 7f99f1612700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99ec14da00 con 0x7f99ec0aae90 2020-11-20T11:37:14.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.257+0000 7f99eb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99ec14e3e0 0x7f99ec1338b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:14.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.257+0000 7f99eb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99ec14e3e0 0x7f99ec1338b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43042/0 (socket says 172.21.15.23:43042) 2020-11-20T11:37:14.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.257+0000 7f99eb7fe700 1 -- 172.21.15.23:0/706558112 learned_addr learned my addr 172.21.15.23:0/706558112 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:37:14.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.257+0000 7f99eb7fe700 1 -- 172.21.15.23:0/706558112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99ec14e3e0 msgr2=0x7f99ec1338b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2020-11-20T11:37:14.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.257+0000 7f99eb7fe700 1 -- 172.21.15.23:0/706558112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99ec14e3e0 msgr2=0x7f99ec1338b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:37:14.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.257+0000 7f99eb7fe700 1 --2- 172.21.15.23:0/706558112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99ec14e3e0 0x7f99ec1338b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:14.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.257+0000 7f99eb7fe700 1 --2- 172.21.15.23:0/706558112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99ec14e3e0 0x7f99ec1338b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:37:14.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.257+0000 7f99e3fff700 1 --2- 172.21.15.23:0/706558112 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f99ec0aae90 0x7f99ec13d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:14.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.257+0000 7f99eaffd700 1 --2- 172.21.15.23:0/706558112 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99ec0aaab0 0x7f99ec0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:14.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.258+0000 7f99e3fff700 1 -- 172.21.15.23:0/706558112 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99ec0aaab0 msgr2=0x7f99ec0a5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:14.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.258+0000 7f99e3fff700 1 --2- 172.21.15.23:0/706558112 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99ec0aaab0 0x7f99ec0a5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:14.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.258+0000 7f99e3fff700 1 -- 172.21.15.23:0/706558112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99ec14e3e0 msgr2=0x7f99ec1338b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:37:14.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.258+0000 7f99e3fff700 1 --2- 172.21.15.23:0/706558112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99ec14e3e0 0x7f99ec1338b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:14.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.258+0000 7f99e3fff700 1 -- 172.21.15.23:0/706558112 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99ec14dca0 con 0x7f99ec0aae90 2020-11-20T11:37:14.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.258+0000 7f99e3fff700 1 --2- 172.21.15.23:0/706558112 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f99ec0aae90 0x7f99ec13d510 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f99d4012000 tx=0x7f99d4015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:14.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.258+0000 7f99e2ffd700 1 -- 172.21.15.23:0/706558112 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d4011b70 con 0x7f99ec0aae90 2020-11-20T11:37:14.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.258+0000 7f99e2ffd700 1 -- 172.21.15.23:0/706558112 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d4011cd0 con 0x7f99ec0aae90 2020-11-20T11:37:14.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.258+0000 7f99f1612700 1 -- 172.21.15.23:0/706558112 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99ec14df40 con 0x7f99ec0aae90 2020-11-20T11:37:14.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.258+0000 7f99e2ffd700 1 -- 172.21.15.23:0/706558112 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d401e670 con 0x7f99ec0aae90 2020-11-20T11:37:14.274 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.258+0000 7f99f1612700 1 -- 172.21.15.23:0/706558112 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99ec133f60 con 0x7f99ec0aae90 2020-11-20T11:37:14.274 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.259+0000 7f99f1612700 1 -- 172.21.15.23:0/706558112 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99ec05c7d0 con 0x7f99ec0aae90 2020-11-20T11:37:14.274 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.259+0000 7f99e2ffd700 1 -- 172.21.15.23:0/706558112 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f99d4025030 con 0x7f99ec0aae90 2020-11-20T11:37:14.274 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.260+0000 7f99e2ffd700 1 --2- 172.21.15.23:0/706558112 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f99c8056d60 0x7f99c8059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:14.274 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.260+0000 7f99e2ffd700 1 -- 172.21.15.23:0/706558112 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f99d4071580 con 0x7f99ec0aae90 2020-11-20T11:37:14.275 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.260+0000 7f99eaffd700 1 --2- 172.21.15.23:0/706558112 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f99c8056d60 0x7f99c8059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:14.275 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.260+0000 7f99eaffd700 1 --2- 172.21.15.23:0/706558112 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f99c8056d60 0x7f99c8059220 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f99dc004610 tx=0x7f99dc007bc0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:14.275 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.264+0000 7f99e2ffd700 1 -- 172.21.15.23:0/706558112 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f99d4046870 con 0x7f99ec0aae90 2020-11-20T11:37:14.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.492+0000 7f99f1612700 1 -- 172.21.15.23:0/706558112 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f99ec0a9620 con 0x7f99ec0aae90 2020-11-20T11:37:14.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.493+0000 7f99e2ffd700 1 -- 172.21.15.23:0/706558112 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1472 (secure 0 0 0) 0x7f99d4023020 con 0x7f99ec0aae90 2020-11-20T11:37:14.496 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":40,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:37:14.498 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.495+0000 7f99f1612700 1 -- 172.21.15.23:0/706558112 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f99c8056d60 msgr2=0x7f99c8059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:14.498 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.495+0000 7f99f1612700 1 --2- 172.21.15.23:0/706558112 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f99c8056d60 0x7f99c8059220 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f99dc004610 tx=0x7f99dc007bc0).stop 2020-11-20T11:37:14.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.495+0000 7f99f1612700 1 -- 172.21.15.23:0/706558112 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f99ec0aae90 msgr2=0x7f99ec13d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:14.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.495+0000 7f99f1612700 1 --2- 172.21.15.23:0/706558112 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f99ec0aae90 0x7f99ec13d510 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f99d4012000 tx=0x7f99d4015000).stop 2020-11-20T11:37:14.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.495+0000 7f99f1612700 1 -- 172.21.15.23:0/706558112 shutdown_connections 2020-11-20T11:37:14.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.495+0000 7f99f1612700 1 --2- 172.21.15.23:0/706558112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99ec14e3e0 0x7f99ec1338b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:14.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.495+0000 7f99f1612700 1 --2- 172.21.15.23:0/706558112 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99ec0aaab0 0x7f99ec0a5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:14.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.495+0000 7f99f1612700 1 --2- 172.21.15.23:0/706558112 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f99c8056d60 0x7f99c8059220 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:14.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.495+0000 7f99f1612700 1 --2- 172.21.15.23:0/706558112 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f99ec0aae90 0x7f99ec13d510 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:14.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.495+0000 7f99f1612700 1 -- 172.21.15.23:0/706558112 >> 172.21.15.23:0/706558112 conn(0x7f99ec0678e0 msgr2=0x7f99ec068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:14.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.495+0000 7f99f1612700 1 -- 172.21.15.23:0/706558112 shutdown_connections 2020-11-20T11:37:14.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:14.495+0000 7f99f1612700 1 -- 172.21.15.23:0/706558112 wait complete. 2020-11-20T11:37:17.513 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:37:17.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.634+0000 7f463cf22700 1 Processor -- start 2020-11-20T11:37:17.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.634+0000 7f463cf22700 1 -- start start 2020-11-20T11:37:17.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.635+0000 7f463cf22700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f46380aaab0 0x7f463813ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:17.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.635+0000 7f463cf22700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46381388b0 con 0x7f463813aeb0 2020-11-20T11:37:17.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.635+0000 7f463cf22700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f463813e600 con 0x7f46380aae90 2020-11-20T11:37:17.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.635+0000 7f463cf22700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f463813e8a0 con 0x7f46380aaab0 2020-11-20T11:37:17.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.635+0000 7f463659c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f46380aaab0 0x7f463813ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:17.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.635+0000 7f4636d9d700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f463813aeb0 0x7f46380a8c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57316/0 (socket says 172.21.15.23:57316) 2020-11-20T11:37:17.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.635+0000 7f4636d9d700 1 -- 172.21.15.23:0/2348619132 learned_addr learned my addr 172.21.15.23:0/2348619132 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:37:17.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.636+0000 7f463659c700 1 -- 172.21.15.23:0/2348619132 >> v2:172.21.15.23:3300/0 conn(0x7f46380aaab0 msgr2=0x7f463813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:37:17.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.636+0000 7f463659c700 1 -- 172.21.15.23:0/2348619132 >> v2:172.21.15.23:3300/0 conn(0x7f46380aaab0 msgr2=0x7f463813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:37:17.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.636+0000 7f463659c700 1 --2- 172.21.15.23:0/2348619132 >> v2:172.21.15.23:3300/0 conn(0x7f46380aaab0 0x7f463813ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:17.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.636+0000 7f463659c700 1 --2- 172.21.15.23:0/2348619132 >> v2:172.21.15.23:3300/0 conn(0x7f46380aaab0 0x7f463813ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:37:17.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.636+0000 7f463559a700 1 -- 172.21.15.23:0/2348619132 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2305072890 0 0) 0x7f463813e600 con 0x7f46380aae90 2020-11-20T11:37:17.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.636+0000 7f463559a700 1 -- 172.21.15.23:0/2348619132 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f461c003280 con 0x7f46380aae90 2020-11-20T11:37:17.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.636+0000 7f463559a700 1 -- 172.21.15.23:0/2348619132 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 522496837 0 0) 0x7f461c003280 con 0x7f46380aae90 2020-11-20T11:37:17.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.636+0000 7f463559a700 1 -- 172.21.15.23:0/2348619132 >> v2:172.21.15.23:3300/0 conn(0x7f46380aaab0 msgr2=0x7f463813ed40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:37:17.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.636+0000 7f463559a700 1 --2- 172.21.15.23:0/2348619132 >> v2:172.21.15.23:3300/0 conn(0x7f46380aaab0 0x7f463813ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:17.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.636+0000 7f463559a700 1 -- 172.21.15.23:0/2348619132 >> v1:172.21.15.23:6789/0 conn(0x7f463813aeb0 legacy=0x7f46380a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:17.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.636+0000 7f463559a700 1 -- 172.21.15.23:0/2348619132 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46380ab6d0 con 0x7f46380aae90 2020-11-20T11:37:17.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.636+0000 7f463559a700 1 -- 172.21.15.23:0/2348619132 <== mon.1 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f462c001520 con 0x7f46380aae90 2020-11-20T11:37:17.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.636+0000 7f463559a700 1 -- 172.21.15.23:0/2348619132 >> v1:172.21.15.155:6789/0 conn(0x7f46380aae90 legacy=0x7f46381381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:17.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.636+0000 7f463559a700 1 --2- 172.21.15.23:0/2348619132 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f461c0037c0 0x7f461c003be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:17.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.636+0000 7f463559a700 1 --2- 172.21.15.23:0/2348619132 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f461c004520 0x7f461c0049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:17.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.636+0000 7f463559a700 1 --2- 172.21.15.23:0/2348619132 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f461c004e90 0x7f461c00b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:17.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.636+0000 7f463559a700 1 -- 172.21.15.23:0/2348619132 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463813e8a0 con 0x7f461c0037c0 2020-11-20T11:37:17.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.636+0000 7f463559a700 1 -- 172.21.15.23:0/2348619132 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f461c005370 con 0x7f461c004520 2020-11-20T11:37:17.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.636+0000 7f463559a700 1 -- 172.21.15.23:0/2348619132 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f461c00bc60 con 0x7f461c004e90 2020-11-20T11:37:17.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.637+0000 7f463659c700 1 --2- 172.21.15.23:0/2348619132 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f461c0037c0 0x7f461c003be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:17.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.637+0000 7f4635d9b700 1 --2- 172.21.15.23:0/2348619132 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f461c004520 0x7f461c0049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:17.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.637+0000 7f4636d9d700 1 --2- 172.21.15.23:0/2348619132 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f461c004e90 0x7f461c00b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:17.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.637+0000 7f463659c700 1 -- 172.21.15.23:0/2348619132 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f461c004e90 msgr2=0x7f461c00b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:37:17.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.637+0000 7f463659c700 1 --2- 172.21.15.23:0/2348619132 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f461c004e90 0x7f461c00b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:17.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.637+0000 7f463659c700 1 -- 172.21.15.23:0/2348619132 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f461c004520 msgr2=0x7f461c0049a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:37:17.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.637+0000 7f463659c700 1 --2- 172.21.15.23:0/2348619132 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f461c004520 0x7f461c0049a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:17.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.637+0000 7f4636d9d700 1 -- 172.21.15.23:0/2348619132 reap_dead start 2020-11-20T11:37:17.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.637+0000 7f463659c700 1 -- 172.21.15.23:0/2348619132 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f461c003280 con 0x7f461c0037c0 2020-11-20T11:37:17.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.637+0000 7f463659c700 1 --2- 172.21.15.23:0/2348619132 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f461c0037c0 0x7f461c003be0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f462800bb20 tx=0x7f4628008040).ready entity=mon.0 client_cookie=c46cb2eb66fd52e6 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:17.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.637+0000 7f463559a700 1 -- 172.21.15.23:0/2348619132 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462800d860 con 0x7f461c0037c0 2020-11-20T11:37:17.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.637+0000 7f463559a700 1 -- 172.21.15.23:0/2348619132 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f462800d9c0 con 0x7f461c0037c0 2020-11-20T11:37:17.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.637+0000 7f463559a700 1 -- 172.21.15.23:0/2348619132 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462800dd60 con 0x7f461c0037c0 2020-11-20T11:37:17.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.638+0000 7f463cf22700 1 -- 172.21.15.23:0/2348619132 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f461c0037c0 msgr2=0x7f461c003be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:17.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.638+0000 7f463cf22700 1 --2- 172.21.15.23:0/2348619132 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f461c0037c0 0x7f461c003be0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f462800bb20 tx=0x7f4628008040).stop 2020-11-20T11:37:17.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.638+0000 7f463cf22700 1 -- 172.21.15.23:0/2348619132 shutdown_connections 2020-11-20T11:37:17.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.638+0000 7f463cf22700 1 --2- 172.21.15.23:0/2348619132 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f461c0037c0 0x7f461c003be0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:17.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.638+0000 7f463cf22700 1 -- 172.21.15.23:0/2348619132 >> 172.21.15.23:0/2348619132 conn(0x7f46380678e0 msgr2=0x7f4638067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:17.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.638+0000 7f463cf22700 1 -- 172.21.15.23:0/2348619132 shutdown_connections 2020-11-20T11:37:17.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.638+0000 7f463cf22700 1 -- 172.21.15.23:0/2348619132 wait complete. 2020-11-20T11:37:17.654 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.639+0000 7f463cf22700 1 Processor -- start 2020-11-20T11:37:17.654 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.639+0000 7f463cf22700 1 -- start start 2020-11-20T11:37:17.654 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.639+0000 7f463cf22700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f461c0037c0 0x7f463813cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:17.654 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.639+0000 7f463cf22700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f463809f280 0x7f46380a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:17.654 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.639+0000 7f463cf22700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46380a7390 0x7f46380a7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:17.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.639+0000 7f463cf22700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463814aa40 con 0x7f46380a7390 2020-11-20T11:37:17.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.639+0000 7f463cf22700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463814ace0 con 0x7f461c0037c0 2020-11-20T11:37:17.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.639+0000 7f463cf22700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f463814af80 con 0x7f463809f280 2020-11-20T11:37:17.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.640+0000 7f4636d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46380a7390 0x7f46380a7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:17.656 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.640+0000 7f463659c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f461c0037c0 0x7f463813cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:17.656 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.640+0000 7f4635d9b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f463809f280 0x7f46380a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:17.656 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.640+0000 7f463659c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f461c0037c0 0x7f463813cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:35970/0 (socket says 172.21.15.23:35970) 2020-11-20T11:37:17.656 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.640+0000 7f4636d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46380a7390 0x7f46380a7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43078/0 (socket says 172.21.15.23:43078) 2020-11-20T11:37:17.656 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.640+0000 7f463659c700 1 -- 172.21.15.23:0/2989782691 learned_addr learned my addr 172.21.15.23:0/2989782691 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:37:17.657 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.640+0000 7f4636d9d700 1 -- 172.21.15.23:0/2989782691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46380a7390 msgr2=0x7f46380a7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2020-11-20T11:37:17.657 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.640+0000 7f4636d9d700 1 -- 172.21.15.23:0/2989782691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46380a7390 msgr2=0x7f46380a7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:37:17.657 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.640+0000 7f4636d9d700 1 --2- 172.21.15.23:0/2989782691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46380a7390 0x7f46380a7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:17.657 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.640+0000 7f4636d9d700 1 --2- 172.21.15.23:0/2989782691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46380a7390 0x7f46380a7810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:37:17.657 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.640+0000 7f4635d9b700 1 -- 172.21.15.23:0/2989782691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f461c0037c0 msgr2=0x7f463813cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:17.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.640+0000 7f4635d9b700 1 --2- 172.21.15.23:0/2989782691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f461c0037c0 0x7f463813cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:17.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.640+0000 7f4635d9b700 1 -- 172.21.15.23:0/2989782691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46380a7390 msgr2=0x7f46380a7810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:37:17.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.640+0000 7f4635d9b700 1 --2- 172.21.15.23:0/2989782691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46380a7390 0x7f46380a7810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:17.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.640+0000 7f4635d9b700 1 -- 172.21.15.23:0/2989782691 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f463814b420 con 0x7f463809f280 2020-11-20T11:37:17.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.640+0000 7f463659c700 1 --2- 172.21.15.23:0/2989782691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f461c0037c0 0x7f463813cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:37:17.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.640+0000 7f4635d9b700 1 --2- 172.21.15.23:0/2989782691 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f463809f280 0x7f46380a5170 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f462c002e00 tx=0x7f462c007680).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:17.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.643+0000 7f46267fc700 1 -- 172.21.15.23:0/2989782691 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462c00d820 con 0x7f463809f280 2020-11-20T11:37:17.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.643+0000 7f46267fc700 1 -- 172.21.15.23:0/2989782691 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f462c00d980 con 0x7f463809f280 2020-11-20T11:37:17.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.643+0000 7f463cf22700 1 -- 172.21.15.23:0/2989782691 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f463813da90 con 0x7f463809f280 2020-11-20T11:37:17.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.643+0000 7f46267fc700 1 -- 172.21.15.23:0/2989782691 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462c00dd00 con 0x7f463809f280 2020-11-20T11:37:17.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.643+0000 7f463cf22700 1 -- 172.21.15.23:0/2989782691 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f463814a810 con 0x7f463809f280 2020-11-20T11:37:17.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.644+0000 7f463cf22700 1 -- 172.21.15.23:0/2989782691 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f463805c7d0 con 0x7f463809f280 2020-11-20T11:37:17.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.644+0000 7f46267fc700 1 -- 172.21.15.23:0/2989782691 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f462c013600 con 0x7f463809f280 2020-11-20T11:37:17.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.645+0000 7f46267fc700 1 --2- 172.21.15.23:0/2989782691 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f460c056d60 0x7f460c059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:17.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.645+0000 7f46267fc700 1 -- 172.21.15.23:0/2989782691 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f462c0703a0 con 0x7f463809f280 2020-11-20T11:37:17.661 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.645+0000 7f463659c700 1 --2- 172.21.15.23:0/2989782691 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f460c056d60 0x7f460c059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:17.661 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.645+0000 7f463659c700 1 --2- 172.21.15.23:0/2989782691 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f460c056d60 0x7f460c059220 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f462800e010 tx=0x7f4628008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:17.661 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.649+0000 7f46267fc700 1 -- 172.21.15.23:0/2989782691 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f462c045610 con 0x7f463809f280 2020-11-20T11:37:17.954 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.951+0000 7f463cf22700 1 -- 172.21.15.23:0/2989782691 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f463814b110 con 0x7f463809f280 2020-11-20T11:37:17.954 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.951+0000 7f46267fc700 1 -- 172.21.15.23:0/2989782691 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1472 (secure 0 0 0) 0x7f462c00de60 con 0x7f463809f280 2020-11-20T11:37:17.955 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":40,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:37:17.956 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.953+0000 7f463cf22700 1 -- 172.21.15.23:0/2989782691 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f460c056d60 msgr2=0x7f460c059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:17.957 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.953+0000 7f463cf22700 1 --2- 172.21.15.23:0/2989782691 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f460c056d60 0x7f460c059220 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f462800e010 tx=0x7f4628008040).stop 2020-11-20T11:37:17.957 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.953+0000 7f463cf22700 1 -- 172.21.15.23:0/2989782691 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f463809f280 msgr2=0x7f46380a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:17.957 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.953+0000 7f463cf22700 1 --2- 172.21.15.23:0/2989782691 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f463809f280 0x7f46380a5170 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f462c002e00 tx=0x7f462c007680).stop 2020-11-20T11:37:17.957 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.953+0000 7f463cf22700 1 -- 172.21.15.23:0/2989782691 shutdown_connections 2020-11-20T11:37:17.957 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.953+0000 7f463cf22700 1 --2- 172.21.15.23:0/2989782691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46380a7390 0x7f46380a7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:17.958 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.953+0000 7f463cf22700 1 --2- 172.21.15.23:0/2989782691 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f461c0037c0 0x7f463813cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:17.958 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.953+0000 7f463cf22700 1 --2- 172.21.15.23:0/2989782691 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f460c056d60 0x7f460c059220 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:17.958 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.953+0000 7f463cf22700 1 --2- 172.21.15.23:0/2989782691 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f463809f280 0x7f46380a5170 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:17.958 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.953+0000 7f463cf22700 1 -- 172.21.15.23:0/2989782691 >> 172.21.15.23:0/2989782691 conn(0x7f46380678e0 msgr2=0x7f46380b1360 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:17.958 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.953+0000 7f463cf22700 1 -- 172.21.15.23:0/2989782691 shutdown_connections 2020-11-20T11:37:17.959 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:17.953+0000 7f463cf22700 1 -- 172.21.15.23:0/2989782691 wait complete. 2020-11-20T11:37:20.975 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:37:21.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.095+0000 7fe98d8f0700 1 Processor -- start 2020-11-20T11:37:21.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.095+0000 7fe98d8f0700 1 -- start start 2020-11-20T11:37:21.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.096+0000 7fe98d8f0700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe9880676a0 0x7fe9880a89c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:21.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.096+0000 7fe98d8f0700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe988065e50 0x7fe9880a8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:21.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.096+0000 7fe98d8f0700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe988138360 con 0x7fe9880aad30 2020-11-20T11:37:21.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.096+0000 7fe98d8f0700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe988067d10 con 0x7fe9880676a0 2020-11-20T11:37:21.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.096+0000 7fe98d8f0700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe988066700 con 0x7fe988065e50 2020-11-20T11:37:21.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.096+0000 7fe98d0ef700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fe9880aad30 0x7fe9880ab1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57340/0 (socket says 172.21.15.23:57340) 2020-11-20T11:37:21.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.096+0000 7fe98d0ef700 1 -- 172.21.15.23:0/2275323879 learned_addr learned my addr 172.21.15.23:0/2275323879 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:37:21.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.096+0000 7fe98c8ee700 1 --2- 172.21.15.23:0/2275323879 >> v2:172.21.15.155:3300/0 conn(0x7fe9880676a0 0x7fe9880a89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:21.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.096+0000 7fe987fff700 1 --2- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe988065e50 0x7fe9880a8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:21.102 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.096+0000 7fe987fff700 1 -- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe988065e50 msgr2=0x7fe9880a8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:37:21.102 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.096+0000 7fe987fff700 1 -- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe988065e50 msgr2=0x7fe9880a8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:37:21.102 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.096+0000 7fe987fff700 1 --2- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe988065e50 0x7fe9880a8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:21.102 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.096+0000 7fe987fff700 1 --2- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe988065e50 0x7fe9880a8eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:37:21.102 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.096+0000 7fe98c8ee700 1 -- 172.21.15.23:0/2275323879 >> v2:172.21.15.155:3300/0 conn(0x7fe9880676a0 msgr2=0x7fe9880a89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:37:21.103 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.096+0000 7fe98c8ee700 1 -- 172.21.15.23:0/2275323879 >> v2:172.21.15.155:3300/0 conn(0x7fe9880676a0 msgr2=0x7fe9880a89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:37:21.103 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.096+0000 7fe98c8ee700 1 --2- 172.21.15.23:0/2275323879 >> v2:172.21.15.155:3300/0 conn(0x7fe9880676a0 0x7fe9880a89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:21.103 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.096+0000 7fe98c8ee700 1 --2- 172.21.15.23:0/2275323879 >> v2:172.21.15.155:3300/0 conn(0x7fe9880676a0 0x7fe9880a89c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:37:21.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.297+0000 7fe98c8ee700 1 --2- 172.21.15.23:0/2275323879 >> v2:172.21.15.155:3300/0 conn(0x7fe9880676a0 0x7fe9880a89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:21.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.297+0000 7fe987fff700 1 --2- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe988065e50 0x7fe9880a8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:21.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.297+0000 7fe98c8ee700 1 -- 172.21.15.23:0/2275323879 >> v2:172.21.15.155:3300/0 conn(0x7fe9880676a0 msgr2=0x7fe9880a89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:37:21.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.297+0000 7fe98c8ee700 1 -- 172.21.15.23:0/2275323879 >> v2:172.21.15.155:3300/0 conn(0x7fe9880676a0 msgr2=0x7fe9880a89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:37:21.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.297+0000 7fe98c8ee700 1 --2- 172.21.15.23:0/2275323879 >> v2:172.21.15.155:3300/0 conn(0x7fe9880676a0 0x7fe9880a89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:21.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.297+0000 7fe987fff700 1 -- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe988065e50 msgr2=0x7fe9880a8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:37:21.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.297+0000 7fe987fff700 1 -- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe988065e50 msgr2=0x7fe9880a8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:37:21.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.297+0000 7fe987fff700 1 --2- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe988065e50 0x7fe9880a8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:21.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.297+0000 7fe98c8ee700 1 --2- 172.21.15.23:0/2275323879 >> v2:172.21.15.155:3300/0 conn(0x7fe9880676a0 0x7fe9880a89c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:37:21.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.297+0000 7fe987fff700 1 --2- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe988065e50 0x7fe9880a8eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:37:21.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.698+0000 7fe98c8ee700 1 --2- 172.21.15.23:0/2275323879 >> v2:172.21.15.155:3300/0 conn(0x7fe9880676a0 0x7fe9880a89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:21.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.698+0000 7fe987fff700 1 --2- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe988065e50 0x7fe9880a8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:21.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.698+0000 7fe98c8ee700 1 -- 172.21.15.23:0/2275323879 >> v2:172.21.15.155:3300/0 conn(0x7fe9880676a0 msgr2=0x7fe9880a89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:37:21.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.698+0000 7fe98c8ee700 1 -- 172.21.15.23:0/2275323879 >> v2:172.21.15.155:3300/0 conn(0x7fe9880676a0 msgr2=0x7fe9880a89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:37:21.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.698+0000 7fe98c8ee700 1 --2- 172.21.15.23:0/2275323879 >> v2:172.21.15.155:3300/0 conn(0x7fe9880676a0 0x7fe9880a89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:21.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.698+0000 7fe987fff700 1 -- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe988065e50 msgr2=0x7fe9880a8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:37:21.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.698+0000 7fe987fff700 1 -- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe988065e50 msgr2=0x7fe9880a8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:37:21.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.698+0000 7fe98c8ee700 1 --2- 172.21.15.23:0/2275323879 >> v2:172.21.15.155:3300/0 conn(0x7fe9880676a0 0x7fe9880a89c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:37:21.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.698+0000 7fe987fff700 1 --2- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe988065e50 0x7fe9880a8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:21.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:21.698+0000 7fe987fff700 1 --2- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe988065e50 0x7fe9880a8eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:37:22.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:22.499+0000 7fe98c8ee700 1 --2- 172.21.15.23:0/2275323879 >> v2:172.21.15.155:3300/0 conn(0x7fe9880676a0 0x7fe9880a89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:22.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:22.499+0000 7fe987fff700 1 --2- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe988065e50 0x7fe9880a8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:22.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:22.499+0000 7fe98c8ee700 1 -- 172.21.15.23:0/2275323879 >> v2:172.21.15.155:3300/0 conn(0x7fe9880676a0 msgr2=0x7fe9880a89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:37:22.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:22.499+0000 7fe987fff700 1 -- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe988065e50 msgr2=0x7fe9880a8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:37:22.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:22.499+0000 7fe98c8ee700 1 -- 172.21.15.23:0/2275323879 >> v2:172.21.15.155:3300/0 conn(0x7fe9880676a0 msgr2=0x7fe9880a89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:37:22.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:22.499+0000 7fe987fff700 1 -- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe988065e50 msgr2=0x7fe9880a8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:37:22.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:22.499+0000 7fe987fff700 1 --2- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe988065e50 0x7fe9880a8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:22.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:22.499+0000 7fe98c8ee700 1 --2- 172.21.15.23:0/2275323879 >> v2:172.21.15.155:3300/0 conn(0x7fe9880676a0 0x7fe9880a89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:22.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:22.499+0000 7fe987fff700 1 --2- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe988065e50 0x7fe9880a8eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:37:22.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:22.499+0000 7fe98c8ee700 1 --2- 172.21.15.23:0/2275323879 >> v2:172.21.15.155:3300/0 conn(0x7fe9880676a0 0x7fe9880a89c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:37:24.099 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:24.096+0000 7fe9867fc700 1 -- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe988065e50 msgr2=0x7fe9880a8eb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:37:24.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:24.096+0000 7fe9867fc700 1 --2- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe988065e50 0x7fe9880a8eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:24.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:24.096+0000 7fe9867fc700 1 -- 172.21.15.23:0/2275323879 >> v2:172.21.15.155:3300/0 conn(0x7fe9880676a0 msgr2=0x7fe9880a89c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:37:24.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:24.096+0000 7fe9867fc700 1 --2- 172.21.15.23:0/2275323879 >> v2:172.21.15.155:3300/0 conn(0x7fe9880676a0 0x7fe9880a89c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:24.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:24.096+0000 7fe9867fc700 1 -- 172.21.15.23:0/2275323879 >> v1:172.21.15.23:6789/0 conn(0x7fe9880aad30 legacy=0x7fe9880ab1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:24.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:24.096+0000 7fe9867fc700 1 --2- 172.21.15.23:0/2275323879 >> v2:172.21.15.23:3300/0 conn(0x7fe970007080 0x7fe970009520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:24.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:24.096+0000 7fe9867fc700 1 -- 172.21.15.23:0/2275323879 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe988138600 con 0x7fe9700033d0 2020-11-20T11:37:24.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:24.096+0000 7fe9867fc700 1 -- 172.21.15.23:0/2275323879 --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe988066700 con 0x7fe970007080 2020-11-20T11:37:24.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:24.096+0000 7fe987fff700 1 --2- 172.21.15.23:0/2275323879 >> v2:172.21.15.23:3300/0 conn(0x7fe970007080 0x7fe970009520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:24.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:24.096+0000 7fe987fff700 1 -- 172.21.15.23:0/2275323879 >> v2:172.21.15.23:3300/0 conn(0x7fe970007080 msgr2=0x7fe970009520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:37:24.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:24.096+0000 7fe987fff700 1 -- 172.21.15.23:0/2275323879 >> v2:172.21.15.23:3300/0 conn(0x7fe970007080 msgr2=0x7fe970009520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:37:24.102 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:24.096+0000 7fe987fff700 1 --2- 172.21.15.23:0/2275323879 >> v2:172.21.15.23:3300/0 conn(0x7fe970007080 0x7fe970009520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:24.102 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:24.096+0000 7fe987fff700 1 --2- 172.21.15.23:0/2275323879 >> v2:172.21.15.23:3300/0 conn(0x7fe970007080 0x7fe970009520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:37:24.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:24.297+0000 7fe987fff700 1 --2- 172.21.15.23:0/2275323879 >> v2:172.21.15.23:3300/0 conn(0x7fe970007080 0x7fe970009520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:24.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:24.297+0000 7fe987fff700 1 -- 172.21.15.23:0/2275323879 >> v2:172.21.15.23:3300/0 conn(0x7fe970007080 msgr2=0x7fe970009520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:37:24.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:24.297+0000 7fe987fff700 1 -- 172.21.15.23:0/2275323879 >> v2:172.21.15.23:3300/0 conn(0x7fe970007080 msgr2=0x7fe970009520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:37:24.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:24.297+0000 7fe987fff700 1 --2- 172.21.15.23:0/2275323879 >> v2:172.21.15.23:3300/0 conn(0x7fe970007080 0x7fe970009520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:24.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:24.297+0000 7fe987fff700 1 --2- 172.21.15.23:0/2275323879 >> v2:172.21.15.23:3300/0 conn(0x7fe970007080 0x7fe970009520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:37:24.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:24.698+0000 7fe987fff700 1 --2- 172.21.15.23:0/2275323879 >> v2:172.21.15.23:3300/0 conn(0x7fe970007080 0x7fe970009520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:24.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:24.698+0000 7fe987fff700 1 -- 172.21.15.23:0/2275323879 >> v2:172.21.15.23:3300/0 conn(0x7fe970007080 msgr2=0x7fe970009520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:37:24.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:24.698+0000 7fe987fff700 1 -- 172.21.15.23:0/2275323879 >> v2:172.21.15.23:3300/0 conn(0x7fe970007080 msgr2=0x7fe970009520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:37:24.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:24.698+0000 7fe987fff700 1 --2- 172.21.15.23:0/2275323879 >> v2:172.21.15.23:3300/0 conn(0x7fe970007080 0x7fe970009520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:24.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:24.698+0000 7fe987fff700 1 --2- 172.21.15.23:0/2275323879 >> v2:172.21.15.23:3300/0 conn(0x7fe970007080 0x7fe970009520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:37:25.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.475+0000 7fe9877fe700 1 -- 172.21.15.23:0/2275323879 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516441626 0 0) 0x7fe988138600 con 0x7fe9700033d0 2020-11-20T11:37:25.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.475+0000 7fe9877fe700 1 -- 172.21.15.23:0/2275323879 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe96c003280 con 0x7fe9700033d0 2020-11-20T11:37:25.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.475+0000 7fe9877fe700 1 -- 172.21.15.23:0/2275323879 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3450489234 0 0) 0x7fe96c003280 con 0x7fe9700033d0 2020-11-20T11:37:25.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.476+0000 7fe9877fe700 1 -- 172.21.15.23:0/2275323879 >> v2:172.21.15.23:3300/0 conn(0x7fe970007080 msgr2=0x7fe970009520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:37:25.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.476+0000 7fe9877fe700 1 --2- 172.21.15.23:0/2275323879 >> v2:172.21.15.23:3300/0 conn(0x7fe970007080 0x7fe970009520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:25.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.476+0000 7fe9877fe700 1 -- 172.21.15.23:0/2275323879 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe97000ac90 con 0x7fe9700033d0 2020-11-20T11:37:25.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.476+0000 7fe9877fe700 1 -- 172.21.15.23:0/2275323879 <== mon.1 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7fe97c00d1c0 con 0x7fe9700033d0 2020-11-20T11:37:25.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.476+0000 7fe9877fe700 1 -- 172.21.15.23:0/2275323879 >> v1:172.21.15.155:6789/0 conn(0x7fe9700033d0 legacy=0x7fe970005850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:25.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.476+0000 7fe98d0ef700 1 -- 172.21.15.23:0/2275323879 reap_dead start 2020-11-20T11:37:25.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.476+0000 7fe9877fe700 1 --2- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe96c003770 0x7fe96c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:25.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.476+0000 7fe9877fe700 1 --2- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe96c004520 0x7fe96c0049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:25.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.476+0000 7fe9877fe700 1 --2- 172.21.15.23:0/2275323879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe96c004e90 0x7fe96c00b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:25.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.476+0000 7fe9877fe700 1 -- 172.21.15.23:0/2275323879 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe988066700 con 0x7fe96c004e90 2020-11-20T11:37:25.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.476+0000 7fe9877fe700 1 -- 172.21.15.23:0/2275323879 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe96c0053a0 con 0x7fe96c003770 2020-11-20T11:37:25.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.476+0000 7fe9877fe700 1 -- 172.21.15.23:0/2275323879 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe96c00bc90 con 0x7fe96c004520 2020-11-20T11:37:25.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.476+0000 7fe98c8ee700 1 --2- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe96c003770 0x7fe96c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:25.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.476+0000 7fe987fff700 1 --2- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe96c004520 0x7fe96c0049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:25.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.476+0000 7fe98c8ee700 1 -- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe96c004520 msgr2=0x7fe96c0049a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:37:25.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.476+0000 7fe98c8ee700 1 --2- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe96c004520 0x7fe96c0049a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:25.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.476+0000 7fe98c8ee700 1 -- 172.21.15.23:0/2275323879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe96c004e90 msgr2=0x7fe96c00b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:37:25.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.476+0000 7fe98c8ee700 1 --2- 172.21.15.23:0/2275323879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe96c004e90 0x7fe96c00b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:37:25.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.476+0000 7fe98c8ee700 1 -- 172.21.15.23:0/2275323879 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe96c003280 con 0x7fe96c003770 2020-11-20T11:37:25.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.476+0000 7fe98c8ee700 1 --2- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe96c003770 0x7fe96c003b70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fe97c0133e0 tx=0x7fe97c008040).ready entity=mon.1 client_cookie=77ab37fcb5266921 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:25.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.477+0000 7fe9877fe700 1 -- 172.21.15.23:0/2275323879 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97c00d7d0 con 0x7fe96c003770 2020-11-20T11:37:25.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.477+0000 7fe9877fe700 1 -- 172.21.15.23:0/2275323879 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe97c016460 con 0x7fe96c003770 2020-11-20T11:37:25.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.477+0000 7fe9877fe700 1 -- 172.21.15.23:0/2275323879 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97c016820 con 0x7fe96c003770 2020-11-20T11:37:25.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.477+0000 7fe98d8f0700 1 -- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe96c003770 msgr2=0x7fe96c003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:25.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.477+0000 7fe98d8f0700 1 --2- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe96c003770 0x7fe96c003b70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fe97c0133e0 tx=0x7fe97c008040).stop 2020-11-20T11:37:25.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.477+0000 7fe98d8f0700 1 -- 172.21.15.23:0/2275323879 shutdown_connections 2020-11-20T11:37:25.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.477+0000 7fe98d8f0700 1 --2- 172.21.15.23:0/2275323879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe96c004e90 0x7fe96c00b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:37:25.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.477+0000 7fe98d8f0700 1 --2- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe96c004520 0x7fe96c0049a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:25.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.477+0000 7fe98d8f0700 1 --2- 172.21.15.23:0/2275323879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe96c003770 0x7fe96c003b70 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:25.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.477+0000 7fe98d8f0700 1 -- 172.21.15.23:0/2275323879 >> 172.21.15.23:0/2275323879 conn(0x7fe9880612c0 msgr2=0x7fe9880616d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:25.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.477+0000 7fe98d8f0700 1 -- 172.21.15.23:0/2275323879 shutdown_connections 2020-11-20T11:37:25.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.477+0000 7fe98d8f0700 1 -- 172.21.15.23:0/2275323879 wait complete. 2020-11-20T11:37:25.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.479+0000 7fe98d8f0700 1 Processor -- start 2020-11-20T11:37:25.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.479+0000 7fe98d8f0700 1 -- start start 2020-11-20T11:37:25.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.480+0000 7fe98d8f0700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe96c003770 0x7fe988132930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:25.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.480+0000 7fe98d8f0700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe96c004520 0x7fe98814d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:25.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.480+0000 7fe98d8f0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe96c004e90 0x7fe98814dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:25.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.480+0000 7fe98d8f0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9880a49a0 con 0x7fe96c004e90 2020-11-20T11:37:25.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.480+0000 7fe98d8f0700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9880a8160 con 0x7fe96c003770 2020-11-20T11:37:25.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.480+0000 7fe98d8f0700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe98814a7c0 con 0x7fe96c004520 2020-11-20T11:37:25.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.480+0000 7fe98d0ef700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe96c004e90 0x7fe98814dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:25.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.480+0000 7fe98c8ee700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe96c003770 0x7fe988132930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:25.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.480+0000 7fe98d0ef700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe96c004e90 0x7fe98814dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43120/0 (socket says 172.21.15.23:43120) 2020-11-20T11:37:25.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.480+0000 7fe98d0ef700 1 -- 172.21.15.23:0/2482954664 learned_addr learned my addr 172.21.15.23:0/2482954664 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:37:25.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.480+0000 7fe987fff700 1 --2- 172.21.15.23:0/2482954664 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe96c004520 0x7fe98814d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:25.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.480+0000 7fe987fff700 1 -- 172.21.15.23:0/2482954664 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe96c004520 msgr2=0x7fe98814d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2020-11-20T11:37:25.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.480+0000 7fe987fff700 1 -- 172.21.15.23:0/2482954664 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe96c004520 msgr2=0x7fe98814d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:37:25.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.480+0000 7fe987fff700 1 --2- 172.21.15.23:0/2482954664 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe96c004520 0x7fe98814d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:25.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.480+0000 7fe987fff700 1 --2- 172.21.15.23:0/2482954664 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe96c004520 0x7fe98814d8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:37:25.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.480+0000 7fe98d0ef700 1 -- 172.21.15.23:0/2482954664 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe96c004520 msgr2=0x7fe98814d8f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:37:25.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.480+0000 7fe98d0ef700 1 --2- 172.21.15.23:0/2482954664 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe96c004520 0x7fe98814d8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:25.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.480+0000 7fe98d0ef700 1 -- 172.21.15.23:0/2482954664 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe96c003770 msgr2=0x7fe988132930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:25.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.480+0000 7fe98d0ef700 1 --2- 172.21.15.23:0/2482954664 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe96c003770 0x7fe988132930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:25.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.480+0000 7fe98d0ef700 1 -- 172.21.15.23:0/2482954664 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe98814aa60 con 0x7fe96c004e90 2020-11-20T11:37:25.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.480+0000 7fe98c8ee700 1 --2- 172.21.15.23:0/2482954664 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe96c003770 0x7fe988132930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:37:25.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.481+0000 7fe98d0ef700 1 --2- 172.21.15.23:0/2482954664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe96c004e90 0x7fe98814dde0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fe988064eb0 tx=0x7fe980007cf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:25.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.482+0000 7fe984ff9700 1 -- 172.21.15.23:0/2482954664 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe980005bc0 con 0x7fe96c004e90 2020-11-20T11:37:25.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.482+0000 7fe984ff9700 1 -- 172.21.15.23:0/2482954664 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe980005d20 con 0x7fe96c004e90 2020-11-20T11:37:25.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.482+0000 7fe98d8f0700 1 -- 172.21.15.23:0/2482954664 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe98814ad00 con 0x7fe96c004e90 2020-11-20T11:37:25.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.482+0000 7fe98d8f0700 1 -- 172.21.15.23:0/2482954664 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe98814a320 con 0x7fe96c004e90 2020-11-20T11:37:25.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.482+0000 7fe984ff9700 1 -- 172.21.15.23:0/2482954664 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe980007300 con 0x7fe96c004e90 2020-11-20T11:37:25.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.483+0000 7fe98d8f0700 1 -- 172.21.15.23:0/2482954664 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe98814aee0 con 0x7fe96c004e90 2020-11-20T11:37:25.497 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.484+0000 7fe984ff9700 1 -- 172.21.15.23:0/2482954664 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe980010070 con 0x7fe96c004e90 2020-11-20T11:37:25.497 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.484+0000 7fe984ff9700 1 --2- 172.21.15.23:0/2482954664 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe960056d60 0x7fe960059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:25.497 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.484+0000 7fe984ff9700 1 -- 172.21.15.23:0/2482954664 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fe98006f710 con 0x7fe96c004e90 2020-11-20T11:37:25.497 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.484+0000 7fe98c8ee700 1 --2- 172.21.15.23:0/2482954664 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe960056d60 0x7fe960059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:25.497 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.485+0000 7fe98c8ee700 1 --2- 172.21.15.23:0/2482954664 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe960056d60 0x7fe960059220 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fe97c006bc0 tx=0x7fe97c008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:25.498 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.490+0000 7fe984ff9700 1 -- 172.21.15.23:0/2482954664 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe98814aee0 con 0x7fe96c004e90 2020-11-20T11:37:25.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.765+0000 7fe98d8f0700 1 -- 172.21.15.23:0/2482954664 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe988066d00 con 0x7fe96c004e90 2020-11-20T11:37:25.769 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.765+0000 7fe984ff9700 1 -- 172.21.15.23:0/2482954664 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1471 (secure 0 0 0) 0x7fe980044a00 con 0x7fe96c004e90 2020-11-20T11:37:25.769 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":44,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:37:25.770 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.767+0000 7fe98d8f0700 1 -- 172.21.15.23:0/2482954664 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe960056d60 msgr2=0x7fe960059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:25.771 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.767+0000 7fe98d8f0700 1 --2- 172.21.15.23:0/2482954664 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe960056d60 0x7fe960059220 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fe97c006bc0 tx=0x7fe97c008040).stop 2020-11-20T11:37:25.771 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.767+0000 7fe98d8f0700 1 -- 172.21.15.23:0/2482954664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe96c004e90 msgr2=0x7fe98814dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:25.771 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.767+0000 7fe98d8f0700 1 --2- 172.21.15.23:0/2482954664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe96c004e90 0x7fe98814dde0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fe988064eb0 tx=0x7fe980007cf0).stop 2020-11-20T11:37:25.771 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.767+0000 7fe98d8f0700 1 -- 172.21.15.23:0/2482954664 shutdown_connections 2020-11-20T11:37:25.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.767+0000 7fe98d8f0700 1 --2- 172.21.15.23:0/2482954664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe96c004e90 0x7fe98814dde0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:25.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.767+0000 7fe98d8f0700 1 --2- 172.21.15.23:0/2482954664 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe96c003770 0x7fe988132930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:25.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.767+0000 7fe98d8f0700 1 --2- 172.21.15.23:0/2482954664 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe960056d60 0x7fe960059220 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:25.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.767+0000 7fe98d8f0700 1 --2- 172.21.15.23:0/2482954664 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe96c004520 0x7fe98814d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:25.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.767+0000 7fe98d8f0700 1 -- 172.21.15.23:0/2482954664 >> 172.21.15.23:0/2482954664 conn(0x7fe9880612c0 msgr2=0x7fe9880616a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:25.773 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.768+0000 7fe98d8f0700 1 -- 172.21.15.23:0/2482954664 shutdown_connections 2020-11-20T11:37:25.773 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:25.768+0000 7fe98d8f0700 1 -- 172.21.15.23:0/2482954664 wait complete. 2020-11-20T11:37:28.784 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:37:28.906 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:28.903+0000 7f0f588ef700 1 Processor -- start 2020-11-20T11:37:28.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:28.904+0000 7f0f588ef700 1 -- start start 2020-11-20T11:37:28.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:28.904+0000 7f0f588ef700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f540a9040 0x7f0f540aaf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:28.909 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:28.904+0000 7f0f588ef700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f540ace70 0x7f0f5413de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:28.909 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:28.904+0000 7f0f588ef700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0f540ad2f0 0x7f0f5413e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:28.909 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:28.904+0000 7f0f588ef700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f540aa8a0 con 0x7f0f540ace70 2020-11-20T11:37:28.909 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:28.904+0000 7f0f588ef700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f540ad860 con 0x7f0f540ad2f0 2020-11-20T11:37:28.910 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:28.904+0000 7f0f588ef700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f540aaa90 con 0x7f0f540a9040 2020-11-20T11:37:28.910 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:28.905+0000 7f0f5259c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f540ace70 0x7f0f5413de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:28.910 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:28.905+0000 7f0f5259c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0f540ace70 0x7f0f5413de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43128/0 (socket says 172.21.15.23:43128) 2020-11-20T11:37:28.910 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:28.905+0000 7f0f5259c700 1 -- 172.21.15.23:0/343666058 learned_addr learned my addr 172.21.15.23:0/343666058 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:37:28.911 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:28.905+0000 7f0f5259c700 1 -- 172.21.15.23:0/343666058 >> v2:172.21.15.23:3300/0 conn(0x7f0f540ace70 msgr2=0x7f0f5413de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:37:28.911 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:28.905+0000 7f0f5259c700 1 -- 172.21.15.23:0/343666058 >> v2:172.21.15.23:3300/0 conn(0x7f0f540ace70 msgr2=0x7f0f5413de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:37:28.911 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:28.905+0000 7f0f5259c700 1 --2- 172.21.15.23:0/343666058 >> v2:172.21.15.23:3300/0 conn(0x7f0f540ace70 0x7f0f5413de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:28.911 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:28.905+0000 7f0f5259c700 1 --2- 172.21.15.23:0/343666058 >> v2:172.21.15.23:3300/0 conn(0x7f0f540ace70 0x7f0f5413de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:37:28.912 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:28.905+0000 7f0f52d9d700 1 --2- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f540a9040 0x7f0f540aaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:28.912 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:28.905+0000 7f0f5359e700 1 --2- 172.21.15.23:0/343666058 >> v2:172.21.15.155:3300/0 conn(0x7f0f540ad2f0 0x7f0f5413e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:28.912 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:28.905+0000 7f0f52d9d700 1 -- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f540a9040 msgr2=0x7f0f540aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:37:28.912 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:28.905+0000 7f0f5359e700 1 -- 172.21.15.23:0/343666058 >> v2:172.21.15.155:3300/0 conn(0x7f0f540ad2f0 msgr2=0x7f0f5413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:37:28.912 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:28.905+0000 7f0f52d9d700 1 -- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f540a9040 msgr2=0x7f0f540aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:37:28.913 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:28.905+0000 7f0f5359e700 1 -- 172.21.15.23:0/343666058 >> v2:172.21.15.155:3300/0 conn(0x7f0f540ad2f0 msgr2=0x7f0f5413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:37:28.913 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:28.905+0000 7f0f52d9d700 1 --2- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f540a9040 0x7f0f540aaf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:28.913 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:28.905+0000 7f0f5359e700 1 --2- 172.21.15.23:0/343666058 >> v2:172.21.15.155:3300/0 conn(0x7f0f540ad2f0 0x7f0f5413e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:28.913 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:28.905+0000 7f0f5359e700 1 --2- 172.21.15.23:0/343666058 >> v2:172.21.15.155:3300/0 conn(0x7f0f540ad2f0 0x7f0f5413e2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:37:28.914 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:28.905+0000 7f0f52d9d700 1 --2- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f540a9040 0x7f0f540aaf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:37:29.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:29.106+0000 7f0f5259c700 1 --2- 172.21.15.23:0/343666058 >> v2:172.21.15.23:3300/0 conn(0x7f0f540ace70 0x7f0f5413de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:29.110 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:29.106+0000 7f0f5259c700 1 -- 172.21.15.23:0/343666058 >> v2:172.21.15.23:3300/0 conn(0x7f0f540ace70 msgr2=0x7f0f5413de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:37:29.110 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:29.106+0000 7f0f5259c700 1 -- 172.21.15.23:0/343666058 >> v2:172.21.15.23:3300/0 conn(0x7f0f540ace70 msgr2=0x7f0f5413de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:37:29.110 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:29.106+0000 7f0f5259c700 1 --2- 172.21.15.23:0/343666058 >> v2:172.21.15.23:3300/0 conn(0x7f0f540ace70 0x7f0f5413de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:29.111 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:29.106+0000 7f0f5259c700 1 --2- 172.21.15.23:0/343666058 >> v2:172.21.15.23:3300/0 conn(0x7f0f540ace70 0x7f0f5413de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:37:29.111 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:29.106+0000 7f0f52d9d700 1 --2- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f540a9040 0x7f0f540aaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:29.111 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:29.106+0000 7f0f5359e700 1 --2- 172.21.15.23:0/343666058 >> v2:172.21.15.155:3300/0 conn(0x7f0f540ad2f0 0x7f0f5413e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:29.111 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:29.106+0000 7f0f52d9d700 1 -- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f540a9040 msgr2=0x7f0f540aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:37:29.111 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:29.106+0000 7f0f52d9d700 1 -- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f540a9040 msgr2=0x7f0f540aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:37:29.112 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:29.106+0000 7f0f5359e700 1 -- 172.21.15.23:0/343666058 >> v2:172.21.15.155:3300/0 conn(0x7f0f540ad2f0 msgr2=0x7f0f5413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:37:29.112 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:29.106+0000 7f0f5359e700 1 -- 172.21.15.23:0/343666058 >> v2:172.21.15.155:3300/0 conn(0x7f0f540ad2f0 msgr2=0x7f0f5413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:37:29.112 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:29.106+0000 7f0f52d9d700 1 --2- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f540a9040 0x7f0f540aaf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:29.113 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:29.106+0000 7f0f5359e700 1 --2- 172.21.15.23:0/343666058 >> v2:172.21.15.155:3300/0 conn(0x7f0f540ad2f0 0x7f0f5413e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:29.113 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:29.106+0000 7f0f52d9d700 1 --2- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f540a9040 0x7f0f540aaf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:37:29.113 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:29.106+0000 7f0f5359e700 1 --2- 172.21.15.23:0/343666058 >> v2:172.21.15.155:3300/0 conn(0x7f0f540ad2f0 0x7f0f5413e2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:37:29.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:29.507+0000 7f0f5259c700 1 --2- 172.21.15.23:0/343666058 >> v2:172.21.15.23:3300/0 conn(0x7f0f540ace70 0x7f0f5413de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:29.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:29.507+0000 7f0f5359e700 1 --2- 172.21.15.23:0/343666058 >> v2:172.21.15.155:3300/0 conn(0x7f0f540ad2f0 0x7f0f5413e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:29.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:29.507+0000 7f0f52d9d700 1 --2- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f540a9040 0x7f0f540aaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:29.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:29.507+0000 7f0f5259c700 1 -- 172.21.15.23:0/343666058 >> v2:172.21.15.23:3300/0 conn(0x7f0f540ace70 msgr2=0x7f0f5413de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:37:29.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:29.507+0000 7f0f5259c700 1 -- 172.21.15.23:0/343666058 >> v2:172.21.15.23:3300/0 conn(0x7f0f540ace70 msgr2=0x7f0f5413de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:37:29.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:29.507+0000 7f0f5259c700 1 --2- 172.21.15.23:0/343666058 >> v2:172.21.15.23:3300/0 conn(0x7f0f540ace70 0x7f0f5413de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:29.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:29.507+0000 7f0f5259c700 1 --2- 172.21.15.23:0/343666058 >> v2:172.21.15.23:3300/0 conn(0x7f0f540ace70 0x7f0f5413de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:37:29.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:29.507+0000 7f0f52d9d700 1 -- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f540a9040 msgr2=0x7f0f540aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:37:29.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:29.507+0000 7f0f5359e700 1 -- 172.21.15.23:0/343666058 >> v2:172.21.15.155:3300/0 conn(0x7f0f540ad2f0 msgr2=0x7f0f5413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:37:29.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:29.507+0000 7f0f5359e700 1 -- 172.21.15.23:0/343666058 >> v2:172.21.15.155:3300/0 conn(0x7f0f540ad2f0 msgr2=0x7f0f5413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:37:29.515 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:29.507+0000 7f0f52d9d700 1 -- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f540a9040 msgr2=0x7f0f540aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:37:29.515 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:29.507+0000 7f0f52d9d700 1 --2- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f540a9040 0x7f0f540aaf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:29.515 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:29.507+0000 7f0f5359e700 1 --2- 172.21.15.23:0/343666058 >> v2:172.21.15.155:3300/0 conn(0x7f0f540ad2f0 0x7f0f5413e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:29.516 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:29.507+0000 7f0f52d9d700 1 --2- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f540a9040 0x7f0f540aaf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:37:29.516 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:29.507+0000 7f0f5359e700 1 --2- 172.21.15.23:0/343666058 >> v2:172.21.15.155:3300/0 conn(0x7f0f540ad2f0 0x7f0f5413e2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:37:30.306 INFO:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:37:30.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:30.308+0000 7f0f5259c700 1 --2- 172.21.15.23:0/343666058 >> v2:172.21.15.23:3300/0 conn(0x7f0f540ace70 0x7f0f5413de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:30.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:30.308+0000 7f0f52d9d700 1 --2- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f540a9040 0x7f0f540aaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:30.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:30.308+0000 7f0f5359e700 1 --2- 172.21.15.23:0/343666058 >> v2:172.21.15.155:3300/0 conn(0x7f0f540ad2f0 0x7f0f5413e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:30.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:30.308+0000 7f0f5259c700 1 -- 172.21.15.23:0/343666058 >> v2:172.21.15.23:3300/0 conn(0x7f0f540ace70 msgr2=0x7f0f5413de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:37:30.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:30.308+0000 7f0f5259c700 1 -- 172.21.15.23:0/343666058 >> v2:172.21.15.23:3300/0 conn(0x7f0f540ace70 msgr2=0x7f0f5413de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:37:30.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:30.308+0000 7f0f5259c700 1 --2- 172.21.15.23:0/343666058 >> v2:172.21.15.23:3300/0 conn(0x7f0f540ace70 0x7f0f5413de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:30.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:30.308+0000 7f0f5359e700 1 -- 172.21.15.23:0/343666058 >> v2:172.21.15.155:3300/0 conn(0x7f0f540ad2f0 msgr2=0x7f0f5413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:37:30.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:30.308+0000 7f0f5259c700 1 --2- 172.21.15.23:0/343666058 >> v2:172.21.15.23:3300/0 conn(0x7f0f540ace70 0x7f0f5413de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:37:30.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:30.308+0000 7f0f5359e700 1 -- 172.21.15.23:0/343666058 >> v2:172.21.15.155:3300/0 conn(0x7f0f540ad2f0 msgr2=0x7f0f5413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:37:30.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:30.308+0000 7f0f52d9d700 1 -- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f540a9040 msgr2=0x7f0f540aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:37:30.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:30.308+0000 7f0f52d9d700 1 -- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f540a9040 msgr2=0x7f0f540aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:37:30.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:30.308+0000 7f0f5359e700 1 --2- 172.21.15.23:0/343666058 >> v2:172.21.15.155:3300/0 conn(0x7f0f540ad2f0 0x7f0f5413e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:30.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:30.308+0000 7f0f52d9d700 1 --2- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f540a9040 0x7f0f540aaf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:30.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:30.308+0000 7f0f5359e700 1 --2- 172.21.15.23:0/343666058 >> v2:172.21.15.155:3300/0 conn(0x7f0f540ad2f0 0x7f0f5413e2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:37:30.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:30.308+0000 7f0f52d9d700 1 --2- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f540a9040 0x7f0f540aaf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:37:30.570 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:37:31.914 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.904+0000 7f0f50d99700 1 -- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f540a9040 msgr2=0x7f0f540aaf30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:37:31.915 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.904+0000 7f0f50d99700 1 --2- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f540a9040 0x7f0f540aaf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:31.915 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.904+0000 7f0f50d99700 1 -- 172.21.15.23:0/343666058 >> v2:172.21.15.155:3300/0 conn(0x7f0f540ad2f0 msgr2=0x7f0f5413e2f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:37:31.915 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.904+0000 7f0f50d99700 1 --2- 172.21.15.23:0/343666058 >> v2:172.21.15.155:3300/0 conn(0x7f0f540ad2f0 0x7f0f5413e2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:31.916 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.904+0000 7f0f50d99700 1 -- 172.21.15.23:0/343666058 >> v2:172.21.15.23:3300/0 conn(0x7f0f540ace70 msgr2=0x7f0f5413de00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:37:31.916 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.904+0000 7f0f50d99700 1 --2- 172.21.15.23:0/343666058 >> v2:172.21.15.23:3300/0 conn(0x7f0f540ace70 0x7f0f5413de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:31.916 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.905+0000 7f0f50d99700 1 -- 172.21.15.23:0/343666058 --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f540a9420 con 0x7f0f40007030 2020-11-20T11:37:31.916 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.905+0000 7f0f50d99700 1 -- 172.21.15.23:0/343666058 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f4000bda0 con 0x7f0f400033a0 2020-11-20T11:37:31.916 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.905+0000 7f0f51d9b700 1 -- 172.21.15.23:0/343666058 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 332426115 0 0) 0x7f0f540a9420 con 0x7f0f40007030 2020-11-20T11:37:31.917 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.905+0000 7f0f51d9b700 1 -- 172.21.15.23:0/343666058 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f34003280 con 0x7f0f40007030 2020-11-20T11:37:31.917 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.905+0000 7f0f51d9b700 1 -- 172.21.15.23:0/343666058 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2690354413 0 0) 0x7f0f34003280 con 0x7f0f40007030 2020-11-20T11:37:31.917 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.905+0000 7f0f51d9b700 1 -- 172.21.15.23:0/343666058 >> v1:172.21.15.155:6789/0 conn(0x7f0f400033a0 legacy=0x7f0f40005800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:31.917 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.905+0000 7f0f51d9b700 1 -- 172.21.15.23:0/343666058 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f4000cfa0 con 0x7f0f40007030 2020-11-20T11:37:31.917 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.905+0000 7f0f51d9b700 1 -- 172.21.15.23:0/343666058 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f0f4800d1c0 con 0x7f0f40007030 2020-11-20T11:37:31.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.905+0000 7f0f51d9b700 1 -- 172.21.15.23:0/343666058 >> v1:172.21.15.23:6789/0 conn(0x7f0f40007030 legacy=0x7f0f400094f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:31.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.905+0000 7f0f5359e700 1 -- 172.21.15.23:0/343666058 reap_dead start 2020-11-20T11:37:31.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.905+0000 7f0f51d9b700 1 --2- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f340037f0 0x7f0f34003bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:31.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.905+0000 7f0f51d9b700 1 --2- 172.21.15.23:0/343666058 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f34004520 0x7f0f340049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:31.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.905+0000 7f0f51d9b700 1 --2- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f34004e90 0x7f0f3400b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:31.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.905+0000 7f0f51d9b700 1 -- 172.21.15.23:0/343666058 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f340053c0 con 0x7f0f34004520 2020-11-20T11:37:31.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.905+0000 7f0f51d9b700 1 -- 172.21.15.23:0/343666058 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3400bc60 con 0x7f0f340037f0 2020-11-20T11:37:31.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.905+0000 7f0f51d9b700 1 -- 172.21.15.23:0/343666058 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3400be20 con 0x7f0f34004e90 2020-11-20T11:37:31.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.906+0000 7f0f5259c700 1 --2- 172.21.15.23:0/343666058 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f34004520 0x7f0f340049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:31.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.906+0000 7f0f5359e700 1 --2- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f34004e90 0x7f0f3400b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:31.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.906+0000 7f0f52d9d700 1 --2- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f340037f0 0x7f0f34003bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:31.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.906+0000 7f0f5259c700 1 -- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f34004e90 msgr2=0x7f0f3400b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:37:31.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.906+0000 7f0f5259c700 1 --2- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f34004e90 0x7f0f3400b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:31.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.906+0000 7f0f5259c700 1 -- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f340037f0 msgr2=0x7f0f34003bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:37:31.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.906+0000 7f0f5259c700 1 --2- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f340037f0 0x7f0f34003bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:31.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.906+0000 7f0f5259c700 1 -- 172.21.15.23:0/343666058 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f34003280 con 0x7f0f34004520 2020-11-20T11:37:31.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.906+0000 7f0f5259c700 1 --2- 172.21.15.23:0/343666058 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f34004520 0x7f0f340049a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f0f48006d00 tx=0x7f0f48008040).ready entity=mon.0 client_cookie=a3775b89545a4577 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:31.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.906+0000 7f0f51d9b700 1 -- 172.21.15.23:0/343666058 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4800d8a0 con 0x7f0f34004520 2020-11-20T11:37:31.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.906+0000 7f0f51d9b700 1 -- 172.21.15.23:0/343666058 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f480155d0 con 0x7f0f34004520 2020-11-20T11:37:31.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.906+0000 7f0f51d9b700 1 -- 172.21.15.23:0/343666058 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f48015950 con 0x7f0f34004520 2020-11-20T11:37:31.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.907+0000 7f0f588ef700 1 -- 172.21.15.23:0/343666058 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f34004520 msgr2=0x7f0f340049a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:31.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.907+0000 7f0f588ef700 1 --2- 172.21.15.23:0/343666058 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f34004520 0x7f0f340049a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f0f48006d00 tx=0x7f0f48008040).stop 2020-11-20T11:37:31.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.907+0000 7f0f588ef700 1 -- 172.21.15.23:0/343666058 shutdown_connections 2020-11-20T11:37:31.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.907+0000 7f0f588ef700 1 --2- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f34004e90 0x7f0f3400b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:31.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.907+0000 7f0f588ef700 1 --2- 172.21.15.23:0/343666058 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f34004520 0x7f0f340049a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:31.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.907+0000 7f0f588ef700 1 --2- 172.21.15.23:0/343666058 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f340037f0 0x7f0f34003bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:31.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.907+0000 7f0f588ef700 1 -- 172.21.15.23:0/343666058 >> 172.21.15.23:0/343666058 conn(0x7f0f540683b0 msgr2=0x7f0f540b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:31.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.907+0000 7f0f588ef700 1 -- 172.21.15.23:0/343666058 shutdown_connections 2020-11-20T11:37:31.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.907+0000 7f0f588ef700 1 -- 172.21.15.23:0/343666058 wait complete. 2020-11-20T11:37:31.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.908+0000 7f0f588ef700 1 Processor -- start 2020-11-20T11:37:31.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.908+0000 7f0f588ef700 1 -- start start 2020-11-20T11:37:31.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.909+0000 7f0f588ef700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f340037f0 0x7f0f5409f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:31.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.909+0000 7f0f588ef700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f34004520 0x7f0f5413f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:31.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.909+0000 7f0f588ef700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f34004e90 0x7f0f5409d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:31.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.909+0000 7f0f588ef700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f540a0960 con 0x7f0f34004e90 2020-11-20T11:37:31.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.909+0000 7f0f588ef700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f540a0c00 con 0x7f0f340037f0 2020-11-20T11:37:31.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.909+0000 7f0f588ef700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f540a0ea0 con 0x7f0f34004520 2020-11-20T11:37:31.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.909+0000 7f0f5359e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f34004e90 0x7f0f5409d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:31.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.909+0000 7f0f5359e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f34004e90 0x7f0f5409d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43164/0 (socket says 172.21.15.23:43164) 2020-11-20T11:37:31.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.909+0000 7f0f5359e700 1 -- 172.21.15.23:0/1513031869 learned_addr learned my addr 172.21.15.23:0/1513031869 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:37:31.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.909+0000 7f0f52d9d700 1 --2- 172.21.15.23:0/1513031869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f340037f0 0x7f0f5409f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:31.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.909+0000 7f0f5259c700 1 --2- 172.21.15.23:0/1513031869 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f34004520 0x7f0f5413f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:31.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.909+0000 7f0f5259c700 1 -- 172.21.15.23:0/1513031869 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f34004520 msgr2=0x7f0f5413f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2020-11-20T11:37:31.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.909+0000 7f0f5259c700 1 -- 172.21.15.23:0/1513031869 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f34004520 msgr2=0x7f0f5413f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:37:31.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.909+0000 7f0f5259c700 1 --2- 172.21.15.23:0/1513031869 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f34004520 0x7f0f5413f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:31.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.909+0000 7f0f5259c700 1 --2- 172.21.15.23:0/1513031869 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f34004520 0x7f0f5413f7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:37:31.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.910+0000 7f0f5359e700 1 -- 172.21.15.23:0/1513031869 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f34004520 msgr2=0x7f0f5413f7c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:37:31.928 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.910+0000 7f0f5359e700 1 --2- 172.21.15.23:0/1513031869 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f34004520 0x7f0f5413f7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:31.928 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.910+0000 7f0f5359e700 1 -- 172.21.15.23:0/1513031869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f340037f0 msgr2=0x7f0f5409f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:31.928 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.910+0000 7f0f5359e700 1 --2- 172.21.15.23:0/1513031869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f340037f0 0x7f0f5409f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:31.928 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.910+0000 7f0f5359e700 1 -- 172.21.15.23:0/1513031869 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f5409e8e0 con 0x7f0f34004e90 2020-11-20T11:37:31.928 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.910+0000 7f0f5359e700 1 --2- 172.21.15.23:0/1513031869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f34004e90 0x7f0f5409d8e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f0f4000ac90 tx=0x7f0f4c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:31.929 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.910+0000 7f0f3affd700 1 -- 172.21.15.23:0/1513031869 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4c010a00 con 0x7f0f34004e90 2020-11-20T11:37:31.929 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.910+0000 7f0f3affd700 1 -- 172.21.15.23:0/1513031869 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f4c00c030 con 0x7f0f34004e90 2020-11-20T11:37:31.929 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.910+0000 7f0f3affd700 1 -- 172.21.15.23:0/1513031869 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4c010d90 con 0x7f0f34004e90 2020-11-20T11:37:31.929 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.910+0000 7f0f588ef700 1 -- 172.21.15.23:0/1513031869 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f5413a9a0 con 0x7f0f34004e90 2020-11-20T11:37:31.930 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.910+0000 7f0f588ef700 1 -- 172.21.15.23:0/1513031869 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f5409df90 con 0x7f0f34004e90 2020-11-20T11:37:31.930 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.911+0000 7f0f588ef700 1 -- 172.21.15.23:0/1513031869 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f540a1060 con 0x7f0f34004e90 2020-11-20T11:37:31.930 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.911+0000 7f0f3affd700 1 -- 172.21.15.23:0/1513031869 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0f4c012040 con 0x7f0f34004e90 2020-11-20T11:37:31.930 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.912+0000 7f0f3affd700 1 --2- 172.21.15.23:0/1513031869 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0f28056d90 0x7f0f28059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:31.931 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.912+0000 7f0f3affd700 1 -- 172.21.15.23:0/1513031869 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f0f4c0703b0 con 0x7f0f34004e90 2020-11-20T11:37:31.931 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.912+0000 7f0f52d9d700 1 --2- 172.21.15.23:0/1513031869 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0f28056d90 0x7f0f28059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:31.932 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.912+0000 7f0f52d9d700 1 --2- 172.21.15.23:0/1513031869 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0f28056d90 0x7f0f28059250 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f0f34003770 tx=0x7f0f44006f90).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:31.932 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:31.916+0000 7f0f3affd700 1 -- 172.21.15.23:0/1513031869 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0f4c0456a0 con 0x7f0f34004e90 2020-11-20T11:37:32.174 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:32.171+0000 7f0f588ef700 1 -- 172.21.15.23:0/1513031869 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0f54057dc0 con 0x7f0f34004e90 2020-11-20T11:37:32.175 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:32.172+0000 7f0f3affd700 1 -- 172.21.15.23:0/1513031869 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1471 (secure 0 0 0) 0x7f0f4c048cc0 con 0x7f0f34004e90 2020-11-20T11:37:32.175 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":48,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:37:32.177 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:32.173+0000 7f0f588ef700 1 -- 172.21.15.23:0/1513031869 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0f28056d90 msgr2=0x7f0f28059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:32.177 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:32.173+0000 7f0f588ef700 1 --2- 172.21.15.23:0/1513031869 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0f28056d90 0x7f0f28059250 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f0f34003770 tx=0x7f0f44006f90).stop 2020-11-20T11:37:32.177 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:32.173+0000 7f0f588ef700 1 -- 172.21.15.23:0/1513031869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f34004e90 msgr2=0x7f0f5409d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:32.177 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:32.173+0000 7f0f588ef700 1 --2- 172.21.15.23:0/1513031869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f34004e90 0x7f0f5409d8e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f0f4000ac90 tx=0x7f0f4c008040).stop 2020-11-20T11:37:32.178 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:32.174+0000 7f0f588ef700 1 -- 172.21.15.23:0/1513031869 shutdown_connections 2020-11-20T11:37:32.178 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:32.174+0000 7f0f588ef700 1 --2- 172.21.15.23:0/1513031869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f34004e90 0x7f0f5409d8e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:32.178 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:32.174+0000 7f0f588ef700 1 --2- 172.21.15.23:0/1513031869 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0f340037f0 0x7f0f5409f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:32.178 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:32.174+0000 7f0f588ef700 1 --2- 172.21.15.23:0/1513031869 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0f28056d90 0x7f0f28059250 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:32.178 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:32.174+0000 7f0f588ef700 1 --2- 172.21.15.23:0/1513031869 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0f34004520 0x7f0f5413f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:32.179 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:32.174+0000 7f0f588ef700 1 -- 172.21.15.23:0/1513031869 >> 172.21.15.23:0/1513031869 conn(0x7f0f540683b0 msgr2=0x7f0f5413c520 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:32.179 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:32.174+0000 7f0f588ef700 1 -- 172.21.15.23:0/1513031869 shutdown_connections 2020-11-20T11:37:32.179 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:32.174+0000 7f0f588ef700 1 -- 172.21.15.23:0/1513031869 wait complete. 2020-11-20T11:37:35.190 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:37:35.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:35.332+0000 7f88d4d33700 1 Processor -- start 2020-11-20T11:37:35.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:35.332+0000 7f88d4d33700 1 -- start start 2020-11-20T11:37:35.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:35.332+0000 7f88d4d33700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f88d0060ca0 0x7f88d013eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:35.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:35.332+0000 7f88d4d33700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d01386f0 con 0x7f88d00608c0 2020-11-20T11:37:35.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:35.332+0000 7f88d4d33700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d013e1b0 con 0x7f88d00a1ac0 2020-11-20T11:37:35.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:35.332+0000 7f88d4d33700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88d013e450 con 0x7f88d0060ca0 2020-11-20T11:37:35.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:35.333+0000 7f88ce59c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f88d00608c0 0x7f88d0137fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57442/0 (socket says 172.21.15.23:57442) 2020-11-20T11:37:35.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:35.333+0000 7f88ce59c700 1 -- 172.21.15.23:0/1439003361 learned_addr learned my addr 172.21.15.23:0/1439003361 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:37:35.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:35.333+0000 7f88cdd9b700 1 --2- 172.21.15.23:0/1439003361 >> v2:172.21.15.155:3300/0 conn(0x7f88d0060ca0 0x7f88d013eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:35.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:35.333+0000 7f88cdd9b700 1 -- 172.21.15.23:0/1439003361 >> v2:172.21.15.155:3300/0 conn(0x7f88d0060ca0 msgr2=0x7f88d013eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:37:35.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:35.333+0000 7f88cdd9b700 1 -- 172.21.15.23:0/1439003361 >> v2:172.21.15.155:3300/0 conn(0x7f88d0060ca0 msgr2=0x7f88d013eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:37:35.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:35.333+0000 7f88cdd9b700 1 --2- 172.21.15.23:0/1439003361 >> v2:172.21.15.155:3300/0 conn(0x7f88d0060ca0 0x7f88d013eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:35.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:35.333+0000 7f88cdd9b700 1 --2- 172.21.15.23:0/1439003361 >> v2:172.21.15.155:3300/0 conn(0x7f88d0060ca0 0x7f88d013eb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:37:35.537 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:35.534+0000 7f88cdd9b700 1 --2- 172.21.15.23:0/1439003361 >> v2:172.21.15.155:3300/0 conn(0x7f88d0060ca0 0x7f88d013eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:35.537 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:35.534+0000 7f88cdd9b700 1 -- 172.21.15.23:0/1439003361 >> v2:172.21.15.155:3300/0 conn(0x7f88d0060ca0 msgr2=0x7f88d013eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:37:35.538 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:35.534+0000 7f88cdd9b700 1 -- 172.21.15.23:0/1439003361 >> v2:172.21.15.155:3300/0 conn(0x7f88d0060ca0 msgr2=0x7f88d013eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:37:35.538 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:35.534+0000 7f88cdd9b700 1 --2- 172.21.15.23:0/1439003361 >> v2:172.21.15.155:3300/0 conn(0x7f88d0060ca0 0x7f88d013eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:35.538 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:35.534+0000 7f88cdd9b700 1 --2- 172.21.15.23:0/1439003361 >> v2:172.21.15.155:3300/0 conn(0x7f88d0060ca0 0x7f88d013eb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:37:35.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:35.935+0000 7f88cdd9b700 1 --2- 172.21.15.23:0/1439003361 >> v2:172.21.15.155:3300/0 conn(0x7f88d0060ca0 0x7f88d013eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:35.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:35.935+0000 7f88cdd9b700 1 -- 172.21.15.23:0/1439003361 >> v2:172.21.15.155:3300/0 conn(0x7f88d0060ca0 msgr2=0x7f88d013eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:37:35.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:35.935+0000 7f88cdd9b700 1 -- 172.21.15.23:0/1439003361 >> v2:172.21.15.155:3300/0 conn(0x7f88d0060ca0 msgr2=0x7f88d013eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:37:35.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:35.935+0000 7f88cdd9b700 1 --2- 172.21.15.23:0/1439003361 >> v2:172.21.15.155:3300/0 conn(0x7f88d0060ca0 0x7f88d013eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:35.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:35.935+0000 7f88cdd9b700 1 --2- 172.21.15.23:0/1439003361 >> v2:172.21.15.155:3300/0 conn(0x7f88d0060ca0 0x7f88d013eb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:37:36.739 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:36.736+0000 7f88cdd9b700 1 --2- 172.21.15.23:0/1439003361 >> v2:172.21.15.155:3300/0 conn(0x7f88d0060ca0 0x7f88d013eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:36.739 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:36.736+0000 7f88cdd9b700 1 -- 172.21.15.23:0/1439003361 >> v2:172.21.15.155:3300/0 conn(0x7f88d0060ca0 msgr2=0x7f88d013eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:37:36.740 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:36.736+0000 7f88cdd9b700 1 -- 172.21.15.23:0/1439003361 >> v2:172.21.15.155:3300/0 conn(0x7f88d0060ca0 msgr2=0x7f88d013eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:37:36.740 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:36.736+0000 7f88cdd9b700 1 --2- 172.21.15.23:0/1439003361 >> v2:172.21.15.155:3300/0 conn(0x7f88d0060ca0 0x7f88d013eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:36.740 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:36.736+0000 7f88cdd9b700 1 --2- 172.21.15.23:0/1439003361 >> v2:172.21.15.155:3300/0 conn(0x7f88d0060ca0 0x7f88d013eb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:37:37.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.496+0000 7f88cd59a700 1 -- 172.21.15.23:0/1439003361 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467665456 0 0) 0x7f88d013e1b0 con 0x7f88d00a1ac0 2020-11-20T11:37:37.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.496+0000 7f88cd59a700 1 -- 172.21.15.23:0/1439003361 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88b4003280 con 0x7f88d00a1ac0 2020-11-20T11:37:37.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.496+0000 7f88cd59a700 1 -- 172.21.15.23:0/1439003361 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4230536971 0 0) 0x7f88b4003280 con 0x7f88d00a1ac0 2020-11-20T11:37:37.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.496+0000 7f88cd59a700 1 -- 172.21.15.23:0/1439003361 >> v2:172.21.15.155:3300/0 conn(0x7f88d0060ca0 msgr2=0x7f88d013eb90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:37:37.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.496+0000 7f88cd59a700 1 --2- 172.21.15.23:0/1439003361 >> v2:172.21.15.155:3300/0 conn(0x7f88d0060ca0 0x7f88d013eb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:37.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.496+0000 7f88cd59a700 1 -- 172.21.15.23:0/1439003361 >> v1:172.21.15.23:6789/0 conn(0x7f88d00608c0 legacy=0x7f88d0137fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:37.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.496+0000 7f88cd59a700 1 -- 172.21.15.23:0/1439003361 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d013e6f0 con 0x7f88d00a1ac0 2020-11-20T11:37:37.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.496+0000 7f88cd59a700 1 -- 172.21.15.23:0/1439003361 <== mon.1 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f88c80018e0 con 0x7f88d00a1ac0 2020-11-20T11:37:37.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.496+0000 7f88cd59a700 1 -- 172.21.15.23:0/1439003361 >> v1:172.21.15.155:6789/0 conn(0x7f88d00a1ac0 legacy=0x7f88d00aabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:37.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.496+0000 7f88cd59a700 1 --2- 172.21.15.23:0/1439003361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f88b4003770 0x7f88b4003b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:37.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.497+0000 7f88cd59a700 1 --2- 172.21.15.23:0/1439003361 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f88b4004520 0x7f88b40049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:37.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.497+0000 7f88cd59a700 1 --2- 172.21.15.23:0/1439003361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88b4004e90 0x7f88b400b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:37.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.497+0000 7f88cd59a700 1 -- 172.21.15.23:0/1439003361 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d013e450 con 0x7f88b4004e90 2020-11-20T11:37:37.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.497+0000 7f88cd59a700 1 -- 172.21.15.23:0/1439003361 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b40042a0 con 0x7f88b4003770 2020-11-20T11:37:37.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.497+0000 7f88cd59a700 1 -- 172.21.15.23:0/1439003361 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b400bc60 con 0x7f88b4004520 2020-11-20T11:37:37.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.497+0000 7f88ce59c700 1 --2- 172.21.15.23:0/1439003361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f88b4003770 0x7f88b4003b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:37.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.497+0000 7f88cdd9b700 1 --2- 172.21.15.23:0/1439003361 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f88b4004520 0x7f88b40049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:37.508 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.497+0000 7f88ce59c700 1 -- 172.21.15.23:0/1439003361 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f88b4004520 msgr2=0x7f88b40049a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:37:37.508 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.497+0000 7f88ce59c700 1 --2- 172.21.15.23:0/1439003361 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f88b4004520 0x7f88b40049a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:37.508 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.497+0000 7f88ce59c700 1 -- 172.21.15.23:0/1439003361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88b4004e90 msgr2=0x7f88b400b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:37:37.509 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.497+0000 7f88ce59c700 1 --2- 172.21.15.23:0/1439003361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88b4004e90 0x7f88b400b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:37:37.509 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.497+0000 7f88ce59c700 1 -- 172.21.15.23:0/1439003361 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b4003280 con 0x7f88b4003770 2020-11-20T11:37:37.509 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.497+0000 7f88ced9d700 1 -- 172.21.15.23:0/1439003361 reap_dead start 2020-11-20T11:37:37.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.497+0000 7f88ce59c700 1 --2- 172.21.15.23:0/1439003361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f88b4003770 0x7f88b4003b90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f88c0001960 tx=0x7f88c0007c80).ready entity=mon.1 client_cookie=75a86f8b33c661c2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:37.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.497+0000 7f88cd59a700 1 -- 172.21.15.23:0/1439003361 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c00072e0 con 0x7f88b4003770 2020-11-20T11:37:37.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.498+0000 7f88cd59a700 1 -- 172.21.15.23:0/1439003361 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c0007440 con 0x7f88b4003770 2020-11-20T11:37:37.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.498+0000 7f88cd59a700 1 -- 172.21.15.23:0/1439003361 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c000c880 con 0x7f88b4003770 2020-11-20T11:37:37.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.498+0000 7f88d4d33700 1 -- 172.21.15.23:0/1439003361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f88b4003770 msgr2=0x7f88b4003b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:37.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.498+0000 7f88d4d33700 1 --2- 172.21.15.23:0/1439003361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f88b4003770 0x7f88b4003b90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f88c0001960 tx=0x7f88c0007c80).stop 2020-11-20T11:37:37.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.498+0000 7f88d4d33700 1 -- 172.21.15.23:0/1439003361 shutdown_connections 2020-11-20T11:37:37.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.498+0000 7f88d4d33700 1 --2- 172.21.15.23:0/1439003361 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f88b4003770 0x7f88b4003b90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:37.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.498+0000 7f88d4d33700 1 -- 172.21.15.23:0/1439003361 >> 172.21.15.23:0/1439003361 conn(0x7f88d0066750 msgr2=0x7f88d00601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:37.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.498+0000 7f88d4d33700 1 -- 172.21.15.23:0/1439003361 shutdown_connections 2020-11-20T11:37:37.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.498+0000 7f88d4d33700 1 -- 172.21.15.23:0/1439003361 wait complete. 2020-11-20T11:37:37.516 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.500+0000 7f88d4d33700 1 Processor -- start 2020-11-20T11:37:37.516 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.500+0000 7f88d4d33700 1 -- start start 2020-11-20T11:37:37.516 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.500+0000 7f88d4d33700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88b4003770 0x7f88d013d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:37.517 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.500+0000 7f88d4d33700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f88d0152430 0x7f88d00a8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:37.517 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.500+0000 7f88d4d33700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f88d01528b0 0x7f88d00a64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:37.517 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.500+0000 7f88d4d33700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d0138bc0 con 0x7f88b4003770 2020-11-20T11:37:37.518 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.500+0000 7f88d4d33700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d00a8850 con 0x7f88d01528b0 2020-11-20T11:37:37.518 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.500+0000 7f88d4d33700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88d0151c30 con 0x7f88d0152430 2020-11-20T11:37:37.518 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.501+0000 7f88cdd9b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f88d0152430 0x7f88d00a8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:37.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.501+0000 7f88cdd9b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f88d0152430 0x7f88d00a8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:53452/0 (socket says 172.21.15.23:53452) 2020-11-20T11:37:37.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.501+0000 7f88cdd9b700 1 -- 172.21.15.23:0/1491446257 learned_addr learned my addr 172.21.15.23:0/1491446257 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:37:37.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.501+0000 7f88ced9d700 1 --2- 172.21.15.23:0/1491446257 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f88d01528b0 0x7f88d00a64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:37.520 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.501+0000 7f88cdd9b700 1 -- 172.21.15.23:0/1491446257 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f88d0152430 msgr2=0x7f88d00a8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2020-11-20T11:37:37.520 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.501+0000 7f88cdd9b700 1 -- 172.21.15.23:0/1491446257 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f88d0152430 msgr2=0x7f88d00a8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:37:37.520 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.501+0000 7f88cdd9b700 1 --2- 172.21.15.23:0/1491446257 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f88d0152430 0x7f88d00a8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:37:37.521 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.501+0000 7f88cdd9b700 1 --2- 172.21.15.23:0/1491446257 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f88d0152430 0x7f88d00a8c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:37:37.521 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.501+0000 7f88ced9d700 1 -- 172.21.15.23:0/1491446257 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f88d0152430 msgr2=0x7f88d00a8c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:37:37.521 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.501+0000 7f88ced9d700 1 --2- 172.21.15.23:0/1491446257 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f88d0152430 0x7f88d00a8c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:37.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.501+0000 7f88ced9d700 1 -- 172.21.15.23:0/1491446257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88b4003770 msgr2=0x7f88d013d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:37.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.501+0000 7f88ced9d700 1 --2- 172.21.15.23:0/1491446257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88b4003770 0x7f88d013d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:37:37.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.501+0000 7f88ced9d700 1 -- 172.21.15.23:0/1491446257 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d013a630 con 0x7f88d01528b0 2020-11-20T11:37:37.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.501+0000 7f88ced9d700 1 --2- 172.21.15.23:0/1491446257 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f88d01528b0 0x7f88d00a64f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f88d00633c0 tx=0x7f88c8003040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:37.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.502+0000 7f88be7fc700 1 -- 172.21.15.23:0/1491446257 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c8007e80 con 0x7f88d01528b0 2020-11-20T11:37:37.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.502+0000 7f88be7fc700 1 -- 172.21.15.23:0/1491446257 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c800aac0 con 0x7f88d01528b0 2020-11-20T11:37:37.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.502+0000 7f88be7fc700 1 -- 172.21.15.23:0/1491446257 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c800ae60 con 0x7f88d01528b0 2020-11-20T11:37:37.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.503+0000 7f88d4d33700 1 -- 172.21.15.23:0/1491446257 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88d0136d00 con 0x7f88d01528b0 2020-11-20T11:37:37.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.503+0000 7f88d4d33700 1 -- 172.21.15.23:0/1491446257 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88d013dce0 con 0x7f88d01528b0 2020-11-20T11:37:37.525 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.504+0000 7f88be7fc700 1 -- 172.21.15.23:0/1491446257 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f88c801d070 con 0x7f88d01528b0 2020-11-20T11:37:37.525 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.504+0000 7f88d4d33700 1 -- 172.21.15.23:0/1491446257 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88d013a470 con 0x7f88d01528b0 2020-11-20T11:37:37.525 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.504+0000 7f88be7fc700 1 --2- 172.21.15.23:0/1491446257 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f88a4056d90 0x7f88a4059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:37.525 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.505+0000 7f88be7fc700 1 -- 172.21.15.23:0/1491446257 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f88c806f730 con 0x7f88d01528b0 2020-11-20T11:37:37.526 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.505+0000 7f88ce59c700 1 --2- 172.21.15.23:0/1491446257 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f88a4056d90 0x7f88a4059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:37.526 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.506+0000 7f88ce59c700 1 --2- 172.21.15.23:0/1491446257 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f88a4056d90 0x7f88a4059250 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f88c0001960 tx=0x7f88c0015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:37.526 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.510+0000 7f88be7fc700 1 -- 172.21.15.23:0/1491446257 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f88c8044910 con 0x7f88d01528b0 2020-11-20T11:37:37.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.750+0000 7f88d4d33700 1 -- 172.21.15.23:0/1491446257 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f88d00a8990 con 0x7f88d01528b0 2020-11-20T11:37:37.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.751+0000 7f88be7fc700 1 -- 172.21.15.23:0/1491446257 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1471 (secure 0 0 0) 0x7f88c8047f30 con 0x7f88d01528b0 2020-11-20T11:37:37.754 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":52,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:37:37.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.753+0000 7f88d4d33700 1 -- 172.21.15.23:0/1491446257 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f88a4056d90 msgr2=0x7f88a4059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:37.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.753+0000 7f88d4d33700 1 --2- 172.21.15.23:0/1491446257 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f88a4056d90 0x7f88a4059250 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f88c0001960 tx=0x7f88c0015000).stop 2020-11-20T11:37:37.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.753+0000 7f88d4d33700 1 -- 172.21.15.23:0/1491446257 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f88d01528b0 msgr2=0x7f88d00a64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:37.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.753+0000 7f88d4d33700 1 --2- 172.21.15.23:0/1491446257 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f88d01528b0 0x7f88d00a64f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f88d00633c0 tx=0x7f88c8003040).stop 2020-11-20T11:37:37.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.753+0000 7f88d4d33700 1 -- 172.21.15.23:0/1491446257 shutdown_connections 2020-11-20T11:37:37.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.753+0000 7f88d4d33700 1 --2- 172.21.15.23:0/1491446257 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f88d01528b0 0x7f88d00a64f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:37.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.753+0000 7f88d4d33700 1 --2- 172.21.15.23:0/1491446257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88b4003770 0x7f88d013d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:37:37.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.753+0000 7f88d4d33700 1 --2- 172.21.15.23:0/1491446257 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f88a4056d90 0x7f88a4059250 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:37.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.753+0000 7f88d4d33700 1 --2- 172.21.15.23:0/1491446257 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f88d0152430 0x7f88d00a8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:37.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.753+0000 7f88d4d33700 1 -- 172.21.15.23:0/1491446257 >> 172.21.15.23:0/1491446257 conn(0x7f88d0066750 msgr2=0x7f88d00601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:37.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.753+0000 7f88d4d33700 1 -- 172.21.15.23:0/1491446257 shutdown_connections 2020-11-20T11:37:37.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:37.753+0000 7f88d4d33700 1 -- 172.21.15.23:0/1491446257 wait complete. 2020-11-20T11:37:40.768 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:37:40.892 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.889+0000 7f6fed31c700 1 Processor -- start 2020-11-20T11:37:40.893 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.890+0000 7f6fed31c700 1 -- start start 2020-11-20T11:37:40.895 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.890+0000 7f6fed31c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6fe80a2460 0x7f6fe8137df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:40.895 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.890+0000 7f6fed31c700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6fe8067a10 0x7f6fe8067e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:40.896 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.890+0000 7f6fed31c700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe80aaab0 con 0x7f6fe80a2080 2020-11-20T11:37:40.896 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.890+0000 7f6fed31c700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fe813aa50 con 0x7f6fe80a2460 2020-11-20T11:37:40.896 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.890+0000 7f6fed31c700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fe80aad50 con 0x7f6fe8067a10 2020-11-20T11:37:40.896 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.890+0000 7f6fe77fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6fe80a2460 0x7f6fe8137df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:40.896 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.890+0000 7f6fe7fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f6fe80a2080 0x7f6fe80aef10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57476/0 (socket says 172.21.15.23:57476) 2020-11-20T11:37:40.897 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.890+0000 7f6fe77fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6fe80a2460 0x7f6fe8137df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43228/0 (socket says 172.21.15.23:43228) 2020-11-20T11:37:40.897 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.891+0000 7f6fe7fff700 1 -- 172.21.15.23:0/383897995 learned_addr learned my addr 172.21.15.23:0/383897995 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:37:40.897 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.891+0000 7f6fecb1b700 1 --2- 172.21.15.23:0/383897995 >> v2:172.21.15.155:3300/0 conn(0x7f6fe8067a10 0x7f6fe8067e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:40.897 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.891+0000 7f6fe77fe700 1 -- 172.21.15.23:0/383897995 >> v2:172.21.15.155:3300/0 conn(0x7f6fe8067a10 msgr2=0x7f6fe8067e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:37:40.897 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.891+0000 7f6fe77fe700 1 --2- 172.21.15.23:0/383897995 >> v2:172.21.15.155:3300/0 conn(0x7f6fe8067a10 0x7f6fe8067e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:40.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.891+0000 7f6fe77fe700 1 -- 172.21.15.23:0/383897995 >> v1:172.21.15.23:6789/0 conn(0x7f6fe80a2080 legacy=0x7f6fe80aef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:40.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.891+0000 7f6fe77fe700 1 -- 172.21.15.23:0/383897995 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe813b920 con 0x7f6fe80a2460 2020-11-20T11:37:40.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.891+0000 7f6fe6ffd700 1 -- 172.21.15.23:0/383897995 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 821752456 0 0) 0x7f6fe80aaab0 con 0x7f6fe80a2080 2020-11-20T11:37:40.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.891+0000 7f6fe77fe700 1 --2- 172.21.15.23:0/383897995 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fe80a2460 0x7f6fe8137df0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f6fe8063640 tx=0x7f6fdc006e30).ready entity=mon.0 client_cookie=b47430029e29ee83 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:40.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.891+0000 7f6fe6ffd700 1 -- 172.21.15.23:0/383897995 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc0097b0 con 0x7f6fe80a2460 2020-11-20T11:37:40.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.891+0000 7f6fe6ffd700 1 -- 172.21.15.23:0/383897995 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fdc000e90 con 0x7f6fe80a2460 2020-11-20T11:37:40.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.891+0000 7f6fe6ffd700 1 -- 172.21.15.23:0/383897995 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc009b20 con 0x7f6fe80a2460 2020-11-20T11:37:40.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.892+0000 7f6fed31c700 1 -- 172.21.15.23:0/383897995 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fe80a2460 msgr2=0x7f6fe8137df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:40.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.892+0000 7f6fed31c700 1 --2- 172.21.15.23:0/383897995 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fe80a2460 0x7f6fe8137df0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f6fe8063640 tx=0x7f6fdc006e30).stop 2020-11-20T11:37:40.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.892+0000 7f6fed31c700 1 -- 172.21.15.23:0/383897995 shutdown_connections 2020-11-20T11:37:40.900 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.892+0000 7f6fed31c700 1 --2- 172.21.15.23:0/383897995 >> v2:172.21.15.155:3300/0 conn(0x7f6fe8067a10 0x7f6fe8067e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:40.900 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.892+0000 7f6fed31c700 1 --2- 172.21.15.23:0/383897995 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fe80a2460 0x7f6fe8137df0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:40.900 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.892+0000 7f6fed31c700 1 -- 172.21.15.23:0/383897995 >> 172.21.15.23:0/383897995 conn(0x7f6fe8063980 msgr2=0x7f6fe8063d90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:40.900 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.892+0000 7f6fed31c700 1 -- 172.21.15.23:0/383897995 shutdown_connections 2020-11-20T11:37:40.900 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.892+0000 7f6fed31c700 1 -- 172.21.15.23:0/383897995 wait complete. 2020-11-20T11:37:40.903 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.893+0000 7f6fed31c700 1 Processor -- start 2020-11-20T11:37:40.903 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.893+0000 7f6fed31c700 1 -- start start 2020-11-20T11:37:40.903 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.894+0000 7f6fed31c700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fe8067a10 0x7f6fe81418d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:40.903 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.894+0000 7f6fed31c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fe80a2080 0x7f6fe80a6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:40.904 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.894+0000 7f6fed31c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fe80a2460 0x7f6fe80a84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:40.904 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.894+0000 7f6fed31c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe80a9320 con 0x7f6fe80a2080 2020-11-20T11:37:40.904 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.894+0000 7f6fed31c700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe80a6d80 con 0x7f6fe80a2460 2020-11-20T11:37:40.904 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.894+0000 7f6fed31c700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe80a8de0 con 0x7f6fe8067a10 2020-11-20T11:37:40.904 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.894+0000 7f6fe77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fe80a2080 0x7f6fe80a6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:40.905 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.894+0000 7f6fe77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fe80a2080 0x7f6fe80a6890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43234/0 (socket says 172.21.15.23:43234) 2020-11-20T11:37:40.905 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.894+0000 7f6fe77fe700 1 -- 172.21.15.23:0/2961665713 learned_addr learned my addr 172.21.15.23:0/2961665713 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:37:40.905 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.894+0000 7f6fecb1b700 1 --2- 172.21.15.23:0/2961665713 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fe80a2460 0x7f6fe80a84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:40.905 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.894+0000 7f6fe7fff700 1 --2- 172.21.15.23:0/2961665713 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fe8067a10 0x7f6fe81418d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:40.905 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.894+0000 7f6fe77fe700 1 -- 172.21.15.23:0/2961665713 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fe8067a10 msgr2=0x7f6fe81418d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:40.906 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.894+0000 7f6fe77fe700 1 --2- 172.21.15.23:0/2961665713 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fe8067a10 0x7f6fe81418d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:40.906 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.894+0000 7f6fe77fe700 1 -- 172.21.15.23:0/2961665713 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fe80a2460 msgr2=0x7f6fe80a84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:40.906 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.894+0000 7f6fe77fe700 1 --2- 172.21.15.23:0/2961665713 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fe80a2460 0x7f6fe80a84d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:40.906 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.894+0000 7f6fe77fe700 1 -- 172.21.15.23:0/2961665713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe80a9080 con 0x7f6fe80a2080 2020-11-20T11:37:40.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.894+0000 7f6fe7fff700 1 --2- 172.21.15.23:0/2961665713 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fe8067a10 0x7f6fe81418d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:37:40.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.894+0000 7f6fe77fe700 1 --2- 172.21.15.23:0/2961665713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fe80a2080 0x7f6fe80a6890 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f6fdc006e70 tx=0x7f6fdc015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:40.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.894+0000 7f6fc7fff700 1 -- 172.21.15.23:0/2961665713 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc011b90 con 0x7f6fe80a2080 2020-11-20T11:37:40.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.895+0000 7f6fc7fff700 1 -- 172.21.15.23:0/2961665713 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fdc011cf0 con 0x7f6fe80a2080 2020-11-20T11:37:40.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.895+0000 7f6fed31c700 1 -- 172.21.15.23:0/2961665713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe813ed60 con 0x7f6fe80a2080 2020-11-20T11:37:40.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.895+0000 7f6fc7fff700 1 -- 172.21.15.23:0/2961665713 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc01e690 con 0x7f6fe80a2080 2020-11-20T11:37:40.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.895+0000 7f6fed31c700 1 -- 172.21.15.23:0/2961665713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe813de10 con 0x7f6fe80a2080 2020-11-20T11:37:40.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.895+0000 7f6fed31c700 1 -- 172.21.15.23:0/2961665713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe80a8c00 con 0x7f6fe80a2080 2020-11-20T11:37:40.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.896+0000 7f6fc7fff700 1 -- 172.21.15.23:0/2961665713 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6fdc025030 con 0x7f6fe80a2080 2020-11-20T11:37:40.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.896+0000 7f6fc7fff700 1 --2- 172.21.15.23:0/2961665713 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6fc0056db0 0x7f6fc0059270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:40.909 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.896+0000 7f6fc7fff700 1 -- 172.21.15.23:0/2961665713 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f6fdc071510 con 0x7f6fe80a2080 2020-11-20T11:37:40.909 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.896+0000 7f6fe7fff700 1 --2- 172.21.15.23:0/2961665713 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6fc0056db0 0x7f6fc0059270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:40.909 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.897+0000 7f6fe7fff700 1 --2- 172.21.15.23:0/2961665713 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6fc0056db0 0x7f6fc0059270 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f6fd8001360 tx=0x7f6fd8009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:40.909 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:40.901+0000 7f6fc7fff700 1 -- 172.21.15.23:0/2961665713 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6fdc046930 con 0x7f6fe80a2080 2020-11-20T11:37:41.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.143+0000 7f6fed31c700 1 -- 172.21.15.23:0/2961665713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6fe8057dc0 con 0x7f6fe80a2080 2020-11-20T11:37:41.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.143+0000 7f6fc7fff700 1 -- 172.21.15.23:0/2961665713 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1477 (secure 0 0 0) 0x7f6fdc023020 con 0x7f6fe80a2080 2020-11-20T11:37:41.147 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:37:41.149 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.145+0000 7f6fed31c700 1 -- 172.21.15.23:0/2961665713 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6fc0056db0 msgr2=0x7f6fc0059270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.149 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.145+0000 7f6fed31c700 1 --2- 172.21.15.23:0/2961665713 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6fc0056db0 0x7f6fc0059270 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f6fd8001360 tx=0x7f6fd8009040).stop 2020-11-20T11:37:41.149 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.145+0000 7f6fed31c700 1 -- 172.21.15.23:0/2961665713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fe80a2080 msgr2=0x7f6fe80a6890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.145+0000 7f6fed31c700 1 --2- 172.21.15.23:0/2961665713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fe80a2080 0x7f6fe80a6890 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f6fdc006e70 tx=0x7f6fdc015000).stop 2020-11-20T11:37:41.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.145+0000 7f6fed31c700 1 -- 172.21.15.23:0/2961665713 shutdown_connections 2020-11-20T11:37:41.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.145+0000 7f6fed31c700 1 --2- 172.21.15.23:0/2961665713 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6fe80a2460 0x7f6fe80a84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:41.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.145+0000 7f6fed31c700 1 --2- 172.21.15.23:0/2961665713 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6fc0056db0 0x7f6fc0059270 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:41.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.145+0000 7f6fed31c700 1 --2- 172.21.15.23:0/2961665713 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6fe8067a10 0x7f6fe81418d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:41.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.145+0000 7f6fed31c700 1 --2- 172.21.15.23:0/2961665713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fe80a2080 0x7f6fe80a6890 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:41.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.145+0000 7f6fed31c700 1 -- 172.21.15.23:0/2961665713 >> 172.21.15.23:0/2961665713 conn(0x7f6fe8063980 msgr2=0x7f6fe8068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:41.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.145+0000 7f6fed31c700 1 -- 172.21.15.23:0/2961665713 shutdown_connections 2020-11-20T11:37:41.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.145+0000 7f6fed31c700 1 -- 172.21.15.23:0/2961665713 wait complete. 2020-11-20T11:37:41.161 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:37:41.162 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:37:41.289 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.286+0000 7f6f065fa700 1 Processor -- start 2020-11-20T11:37:41.290 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.287+0000 7f6f065fa700 1 -- start start 2020-11-20T11:37:41.290 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.287+0000 7f6f065fa700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f000ace70 0x7f6f0013a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:41.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.287+0000 7f6f065fa700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f000ad860 con 0x7f6f000ad2f0 2020-11-20T11:37:41.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.287+0000 7f6f065fa700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f000aaa90 con 0x7f6f000a9040 2020-11-20T11:37:41.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.287+0000 7f6f065fa700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f000aa8a0 con 0x7f6f000ace70 2020-11-20T11:37:41.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.288+0000 7f6f05df9700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f6f000ad2f0 0x7f6f0013ad20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57494/0 (socket says 172.21.15.23:57494) 2020-11-20T11:37:41.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.288+0000 7f6f05df9700 1 -- 172.21.15.23:0/1111958467 learned_addr learned my addr 172.21.15.23:0/1111958467 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:37:41.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.288+0000 7f6f04df7700 1 --2- 172.21.15.23:0/1111958467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f000ace70 0x7f6f0013a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:41.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.288+0000 7f6eeffff700 1 -- 172.21.15.23:0/1111958467 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3482973390 0 0) 0x7f6f000ad860 con 0x7f6f000ad2f0 2020-11-20T11:37:41.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.288+0000 7f6eeffff700 1 -- 172.21.15.23:0/1111958467 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ee8003280 con 0x7f6f000ad2f0 2020-11-20T11:37:41.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.288+0000 7f6eeffff700 1 -- 172.21.15.23:0/1111958467 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1347866221 0 0) 0x7f6f000aaa90 con 0x7f6f000a9040 2020-11-20T11:37:41.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.288+0000 7f6eeffff700 1 -- 172.21.15.23:0/1111958467 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f000ad860 con 0x7f6f000a9040 2020-11-20T11:37:41.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.288+0000 7f6eeffff700 1 -- 172.21.15.23:0/1111958467 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4037224192 0 0) 0x7f6ee8003280 con 0x7f6f000ad2f0 2020-11-20T11:37:41.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.288+0000 7f6eeffff700 1 -- 172.21.15.23:0/1111958467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f000ace70 msgr2=0x7f6f0013a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:37:41.298 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.288+0000 7f6eeffff700 1 --2- 172.21.15.23:0/1111958467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f000ace70 0x7f6f0013a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:41.298 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.288+0000 7f6eeffff700 1 -- 172.21.15.23:0/1111958467 >> v1:172.21.15.155:6789/0 conn(0x7f6f000a9040 legacy=0x7f6f000aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.298 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.288+0000 7f6eeffff700 1 -- 172.21.15.23:0/1111958467 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f000a9420 con 0x7f6f000ad2f0 2020-11-20T11:37:41.298 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.288+0000 7f6eeffff700 1 -- 172.21.15.23:0/1111958467 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f6ef8001820 con 0x7f6f000ad2f0 2020-11-20T11:37:41.298 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.288+0000 7f6f04df7700 1 --2- 172.21.15.23:0/1111958467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f000ace70 0x7f6f0013a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:37:41.299 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.288+0000 7f6eeffff700 1 -- 172.21.15.23:0/1111958467 >> v1:172.21.15.23:6789/0 conn(0x7f6f000ad2f0 legacy=0x7f6f0013ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.299 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.288+0000 7f6eeffff700 1 --2- 172.21.15.23:0/1111958467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ee8003930 0x7f6ee8003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:41.299 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.288+0000 7f6eeffff700 1 --2- 172.21.15.23:0/1111958467 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee8004520 0x7f6ee80049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:41.299 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.288+0000 7f6eeffff700 1 --2- 172.21.15.23:0/1111958467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f000ace70 0x7f6ee8004e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:41.299 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.288+0000 7f6eeffff700 1 -- 172.21.15.23:0/1111958467 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee8005380 con 0x7f6ee8004520 2020-11-20T11:37:41.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.288+0000 7f6eeffff700 1 -- 172.21.15.23:0/1111958467 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee800b770 con 0x7f6ee8003930 2020-11-20T11:37:41.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.288+0000 7f6eeffff700 1 -- 172.21.15.23:0/1111958467 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee800b8b0 con 0x7f6f000ace70 2020-11-20T11:37:41.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.289+0000 7f6f04df7700 1 --2- 172.21.15.23:0/1111958467 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee8004520 0x7f6ee80049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:41.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.289+0000 7f6f055f8700 1 --2- 172.21.15.23:0/1111958467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ee8003930 0x7f6ee8003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:41.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.289+0000 7f6f05df9700 1 --2- 172.21.15.23:0/1111958467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f000ace70 0x7f6ee8004e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:41.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.289+0000 7f6f04df7700 1 -- 172.21.15.23:0/1111958467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f000ace70 msgr2=0x7f6ee8004e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:37:41.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.289+0000 7f6f04df7700 1 --2- 172.21.15.23:0/1111958467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f000ace70 0x7f6ee8004e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:41.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.289+0000 7f6f04df7700 1 -- 172.21.15.23:0/1111958467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ee8003930 msgr2=0x7f6ee8003d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:37:41.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.289+0000 7f6f04df7700 1 --2- 172.21.15.23:0/1111958467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ee8003930 0x7f6ee8003d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:41.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.289+0000 7f6f04df7700 1 -- 172.21.15.23:0/1111958467 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee8003280 con 0x7f6ee8004520 2020-11-20T11:37:41.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.289+0000 7f6f05df9700 1 --2- 172.21.15.23:0/1111958467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f000ace70 0x7f6ee8004e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:37:41.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.289+0000 7f6f055f8700 1 --2- 172.21.15.23:0/1111958467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ee8003930 0x7f6ee8003d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:37:41.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.289+0000 7f6f04df7700 1 --2- 172.21.15.23:0/1111958467 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee8004520 0x7f6ee80049a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f6ef000bef0 tx=0x7f6ef0006e30).ready entity=mon.0 client_cookie=e13aeba4c1a10437 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:41.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.289+0000 7f6eeffff700 1 -- 172.21.15.23:0/1111958467 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef000b5d0 con 0x7f6ee8004520 2020-11-20T11:37:41.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.289+0000 7f6eeffff700 1 -- 172.21.15.23:0/1111958467 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef000ed60 con 0x7f6ee8004520 2020-11-20T11:37:41.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.289+0000 7f6eeffff700 1 -- 172.21.15.23:0/1111958467 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef0012600 con 0x7f6ee8004520 2020-11-20T11:37:41.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.290+0000 7f6f065fa700 1 -- 172.21.15.23:0/1111958467 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee8004520 msgr2=0x7f6ee80049a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.290+0000 7f6f065fa700 1 --2- 172.21.15.23:0/1111958467 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6ee8004520 0x7f6ee80049a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f6ef000bef0 tx=0x7f6ef0006e30).stop 2020-11-20T11:37:41.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.290+0000 7f6f05df9700 1 -- 172.21.15.23:0/1111958467 reap_dead start 2020-11-20T11:37:41.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.290+0000 7f6f065fa700 1 -- 172.21.15.23:0/1111958467 shutdown_connections 2020-11-20T11:37:41.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.290+0000 7f6f065fa700 1 -- 172.21.15.23:0/1111958467 >> 172.21.15.23:0/1111958467 conn(0x7f6f000683b0 msgr2=0x7f6f000b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:41.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.290+0000 7f6f065fa700 1 -- 172.21.15.23:0/1111958467 shutdown_connections 2020-11-20T11:37:41.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.290+0000 7f6f065fa700 1 -- 172.21.15.23:0/1111958467 wait complete. 2020-11-20T11:37:41.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.291+0000 7f6f065fa700 1 Processor -- start 2020-11-20T11:37:41.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.291+0000 7f6f065fa700 1 -- start start 2020-11-20T11:37:41.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.292+0000 7f6f065fa700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f000a1210 0x7f6f000a15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:41.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.292+0000 7f6f065fa700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f0009f8e0 0x7f6f00149b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:41.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.292+0000 7f6f065fa700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f000a6c50 0x7f6f000a70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:41.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.292+0000 7f6f065fa700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0009fd60 con 0x7f6f000a6c50 2020-11-20T11:37:41.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.292+0000 7f6f065fa700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0009e270 con 0x7f6f000a1210 2020-11-20T11:37:41.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.292+0000 7f6f065fa700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0009e510 con 0x7f6f0009f8e0 2020-11-20T11:37:41.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.292+0000 7f6f05df9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f000a6c50 0x7f6f000a70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:41.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.292+0000 7f6f05df9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f000a6c50 0x7f6f000a70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43256/0 (socket says 172.21.15.23:43256) 2020-11-20T11:37:41.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.292+0000 7f6f05df9700 1 -- 172.21.15.23:0/2690772019 learned_addr learned my addr 172.21.15.23:0/2690772019 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:37:41.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.292+0000 7f6f055f8700 1 --2- 172.21.15.23:0/2690772019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f000a1210 0x7f6f000a15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:41.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.292+0000 7f6f04df7700 1 --2- 172.21.15.23:0/2690772019 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f0009f8e0 0x7f6f00149b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:41.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.292+0000 7f6f05df9700 1 -- 172.21.15.23:0/2690772019 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f0009f8e0 msgr2=0x7f6f00149b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.292+0000 7f6f05df9700 1 --2- 172.21.15.23:0/2690772019 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f0009f8e0 0x7f6f00149b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:41.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.292+0000 7f6f05df9700 1 -- 172.21.15.23:0/2690772019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f000a1210 msgr2=0x7f6f000a15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.292+0000 7f6f05df9700 1 --2- 172.21.15.23:0/2690772019 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f000a1210 0x7f6f000a15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:41.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.292+0000 7f6f05df9700 1 -- 172.21.15.23:0/2690772019 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f0009e7b0 con 0x7f6f000a6c50 2020-11-20T11:37:41.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.292+0000 7f6f05df9700 1 --2- 172.21.15.23:0/2690772019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f000a6c50 0x7f6f000a70b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f6f00063150 tx=0x7f6f000aa780).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:41.308 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.292+0000 7f6eed7fa700 1 -- 172.21.15.23:0/2690772019 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef800c880 con 0x7f6f000a6c50 2020-11-20T11:37:41.308 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.293+0000 7f6eed7fa700 1 -- 172.21.15.23:0/2690772019 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef800c9e0 con 0x7f6f000a6c50 2020-11-20T11:37:41.308 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.293+0000 7f6f065fa700 1 -- 172.21.15.23:0/2690772019 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f0013e660 con 0x7f6f000a6c50 2020-11-20T11:37:41.308 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.293+0000 7f6eed7fa700 1 -- 172.21.15.23:0/2690772019 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef800cd80 con 0x7f6f000a6c50 2020-11-20T11:37:41.308 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.293+0000 7f6f065fa700 1 -- 172.21.15.23:0/2690772019 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f0009d610 con 0x7f6f000a6c50 2020-11-20T11:37:41.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.293+0000 7f6f065fa700 1 --2- 172.21.15.23:0/2690772019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f0014d110 0x7f6f0014d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:41.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.293+0000 7f6f065fa700 1 -- 172.21.15.23:0/2690772019 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f00069c00 con 0x7f6f0014d110 2020-11-20T11:37:41.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.293+0000 7f6f055f8700 1 --2- 172.21.15.23:0/2690772019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f0014d110 0x7f6f0014d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:41.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.294+0000 7f6f055f8700 1 -- 172.21.15.23:0/2690772019 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6f00057dc0 con 0x7f6f0014d110 2020-11-20T11:37:41.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.294+0000 7f6f055f8700 1 --2- 172.21.15.23:0/2690772019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f0014d110 0x7f6f0014d4f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f6efc000c00 tx=0x7f6efc009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:41.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.294+0000 7f6eed7fa700 1 -- 172.21.15.23:0/2690772019 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6ef8014640 con 0x7f6f000a6c50 2020-11-20T11:37:41.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.294+0000 7f6eed7fa700 1 --2- 172.21.15.23:0/2690772019 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6ed8056d00 0x7f6ed80591c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:41.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.294+0000 7f6eed7fa700 1 -- 172.21.15.23:0/2690772019 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f6ef806fd10 con 0x7f6f000a6c50 2020-11-20T11:37:41.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.294+0000 7f6eed7fa700 1 -- 172.21.15.23:0/2690772019 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6efc00f960 con 0x7f6f0014d110 2020-11-20T11:37:41.313 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.294+0000 7f6eed7fa700 1 -- 172.21.15.23:0/2690772019 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f6efc00fac0 con 0x7f6f0014d110 2020-11-20T11:37:41.313 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.294+0000 7f6eed7fa700 1 -- 172.21.15.23:0/2690772019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f0014d110 msgr2=0x7f6f0014d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.313 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.294+0000 7f6eed7fa700 1 --2- 172.21.15.23:0/2690772019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f0014d110 0x7f6f0014d4f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f6efc000c00 tx=0x7f6efc009040).stop 2020-11-20T11:37:41.313 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.294+0000 7f6eed7fa700 1 -- 172.21.15.23:0/2690772019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f0014d110 msgr2=0x7f6f0014d4f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:37:41.314 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.295+0000 7f6eed7fa700 1 --2- 172.21.15.23:0/2690772019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f0014d110 0x7f6f0014d4f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:41.314 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.295+0000 7f6f04df7700 1 --2- 172.21.15.23:0/2690772019 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6ed8056d00 0x7f6ed80591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:41.314 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.295+0000 7f6f04df7700 1 --2- 172.21.15.23:0/2690772019 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6ed8056d00 0x7f6ed80591c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f6f0005f8e0 tx=0x7f6ef0019000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:41.314 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.301+0000 7f6f065fa700 1 --2- 172.21.15.23:0/2690772019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f0014d9e0 0x7f6f0013bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:41.314 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.301+0000 7f6f065fa700 1 -- 172.21.15.23:0/2690772019 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f00044710 con 0x7f6f0014d9e0 2020-11-20T11:37:41.315 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.301+0000 7f6f055f8700 1 --2- 172.21.15.23:0/2690772019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f0014d9e0 0x7f6f0013bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:41.315 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.302+0000 7f6f055f8700 1 -- 172.21.15.23:0/2690772019 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6f0005c5a0 con 0x7f6f0014d9e0 2020-11-20T11:37:41.315 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.302+0000 7f6f055f8700 1 --2- 172.21.15.23:0/2690772019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f0014d9e0 0x7f6f0013bc30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f6efc018000 tx=0x7f6efc009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:41.315 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.302+0000 7f6eed7fa700 1 -- 172.21.15.23:0/2690772019 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6efc025730 con 0x7f6f0014d9e0 2020-11-20T11:37:41.316 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.302+0000 7f6eed7fa700 1 -- 172.21.15.23:0/2690772019 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4445 (secure 0 0 0) 0x7f6efc00f750 con 0x7f6f0014d9e0 2020-11-20T11:37:41.316 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.302+0000 7f6eed7fa700 1 -- 172.21.15.23:0/2690772019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f0014d9e0 msgr2=0x7f6f0013bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.316 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.302+0000 7f6eed7fa700 1 --2- 172.21.15.23:0/2690772019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f0014d9e0 0x7f6f0013bc30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f6efc018000 tx=0x7f6efc009040).stop 2020-11-20T11:37:41.316 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.302+0000 7f6eed7fa700 1 -- 172.21.15.23:0/2690772019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f0014d9e0 msgr2=0x7f6f0013bc30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:37:41.316 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.302+0000 7f6eed7fa700 1 --2- 172.21.15.23:0/2690772019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f0014d9e0 0x7f6f0013bc30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:41.317 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.303+0000 7f6f065fa700 1 -- 172.21.15.23:0/2690772019 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6ed8056d00 msgr2=0x7f6ed80591c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.317 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.303+0000 7f6f065fa700 1 --2- 172.21.15.23:0/2690772019 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6ed8056d00 0x7f6ed80591c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f6f0005f8e0 tx=0x7f6ef0019000).stop 2020-11-20T11:37:41.317 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.303+0000 7f6f065fa700 1 -- 172.21.15.23:0/2690772019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f000a6c50 msgr2=0x7f6f000a70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.317 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.303+0000 7f6f065fa700 1 --2- 172.21.15.23:0/2690772019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f000a6c50 0x7f6f000a70b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f6f00063150 tx=0x7f6f000aa780).stop 2020-11-20T11:37:41.317 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.303+0000 7f6f05df9700 1 -- 172.21.15.23:0/2690772019 reap_dead start 2020-11-20T11:37:41.318 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.303+0000 7f6f05df9700 1 -- 172.21.15.23:0/2690772019 reap_dead start 2020-11-20T11:37:41.318 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.303+0000 7f6f065fa700 1 -- 172.21.15.23:0/2690772019 shutdown_connections 2020-11-20T11:37:41.318 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.303+0000 7f6f065fa700 1 -- 172.21.15.23:0/2690772019 >> 172.21.15.23:0/2690772019 conn(0x7f6f000683b0 msgr2=0x7f6f000aae50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:41.318 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.303+0000 7f6f065fa700 1 -- 172.21.15.23:0/2690772019 shutdown_connections 2020-11-20T11:37:41.318 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.303+0000 7f6f065fa700 1 -- 172.21.15.23:0/2690772019 wait complete. 2020-11-20T11:37:41.319 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:37:41.319 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:37:41.319 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:37:41.319 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:37:41.320 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 56, 2020-11-20T11:37:41.320 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:37:41.320 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:37:41.320 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:37:41.320 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:37:41.321 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:41.321 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 1, 2020-11-20T11:37:41.321 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:37:41.321 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:37:41.322 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:37:41.322 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:37:41.322 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:37:41.322 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:37:41.322 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:37:41.323 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:37:41.323 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:37:41.323 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:37:41.323 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:37:41.323 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:41.324 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:37:41.324 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:37:41.324 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:37:41.324 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:37:41.324 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:37:41.325 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:37:41.325 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:37:41.325 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:37:41.325 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:37:41.325 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:37:41.326 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:41.326 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.326 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:37:41.326 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:37:41.326 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:37:41.327 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:37:41.327 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:37:41.327 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:37:41.327 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:37:41.327 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:37:41.327 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:37:41.328 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:37:41.328 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:37:41.328 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:37:41.328 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:37:41.328 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:37:41.329 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:37:41.329 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:37:41.329 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:37:41.329 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:37:41.329 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:37:41.330 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:37:41.330 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:37:41.330 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:37:41.330 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:37:41.330 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:41.331 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:37:41.331 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.331 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:37:41.331 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.331 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:37:41.331 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:37:41.332 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:37:41.332 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:37:41.332 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.332 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:37:41.332 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:37:41.333 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:41.333 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.333 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.333 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:37:41.333 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:37:41.334 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:41.334 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:41.334 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:41.334 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.334 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:37:41.335 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:37:41.335 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:37:41.335 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:37:41.335 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:37:41.335 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.335 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.336 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:37:41.336 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:37:41.336 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:37:41.336 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:37:41.336 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.337 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:37:41.337 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:37:41.337 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:41.337 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.337 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.338 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:37:41.338 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:37:41.338 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:41.338 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:41.338 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:41.339 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.339 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:37:41.339 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:37:41.339 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:37:41.339 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:37:41.340 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:37:41.340 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.340 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.340 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:37:41.340 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:37:41.341 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:37:41.341 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:37:41.341 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.341 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:37:41.341 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:37:41.342 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:41.342 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.342 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.342 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:37:41.342 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:37:41.342 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:41.343 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:41.343 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:41.343 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.343 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:37:41.343 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:37:41.344 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:37:41.344 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:37:41.344 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:37:41.344 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:41.344 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:41.345 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.345 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:37:41.345 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:37:41.345 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.345 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:37:41.346 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:37:41.346 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:37:41.346 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:41.346 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:41.346 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:37:41.347 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.347 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:37:41.347 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:37:41.347 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:37:41.347 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:41.348 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:41.348 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:37:41.348 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.348 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:37:41.348 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:37:41.348 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:37:41.349 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:41.349 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:41.349 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.349 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:37:41.350 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:37:41.351 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:37:41.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.478+0000 7ff19f0a2700 1 Processor -- start 2020-11-20T11:37:41.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.478+0000 7ff19f0a2700 1 -- start start 2020-11-20T11:37:41.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.479+0000 7ff19f0a2700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff1980aaab0 0x7ff19813ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:41.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.479+0000 7ff19f0a2700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff19813aeb0 0x7ff1980a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:41.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.479+0000 7ff19f0a2700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1981388b0 con 0x7ff1980aae90 2020-11-20T11:37:41.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.479+0000 7ff19f0a2700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff19813e600 con 0x7ff1980aaab0 2020-11-20T11:37:41.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.479+0000 7ff19f0a2700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff19813e8a0 con 0x7ff19813aeb0 2020-11-20T11:37:41.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.479+0000 7ff19d63f700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff19813aeb0 0x7ff1980a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:41.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.479+0000 7ff19ce3e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff1980aaab0 0x7ff19813ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:41.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.479+0000 7ff19ce3e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff1980aaab0 0x7ff19813ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:36160/0 (socket says 172.21.15.23:36160) 2020-11-20T11:37:41.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.479+0000 7ff19d63f700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff19813aeb0 0x7ff1980a8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:53508/0 (socket says 172.21.15.23:53508) 2020-11-20T11:37:41.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.479+0000 7ff19ce3e700 1 -- 172.21.15.23:0/4232311447 learned_addr learned my addr 172.21.15.23:0/4232311447 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:37:41.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.479+0000 7ff197fff700 1 --1- 172.21.15.23:0/4232311447 >> v1:172.21.15.155:6789/0 conn(0x7ff1980aae90 0x7ff1981381a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:35488/0 (socket says 172.21.15.23:35488) 2020-11-20T11:37:41.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.479+0000 7ff19ce3e700 1 -- 172.21.15.23:0/4232311447 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff19813aeb0 msgr2=0x7ff1980a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:37:41.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.479+0000 7ff19ce3e700 1 --2- 172.21.15.23:0/4232311447 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff19813aeb0 0x7ff1980a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:41.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.479+0000 7ff19ce3e700 1 -- 172.21.15.23:0/4232311447 >> v1:172.21.15.155:6789/0 conn(0x7ff1980aae90 legacy=0x7ff1981381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.479+0000 7ff19ce3e700 1 -- 172.21.15.23:0/4232311447 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1980ab6d0 con 0x7ff1980aaab0 2020-11-20T11:37:41.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.479+0000 7ff19d63f700 1 --2- 172.21.15.23:0/4232311447 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff19813aeb0 0x7ff1980a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:37:41.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.479+0000 7ff19ce3e700 1 --2- 172.21.15.23:0/4232311447 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1980aaab0 0x7ff19813ed40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7ff18c009a10 tx=0x7ff18c006ef0).ready entity=mon.1 client_cookie=e433fd6611b62db3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:41.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.480+0000 7ff1977fe700 1 -- 172.21.15.23:0/4232311447 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff18c00b860 con 0x7ff1980aaab0 2020-11-20T11:37:41.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.480+0000 7ff1977fe700 1 -- 172.21.15.23:0/4232311447 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff18c000e90 con 0x7ff1980aaab0 2020-11-20T11:37:41.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.480+0000 7ff1977fe700 1 -- 172.21.15.23:0/4232311447 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff18c00bbd0 con 0x7ff1980aaab0 2020-11-20T11:37:41.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.480+0000 7ff19f0a2700 1 -- 172.21.15.23:0/4232311447 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1980aaab0 msgr2=0x7ff19813ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.480+0000 7ff19f0a2700 1 --2- 172.21.15.23:0/4232311447 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1980aaab0 0x7ff19813ed40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7ff18c009a10 tx=0x7ff18c006ef0).stop 2020-11-20T11:37:41.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.480+0000 7ff19f0a2700 1 -- 172.21.15.23:0/4232311447 shutdown_connections 2020-11-20T11:37:41.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.480+0000 7ff19f0a2700 1 --2- 172.21.15.23:0/4232311447 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1980aaab0 0x7ff19813ed40 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:41.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.480+0000 7ff19f0a2700 1 --2- 172.21.15.23:0/4232311447 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff19813aeb0 0x7ff1980a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:41.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.480+0000 7ff19f0a2700 1 -- 172.21.15.23:0/4232311447 >> 172.21.15.23:0/4232311447 conn(0x7ff1980678e0 msgr2=0x7ff198067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:41.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.480+0000 7ff19f0a2700 1 -- 172.21.15.23:0/4232311447 shutdown_connections 2020-11-20T11:37:41.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.480+0000 7ff19f0a2700 1 -- 172.21.15.23:0/4232311447 wait complete. 2020-11-20T11:37:41.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.481+0000 7ff19f0a2700 1 Processor -- start 2020-11-20T11:37:41.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.482+0000 7ff19f0a2700 1 -- start start 2020-11-20T11:37:41.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.482+0000 7ff19f0a2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1980aaab0 0x7ff1980af9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:41.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.482+0000 7ff19f0a2700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1980aae90 0x7ff198140960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:41.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.482+0000 7ff19f0a2700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff198141170 0x7ff1980b09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:41.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.482+0000 7ff19f0a2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1980affd0 con 0x7ff1980aaab0 2020-11-20T11:37:41.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.482+0000 7ff19f0a2700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1980b0530 con 0x7ff1980aae90 2020-11-20T11:37:41.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.482+0000 7ff19f0a2700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff19814a450 con 0x7ff198141170 2020-11-20T11:37:41.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.482+0000 7ff19ce3e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1980aaab0 0x7ff1980af9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:41.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.482+0000 7ff19ce3e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1980aaab0 0x7ff1980af9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43270/0 (socket says 172.21.15.23:43270) 2020-11-20T11:37:41.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.482+0000 7ff19ce3e700 1 -- 172.21.15.23:0/3491058594 learned_addr learned my addr 172.21.15.23:0/3491058594 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:37:41.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.482+0000 7ff197fff700 1 --2- 172.21.15.23:0/3491058594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1980aae90 0x7ff198140960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:41.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.483+0000 7ff19d63f700 1 --2- 172.21.15.23:0/3491058594 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff198141170 0x7ff1980b09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:41.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.483+0000 7ff19ce3e700 1 -- 172.21.15.23:0/3491058594 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff198141170 msgr2=0x7ff1980b09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.483+0000 7ff19ce3e700 1 --2- 172.21.15.23:0/3491058594 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff198141170 0x7ff1980b09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:41.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.483+0000 7ff19ce3e700 1 -- 172.21.15.23:0/3491058594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1980aae90 msgr2=0x7ff198140960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.483+0000 7ff19ce3e700 1 --2- 172.21.15.23:0/3491058594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1980aae90 0x7ff198140960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:41.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.483+0000 7ff19ce3e700 1 -- 172.21.15.23:0/3491058594 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff19813dc50 con 0x7ff1980aaab0 2020-11-20T11:37:41.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.483+0000 7ff197fff700 1 --2- 172.21.15.23:0/3491058594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1980aae90 0x7ff198140960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:37:41.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.483+0000 7ff19ce3e700 1 --2- 172.21.15.23:0/3491058594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1980aaab0 0x7ff1980af9f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7ff198062f90 tx=0x7ff18c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:41.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.483+0000 7ff194ff9700 1 -- 172.21.15.23:0/3491058594 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff18c011d50 con 0x7ff1980aaab0 2020-11-20T11:37:41.497 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.483+0000 7ff194ff9700 1 -- 172.21.15.23:0/3491058594 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff18c01d480 con 0x7ff1980aaab0 2020-11-20T11:37:41.497 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.483+0000 7ff19f0a2700 1 -- 172.21.15.23:0/3491058594 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1980b0ec0 con 0x7ff1980aaab0 2020-11-20T11:37:41.497 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.483+0000 7ff194ff9700 1 -- 172.21.15.23:0/3491058594 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff18c01d820 con 0x7ff1980aaab0 2020-11-20T11:37:41.497 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.483+0000 7ff19f0a2700 1 -- 172.21.15.23:0/3491058594 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff19814ab60 con 0x7ff1980aaab0 2020-11-20T11:37:41.498 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.484+0000 7ff19f0a2700 1 --2- 172.21.15.23:0/3491058594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff19813aeb0 0x7ff19814ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:41.498 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.484+0000 7ff19f0a2700 1 -- 172.21.15.23:0/3491058594 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1980b1190 con 0x7ff19813aeb0 2020-11-20T11:37:41.498 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.484+0000 7ff197fff700 1 --2- 172.21.15.23:0/3491058594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff19813aeb0 0x7ff19814ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:41.498 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.484+0000 7ff194ff9700 1 -- 172.21.15.23:0/3491058594 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff18c017070 con 0x7ff1980aaab0 2020-11-20T11:37:41.498 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.485+0000 7ff197fff700 1 -- 172.21.15.23:0/3491058594 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1980b03b0 con 0x7ff19813aeb0 2020-11-20T11:37:41.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.485+0000 7ff194ff9700 1 --2- 172.21.15.23:0/3491058594 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff170056d60 0x7ff170059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:41.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.485+0000 7ff194ff9700 1 -- 172.21.15.23:0/3491058594 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7ff18c071bc0 con 0x7ff1980aaab0 2020-11-20T11:37:41.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.485+0000 7ff197fff700 1 --2- 172.21.15.23:0/3491058594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff19813aeb0 0x7ff19814ad50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7ff198138650 tx=0x7ff184007df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:41.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.485+0000 7ff19d63f700 1 --2- 172.21.15.23:0/3491058594 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff170056d60 0x7ff170059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:41.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.485+0000 7ff194ff9700 1 -- 172.21.15.23:0/3491058594 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff184010520 con 0x7ff19813aeb0 2020-11-20T11:37:41.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.485+0000 7ff194ff9700 1 -- 172.21.15.23:0/3491058594 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7ff184010680 con 0x7ff19813aeb0 2020-11-20T11:37:41.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.485+0000 7ff194ff9700 1 -- 172.21.15.23:0/3491058594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff19813aeb0 msgr2=0x7ff19814ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.485+0000 7ff194ff9700 1 --2- 172.21.15.23:0/3491058594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff19813aeb0 0x7ff19814ad50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7ff198138650 tx=0x7ff184007df0).stop 2020-11-20T11:37:41.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.485+0000 7ff194ff9700 1 -- 172.21.15.23:0/3491058594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff19813aeb0 msgr2=0x7ff19814ad50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:37:41.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.485+0000 7ff194ff9700 1 --2- 172.21.15.23:0/3491058594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff19813aeb0 0x7ff19814ad50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:41.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.485+0000 7ff19d63f700 1 --2- 172.21.15.23:0/3491058594 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff170056d60 0x7ff170059220 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7ff190009490 tx=0x7ff190008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:41.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.494+0000 7ff19f0a2700 1 --2- 172.21.15.23:0/3491058594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1980601b0 0x7ff19814da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:41.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.494+0000 7ff19f0a2700 1 -- 172.21.15.23:0/3491058594 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff19805c7d0 con 0x7ff1980601b0 2020-11-20T11:37:41.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.494+0000 7ff197fff700 1 --2- 172.21.15.23:0/3491058594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1980601b0 0x7ff19814da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:41.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.495+0000 7ff197fff700 1 -- 172.21.15.23:0/3491058594 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff198044710 con 0x7ff1980601b0 2020-11-20T11:37:41.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.495+0000 7ff197fff700 1 --2- 172.21.15.23:0/3491058594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1980601b0 0x7ff19814da20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7ff184014000 tx=0x7ff184015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:41.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.495+0000 7ff194ff9700 1 -- 172.21.15.23:0/3491058594 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1840135e0 con 0x7ff1980601b0 2020-11-20T11:37:41.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.495+0000 7ff194ff9700 1 -- 172.21.15.23:0/3491058594 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4783 (secure 0 0 0) 0x7ff184010310 con 0x7ff1980601b0 2020-11-20T11:37:41.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.495+0000 7ff194ff9700 1 -- 172.21.15.23:0/3491058594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1980601b0 msgr2=0x7ff19814da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.495+0000 7ff194ff9700 1 --2- 172.21.15.23:0/3491058594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1980601b0 0x7ff19814da20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7ff184014000 tx=0x7ff184015000).stop 2020-11-20T11:37:41.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.495+0000 7ff194ff9700 1 -- 172.21.15.23:0/3491058594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1980601b0 msgr2=0x7ff19814da20 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:37:41.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.495+0000 7ff194ff9700 1 --2- 172.21.15.23:0/3491058594 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1980601b0 0x7ff19814da20 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:41.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.496+0000 7ff19f0a2700 1 -- 172.21.15.23:0/3491058594 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff170056d60 msgr2=0x7ff170059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.496+0000 7ff19f0a2700 1 --2- 172.21.15.23:0/3491058594 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff170056d60 0x7ff170059220 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7ff190009490 tx=0x7ff190008040).stop 2020-11-20T11:37:41.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.496+0000 7ff19f0a2700 1 -- 172.21.15.23:0/3491058594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1980aaab0 msgr2=0x7ff1980af9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.496+0000 7ff19f0a2700 1 --2- 172.21.15.23:0/3491058594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1980aaab0 0x7ff1980af9f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7ff198062f90 tx=0x7ff18c016000).stop 2020-11-20T11:37:41.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.496+0000 7ff19d63f700 1 -- 172.21.15.23:0/3491058594 reap_dead start 2020-11-20T11:37:41.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.496+0000 7ff19d63f700 1 -- 172.21.15.23:0/3491058594 reap_dead start 2020-11-20T11:37:41.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.496+0000 7ff19f0a2700 1 -- 172.21.15.23:0/3491058594 shutdown_connections 2020-11-20T11:37:41.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.496+0000 7ff19f0a2700 1 -- 172.21.15.23:0/3491058594 >> 172.21.15.23:0/3491058594 conn(0x7ff1980678e0 msgr2=0x7ff1980aba00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:41.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.496+0000 7ff19f0a2700 1 -- 172.21.15.23:0/3491058594 shutdown_connections 2020-11-20T11:37:41.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.496+0000 7ff19f0a2700 1 -- 172.21.15.23:0/3491058594 wait complete. 2020-11-20T11:37:41.507 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:37:41.507 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:37:41.507 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:37:41.508 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:37:41.508 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 56, 2020-11-20T11:37:41.508 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:37:41.508 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:37:41.508 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:37:41.509 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:37:41.509 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:41.509 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 1, 2020-11-20T11:37:41.509 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:37:41.510 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:37:41.510 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:37:41.510 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:37:41.510 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:37:41.510 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:37:41.511 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:37:41.511 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:37:41.511 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:37:41.511 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:37:41.511 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:37:41.512 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:41.512 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:37:41.512 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:37:41.512 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:37:41.512 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:37:41.513 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:37:41.513 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:37:41.513 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:37:41.513 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:37:41.514 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:37:41.514 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:37:41.514 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:41.514 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.514 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:37:41.515 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:37:41.515 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:37:41.515 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:37:41.515 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:37:41.515 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:37:41.516 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:37:41.516 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:37:41.516 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:37:41.516 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:37:41.516 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:37:41.517 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:37:41.517 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:37:41.517 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:37:41.517 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:37:41.517 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:37:41.518 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:37:41.518 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:37:41.518 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:37:41.518 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:37:41.518 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:37:41.519 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:37:41.519 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:37:41.519 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:41.519 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:37:41.520 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.520 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:37:41.520 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.520 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:37:41.520 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:37:41.521 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:37:41.521 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:37:41.521 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.521 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:37:41.521 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:37:41.522 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:41.522 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.522 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.522 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:37:41.522 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:37:41.523 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:41.523 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:41.523 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:41.523 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.523 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:37:41.524 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:37:41.524 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:37:41.524 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:37:41.524 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:37:41.524 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.525 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.525 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:37:41.525 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:37:41.525 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:37:41.526 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:37:41.526 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.526 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:37:41.526 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:37:41.526 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:41.527 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.527 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.527 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:37:41.527 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:37:41.527 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:41.528 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:41.528 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:41.528 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.528 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:37:41.528 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:37:41.529 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:37:41.529 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:37:41.529 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:37:41.529 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.529 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.530 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:37:41.530 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:37:41.530 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:37:41.530 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:37:41.530 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.531 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:37:41.531 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:37:41.531 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:41.531 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.531 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.532 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:37:41.532 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:37:41.532 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:41.532 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:41.532 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:41.533 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.533 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:37:41.533 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:37:41.533 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:37:41.533 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:37:41.534 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:37:41.534 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:41.534 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:41.534 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.534 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:37:41.535 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:37:41.535 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.535 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:37:41.535 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:37:41.535 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:37:41.536 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:41.536 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:41.536 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:37:41.536 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.536 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:37:41.537 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:37:41.537 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:37:41.537 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:41.537 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:41.537 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:37:41.538 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.538 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:37:41.538 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:37:41.538 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:37:41.538 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:41.539 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:41.539 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:37:41.539 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.539 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:37:41.539 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:37:41.540 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:37:41.540 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:41.540 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:41.540 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:37:41.540 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.541 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:37:41.541 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:37:41.541 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:37:41.541 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:41.541 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:41.542 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.542 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:37:41.542 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:37:41.543 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:37:41.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.665+0000 7f2ee3ffe700 1 Processor -- start 2020-11-20T11:37:41.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.665+0000 7f2ee3ffe700 1 -- start start 2020-11-20T11:37:41.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.666+0000 7f2ee3ffe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f2edc13aeb0 0x7f2edc0a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:41.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.666+0000 7f2ee3ffe700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2edc1388b0 con 0x7f2edc0aaab0 2020-11-20T11:37:41.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.666+0000 7f2ee3ffe700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2edc13e600 con 0x7f2edc0aae90 2020-11-20T11:37:41.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.666+0000 7f2ee3ffe700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2edc13e8a0 con 0x7f2edc13aeb0 2020-11-20T11:37:41.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.666+0000 7f2ee1d9a700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f2edc0aaab0 0x7f2edc13ed40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57532/0 (socket says 172.21.15.23:57532) 2020-11-20T11:37:41.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.666+0000 7f2ee259b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f2edc13aeb0 0x7f2edc0a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:41.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.666+0000 7f2ee1d9a700 1 -- 172.21.15.23:0/1303013612 learned_addr learned my addr 172.21.15.23:0/1303013612 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:37:41.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.667+0000 7f2ee259b700 1 -- 172.21.15.23:0/1303013612 >> v1:172.21.15.155:6789/0 conn(0x7f2edc0aae90 legacy=0x7f2edc1381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.667+0000 7f2ee259b700 1 -- 172.21.15.23:0/1303013612 >> v1:172.21.15.23:6789/0 conn(0x7f2edc0aaab0 legacy=0x7f2edc13ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.667+0000 7f2ee259b700 1 -- 172.21.15.23:0/1303013612 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2edc0ab6d0 con 0x7f2edc13aeb0 2020-11-20T11:37:41.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.667+0000 7f2ee0d98700 1 -- 172.21.15.23:0/1303013612 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1063525902 0 0) 0x7f2edc1388b0 con 0x7f2edc0aaab0 2020-11-20T11:37:41.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.667+0000 7f2ee259b700 1 --2- 172.21.15.23:0/1303013612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2edc13aeb0 0x7f2edc0a8c60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f2ed4009ce0 tx=0x7f2ed4009220).ready entity=mon.0 client_cookie=54115a8dae5bf23b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:41.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.667+0000 7f2ee0d98700 1 -- 172.21.15.23:0/1303013612 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed400bbf0 con 0x7f2edc13aeb0 2020-11-20T11:37:41.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.667+0000 7f2ee0d98700 1 -- 172.21.15.23:0/1303013612 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed400bd50 con 0x7f2edc13aeb0 2020-11-20T11:37:41.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.667+0000 7f2ee0d98700 1 -- 172.21.15.23:0/1303013612 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed4011610 con 0x7f2edc13aeb0 2020-11-20T11:37:41.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.668+0000 7f2ee3ffe700 1 -- 172.21.15.23:0/1303013612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2edc13aeb0 msgr2=0x7f2edc0a8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.668+0000 7f2ee3ffe700 1 --2- 172.21.15.23:0/1303013612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2edc13aeb0 0x7f2edc0a8c60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f2ed4009ce0 tx=0x7f2ed4009220).stop 2020-11-20T11:37:41.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.668+0000 7f2ee3ffe700 1 -- 172.21.15.23:0/1303013612 shutdown_connections 2020-11-20T11:37:41.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.668+0000 7f2ee3ffe700 1 --2- 172.21.15.23:0/1303013612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2edc13aeb0 0x7f2edc0a8c60 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:41.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.668+0000 7f2ee3ffe700 1 -- 172.21.15.23:0/1303013612 >> 172.21.15.23:0/1303013612 conn(0x7f2edc0678e0 msgr2=0x7f2edc067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:41.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.668+0000 7f2ee3ffe700 1 -- 172.21.15.23:0/1303013612 shutdown_connections 2020-11-20T11:37:41.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.668+0000 7f2ee3ffe700 1 -- 172.21.15.23:0/1303013612 wait complete. 2020-11-20T11:37:41.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.669+0000 7f2ee3ffe700 1 Processor -- start 2020-11-20T11:37:41.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.669+0000 7f2ee3ffe700 1 -- start start 2020-11-20T11:37:41.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.670+0000 7f2ee3ffe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2edc0aaab0 0x7f2edc135ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:41.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.670+0000 7f2ee3ffe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2edc0aae90 0x7f2edc0a7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:41.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.670+0000 7f2ee3ffe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2edc09efa0 0x7f2edc0a6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:41.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.670+0000 7f2ee3ffe700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc0a6120 con 0x7f2edc0aaab0 2020-11-20T11:37:41.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.670+0000 7f2ee3ffe700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc0a7740 con 0x7f2edc09efa0 2020-11-20T11:37:41.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.670+0000 7f2ee3ffe700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc0a79d0 con 0x7f2edc0aae90 2020-11-20T11:37:41.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.670+0000 7f2ee1d9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2edc0aaab0 0x7f2edc135ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:41.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.670+0000 7f2ee1d9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2edc0aaab0 0x7f2edc135ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43288/0 (socket says 172.21.15.23:43288) 2020-11-20T11:37:41.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.670+0000 7f2ee1d9a700 1 -- 172.21.15.23:0/103121683 learned_addr learned my addr 172.21.15.23:0/103121683 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:37:41.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.670+0000 7f2ee1599700 1 --2- 172.21.15.23:0/103121683 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2edc0aae90 0x7f2edc0a7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:41.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.670+0000 7f2ee259b700 1 --2- 172.21.15.23:0/103121683 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2edc09efa0 0x7f2edc0a6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:41.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.671+0000 7f2ee1d9a700 1 -- 172.21.15.23:0/103121683 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2edc0aae90 msgr2=0x7f2edc0a7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.671+0000 7f2ee1d9a700 1 --2- 172.21.15.23:0/103121683 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2edc0aae90 0x7f2edc0a7250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:41.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.671+0000 7f2ee1d9a700 1 -- 172.21.15.23:0/103121683 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2edc09efa0 msgr2=0x7f2edc0a6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.671+0000 7f2ee1d9a700 1 --2- 172.21.15.23:0/103121683 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2edc09efa0 0x7f2edc0a6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:41.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.671+0000 7f2ee1d9a700 1 -- 172.21.15.23:0/103121683 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2edc0a7c70 con 0x7f2edc0aaab0 2020-11-20T11:37:41.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.671+0000 7f2ee1d9a700 1 --2- 172.21.15.23:0/103121683 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2edc0aaab0 0x7f2edc135ff0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f2ed800c710 tx=0x7f2ed8007aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:41.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.671+0000 7f2ec9ffb700 1 -- 172.21.15.23:0/103121683 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed80072e0 con 0x7f2edc0aaab0 2020-11-20T11:37:41.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.671+0000 7f2ec9ffb700 1 -- 172.21.15.23:0/103121683 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed8007440 con 0x7f2edc0aaab0 2020-11-20T11:37:41.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.671+0000 7f2ec9ffb700 1 -- 172.21.15.23:0/103121683 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8005860 con 0x7f2edc0aaab0 2020-11-20T11:37:41.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.671+0000 7f2ee3ffe700 1 -- 172.21.15.23:0/103121683 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2edc0a6a70 con 0x7f2edc0aaab0 2020-11-20T11:37:41.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.671+0000 7f2ee3ffe700 1 -- 172.21.15.23:0/103121683 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2edc14dd70 con 0x7f2edc0aaab0 2020-11-20T11:37:41.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.672+0000 7f2ee3ffe700 1 --2- 172.21.15.23:0/103121683 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2edc13aeb0 0x7f2edc14e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:41.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.672+0000 7f2ee3ffe700 1 -- 172.21.15.23:0/103121683 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc0a62e0 con 0x7f2edc13aeb0 2020-11-20T11:37:41.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.672+0000 7f2ec9ffb700 1 -- 172.21.15.23:0/103121683 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2ed8010070 con 0x7f2edc0aaab0 2020-11-20T11:37:41.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.672+0000 7f2ee1599700 1 --2- 172.21.15.23:0/103121683 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2edc13aeb0 0x7f2edc14e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:41.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.673+0000 7f2ec9ffb700 1 --2- 172.21.15.23:0/103121683 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2eb4056d60 0x7f2eb4059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:41.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.673+0000 7f2ee1599700 1 -- 172.21.15.23:0/103121683 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2edc05c7d0 con 0x7f2edc13aeb0 2020-11-20T11:37:41.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.673+0000 7f2ec9ffb700 1 -- 172.21.15.23:0/103121683 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f2ed806f760 con 0x7f2edc0aaab0 2020-11-20T11:37:41.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.673+0000 7f2ee1599700 1 --2- 172.21.15.23:0/103121683 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2edc13aeb0 0x7f2edc14e040 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f2ecc00f820 tx=0x7f2ecc007bd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:41.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.673+0000 7f2ee259b700 1 --2- 172.21.15.23:0/103121683 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2eb4056d60 0x7f2eb4059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:41.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.673+0000 7f2ec9ffb700 1 -- 172.21.15.23:0/103121683 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc003040 con 0x7f2edc13aeb0 2020-11-20T11:37:41.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.673+0000 7f2ec9ffb700 1 -- 172.21.15.23:0/103121683 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f2ecc007030 con 0x7f2edc13aeb0 2020-11-20T11:37:41.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.673+0000 7f2ee259b700 1 --2- 172.21.15.23:0/103121683 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2eb4056d60 0x7f2eb4059220 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f2edc0af230 tx=0x7f2ed4016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:41.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.673+0000 7f2ec9ffb700 1 -- 172.21.15.23:0/103121683 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2edc13aeb0 msgr2=0x7f2edc14e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.673+0000 7f2ec9ffb700 1 --2- 172.21.15.23:0/103121683 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2edc13aeb0 0x7f2edc14e040 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f2ecc00f820 tx=0x7f2ecc007bd0).stop 2020-11-20T11:37:41.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.673+0000 7f2ec9ffb700 1 -- 172.21.15.23:0/103121683 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2edc13aeb0 msgr2=0x7f2edc14e040 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:37:41.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.673+0000 7f2ec9ffb700 1 --2- 172.21.15.23:0/103121683 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2edc13aeb0 0x7f2edc14e040 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:41.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.681+0000 7f2ee3ffe700 1 --2- 172.21.15.23:0/103121683 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2edc0601b0 0x7f2edc14e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:41.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.681+0000 7f2ee3ffe700 1 -- 172.21.15.23:0/103121683 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc0a6420 con 0x7f2edc0601b0 2020-11-20T11:37:41.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.682+0000 7f2ee1599700 1 --2- 172.21.15.23:0/103121683 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2edc0601b0 0x7f2edc14e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:41.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.682+0000 7f2ee1599700 1 -- 172.21.15.23:0/103121683 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2edc0a83d0 con 0x7f2edc0601b0 2020-11-20T11:37:41.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.682+0000 7f2ee1599700 1 --2- 172.21.15.23:0/103121683 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2edc0601b0 0x7f2edc14e530 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f2ecc00f820 tx=0x7f2ecc018000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:41.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.682+0000 7f2ec9ffb700 1 -- 172.21.15.23:0/103121683 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc006b10 con 0x7f2edc0601b0 2020-11-20T11:37:41.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.682+0000 7f2ec9ffb700 1 -- 172.21.15.23:0/103121683 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4273 (secure 0 0 0) 0x7f2ecc00d490 con 0x7f2edc0601b0 2020-11-20T11:37:41.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.682+0000 7f2ec9ffb700 1 -- 172.21.15.23:0/103121683 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2edc0601b0 msgr2=0x7f2edc14e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.682+0000 7f2ec9ffb700 1 --2- 172.21.15.23:0/103121683 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2edc0601b0 0x7f2edc14e530 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f2ecc00f820 tx=0x7f2ecc018000).stop 2020-11-20T11:37:41.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.682+0000 7f2ec9ffb700 1 -- 172.21.15.23:0/103121683 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2edc0601b0 msgr2=0x7f2edc14e530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:37:41.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.682+0000 7f2ec9ffb700 1 --2- 172.21.15.23:0/103121683 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2edc0601b0 0x7f2edc14e530 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:41.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.683+0000 7f2ee3ffe700 1 -- 172.21.15.23:0/103121683 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2eb4056d60 msgr2=0x7f2eb4059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.683+0000 7f2ee3ffe700 1 --2- 172.21.15.23:0/103121683 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2eb4056d60 0x7f2eb4059220 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f2edc0af230 tx=0x7f2ed4016000).stop 2020-11-20T11:37:41.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.683+0000 7f2ee3ffe700 1 -- 172.21.15.23:0/103121683 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2edc0aaab0 msgr2=0x7f2edc135ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.683+0000 7f2ee3ffe700 1 --2- 172.21.15.23:0/103121683 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2edc0aaab0 0x7f2edc135ff0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f2ed800c710 tx=0x7f2ed8007aa0).stop 2020-11-20T11:37:41.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.683+0000 7f2ee259b700 1 -- 172.21.15.23:0/103121683 reap_dead start 2020-11-20T11:37:41.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.683+0000 7f2ee259b700 1 -- 172.21.15.23:0/103121683 reap_dead start 2020-11-20T11:37:41.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.683+0000 7f2ee3ffe700 1 -- 172.21.15.23:0/103121683 shutdown_connections 2020-11-20T11:37:41.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.683+0000 7f2ee3ffe700 1 -- 172.21.15.23:0/103121683 >> 172.21.15.23:0/103121683 conn(0x7f2edc0678e0 msgr2=0x7f2edc0ac4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:41.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.683+0000 7f2ee3ffe700 1 -- 172.21.15.23:0/103121683 shutdown_connections 2020-11-20T11:37:41.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.683+0000 7f2ee3ffe700 1 -- 172.21.15.23:0/103121683 wait complete. 2020-11-20T11:37:41.692 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:37:41.693 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:37:41.693 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:37:41.693 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:37:41.693 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 56, 2020-11-20T11:37:41.693 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:37:41.694 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:37:41.694 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:37:41.694 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:37:41.694 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:41.694 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 2, 2020-11-20T11:37:41.695 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:37:41.695 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:37:41.695 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:37:41.695 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:37:41.695 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:37:41.696 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:37:41.696 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:37:41.696 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:37:41.696 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:37:41.696 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:37:41.697 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:37:41.697 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:41.697 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:37:41.697 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:37:41.697 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:37:41.698 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:37:41.698 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:37:41.698 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:37:41.698 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:37:41.698 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:37:41.699 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:37:41.699 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:37:41.699 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:41.699 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.699 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:37:41.700 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:37:41.700 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:37:41.700 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:37:41.700 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:37:41.700 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:37:41.701 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:37:41.701 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:37:41.701 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:37:41.701 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:37:41.702 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:37:41.702 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:37:41.702 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:37:41.702 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:37:41.702 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:37:41.703 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:37:41.703 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:37:41.703 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:37:41.703 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:37:41.703 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:37:41.704 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:37:41.704 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:37:41.704 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:37:41.704 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:41.704 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:37:41.705 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.705 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:37:41.705 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.705 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:37:41.705 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:37:41.706 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:37:41.706 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:37:41.706 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.706 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:37:41.706 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:37:41.707 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:41.707 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.707 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.707 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:37:41.707 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:37:41.707 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:41.708 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:41.708 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:41.708 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.708 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:37:41.708 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:37:41.709 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:37:41.709 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:37:41.709 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:37:41.709 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.709 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.710 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:37:41.710 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:37:41.710 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:37:41.710 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:37:41.710 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.711 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:37:41.711 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:37:41.711 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:41.711 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.712 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.712 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:37:41.712 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:37:41.712 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:41.712 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:41.712 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:41.713 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.713 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:37:41.713 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:37:41.713 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:37:41.714 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:37:41.714 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:37:41.714 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.714 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.714 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:37:41.715 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:37:41.715 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:37:41.715 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:37:41.715 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.715 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:37:41.716 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:37:41.716 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:41.716 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.716 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.716 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:37:41.716 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:37:41.717 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:41.717 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:41.717 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:41.717 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.718 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:37:41.718 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:37:41.718 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:37:41.718 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:37:41.718 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:37:41.719 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:41.719 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:41.719 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.719 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:37:41.719 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:37:41.719 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.720 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:37:41.720 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:37:41.720 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:37:41.720 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:41.721 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:41.721 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:37:41.721 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:41.721 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:37:41.721 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:37:41.722 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:37:41.722 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:41.722 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:41.722 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:41.722 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:37:41.723 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:37:41.724 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2020-11-20T11:37:41.724 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2020-11-20T11:37:41.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.867+0000 7f796e0d2700 1 Processor -- start 2020-11-20T11:37:41.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.867+0000 7f796e0d2700 1 -- start start 2020-11-20T11:37:41.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.867+0000 7f796e0d2700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7968066210 0x7f796813adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:41.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.867+0000 7f796e0d2700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f79680665f0 0x7f79680ab2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:41.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.868+0000 7f796e0d2700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f796813bd10 con 0x7f79680a8ff0 2020-11-20T11:37:41.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.868+0000 7f796e0d2700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79680aab60 con 0x7f79680665f0 2020-11-20T11:37:41.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.868+0000 7f796e0d2700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79680aae00 con 0x7f7968066210 2020-11-20T11:37:41.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.868+0000 7f796d8d1700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f79680a8ff0 0x7f7968137f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57554/0 (socket says 172.21.15.23:57554) 2020-11-20T11:37:41.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.868+0000 7f796d8d1700 1 -- 172.21.15.23:0/1734675183 learned_addr learned my addr 172.21.15.23:0/1734675183 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:37:41.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.868+0000 7f796c8cf700 1 --2- 172.21.15.23:0/1734675183 >> v2:172.21.15.155:3300/0 conn(0x7f79680665f0 0x7f79680ab2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:41.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.868+0000 7f796d0d0700 1 --2- 172.21.15.23:0/1734675183 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7968066210 0x7f796813adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:41.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.868+0000 7f7957fff700 1 -- 172.21.15.23:0/1734675183 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1850498451 0 0) 0x7f796813bd10 con 0x7f79680a8ff0 2020-11-20T11:37:41.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.868+0000 7f7957fff700 1 -- 172.21.15.23:0/1734675183 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7950003280 con 0x7f79680a8ff0 2020-11-20T11:37:41.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.868+0000 7f7957fff700 1 -- 172.21.15.23:0/1734675183 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1413124844 0 0) 0x7f7950003280 con 0x7f79680a8ff0 2020-11-20T11:37:41.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.868+0000 7f796c8cf700 1 -- 172.21.15.23:0/1734675183 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7968066210 msgr2=0x7f796813adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:37:41.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.868+0000 7f796c8cf700 1 --2- 172.21.15.23:0/1734675183 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7968066210 0x7f796813adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:41.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.868+0000 7f796c8cf700 1 -- 172.21.15.23:0/1734675183 >> v1:172.21.15.23:6789/0 conn(0x7f79680a8ff0 legacy=0x7f7968137f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.868+0000 7f796c8cf700 1 -- 172.21.15.23:0/1734675183 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79680aa930 con 0x7f79680665f0 2020-11-20T11:37:41.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.868+0000 7f796d0d0700 1 --2- 172.21.15.23:0/1734675183 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7968066210 0x7f796813adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:37:41.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.868+0000 7f796c8cf700 1 --2- 172.21.15.23:0/1734675183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79680665f0 0x7f79680ab2a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f7958006fd0 tx=0x7f7958006e30).ready entity=mon.1 client_cookie=26603c3fd6a69384 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:41.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.869+0000 7f7957fff700 1 -- 172.21.15.23:0/1734675183 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79600018a0 con 0x7f79680665f0 2020-11-20T11:37:41.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.869+0000 7f7957fff700 1 -- 172.21.15.23:0/1734675183 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7958000e90 con 0x7f79680665f0 2020-11-20T11:37:41.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.869+0000 7f7957fff700 1 -- 172.21.15.23:0/1734675183 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7958011610 con 0x7f79680665f0 2020-11-20T11:37:41.882 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.869+0000 7f796e0d2700 1 -- 172.21.15.23:0/1734675183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79680665f0 msgr2=0x7f79680ab2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.883 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.869+0000 7f796e0d2700 1 --2- 172.21.15.23:0/1734675183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79680665f0 0x7f79680ab2a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f7958006fd0 tx=0x7f7958006e30).stop 2020-11-20T11:37:41.884 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.869+0000 7f796e0d2700 1 -- 172.21.15.23:0/1734675183 shutdown_connections 2020-11-20T11:37:41.885 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.869+0000 7f796e0d2700 1 --2- 172.21.15.23:0/1734675183 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7968066210 0x7f796813adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:41.885 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.869+0000 7f796e0d2700 1 --2- 172.21.15.23:0/1734675183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79680665f0 0x7f79680ab2a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:41.886 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.869+0000 7f796e0d2700 1 -- 172.21.15.23:0/1734675183 >> 172.21.15.23:0/1734675183 conn(0x7f79680612c0 msgr2=0x7f79680616d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:41.887 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.869+0000 7f796e0d2700 1 -- 172.21.15.23:0/1734675183 shutdown_connections 2020-11-20T11:37:41.888 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.869+0000 7f796e0d2700 1 -- 172.21.15.23:0/1734675183 wait complete. 2020-11-20T11:37:41.890 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.870+0000 7f796e0d2700 1 Processor -- start 2020-11-20T11:37:41.891 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.870+0000 7f796e0d2700 1 -- start start 2020-11-20T11:37:41.896 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.871+0000 7f796e0d2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7968066210 0x7f7968141c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:41.897 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.871+0000 7f796e0d2700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79680665f0 0x7f79680a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:41.897 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.871+0000 7f796e0d2700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79680a8ff0 0x7f79681361a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:41.897 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.871+0000 7f796e0d2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79680a4470 con 0x7f7968066210 2020-11-20T11:37:41.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.871+0000 7f796e0d2700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79680a2de0 con 0x7f79680a8ff0 2020-11-20T11:37:41.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.871+0000 7f796e0d2700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79680a3080 con 0x7f79680665f0 2020-11-20T11:37:41.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.871+0000 7f796d0d0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7968066210 0x7f7968141c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:41.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.871+0000 7f796d0d0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7968066210 0x7f7968141c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43306/0 (socket says 172.21.15.23:43306) 2020-11-20T11:37:41.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.871+0000 7f796d0d0700 1 -- 172.21.15.23:0/2240324917 learned_addr learned my addr 172.21.15.23:0/2240324917 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:37:41.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.871+0000 7f796c8cf700 1 --2- 172.21.15.23:0/2240324917 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79680665f0 0x7f79680a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:41.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.871+0000 7f796d8d1700 1 --2- 172.21.15.23:0/2240324917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79680a8ff0 0x7f79681361a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:41.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.871+0000 7f796d0d0700 1 -- 172.21.15.23:0/2240324917 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79680665f0 msgr2=0x7f79680a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.871+0000 7f796d0d0700 1 --2- 172.21.15.23:0/2240324917 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79680665f0 0x7f79680a3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:41.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.871+0000 7f796d0d0700 1 -- 172.21.15.23:0/2240324917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79680a8ff0 msgr2=0x7f79681361a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:41.900 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.871+0000 7f796d0d0700 1 --2- 172.21.15.23:0/2240324917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79680a8ff0 0x7f79681361a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:41.900 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.871+0000 7f796d0d0700 1 -- 172.21.15.23:0/2240324917 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79680a3320 con 0x7f7968066210 2020-11-20T11:37:41.900 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.872+0000 7f796d0d0700 1 --2- 172.21.15.23:0/2240324917 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7968066210 0x7f7968141c50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f796400ef50 tx=0x7f7964006e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:41.900 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.872+0000 7f79557fa700 1 -- 172.21.15.23:0/2240324917 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796400cab0 con 0x7f7968066210 2020-11-20T11:37:41.900 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.872+0000 7f79557fa700 1 -- 172.21.15.23:0/2240324917 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f796400cc10 con 0x7f7968066210 2020-11-20T11:37:41.901 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.872+0000 7f796e0d2700 1 -- 172.21.15.23:0/2240324917 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79680a2860 con 0x7f7968066210 2020-11-20T11:37:41.901 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.872+0000 7f79557fa700 1 -- 172.21.15.23:0/2240324917 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79640123f0 con 0x7f7968066210 2020-11-20T11:37:41.901 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.872+0000 7f796e0d2700 1 -- 172.21.15.23:0/2240324917 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7968136690 con 0x7f7968066210 2020-11-20T11:37:41.901 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.873+0000 7f796e0d2700 1 -- 172.21.15.23:0/2240324917 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7968062760 con 0x7f7968066210 2020-11-20T11:37:41.902 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.873+0000 7f79557fa700 1 -- 172.21.15.23:0/2240324917 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7964012760 con 0x7f7968066210 2020-11-20T11:37:41.902 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.873+0000 7f79557fa700 1 --2- 172.21.15.23:0/2240324917 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7940056d60 0x7f7940059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:41.902 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.873+0000 7f79557fa700 1 -- 172.21.15.23:0/2240324917 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f79640706b0 con 0x7f7968066210 2020-11-20T11:37:41.902 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.874+0000 7f796c8cf700 1 --2- 172.21.15.23:0/2240324917 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7940056d60 0x7f7940059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:41.902 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.874+0000 7f796c8cf700 1 --2- 172.21.15.23:0/2240324917 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7940056d60 0x7f7940059220 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f7958006fd0 tx=0x7f7958015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:41.903 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:41.877+0000 7f79557fa700 1 -- 172.21.15.23:0/2240324917 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f79640459a0 con 0x7f7968066210 2020-11-20T11:37:42.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:42.102+0000 7f796e0d2700 1 -- 172.21.15.23:0/2240324917 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f7968057dc0 con 0x7f7968066210 2020-11-20T11:37:42.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:42.102+0000 7f79557fa700 1 -- 172.21.15.23:0/2240324917 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f7964048fc0 con 0x7f7968066210 2020-11-20T11:37:42.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:42.104+0000 7f796e0d2700 1 -- 172.21.15.23:0/2240324917 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7940056d60 msgr2=0x7f7940059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:42.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:42.104+0000 7f796e0d2700 1 --2- 172.21.15.23:0/2240324917 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7940056d60 0x7f7940059220 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f7958006fd0 tx=0x7f7958015000).stop 2020-11-20T11:37:42.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:42.104+0000 7f796e0d2700 1 -- 172.21.15.23:0/2240324917 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7968066210 msgr2=0x7f7968141c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:42.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:42.104+0000 7f796e0d2700 1 --2- 172.21.15.23:0/2240324917 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7968066210 0x7f7968141c50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f796400ef50 tx=0x7f7964006e30).stop 2020-11-20T11:37:42.110 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:42.105+0000 7f796e0d2700 1 -- 172.21.15.23:0/2240324917 shutdown_connections 2020-11-20T11:37:42.110 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:42.105+0000 7f796e0d2700 1 --2- 172.21.15.23:0/2240324917 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79680a8ff0 0x7f79681361a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:42.110 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:42.105+0000 7f796e0d2700 1 --2- 172.21.15.23:0/2240324917 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7968066210 0x7f7968141c50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:42.110 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:42.105+0000 7f796e0d2700 1 --2- 172.21.15.23:0/2240324917 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7940056d60 0x7f7940059220 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:42.111 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:42.105+0000 7f796e0d2700 1 --2- 172.21.15.23:0/2240324917 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79680665f0 0x7f79680a3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:42.111 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:42.105+0000 7f796e0d2700 1 -- 172.21.15.23:0/2240324917 >> 172.21.15.23:0/2240324917 conn(0x7f79680612c0 msgr2=0x7f79680669d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:42.111 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:42.105+0000 7f796e0d2700 1 -- 172.21.15.23:0/2240324917 shutdown_connections 2020-11-20T11:37:42.112 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:42.105+0000 7f796e0d2700 1 -- 172.21.15.23:0/2240324917 wait complete. 2020-11-20T11:37:42.122 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2020-11-20T11:37:52.123 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:37:52.124 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:37:52.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.246+0000 7f21893cf700 1 Processor -- start 2020-11-20T11:37:52.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.246+0000 7f21893cf700 1 -- start start 2020-11-20T11:37:52.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.247+0000 7f21893cf700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2184137870 0x7f2184137cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:52.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.247+0000 7f21893cf700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f2184133b90 0x7f2184134020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:52.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.247+0000 7f21893cf700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21840a9000 con 0x7f2184068260 2020-11-20T11:37:52.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.247+0000 7f21893cf700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2184134c40 con 0x7f2184133b90 2020-11-20T11:37:52.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.247+0000 7f21893cf700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21840663a0 con 0x7f2184137870 2020-11-20T11:37:52.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.247+0000 7f21837fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f2184133b90 0x7f2184134020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:52.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.247+0000 7f2182ffd700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f2184068260 0x7f2184065780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57564/0 (socket says 172.21.15.23:57564) 2020-11-20T11:37:52.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.247+0000 7f2182ffd700 1 -- 172.21.15.23:0/1954362979 learned_addr learned my addr 172.21.15.23:0/1954362979 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:37:52.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.247+0000 7f21827fc700 1 --2- 172.21.15.23:0/1954362979 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2184137870 0x7f2184137cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:52.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.248+0000 7f21837fe700 1 -- 172.21.15.23:0/1954362979 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2184137870 msgr2=0x7f2184137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:37:52.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.248+0000 7f21837fe700 1 --2- 172.21.15.23:0/1954362979 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2184137870 0x7f2184137cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:52.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.248+0000 7f21837fe700 1 -- 172.21.15.23:0/1954362979 >> v1:172.21.15.23:6789/0 conn(0x7f2184068260 legacy=0x7f2184065780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:52.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.248+0000 7f2181ffb700 1 -- 172.21.15.23:0/1954362979 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1843124281 0 0) 0x7f21840a9000 con 0x7f2184068260 2020-11-20T11:37:52.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.248+0000 7f21837fe700 1 -- 172.21.15.23:0/1954362979 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2184137170 con 0x7f2184133b90 2020-11-20T11:37:52.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.248+0000 7f21837fe700 1 --2- 172.21.15.23:0/1954362979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2184133b90 0x7f2184134020 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f217c009d40 tx=0x7f217c009220).ready entity=mon.0 client_cookie=967216ca5ea69be6 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:52.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.248+0000 7f2181ffb700 1 -- 172.21.15.23:0/1954362979 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217c00bb70 con 0x7f2184133b90 2020-11-20T11:37:52.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.248+0000 7f2181ffb700 1 -- 172.21.15.23:0/1954362979 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f217c00bcd0 con 0x7f2184133b90 2020-11-20T11:37:52.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.248+0000 7f2181ffb700 1 -- 172.21.15.23:0/1954362979 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217c011610 con 0x7f2184133b90 2020-11-20T11:37:52.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.248+0000 7f21893cf700 1 -- 172.21.15.23:0/1954362979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2184133b90 msgr2=0x7f2184134020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:52.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.248+0000 7f21893cf700 1 --2- 172.21.15.23:0/1954362979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2184133b90 0x7f2184134020 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f217c009d40 tx=0x7f217c009220).stop 2020-11-20T11:37:52.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.248+0000 7f21893cf700 1 -- 172.21.15.23:0/1954362979 shutdown_connections 2020-11-20T11:37:52.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.248+0000 7f21893cf700 1 --2- 172.21.15.23:0/1954362979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2184133b90 0x7f2184134020 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:52.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.248+0000 7f21893cf700 1 --2- 172.21.15.23:0/1954362979 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2184137870 0x7f2184137cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:52.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.248+0000 7f21893cf700 1 -- 172.21.15.23:0/1954362979 >> 172.21.15.23:0/1954362979 conn(0x7f21840ad7a0 msgr2=0x7f21840adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:52.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.248+0000 7f21893cf700 1 -- 172.21.15.23:0/1954362979 shutdown_connections 2020-11-20T11:37:52.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.249+0000 7f21893cf700 1 -- 172.21.15.23:0/1954362979 wait complete. 2020-11-20T11:37:52.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.250+0000 7f21893cf700 1 Processor -- start 2020-11-20T11:37:52.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.250+0000 7f21893cf700 1 -- start start 2020-11-20T11:37:52.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.251+0000 7f21893cf700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2184068260 0x7f21840aa210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:52.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.251+0000 7f21893cf700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2184133b90 0x7f21841290e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:52.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.251+0000 7f21893cf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2184137870 0x7f21841295d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:52.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.251+0000 7f21893cf700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21840a19a0 con 0x7f2184137870 2020-11-20T11:37:52.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.251+0000 7f21893cf700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21840a3170 con 0x7f2184133b90 2020-11-20T11:37:52.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.251+0000 7f21893cf700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21840a33b0 con 0x7f2184068260 2020-11-20T11:37:52.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.251+0000 7f21837fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2184137870 0x7f21841295d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:52.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.251+0000 7f21837fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2184137870 0x7f21841295d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43324/0 (socket says 172.21.15.23:43324) 2020-11-20T11:37:52.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.251+0000 7f21837fe700 1 -- 172.21.15.23:0/2408792493 learned_addr learned my addr 172.21.15.23:0/2408792493 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:37:52.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.251+0000 7f2182ffd700 1 --2- 172.21.15.23:0/2408792493 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2184068260 0x7f21840aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:52.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.251+0000 7f21827fc700 1 --2- 172.21.15.23:0/2408792493 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2184133b90 0x7f21841290e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:52.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.251+0000 7f21837fe700 1 -- 172.21.15.23:0/2408792493 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2184068260 msgr2=0x7f21840aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:52.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.251+0000 7f21837fe700 1 --2- 172.21.15.23:0/2408792493 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2184068260 0x7f21840aa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:52.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.251+0000 7f21837fe700 1 -- 172.21.15.23:0/2408792493 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2184133b90 msgr2=0x7f21841290e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:52.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.251+0000 7f21837fe700 1 --2- 172.21.15.23:0/2408792493 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2184133b90 0x7f21841290e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:52.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.251+0000 7f21837fe700 1 -- 172.21.15.23:0/2408792493 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21840a36a0 con 0x7f2184137870 2020-11-20T11:37:52.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.251+0000 7f21837fe700 1 --2- 172.21.15.23:0/2408792493 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2184137870 0x7f21841295d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f217c0090b0 tx=0x7f217c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:52.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.251+0000 7f216affd700 1 -- 172.21.15.23:0/2408792493 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217c00bb70 con 0x7f2184137870 2020-11-20T11:37:52.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.252+0000 7f216affd700 1 -- 172.21.15.23:0/2408792493 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f217c009e80 con 0x7f2184137870 2020-11-20T11:37:52.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.252+0000 7f21893cf700 1 -- 172.21.15.23:0/2408792493 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21840a15b0 con 0x7f2184137870 2020-11-20T11:37:52.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.252+0000 7f216affd700 1 -- 172.21.15.23:0/2408792493 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217c010e50 con 0x7f2184137870 2020-11-20T11:37:52.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.252+0000 7f21893cf700 1 -- 172.21.15.23:0/2408792493 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21840ab0c0 con 0x7f2184137870 2020-11-20T11:37:52.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.253+0000 7f21893cf700 1 -- 172.21.15.23:0/2408792493 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21840a3830 con 0x7f2184137870 2020-11-20T11:37:52.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.253+0000 7f216affd700 1 -- 172.21.15.23:0/2408792493 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f217c017040 con 0x7f2184137870 2020-11-20T11:37:52.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.253+0000 7f216affd700 1 --2- 172.21.15.23:0/2408792493 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2160056db0 0x7f2160059270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:52.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.253+0000 7f216affd700 1 -- 172.21.15.23:0/2408792493 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f217c0722d0 con 0x7f2184137870 2020-11-20T11:37:52.266 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.253+0000 7f2182ffd700 1 --2- 172.21.15.23:0/2408792493 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2160056db0 0x7f2160059270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:52.266 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.254+0000 7f2182ffd700 1 --2- 172.21.15.23:0/2408792493 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2160056db0 0x7f2160059270 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f21740011e0 tx=0x7f2174009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:52.266 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.257+0000 7f216affd700 1 -- 172.21.15.23:0/2408792493 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f217c047640 con 0x7f2184137870 2020-11-20T11:37:52.533 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.530+0000 7f21893cf700 1 -- 172.21.15.23:0/2408792493 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2184062fe0 con 0x7f2184137870 2020-11-20T11:37:52.534 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.530+0000 7f216affd700 1 -- 172.21.15.23:0/2408792493 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7f217c015020 con 0x7f2184137870 2020-11-20T11:37:52.534 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:37:52.536 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.532+0000 7f21893cf700 1 -- 172.21.15.23:0/2408792493 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2160056db0 msgr2=0x7f2160059270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:52.536 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.532+0000 7f21893cf700 1 --2- 172.21.15.23:0/2408792493 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2160056db0 0x7f2160059270 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f21740011e0 tx=0x7f2174009040).stop 2020-11-20T11:37:52.536 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.532+0000 7f21893cf700 1 -- 172.21.15.23:0/2408792493 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2184137870 msgr2=0x7f21841295d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:52.536 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.532+0000 7f21893cf700 1 --2- 172.21.15.23:0/2408792493 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2184137870 0x7f21841295d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f217c0090b0 tx=0x7f217c016000).stop 2020-11-20T11:37:52.536 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.532+0000 7f21893cf700 1 -- 172.21.15.23:0/2408792493 shutdown_connections 2020-11-20T11:37:52.537 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.532+0000 7f21893cf700 1 --2- 172.21.15.23:0/2408792493 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2184137870 0x7f21841295d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:52.537 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.532+0000 7f21893cf700 1 --2- 172.21.15.23:0/2408792493 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2160056db0 0x7f2160059270 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:52.537 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.532+0000 7f21893cf700 1 --2- 172.21.15.23:0/2408792493 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2184068260 0x7f21840aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:52.537 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.532+0000 7f21893cf700 1 --2- 172.21.15.23:0/2408792493 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2184133b90 0x7f21841290e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:52.538 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.532+0000 7f21893cf700 1 -- 172.21.15.23:0/2408792493 >> 172.21.15.23:0/2408792493 conn(0x7f21840ad7a0 msgr2=0x7f2184068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:52.538 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.533+0000 7f21893cf700 1 -- 172.21.15.23:0/2408792493 shutdown_connections 2020-11-20T11:37:52.538 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.533+0000 7f21893cf700 1 -- 172.21.15.23:0/2408792493 wait complete. 2020-11-20T11:37:52.547 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:37:52.547 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2020-11-20T11:37:52.548 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:37:52.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.691+0000 7f1c3107d700 1 Processor -- start 2020-11-20T11:37:52.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.691+0000 7f1c3107d700 1 -- start start 2020-11-20T11:37:52.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.691+0000 7f1c3107d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1c2c061380 0x7f1c2c0a5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:52.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.691+0000 7f1c3107d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1c2c0a9bf0 0x7f1c2c0a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:52.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.691+0000 7f1c3107d700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c2c0a6bf0 con 0x7f1c2c0a1500 2020-11-20T11:37:52.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.691+0000 7f1c3107d700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c2c0a6e20 con 0x7f1c2c061380 2020-11-20T11:37:52.707 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.691+0000 7f1c3107d700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c0a70c0 con 0x7f1c2c0a9bf0 2020-11-20T11:37:52.707 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.692+0000 7f1c2bfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1c2c061380 0x7f1c2c0a5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:52.707 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.692+0000 7f1c3087c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f1c2c0a1500 0x7f1c2c0a4f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57582/0 (socket says 172.21.15.23:57582) 2020-11-20T11:37:52.708 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.692+0000 7f1c3087c700 1 -- 172.21.15.23:0/2453483840 learned_addr learned my addr 172.21.15.23:0/2453483840 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:37:52.708 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.692+0000 7f1c2b7fe700 1 --2- 172.21.15.23:0/2453483840 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1c2c0a9bf0 0x7f1c2c0a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:52.708 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.692+0000 7f1c2affd700 1 -- 172.21.15.23:0/2453483840 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3030228146 0 0) 0x7f1c2c0a6bf0 con 0x7f1c2c0a1500 2020-11-20T11:37:52.709 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.692+0000 7f1c2bfff700 1 -- 172.21.15.23:0/2453483840 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1c2c0a9bf0 msgr2=0x7f1c2c0a4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:37:52.709 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.692+0000 7f1c2bfff700 1 --2- 172.21.15.23:0/2453483840 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1c2c0a9bf0 0x7f1c2c0a4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:52.709 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.692+0000 7f1c2bfff700 1 -- 172.21.15.23:0/2453483840 >> v1:172.21.15.23:6789/0 conn(0x7f1c2c0a1500 legacy=0x7f1c2c0a4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:52.710 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.692+0000 7f1c2bfff700 1 -- 172.21.15.23:0/2453483840 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c2c0aaaf0 con 0x7f1c2c061380 2020-11-20T11:37:52.710 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.692+0000 7f1c2bfff700 1 --2- 172.21.15.23:0/2453483840 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c2c061380 0x7f1c2c0a5c00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f1c1c006fd0 tx=0x7f1c1c006e30).ready entity=mon.0 client_cookie=5c9f8685e7f4da57 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:52.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.692+0000 7f1c2affd700 1 -- 172.21.15.23:0/2453483840 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c0097e0 con 0x7f1c2c061380 2020-11-20T11:37:52.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.692+0000 7f1c2affd700 1 -- 172.21.15.23:0/2453483840 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c1c000e90 con 0x7f1c2c061380 2020-11-20T11:37:52.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.692+0000 7f1c2affd700 1 -- 172.21.15.23:0/2453483840 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c009b50 con 0x7f1c2c061380 2020-11-20T11:37:52.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.693+0000 7f1c3107d700 1 -- 172.21.15.23:0/2453483840 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c2c061380 msgr2=0x7f1c2c0a5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:52.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.693+0000 7f1c3107d700 1 --2- 172.21.15.23:0/2453483840 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c2c061380 0x7f1c2c0a5c00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f1c1c006fd0 tx=0x7f1c1c006e30).stop 2020-11-20T11:37:52.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.693+0000 7f1c3107d700 1 -- 172.21.15.23:0/2453483840 shutdown_connections 2020-11-20T11:37:52.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.693+0000 7f1c3107d700 1 --2- 172.21.15.23:0/2453483840 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c2c061380 0x7f1c2c0a5c00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:52.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.693+0000 7f1c3107d700 1 --2- 172.21.15.23:0/2453483840 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1c2c0a9bf0 0x7f1c2c0a4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:52.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.693+0000 7f1c3107d700 1 -- 172.21.15.23:0/2453483840 >> 172.21.15.23:0/2453483840 conn(0x7f1c2c0ae0a0 msgr2=0x7f1c2c0aa0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:52.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.693+0000 7f1c3107d700 1 -- 172.21.15.23:0/2453483840 shutdown_connections 2020-11-20T11:37:52.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.693+0000 7f1c3107d700 1 -- 172.21.15.23:0/2453483840 wait complete. 2020-11-20T11:37:52.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.694+0000 7f1c3107d700 1 Processor -- start 2020-11-20T11:37:52.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.694+0000 7f1c3107d700 1 -- start start 2020-11-20T11:37:52.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.695+0000 7f1c3107d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1c2c061380 0x7f1c2c138900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:52.716 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.695+0000 7f1c3107d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1c2c0a1500 0x7f1c2c14a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:52.716 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.695+0000 7f1c3107d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c2c0a9bf0 0x7f1c2c14a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:52.716 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.695+0000 7f1c3107d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c139630 con 0x7f1c2c0a9bf0 2020-11-20T11:37:52.717 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.695+0000 7f1c3107d700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c14b000 con 0x7f1c2c0a1500 2020-11-20T11:37:52.717 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.695+0000 7f1c3107d700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c14b2a0 con 0x7f1c2c061380 2020-11-20T11:37:52.717 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.695+0000 7f1c3087c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c2c0a9bf0 0x7f1c2c14a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:52.718 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.695+0000 7f1c3087c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c2c0a9bf0 0x7f1c2c14a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43338/0 (socket says 172.21.15.23:43338) 2020-11-20T11:37:52.718 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.695+0000 7f1c3087c700 1 -- 172.21.15.23:0/1814694212 learned_addr learned my addr 172.21.15.23:0/1814694212 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:37:52.718 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.695+0000 7f1c2bfff700 1 --2- 172.21.15.23:0/1814694212 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1c2c061380 0x7f1c2c138900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:52.719 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.695+0000 7f1c2b7fe700 1 --2- 172.21.15.23:0/1814694212 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1c2c0a1500 0x7f1c2c14a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:52.719 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.695+0000 7f1c3087c700 1 -- 172.21.15.23:0/1814694212 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1c2c061380 msgr2=0x7f1c2c138900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:52.719 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.695+0000 7f1c3087c700 1 --2- 172.21.15.23:0/1814694212 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1c2c061380 0x7f1c2c138900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:52.720 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.695+0000 7f1c3087c700 1 -- 172.21.15.23:0/1814694212 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1c2c0a1500 msgr2=0x7f1c2c14a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:52.720 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.695+0000 7f1c3087c700 1 --2- 172.21.15.23:0/1814694212 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1c2c0a1500 0x7f1c2c14a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:52.720 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.695+0000 7f1c3087c700 1 -- 172.21.15.23:0/1814694212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c2c14d9f0 con 0x7f1c2c0a9bf0 2020-11-20T11:37:52.721 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.695+0000 7f1c3087c700 1 --2- 172.21.15.23:0/1814694212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c2c0a9bf0 0x7f1c2c14a970 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f1c2400cab0 tx=0x7f1c24007da0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:52.721 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.696+0000 7f1c0bfff700 1 -- 172.21.15.23:0/1814694212 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2400ce90 con 0x7f1c2c0a9bf0 2020-11-20T11:37:52.722 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.696+0000 7f1c0bfff700 1 -- 172.21.15.23:0/1814694212 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c24005bc0 con 0x7f1c2c0a9bf0 2020-11-20T11:37:52.722 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.696+0000 7f1c0bfff700 1 -- 172.21.15.23:0/1814694212 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c24007110 con 0x7f1c2c0a9bf0 2020-11-20T11:37:52.722 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.696+0000 7f1c3107d700 1 -- 172.21.15.23:0/1814694212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c2c136850 con 0x7f1c2c0a9bf0 2020-11-20T11:37:52.723 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.696+0000 7f1c3107d700 1 -- 172.21.15.23:0/1814694212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c2c136d70 con 0x7f1c2c0a9bf0 2020-11-20T11:37:52.723 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.697+0000 7f1c3107d700 1 --2- 172.21.15.23:0/1814694212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c2c061980 0x7f1c2c14ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:52.723 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.697+0000 7f1c3107d700 1 -- 172.21.15.23:0/1814694212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c14ae60 con 0x7f1c2c061980 2020-11-20T11:37:52.724 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.697+0000 7f1c2bfff700 1 --2- 172.21.15.23:0/1814694212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c2c061980 0x7f1c2c14ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:52.724 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.697+0000 7f1c2bfff700 1 -- 172.21.15.23:0/1814694212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1c2c069c70 con 0x7f1c2c061980 2020-11-20T11:37:52.724 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.697+0000 7f1c2bfff700 1 --2- 172.21.15.23:0/1814694212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c2c061980 0x7f1c2c14ddb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f1c1c006fd0 tx=0x7f1c1c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:52.725 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.697+0000 7f1c0bfff700 1 -- 172.21.15.23:0/1814694212 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1c24010030 con 0x7f1c2c0a9bf0 2020-11-20T11:37:52.725 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.698+0000 7f1c0bfff700 1 --2- 172.21.15.23:0/1814694212 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1c04056d90 0x7f1c04059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:52.726 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.698+0000 7f1c0bfff700 1 -- 172.21.15.23:0/1814694212 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f1c2406feb0 con 0x7f1c2c0a9bf0 2020-11-20T11:37:52.726 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.698+0000 7f1c0bfff700 1 -- 172.21.15.23:0/1814694212 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c011bd0 con 0x7f1c2c061980 2020-11-20T11:37:52.726 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.698+0000 7f1c0bfff700 1 -- 172.21.15.23:0/1814694212 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f1c1c011d30 con 0x7f1c2c061980 2020-11-20T11:37:52.726 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.698+0000 7f1c2b7fe700 1 --2- 172.21.15.23:0/1814694212 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1c04056d90 0x7f1c04059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:52.727 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.698+0000 7f1c0bfff700 1 -- 172.21.15.23:0/1814694212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c2c061980 msgr2=0x7f1c2c14ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:52.727 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.698+0000 7f1c0bfff700 1 --2- 172.21.15.23:0/1814694212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c2c061980 0x7f1c2c14ddb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f1c1c006fd0 tx=0x7f1c1c015000).stop 2020-11-20T11:37:52.727 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.698+0000 7f1c0bfff700 1 -- 172.21.15.23:0/1814694212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c2c061980 msgr2=0x7f1c2c14ddb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:37:52.728 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.698+0000 7f1c0bfff700 1 --2- 172.21.15.23:0/1814694212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c2c061980 0x7f1c2c14ddb0 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0x7f1c1c006fd0 tx=0x7f1c1c015000).stop 2020-11-20T11:37:52.728 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.698+0000 7f1c2b7fe700 1 --2- 172.21.15.23:0/1814694212 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1c04056d90 0x7f1c04059250 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f1c2c09dce0 tx=0x7f1c20006e30).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:52.732 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.706+0000 7f1c3107d700 1 --2- 172.21.15.23:0/1814694212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c2c14e2a0 0x7f1c2c0a7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:52.733 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.706+0000 7f1c3107d700 1 -- 172.21.15.23:0/1814694212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c13a730 con 0x7f1c2c14e2a0 2020-11-20T11:37:52.733 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.706+0000 7f1c2bfff700 1 --2- 172.21.15.23:0/1814694212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c2c14e2a0 0x7f1c2c0a7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:52.734 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.706+0000 7f1c2bfff700 1 -- 172.21.15.23:0/1814694212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1c2c14db80 con 0x7f1c2c14e2a0 2020-11-20T11:37:52.734 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.707+0000 7f1c2bfff700 1 --2- 172.21.15.23:0/1814694212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c2c14e2a0 0x7f1c2c0a7fc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f1c1c009fd0 tx=0x7f1c1c021070).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:52.734 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.707+0000 7f1c0bfff700 1 -- 172.21.15.23:0/1814694212 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c02d860 con 0x7f1c2c14e2a0 2020-11-20T11:37:52.735 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.707+0000 7f1c0bfff700 1 -- 172.21.15.23:0/1814694212 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4446 (secure 0 0 0) 0x7f1c1c0119c0 con 0x7f1c2c14e2a0 2020-11-20T11:37:52.735 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.707+0000 7f1c0bfff700 1 -- 172.21.15.23:0/1814694212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c2c14e2a0 msgr2=0x7f1c2c0a7fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:52.735 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.707+0000 7f1c0bfff700 1 --2- 172.21.15.23:0/1814694212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c2c14e2a0 0x7f1c2c0a7fc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f1c1c009fd0 tx=0x7f1c1c021070).stop 2020-11-20T11:37:52.735 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.707+0000 7f1c0bfff700 1 -- 172.21.15.23:0/1814694212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c2c14e2a0 msgr2=0x7f1c2c0a7fc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:37:52.736 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.707+0000 7f1c0bfff700 1 --2- 172.21.15.23:0/1814694212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c2c14e2a0 0x7f1c2c0a7fc0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:52.736 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.707+0000 7f1c3107d700 1 -- 172.21.15.23:0/1814694212 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1c04056d90 msgr2=0x7f1c04059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:52.736 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.707+0000 7f1c3107d700 1 --2- 172.21.15.23:0/1814694212 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1c04056d90 0x7f1c04059250 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f1c2c09dce0 tx=0x7f1c20006e30).stop 2020-11-20T11:37:52.737 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.707+0000 7f1c3107d700 1 -- 172.21.15.23:0/1814694212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c2c0a9bf0 msgr2=0x7f1c2c14a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:52.737 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.707+0000 7f1c3107d700 1 --2- 172.21.15.23:0/1814694212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c2c0a9bf0 0x7f1c2c14a970 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f1c2400cab0 tx=0x7f1c24007da0).stop 2020-11-20T11:37:52.737 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.707+0000 7f1c3087c700 1 -- 172.21.15.23:0/1814694212 reap_dead start 2020-11-20T11:37:52.738 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.707+0000 7f1c3087c700 1 -- 172.21.15.23:0/1814694212 reap_dead start 2020-11-20T11:37:52.738 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.707+0000 7f1c3107d700 1 -- 172.21.15.23:0/1814694212 shutdown_connections 2020-11-20T11:37:52.739 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.707+0000 7f1c3107d700 1 -- 172.21.15.23:0/1814694212 >> 172.21.15.23:0/1814694212 conn(0x7f1c2c0ae0a0 msgr2=0x7f1c2c0ab430 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:52.739 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.707+0000 7f1c3107d700 1 -- 172.21.15.23:0/1814694212 shutdown_connections 2020-11-20T11:37:52.739 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.707+0000 7f1c3107d700 1 -- 172.21.15.23:0/1814694212 wait complete. 2020-11-20T11:37:52.740 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:37:52.740 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:37:52.741 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:37:52.741 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:37:52.741 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 56, 2020-11-20T11:37:52.742 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:37:52.742 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:37:52.742 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:37:52.742 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:37:52.743 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:52.743 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 13, 2020-11-20T11:37:52.744 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:37:52.744 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:37:52.744 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:37:52.745 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:37:52.745 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:37:52.745 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:37:52.746 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:37:52.746 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:37:52.746 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:37:52.747 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:37:52.747 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:37:52.747 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:52.748 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:37:52.748 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:37:52.748 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:37:52.749 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:37:52.749 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:37:52.749 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:37:52.750 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:37:52.750 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:37:52.750 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:37:52.751 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:37:52.751 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:52.751 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:52.752 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:37:52.752 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:37:52.752 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:37:52.753 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:37:52.753 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:37:52.754 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:37:52.754 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:37:52.754 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:37:52.755 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:37:52.755 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:37:52.755 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:37:52.756 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:37:52.756 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:37:52.756 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:37:52.757 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:37:52.757 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:37:52.758 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:37:52.758 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:37:52.758 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:37:52.759 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:37:52.759 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:37:52.759 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:37:52.760 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:37:52.760 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:52.760 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:37:52.761 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:52.761 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:37:52.761 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:52.762 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:37:52.762 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:37:52.762 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:37:52.763 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:37:52.763 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:52.763 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:37:52.764 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:37:52.764 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:52.765 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:52.765 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:52.765 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:37:52.766 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:37:52.766 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:52.766 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:52.767 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:52.767 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:52.767 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:37:52.768 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:37:52.768 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:37:52.768 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:37:52.769 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:37:52.769 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:52.770 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:52.770 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:37:52.770 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:37:52.770 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:37:52.771 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:37:52.771 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:52.772 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:37:52.772 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:37:52.772 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:52.772 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:52.773 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:52.773 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:37:52.774 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:37:52.774 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:52.774 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:52.775 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:52.775 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:52.775 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:37:52.776 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:37:52.776 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:37:52.776 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:37:52.777 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:37:52.777 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:52.778 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:52.778 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:37:52.778 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:37:52.779 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:37:52.779 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:37:52.779 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:52.780 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:37:52.780 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:37:52.780 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:52.781 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:52.781 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:52.781 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:37:52.782 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:37:52.782 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:52.782 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:52.783 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:52.783 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:52.783 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:37:52.784 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:37:52.784 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:37:52.784 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:37:52.785 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:37:52.785 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:52.786 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:52.786 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:52.786 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:37:52.787 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:37:52.787 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:52.787 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:37:52.788 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:37:52.788 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:37:52.788 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:52.789 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:52.789 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:37:52.789 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:52.790 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:37:52.790 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:37:52.791 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:37:52.791 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:52.791 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:52.792 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:37:52.792 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:52.792 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:37:52.792 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:37:52.793 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:37:52.793 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:52.794 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:52.794 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:52.794 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:37:52.795 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:37:52.795 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:37:52.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.929+0000 7f902bbb8700 1 Processor -- start 2020-11-20T11:37:52.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.930+0000 7f902bbb8700 1 -- start start 2020-11-20T11:37:52.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.930+0000 7f902bbb8700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f90240aae90 0x7f90241381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:52.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.930+0000 7f902bbb8700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f902413aeb0 0x7f90240a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:52.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.930+0000 7f902bbb8700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f902413e600 con 0x7f90240aaab0 2020-11-20T11:37:52.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.930+0000 7f902bbb8700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f902413e8a0 con 0x7f90240aae90 2020-11-20T11:37:52.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.930+0000 7f902bbb8700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90240ab6d0 con 0x7f902413aeb0 2020-11-20T11:37:52.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.930+0000 7f9029153700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f90240aae90 0x7f90241381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:52.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.930+0000 7f9029954700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f90240aaab0 0x7f902413ed40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57596/0 (socket says 172.21.15.23:57596) 2020-11-20T11:37:52.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.930+0000 7f9029954700 1 -- 172.21.15.23:0/368729259 learned_addr learned my addr 172.21.15.23:0/368729259 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:37:52.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.931+0000 7f902a155700 1 --2- 172.21.15.23:0/368729259 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f902413aeb0 0x7f90240a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:52.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.931+0000 7f9029153700 1 -- 172.21.15.23:0/368729259 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f902413aeb0 msgr2=0x7f90240a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:37:52.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.931+0000 7f9029153700 1 --2- 172.21.15.23:0/368729259 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f902413aeb0 0x7f90240a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:52.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.931+0000 7f9029153700 1 -- 172.21.15.23:0/368729259 >> v1:172.21.15.23:6789/0 conn(0x7f90240aaab0 legacy=0x7f902413ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:52.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.931+0000 7f9029153700 1 -- 172.21.15.23:0/368729259 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9024138690 con 0x7f90240aae90 2020-11-20T11:37:52.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.931+0000 7f9028952700 1 -- 172.21.15.23:0/368729259 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4160205212 0 0) 0x7f902413e600 con 0x7f90240aaab0 2020-11-20T11:37:52.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.931+0000 7f9029153700 1 --2- 172.21.15.23:0/368729259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90240aae90 0x7f90241381a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f9018006fd0 tx=0x7f9018006e30).ready entity=mon.0 client_cookie=d5562f8fc543daf2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:52.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.931+0000 7f9028952700 1 -- 172.21.15.23:0/368729259 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90180097e0 con 0x7f90240aae90 2020-11-20T11:37:52.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.931+0000 7f9028952700 1 -- 172.21.15.23:0/368729259 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9018000e90 con 0x7f90240aae90 2020-11-20T11:37:52.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.931+0000 7f9028952700 1 -- 172.21.15.23:0/368729259 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9018009b50 con 0x7f90240aae90 2020-11-20T11:37:52.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.932+0000 7f902bbb8700 1 -- 172.21.15.23:0/368729259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90240aae90 msgr2=0x7f90241381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:52.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.932+0000 7f902bbb8700 1 --2- 172.21.15.23:0/368729259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90240aae90 0x7f90241381a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f9018006fd0 tx=0x7f9018006e30).stop 2020-11-20T11:37:52.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.932+0000 7f902bbb8700 1 -- 172.21.15.23:0/368729259 shutdown_connections 2020-11-20T11:37:52.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.932+0000 7f902bbb8700 1 --2- 172.21.15.23:0/368729259 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f902413aeb0 0x7f90240a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:52.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.932+0000 7f902bbb8700 1 --2- 172.21.15.23:0/368729259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90240aae90 0x7f90241381a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:52.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.932+0000 7f902bbb8700 1 -- 172.21.15.23:0/368729259 >> 172.21.15.23:0/368729259 conn(0x7f90240678e0 msgr2=0x7f9024067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:52.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.932+0000 7f902bbb8700 1 -- 172.21.15.23:0/368729259 shutdown_connections 2020-11-20T11:37:52.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.932+0000 7f902bbb8700 1 -- 172.21.15.23:0/368729259 wait complete. 2020-11-20T11:37:52.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.933+0000 7f902bbb8700 1 Processor -- start 2020-11-20T11:37:52.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.933+0000 7f902bbb8700 1 -- start start 2020-11-20T11:37:52.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.933+0000 7f902bbb8700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90240aaab0 0x7f9024132bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:52.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.933+0000 7f902bbb8700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90240aae90 0x7f90240a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:52.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.933+0000 7f902bbb8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f902413d510 0x7f902413c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:52.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.933+0000 7f902bbb8700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902413e310 con 0x7f902413d510 2020-11-20T11:37:52.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.933+0000 7f902bbb8700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902413da20 con 0x7f90240aaab0 2020-11-20T11:37:52.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.933+0000 7f902bbb8700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f902413dcc0 con 0x7f90240aae90 2020-11-20T11:37:52.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.933+0000 7f902a155700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f902413d510 0x7f902413c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:52.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.934+0000 7f902a155700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f902413d510 0x7f902413c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43356/0 (socket says 172.21.15.23:43356) 2020-11-20T11:37:52.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.934+0000 7f902a155700 1 -- 172.21.15.23:0/2229560927 learned_addr learned my addr 172.21.15.23:0/2229560927 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:37:52.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.934+0000 7f9029153700 1 --2- 172.21.15.23:0/2229560927 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90240aae90 0x7f90240a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:52.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.934+0000 7f9029954700 1 --2- 172.21.15.23:0/2229560927 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90240aaab0 0x7f9024132bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:52.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.934+0000 7f902a155700 1 -- 172.21.15.23:0/2229560927 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90240aae90 msgr2=0x7f90240a62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:52.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.934+0000 7f902a155700 1 --2- 172.21.15.23:0/2229560927 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90240aae90 0x7f90240a62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:52.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.934+0000 7f902a155700 1 -- 172.21.15.23:0/2229560927 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90240aaab0 msgr2=0x7f9024132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:52.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.934+0000 7f902a155700 1 --2- 172.21.15.23:0/2229560927 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90240aaab0 0x7f9024132bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:52.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.934+0000 7f902a155700 1 -- 172.21.15.23:0/2229560927 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f902413c9a0 con 0x7f902413d510 2020-11-20T11:37:52.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.934+0000 7f902a155700 1 --2- 172.21.15.23:0/2229560927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f902413d510 0x7f902413c4b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f901400d830 tx=0x7f9014008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:52.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.934+0000 7f9021ffb700 1 -- 172.21.15.23:0/2229560927 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9014011590 con 0x7f902413d510 2020-11-20T11:37:52.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.934+0000 7f9021ffb700 1 -- 172.21.15.23:0/2229560927 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90140116f0 con 0x7f902413d510 2020-11-20T11:37:52.949 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.934+0000 7f902bbb8700 1 -- 172.21.15.23:0/2229560927 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f902413cbf0 con 0x7f902413d510 2020-11-20T11:37:52.949 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.934+0000 7f9021ffb700 1 -- 172.21.15.23:0/2229560927 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9014011a90 con 0x7f902413d510 2020-11-20T11:37:52.949 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.934+0000 7f902bbb8700 1 -- 172.21.15.23:0/2229560927 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9024130870 con 0x7f902413d510 2020-11-20T11:37:52.949 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.935+0000 7f902bbb8700 1 --2- 172.21.15.23:0/2229560927 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f902413aeb0 0x7f90240acf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:52.950 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.935+0000 7f902bbb8700 1 -- 172.21.15.23:0/2229560927 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902405c7d0 con 0x7f902413aeb0 2020-11-20T11:37:52.950 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.936+0000 7f9029954700 1 --2- 172.21.15.23:0/2229560927 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f902413aeb0 0x7f90240acf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:52.950 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.936+0000 7f9029954700 1 -- 172.21.15.23:0/2229560927 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9024044710 con 0x7f902413aeb0 2020-11-20T11:37:52.950 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.936+0000 7f9029954700 1 --2- 172.21.15.23:0/2229560927 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f902413aeb0 0x7f90240acf60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f901c001f50 tx=0x7f901c007dc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:52.950 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.936+0000 7f9021ffb700 1 -- 172.21.15.23:0/2229560927 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9014012420 con 0x7f902413d510 2020-11-20T11:37:52.951 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.936+0000 7f9021ffb700 1 --2- 172.21.15.23:0/2229560927 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8ffc056d60 0x7f8ffc059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:52.951 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.937+0000 7f9021ffb700 1 -- 172.21.15.23:0/2229560927 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f901406fb50 con 0x7f902413d510 2020-11-20T11:37:52.951 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.937+0000 7f9021ffb700 1 -- 172.21.15.23:0/2229560927 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901c007820 con 0x7f902413aeb0 2020-11-20T11:37:52.951 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.937+0000 7f9029153700 1 --2- 172.21.15.23:0/2229560927 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8ffc056d60 0x7f8ffc059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:52.951 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.937+0000 7f9021ffb700 1 -- 172.21.15.23:0/2229560927 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f901c00fae0 con 0x7f902413aeb0 2020-11-20T11:37:52.952 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.937+0000 7f9021ffb700 1 -- 172.21.15.23:0/2229560927 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f902413aeb0 msgr2=0x7f90240acf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:52.952 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.937+0000 7f9021ffb700 1 --2- 172.21.15.23:0/2229560927 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f902413aeb0 0x7f90240acf60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f901c001f50 tx=0x7f901c007dc0).stop 2020-11-20T11:37:52.952 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.937+0000 7f9021ffb700 1 -- 172.21.15.23:0/2229560927 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f902413aeb0 msgr2=0x7f90240acf60 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:37:52.952 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.937+0000 7f9021ffb700 1 --2- 172.21.15.23:0/2229560927 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f902413aeb0 0x7f90240acf60 secure :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0x7f901c001f50 tx=0x7f901c007dc0).stop 2020-11-20T11:37:52.952 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.937+0000 7f9029153700 1 --2- 172.21.15.23:0/2229560927 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8ffc056d60 0x7f8ffc059220 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f9018006e70 tx=0x7f9018015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:52.954 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.943+0000 7f902bbb8700 1 --2- 172.21.15.23:0/2229560927 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90240601b0 0x7f90240b0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:52.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.943+0000 7f902bbb8700 1 -- 172.21.15.23:0/2229560927 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902405c280 con 0x7f90240601b0 2020-11-20T11:37:52.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.944+0000 7f9029954700 1 --2- 172.21.15.23:0/2229560927 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90240601b0 0x7f90240b0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:52.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.944+0000 7f9029954700 1 -- 172.21.15.23:0/2229560927 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f902405cd10 con 0x7f90240601b0 2020-11-20T11:37:52.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.944+0000 7f9029954700 1 --2- 172.21.15.23:0/2229560927 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90240601b0 0x7f90240b0140 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f901c001f50 tx=0x7f901c019000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:52.956 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.944+0000 7f9021ffb700 1 -- 172.21.15.23:0/2229560927 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f901c024480 con 0x7f90240601b0 2020-11-20T11:37:52.956 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.945+0000 7f9021ffb700 1 -- 172.21.15.23:0/2229560927 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4784 (secure 0 0 0) 0x7f901c002030 con 0x7f90240601b0 2020-11-20T11:37:52.956 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.945+0000 7f9021ffb700 1 -- 172.21.15.23:0/2229560927 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90240601b0 msgr2=0x7f90240b0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:52.956 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.945+0000 7f9021ffb700 1 --2- 172.21.15.23:0/2229560927 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90240601b0 0x7f90240b0140 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f901c001f50 tx=0x7f901c019000).stop 2020-11-20T11:37:52.956 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.945+0000 7f9021ffb700 1 -- 172.21.15.23:0/2229560927 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90240601b0 msgr2=0x7f90240b0140 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:37:52.957 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.945+0000 7f9021ffb700 1 --2- 172.21.15.23:0/2229560927 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90240601b0 0x7f90240b0140 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:52.957 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.945+0000 7f902bbb8700 1 -- 172.21.15.23:0/2229560927 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8ffc056d60 msgr2=0x7f8ffc059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:52.957 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.945+0000 7f902bbb8700 1 --2- 172.21.15.23:0/2229560927 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8ffc056d60 0x7f8ffc059220 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f9018006e70 tx=0x7f9018015000).stop 2020-11-20T11:37:52.957 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.945+0000 7f902bbb8700 1 -- 172.21.15.23:0/2229560927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f902413d510 msgr2=0x7f902413c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:52.958 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.945+0000 7f902bbb8700 1 --2- 172.21.15.23:0/2229560927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f902413d510 0x7f902413c4b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f901400d830 tx=0x7f9014008040).stop 2020-11-20T11:37:52.958 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.945+0000 7f902a155700 1 -- 172.21.15.23:0/2229560927 reap_dead start 2020-11-20T11:37:52.958 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.945+0000 7f902a155700 1 -- 172.21.15.23:0/2229560927 reap_dead start 2020-11-20T11:37:52.958 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.945+0000 7f902bbb8700 1 -- 172.21.15.23:0/2229560927 shutdown_connections 2020-11-20T11:37:52.958 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.945+0000 7f902bbb8700 1 -- 172.21.15.23:0/2229560927 >> 172.21.15.23:0/2229560927 conn(0x7f90240678e0 msgr2=0x7f9024068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:52.959 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.945+0000 7f902bbb8700 1 -- 172.21.15.23:0/2229560927 shutdown_connections 2020-11-20T11:37:52.959 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:52.945+0000 7f902bbb8700 1 -- 172.21.15.23:0/2229560927 wait complete. 2020-11-20T11:37:52.959 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:37:52.959 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:37:52.959 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:37:52.960 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:37:52.960 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 56, 2020-11-20T11:37:52.960 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:37:52.960 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:37:52.961 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:37:52.961 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:37:52.961 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:52.961 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 13, 2020-11-20T11:37:52.961 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:37:52.962 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:37:52.962 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:37:52.962 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:37:52.962 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:37:52.962 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:37:52.963 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:37:52.963 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:37:52.963 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:37:52.963 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:37:52.963 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:37:52.964 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:52.964 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:37:52.964 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:37:52.964 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:37:52.964 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:37:52.965 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:37:52.965 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:37:52.965 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:37:52.965 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:37:52.965 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:37:52.966 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:37:52.966 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:52.966 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:52.966 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:37:52.966 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:37:52.967 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:37:52.967 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:37:52.967 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:37:52.967 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:37:52.967 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:37:52.968 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:37:52.968 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:37:52.968 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:37:52.968 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:37:52.968 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:37:52.969 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:37:52.969 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:37:52.969 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:37:52.969 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:37:52.969 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:37:52.970 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:37:52.970 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:37:52.970 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:37:52.970 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:37:52.970 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:37:52.971 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:37:52.971 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:52.971 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:37:52.971 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:52.971 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:37:52.972 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:52.972 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:37:52.972 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:37:52.972 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:37:52.972 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:37:52.973 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:52.973 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:37:52.973 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:37:52.973 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:52.973 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:52.974 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:52.974 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:37:52.974 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:37:52.974 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:52.974 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:52.975 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:52.975 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:52.975 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:37:52.975 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:37:52.975 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:37:52.976 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:37:52.976 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:37:52.976 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:52.976 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:52.976 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:37:52.977 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:37:52.977 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:37:52.977 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:37:52.977 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:52.977 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:37:52.978 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:37:52.978 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:52.978 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:52.978 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:52.978 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:37:52.979 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:37:52.979 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:52.979 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:52.979 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:52.979 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:52.980 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:37:52.980 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:37:52.980 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:37:52.980 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:37:52.980 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:37:52.981 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:52.981 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:52.981 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:37:52.981 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:37:52.981 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:37:52.982 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:37:52.982 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:52.982 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:37:52.982 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:37:52.982 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:52.983 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:52.983 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:52.983 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:37:52.983 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:37:52.983 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:52.984 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:52.984 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:52.984 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:52.984 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:37:52.984 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:37:52.985 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:37:52.985 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:37:52.985 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:37:52.985 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:52.985 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:52.986 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:52.986 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:37:52.986 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:37:52.986 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:52.986 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:37:52.987 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:37:52.987 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:37:52.987 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:52.987 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:52.987 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:37:52.988 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:52.988 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:37:52.988 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:37:52.988 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:37:52.988 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:52.989 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:52.989 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:37:52.989 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:52.989 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:37:52.989 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:37:52.990 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:37:52.990 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:52.990 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:52.990 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:37:52.990 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:52.991 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:37:52.991 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:37:52.991 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:37:52.991 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:52.991 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:52.992 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:37:52.992 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:52.992 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:37:52.992 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:37:52.992 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:37:52.993 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:52.993 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:52.993 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:52.993 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:37:52.993 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:37:52.995 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:37:53.129 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.126+0000 7efdc4b31700 1 Processor -- start 2020-11-20T11:37:53.129 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.126+0000 7efdc4b31700 1 -- start start 2020-11-20T11:37:53.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.126+0000 7efdc4b31700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7efdc0061b90 0x7efdc00a92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:53.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.126+0000 7efdc4b31700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdc013b920 0x7efdc01427e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:53.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.126+0000 7efdc4b31700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc00a9c80 con 0x7efdc00a97d0 2020-11-20T11:37:53.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.126+0000 7efdc4b31700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdc0067190 con 0x7efdc0061b90 2020-11-20T11:37:53.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.126+0000 7efdc4b31700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc0067430 con 0x7efdc013b920 2020-11-20T11:37:53.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.127+0000 7efdbe59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7efdc0061b90 0x7efdc00a92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:53.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.127+0000 7efdbdd9b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7efdc00a97d0 0x7efdc013b210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57616/0 (socket says 172.21.15.23:57616) 2020-11-20T11:37:53.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.127+0000 7efdbdd9b700 1 -- 172.21.15.23:0/2410531963 learned_addr learned my addr 172.21.15.23:0/2410531963 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:37:53.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.127+0000 7efdbed9d700 1 --2- 172.21.15.23:0/2410531963 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdc013b920 0x7efdc01427e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:53.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.127+0000 7efdbe59c700 1 -- 172.21.15.23:0/2410531963 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdc013b920 msgr2=0x7efdc01427e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:37:53.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.127+0000 7efdbe59c700 1 --2- 172.21.15.23:0/2410531963 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdc013b920 0x7efdc01427e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:53.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.127+0000 7efdbe59c700 1 -- 172.21.15.23:0/2410531963 >> v1:172.21.15.23:6789/0 conn(0x7efdc00a97d0 legacy=0x7efdc013b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:53.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.127+0000 7efdbe59c700 1 -- 172.21.15.23:0/2410531963 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc0067860 con 0x7efdc0061b90 2020-11-20T11:37:53.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.127+0000 7efdbed9d700 1 --2- 172.21.15.23:0/2410531963 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdc013b920 0x7efdc01427e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:37:53.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.127+0000 7efdbe59c700 1 --2- 172.21.15.23:0/2410531963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdc0061b90 0x7efdc00a92e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7efdb000dc10 tx=0x7efdb0006ec0).ready entity=mon.0 client_cookie=83afd1f889472df4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:53.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.127+0000 7efdbd59a700 1 -- 172.21.15.23:0/2410531963 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb000bb00 con 0x7efdc0061b90 2020-11-20T11:37:53.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.127+0000 7efdbd59a700 1 -- 172.21.15.23:0/2410531963 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb0000e90 con 0x7efdc0061b90 2020-11-20T11:37:53.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.127+0000 7efdbd59a700 1 -- 172.21.15.23:0/2410531963 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb000be70 con 0x7efdc0061b90 2020-11-20T11:37:53.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.128+0000 7efdc4b31700 1 -- 172.21.15.23:0/2410531963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdc0061b90 msgr2=0x7efdc00a92e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:53.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.128+0000 7efdc4b31700 1 --2- 172.21.15.23:0/2410531963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdc0061b90 0x7efdc00a92e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7efdb000dc10 tx=0x7efdb0006ec0).stop 2020-11-20T11:37:53.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.128+0000 7efdc4b31700 1 -- 172.21.15.23:0/2410531963 shutdown_connections 2020-11-20T11:37:53.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.128+0000 7efdc4b31700 1 --2- 172.21.15.23:0/2410531963 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdc013b920 0x7efdc01427e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:53.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.128+0000 7efdc4b31700 1 --2- 172.21.15.23:0/2410531963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdc0061b90 0x7efdc00a92e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:53.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.128+0000 7efdc4b31700 1 -- 172.21.15.23:0/2410531963 >> 172.21.15.23:0/2410531963 conn(0x7efdc0063640 msgr2=0x7efdc00612c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:53.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.128+0000 7efdc4b31700 1 -- 172.21.15.23:0/2410531963 shutdown_connections 2020-11-20T11:37:53.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.128+0000 7efdc4b31700 1 -- 172.21.15.23:0/2410531963 wait complete. 2020-11-20T11:37:53.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.129+0000 7efdc4b31700 1 Processor -- start 2020-11-20T11:37:53.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.129+0000 7efdc4b31700 1 -- start start 2020-11-20T11:37:53.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.130+0000 7efdc4b31700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efdc0061b90 0x7efdc00a7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:53.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.130+0000 7efdc4b31700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdc00a97d0 0x7efdc00a4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:53.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.130+0000 7efdc4b31700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdc013b920 0x7efdc0149b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:53.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.130+0000 7efdc4b31700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc0138fd0 con 0x7efdc013b920 2020-11-20T11:37:53.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.130+0000 7efdc4b31700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc01494f0 con 0x7efdc0061b90 2020-11-20T11:37:53.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.130+0000 7efdc4b31700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc009f900 con 0x7efdc00a97d0 2020-11-20T11:37:53.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.130+0000 7efdbed9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdc013b920 0x7efdc0149b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:53.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.130+0000 7efdbed9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdc013b920 0x7efdc0149b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43374/0 (socket says 172.21.15.23:43374) 2020-11-20T11:37:53.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.130+0000 7efdbed9d700 1 -- 172.21.15.23:0/2260935793 learned_addr learned my addr 172.21.15.23:0/2260935793 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:37:53.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.131+0000 7efdbe59c700 1 --2- 172.21.15.23:0/2260935793 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efdc0061b90 0x7efdc00a7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:53.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.131+0000 7efdbdd9b700 1 --2- 172.21.15.23:0/2260935793 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdc00a97d0 0x7efdc00a4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:53.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.131+0000 7efdbed9d700 1 -- 172.21.15.23:0/2260935793 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdc00a97d0 msgr2=0x7efdc00a4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:53.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.131+0000 7efdbed9d700 1 --2- 172.21.15.23:0/2260935793 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdc00a97d0 0x7efdc00a4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:53.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.131+0000 7efdbed9d700 1 -- 172.21.15.23:0/2260935793 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efdc0061b90 msgr2=0x7efdc00a7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:53.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.131+0000 7efdbed9d700 1 --2- 172.21.15.23:0/2260935793 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efdc0061b90 0x7efdc00a7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:53.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.131+0000 7efdbed9d700 1 -- 172.21.15.23:0/2260935793 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc009fba0 con 0x7efdc013b920 2020-11-20T11:37:53.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.131+0000 7efdbdd9b700 1 --2- 172.21.15.23:0/2260935793 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdc00a97d0 0x7efdc00a4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:37:53.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.131+0000 7efdbe59c700 1 --2- 172.21.15.23:0/2260935793 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efdc0061b90 0x7efdc00a7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:37:53.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.131+0000 7efdbed9d700 1 --2- 172.21.15.23:0/2260935793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdc013b920 0x7efdc0149b50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7efdb800cb20 tx=0x7efdb8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:53.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.131+0000 7efdae7fc700 1 -- 172.21.15.23:0/2260935793 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb80078b0 con 0x7efdc013b920 2020-11-20T11:37:53.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.131+0000 7efdae7fc700 1 -- 172.21.15.23:0/2260935793 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb8007a10 con 0x7efdc013b920 2020-11-20T11:37:53.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.131+0000 7efdc4b31700 1 -- 172.21.15.23:0/2260935793 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc00a3ef0 con 0x7efdc013b920 2020-11-20T11:37:53.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.131+0000 7efdae7fc700 1 -- 172.21.15.23:0/2260935793 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb8007d90 con 0x7efdc013b920 2020-11-20T11:37:53.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.131+0000 7efdc4b31700 1 -- 172.21.15.23:0/2260935793 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc009f460 con 0x7efdc013b920 2020-11-20T11:37:53.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.132+0000 7efdc4b31700 1 --2- 172.21.15.23:0/2260935793 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdc005c070 0x7efdc014a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:53.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.132+0000 7efdc4b31700 1 -- 172.21.15.23:0/2260935793 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc0139760 con 0x7efdc005c070 2020-11-20T11:37:53.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.132+0000 7efdbe59c700 1 --2- 172.21.15.23:0/2260935793 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdc005c070 0x7efdc014a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:53.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.132+0000 7efdae7fc700 1 -- 172.21.15.23:0/2260935793 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7efdb8011620 con 0x7efdc013b920 2020-11-20T11:37:53.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.133+0000 7efdbe59c700 1 -- 172.21.15.23:0/2260935793 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efdc00a4080 con 0x7efdc005c070 2020-11-20T11:37:53.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.133+0000 7efdbe59c700 1 --2- 172.21.15.23:0/2260935793 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdc005c070 0x7efdc014a040 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7efdb0014000 tx=0x7efdb0006ec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:53.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.133+0000 7efdae7fc700 1 --2- 172.21.15.23:0/2260935793 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efd94056d90 0x7efd94059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:53.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.133+0000 7efdae7fc700 1 -- 172.21.15.23:0/2260935793 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7efdb80705b0 con 0x7efdc013b920 2020-11-20T11:37:53.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.133+0000 7efdae7fc700 1 -- 172.21.15.23:0/2260935793 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb001a660 con 0x7efdc005c070 2020-11-20T11:37:53.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.133+0000 7efdbdd9b700 1 --2- 172.21.15.23:0/2260935793 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efd94056d90 0x7efd94059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:53.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.133+0000 7efdae7fc700 1 -- 172.21.15.23:0/2260935793 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7efdb0021040 con 0x7efdc005c070 2020-11-20T11:37:53.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.133+0000 7efdae7fc700 1 -- 172.21.15.23:0/2260935793 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdc005c070 msgr2=0x7efdc014a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:53.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.133+0000 7efdae7fc700 1 --2- 172.21.15.23:0/2260935793 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdc005c070 0x7efdc014a040 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7efdb0014000 tx=0x7efdb0006ec0).stop 2020-11-20T11:37:53.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.133+0000 7efdae7fc700 1 -- 172.21.15.23:0/2260935793 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdc005c070 msgr2=0x7efdc014a040 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:37:53.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.133+0000 7efdae7fc700 1 --2- 172.21.15.23:0/2260935793 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdc005c070 0x7efdc014a040 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:53.148 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.133+0000 7efdbdd9b700 1 --2- 172.21.15.23:0/2260935793 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efd94056d90 0x7efd94059250 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7efdc00a38f0 tx=0x7efdb4009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:53.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.142+0000 7efdc4b31700 1 --2- 172.21.15.23:0/2260935793 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdc00622d0 0x7efdc00a1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:53.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.142+0000 7efdc4b31700 1 -- 172.21.15.23:0/2260935793 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc0044710 con 0x7efdc00622d0 2020-11-20T11:37:53.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.143+0000 7efdbe59c700 1 --2- 172.21.15.23:0/2260935793 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdc00622d0 0x7efdc00a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:53.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.143+0000 7efdbe59c700 1 -- 172.21.15.23:0/2260935793 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7efdc009f6b0 con 0x7efdc00622d0 2020-11-20T11:37:53.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.143+0000 7efdbe59c700 1 --2- 172.21.15.23:0/2260935793 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdc00622d0 0x7efdc00a1e30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7efdc01398a0 tx=0x7efdb0022000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:53.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.143+0000 7efdae7fc700 1 -- 172.21.15.23:0/2260935793 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb001ae30 con 0x7efdc00622d0 2020-11-20T11:37:53.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.144+0000 7efdae7fc700 1 -- 172.21.15.23:0/2260935793 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4274 (secure 0 0 0) 0x7efdb001a450 con 0x7efdc00622d0 2020-11-20T11:37:53.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.144+0000 7efdae7fc700 1 -- 172.21.15.23:0/2260935793 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdc00622d0 msgr2=0x7efdc00a1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:53.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.144+0000 7efdae7fc700 1 --2- 172.21.15.23:0/2260935793 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdc00622d0 0x7efdc00a1e30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7efdc01398a0 tx=0x7efdb0022000).stop 2020-11-20T11:37:53.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.144+0000 7efdae7fc700 1 -- 172.21.15.23:0/2260935793 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdc00622d0 msgr2=0x7efdc00a1e30 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:37:53.152 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.144+0000 7efdae7fc700 1 --2- 172.21.15.23:0/2260935793 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdc00622d0 0x7efdc00a1e30 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0x7efdc01398a0 tx=0x7efdb0022000).stop 2020-11-20T11:37:53.152 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.144+0000 7efdc4b31700 1 -- 172.21.15.23:0/2260935793 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efd94056d90 msgr2=0x7efd94059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:53.152 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.144+0000 7efdc4b31700 1 --2- 172.21.15.23:0/2260935793 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efd94056d90 0x7efd94059250 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7efdc00a38f0 tx=0x7efdb4009040).stop 2020-11-20T11:37:53.152 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.144+0000 7efdc4b31700 1 -- 172.21.15.23:0/2260935793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdc013b920 msgr2=0x7efdc0149b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:53.152 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.144+0000 7efdc4b31700 1 --2- 172.21.15.23:0/2260935793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdc013b920 0x7efdc0149b50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7efdb800cb20 tx=0x7efdb8008040).stop 2020-11-20T11:37:53.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.144+0000 7efdbed9d700 1 -- 172.21.15.23:0/2260935793 reap_dead start 2020-11-20T11:37:53.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.144+0000 7efdbed9d700 1 -- 172.21.15.23:0/2260935793 reap_dead start 2020-11-20T11:37:53.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.144+0000 7efdc4b31700 1 -- 172.21.15.23:0/2260935793 shutdown_connections 2020-11-20T11:37:53.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.144+0000 7efdc4b31700 1 -- 172.21.15.23:0/2260935793 >> 172.21.15.23:0/2260935793 conn(0x7efdc0063640 msgr2=0x7efdc00aa320 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:53.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.144+0000 7efdc4b31700 1 -- 172.21.15.23:0/2260935793 shutdown_connections 2020-11-20T11:37:53.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.144+0000 7efdc4b31700 1 -- 172.21.15.23:0/2260935793 wait complete. 2020-11-20T11:37:53.154 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:37:53.154 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:37:53.154 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:37:53.155 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:37:53.155 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 56, 2020-11-20T11:37:53.155 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:37:53.155 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:37:53.155 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:37:53.156 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:37:53.156 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:53.156 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 13, 2020-11-20T11:37:53.156 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:37:53.156 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:37:53.157 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:37:53.157 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:37:53.157 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:37:53.157 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:37:53.157 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:37:53.158 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:37:53.158 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:37:53.158 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:37:53.158 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:37:53.158 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:53.159 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:37:53.159 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:37:53.159 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:37:53.159 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:37:53.159 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:37:53.159 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:37:53.160 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:37:53.160 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:37:53.160 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:37:53.160 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:37:53.160 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:53.161 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:53.161 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:37:53.161 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:37:53.161 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:37:53.161 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:37:53.162 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:37:53.162 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:37:53.162 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:37:53.162 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:37:53.162 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:37:53.163 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:37:53.163 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:37:53.163 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:37:53.163 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:37:53.163 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:37:53.164 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:37:53.164 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:37:53.164 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:37:53.164 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:37:53.164 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:37:53.165 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:37:53.165 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:37:53.165 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:37:53.165 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:37:53.165 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:53.166 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:37:53.166 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:53.166 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:37:53.166 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:53.166 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:37:53.167 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:37:53.167 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:37:53.167 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:37:53.167 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:53.167 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:37:53.168 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:37:53.168 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:53.168 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:53.168 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:53.168 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:37:53.169 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:37:53.169 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:53.169 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:53.169 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:53.169 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:53.170 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:37:53.170 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:37:53.170 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:37:53.170 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:37:53.170 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:37:53.171 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:53.171 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:53.171 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:37:53.171 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:37:53.171 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:37:53.171 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:37:53.172 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:53.172 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:37:53.172 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:37:53.172 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:53.172 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:53.173 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:53.173 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:37:53.173 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:37:53.173 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:53.173 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:53.174 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:53.174 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:53.174 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:37:53.174 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:37:53.174 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:37:53.175 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:37:53.175 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:37:53.175 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:53.175 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:53.175 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:37:53.176 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:37:53.176 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:37:53.176 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:37:53.176 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:53.176 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:37:53.177 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:37:53.177 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:53.177 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:53.177 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:53.177 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:37:53.178 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:37:53.178 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:37:53.178 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:53.178 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:53.178 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:53.179 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:37:53.179 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:37:53.179 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:37:53.179 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:37:53.179 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:37:53.179 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:53.180 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:53.180 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:53.180 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:37:53.180 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:37:53.180 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:53.181 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:37:53.181 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:37:53.181 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:37:53.181 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:53.181 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:37:53.182 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:37:53.182 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:37:53.182 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:37:53.182 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:37:53.182 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:37:53.183 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:37:53.183 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:37:53.183 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:37:53.183 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:37:53.183 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:37:53.185 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2020-11-20T11:37:53.185 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2020-11-20T11:37:53.185 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2020-11-20T11:37:53.185 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2020-11-20T11:37:53.186 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2020-11-20T11:37:53.186 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T11:37:53.198 INFO:tasks.ceph.mon.c:Stopped 2020-11-20T11:37:53.199 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2020-11-20T11:37:53.199 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:37:53.328 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.325+0000 7f53a16e6700 1 Processor -- start 2020-11-20T11:37:53.329 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.325+0000 7f53a16e6700 1 -- start start 2020-11-20T11:37:53.330 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.326+0000 7f53a16e6700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f539c0aaab0 0x7f539c13ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:53.331 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.326+0000 7f53a16e6700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f539c0aae90 0x7f539c1381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:53.331 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.326+0000 7f53a16e6700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f539c13aeb0 0x7f539c0a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:53.331 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.326+0000 7f53a16e6700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f539c13e600 con 0x7f539c13aeb0 2020-11-20T11:37:53.331 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.326+0000 7f53a16e6700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f539c13e8a0 con 0x7f539c0aaab0 2020-11-20T11:37:53.332 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.326+0000 7f53a16e6700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f539c0ab6d0 con 0x7f539c0aae90 2020-11-20T11:37:53.332 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.326+0000 7f539a7fc700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f539c0aae90 msgr2=0x7f539c1381a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:37:53.332 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.326+0000 7f539b7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f539c13aeb0 0x7f539c0a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:53.332 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.326+0000 7f539a7fc700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f539c0aae90 0x7f539c1381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:37:53.332 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.326+0000 7f539b7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f539c13aeb0 0x7f539c0a8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43384/0 (socket says 172.21.15.23:43384) 2020-11-20T11:37:53.333 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.326+0000 7f539b7fe700 1 -- 172.21.15.23:0/3011136628 learned_addr learned my addr 172.21.15.23:0/3011136628 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:37:53.333 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.326+0000 7f539affd700 1 --2- 172.21.15.23:0/3011136628 >> v2:172.21.15.155:3300/0 conn(0x7f539c0aaab0 0x7f539c13ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:53.333 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.326+0000 7f539b7fe700 1 -- 172.21.15.23:0/3011136628 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f539c0aae90 msgr2=0x7f539c1381a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:37:53.333 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.326+0000 7f539b7fe700 1 --2- 172.21.15.23:0/3011136628 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f539c0aae90 0x7f539c1381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:37:53.333 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.327+0000 7f539b7fe700 1 -- 172.21.15.23:0/3011136628 >> v2:172.21.15.155:3300/0 conn(0x7f539c0aaab0 msgr2=0x7f539c13ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:37:53.334 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.327+0000 7f539b7fe700 1 --2- 172.21.15.23:0/3011136628 >> v2:172.21.15.155:3300/0 conn(0x7f539c0aaab0 0x7f539c13ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:53.334 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.327+0000 7f539b7fe700 1 -- 172.21.15.23:0/3011136628 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f539c13f230 con 0x7f539c13aeb0 2020-11-20T11:37:53.334 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.327+0000 7f539b7fe700 1 --2- 172.21.15.23:0/3011136628 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f539c13aeb0 0x7f539c0a8c60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f539c0b23d0 tx=0x7f53940092d0).ready entity=mon.0 client_cookie=7449b7d4d662e219 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:53.334 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.327+0000 7f5399ffb700 1 -- 172.21.15.23:0/3011136628 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539400bbf0 con 0x7f539c13aeb0 2020-11-20T11:37:53.334 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.327+0000 7f5399ffb700 1 -- 172.21.15.23:0/3011136628 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f539400bd50 con 0x7f539c13aeb0 2020-11-20T11:37:53.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.327+0000 7f5399ffb700 1 -- 172.21.15.23:0/3011136628 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5394011610 con 0x7f539c13aeb0 2020-11-20T11:37:53.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.327+0000 7f53a16e6700 1 -- 172.21.15.23:0/3011136628 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f539c13aeb0 msgr2=0x7f539c0a8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:53.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.327+0000 7f53a16e6700 1 --2- 172.21.15.23:0/3011136628 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f539c13aeb0 0x7f539c0a8c60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f539c0b23d0 tx=0x7f53940092d0).stop 2020-11-20T11:37:53.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.327+0000 7f53a16e6700 1 -- 172.21.15.23:0/3011136628 shutdown_connections 2020-11-20T11:37:53.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.327+0000 7f53a16e6700 1 --2- 172.21.15.23:0/3011136628 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f539c13aeb0 0x7f539c0a8c60 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:53.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.327+0000 7f53a16e6700 1 --2- 172.21.15.23:0/3011136628 >> v2:172.21.15.155:3300/0 conn(0x7f539c0aaab0 0x7f539c13ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:53.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.327+0000 7f53a16e6700 1 --2- 172.21.15.23:0/3011136628 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f539c0aae90 0x7f539c1381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:37:53.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.327+0000 7f53a16e6700 1 -- 172.21.15.23:0/3011136628 >> 172.21.15.23:0/3011136628 conn(0x7f539c0678e0 msgr2=0x7f539c067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:53.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.327+0000 7f53a16e6700 1 -- 172.21.15.23:0/3011136628 shutdown_connections 2020-11-20T11:37:53.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.327+0000 7f53a16e6700 1 -- 172.21.15.23:0/3011136628 wait complete. 2020-11-20T11:37:53.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.329+0000 7f53a16e6700 1 Processor -- start 2020-11-20T11:37:53.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.329+0000 7f53a16e6700 1 -- start start 2020-11-20T11:37:53.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.330+0000 7f53a16e6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f539c0aaab0 0x7f539c09ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:53.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.330+0000 7f53a16e6700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f539c0aae90 0x7f539c0a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:53.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.330+0000 7f53a16e6700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f539c14a030 0x7f539c13d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:53.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.330+0000 7f53a16e6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f539c14afe0 con 0x7f539c0aaab0 2020-11-20T11:37:53.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.330+0000 7f53a16e6700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f539c0a5210 con 0x7f539c14a030 2020-11-20T11:37:53.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.330+0000 7f53a16e6700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f539c0a54b0 con 0x7f539c0aae90 2020-11-20T11:37:53.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.330+0000 7f539affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f539c0aaab0 0x7f539c09ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:53.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.330+0000 7f539a7fc700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f539c0aae90 msgr2=0x7f539c0a63a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:37:53.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.330+0000 7f539a7fc700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f539c0aae90 0x7f539c0a63a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:37:53.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.330+0000 7f539affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f539c0aaab0 0x7f539c09ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43386/0 (socket says 172.21.15.23:43386) 2020-11-20T11:37:53.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.330+0000 7f539affd700 1 -- 172.21.15.23:0/1897346379 learned_addr learned my addr 172.21.15.23:0/1897346379 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:37:53.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.330+0000 7f539b7fe700 1 --2- 172.21.15.23:0/1897346379 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f539c14a030 0x7f539c13d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:53.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.330+0000 7f539affd700 1 -- 172.21.15.23:0/1897346379 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f539c0aae90 msgr2=0x7f539c0a63a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:37:53.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.330+0000 7f539affd700 1 --2- 172.21.15.23:0/1897346379 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f539c0aae90 0x7f539c0a63a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:37:53.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.330+0000 7f539affd700 1 -- 172.21.15.23:0/1897346379 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f539c14a030 msgr2=0x7f539c13d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:53.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.330+0000 7f539affd700 1 --2- 172.21.15.23:0/1897346379 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f539c14a030 0x7f539c13d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:53.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.330+0000 7f539affd700 1 -- 172.21.15.23:0/1897346379 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f539c0a5750 con 0x7f539c0aaab0 2020-11-20T11:37:53.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.330+0000 7f539b7fe700 1 --2- 172.21.15.23:0/1897346379 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f539c14a030 0x7f539c13d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:37:53.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.330+0000 7f539affd700 1 --2- 172.21.15.23:0/1897346379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f539c0aaab0 0x7f539c09ddb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f538c00eba0 tx=0x7f538c006e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:53.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.330+0000 7f5382ffd700 1 -- 172.21.15.23:0/1897346379 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538c00c9d0 con 0x7f539c0aaab0 2020-11-20T11:37:53.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.331+0000 7f5382ffd700 1 -- 172.21.15.23:0/1897346379 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538c00cb30 con 0x7f539c0aaab0 2020-11-20T11:37:53.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.331+0000 7f53a16e6700 1 -- 172.21.15.23:0/1897346379 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f539c0a4e20 con 0x7f539c0aaab0 2020-11-20T11:37:53.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.331+0000 7f5382ffd700 1 -- 172.21.15.23:0/1897346379 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538c0123f0 con 0x7f539c0aaab0 2020-11-20T11:37:53.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.331+0000 7f53a16e6700 1 -- 172.21.15.23:0/1897346379 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f539c0a5bf0 con 0x7f539c0aaab0 2020-11-20T11:37:53.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.332+0000 7f53a16e6700 1 -- 172.21.15.23:0/1897346379 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f539c05c7d0 con 0x7f539c0aaab0 2020-11-20T11:37:53.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.332+0000 7f5382ffd700 1 -- 172.21.15.23:0/1897346379 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f538c0127a0 con 0x7f539c0aaab0 2020-11-20T11:37:53.345 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.332+0000 7f5382ffd700 1 --2- 172.21.15.23:0/1897346379 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5378056d10 0x7f53780591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:53.345 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.332+0000 7f5382ffd700 1 -- 172.21.15.23:0/1897346379 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f538c070480 con 0x7f539c0aaab0 2020-11-20T11:37:53.345 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.333+0000 7f539a7fc700 1 --2- 172.21.15.23:0/1897346379 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5378056d10 0x7f53780591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:53.345 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.333+0000 7f539a7fc700 1 --2- 172.21.15.23:0/1897346379 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5378056d10 0x7f53780591d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f5390006fd0 tx=0x7f5390006b20).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:53.345 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.336+0000 7f5382ffd700 1 -- 172.21.15.23:0/1897346379 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f538c045770 con 0x7f539c0aaab0 2020-11-20T11:37:53.605 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.602+0000 7f53a16e6700 1 -- 172.21.15.23:0/1897346379 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f539c0a58e0 con 0x7f539c0aaab0 2020-11-20T11:37:53.605 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.602+0000 7f5382ffd700 1 -- 172.21.15.23:0/1897346379 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7f538c048d90 con 0x7f539c0aaab0 2020-11-20T11:37:53.606 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:37:53.608 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.604+0000 7f53a16e6700 1 -- 172.21.15.23:0/1897346379 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5378056d10 msgr2=0x7f53780591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:53.608 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.604+0000 7f53a16e6700 1 --2- 172.21.15.23:0/1897346379 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5378056d10 0x7f53780591d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f5390006fd0 tx=0x7f5390006b20).stop 2020-11-20T11:37:53.608 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.604+0000 7f53a16e6700 1 -- 172.21.15.23:0/1897346379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f539c0aaab0 msgr2=0x7f539c09ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:53.608 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.604+0000 7f53a16e6700 1 --2- 172.21.15.23:0/1897346379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f539c0aaab0 0x7f539c09ddb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f538c00eba0 tx=0x7f538c006e30).stop 2020-11-20T11:37:53.609 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.604+0000 7f53a16e6700 1 -- 172.21.15.23:0/1897346379 shutdown_connections 2020-11-20T11:37:53.609 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.604+0000 7f53a16e6700 1 --2- 172.21.15.23:0/1897346379 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f539c14a030 0x7f539c13d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:53.609 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.604+0000 7f53a16e6700 1 --2- 172.21.15.23:0/1897346379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f539c0aaab0 0x7f539c09ddb0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:53.609 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.604+0000 7f53a16e6700 1 --2- 172.21.15.23:0/1897346379 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5378056d10 0x7f53780591d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:53.609 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.604+0000 7f53a16e6700 1 --2- 172.21.15.23:0/1897346379 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f539c0aae90 0x7f539c0a63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:37:53.610 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.604+0000 7f53a16e6700 1 -- 172.21.15.23:0/1897346379 >> 172.21.15.23:0/1897346379 conn(0x7f539c0678e0 msgr2=0x7f539c0ae220 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:53.610 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.605+0000 7f53a16e6700 1 -- 172.21.15.23:0/1897346379 shutdown_connections 2020-11-20T11:37:53.611 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:53.605+0000 7f53a16e6700 1 -- 172.21.15.23:0/1897346379 wait complete. 2020-11-20T11:37:56.621 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:37:56.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.742+0000 7f0944f7e700 1 Processor -- start 2020-11-20T11:37:56.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.742+0000 7f0944f7e700 1 -- start start 2020-11-20T11:37:56.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.743+0000 7f0944f7e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f09400676a0 0x7f09400a89c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:56.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.743+0000 7f0944f7e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f09400ab440 0x7f09400a8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:56.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.743+0000 7f0944f7e700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0940138360 con 0x7f0940065e50 2020-11-20T11:37:56.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.743+0000 7f0944f7e700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0940067d10 con 0x7f09400676a0 2020-11-20T11:37:56.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.743+0000 7f0944f7e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0940066700 con 0x7f09400ab440 2020-11-20T11:37:56.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.743+0000 7f093ffff700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f09400ab440 msgr2=0x7f09400a8eb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:37:56.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.743+0000 7f093ffff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f09400ab440 0x7f09400a8eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:37:56.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.743+0000 7f093f7fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f09400676a0 0x7f09400a89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:56.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.743+0000 7f093f7fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f09400676a0 0x7f09400a89c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:36300/0 (socket says 172.21.15.23:36300) 2020-11-20T11:37:56.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.743+0000 7f093f7fe700 1 -- 172.21.15.23:0/300716528 learned_addr learned my addr 172.21.15.23:0/300716528 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:37:56.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.743+0000 7f093f7fe700 1 -- 172.21.15.23:0/300716528 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f09400ab440 msgr2=0x7f09400a8eb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:37:56.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.743+0000 7f093f7fe700 1 --2- 172.21.15.23:0/300716528 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f09400ab440 0x7f09400a8eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:37:56.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.744+0000 7f093f7fe700 1 -- 172.21.15.23:0/300716528 >> v1:172.21.15.155:6789/0 conn(0x7f0940065e50 legacy=0x7f09400aad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:56.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.744+0000 7f093f7fe700 1 -- 172.21.15.23:0/300716528 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0940138600 con 0x7f09400676a0 2020-11-20T11:37:56.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.744+0000 7f093f7fe700 1 --2- 172.21.15.23:0/300716528 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09400676a0 0x7f09400a89c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f0930009a10 tx=0x7f0930006ef0).ready entity=mon.1 client_cookie=2b711e38f12a9756 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:56.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.744+0000 7f093e7fc700 1 -- 172.21.15.23:0/300716528 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093000b860 con 0x7f09400676a0 2020-11-20T11:37:56.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.744+0000 7f093e7fc700 1 -- 172.21.15.23:0/300716528 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0930000e90 con 0x7f09400676a0 2020-11-20T11:37:56.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.744+0000 7f093e7fc700 1 -- 172.21.15.23:0/300716528 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093000bbd0 con 0x7f09400676a0 2020-11-20T11:37:56.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.744+0000 7f0944f7e700 1 -- 172.21.15.23:0/300716528 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09400676a0 msgr2=0x7f09400a89c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:56.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.744+0000 7f0944f7e700 1 --2- 172.21.15.23:0/300716528 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09400676a0 0x7f09400a89c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f0930009a10 tx=0x7f0930006ef0).stop 2020-11-20T11:37:56.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.744+0000 7f0944f7e700 1 -- 172.21.15.23:0/300716528 shutdown_connections 2020-11-20T11:37:56.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.744+0000 7f0944f7e700 1 --2- 172.21.15.23:0/300716528 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f09400676a0 0x7f09400a89c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:56.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.744+0000 7f0944f7e700 1 --2- 172.21.15.23:0/300716528 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f09400ab440 0x7f09400a8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:37:56.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.744+0000 7f0944f7e700 1 -- 172.21.15.23:0/300716528 >> 172.21.15.23:0/300716528 conn(0x7f09400612c0 msgr2=0x7f09400616d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:56.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.744+0000 7f0944f7e700 1 -- 172.21.15.23:0/300716528 shutdown_connections 2020-11-20T11:37:56.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.744+0000 7f0944f7e700 1 -- 172.21.15.23:0/300716528 wait complete. 2020-11-20T11:37:56.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.746+0000 7f0944f7e700 1 Processor -- start 2020-11-20T11:37:56.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.746+0000 7f0944f7e700 1 -- start start 2020-11-20T11:37:56.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.746+0000 7f0944f7e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0940065e50 0x7f09400a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:56.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.746+0000 7f0944f7e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f09400676a0 0x7f09400a71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:56.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.746+0000 7f0944f7e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09400ab440 0x7f09400a76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:56.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.746+0000 7f0944f7e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09400a7bd0 con 0x7f09400ab440 2020-11-20T11:37:56.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.746+0000 7f0944f7e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09400a48b0 con 0x7f0940065e50 2020-11-20T11:37:56.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.746+0000 7f0944f7e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09400a4b50 con 0x7f09400676a0 2020-11-20T11:37:56.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.746+0000 7f093effd700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f09400676a0 msgr2=0x7f09400a71f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:37:56.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.747+0000 7f093effd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f09400676a0 0x7f09400a71f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:37:56.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.747+0000 7f093ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09400ab440 0x7f09400a76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:56.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.747+0000 7f093ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09400ab440 0x7f09400a76e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43414/0 (socket says 172.21.15.23:43414) 2020-11-20T11:37:56.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.747+0000 7f093ffff700 1 -- 172.21.15.23:0/154262548 learned_addr learned my addr 172.21.15.23:0/154262548 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:37:56.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.747+0000 7f093f7fe700 1 --2- 172.21.15.23:0/154262548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0940065e50 0x7f09400a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:56.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.747+0000 7f093ffff700 1 -- 172.21.15.23:0/154262548 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f09400676a0 msgr2=0x7f09400a71f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:37:56.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.747+0000 7f093ffff700 1 --2- 172.21.15.23:0/154262548 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f09400676a0 0x7f09400a71f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:37:56.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.747+0000 7f093ffff700 1 -- 172.21.15.23:0/154262548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0940065e50 msgr2=0x7f09400a60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:56.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.747+0000 7f093ffff700 1 --2- 172.21.15.23:0/154262548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0940065e50 0x7f09400a60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:56.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.747+0000 7f093ffff700 1 -- 172.21.15.23:0/154262548 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09400a53e0 con 0x7f09400ab440 2020-11-20T11:37:56.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.747+0000 7f093ffff700 1 --2- 172.21.15.23:0/154262548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09400ab440 0x7f09400a76e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f093800bab0 tx=0x7f09380091a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:56.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.747+0000 7f091f7fe700 1 -- 172.21.15.23:0/154262548 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0938003c50 con 0x7f09400ab440 2020-11-20T11:37:56.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.747+0000 7f091f7fe700 1 -- 172.21.15.23:0/154262548 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0938003dd0 con 0x7f09400ab440 2020-11-20T11:37:56.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.747+0000 7f0944f7e700 1 -- 172.21.15.23:0/154262548 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09400a43a0 con 0x7f09400ab440 2020-11-20T11:37:56.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.747+0000 7f091f7fe700 1 -- 172.21.15.23:0/154262548 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0938006170 con 0x7f09400ab440 2020-11-20T11:37:56.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.747+0000 7f0944f7e700 1 -- 172.21.15.23:0/154262548 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09400a2bf0 con 0x7f09400ab440 2020-11-20T11:37:56.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.748+0000 7f0944f7e700 1 -- 172.21.15.23:0/154262548 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09400a4d10 con 0x7f09400ab440 2020-11-20T11:37:56.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.749+0000 7f091f7fe700 1 -- 172.21.15.23:0/154262548 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0938006560 con 0x7f09400ab440 2020-11-20T11:37:56.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.749+0000 7f091f7fe700 1 --2- 172.21.15.23:0/154262548 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0918056d10 0x7f09180591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:37:56.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.749+0000 7f091f7fe700 1 -- 172.21.15.23:0/154262548 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f0938011020 con 0x7f09400ab440 2020-11-20T11:37:56.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.749+0000 7f093f7fe700 1 --2- 172.21.15.23:0/154262548 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0918056d10 0x7f09180591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:37:56.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.750+0000 7f093f7fe700 1 --2- 172.21.15.23:0/154262548 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0918056d10 0x7f09180591d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f0930006f30 tx=0x7f0930016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:37:56.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:56.753+0000 7f091f7fe700 1 -- 172.21.15.23:0/154262548 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0938044d40 con 0x7f09400ab440 2020-11-20T11:37:57.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:57.010+0000 7f0944f7e700 1 -- 172.21.15.23:0/154262548 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f09400a1930 con 0x7f09400ab440 2020-11-20T11:37:57.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:57.010+0000 7f091f7fe700 1 -- 172.21.15.23:0/154262548 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7f0938048360 con 0x7f09400ab440 2020-11-20T11:37:57.014 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:37:57.015 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:57.012+0000 7f0944f7e700 1 -- 172.21.15.23:0/154262548 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0918056d10 msgr2=0x7f09180591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:57.015 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:57.012+0000 7f0944f7e700 1 --2- 172.21.15.23:0/154262548 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0918056d10 0x7f09180591d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f0930006f30 tx=0x7f0930016000).stop 2020-11-20T11:37:57.015 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:57.012+0000 7f0944f7e700 1 -- 172.21.15.23:0/154262548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09400ab440 msgr2=0x7f09400a76e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:37:57.016 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:57.012+0000 7f0944f7e700 1 --2- 172.21.15.23:0/154262548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09400ab440 0x7f09400a76e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f093800bab0 tx=0x7f09380091a0).stop 2020-11-20T11:37:57.016 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:57.012+0000 7f0944f7e700 1 -- 172.21.15.23:0/154262548 shutdown_connections 2020-11-20T11:37:57.016 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:57.012+0000 7f0944f7e700 1 --2- 172.21.15.23:0/154262548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09400ab440 0x7f09400a76e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:57.016 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:57.012+0000 7f0944f7e700 1 --2- 172.21.15.23:0/154262548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0940065e50 0x7f09400a60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:57.016 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:57.012+0000 7f0944f7e700 1 --2- 172.21.15.23:0/154262548 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0918056d10 0x7f09180591d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:37:57.017 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:57.012+0000 7f0944f7e700 1 --2- 172.21.15.23:0/154262548 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f09400676a0 0x7f09400a71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:37:57.017 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:57.012+0000 7f0944f7e700 1 -- 172.21.15.23:0/154262548 >> 172.21.15.23:0/154262548 conn(0x7f09400612c0 msgr2=0x7f09400616a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:37:57.017 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:57.012+0000 7f0944f7e700 1 -- 172.21.15.23:0/154262548 shutdown_connections 2020-11-20T11:37:57.017 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:37:57.012+0000 7f0944f7e700 1 -- 172.21.15.23:0/154262548 wait complete. 2020-11-20T11:38:00.027 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:38:00.152 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.149+0000 7f6293326700 1 Processor -- start 2020-11-20T11:38:00.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.149+0000 7f6293326700 1 -- start start 2020-11-20T11:38:00.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.150+0000 7f6293326700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f628c0ad2f0 0x7f628c13af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:00.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.150+0000 7f6293326700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628c0ad860 con 0x7f628c0a9040 2020-11-20T11:38:00.155 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.150+0000 7f6293326700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628c0aaa90 con 0x7f628c0ace70 2020-11-20T11:38:00.155 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.150+0000 7f6293326700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f628c0aa8a0 con 0x7f628c0ad2f0 2020-11-20T11:38:00.155 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.150+0000 7f6292324700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f628c0a9040 0x7f628c0aaf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57670/0 (socket says 172.21.15.23:57670) 2020-11-20T11:38:00.155 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.150+0000 7f6292324700 1 -- 172.21.15.23:0/3983538117 learned_addr learned my addr 172.21.15.23:0/3983538117 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:38:00.155 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.151+0000 7f6292b25700 1 --2- 172.21.15.23:0/3983538117 >> v2:172.21.15.155:3300/0 conn(0x7f628c0ad2f0 0x7f628c13af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:00.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.151+0000 7f6292b25700 1 -- 172.21.15.23:0/3983538117 >> v1:172.21.15.155:6789/0 conn(0x7f628c0ace70 legacy=0x7f628c13a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:00.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.151+0000 7f6292b25700 1 -- 172.21.15.23:0/3983538117 >> v1:172.21.15.23:6789/0 conn(0x7f628c0a9040 legacy=0x7f628c0aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:00.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.151+0000 7f6292b25700 1 -- 172.21.15.23:0/3983538117 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f628c0a9420 con 0x7f628c0ad2f0 2020-11-20T11:38:00.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.151+0000 7f6292b25700 1 --2- 172.21.15.23:0/3983538117 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f628c0ad2f0 0x7f628c13af40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f6284009ce0 tx=0x7f6284009220).ready entity=mon.1 client_cookie=143df0430bda12c8 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:00.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.151+0000 7f6291322700 1 -- 172.21.15.23:0/3983538117 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628400bbf0 con 0x7f628c0ad2f0 2020-11-20T11:38:00.157 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.151+0000 7f6291322700 1 -- 172.21.15.23:0/3983538117 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f628400bd50 con 0x7f628c0ad2f0 2020-11-20T11:38:00.157 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.151+0000 7f6291322700 1 -- 172.21.15.23:0/3983538117 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6284011610 con 0x7f628c0ad2f0 2020-11-20T11:38:00.160 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.152+0000 7f6293326700 1 -- 172.21.15.23:0/3983538117 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f628c0ad2f0 msgr2=0x7f628c13af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:00.160 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.152+0000 7f6293326700 1 --2- 172.21.15.23:0/3983538117 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f628c0ad2f0 0x7f628c13af40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f6284009ce0 tx=0x7f6284009220).stop 2020-11-20T11:38:00.160 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.152+0000 7f6293326700 1 -- 172.21.15.23:0/3983538117 shutdown_connections 2020-11-20T11:38:00.160 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.152+0000 7f6293326700 1 --2- 172.21.15.23:0/3983538117 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f628c0ad2f0 0x7f628c13af40 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:00.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.152+0000 7f6293326700 1 -- 172.21.15.23:0/3983538117 >> 172.21.15.23:0/3983538117 conn(0x7f628c0683b0 msgr2=0x7f628c0b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:00.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.152+0000 7f6293326700 1 -- 172.21.15.23:0/3983538117 shutdown_connections 2020-11-20T11:38:00.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.152+0000 7f6293326700 1 -- 172.21.15.23:0/3983538117 wait complete. 2020-11-20T11:38:00.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.153+0000 7f6293326700 1 Processor -- start 2020-11-20T11:38:00.162 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.153+0000 7f6293326700 1 -- start start 2020-11-20T11:38:00.162 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.153+0000 7f6293326700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f628c0a9040 0x7f628c0a1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:00.162 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.153+0000 7f6293326700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f628c0ace70 0x7f628c09fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:00.162 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.153+0000 7f6293326700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f628c0ad2f0 0x7f628c1499f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:00.162 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.154+0000 7f6293326700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628c09f310 con 0x7f628c0ad2f0 2020-11-20T11:38:00.163 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.154+0000 7f6293326700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628c09e930 con 0x7f628c0ace70 2020-11-20T11:38:00.163 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.154+0000 7f6293326700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f628c09ebd0 con 0x7f628c0a9040 2020-11-20T11:38:00.163 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.154+0000 7f6292b25700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f628c0ad2f0 0x7f628c1499f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:00.163 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.154+0000 7f6292b25700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f628c0ad2f0 0x7f628c1499f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43428/0 (socket says 172.21.15.23:43428) 2020-11-20T11:38:00.163 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.154+0000 7f6292b25700 1 -- 172.21.15.23:0/2277025083 learned_addr learned my addr 172.21.15.23:0/2277025083 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:38:00.164 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.154+0000 7f6292324700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f628c0a9040 msgr2=0x7f628c0a1820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:38:00.164 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.154+0000 7f6292324700 1 --2- 172.21.15.23:0/2277025083 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f628c0a9040 0x7f628c0a1820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:38:00.164 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.154+0000 7f6291b23700 1 --2- 172.21.15.23:0/2277025083 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f628c0ace70 0x7f628c09fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:00.164 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.154+0000 7f6292b25700 1 -- 172.21.15.23:0/2277025083 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f628c0a9040 msgr2=0x7f628c0a1820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:38:00.164 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.154+0000 7f6292b25700 1 --2- 172.21.15.23:0/2277025083 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f628c0a9040 0x7f628c0a1820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:38:00.165 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.154+0000 7f6292b25700 1 -- 172.21.15.23:0/2277025083 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f628c0ace70 msgr2=0x7f628c09fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:00.165 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.154+0000 7f6292b25700 1 --2- 172.21.15.23:0/2277025083 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f628c0ace70 0x7f628c09fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:00.165 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.154+0000 7f6292b25700 1 -- 172.21.15.23:0/2277025083 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f628c09ee70 con 0x7f628c0ad2f0 2020-11-20T11:38:00.165 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.154+0000 7f6291b23700 1 --2- 172.21.15.23:0/2277025083 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f628c0ace70 0x7f628c09fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:38:00.166 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.154+0000 7f6292b25700 1 --2- 172.21.15.23:0/2277025083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f628c0ad2f0 0x7f628c1499f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f628c0ad790 tx=0x7f6284016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:00.166 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.154+0000 7f627a7fc700 1 -- 172.21.15.23:0/2277025083 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628400bec0 con 0x7f628c0ad2f0 2020-11-20T11:38:00.166 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.154+0000 7f627a7fc700 1 -- 172.21.15.23:0/2277025083 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6284009e20 con 0x7f628c0ad2f0 2020-11-20T11:38:00.166 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.154+0000 7f6293326700 1 -- 172.21.15.23:0/2277025083 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f628c09e5b0 con 0x7f628c0ad2f0 2020-11-20T11:38:00.166 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.154+0000 7f627a7fc700 1 -- 172.21.15.23:0/2277025083 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6284010e10 con 0x7f628c0ad2f0 2020-11-20T11:38:00.167 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.154+0000 7f6293326700 1 -- 172.21.15.23:0/2277025083 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f628c149ee0 con 0x7f628c0ad2f0 2020-11-20T11:38:00.167 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.155+0000 7f6293326700 1 -- 172.21.15.23:0/2277025083 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f628c057dc0 con 0x7f628c0ad2f0 2020-11-20T11:38:00.167 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.156+0000 7f627a7fc700 1 -- 172.21.15.23:0/2277025083 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6284017070 con 0x7f628c0ad2f0 2020-11-20T11:38:00.167 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.156+0000 7f627a7fc700 1 --2- 172.21.15.23:0/2277025083 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6264056d10 0x7f62640591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:00.167 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.156+0000 7f627a7fc700 1 -- 172.21.15.23:0/2277025083 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f62840725d0 con 0x7f628c0ad2f0 2020-11-20T11:38:00.168 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.156+0000 7f6292324700 1 --2- 172.21.15.23:0/2277025083 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6264056d10 0x7f62640591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:00.168 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.157+0000 7f6292324700 1 --2- 172.21.15.23:0/2277025083 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6264056d10 0x7f62640591d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f6288000fa0 tx=0x7f62880077b0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:00.168 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.160+0000 7f627a7fc700 1 -- 172.21.15.23:0/2277025083 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f62840478c0 con 0x7f628c0ad2f0 2020-11-20T11:38:00.408 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.405+0000 7f6293326700 1 -- 172.21.15.23:0/2277025083 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f628c09e7a0 con 0x7f628c0ad2f0 2020-11-20T11:38:00.408 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.405+0000 7f627a7fc700 1 -- 172.21.15.23:0/2277025083 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7f6284015020 con 0x7f628c0ad2f0 2020-11-20T11:38:00.408 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:38:00.410 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.407+0000 7f6293326700 1 -- 172.21.15.23:0/2277025083 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6264056d10 msgr2=0x7f62640591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:00.410 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.407+0000 7f6293326700 1 --2- 172.21.15.23:0/2277025083 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6264056d10 0x7f62640591d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f6288000fa0 tx=0x7f62880077b0).stop 2020-11-20T11:38:00.411 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.407+0000 7f6293326700 1 -- 172.21.15.23:0/2277025083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f628c0ad2f0 msgr2=0x7f628c1499f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:00.411 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.407+0000 7f6293326700 1 --2- 172.21.15.23:0/2277025083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f628c0ad2f0 0x7f628c1499f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f628c0ad790 tx=0x7f6284016000).stop 2020-11-20T11:38:00.411 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.407+0000 7f6293326700 1 -- 172.21.15.23:0/2277025083 shutdown_connections 2020-11-20T11:38:00.411 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.407+0000 7f6293326700 1 --2- 172.21.15.23:0/2277025083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f628c0ad2f0 0x7f628c1499f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:00.411 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.407+0000 7f6293326700 1 --2- 172.21.15.23:0/2277025083 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6264056d10 0x7f62640591d0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:00.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.407+0000 7f6293326700 1 --2- 172.21.15.23:0/2277025083 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f628c0a9040 0x7f628c0a1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:38:00.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.407+0000 7f6293326700 1 --2- 172.21.15.23:0/2277025083 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f628c0ace70 0x7f628c09fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:00.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.407+0000 7f6293326700 1 -- 172.21.15.23:0/2277025083 >> 172.21.15.23:0/2277025083 conn(0x7f628c0683b0 msgr2=0x7f628c0ac330 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:00.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.407+0000 7f6293326700 1 -- 172.21.15.23:0/2277025083 shutdown_connections 2020-11-20T11:38:00.413 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:00.407+0000 7f6293326700 1 -- 172.21.15.23:0/2277025083 wait complete. 2020-11-20T11:38:00.635 INFO:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:38:00.637 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:38:03.423 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:38:03.560 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:03.557+0000 7f6166e7e700 1 Processor -- start 2020-11-20T11:38:03.561 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:03.557+0000 7f6166e7e700 1 -- start start 2020-11-20T11:38:03.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:03.558+0000 7f6166e7e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f616005bc80 0x7f61600a9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:03.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:03.558+0000 7f6166e7e700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6160065780 0x7f61600601b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:03.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:03.558+0000 7f6166e7e700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6160067430 con 0x7f61600a9cf0 2020-11-20T11:38:03.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:03.558+0000 7f6166e7e700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61600a8ca0 con 0x7f6160065780 2020-11-20T11:38:03.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:03.558+0000 7f6166e7e700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f616005c060 con 0x7f616005bc80 2020-11-20T11:38:03.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:03.558+0000 7f615ffff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6160065780 0x7f61600601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:03.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:03.558+0000 7f615ffff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6160065780 0x7f61600601b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43436/0 (socket says 172.21.15.23:43436) 2020-11-20T11:38:03.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:03.558+0000 7f615ffff700 1 -- 172.21.15.23:0/2348130241 learned_addr learned my addr 172.21.15.23:0/2348130241 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:38:03.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:03.558+0000 7f6164c1a700 1 --2- 172.21.15.23:0/2348130241 >> v2:172.21.15.155:3300/0 conn(0x7f616005bc80 0x7f61600a9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:03.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:03.558+0000 7f615ffff700 1 -- 172.21.15.23:0/2348130241 >> v2:172.21.15.23:3300/0 conn(0x7f6160065780 msgr2=0x7f61600601b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:38:03.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:03.558+0000 7f615ffff700 1 -- 172.21.15.23:0/2348130241 >> v2:172.21.15.23:3300/0 conn(0x7f6160065780 msgr2=0x7f61600601b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:38:03.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:03.558+0000 7f615ffff700 1 --2- 172.21.15.23:0/2348130241 >> v2:172.21.15.23:3300/0 conn(0x7f6160065780 0x7f61600601b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:38:03.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:03.558+0000 7f615ffff700 1 --2- 172.21.15.23:0/2348130241 >> v2:172.21.15.23:3300/0 conn(0x7f6160065780 0x7f61600601b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:38:03.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:03.559+0000 7f6164c1a700 1 -- 172.21.15.23:0/2348130241 >> v2:172.21.15.155:3300/0 conn(0x7f616005bc80 msgr2=0x7f61600a9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:38:03.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:03.559+0000 7f6164c1a700 1 -- 172.21.15.23:0/2348130241 >> v2:172.21.15.155:3300/0 conn(0x7f616005bc80 msgr2=0x7f61600a9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:38:03.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:03.559+0000 7f6164c1a700 1 --2- 172.21.15.23:0/2348130241 >> v2:172.21.15.155:3300/0 conn(0x7f616005bc80 0x7f61600a9140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:38:03.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:03.559+0000 7f6164c1a700 1 --2- 172.21.15.23:0/2348130241 >> v2:172.21.15.155:3300/0 conn(0x7f616005bc80 0x7f61600a9140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:38:03.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:03.759+0000 7f615ffff700 1 --2- 172.21.15.23:0/2348130241 >> v2:172.21.15.23:3300/0 conn(0x7f6160065780 0x7f61600601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:03.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:03.759+0000 7f615ffff700 1 -- 172.21.15.23:0/2348130241 >> v2:172.21.15.23:3300/0 conn(0x7f6160065780 msgr2=0x7f61600601b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:38:03.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:03.759+0000 7f615ffff700 1 -- 172.21.15.23:0/2348130241 >> v2:172.21.15.23:3300/0 conn(0x7f6160065780 msgr2=0x7f61600601b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:38:03.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:03.759+0000 7f615ffff700 1 --2- 172.21.15.23:0/2348130241 >> v2:172.21.15.23:3300/0 conn(0x7f6160065780 0x7f61600601b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:38:03.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:03.760+0000 7f615ffff700 1 --2- 172.21.15.23:0/2348130241 >> v2:172.21.15.23:3300/0 conn(0x7f6160065780 0x7f61600601b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:38:03.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:03.760+0000 7f6164c1a700 1 --2- 172.21.15.23:0/2348130241 >> v2:172.21.15.155:3300/0 conn(0x7f616005bc80 0x7f61600a9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:03.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:03.760+0000 7f6164c1a700 1 -- 172.21.15.23:0/2348130241 >> v2:172.21.15.155:3300/0 conn(0x7f616005bc80 msgr2=0x7f61600a9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:38:03.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:03.760+0000 7f6164c1a700 1 -- 172.21.15.23:0/2348130241 >> v2:172.21.15.155:3300/0 conn(0x7f616005bc80 msgr2=0x7f61600a9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:38:03.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:03.760+0000 7f6164c1a700 1 --2- 172.21.15.23:0/2348130241 >> v2:172.21.15.155:3300/0 conn(0x7f616005bc80 0x7f61600a9140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:38:03.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:03.760+0000 7f6164c1a700 1 --2- 172.21.15.23:0/2348130241 >> v2:172.21.15.155:3300/0 conn(0x7f616005bc80 0x7f61600a9140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:38:04.164 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:04.161+0000 7f615ffff700 1 --2- 172.21.15.23:0/2348130241 >> v2:172.21.15.23:3300/0 conn(0x7f6160065780 0x7f61600601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:04.165 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:04.161+0000 7f615ffff700 1 -- 172.21.15.23:0/2348130241 >> v2:172.21.15.23:3300/0 conn(0x7f6160065780 msgr2=0x7f61600601b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:38:04.165 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:04.161+0000 7f615ffff700 1 -- 172.21.15.23:0/2348130241 >> v2:172.21.15.23:3300/0 conn(0x7f6160065780 msgr2=0x7f61600601b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:38:04.165 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:04.161+0000 7f615ffff700 1 --2- 172.21.15.23:0/2348130241 >> v2:172.21.15.23:3300/0 conn(0x7f6160065780 0x7f61600601b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:38:04.165 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:04.161+0000 7f615ffff700 1 --2- 172.21.15.23:0/2348130241 >> v2:172.21.15.23:3300/0 conn(0x7f6160065780 0x7f61600601b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:38:04.165 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:04.161+0000 7f6164c1a700 1 --2- 172.21.15.23:0/2348130241 >> v2:172.21.15.155:3300/0 conn(0x7f616005bc80 0x7f61600a9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:04.166 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:04.161+0000 7f6164c1a700 1 -- 172.21.15.23:0/2348130241 >> v2:172.21.15.155:3300/0 conn(0x7f616005bc80 msgr2=0x7f61600a9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:38:04.166 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:04.161+0000 7f6164c1a700 1 -- 172.21.15.23:0/2348130241 >> v2:172.21.15.155:3300/0 conn(0x7f616005bc80 msgr2=0x7f61600a9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:38:04.166 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:04.161+0000 7f6164c1a700 1 --2- 172.21.15.23:0/2348130241 >> v2:172.21.15.155:3300/0 conn(0x7f616005bc80 0x7f61600a9140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:38:04.166 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:04.161+0000 7f6164c1a700 1 --2- 172.21.15.23:0/2348130241 >> v2:172.21.15.155:3300/0 conn(0x7f616005bc80 0x7f61600a9140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:38:04.966 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:04.962+0000 7f615ffff700 1 --2- 172.21.15.23:0/2348130241 >> v2:172.21.15.23:3300/0 conn(0x7f6160065780 0x7f61600601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:04.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:04.962+0000 7f615ffff700 1 -- 172.21.15.23:0/2348130241 >> v2:172.21.15.23:3300/0 conn(0x7f6160065780 msgr2=0x7f61600601b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:38:04.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:04.962+0000 7f615ffff700 1 -- 172.21.15.23:0/2348130241 >> v2:172.21.15.23:3300/0 conn(0x7f6160065780 msgr2=0x7f61600601b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:38:04.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:04.962+0000 7f615ffff700 1 --2- 172.21.15.23:0/2348130241 >> v2:172.21.15.23:3300/0 conn(0x7f6160065780 0x7f61600601b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:38:04.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:04.962+0000 7f615ffff700 1 --2- 172.21.15.23:0/2348130241 >> v2:172.21.15.23:3300/0 conn(0x7f6160065780 0x7f61600601b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:38:04.968 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:04.962+0000 7f6164c1a700 1 --2- 172.21.15.23:0/2348130241 >> v2:172.21.15.155:3300/0 conn(0x7f616005bc80 0x7f61600a9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:04.968 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:04.962+0000 7f6164c1a700 1 -- 172.21.15.23:0/2348130241 >> v2:172.21.15.155:3300/0 conn(0x7f616005bc80 msgr2=0x7f61600a9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:38:04.969 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:04.962+0000 7f6164c1a700 1 -- 172.21.15.23:0/2348130241 >> v2:172.21.15.155:3300/0 conn(0x7f616005bc80 msgr2=0x7f61600a9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:38:04.969 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:04.962+0000 7f6164c1a700 1 --2- 172.21.15.23:0/2348130241 >> v2:172.21.15.155:3300/0 conn(0x7f616005bc80 0x7f61600a9140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:38:04.969 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:04.962+0000 7f6164c1a700 1 --2- 172.21.15.23:0/2348130241 >> v2:172.21.15.155:3300/0 conn(0x7f616005bc80 0x7f61600a9140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:38:06.561 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:06.558+0000 7f615e7fc700 1 -- 172.21.15.23:0/2348130241 >> v2:172.21.15.155:3300/0 conn(0x7f616005bc80 msgr2=0x7f61600a9140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:38:06.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:06.558+0000 7f615e7fc700 1 --2- 172.21.15.23:0/2348130241 >> v2:172.21.15.155:3300/0 conn(0x7f616005bc80 0x7f61600a9140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:06.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:06.558+0000 7f615e7fc700 1 -- 172.21.15.23:0/2348130241 >> v2:172.21.15.23:3300/0 conn(0x7f6160065780 msgr2=0x7f61600601b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:38:06.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:06.558+0000 7f615e7fc700 1 --2- 172.21.15.23:0/2348130241 >> v2:172.21.15.23:3300/0 conn(0x7f6160065780 0x7f61600601b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:06.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:06.558+0000 7f615e7fc700 1 -- 172.21.15.23:0/2348130241 >> v1:172.21.15.155:6789/0 conn(0x7f61600a9cf0 legacy=0x7f61601379e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:06.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:06.558+0000 7f615e7fc700 1 --2- 172.21.15.23:0/2348130241 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6148006ff0 0x7f61480094b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:06.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:06.558+0000 7f615e7fc700 1 -- 172.21.15.23:0/2348130241 --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61600a8810 con 0x7f61480033a0 2020-11-20T11:38:06.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:06.558+0000 7f615e7fc700 1 -- 172.21.15.23:0/2348130241 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61600a8ca0 con 0x7f6148006ff0 2020-11-20T11:38:06.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:06.558+0000 7f615ffff700 1 -- 172.21.15.23:0/2348130241 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6148006ff0 msgr2=0x7f61480094b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:38:06.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:06.558+0000 7f615ffff700 1 --2- 172.21.15.23:0/2348130241 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6148006ff0 0x7f61480094b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:38:06.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:06.759+0000 7f615ffff700 1 -- 172.21.15.23:0/2348130241 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6148006ff0 msgr2=0x7f61480094b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:38:06.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:06.759+0000 7f615ffff700 1 --2- 172.21.15.23:0/2348130241 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6148006ff0 0x7f61480094b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:38:07.163 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.160+0000 7f615ffff700 1 -- 172.21.15.23:0/2348130241 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6148006ff0 msgr2=0x7f61480094b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:38:07.163 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.160+0000 7f615ffff700 1 --2- 172.21.15.23:0/2348130241 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6148006ff0 0x7f61480094b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:38:07.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.671+0000 7f615f7fe700 1 -- 172.21.15.23:0/2348130241 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 388156389 0 0) 0x7f61600a8810 con 0x7f61480033a0 2020-11-20T11:38:07.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.671+0000 7f615f7fe700 1 -- 172.21.15.23:0/2348130241 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6144003280 con 0x7f61480033a0 2020-11-20T11:38:07.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.671+0000 7f615f7fe700 1 -- 172.21.15.23:0/2348130241 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3002597133 0 0) 0x7f6144003280 con 0x7f61480033a0 2020-11-20T11:38:07.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.672+0000 7f615f7fe700 1 -- 172.21.15.23:0/2348130241 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6148006ff0 msgr2=0x7f61480094b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:38:07.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.672+0000 7f615f7fe700 1 --2- 172.21.15.23:0/2348130241 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6148006ff0 0x7f61480094b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:38:07.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.672+0000 7f615f7fe700 1 -- 172.21.15.23:0/2348130241 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f614800ac00 con 0x7f61480033a0 2020-11-20T11:38:07.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.672+0000 7f615f7fe700 1 -- 172.21.15.23:0/2348130241 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f615400d3b0 con 0x7f61480033a0 2020-11-20T11:38:07.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.672+0000 7f615f7fe700 1 -- 172.21.15.23:0/2348130241 >> v1:172.21.15.23:6789/0 conn(0x7f61480033a0 legacy=0x7f6148005820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:07.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.672+0000 7f616541b700 1 -- 172.21.15.23:0/2348130241 reap_dead start 2020-11-20T11:38:07.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.672+0000 7f615f7fe700 1 --2- 172.21.15.23:0/2348130241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6144003770 0x7f6144003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:07.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.672+0000 7f615f7fe700 1 --2- 172.21.15.23:0/2348130241 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6144004080 0x7f6144004560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:07.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.672+0000 7f615f7fe700 1 --2- 172.21.15.23:0/2348130241 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6144004c00 0x7f614400b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:07.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.672+0000 7f615f7fe700 1 -- 172.21.15.23:0/2348130241 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61600a8ca0 con 0x7f6144004080 2020-11-20T11:38:07.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.672+0000 7f615f7fe700 1 -- 172.21.15.23:0/2348130241 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61440052a0 con 0x7f6144003770 2020-11-20T11:38:07.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.672+0000 7f615f7fe700 1 -- 172.21.15.23:0/2348130241 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f614400bc60 con 0x7f6144004c00 2020-11-20T11:38:07.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.672+0000 7f616541b700 1 -- 172.21.15.23:0/2348130241 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6144004c00 msgr2=0x7f614400b770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:38:07.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.672+0000 7f615ffff700 1 --2- 172.21.15.23:0/2348130241 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6144004080 0x7f6144004560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:07.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.672+0000 7f616541b700 1 --2- 172.21.15.23:0/2348130241 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6144004c00 0x7f614400b770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:38:07.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.672+0000 7f6164c1a700 1 --2- 172.21.15.23:0/2348130241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6144003770 0x7f6144003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:07.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.672+0000 7f615ffff700 1 -- 172.21.15.23:0/2348130241 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6144004c00 msgr2=0x7f614400b770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:38:07.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.672+0000 7f615ffff700 1 --2- 172.21.15.23:0/2348130241 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6144004c00 0x7f614400b770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:38:07.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.672+0000 7f615ffff700 1 -- 172.21.15.23:0/2348130241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6144003770 msgr2=0x7f6144003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:38:07.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.672+0000 7f615ffff700 1 --2- 172.21.15.23:0/2348130241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6144003770 0x7f6144003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:07.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.672+0000 7f615ffff700 1 -- 172.21.15.23:0/2348130241 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6144003280 con 0x7f6144004080 2020-11-20T11:38:07.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.672+0000 7f615ffff700 1 --2- 172.21.15.23:0/2348130241 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6144004080 0x7f6144004560 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f614c011ca0 tx=0x7f614c008040).ready entity=mon.0 client_cookie=11ed5350203c592a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:07.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.672+0000 7f615f7fe700 1 -- 172.21.15.23:0/2348130241 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614c013c00 con 0x7f6144004080 2020-11-20T11:38:07.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.672+0000 7f615f7fe700 1 -- 172.21.15.23:0/2348130241 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f614c013d60 con 0x7f6144004080 2020-11-20T11:38:07.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.672+0000 7f615f7fe700 1 -- 172.21.15.23:0/2348130241 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614c016690 con 0x7f6144004080 2020-11-20T11:38:07.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.673+0000 7f6166e7e700 1 -- 172.21.15.23:0/2348130241 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6144004080 msgr2=0x7f6144004560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:07.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.673+0000 7f6166e7e700 1 --2- 172.21.15.23:0/2348130241 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6144004080 0x7f6144004560 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f614c011ca0 tx=0x7f614c008040).stop 2020-11-20T11:38:07.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.673+0000 7f6166e7e700 1 -- 172.21.15.23:0/2348130241 shutdown_connections 2020-11-20T11:38:07.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.673+0000 7f6166e7e700 1 --2- 172.21.15.23:0/2348130241 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6144004c00 0x7f614400b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:38:07.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.673+0000 7f6166e7e700 1 --2- 172.21.15.23:0/2348130241 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6144004080 0x7f6144004560 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0x7f614c011ca0 tx=0x7f614c008040).stop 2020-11-20T11:38:07.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.673+0000 7f6166e7e700 1 --2- 172.21.15.23:0/2348130241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6144003770 0x7f6144003b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:07.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.673+0000 7f6166e7e700 1 -- 172.21.15.23:0/2348130241 >> 172.21.15.23:0/2348130241 conn(0x7f61600ad7a0 msgr2=0x7f61600adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:07.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.674+0000 7f6166e7e700 1 -- 172.21.15.23:0/2348130241 shutdown_connections 2020-11-20T11:38:07.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.674+0000 7f6166e7e700 1 -- 172.21.15.23:0/2348130241 wait complete. 2020-11-20T11:38:07.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.675+0000 7f6166e7e700 1 Processor -- start 2020-11-20T11:38:07.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.675+0000 7f6166e7e700 1 -- start start 2020-11-20T11:38:07.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.676+0000 7f6166e7e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6144003770 0x7f616014d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:07.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.676+0000 7f6166e7e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6144004c00 0x7f61601417c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:07.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.676+0000 7f6166e7e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f616013f3b0 0x7f616013eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:07.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.676+0000 7f6166e7e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6160142390 con 0x7f6144003770 2020-11-20T11:38:07.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.676+0000 7f6166e7e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6160140180 con 0x7f6144004c00 2020-11-20T11:38:07.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.676+0000 7f6166e7e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f616013e430 con 0x7f616013f3b0 2020-11-20T11:38:07.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.676+0000 7f616541b700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f616013f3b0 msgr2=0x7f616013eb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:38:07.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.676+0000 7f616541b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f616013f3b0 0x7f616013eb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:38:07.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.677+0000 7f615ffff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6144004c00 0x7f61601417c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:07.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.677+0000 7f615ffff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6144004c00 0x7f61601417c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:36366/0 (socket says 172.21.15.23:36366) 2020-11-20T11:38:07.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.677+0000 7f615ffff700 1 -- 172.21.15.23:0/826647430 learned_addr learned my addr 172.21.15.23:0/826647430 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:38:07.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.677+0000 7f615ffff700 1 -- 172.21.15.23:0/826647430 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f616013f3b0 msgr2=0x7f616013eb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:38:07.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.677+0000 7f615ffff700 1 --2- 172.21.15.23:0/826647430 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f616013f3b0 0x7f616013eb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:38:07.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.677+0000 7f615ffff700 1 -- 172.21.15.23:0/826647430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6144003770 msgr2=0x7f616014d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:07.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.677+0000 7f615ffff700 1 --2- 172.21.15.23:0/826647430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6144003770 0x7f616014d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:38:07.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.677+0000 7f615ffff700 1 -- 172.21.15.23:0/826647430 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f616013e6d0 con 0x7f6144004c00 2020-11-20T11:38:07.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.677+0000 7f615ffff700 1 --2- 172.21.15.23:0/826647430 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6144004c00 0x7f61601417c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f614c008010 tx=0x7f614c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:07.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.677+0000 7f615cff9700 1 -- 172.21.15.23:0/826647430 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614c0067a0 con 0x7f6144004c00 2020-11-20T11:38:07.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.677+0000 7f615cff9700 1 -- 172.21.15.23:0/826647430 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f614c0207f0 con 0x7f6144004c00 2020-11-20T11:38:07.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.678+0000 7f615cff9700 1 -- 172.21.15.23:0/826647430 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614c020b70 con 0x7f6144004c00 2020-11-20T11:38:07.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.678+0000 7f6166e7e700 1 -- 172.21.15.23:0/826647430 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f616013fcb0 con 0x7f6144004c00 2020-11-20T11:38:07.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.678+0000 7f6166e7e700 1 -- 172.21.15.23:0/826647430 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6160132b50 con 0x7f6144004c00 2020-11-20T11:38:07.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.679+0000 7f6166e7e700 1 -- 172.21.15.23:0/826647430 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f616013e860 con 0x7f6144004c00 2020-11-20T11:38:07.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.679+0000 7f615cff9700 1 -- 172.21.15.23:0/826647430 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f614c02f020 con 0x7f6144004c00 2020-11-20T11:38:07.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.679+0000 7f615cff9700 1 --2- 172.21.15.23:0/826647430 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6138056d10 0x7f61380591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:07.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.680+0000 7f615cff9700 1 -- 172.21.15.23:0/826647430 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f614c072810 con 0x7f6144004c00 2020-11-20T11:38:07.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.680+0000 7f6164c1a700 1 --2- 172.21.15.23:0/826647430 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6138056d10 0x7f61380591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:07.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.680+0000 7f6164c1a700 1 --2- 172.21.15.23:0/826647430 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6138056d10 0x7f61380591d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f6154006bc0 tx=0x7f6154008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:07.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.684+0000 7f615cff9700 1 -- 172.21.15.23:0/826647430 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f614c04b530 con 0x7f6144004c00 2020-11-20T11:38:07.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.920+0000 7f6166e7e700 1 -- 172.21.15.23:0/826647430 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6160062fe0 con 0x7f6144004c00 2020-11-20T11:38:07.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.921+0000 7f615cff9700 1 -- 172.21.15.23:0/826647430 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1471 (secure 0 0 0) 0x7f614c04b120 con 0x7f6144004c00 2020-11-20T11:38:07.924 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":60,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:38:07.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.922+0000 7f6166e7e700 1 -- 172.21.15.23:0/826647430 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6138056d10 msgr2=0x7f61380591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:07.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.922+0000 7f6166e7e700 1 --2- 172.21.15.23:0/826647430 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6138056d10 0x7f61380591d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f6154006bc0 tx=0x7f6154008040).stop 2020-11-20T11:38:07.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.922+0000 7f6166e7e700 1 -- 172.21.15.23:0/826647430 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6144004c00 msgr2=0x7f61601417c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:07.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.922+0000 7f6166e7e700 1 --2- 172.21.15.23:0/826647430 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6144004c00 0x7f61601417c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f614c008010 tx=0x7f614c008040).stop 2020-11-20T11:38:07.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.922+0000 7f6166e7e700 1 -- 172.21.15.23:0/826647430 shutdown_connections 2020-11-20T11:38:07.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.922+0000 7f6166e7e700 1 --2- 172.21.15.23:0/826647430 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6138056d10 0x7f61380591d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:07.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.922+0000 7f6166e7e700 1 --2- 172.21.15.23:0/826647430 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f616013f3b0 0x7f616013eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:38:07.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.922+0000 7f6166e7e700 1 --2- 172.21.15.23:0/826647430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6144003770 0x7f616014d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:38:07.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.922+0000 7f6166e7e700 1 --2- 172.21.15.23:0/826647430 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6144004c00 0x7f61601417c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:07.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.922+0000 7f6166e7e700 1 -- 172.21.15.23:0/826647430 >> 172.21.15.23:0/826647430 conn(0x7f61600ad7a0 msgr2=0x7f6160068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:07.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.922+0000 7f6166e7e700 1 -- 172.21.15.23:0/826647430 shutdown_connections 2020-11-20T11:38:07.928 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:07.922+0000 7f6166e7e700 1 -- 172.21.15.23:0/826647430 wait complete. 2020-11-20T11:38:07.937 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2020-11-20T11:38:07.937 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:38:08.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.057+0000 7fd93d9da700 1 Processor -- start 2020-11-20T11:38:08.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.057+0000 7fd93d9da700 1 -- start start 2020-11-20T11:38:08.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.058+0000 7fd93d9da700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd938061380 0x7fd9380a8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:08.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.058+0000 7fd93d9da700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd9380a7410 0x7fd93813efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:08.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.058+0000 7fd93d9da700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9380a5000 con 0x7fd9380a8d70 2020-11-20T11:38:08.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.058+0000 7fd93d9da700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9380a78a0 con 0x7fd938061380 2020-11-20T11:38:08.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.058+0000 7fd93d9da700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9380a5d30 con 0x7fd9380a7410 2020-11-20T11:38:08.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.058+0000 7fd93c9d8700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd938061380 0x7fd9380a8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:08.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.058+0000 7fd93c9d8700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd938061380 0x7fd9380a8880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43476/0 (socket says 172.21.15.23:43476) 2020-11-20T11:38:08.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.058+0000 7fd93c9d8700 1 -- 172.21.15.23:0/114537568 learned_addr learned my addr 172.21.15.23:0/114537568 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:38:08.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.058+0000 7fd93d1d9700 1 --2- 172.21.15.23:0/114537568 >> v2:172.21.15.155:3300/0 conn(0x7fd9380a7410 0x7fd93813efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:08.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.058+0000 7fd93c9d8700 1 -- 172.21.15.23:0/114537568 >> v2:172.21.15.155:3300/0 conn(0x7fd9380a7410 msgr2=0x7fd93813efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:38:08.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.058+0000 7fd93c9d8700 1 --2- 172.21.15.23:0/114537568 >> v2:172.21.15.155:3300/0 conn(0x7fd9380a7410 0x7fd93813efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:08.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.058+0000 7fd93c9d8700 1 -- 172.21.15.23:0/114537568 >> v1:172.21.15.155:6789/0 conn(0x7fd9380a8d70 legacy=0x7fd93813e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:38:08.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.058+0000 7fd93c9d8700 1 -- 172.21.15.23:0/114537568 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9380a5fd0 con 0x7fd938061380 2020-11-20T11:38:08.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.058+0000 7fd93d1d9700 1 --2- 172.21.15.23:0/114537568 >> v2:172.21.15.155:3300/0 conn(0x7fd9380a7410 0x7fd93813efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:38:08.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.058+0000 7fd93c9d8700 1 --2- 172.21.15.23:0/114537568 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd938061380 0x7fd9380a8880 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fd93805c060 tx=0x7fd92c006ef0).ready entity=mon.0 client_cookie=80830ad8347248b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:08.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.058+0000 7fd9377fe700 1 -- 172.21.15.23:0/114537568 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92c00b820 con 0x7fd938061380 2020-11-20T11:38:08.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.059+0000 7fd9377fe700 1 -- 172.21.15.23:0/114537568 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd92c000e90 con 0x7fd938061380 2020-11-20T11:38:08.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.059+0000 7fd9377fe700 1 -- 172.21.15.23:0/114537568 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92c00bb90 con 0x7fd938061380 2020-11-20T11:38:08.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.059+0000 7fd93d9da700 1 -- 172.21.15.23:0/114537568 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd938061380 msgr2=0x7fd9380a8880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:08.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.059+0000 7fd93d9da700 1 --2- 172.21.15.23:0/114537568 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd938061380 0x7fd9380a8880 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fd93805c060 tx=0x7fd92c006ef0).stop 2020-11-20T11:38:08.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.059+0000 7fd93d9da700 1 -- 172.21.15.23:0/114537568 shutdown_connections 2020-11-20T11:38:08.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.059+0000 7fd93d9da700 1 --2- 172.21.15.23:0/114537568 >> v2:172.21.15.155:3300/0 conn(0x7fd9380a7410 0x7fd93813efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:08.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.059+0000 7fd93d9da700 1 --2- 172.21.15.23:0/114537568 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd938061380 0x7fd9380a8880 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0x7fd93805c060 tx=0x7fd92c006ef0).stop 2020-11-20T11:38:08.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.059+0000 7fd93d9da700 1 -- 172.21.15.23:0/114537568 >> 172.21.15.23:0/114537568 conn(0x7fd9380ae0a0 msgr2=0x7fd9380aa0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:08.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.059+0000 7fd93d9da700 1 -- 172.21.15.23:0/114537568 shutdown_connections 2020-11-20T11:38:08.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.059+0000 7fd93d9da700 1 -- 172.21.15.23:0/114537568 wait complete. 2020-11-20T11:38:08.070 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.060+0000 7fd93d9da700 1 Processor -- start 2020-11-20T11:38:08.071 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.060+0000 7fd93d9da700 1 -- start start 2020-11-20T11:38:08.071 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.061+0000 7fd93d9da700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9380a7410 0x7fd9380a27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:08.071 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.061+0000 7fd93d9da700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9380a8d70 0x7fd93814a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:08.071 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.061+0000 7fd93d9da700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd93811c5f0 0x7fd93813ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:08.072 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.061+0000 7fd93d9da700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93814dba0 con 0x7fd9380a8d70 2020-11-20T11:38:08.072 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.061+0000 7fd93d9da700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93814de40 con 0x7fd93811c5f0 2020-11-20T11:38:08.072 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.061+0000 7fd93d9da700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd93814e0e0 con 0x7fd9380a7410 2020-11-20T11:38:08.072 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.061+0000 7fd937fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9380a8d70 0x7fd93814a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:08.072 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.061+0000 7fd937fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9380a8d70 0x7fd93814a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43484/0 (socket says 172.21.15.23:43484) 2020-11-20T11:38:08.073 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.061+0000 7fd93c9d8700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9380a7410 msgr2=0x7fd9380a27a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:38:08.073 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.061+0000 7fd937fff700 1 -- 172.21.15.23:0/4055229398 learned_addr learned my addr 172.21.15.23:0/4055229398 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:38:08.073 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.061+0000 7fd93c9d8700 1 --2- 172.21.15.23:0/4055229398 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9380a7410 0x7fd9380a27a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:38:08.073 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.061+0000 7fd93d1d9700 1 --2- 172.21.15.23:0/4055229398 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd93811c5f0 0x7fd93813ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:08.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.062+0000 7fd937fff700 1 -- 172.21.15.23:0/4055229398 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9380a7410 msgr2=0x7fd9380a27a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:38:08.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.062+0000 7fd937fff700 1 --2- 172.21.15.23:0/4055229398 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd9380a7410 0x7fd9380a27a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:38:08.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.062+0000 7fd937fff700 1 -- 172.21.15.23:0/4055229398 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd93811c5f0 msgr2=0x7fd93813ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:08.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.062+0000 7fd937fff700 1 --2- 172.21.15.23:0/4055229398 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd93811c5f0 0x7fd93813ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:08.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.062+0000 7fd937fff700 1 -- 172.21.15.23:0/4055229398 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd93814d890 con 0x7fd9380a8d70 2020-11-20T11:38:08.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.062+0000 7fd937fff700 1 --2- 172.21.15.23:0/4055229398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9380a8d70 0x7fd93814a2c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fd9240096e0 tx=0x7fd924006d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:08.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.062+0000 7fd934ff9700 1 -- 172.21.15.23:0/4055229398 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92400d4a0 con 0x7fd9380a8d70 2020-11-20T11:38:08.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.062+0000 7fd934ff9700 1 -- 172.21.15.23:0/4055229398 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd92400d620 con 0x7fd9380a8d70 2020-11-20T11:38:08.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.062+0000 7fd93d9da700 1 -- 172.21.15.23:0/4055229398 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9380a2c90 con 0x7fd9380a8d70 2020-11-20T11:38:08.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.062+0000 7fd934ff9700 1 -- 172.21.15.23:0/4055229398 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92400d9e0 con 0x7fd9380a8d70 2020-11-20T11:38:08.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.062+0000 7fd93d9da700 1 -- 172.21.15.23:0/4055229398 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd938067800 con 0x7fd9380a8d70 2020-11-20T11:38:08.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.063+0000 7fd93d9da700 1 --2- 172.21.15.23:0/4055229398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd938061980 0x7fd938140460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:08.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.063+0000 7fd93d9da700 1 -- 172.21.15.23:0/4055229398 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93814a9f0 con 0x7fd938061980 2020-11-20T11:38:08.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.063+0000 7fd93c9d8700 1 --2- 172.21.15.23:0/4055229398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd938061980 0x7fd938140460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:08.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.063+0000 7fd93c9d8700 1 -- 172.21.15.23:0/4055229398 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd938069c70 con 0x7fd938061980 2020-11-20T11:38:08.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.063+0000 7fd934ff9700 1 -- 172.21.15.23:0/4055229398 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd92401b030 con 0x7fd9380a8d70 2020-11-20T11:38:08.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.063+0000 7fd93c9d8700 1 --2- 172.21.15.23:0/4055229398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd938061980 0x7fd938140460 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fd93805c060 tx=0x7fd92c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:08.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.064+0000 7fd934ff9700 1 --2- 172.21.15.23:0/4055229398 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd910056d10 0x7fd9100591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:08.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.064+0000 7fd934ff9700 1 -- 172.21.15.23:0/4055229398 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fd92406fe50 con 0x7fd9380a8d70 2020-11-20T11:38:08.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.064+0000 7fd934ff9700 1 -- 172.21.15.23:0/4055229398 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92c011ba0 con 0x7fd938061980 2020-11-20T11:38:08.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.064+0000 7fd934ff9700 1 -- 172.21.15.23:0/4055229398 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fd92c011d00 con 0x7fd938061980 2020-11-20T11:38:08.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.064+0000 7fd934ff9700 1 -- 172.21.15.23:0/4055229398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd938061980 msgr2=0x7fd938140460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:08.079 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.064+0000 7fd934ff9700 1 --2- 172.21.15.23:0/4055229398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd938061980 0x7fd938140460 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fd93805c060 tx=0x7fd92c016000).stop 2020-11-20T11:38:08.079 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.064+0000 7fd93d1d9700 1 --2- 172.21.15.23:0/4055229398 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd910056d10 0x7fd9100591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:08.079 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.064+0000 7fd934ff9700 1 -- 172.21.15.23:0/4055229398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd938061980 msgr2=0x7fd938140460 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:38:08.079 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.064+0000 7fd934ff9700 1 --2- 172.21.15.23:0/4055229398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd938061980 0x7fd938140460 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0x7fd93805c060 tx=0x7fd92c016000).stop 2020-11-20T11:38:08.079 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.064+0000 7fd93d1d9700 1 --2- 172.21.15.23:0/4055229398 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd910056d10 0x7fd9100591d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fd93000ad70 tx=0x7fd930008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:08.081 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.072+0000 7fd93d9da700 1 --2- 172.21.15.23:0/4055229398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd93814e2a0 0x7fd9380665a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:08.082 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.072+0000 7fd93d9da700 1 -- 172.21.15.23:0/4055229398 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd938057dc0 con 0x7fd93814e2a0 2020-11-20T11:38:08.082 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.072+0000 7fd93c9d8700 1 --2- 172.21.15.23:0/4055229398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd93814e2a0 0x7fd9380665a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:08.082 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.073+0000 7fd93c9d8700 1 -- 172.21.15.23:0/4055229398 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd938044710 con 0x7fd93814e2a0 2020-11-20T11:38:08.082 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.073+0000 7fd93c9d8700 1 --2- 172.21.15.23:0/4055229398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd93814e2a0 0x7fd9380665a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fd92c00c010 tx=0x7fd92c01e000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:08.082 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.073+0000 7fd934ff9700 1 -- 172.21.15.23:0/4055229398 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92c027890 con 0x7fd93814e2a0 2020-11-20T11:38:08.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.073+0000 7fd934ff9700 1 -- 172.21.15.23:0/4055229398 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4434 (secure 0 0 0) 0x7fd92c011990 con 0x7fd93814e2a0 2020-11-20T11:38:08.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.073+0000 7fd934ff9700 1 -- 172.21.15.23:0/4055229398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd93814e2a0 msgr2=0x7fd9380665a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:08.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.073+0000 7fd934ff9700 1 --2- 172.21.15.23:0/4055229398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd93814e2a0 0x7fd9380665a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fd92c00c010 tx=0x7fd92c01e000).stop 2020-11-20T11:38:08.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.073+0000 7fd934ff9700 1 -- 172.21.15.23:0/4055229398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd93814e2a0 msgr2=0x7fd9380665a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:38:08.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.073+0000 7fd934ff9700 1 --2- 172.21.15.23:0/4055229398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd93814e2a0 0x7fd9380665a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:08.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.074+0000 7fd93d9da700 1 -- 172.21.15.23:0/4055229398 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd910056d10 msgr2=0x7fd9100591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:08.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.074+0000 7fd93d9da700 1 --2- 172.21.15.23:0/4055229398 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd910056d10 0x7fd9100591d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fd93000ad70 tx=0x7fd930008040).stop 2020-11-20T11:38:08.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.074+0000 7fd93d9da700 1 -- 172.21.15.23:0/4055229398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9380a8d70 msgr2=0x7fd93814a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:08.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.074+0000 7fd93d9da700 1 --2- 172.21.15.23:0/4055229398 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9380a8d70 0x7fd93814a2c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fd9240096e0 tx=0x7fd924006d90).stop 2020-11-20T11:38:08.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.074+0000 7fd93d1d9700 1 -- 172.21.15.23:0/4055229398 reap_dead start 2020-11-20T11:38:08.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.074+0000 7fd93d1d9700 1 -- 172.21.15.23:0/4055229398 reap_dead start 2020-11-20T11:38:08.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.074+0000 7fd93d9da700 1 -- 172.21.15.23:0/4055229398 shutdown_connections 2020-11-20T11:38:08.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.074+0000 7fd93d9da700 1 -- 172.21.15.23:0/4055229398 >> 172.21.15.23:0/4055229398 conn(0x7fd9380ae0a0 msgr2=0x7fd9380acaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:08.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.074+0000 7fd93d9da700 1 -- 172.21.15.23:0/4055229398 shutdown_connections 2020-11-20T11:38:08.086 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.074+0000 7fd93d9da700 1 -- 172.21.15.23:0/4055229398 wait complete. 2020-11-20T11:38:08.086 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:38:08.086 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:38:08.086 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:38:08.087 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:38:08.087 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 60, 2020-11-20T11:38:08.087 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:38:08.087 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:38:08.087 INFO:teuthology.orchestra.run.smithi023.stdout: 1 2020-11-20T11:38:08.088 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:08.088 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 0, 2020-11-20T11:38:08.088 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:38:08.088 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:38:08.088 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:38:08.089 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:38:08.089 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:38:08.089 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:38:08.089 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:38:08.089 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:38:08.090 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:38:08.090 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:38:08.090 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:38:08.090 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:08.090 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:38:08.091 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:38:08.091 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:38:08.091 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:38:08.091 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:38:08.092 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:38:08.092 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:38:08.092 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:38:08.092 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:38:08.092 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:38:08.093 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:08.093 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:08.093 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:38:08.093 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:38:08.093 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:38:08.094 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:38:08.094 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:38:08.094 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:38:08.094 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:38:08.094 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:38:08.095 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:38:08.095 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:38:08.095 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:38:08.095 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:38:08.095 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:38:08.096 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:38:08.096 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:38:08.096 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:38:08.097 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:38:08.097 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:38:08.097 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:38:08.097 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:38:08.098 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:38:08.098 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:38:08.098 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:38:08.098 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:08.098 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:38:08.099 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:08.099 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:38:08.099 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:08.099 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:38:08.099 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:38:08.100 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:38:08.100 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:38:08.100 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:08.100 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:38:08.100 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:38:08.101 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:08.101 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:08.101 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:08.101 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:38:08.102 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:38:08.102 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:08.102 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:08.102 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:08.102 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:08.103 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:38:08.103 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:38:08.103 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:38:08.103 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:38:08.103 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:38:08.104 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:08.104 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:08.104 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:38:08.104 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:38:08.104 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:38:08.105 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:38:08.105 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:08.105 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:38:08.105 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:38:08.105 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:08.106 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:08.106 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:08.106 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:38:08.106 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:38:08.106 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:08.107 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:08.107 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:08.107 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:08.107 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:38:08.107 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:38:08.108 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:38:08.108 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:38:08.108 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:38:08.108 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:08.108 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:08.109 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:38:08.109 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:38:08.109 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:38:08.109 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:38:08.110 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:08.110 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:38:08.110 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:38:08.110 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:08.110 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:08.110 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:08.111 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:38:08.111 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:38:08.111 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:08.111 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:08.111 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:08.112 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:08.112 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:38:08.112 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:38:08.112 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:38:08.113 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:38:08.113 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:38:08.113 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:08.113 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:08.113 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:08.114 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:38:08.114 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:38:08.114 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:08.114 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:38:08.114 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:38:08.115 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:38:08.115 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:08.115 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:08.115 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:38:08.115 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:08.115 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:38:08.116 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:38:08.116 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:38:08.116 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:08.116 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:08.117 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:38:08.117 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:08.117 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:38:08.117 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:38:08.117 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:38:08.118 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:08.118 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:08.118 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:08.119 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:38:08.119 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:38:08.120 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:38:08.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.246+0000 7f9d0d46a700 1 Processor -- start 2020-11-20T11:38:08.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.246+0000 7f9d0d46a700 1 -- start start 2020-11-20T11:38:08.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.246+0000 7f9d0d46a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f9d080aae90 0x7f9d081381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:08.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.246+0000 7f9d0d46a700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0813e600 con 0x7f9d080aaab0 2020-11-20T11:38:08.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.246+0000 7f9d0d46a700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d0813e8a0 con 0x7f9d0813aeb0 2020-11-20T11:38:08.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.246+0000 7f9d0d46a700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d080ab6d0 con 0x7f9d080aae90 2020-11-20T11:38:08.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.246+0000 7f9d067fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f9d080aae90 0x7f9d081381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:08.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.246+0000 7f9d06ffd700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f9d080aaab0 0x7f9d0813ed40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57744/0 (socket says 172.21.15.23:57744) 2020-11-20T11:38:08.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.246+0000 7f9d06ffd700 1 -- 172.21.15.23:0/1194877581 learned_addr learned my addr 172.21.15.23:0/1194877581 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:38:08.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.247+0000 7f9d067fc700 1 -- 172.21.15.23:0/1194877581 >> v1:172.21.15.155:6789/0 conn(0x7f9d0813aeb0 legacy=0x7f9d080a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:08.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.247+0000 7f9d067fc700 1 -- 172.21.15.23:0/1194877581 >> v1:172.21.15.23:6789/0 conn(0x7f9d080aaab0 legacy=0x7f9d0813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:08.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.247+0000 7f9d067fc700 1 -- 172.21.15.23:0/1194877581 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d080a9370 con 0x7f9d080aae90 2020-11-20T11:38:08.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.247+0000 7f9d067fc700 1 --2- 172.21.15.23:0/1194877581 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d080aae90 0x7f9d081381a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f9cfc006fd0 tx=0x7f9cfc006e30).ready entity=mon.0 client_cookie=392bfa3287bee80a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:08.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.247+0000 7f9d05ffb700 1 -- 172.21.15.23:0/1194877581 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc0097e0 con 0x7f9d080aae90 2020-11-20T11:38:08.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.247+0000 7f9d05ffb700 1 -- 172.21.15.23:0/1194877581 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cfc000e90 con 0x7f9d080aae90 2020-11-20T11:38:08.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.247+0000 7f9d05ffb700 1 -- 172.21.15.23:0/1194877581 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc009b50 con 0x7f9d080aae90 2020-11-20T11:38:08.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.248+0000 7f9d0d46a700 1 -- 172.21.15.23:0/1194877581 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d080aae90 msgr2=0x7f9d081381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:08.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.248+0000 7f9d0d46a700 1 --2- 172.21.15.23:0/1194877581 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d080aae90 0x7f9d081381a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f9cfc006fd0 tx=0x7f9cfc006e30).stop 2020-11-20T11:38:08.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.248+0000 7f9d0d46a700 1 -- 172.21.15.23:0/1194877581 shutdown_connections 2020-11-20T11:38:08.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.248+0000 7f9d0d46a700 1 --2- 172.21.15.23:0/1194877581 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d080aae90 0x7f9d081381a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:08.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.248+0000 7f9d0d46a700 1 -- 172.21.15.23:0/1194877581 >> 172.21.15.23:0/1194877581 conn(0x7f9d080678e0 msgr2=0x7f9d08067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:08.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.248+0000 7f9d0d46a700 1 -- 172.21.15.23:0/1194877581 shutdown_connections 2020-11-20T11:38:08.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.248+0000 7f9d0d46a700 1 -- 172.21.15.23:0/1194877581 wait complete. 2020-11-20T11:38:08.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.249+0000 7f9d0d46a700 1 Processor -- start 2020-11-20T11:38:08.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.249+0000 7f9d0d46a700 1 -- start start 2020-11-20T11:38:08.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.250+0000 7f9d0d46a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d080aaab0 0x7f9d0809e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:08.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.250+0000 7f9d0d46a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d080aae90 0x7f9d0809fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:08.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.250+0000 7f9d0d46a700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d0814a3d0 0x7f9d080a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:08.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.250+0000 7f9d0d46a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0813e310 con 0x7f9d080aaab0 2020-11-20T11:38:08.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.250+0000 7f9d0d46a700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d080a73e0 con 0x7f9d080aae90 2020-11-20T11:38:08.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.250+0000 7f9d0d46a700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d080a7680 con 0x7f9d0814a3d0 2020-11-20T11:38:08.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.250+0000 7f9d06ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d080aaab0 0x7f9d0809e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:08.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.250+0000 7f9d077fe700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d0814a3d0 msgr2=0x7f9d080a63b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:38:08.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.250+0000 7f9d077fe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d0814a3d0 0x7f9d080a63b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:38:08.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.250+0000 7f9d06ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d080aaab0 0x7f9d0809e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43500/0 (socket says 172.21.15.23:43500) 2020-11-20T11:38:08.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.250+0000 7f9d06ffd700 1 -- 172.21.15.23:0/1711759866 learned_addr learned my addr 172.21.15.23:0/1711759866 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:38:08.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.250+0000 7f9d067fc700 1 --2- 172.21.15.23:0/1711759866 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d080aae90 0x7f9d0809fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:08.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.250+0000 7f9d06ffd700 1 -- 172.21.15.23:0/1711759866 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d0814a3d0 msgr2=0x7f9d080a63b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:38:08.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.250+0000 7f9d06ffd700 1 --2- 172.21.15.23:0/1711759866 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d0814a3d0 0x7f9d080a63b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:38:08.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.250+0000 7f9d06ffd700 1 -- 172.21.15.23:0/1711759866 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d080aae90 msgr2=0x7f9d0809fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:08.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.250+0000 7f9d06ffd700 1 --2- 172.21.15.23:0/1711759866 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d080aae90 0x7f9d0809fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:08.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.250+0000 7f9d06ffd700 1 -- 172.21.15.23:0/1711759866 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d080a7920 con 0x7f9d080aaab0 2020-11-20T11:38:08.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.250+0000 7f9d06ffd700 1 --2- 172.21.15.23:0/1711759866 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d080aaab0 0x7f9d0809e530 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f9cf800c710 tx=0x7f9cf8007c80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:08.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.250+0000 7f9ceeffd700 1 -- 172.21.15.23:0/1711759866 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf80072e0 con 0x7f9d080aaab0 2020-11-20T11:38:08.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.250+0000 7f9ceeffd700 1 -- 172.21.15.23:0/1711759866 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf8007440 con 0x7f9d080aaab0 2020-11-20T11:38:08.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.250+0000 7f9ceeffd700 1 -- 172.21.15.23:0/1711759866 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8005840 con 0x7f9d080aaab0 2020-11-20T11:38:08.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.250+0000 7f9d0d46a700 1 -- 172.21.15.23:0/1711759866 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d080a8170 con 0x7f9d080aaab0 2020-11-20T11:38:08.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.250+0000 7f9d0d46a700 1 -- 172.21.15.23:0/1711759866 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d080a68a0 con 0x7f9d080aaab0 2020-11-20T11:38:08.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.251+0000 7f9d0d46a700 1 --2- 172.21.15.23:0/1711759866 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d0813aeb0 0x7f9d0814d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:08.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.251+0000 7f9d0d46a700 1 -- 172.21.15.23:0/1711759866 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d080a7f80 con 0x7f9d0813aeb0 2020-11-20T11:38:08.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.252+0000 7f9ceeffd700 1 -- 172.21.15.23:0/1711759866 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9cf8010070 con 0x7f9d080aaab0 2020-11-20T11:38:08.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.252+0000 7f9d067fc700 1 --2- 172.21.15.23:0/1711759866 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d0813aeb0 0x7f9d0814d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:08.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.252+0000 7f9ceeffd700 1 --2- 172.21.15.23:0/1711759866 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9ce4056d10 0x7f9ce40591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:08.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.252+0000 7f9d067fc700 1 -- 172.21.15.23:0/1711759866 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d0805c7d0 con 0x7f9d0813aeb0 2020-11-20T11:38:08.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.252+0000 7f9ceeffd700 1 -- 172.21.15.23:0/1711759866 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f9cf806f730 con 0x7f9d080aaab0 2020-11-20T11:38:08.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.252+0000 7f9d067fc700 1 --2- 172.21.15.23:0/1711759866 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d0813aeb0 0x7f9d0814d890 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f9cfc006fd0 tx=0x7f9cfc015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:08.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.252+0000 7f9d077fe700 1 --2- 172.21.15.23:0/1711759866 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9ce4056d10 0x7f9ce40591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:08.266 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.252+0000 7f9ceeffd700 1 -- 172.21.15.23:0/1711759866 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc011c70 con 0x7f9d0813aeb0 2020-11-20T11:38:08.266 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.253+0000 7f9d077fe700 1 --2- 172.21.15.23:0/1711759866 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9ce4056d10 0x7f9ce40591d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f9d080af230 tx=0x7f9d00007b70).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:08.266 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.253+0000 7f9ceeffd700 1 -- 172.21.15.23:0/1711759866 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f9cfc00bea0 con 0x7f9d0813aeb0 2020-11-20T11:38:08.266 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.253+0000 7f9ceeffd700 1 -- 172.21.15.23:0/1711759866 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d0813aeb0 msgr2=0x7f9d0814d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:08.266 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.253+0000 7f9ceeffd700 1 --2- 172.21.15.23:0/1711759866 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d0813aeb0 0x7f9d0814d890 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f9cfc006fd0 tx=0x7f9cfc015000).stop 2020-11-20T11:38:08.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.253+0000 7f9ceeffd700 1 -- 172.21.15.23:0/1711759866 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d0813aeb0 msgr2=0x7f9d0814d890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:38:08.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.253+0000 7f9ceeffd700 1 --2- 172.21.15.23:0/1711759866 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d0813aeb0 0x7f9d0814d890 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:08.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.260+0000 7f9d0d46a700 1 --2- 172.21.15.23:0/1711759866 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d080601b0 0x7f9d0814dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:08.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.260+0000 7f9d0d46a700 1 -- 172.21.15.23:0/1711759866 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d08135210 con 0x7f9d080601b0 2020-11-20T11:38:08.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.260+0000 7f9d067fc700 1 --2- 172.21.15.23:0/1711759866 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d080601b0 0x7f9d0814dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:08.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.260+0000 7f9d067fc700 1 -- 172.21.15.23:0/1711759866 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9d08044710 con 0x7f9d080601b0 2020-11-20T11:38:08.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.260+0000 7f9d067fc700 1 --2- 172.21.15.23:0/1711759866 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d080601b0 0x7f9d0814dd80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f9d08148280 tx=0x7f9cfc021070).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:08.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.261+0000 7f9ceeffd700 1 -- 172.21.15.23:0/1711759866 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc02db20 con 0x7f9d080601b0 2020-11-20T11:38:08.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.261+0000 7f9ceeffd700 1 -- 172.21.15.23:0/1711759866 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4772 (secure 0 0 0) 0x7f9cfc011a60 con 0x7f9d080601b0 2020-11-20T11:38:08.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.261+0000 7f9ceeffd700 1 -- 172.21.15.23:0/1711759866 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d080601b0 msgr2=0x7f9d0814dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:08.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.261+0000 7f9ceeffd700 1 --2- 172.21.15.23:0/1711759866 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d080601b0 0x7f9d0814dd80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f9d08148280 tx=0x7f9cfc021070).stop 2020-11-20T11:38:08.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.261+0000 7f9ceeffd700 1 -- 172.21.15.23:0/1711759866 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d080601b0 msgr2=0x7f9d0814dd80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:38:08.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.261+0000 7f9ceeffd700 1 --2- 172.21.15.23:0/1711759866 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d080601b0 0x7f9d0814dd80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:08.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.261+0000 7f9d0d46a700 1 -- 172.21.15.23:0/1711759866 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9ce4056d10 msgr2=0x7f9ce40591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:08.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.261+0000 7f9d0d46a700 1 --2- 172.21.15.23:0/1711759866 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9ce4056d10 0x7f9ce40591d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f9d080af230 tx=0x7f9d00007b70).stop 2020-11-20T11:38:08.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.261+0000 7f9d0d46a700 1 -- 172.21.15.23:0/1711759866 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d080aaab0 msgr2=0x7f9d0809e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:08.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.261+0000 7f9d0d46a700 1 --2- 172.21.15.23:0/1711759866 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d080aaab0 0x7f9d0809e530 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f9cf800c710 tx=0x7f9cf8007c80).stop 2020-11-20T11:38:08.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.261+0000 7f9d077fe700 1 -- 172.21.15.23:0/1711759866 reap_dead start 2020-11-20T11:38:08.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.261+0000 7f9d077fe700 1 -- 172.21.15.23:0/1711759866 reap_dead start 2020-11-20T11:38:08.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.261+0000 7f9d0d46a700 1 -- 172.21.15.23:0/1711759866 shutdown_connections 2020-11-20T11:38:08.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.261+0000 7f9d0d46a700 1 -- 172.21.15.23:0/1711759866 >> 172.21.15.23:0/1711759866 conn(0x7f9d080678e0 msgr2=0x7f9d08138e20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:08.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.261+0000 7f9d0d46a700 1 -- 172.21.15.23:0/1711759866 shutdown_connections 2020-11-20T11:38:08.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:08.261+0000 7f9d0d46a700 1 -- 172.21.15.23:0/1711759866 wait complete. 2020-11-20T11:38:08.273 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:38:08.273 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:38:08.274 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:38:08.274 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:38:08.274 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 60, 2020-11-20T11:38:08.274 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:38:08.274 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:38:08.275 INFO:teuthology.orchestra.run.smithi023.stdout: 1 2020-11-20T11:38:08.275 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:08.275 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 0, 2020-11-20T11:38:08.275 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:38:08.275 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:38:08.276 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:38:08.276 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:38:08.276 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:38:08.276 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:38:08.276 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:38:08.277 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:38:08.277 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:38:08.277 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:38:08.277 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:38:08.278 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:08.278 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:38:08.278 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:38:08.278 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:38:08.279 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:38:08.279 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:38:08.279 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:38:08.279 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:38:08.279 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:38:08.280 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:38:08.280 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:38:08.280 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:08.280 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:08.280 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:38:08.281 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:38:08.281 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:38:08.281 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:38:08.281 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:38:08.282 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:38:08.282 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:38:08.282 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:38:08.282 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:38:08.282 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:38:08.283 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:38:08.283 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:38:08.283 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:38:08.283 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:38:08.283 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:38:08.284 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:38:08.284 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:38:08.284 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:38:08.284 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:38:08.284 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:38:08.285 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:38:08.285 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:38:08.285 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:38:08.285 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:08.285 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:38:08.286 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:08.286 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:38:08.286 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:08.286 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:38:08.287 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:38:08.287 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:38:08.287 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:38:08.287 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:08.287 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:38:08.288 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:38:08.288 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:08.288 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:08.288 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:08.288 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:38:08.289 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:38:08.289 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:08.289 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:08.289 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:08.289 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:08.290 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:38:08.290 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:38:08.290 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:38:08.290 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:38:08.290 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:38:08.291 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:08.291 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:08.291 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:38:08.291 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:38:08.291 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:38:08.292 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:38:08.292 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:08.292 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:38:08.292 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:38:08.292 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:08.293 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:08.293 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:08.293 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:38:08.293 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:38:08.293 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:08.294 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:08.294 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:08.294 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:08.294 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:38:08.294 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:38:08.295 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:38:08.295 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:38:08.295 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:38:08.295 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:08.295 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:08.296 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:38:08.296 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:38:08.296 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:38:08.296 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:38:08.297 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:08.297 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:38:08.297 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:38:08.297 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:08.297 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:08.298 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:08.298 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:38:08.298 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:38:08.298 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:08.298 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:08.299 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:08.299 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:08.299 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:38:08.299 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:38:08.299 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:38:08.300 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:38:08.300 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:38:08.300 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:08.300 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:08.300 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:08.301 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:38:08.301 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:38:08.301 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:08.302 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:38:08.302 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:38:08.302 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:38:08.302 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:08.303 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:08.303 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:38:08.303 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:08.303 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:38:08.304 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:38:08.304 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:38:08.304 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:08.304 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:08.304 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:38:08.305 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:08.305 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:38:08.305 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:38:08.305 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:38:08.305 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:08.306 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:08.306 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:38:08.306 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:08.306 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:38:08.307 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:38:08.307 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:38:08.307 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:08.307 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:08.307 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:38:08.308 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:08.308 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:38:08.308 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:38:08.308 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:38:08.308 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:08.309 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:08.309 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:08.309 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:38:08.309 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:38:08.310 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2020-11-20T11:38:28.315 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2020-11-20T11:38:28.316 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2020-11-20T11:38:28.316 INFO:tasks.ceph.mon.c:Restarting daemon 2020-11-20T11:38:28.316 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2020-11-20T11:38:28.319 INFO:tasks.ceph.mon.c:Started 2020-11-20T11:38:28.319 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:38:28.320 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:38:28.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.437+0000 7f41dde61700 1 Processor -- start 2020-11-20T11:38:28.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.437+0000 7f41dde61700 1 -- start start 2020-11-20T11:38:28.442 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.438+0000 7f41dde61700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f41d80a97d0 0x7f41d81344c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:28.442 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.438+0000 7f41dde61700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d80a51f0 con 0x7f41d8061380 2020-11-20T11:38:28.442 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.438+0000 7f41dde61700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d80a5490 con 0x7f41d80a9370 2020-11-20T11:38:28.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.438+0000 7f41dde61700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41d80a5730 con 0x7f41d80a97d0 2020-11-20T11:38:28.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.438+0000 7f41d7fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f41d80a97d0 0x7f41d81344c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:28.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.438+0000 7f41d77fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f41d8061380 0x7f41d80a6c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57764/0 (socket says 172.21.15.23:57764) 2020-11-20T11:38:28.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.438+0000 7f41d77fe700 1 -- 172.21.15.23:0/4179619545 learned_addr learned my addr 172.21.15.23:0/4179619545 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:38:28.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.438+0000 7f41d7fff700 1 -- 172.21.15.23:0/4179619545 >> v1:172.21.15.155:6789/0 conn(0x7f41d80a9370 legacy=0x7f41d8133db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:28.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.438+0000 7f41d7fff700 1 -- 172.21.15.23:0/4179619545 >> v1:172.21.15.23:6789/0 conn(0x7f41d8061380 legacy=0x7f41d80a6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:28.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.438+0000 7f41d67fc700 1 -- 172.21.15.23:0/4179619545 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1796857555 0 0) 0x7f41d80a5490 con 0x7f41d80a9370 2020-11-20T11:38:28.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.438+0000 7f41d7fff700 1 -- 172.21.15.23:0/4179619545 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d8061760 con 0x7f41d80a97d0 2020-11-20T11:38:28.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.438+0000 7f41d7fff700 1 --2- 172.21.15.23:0/4179619545 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41d80a97d0 0x7f41d81344c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f41d0009ce0 tx=0x7f41d0009220).ready entity=mon.0 client_cookie=8e5f824e874c33c6 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:28.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.439+0000 7f41d67fc700 1 -- 172.21.15.23:0/4179619545 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d000baf0 con 0x7f41d80a97d0 2020-11-20T11:38:28.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.439+0000 7f41d67fc700 1 -- 172.21.15.23:0/4179619545 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d000bc50 con 0x7f41d80a97d0 2020-11-20T11:38:28.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.439+0000 7f41d67fc700 1 -- 172.21.15.23:0/4179619545 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d0011400 con 0x7f41d80a97d0 2020-11-20T11:38:28.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.439+0000 7f41dde61700 1 -- 172.21.15.23:0/4179619545 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41d80a97d0 msgr2=0x7f41d81344c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:28.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.439+0000 7f41dde61700 1 --2- 172.21.15.23:0/4179619545 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41d80a97d0 0x7f41d81344c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f41d0009ce0 tx=0x7f41d0009220).stop 2020-11-20T11:38:28.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.439+0000 7f41dde61700 1 -- 172.21.15.23:0/4179619545 shutdown_connections 2020-11-20T11:38:28.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.439+0000 7f41dde61700 1 --2- 172.21.15.23:0/4179619545 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41d80a97d0 0x7f41d81344c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:28.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.439+0000 7f41dde61700 1 -- 172.21.15.23:0/4179619545 >> 172.21.15.23:0/4179619545 conn(0x7f41d80ae280 msgr2=0x7f41d80aa290 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:28.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.439+0000 7f41dde61700 1 -- 172.21.15.23:0/4179619545 shutdown_connections 2020-11-20T11:38:28.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.439+0000 7f41dde61700 1 -- 172.21.15.23:0/4179619545 wait complete. 2020-11-20T11:38:28.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.440+0000 7f41dde61700 1 Processor -- start 2020-11-20T11:38:28.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.440+0000 7f41dde61700 1 -- start start 2020-11-20T11:38:28.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.441+0000 7f41dde61700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41d8061380 0x7f41d814bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:28.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.441+0000 7f41dde61700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41d80a9370 0x7f41d80657d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:28.450 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.441+0000 7f41dde61700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41d80a97d0 0x7f41d80abae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:28.450 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.441+0000 7f41dde61700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d81321c0 con 0x7f41d80a97d0 2020-11-20T11:38:28.450 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.441+0000 7f41dde61700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d814b5d0 con 0x7f41d8061380 2020-11-20T11:38:28.450 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.441+0000 7f41dde61700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d814b870 con 0x7f41d80a9370 2020-11-20T11:38:28.450 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.441+0000 7f41d7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41d80a97d0 0x7f41d80abae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:28.451 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.441+0000 7f41d7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41d80a97d0 0x7f41d80abae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43524/0 (socket says 172.21.15.23:43524) 2020-11-20T11:38:28.451 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.441+0000 7f41d7fff700 1 -- 172.21.15.23:0/3523429328 learned_addr learned my addr 172.21.15.23:0/3523429328 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:38:28.451 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.441+0000 7f41d77fe700 1 --2- 172.21.15.23:0/3523429328 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41d8061380 0x7f41d814bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:28.451 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.441+0000 7f41d6ffd700 1 --2- 172.21.15.23:0/3523429328 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41d80a9370 0x7f41d80657d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:28.451 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.441+0000 7f41d6ffd700 1 -- 172.21.15.23:0/3523429328 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41d80a9370 msgr2=0x7f41d80657d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2020-11-20T11:38:28.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.441+0000 7f41d6ffd700 1 -- 172.21.15.23:0/3523429328 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41d80a9370 msgr2=0x7f41d80657d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:38:28.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.441+0000 7f41d6ffd700 1 --2- 172.21.15.23:0/3523429328 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41d80a9370 0x7f41d80657d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:38:28.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.441+0000 7f41d6ffd700 1 --2- 172.21.15.23:0/3523429328 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41d80a9370 0x7f41d80657d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:38:28.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.441+0000 7f41d7fff700 1 -- 172.21.15.23:0/3523429328 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41d80a9370 msgr2=0x7f41d80657d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:38:28.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.441+0000 7f41d7fff700 1 --2- 172.21.15.23:0/3523429328 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41d80a9370 0x7f41d80657d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:28.453 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.441+0000 7f41d7fff700 1 -- 172.21.15.23:0/3523429328 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41d8061380 msgr2=0x7f41d814bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:28.453 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.441+0000 7f41d7fff700 1 --2- 172.21.15.23:0/3523429328 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41d8061380 0x7f41d814bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:28.453 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.442+0000 7f41d7fff700 1 -- 172.21.15.23:0/3523429328 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d8065cc0 con 0x7f41d80a97d0 2020-11-20T11:38:28.453 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.442+0000 7f41d7fff700 1 --2- 172.21.15.23:0/3523429328 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41d80a97d0 0x7f41d80abae0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f41d00090b0 tx=0x7f41d0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:28.454 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.442+0000 7f41b77fe700 1 -- 172.21.15.23:0/3523429328 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d0010700 con 0x7f41d80a97d0 2020-11-20T11:38:28.454 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.442+0000 7f41b77fe700 1 -- 172.21.15.23:0/3523429328 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d0010860 con 0x7f41d80a97d0 2020-11-20T11:38:28.454 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.442+0000 7f41b77fe700 1 -- 172.21.15.23:0/3523429328 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d0010be0 con 0x7f41d80a97d0 2020-11-20T11:38:28.454 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.442+0000 7f41dde61700 1 -- 172.21.15.23:0/3523429328 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41d80abfd0 con 0x7f41d80a97d0 2020-11-20T11:38:28.454 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.442+0000 7f41dde61700 1 -- 172.21.15.23:0/3523429328 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41d8132df0 con 0x7f41d80a97d0 2020-11-20T11:38:28.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.443+0000 7f41dde61700 1 -- 172.21.15.23:0/3523429328 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41d8061e10 con 0x7f41d80a97d0 2020-11-20T11:38:28.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.443+0000 7f41b77fe700 1 -- 172.21.15.23:0/3523429328 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f41d0017070 con 0x7f41d80a97d0 2020-11-20T11:38:28.456 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.443+0000 7f41b77fe700 1 --2- 172.21.15.23:0/3523429328 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f41b0056d60 0x7f41b0059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:28.456 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.444+0000 7f41b77fe700 1 -- 172.21.15.23:0/3523429328 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f41d0072880 con 0x7f41d80a97d0 2020-11-20T11:38:28.456 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.444+0000 7f41d77fe700 1 --2- 172.21.15.23:0/3523429328 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f41b0056d60 0x7f41b0059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:28.456 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.444+0000 7f41d77fe700 1 --2- 172.21.15.23:0/3523429328 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f41b0056d60 0x7f41b0059220 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f41c80055e0 tx=0x7f41c8009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:28.457 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.447+0000 7f41b77fe700 1 -- 172.21.15.23:0/3523429328 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f41d0047b70 con 0x7f41d80a97d0 2020-11-20T11:38:28.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.691+0000 7f41dde61700 1 -- 172.21.15.23:0/3523429328 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f41d80ac160 con 0x7f41d80a97d0 2020-11-20T11:38:28.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.691+0000 7f41b77fe700 1 -- 172.21.15.23:0/3523429328 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1472 (secure 0 0 0) 0x7f41d0015020 con 0x7f41d80a97d0 2020-11-20T11:38:28.694 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":60,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:38:28.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.692+0000 7f41dde61700 1 -- 172.21.15.23:0/3523429328 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f41b0056d60 msgr2=0x7f41b0059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:28.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.692+0000 7f41dde61700 1 --2- 172.21.15.23:0/3523429328 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f41b0056d60 0x7f41b0059220 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f41c80055e0 tx=0x7f41c8009040).stop 2020-11-20T11:38:28.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.692+0000 7f41dde61700 1 -- 172.21.15.23:0/3523429328 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41d80a97d0 msgr2=0x7f41d80abae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:28.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.692+0000 7f41dde61700 1 --2- 172.21.15.23:0/3523429328 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41d80a97d0 0x7f41d80abae0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f41d00090b0 tx=0x7f41d0016000).stop 2020-11-20T11:38:28.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.692+0000 7f41dde61700 1 -- 172.21.15.23:0/3523429328 shutdown_connections 2020-11-20T11:38:28.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.692+0000 7f41dde61700 1 --2- 172.21.15.23:0/3523429328 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41d80a97d0 0x7f41d80abae0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:28.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.692+0000 7f41dde61700 1 --2- 172.21.15.23:0/3523429328 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41d8061380 0x7f41d814bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:28.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.692+0000 7f41dde61700 1 --2- 172.21.15.23:0/3523429328 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f41b0056d60 0x7f41b0059220 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:28.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.692+0000 7f41dde61700 1 --2- 172.21.15.23:0/3523429328 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41d80a9370 0x7f41d80657d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:28.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.693+0000 7f41dde61700 1 -- 172.21.15.23:0/3523429328 >> 172.21.15.23:0/3523429328 conn(0x7f41d80ae280 msgr2=0x7f41d805bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:28.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.693+0000 7f41dde61700 1 -- 172.21.15.23:0/3523429328 shutdown_connections 2020-11-20T11:38:28.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:28.693+0000 7f41dde61700 1 -- 172.21.15.23:0/3523429328 wait complete. 2020-11-20T11:38:30.708 INFO:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:38:30.710 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:38:31.709 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:38:31.832 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.829+0000 7f3cad73e700 1 Processor -- start 2020-11-20T11:38:31.832 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.829+0000 7f3cad73e700 1 -- start start 2020-11-20T11:38:31.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.829+0000 7f3cad73e700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3ca80aaab0 0x7f3ca813ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:31.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.829+0000 7f3cad73e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ca80aae90 0x7f3ca81381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:31.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.829+0000 7f3cad73e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3ca813aeb0 0x7f3ca80a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:31.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.829+0000 7f3cad73e700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ca813e600 con 0x7f3ca80aaab0 2020-11-20T11:38:31.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.829+0000 7f3cad73e700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ca813e8a0 con 0x7f3ca813aeb0 2020-11-20T11:38:31.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.829+0000 7f3cad73e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca80ab6d0 con 0x7f3ca80aae90 2020-11-20T11:38:31.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.830+0000 7f3ca6ffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3ca80aaab0 0x7f3ca813ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:31.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.830+0000 7f3ca6ffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3ca80aaab0 0x7f3ca813ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43528/0 (socket says 172.21.15.23:43528) 2020-11-20T11:38:31.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.830+0000 7f3ca6ffd700 1 -- 172.21.15.23:0/1597332160 learned_addr learned my addr 172.21.15.23:0/1597332160 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:38:31.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.830+0000 7f3ca67fc700 1 --2- 172.21.15.23:0/1597332160 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ca80aae90 0x7f3ca81381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:31.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.830+0000 7f3ca77fe700 1 --2- 172.21.15.23:0/1597332160 >> v2:172.21.15.155:3300/0 conn(0x7f3ca813aeb0 0x7f3ca80a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:31.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.830+0000 7f3ca67fc700 1 -- 172.21.15.23:0/1597332160 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ca80aae90 msgr2=0x7f3ca81381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:38:31.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.830+0000 7f3ca67fc700 1 -- 172.21.15.23:0/1597332160 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ca80aae90 msgr2=0x7f3ca81381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:38:31.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.830+0000 7f3ca6ffd700 1 -- 172.21.15.23:0/1597332160 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ca80aae90 msgr2=0x7f3ca81381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:38:31.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.830+0000 7f3ca67fc700 1 --2- 172.21.15.23:0/1597332160 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ca80aae90 0x7f3ca81381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:38:31.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.830+0000 7f3ca67fc700 1 --2- 172.21.15.23:0/1597332160 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ca80aae90 0x7f3ca81381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:38:31.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.830+0000 7f3ca6ffd700 1 --2- 172.21.15.23:0/1597332160 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ca80aae90 0x7f3ca81381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:31.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.830+0000 7f3ca6ffd700 1 -- 172.21.15.23:0/1597332160 >> v2:172.21.15.155:3300/0 conn(0x7f3ca813aeb0 msgr2=0x7f3ca80a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:38:31.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.830+0000 7f3ca6ffd700 1 --2- 172.21.15.23:0/1597332160 >> v2:172.21.15.155:3300/0 conn(0x7f3ca813aeb0 0x7f3ca80a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:31.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.830+0000 7f3ca6ffd700 1 -- 172.21.15.23:0/1597332160 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca813f230 con 0x7f3ca80aaab0 2020-11-20T11:38:31.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.830+0000 7f3ca77fe700 1 --2- 172.21.15.23:0/1597332160 >> v2:172.21.15.155:3300/0 conn(0x7f3ca813aeb0 0x7f3ca80a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:38:31.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.830+0000 7f3ca6ffd700 1 --2- 172.21.15.23:0/1597332160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ca80aaab0 0x7f3ca813ed40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f3ca80acf30 tx=0x7f3c98006ef0).ready entity=mon.0 client_cookie=51287c1f2d3caef server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:31.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.830+0000 7f3ca5ffb700 1 -- 172.21.15.23:0/1597332160 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9800b860 con 0x7f3ca80aaab0 2020-11-20T11:38:31.850 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.830+0000 7f3ca5ffb700 1 -- 172.21.15.23:0/1597332160 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c98000e90 con 0x7f3ca80aaab0 2020-11-20T11:38:31.850 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.830+0000 7f3ca5ffb700 1 -- 172.21.15.23:0/1597332160 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9800bbd0 con 0x7f3ca80aaab0 2020-11-20T11:38:31.850 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.831+0000 7f3cad73e700 1 -- 172.21.15.23:0/1597332160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ca80aaab0 msgr2=0x7f3ca813ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:31.851 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.831+0000 7f3cad73e700 1 --2- 172.21.15.23:0/1597332160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ca80aaab0 0x7f3ca813ed40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f3ca80acf30 tx=0x7f3c98006ef0).stop 2020-11-20T11:38:31.851 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.831+0000 7f3cad73e700 1 -- 172.21.15.23:0/1597332160 shutdown_connections 2020-11-20T11:38:31.852 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.831+0000 7f3cad73e700 1 --2- 172.21.15.23:0/1597332160 >> v2:172.21.15.155:3300/0 conn(0x7f3ca813aeb0 0x7f3ca80a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:31.852 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.831+0000 7f3cad73e700 1 --2- 172.21.15.23:0/1597332160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ca80aaab0 0x7f3ca813ed40 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:31.852 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.831+0000 7f3cad73e700 1 --2- 172.21.15.23:0/1597332160 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ca80aae90 0x7f3ca81381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:31.853 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.831+0000 7f3cad73e700 1 -- 172.21.15.23:0/1597332160 >> 172.21.15.23:0/1597332160 conn(0x7f3ca80678e0 msgr2=0x7f3ca8067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:31.853 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.831+0000 7f3cad73e700 1 -- 172.21.15.23:0/1597332160 shutdown_connections 2020-11-20T11:38:31.853 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.831+0000 7f3cad73e700 1 -- 172.21.15.23:0/1597332160 wait complete. 2020-11-20T11:38:31.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.832+0000 7f3cad73e700 1 Processor -- start 2020-11-20T11:38:31.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.832+0000 7f3cad73e700 1 -- start start 2020-11-20T11:38:31.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.833+0000 7f3cad73e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ca80aaab0 0x7f3ca80a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:31.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.833+0000 7f3cad73e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ca80aae90 0x7f3ca80a5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:31.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.833+0000 7f3cad73e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ca814a190 0x7f3ca814a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:31.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.833+0000 7f3cad73e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca80a6e60 con 0x7f3ca80aaab0 2020-11-20T11:38:31.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.833+0000 7f3cad73e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca814ab00 con 0x7f3ca80aae90 2020-11-20T11:38:31.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.833+0000 7f3cad73e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ca80a5820 con 0x7f3ca814a190 2020-11-20T11:38:31.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.833+0000 7f3ca6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ca80aaab0 0x7f3ca80a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:31.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.833+0000 7f3ca6ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ca80aaab0 0x7f3ca80a6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43534/0 (socket says 172.21.15.23:43534) 2020-11-20T11:38:31.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.833+0000 7f3ca6ffd700 1 -- 172.21.15.23:0/257807811 learned_addr learned my addr 172.21.15.23:0/257807811 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:38:31.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.833+0000 7f3ca77fe700 1 --2- 172.21.15.23:0/257807811 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ca814a190 0x7f3ca814a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:31.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.833+0000 7f3ca67fc700 1 --2- 172.21.15.23:0/257807811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ca80aae90 0x7f3ca80a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:31.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.833+0000 7f3ca77fe700 1 -- 172.21.15.23:0/257807811 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ca814a190 msgr2=0x7f3ca814a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2020-11-20T11:38:31.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.833+0000 7f3ca77fe700 1 -- 172.21.15.23:0/257807811 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ca814a190 msgr2=0x7f3ca814a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:38:31.859 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.833+0000 7f3ca77fe700 1 --2- 172.21.15.23:0/257807811 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ca814a190 0x7f3ca814a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:38:31.859 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.833+0000 7f3ca77fe700 1 --2- 172.21.15.23:0/257807811 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ca814a190 0x7f3ca814a610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:38:31.859 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.834+0000 7f3ca6ffd700 1 -- 172.21.15.23:0/257807811 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ca814a190 msgr2=0x7f3ca814a610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:38:31.860 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.834+0000 7f3ca6ffd700 1 --2- 172.21.15.23:0/257807811 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ca814a190 0x7f3ca814a610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:31.860 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.834+0000 7f3ca6ffd700 1 -- 172.21.15.23:0/257807811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ca80aae90 msgr2=0x7f3ca80a5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:31.860 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.834+0000 7f3ca6ffd700 1 --2- 172.21.15.23:0/257807811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ca80aae90 0x7f3ca80a5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:31.861 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.834+0000 7f3ca6ffd700 1 -- 172.21.15.23:0/257807811 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca80a6bc0 con 0x7f3ca80aaab0 2020-11-20T11:38:31.861 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.834+0000 7f3ca6ffd700 1 --2- 172.21.15.23:0/257807811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ca80aaab0 0x7f3ca80a6250 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f3c9800b3c0 tx=0x7f3c98016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:31.861 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.834+0000 7f3c8effd700 1 -- 172.21.15.23:0/257807811 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c98011cc0 con 0x7f3ca80aaab0 2020-11-20T11:38:31.862 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.834+0000 7f3c8effd700 1 -- 172.21.15.23:0/257807811 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c98011e20 con 0x7f3ca80aaab0 2020-11-20T11:38:31.862 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.834+0000 7f3c8effd700 1 -- 172.21.15.23:0/257807811 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9801c6a0 con 0x7f3ca80aaab0 2020-11-20T11:38:31.862 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.834+0000 7f3cad73e700 1 -- 172.21.15.23:0/257807811 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ca80a5ab0 con 0x7f3ca80aaab0 2020-11-20T11:38:31.863 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.834+0000 7f3cad73e700 1 -- 172.21.15.23:0/257807811 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ca809de20 con 0x7f3ca80aaab0 2020-11-20T11:38:31.863 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.835+0000 7f3cad73e700 1 -- 172.21.15.23:0/257807811 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ca805c7d0 con 0x7f3ca80aaab0 2020-11-20T11:38:31.863 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.835+0000 7f3c8effd700 1 -- 172.21.15.23:0/257807811 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3c98017070 con 0x7f3ca80aaab0 2020-11-20T11:38:31.864 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.836+0000 7f3c8effd700 1 --2- 172.21.15.23:0/257807811 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3c84056d60 0x7f3c84059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:31.864 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.836+0000 7f3c8effd700 1 -- 172.21.15.23:0/257807811 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f3c98071ad0 con 0x7f3ca80aaab0 2020-11-20T11:38:31.864 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.836+0000 7f3ca67fc700 1 --2- 172.21.15.23:0/257807811 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3c84056d60 0x7f3c84059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:31.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.836+0000 7f3ca67fc700 1 --2- 172.21.15.23:0/257807811 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3c84056d60 0x7f3c84059220 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f3ca80ae450 tx=0x7f3c9c008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:31.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:31.841+0000 7f3c8effd700 1 -- 172.21.15.23:0/257807811 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3c98046da0 con 0x7f3ca80aaab0 2020-11-20T11:38:32.116 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:32.113+0000 7f3cad73e700 1 -- 172.21.15.23:0/257807811 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3ca80a5c40 con 0x7f3ca80aaab0 2020-11-20T11:38:32.116 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:32.113+0000 7f3c8effd700 1 -- 172.21.15.23:0/257807811 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1472 (secure 0 0 0) 0x7f3c98015070 con 0x7f3ca80aaab0 2020-11-20T11:38:32.117 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":60,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:38:32.118 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:32.115+0000 7f3cad73e700 1 -- 172.21.15.23:0/257807811 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3c84056d60 msgr2=0x7f3c84059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:32.118 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:32.115+0000 7f3cad73e700 1 --2- 172.21.15.23:0/257807811 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3c84056d60 0x7f3c84059220 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f3ca80ae450 tx=0x7f3c9c008040).stop 2020-11-20T11:38:32.119 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:32.115+0000 7f3cad73e700 1 -- 172.21.15.23:0/257807811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ca80aaab0 msgr2=0x7f3ca80a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:32.119 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:32.115+0000 7f3cad73e700 1 --2- 172.21.15.23:0/257807811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ca80aaab0 0x7f3ca80a6250 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f3c9800b3c0 tx=0x7f3c98016000).stop 2020-11-20T11:38:32.119 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:32.115+0000 7f3cad73e700 1 -- 172.21.15.23:0/257807811 shutdown_connections 2020-11-20T11:38:32.119 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:32.115+0000 7f3cad73e700 1 --2- 172.21.15.23:0/257807811 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3c84056d60 0x7f3c84059220 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:32.119 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:32.115+0000 7f3cad73e700 1 --2- 172.21.15.23:0/257807811 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3ca814a190 0x7f3ca814a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:32.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:32.115+0000 7f3cad73e700 1 --2- 172.21.15.23:0/257807811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ca80aaab0 0x7f3ca80a6250 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:32.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:32.115+0000 7f3cad73e700 1 --2- 172.21.15.23:0/257807811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ca80aae90 0x7f3ca80a5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:32.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:32.115+0000 7f3cad73e700 1 -- 172.21.15.23:0/257807811 >> 172.21.15.23:0/257807811 conn(0x7f3ca80678e0 msgr2=0x7f3ca80aba00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:32.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:32.115+0000 7f3cad73e700 1 -- 172.21.15.23:0/257807811 shutdown_connections 2020-11-20T11:38:32.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:32.115+0000 7f3cad73e700 1 -- 172.21.15.23:0/257807811 wait complete. 2020-11-20T11:38:35.130 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:38:35.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:35.250+0000 7fdc1fa25700 1 Processor -- start 2020-11-20T11:38:35.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:35.251+0000 7fdc1fa25700 1 -- start start 2020-11-20T11:38:35.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:35.252+0000 7fdc1fa25700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fdc180a4c20 0x7fdc1813e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:35.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:35.252+0000 7fdc1fa25700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc180b1a20 con 0x7fdc180ab2a0 2020-11-20T11:38:35.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:35.252+0000 7fdc1fa25700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc180a8b90 con 0x7fdc180ad8d0 2020-11-20T11:38:35.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:35.252+0000 7fdc1fa25700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc1813f4a0 con 0x7fdc180a4c20 2020-11-20T11:38:35.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:35.252+0000 7fdc1d7c1700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fdc180ab2a0 0x7fdc180a4510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57794/0 (socket says 172.21.15.23:57794) 2020-11-20T11:38:35.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:35.252+0000 7fdc1d7c1700 1 -- 172.21.15.23:0/744614049 learned_addr learned my addr 172.21.15.23:0/744614049 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:38:35.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:35.252+0000 7fdc1dfc2700 1 --2- 172.21.15.23:0/744614049 >> v2:172.21.15.155:3300/0 conn(0x7fdc180a4c20 0x7fdc1813e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:35.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:35.252+0000 7fdc1dfc2700 1 -- 172.21.15.23:0/744614049 >> v2:172.21.15.155:3300/0 conn(0x7fdc180a4c20 msgr2=0x7fdc1813e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:38:35.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:35.252+0000 7fdc1dfc2700 1 -- 172.21.15.23:0/744614049 >> v2:172.21.15.155:3300/0 conn(0x7fdc180a4c20 msgr2=0x7fdc1813e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:38:35.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:35.252+0000 7fdc1dfc2700 1 --2- 172.21.15.23:0/744614049 >> v2:172.21.15.155:3300/0 conn(0x7fdc180a4c20 0x7fdc1813e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:38:35.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:35.252+0000 7fdc1dfc2700 1 --2- 172.21.15.23:0/744614049 >> v2:172.21.15.155:3300/0 conn(0x7fdc180a4c20 0x7fdc1813e570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:38:35.457 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:35.453+0000 7fdc1dfc2700 1 --2- 172.21.15.23:0/744614049 >> v2:172.21.15.155:3300/0 conn(0x7fdc180a4c20 0x7fdc1813e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:35.457 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:35.453+0000 7fdc1dfc2700 1 -- 172.21.15.23:0/744614049 >> v2:172.21.15.155:3300/0 conn(0x7fdc180a4c20 msgr2=0x7fdc1813e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:38:35.458 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:35.453+0000 7fdc1dfc2700 1 -- 172.21.15.23:0/744614049 >> v2:172.21.15.155:3300/0 conn(0x7fdc180a4c20 msgr2=0x7fdc1813e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:38:35.458 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:35.453+0000 7fdc1dfc2700 1 --2- 172.21.15.23:0/744614049 >> v2:172.21.15.155:3300/0 conn(0x7fdc180a4c20 0x7fdc1813e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:38:35.458 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:35.453+0000 7fdc1dfc2700 1 --2- 172.21.15.23:0/744614049 >> v2:172.21.15.155:3300/0 conn(0x7fdc180a4c20 0x7fdc1813e570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:38:35.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:35.854+0000 7fdc1dfc2700 1 --2- 172.21.15.23:0/744614049 >> v2:172.21.15.155:3300/0 conn(0x7fdc180a4c20 0x7fdc1813e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:35.859 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:35.854+0000 7fdc1dfc2700 1 -- 172.21.15.23:0/744614049 >> v2:172.21.15.155:3300/0 conn(0x7fdc180a4c20 msgr2=0x7fdc1813e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:38:35.859 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:35.854+0000 7fdc1dfc2700 1 -- 172.21.15.23:0/744614049 >> v2:172.21.15.155:3300/0 conn(0x7fdc180a4c20 msgr2=0x7fdc1813e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:38:35.859 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:35.854+0000 7fdc1dfc2700 1 --2- 172.21.15.23:0/744614049 >> v2:172.21.15.155:3300/0 conn(0x7fdc180a4c20 0x7fdc1813e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:38:35.859 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:35.854+0000 7fdc1dfc2700 1 --2- 172.21.15.23:0/744614049 >> v2:172.21.15.155:3300/0 conn(0x7fdc180a4c20 0x7fdc1813e570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:38:36.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:36.655+0000 7fdc1dfc2700 1 --2- 172.21.15.23:0/744614049 >> v2:172.21.15.155:3300/0 conn(0x7fdc180a4c20 0x7fdc1813e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:36.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:36.655+0000 7fdc1dfc2700 1 -- 172.21.15.23:0/744614049 >> v2:172.21.15.155:3300/0 conn(0x7fdc180a4c20 msgr2=0x7fdc1813e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:38:36.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:36.655+0000 7fdc1dfc2700 1 -- 172.21.15.23:0/744614049 >> v2:172.21.15.155:3300/0 conn(0x7fdc180a4c20 msgr2=0x7fdc1813e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:38:36.661 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:36.655+0000 7fdc1dfc2700 1 --2- 172.21.15.23:0/744614049 >> v2:172.21.15.155:3300/0 conn(0x7fdc180a4c20 0x7fdc1813e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:38:36.661 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:36.655+0000 7fdc1dfc2700 1 --2- 172.21.15.23:0/744614049 >> v2:172.21.15.155:3300/0 conn(0x7fdc180a4c20 0x7fdc1813e570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:38:38.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.252+0000 7fdc06ffd700 1 -- 172.21.15.23:0/744614049 >> v2:172.21.15.155:3300/0 conn(0x7fdc180a4c20 msgr2=0x7fdc1813e570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:38:38.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.252+0000 7fdc06ffd700 1 --2- 172.21.15.23:0/744614049 >> v2:172.21.15.155:3300/0 conn(0x7fdc180a4c20 0x7fdc1813e570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:38.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.252+0000 7fdc06ffd700 1 -- 172.21.15.23:0/744614049 >> v1:172.21.15.155:6789/0 conn(0x7fdc180ad8d0 legacy=0x7fdc181438a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:38.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.252+0000 7fdc06ffd700 1 -- 172.21.15.23:0/744614049 >> v1:172.21.15.23:6789/0 conn(0x7fdc180ab2a0 legacy=0x7fdc180a4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:38.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.252+0000 7fdc06ffd700 1 --2- 172.21.15.23:0/744614049 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdbf8003440 0x7fdbf8005890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:38.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.252+0000 7fdc06ffd700 1 --2- 172.21.15.23:0/744614049 >> v2:172.21.15.23:3300/0 conn(0x7fdbf8005ed0 0x7fdbf8008350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:38.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.252+0000 7fdc06ffd700 1 -- 172.21.15.23:0/744614049 --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbf8008980 con 0x7fdbf8005ed0 2020-11-20T11:38:38.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.252+0000 7fdc06ffd700 1 -- 172.21.15.23:0/744614049 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf8008af0 con 0x7fdbf8003440 2020-11-20T11:38:38.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.252+0000 7fdc1cfc0700 1 --2- 172.21.15.23:0/744614049 >> v2:172.21.15.23:3300/0 conn(0x7fdbf8005ed0 0x7fdbf8008350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:38.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.252+0000 7fdc1d7c1700 1 --2- 172.21.15.23:0/744614049 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdbf8003440 0x7fdbf8005890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:38.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.252+0000 7fdc1cfc0700 1 -- 172.21.15.23:0/744614049 >> v2:172.21.15.23:3300/0 conn(0x7fdbf8005ed0 msgr2=0x7fdbf8008350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:38:38.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.252+0000 7fdc1cfc0700 1 -- 172.21.15.23:0/744614049 >> v2:172.21.15.23:3300/0 conn(0x7fdbf8005ed0 msgr2=0x7fdbf8008350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:38:38.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.252+0000 7fdc1cfc0700 1 --2- 172.21.15.23:0/744614049 >> v2:172.21.15.23:3300/0 conn(0x7fdbf8005ed0 0x7fdbf8008350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:38:38.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.252+0000 7fdc1cfc0700 1 --2- 172.21.15.23:0/744614049 >> v2:172.21.15.23:3300/0 conn(0x7fdbf8005ed0 0x7fdbf8008350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:38:38.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.252+0000 7fdc1d7c1700 1 -- 172.21.15.23:0/744614049 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdbf8003440 msgr2=0x7fdbf8005890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:38:38.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.252+0000 7fdc1d7c1700 1 -- 172.21.15.23:0/744614049 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdbf8003440 msgr2=0x7fdbf8005890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:38:38.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.252+0000 7fdc1d7c1700 1 --2- 172.21.15.23:0/744614049 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdbf8003440 0x7fdbf8005890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:38:38.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.252+0000 7fdc1d7c1700 1 --2- 172.21.15.23:0/744614049 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdbf8003440 0x7fdbf8005890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:38:38.457 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.453+0000 7fdc1cfc0700 1 --2- 172.21.15.23:0/744614049 >> v2:172.21.15.23:3300/0 conn(0x7fdbf8005ed0 0x7fdbf8008350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:38.458 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.453+0000 7fdc1cfc0700 1 -- 172.21.15.23:0/744614049 >> v2:172.21.15.23:3300/0 conn(0x7fdbf8005ed0 msgr2=0x7fdbf8008350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:38:38.458 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.453+0000 7fdc1cfc0700 1 -- 172.21.15.23:0/744614049 >> v2:172.21.15.23:3300/0 conn(0x7fdbf8005ed0 msgr2=0x7fdbf8008350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:38:38.458 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.453+0000 7fdc1cfc0700 1 --2- 172.21.15.23:0/744614049 >> v2:172.21.15.23:3300/0 conn(0x7fdbf8005ed0 0x7fdbf8008350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:38:38.458 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.453+0000 7fdc1cfc0700 1 --2- 172.21.15.23:0/744614049 >> v2:172.21.15.23:3300/0 conn(0x7fdbf8005ed0 0x7fdbf8008350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:38:38.459 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.454+0000 7fdc1d7c1700 1 --2- 172.21.15.23:0/744614049 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdbf8003440 0x7fdbf8005890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:38.459 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.455+0000 7fdc1d7c1700 1 -- 172.21.15.23:0/744614049 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdbf8003440 msgr2=0x7fdbf8005890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:38:38.460 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.455+0000 7fdc1d7c1700 1 -- 172.21.15.23:0/744614049 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdbf8003440 msgr2=0x7fdbf8005890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:38:38.460 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.455+0000 7fdc1d7c1700 1 --2- 172.21.15.23:0/744614049 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdbf8003440 0x7fdbf8005890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:38:38.460 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.455+0000 7fdc1d7c1700 1 --2- 172.21.15.23:0/744614049 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdbf8003440 0x7fdbf8005890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:38:38.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.854+0000 7fdc1cfc0700 1 --2- 172.21.15.23:0/744614049 >> v2:172.21.15.23:3300/0 conn(0x7fdbf8005ed0 0x7fdbf8008350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:38.859 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.854+0000 7fdc1cfc0700 1 -- 172.21.15.23:0/744614049 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdbf8003440 msgr2=0x7fdbf8005890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:38:38.859 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.854+0000 7fdc1cfc0700 1 --2- 172.21.15.23:0/744614049 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdbf8003440 0x7fdbf8005890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:38.859 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.854+0000 7fdc1cfc0700 1 -- 172.21.15.23:0/744614049 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc180b1c50 con 0x7fdbf8005ed0 2020-11-20T11:38:38.859 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.854+0000 7fdc1cfc0700 1 --2- 172.21.15.23:0/744614049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdbf8005ed0 0x7fdbf8008350 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fdc080079e0 tx=0x7fdc08009040).ready entity=mon.0 client_cookie=4bd8f24a70b37d6f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:38.860 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.854+0000 7fdc07fff700 1 -- 172.21.15.23:0/744614049 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc080125a0 con 0x7fdbf8005ed0 2020-11-20T11:38:38.860 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.854+0000 7fdc07fff700 1 -- 172.21.15.23:0/744614049 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc08012700 con 0x7fdbf8005ed0 2020-11-20T11:38:38.860 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.854+0000 7fdc07fff700 1 -- 172.21.15.23:0/744614049 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc08012a80 con 0x7fdbf8005ed0 2020-11-20T11:38:38.864 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.855+0000 7fdc1fa25700 1 -- 172.21.15.23:0/744614049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdbf8005ed0 msgr2=0x7fdbf8008350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:38.864 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.855+0000 7fdc1fa25700 1 --2- 172.21.15.23:0/744614049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdbf8005ed0 0x7fdbf8008350 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fdc080079e0 tx=0x7fdc08009040).stop 2020-11-20T11:38:38.864 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.855+0000 7fdc1dfc2700 1 -- 172.21.15.23:0/744614049 reap_dead start 2020-11-20T11:38:38.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.855+0000 7fdc1fa25700 1 -- 172.21.15.23:0/744614049 shutdown_connections 2020-11-20T11:38:38.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.855+0000 7fdc1fa25700 1 --2- 172.21.15.23:0/744614049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdbf8005ed0 0x7fdbf8008350 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:38.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.855+0000 7fdc1fa25700 1 -- 172.21.15.23:0/744614049 >> 172.21.15.23:0/744614049 conn(0x7fdc180683b0 msgr2=0x7fdc180b1310 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:38.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.855+0000 7fdc1fa25700 1 -- 172.21.15.23:0/744614049 shutdown_connections 2020-11-20T11:38:38.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.855+0000 7fdc1fa25700 1 -- 172.21.15.23:0/744614049 wait complete. 2020-11-20T11:38:38.866 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.856+0000 7fdc1fa25700 1 Processor -- start 2020-11-20T11:38:38.866 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.856+0000 7fdc1fa25700 1 -- start start 2020-11-20T11:38:38.866 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.857+0000 7fdc1fa25700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbf8005ed0 0x7fdc180a04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:38.866 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.857+0000 7fdc1fa25700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc180a2960 0x7fdc18141f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:38.866 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.857+0000 7fdc1fa25700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc1809f720 0x7fdc1809fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:38.867 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.857+0000 7fdc1fa25700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc180a0b60 con 0x7fdc180a2960 2020-11-20T11:38:38.867 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.857+0000 7fdc1fa25700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc180a0e00 con 0x7fdbf8005ed0 2020-11-20T11:38:38.867 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.857+0000 7fdc1fa25700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc180a0070 con 0x7fdc1809f720 2020-11-20T11:38:38.867 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.857+0000 7fdc1cfc0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc180a2960 0x7fdc18141f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:38.867 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.857+0000 7fdc1cfc0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc180a2960 0x7fdc18141f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43568/0 (socket says 172.21.15.23:43568) 2020-11-20T11:38:38.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.857+0000 7fdc1cfc0700 1 -- 172.21.15.23:0/2931912799 learned_addr learned my addr 172.21.15.23:0/2931912799 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:38:38.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.858+0000 7fdc1d7c1700 1 --2- 172.21.15.23:0/2931912799 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbf8005ed0 0x7fdc180a04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:38.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.858+0000 7fdc1dfc2700 1 --2- 172.21.15.23:0/2931912799 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc1809f720 0x7fdc1809fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:38.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.858+0000 7fdc1cfc0700 1 -- 172.21.15.23:0/2931912799 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc1809f720 msgr2=0x7fdc1809fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:38.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.858+0000 7fdc1cfc0700 1 --2- 172.21.15.23:0/2931912799 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc1809f720 0x7fdc1809fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:38.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.858+0000 7fdc1cfc0700 1 -- 172.21.15.23:0/2931912799 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbf8005ed0 msgr2=0x7fdc180a04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:38.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.858+0000 7fdc1cfc0700 1 --2- 172.21.15.23:0/2931912799 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbf8005ed0 0x7fdc180a04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:38.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.858+0000 7fdc1cfc0700 1 -- 172.21.15.23:0/2931912799 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc1809d4a0 con 0x7fdc180a2960 2020-11-20T11:38:38.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.858+0000 7fdc1cfc0700 1 --2- 172.21.15.23:0/2931912799 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc180a2960 0x7fdc18141f40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fdc08003f10 tx=0x7fdc08009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:38.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.858+0000 7fdc057fa700 1 -- 172.21.15.23:0/2931912799 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc08026070 con 0x7fdc180a2960 2020-11-20T11:38:38.870 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.858+0000 7fdc057fa700 1 -- 172.21.15.23:0/2931912799 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc08014e70 con 0x7fdc180a2960 2020-11-20T11:38:38.870 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.858+0000 7fdc1fa25700 1 -- 172.21.15.23:0/2931912799 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc1809d6f0 con 0x7fdc180a2960 2020-11-20T11:38:38.870 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.858+0000 7fdc057fa700 1 -- 172.21.15.23:0/2931912799 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc08007450 con 0x7fdc180a2960 2020-11-20T11:38:38.870 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.858+0000 7fdc1fa25700 1 -- 172.21.15.23:0/2931912799 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc1809dc60 con 0x7fdc180a2960 2020-11-20T11:38:38.870 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.859+0000 7fdc1fa25700 1 -- 172.21.15.23:0/2931912799 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc180a09b0 con 0x7fdc180a2960 2020-11-20T11:38:38.871 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.860+0000 7fdc057fa700 1 -- 172.21.15.23:0/2931912799 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fdc08006740 con 0x7fdc180a2960 2020-11-20T11:38:38.871 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.860+0000 7fdc057fa700 1 --2- 172.21.15.23:0/2931912799 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fdbf0056d60 0x7fdbf0059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:38.871 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.860+0000 7fdc057fa700 1 -- 172.21.15.23:0/2931912799 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fdc080721e0 con 0x7fdc180a2960 2020-11-20T11:38:38.871 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.860+0000 7fdc1d7c1700 1 --2- 172.21.15.23:0/2931912799 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fdbf0056d60 0x7fdbf0059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:38.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.861+0000 7fdc1d7c1700 1 --2- 172.21.15.23:0/2931912799 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fdbf0056d60 0x7fdbf0059220 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fdc14007a60 tx=0x7fdc14009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:38.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:38.864+0000 7fdc057fa700 1 -- 172.21.15.23:0/2931912799 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fdc080474f0 con 0x7fdc180a2960 2020-11-20T11:38:39.129 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:39.125+0000 7fdc1fa25700 1 -- 172.21.15.23:0/2931912799 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fdc180a09b0 con 0x7fdc180a2960 2020-11-20T11:38:39.129 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:39.125+0000 7fdc057fa700 1 -- 172.21.15.23:0/2931912799 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1471 (secure 0 0 0) 0x7fdc0804aa90 con 0x7fdc180a2960 2020-11-20T11:38:39.130 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":64,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:38:39.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:39.126+0000 7fdc1fa25700 1 -- 172.21.15.23:0/2931912799 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fdbf0056d60 msgr2=0x7fdbf0059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:39.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:39.126+0000 7fdc1fa25700 1 --2- 172.21.15.23:0/2931912799 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fdbf0056d60 0x7fdbf0059220 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fdc14007a60 tx=0x7fdc14009040).stop 2020-11-20T11:38:39.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:39.126+0000 7fdc1fa25700 1 -- 172.21.15.23:0/2931912799 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc180a2960 msgr2=0x7fdc18141f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:39.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:39.127+0000 7fdc1fa25700 1 --2- 172.21.15.23:0/2931912799 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc180a2960 0x7fdc18141f40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fdc08003f10 tx=0x7fdc08009040).stop 2020-11-20T11:38:39.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:39.127+0000 7fdc1fa25700 1 -- 172.21.15.23:0/2931912799 shutdown_connections 2020-11-20T11:38:39.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:39.127+0000 7fdc1fa25700 1 --2- 172.21.15.23:0/2931912799 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fdbf0056d60 0x7fdbf0059220 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:39.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:39.127+0000 7fdc1fa25700 1 --2- 172.21.15.23:0/2931912799 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc1809f720 0x7fdc1809fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:39.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:39.127+0000 7fdc1fa25700 1 --2- 172.21.15.23:0/2931912799 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdbf8005ed0 0x7fdc180a04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:39.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:39.127+0000 7fdc1fa25700 1 --2- 172.21.15.23:0/2931912799 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc180a2960 0x7fdc18141f40 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:39.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:39.127+0000 7fdc1fa25700 1 -- 172.21.15.23:0/2931912799 >> 172.21.15.23:0/2931912799 conn(0x7fdc180683b0 msgr2=0x7fdc180ae5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:39.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:39.127+0000 7fdc1fa25700 1 -- 172.21.15.23:0/2931912799 shutdown_connections 2020-11-20T11:38:39.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:39.127+0000 7fdc1fa25700 1 -- 172.21.15.23:0/2931912799 wait complete. 2020-11-20T11:38:42.143 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:38:42.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.263+0000 7fcf2b59e700 1 Processor -- start 2020-11-20T11:38:42.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.263+0000 7fcf2b59e700 1 -- start start 2020-11-20T11:38:42.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.264+0000 7fcf2b59e700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fcf2c0ace70 0x7fcf2c13a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:42.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.264+0000 7fcf2b59e700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf2c0ad860 con 0x7fcf2c0ad2f0 2020-11-20T11:38:42.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.264+0000 7fcf2b59e700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf2c0aaa90 con 0x7fcf2c0a9040 2020-11-20T11:38:42.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.264+0000 7fcf2b59e700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf2c0aa8a0 con 0x7fcf2c0ace70 2020-11-20T11:38:42.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.264+0000 7fcf29d9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fcf2c0ace70 0x7fcf2c13a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:42.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.264+0000 7fcf2ad9d700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fcf2c0ad2f0 0x7fcf2c13ad20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57828/0 (socket says 172.21.15.23:57828) 2020-11-20T11:38:42.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.264+0000 7fcf2ad9d700 1 -- 172.21.15.23:0/1553860126 learned_addr learned my addr 172.21.15.23:0/1553860126 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:38:42.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.265+0000 7fcf29d9b700 1 -- 172.21.15.23:0/1553860126 >> v1:172.21.15.155:6789/0 conn(0x7fcf2c0a9040 legacy=0x7fcf2c0aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:42.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.265+0000 7fcf29d9b700 1 -- 172.21.15.23:0/1553860126 >> v1:172.21.15.23:6789/0 conn(0x7fcf2c0ad2f0 legacy=0x7fcf2c13ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:42.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.265+0000 7fcf29d9b700 1 -- 172.21.15.23:0/1553860126 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf2c0a9420 con 0x7fcf2c0ace70 2020-11-20T11:38:42.274 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.265+0000 7fcf29d9b700 1 --2- 172.21.15.23:0/1553860126 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf2c0ace70 0x7fcf2c13a830 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fcf20006fd0 tx=0x7fcf20006e30).ready entity=mon.0 client_cookie=c16228af2134a670 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:42.274 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.265+0000 7fcf2959a700 1 -- 172.21.15.23:0/1553860126 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf200097e0 con 0x7fcf2c0ace70 2020-11-20T11:38:42.274 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.265+0000 7fcf2959a700 1 -- 172.21.15.23:0/1553860126 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf20000e90 con 0x7fcf2c0ace70 2020-11-20T11:38:42.274 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.265+0000 7fcf2959a700 1 -- 172.21.15.23:0/1553860126 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf20009b50 con 0x7fcf2c0ace70 2020-11-20T11:38:42.275 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.266+0000 7fcf2b59e700 1 -- 172.21.15.23:0/1553860126 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf2c0ace70 msgr2=0x7fcf2c13a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:42.275 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.266+0000 7fcf2b59e700 1 --2- 172.21.15.23:0/1553860126 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf2c0ace70 0x7fcf2c13a830 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fcf20006fd0 tx=0x7fcf20006e30).stop 2020-11-20T11:38:42.275 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.266+0000 7fcf2b59e700 1 -- 172.21.15.23:0/1553860126 shutdown_connections 2020-11-20T11:38:42.275 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.266+0000 7fcf2b59e700 1 --2- 172.21.15.23:0/1553860126 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf2c0ace70 0x7fcf2c13a830 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:42.275 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.266+0000 7fcf2b59e700 1 -- 172.21.15.23:0/1553860126 >> 172.21.15.23:0/1553860126 conn(0x7fcf2c0683b0 msgr2=0x7fcf2c0b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:42.275 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.266+0000 7fcf2b59e700 1 -- 172.21.15.23:0/1553860126 shutdown_connections 2020-11-20T11:38:42.276 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.266+0000 7fcf2b59e700 1 -- 172.21.15.23:0/1553860126 wait complete. 2020-11-20T11:38:42.276 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.267+0000 7fcf2b59e700 1 Processor -- start 2020-11-20T11:38:42.276 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.267+0000 7fcf2b59e700 1 -- start start 2020-11-20T11:38:42.279 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.267+0000 7fcf2b59e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcf2c0a9040 0x7fcf2c135370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:42.279 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.267+0000 7fcf2b59e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf2c0ace70 0x7fcf2c13dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:42.279 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.267+0000 7fcf2b59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf2c0ad2f0 0x7fcf2c13e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:42.280 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.267+0000 7fcf2b59e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c13efb0 con 0x7fcf2c0ad2f0 2020-11-20T11:38:42.280 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.267+0000 7fcf2b59e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c13d870 con 0x7fcf2c0ace70 2020-11-20T11:38:42.280 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.267+0000 7fcf2b59e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c13f250 con 0x7fcf2c0a9040 2020-11-20T11:38:42.280 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.268+0000 7fcf2ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf2c0ad2f0 0x7fcf2c13e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:42.280 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.268+0000 7fcf2ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf2c0ad2f0 0x7fcf2c13e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43584/0 (socket says 172.21.15.23:43584) 2020-11-20T11:38:42.281 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.268+0000 7fcf2ad9d700 1 -- 172.21.15.23:0/1328461298 learned_addr learned my addr 172.21.15.23:0/1328461298 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:38:42.281 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.268+0000 7fcf29d9b700 1 --2- 172.21.15.23:0/1328461298 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf2c0ace70 0x7fcf2c13dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:42.281 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.268+0000 7fcf2a59c700 1 --2- 172.21.15.23:0/1328461298 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcf2c0a9040 0x7fcf2c135370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:42.281 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.268+0000 7fcf2a59c700 1 -- 172.21.15.23:0/1328461298 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcf2c0a9040 msgr2=0x7fcf2c135370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2020-11-20T11:38:42.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.268+0000 7fcf2a59c700 1 -- 172.21.15.23:0/1328461298 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcf2c0a9040 msgr2=0x7fcf2c135370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:38:42.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.268+0000 7fcf2a59c700 1 --2- 172.21.15.23:0/1328461298 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcf2c0a9040 0x7fcf2c135370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:38:42.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.268+0000 7fcf2a59c700 1 --2- 172.21.15.23:0/1328461298 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcf2c0a9040 0x7fcf2c135370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:38:42.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.268+0000 7fcf2ad9d700 1 -- 172.21.15.23:0/1328461298 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcf2c0a9040 msgr2=0x7fcf2c135370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:38:42.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.268+0000 7fcf2ad9d700 1 --2- 172.21.15.23:0/1328461298 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcf2c0a9040 0x7fcf2c135370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:42.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.268+0000 7fcf2ad9d700 1 -- 172.21.15.23:0/1328461298 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf2c0ace70 msgr2=0x7fcf2c13dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:42.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.268+0000 7fcf2ad9d700 1 --2- 172.21.15.23:0/1328461298 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf2c0ace70 0x7fcf2c13dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:42.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.268+0000 7fcf2ad9d700 1 -- 172.21.15.23:0/1328461298 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf2c13e200 con 0x7fcf2c0ad2f0 2020-11-20T11:38:42.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.268+0000 7fcf2ad9d700 1 --2- 172.21.15.23:0/1328461298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf2c0ad2f0 0x7fcf2c13e6a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fcf2400ca10 tx=0x7fcf240013a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:42.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.268+0000 7fcf1a7fc700 1 -- 172.21.15.23:0/1328461298 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf24005bc0 con 0x7fcf2c0ad2f0 2020-11-20T11:38:42.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.268+0000 7fcf1a7fc700 1 -- 172.21.15.23:0/1328461298 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf24005d20 con 0x7fcf2c0ad2f0 2020-11-20T11:38:42.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.268+0000 7fcf2b59e700 1 -- 172.21.15.23:0/1328461298 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf2c09e360 con 0x7fcf2c0ad2f0 2020-11-20T11:38:42.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.268+0000 7fcf1a7fc700 1 -- 172.21.15.23:0/1328461298 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf24007320 con 0x7fcf2c0ad2f0 2020-11-20T11:38:42.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.268+0000 7fcf2b59e700 1 -- 172.21.15.23:0/1328461298 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf2c13ed50 con 0x7fcf2c0ad2f0 2020-11-20T11:38:42.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.270+0000 7fcf2b59e700 1 -- 172.21.15.23:0/1328461298 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf2c13b6d0 con 0x7fcf2c0ad2f0 2020-11-20T11:38:42.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.270+0000 7fcf1a7fc700 1 -- 172.21.15.23:0/1328461298 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fcf24010030 con 0x7fcf2c0ad2f0 2020-11-20T11:38:42.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.270+0000 7fcf1a7fc700 1 --2- 172.21.15.23:0/1328461298 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcf00056d60 0x7fcf00059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:42.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.270+0000 7fcf1a7fc700 1 -- 172.21.15.23:0/1328461298 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fcf2406ff90 con 0x7fcf2c0ad2f0 2020-11-20T11:38:42.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.270+0000 7fcf2a59c700 1 --2- 172.21.15.23:0/1328461298 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcf00056d60 0x7fcf00059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:42.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.271+0000 7fcf2a59c700 1 --2- 172.21.15.23:0/1328461298 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcf00056d60 0x7fcf00059220 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fcf1c0079d0 tx=0x7fcf1c009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:42.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.275+0000 7fcf1a7fc700 1 -- 172.21.15.23:0/1328461298 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fcf24045280 con 0x7fcf2c0ad2f0 2020-11-20T11:38:42.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.519+0000 7fcf2b59e700 1 -- 172.21.15.23:0/1328461298 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fcf2c13da30 con 0x7fcf2c0ad2f0 2020-11-20T11:38:42.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.520+0000 7fcf1a7fc700 1 -- 172.21.15.23:0/1328461298 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1477 (secure 0 0 0) 0x7fcf240488a0 con 0x7fcf2c0ad2f0 2020-11-20T11:38:42.524 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:38:42.526 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.522+0000 7fcf2b59e700 1 -- 172.21.15.23:0/1328461298 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcf00056d60 msgr2=0x7fcf00059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:42.527 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.522+0000 7fcf2b59e700 1 --2- 172.21.15.23:0/1328461298 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcf00056d60 0x7fcf00059220 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fcf1c0079d0 tx=0x7fcf1c009040).stop 2020-11-20T11:38:42.527 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.522+0000 7fcf2b59e700 1 -- 172.21.15.23:0/1328461298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf2c0ad2f0 msgr2=0x7fcf2c13e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:42.527 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.522+0000 7fcf2b59e700 1 --2- 172.21.15.23:0/1328461298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf2c0ad2f0 0x7fcf2c13e6a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fcf2400ca10 tx=0x7fcf240013a0).stop 2020-11-20T11:38:42.527 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.523+0000 7fcf2b59e700 1 -- 172.21.15.23:0/1328461298 shutdown_connections 2020-11-20T11:38:42.527 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.523+0000 7fcf2b59e700 1 --2- 172.21.15.23:0/1328461298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf2c0ad2f0 0x7fcf2c13e6a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:42.528 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.523+0000 7fcf2b59e700 1 --2- 172.21.15.23:0/1328461298 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcf00056d60 0x7fcf00059220 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:42.528 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.523+0000 7fcf2b59e700 1 --2- 172.21.15.23:0/1328461298 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcf2c0a9040 0x7fcf2c135370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:42.528 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.523+0000 7fcf2b59e700 1 --2- 172.21.15.23:0/1328461298 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcf2c0ace70 0x7fcf2c13dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:42.528 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.523+0000 7fcf2b59e700 1 -- 172.21.15.23:0/1328461298 >> 172.21.15.23:0/1328461298 conn(0x7fcf2c0683b0 msgr2=0x7fcf2c13fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:42.529 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.523+0000 7fcf2b59e700 1 -- 172.21.15.23:0/1328461298 shutdown_connections 2020-11-20T11:38:42.529 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.523+0000 7fcf2b59e700 1 -- 172.21.15.23:0/1328461298 wait complete. 2020-11-20T11:38:42.541 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:38:42.542 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:38:42.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.663+0000 7f484d897700 1 Processor -- start 2020-11-20T11:38:42.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.663+0000 7f484d897700 1 -- start start 2020-11-20T11:38:42.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.664+0000 7f484d897700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4848061380 0x7f484813afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:42.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.664+0000 7f484d897700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f48480aacf0 0x7f4848138160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:42.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.664+0000 7f484d897700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48480a1c30 con 0x7f48480a9810 2020-11-20T11:38:42.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.664+0000 7f484d897700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48480a24f0 con 0x7f48480aacf0 2020-11-20T11:38:42.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.664+0000 7f484d897700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48480a9db0 con 0x7f4848061380 2020-11-20T11:38:42.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.664+0000 7f48477fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f48480aacf0 0x7f4848138160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:42.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.664+0000 7f48477fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f48480aacf0 0x7f4848138160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43592/0 (socket says 172.21.15.23:43592) 2020-11-20T11:38:42.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.664+0000 7f48477fe700 1 -- 172.21.15.23:0/3002352991 learned_addr learned my addr 172.21.15.23:0/3002352991 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:38:42.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.664+0000 7f4846ffd700 1 --2- 172.21.15.23:0/3002352991 >> v2:172.21.15.155:3300/0 conn(0x7f4848061380 0x7f484813afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:42.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.664+0000 7f48477fe700 1 -- 172.21.15.23:0/3002352991 >> v2:172.21.15.155:3300/0 conn(0x7f4848061380 msgr2=0x7f484813afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:38:42.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.664+0000 7f48477fe700 1 --2- 172.21.15.23:0/3002352991 >> v2:172.21.15.155:3300/0 conn(0x7f4848061380 0x7f484813afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:42.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.664+0000 7f48477fe700 1 -- 172.21.15.23:0/3002352991 >> v1:172.21.15.155:6789/0 conn(0x7f48480a9810 legacy=0x7f4848137a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:42.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.664+0000 7f48477fe700 1 -- 172.21.15.23:0/3002352991 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484813ab40 con 0x7f48480aacf0 2020-11-20T11:38:42.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.664+0000 7f48477fe700 1 --2- 172.21.15.23:0/3002352991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48480aacf0 0x7f4848138160 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f484805cbf0 tx=0x7f48400092d0).ready entity=mon.0 client_cookie=b59c3dfe72e5ee8b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:42.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.665+0000 7f4845ffb700 1 -- 172.21.15.23:0/3002352991 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484000bbf0 con 0x7f48480aacf0 2020-11-20T11:38:42.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.665+0000 7f4845ffb700 1 -- 172.21.15.23:0/3002352991 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484000bd50 con 0x7f48480aacf0 2020-11-20T11:38:42.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.665+0000 7f4845ffb700 1 -- 172.21.15.23:0/3002352991 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4840011610 con 0x7f48480aacf0 2020-11-20T11:38:42.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.665+0000 7f484d897700 1 -- 172.21.15.23:0/3002352991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48480aacf0 msgr2=0x7f4848138160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:42.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.665+0000 7f484d897700 1 --2- 172.21.15.23:0/3002352991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48480aacf0 0x7f4848138160 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f484805cbf0 tx=0x7f48400092d0).stop 2020-11-20T11:38:42.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.665+0000 7f484d897700 1 -- 172.21.15.23:0/3002352991 shutdown_connections 2020-11-20T11:38:42.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.665+0000 7f484d897700 1 --2- 172.21.15.23:0/3002352991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48480aacf0 0x7f4848138160 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:42.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.665+0000 7f484d897700 1 --2- 172.21.15.23:0/3002352991 >> v2:172.21.15.155:3300/0 conn(0x7f4848061380 0x7f484813afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:42.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.665+0000 7f484d897700 1 -- 172.21.15.23:0/3002352991 >> 172.21.15.23:0/3002352991 conn(0x7f48480ae120 msgr2=0x7f48480aa0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:42.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.665+0000 7f484d897700 1 -- 172.21.15.23:0/3002352991 shutdown_connections 2020-11-20T11:38:42.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.665+0000 7f484d897700 1 -- 172.21.15.23:0/3002352991 wait complete. 2020-11-20T11:38:42.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.666+0000 7f484d897700 1 Processor -- start 2020-11-20T11:38:42.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.667+0000 7f484d897700 1 -- start start 2020-11-20T11:38:42.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.667+0000 7f484d897700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4848061380 0x7f48480a4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:42.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.667+0000 7f484d897700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48480a9810 0x7f484814d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:42.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.667+0000 7f484d897700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f48480aacf0 0x7f4848066d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:42.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.667+0000 7f484d897700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48480657e0 con 0x7f48480a9810 2020-11-20T11:38:42.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.667+0000 7f484d897700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4848065a80 con 0x7f4848061380 2020-11-20T11:38:42.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.667+0000 7f484d897700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4848065d20 con 0x7f48480aacf0 2020-11-20T11:38:42.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.667+0000 7f48467fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48480a9810 0x7f484814d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:42.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.667+0000 7f48467fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48480a9810 0x7f484814d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43596/0 (socket says 172.21.15.23:43596) 2020-11-20T11:38:42.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.667+0000 7f48467fc700 1 -- 172.21.15.23:0/3905283715 learned_addr learned my addr 172.21.15.23:0/3905283715 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:38:42.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.667+0000 7f48477fe700 1 --2- 172.21.15.23:0/3905283715 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f48480aacf0 0x7f4848066d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:42.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.667+0000 7f4846ffd700 1 --2- 172.21.15.23:0/3905283715 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4848061380 0x7f48480a4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:42.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.667+0000 7f48477fe700 1 -- 172.21.15.23:0/3905283715 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f48480aacf0 msgr2=0x7f4848066d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2020-11-20T11:38:42.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.667+0000 7f48477fe700 1 -- 172.21.15.23:0/3905283715 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f48480aacf0 msgr2=0x7f4848066d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:38:42.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.667+0000 7f48477fe700 1 --2- 172.21.15.23:0/3905283715 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f48480aacf0 0x7f4848066d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:38:42.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.668+0000 7f48477fe700 1 --2- 172.21.15.23:0/3905283715 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f48480aacf0 0x7f4848066d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:38:42.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.668+0000 7f48467fc700 1 -- 172.21.15.23:0/3905283715 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f48480aacf0 msgr2=0x7f4848066d30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:38:42.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.668+0000 7f48467fc700 1 --2- 172.21.15.23:0/3905283715 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f48480aacf0 0x7f4848066d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:42.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.668+0000 7f48467fc700 1 -- 172.21.15.23:0/3905283715 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4848061380 msgr2=0x7f48480a4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:42.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.668+0000 7f48467fc700 1 --2- 172.21.15.23:0/3905283715 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4848061380 0x7f48480a4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:42.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.668+0000 7f48467fc700 1 -- 172.21.15.23:0/3905283715 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484814daf0 con 0x7f48480a9810 2020-11-20T11:38:42.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.668+0000 7f48467fc700 1 --2- 172.21.15.23:0/3905283715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48480a9810 0x7f484814d600 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f483c00c710 tx=0x7f483c007c90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:42.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.668+0000 7f482effd700 1 -- 172.21.15.23:0/3905283715 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f483c007020 con 0x7f48480a9810 2020-11-20T11:38:42.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.668+0000 7f482effd700 1 -- 172.21.15.23:0/3905283715 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f483c001e90 con 0x7f48480a9810 2020-11-20T11:38:42.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.668+0000 7f482effd700 1 -- 172.21.15.23:0/3905283715 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f483c0073b0 con 0x7f48480a9810 2020-11-20T11:38:42.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.668+0000 7f484d897700 1 -- 172.21.15.23:0/3905283715 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48481387d0 con 0x7f48480a9810 2020-11-20T11:38:42.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.668+0000 7f484d897700 1 -- 172.21.15.23:0/3905283715 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48480661c0 con 0x7f48480a9810 2020-11-20T11:38:42.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.669+0000 7f482cff9700 1 --2- 172.21.15.23:0/3905283715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4818002a50 0x7f4818004f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:42.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.669+0000 7f482cff9700 1 -- 172.21.15.23:0/3905283715 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4818007bb0 con 0x7f4818002a50 2020-11-20T11:38:42.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.669+0000 7f4846ffd700 1 --2- 172.21.15.23:0/3905283715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4818002a50 0x7f4818004f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:42.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.669+0000 7f4846ffd700 1 -- 172.21.15.23:0/3905283715 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4818007d70 con 0x7f4818002a50 2020-11-20T11:38:42.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.669+0000 7f4846ffd700 1 --2- 172.21.15.23:0/3905283715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4818002a50 0x7f4818004f10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f4838011cb0 tx=0x7f4838006e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:42.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.670+0000 7f482effd700 1 -- 172.21.15.23:0/3905283715 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f483c010030 con 0x7f48480a9810 2020-11-20T11:38:42.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.670+0000 7f482effd700 1 --2- 172.21.15.23:0/3905283715 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4824056d60 0x7f4824059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:42.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.670+0000 7f482effd700 1 -- 172.21.15.23:0/3905283715 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f483c06f6c0 con 0x7f48480a9810 2020-11-20T11:38:42.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.670+0000 7f482effd700 1 -- 172.21.15.23:0/3905283715 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4838011ec0 con 0x7f4818002a50 2020-11-20T11:38:42.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.670+0000 7f482effd700 1 -- 172.21.15.23:0/3905283715 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f483800fd60 con 0x7f4818002a50 2020-11-20T11:38:42.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.670+0000 7f482effd700 1 -- 172.21.15.23:0/3905283715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4818002a50 msgr2=0x7f4818004f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:42.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.670+0000 7f482effd700 1 --2- 172.21.15.23:0/3905283715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4818002a50 0x7f4818004f10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f4838011cb0 tx=0x7f4838006e30).stop 2020-11-20T11:38:42.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.670+0000 7f482effd700 1 -- 172.21.15.23:0/3905283715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4818002a50 msgr2=0x7f4818004f10 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:38:42.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.670+0000 7f482effd700 1 --2- 172.21.15.23:0/3905283715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4818002a50 0x7f4818004f10 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:42.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.670+0000 7f48477fe700 1 --2- 172.21.15.23:0/3905283715 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4824056d60 0x7f4824059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:42.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.671+0000 7f48477fe700 1 --2- 172.21.15.23:0/3905283715 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4824056d60 0x7f4824059220 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f48480ab120 tx=0x7f4840016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:42.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.677+0000 7f482cff9700 1 --2- 172.21.15.23:0/3905283715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4818006c80 0x7f48180070f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:42.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.677+0000 7f482cff9700 1 -- 172.21.15.23:0/3905283715 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4818007700 con 0x7f4818006c80 2020-11-20T11:38:42.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.677+0000 7f4846ffd700 1 --2- 172.21.15.23:0/3905283715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4818006c80 0x7f48180070f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:42.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.678+0000 7f4846ffd700 1 -- 172.21.15.23:0/3905283715 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f48180078c0 con 0x7f4818006c80 2020-11-20T11:38:42.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.678+0000 7f4846ffd700 1 --2- 172.21.15.23:0/3905283715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4818006c80 0x7f48180070f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f4838018000 tx=0x7f483801b000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:42.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.678+0000 7f482effd700 1 -- 172.21.15.23:0/3905283715 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4838006240 con 0x7f4818006c80 2020-11-20T11:38:42.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.678+0000 7f482effd700 1 -- 172.21.15.23:0/3905283715 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4445 (secure 0 0 0) 0x7f4838027070 con 0x7f4818006c80 2020-11-20T11:38:42.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.678+0000 7f482effd700 1 -- 172.21.15.23:0/3905283715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4818006c80 msgr2=0x7f48180070f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:42.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.678+0000 7f482effd700 1 --2- 172.21.15.23:0/3905283715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4818006c80 0x7f48180070f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f4838018000 tx=0x7f483801b000).stop 2020-11-20T11:38:42.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.678+0000 7f482effd700 1 -- 172.21.15.23:0/3905283715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4818006c80 msgr2=0x7f48180070f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:38:42.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.678+0000 7f482effd700 1 --2- 172.21.15.23:0/3905283715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4818006c80 0x7f48180070f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:42.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.678+0000 7f482cff9700 1 -- 172.21.15.23:0/3905283715 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4824056d60 msgr2=0x7f4824059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:42.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.678+0000 7f482cff9700 1 --2- 172.21.15.23:0/3905283715 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4824056d60 0x7f4824059220 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f48480ab120 tx=0x7f4840016000).stop 2020-11-20T11:38:42.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.678+0000 7f482cff9700 1 -- 172.21.15.23:0/3905283715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48480a9810 msgr2=0x7f484814d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:42.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.678+0000 7f482cff9700 1 --2- 172.21.15.23:0/3905283715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48480a9810 0x7f484814d600 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f483c00c710 tx=0x7f483c007c90).stop 2020-11-20T11:38:42.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.678+0000 7f48477fe700 1 -- 172.21.15.23:0/3905283715 reap_dead start 2020-11-20T11:38:42.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.678+0000 7f482cff9700 1 -- 172.21.15.23:0/3905283715 shutdown_connections 2020-11-20T11:38:42.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.678+0000 7f482cff9700 1 --2- 172.21.15.23:0/3905283715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48480a9810 0x7f484814d600 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:42.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.679+0000 7f482cff9700 1 -- 172.21.15.23:0/3905283715 >> 172.21.15.23:0/3905283715 conn(0x7f48480ae120 msgr2=0x7f4848139560 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:42.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.679+0000 7f482cff9700 1 -- 172.21.15.23:0/3905283715 shutdown_connections 2020-11-20T11:38:42.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.679+0000 7f482cff9700 1 -- 172.21.15.23:0/3905283715 wait complete. 2020-11-20T11:38:42.694 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:38:42.694 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:38:42.694 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:38:42.694 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:38:42.694 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 66, 2020-11-20T11:38:42.695 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:38:42.695 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:38:42.695 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:38:42.695 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:38:42.695 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:42.696 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 2, 2020-11-20T11:38:42.696 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:38:42.696 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:38:42.696 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:38:42.696 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:38:42.697 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:38:42.697 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:38:42.697 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:38:42.697 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:38:42.698 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:38:42.698 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:38:42.698 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:38:42.698 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:42.698 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:38:42.699 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:38:42.699 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:38:42.699 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:38:42.699 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:38:42.699 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:38:42.700 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:38:42.700 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:38:42.700 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:38:42.700 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:38:42.700 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:42.701 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:42.701 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:38:42.701 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:38:42.701 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:38:42.701 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:38:42.702 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:38:42.702 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:38:42.702 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:38:42.702 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:38:42.703 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:38:42.703 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:38:42.703 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:38:42.703 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:38:42.704 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:38:42.704 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:38:42.704 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:38:42.704 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:38:42.704 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:38:42.705 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:38:42.705 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:38:42.705 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:38:42.705 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:38:42.705 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:38:42.706 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:38:42.706 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:42.706 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:38:42.706 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:42.706 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:38:42.707 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:42.707 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:38:42.707 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:38:42.707 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:38:42.707 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:38:42.708 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:42.708 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:38:42.708 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:38:42.708 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:42.709 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:42.709 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:42.709 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:38:42.709 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:38:42.709 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:42.710 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:42.710 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:42.710 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:42.710 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:38:42.711 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:38:42.711 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:38:42.711 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:38:42.711 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:38:42.711 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:42.712 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:42.712 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:38:42.712 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:38:42.712 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:38:42.712 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:38:42.713 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:42.713 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:38:42.713 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:38:42.713 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:42.714 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:42.714 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:42.714 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:38:42.714 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:38:42.715 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:42.715 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:42.715 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:42.715 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:42.715 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:38:42.715 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:38:42.716 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:38:42.716 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:38:42.716 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:38:42.716 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:42.717 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:42.717 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:38:42.717 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:38:42.717 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:38:42.717 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:38:42.717 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:42.718 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:38:42.718 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:38:42.718 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:42.718 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:42.719 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:42.719 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:38:42.719 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:38:42.719 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:42.719 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:42.720 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:42.720 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:42.720 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:38:42.720 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:38:42.720 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:38:42.721 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:38:42.721 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:38:42.721 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:42.721 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:42.721 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:42.722 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:38:42.722 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:38:42.722 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:42.722 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:38:42.722 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:38:42.723 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:38:42.723 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:42.723 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:42.723 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:38:42.723 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:42.724 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:38:42.724 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:38:42.724 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:38:42.724 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:42.725 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:42.725 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:38:42.725 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:42.725 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:38:42.725 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:38:42.726 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:38:42.726 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:42.726 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:42.726 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:42.726 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:38:42.727 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:38:42.728 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:38:42.851 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.847+0000 7fb3b8b1b700 1 Processor -- start 2020-11-20T11:38:42.851 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.847+0000 7fb3b8b1b700 1 -- start start 2020-11-20T11:38:42.853 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.847+0000 7fb3b8b1b700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb3b4061380 0x7fb3b40a9210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:42.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.847+0000 7fb3b8b1b700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b413b800 con 0x7fb3b40a9700 2020-11-20T11:38:42.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.847+0000 7fb3b8b1b700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b40a6e60 con 0x7fb3b40a4e00 2020-11-20T11:38:42.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.847+0000 7fb3b8b1b700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3b40a75a0 con 0x7fb3b4061380 2020-11-20T11:38:42.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.848+0000 7fb3b3fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fb3b40a9700 0x7fb3b413b0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57860/0 (socket says 172.21.15.23:57860) 2020-11-20T11:38:42.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.848+0000 7fb3b3fff700 1 -- 172.21.15.23:0/4264733425 learned_addr learned my addr 172.21.15.23:0/4264733425 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:38:42.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.848+0000 7fb3b37fe700 1 --2- 172.21.15.23:0/4264733425 >> v2:172.21.15.155:3300/0 conn(0x7fb3b4061380 0x7fb3b40a9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:42.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.848+0000 7fb3b27fc700 1 -- 172.21.15.23:0/4264733425 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3435402477 0 0) 0x7fb3b413b800 con 0x7fb3b40a9700 2020-11-20T11:38:42.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.848+0000 7fb3b27fc700 1 -- 172.21.15.23:0/4264733425 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3a0003280 con 0x7fb3b40a9700 2020-11-20T11:38:42.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.848+0000 7fb3b37fe700 1 -- 172.21.15.23:0/4264733425 >> v1:172.21.15.155:6789/0 conn(0x7fb3b40a4e00 legacy=0x7fb3b413a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:42.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.848+0000 7fb3b37fe700 1 -- 172.21.15.23:0/4264733425 >> v1:172.21.15.23:6789/0 conn(0x7fb3b40a9700 legacy=0x7fb3b413b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:42.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.848+0000 7fb3b27fc700 1 -- 172.21.15.23:0/4264733425 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755126594 0 0) 0x7fb3b40a6e60 con 0x7fb3b40a4e00 2020-11-20T11:38:42.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.848+0000 7fb3b37fe700 1 -- 172.21.15.23:0/4264733425 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b40a7100 con 0x7fb3b4061380 2020-11-20T11:38:42.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.848+0000 7fb3b37fe700 1 --2- 172.21.15.23:0/4264733425 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3b4061380 0x7fb3b40a9210 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fb3a4006fd0 tx=0x7fb3a4006e30).ready entity=mon.1 client_cookie=5f657fd303c19617 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:42.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.848+0000 7fb3b27fc700 1 -- 172.21.15.23:0/4264733425 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a400bec0 con 0x7fb3b4061380 2020-11-20T11:38:42.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.848+0000 7fb3b27fc700 1 -- 172.21.15.23:0/4264733425 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a4000e90 con 0x7fb3b4061380 2020-11-20T11:38:42.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.848+0000 7fb3b27fc700 1 -- 172.21.15.23:0/4264733425 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a40099f0 con 0x7fb3b4061380 2020-11-20T11:38:42.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.849+0000 7fb3b8b1b700 1 -- 172.21.15.23:0/4264733425 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3b4061380 msgr2=0x7fb3b40a9210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:42.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.849+0000 7fb3b8b1b700 1 --2- 172.21.15.23:0/4264733425 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3b4061380 0x7fb3b40a9210 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fb3a4006fd0 tx=0x7fb3a4006e30).stop 2020-11-20T11:38:42.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.849+0000 7fb3b8b1b700 1 -- 172.21.15.23:0/4264733425 shutdown_connections 2020-11-20T11:38:42.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.849+0000 7fb3b8b1b700 1 --2- 172.21.15.23:0/4264733425 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3b4061380 0x7fb3b40a9210 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:42.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.849+0000 7fb3b8b1b700 1 -- 172.21.15.23:0/4264733425 >> 172.21.15.23:0/4264733425 conn(0x7fb3b40ae0a0 msgr2=0x7fb3b40aa0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:42.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.849+0000 7fb3b8b1b700 1 -- 172.21.15.23:0/4264733425 shutdown_connections 2020-11-20T11:38:42.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.849+0000 7fb3b8b1b700 1 -- 172.21.15.23:0/4264733425 wait complete. 2020-11-20T11:38:42.861 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.850+0000 7fb3b8b1b700 1 Processor -- start 2020-11-20T11:38:42.862 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.850+0000 7fb3b8b1b700 1 -- start start 2020-11-20T11:38:42.862 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.851+0000 7fb3b8b1b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3b4061380 0x7fb3b409f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:42.862 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.851+0000 7fb3b8b1b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3b40a4e00 0x7fb3b4066820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:42.862 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.851+0000 7fb3b8b1b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3b40a9700 0x7fb3b4136ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:42.863 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.851+0000 7fb3b8b1b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b4067ac0 con 0x7fb3b4061380 2020-11-20T11:38:42.863 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.851+0000 7fb3b8b1b700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b4067d60 con 0x7fb3b40a9700 2020-11-20T11:38:42.863 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.851+0000 7fb3b8b1b700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b4066d10 con 0x7fb3b40a4e00 2020-11-20T11:38:42.863 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.851+0000 7fb3b37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3b4061380 0x7fb3b409f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:42.863 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.851+0000 7fb3b37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3b4061380 0x7fb3b409f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43612/0 (socket says 172.21.15.23:43612) 2020-11-20T11:38:42.864 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.851+0000 7fb3b37fe700 1 -- 172.21.15.23:0/1252771995 learned_addr learned my addr 172.21.15.23:0/1252771995 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:38:42.864 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.851+0000 7fb3b2ffd700 1 --2- 172.21.15.23:0/1252771995 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3b40a4e00 0x7fb3b4066820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:42.864 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.851+0000 7fb3b3fff700 1 --2- 172.21.15.23:0/1252771995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3b40a9700 0x7fb3b4136ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:42.864 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.851+0000 7fb3b2ffd700 1 -- 172.21.15.23:0/1252771995 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3b40a4e00 msgr2=0x7fb3b4066820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2020-11-20T11:38:42.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.851+0000 7fb3b2ffd700 1 -- 172.21.15.23:0/1252771995 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3b40a4e00 msgr2=0x7fb3b4066820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:38:42.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.851+0000 7fb3b2ffd700 1 --2- 172.21.15.23:0/1252771995 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3b40a4e00 0x7fb3b4066820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:38:42.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.851+0000 7fb3b2ffd700 1 --2- 172.21.15.23:0/1252771995 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3b40a4e00 0x7fb3b4066820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:38:42.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.851+0000 7fb3b37fe700 1 -- 172.21.15.23:0/1252771995 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3b40a4e00 msgr2=0x7fb3b4066820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:38:42.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.851+0000 7fb3b37fe700 1 --2- 172.21.15.23:0/1252771995 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3b40a4e00 0x7fb3b4066820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:42.866 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.851+0000 7fb3b37fe700 1 -- 172.21.15.23:0/1252771995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3b40a9700 msgr2=0x7fb3b4136ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:42.866 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.851+0000 7fb3b37fe700 1 --2- 172.21.15.23:0/1252771995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3b40a9700 0x7fb3b4136ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:42.866 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.851+0000 7fb3b37fe700 1 -- 172.21.15.23:0/1252771995 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b40a29a0 con 0x7fb3b4061380 2020-11-20T11:38:42.866 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.851+0000 7fb3b37fe700 1 --2- 172.21.15.23:0/1252771995 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3b4061380 0x7fb3b409f000 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fb3b4062300 tx=0x7fb3a4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:42.866 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.852+0000 7fb3937fe700 1 -- 172.21.15.23:0/1252771995 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a4011bc0 con 0x7fb3b4061380 2020-11-20T11:38:42.867 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.852+0000 7fb3937fe700 1 -- 172.21.15.23:0/1252771995 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a4011d20 con 0x7fb3b4061380 2020-11-20T11:38:42.867 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.852+0000 7fb3b8b1b700 1 -- 172.21.15.23:0/1252771995 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3b409fd10 con 0x7fb3b4061380 2020-11-20T11:38:42.867 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.852+0000 7fb3937fe700 1 -- 172.21.15.23:0/1252771995 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a401e690 con 0x7fb3b4061380 2020-11-20T11:38:42.867 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.852+0000 7fb3b8b1b700 1 -- 172.21.15.23:0/1252771995 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3b4068440 con 0x7fb3b4061380 2020-11-20T11:38:42.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.853+0000 7fb3937fe700 1 -- 172.21.15.23:0/1252771995 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb3a4025030 con 0x7fb3b4061380 2020-11-20T11:38:42.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.853+0000 7fb3b8b1b700 1 --2- 172.21.15.23:0/1252771995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3b4061980 0x7fb3b41373d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:42.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.853+0000 7fb3b8b1b700 1 -- 172.21.15.23:0/1252771995 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b4067940 con 0x7fb3b4061980 2020-11-20T11:38:42.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.853+0000 7fb3937fe700 1 --2- 172.21.15.23:0/1252771995 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb38c056d60 0x7fb38c059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:42.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.853+0000 7fb3b2ffd700 1 --2- 172.21.15.23:0/1252771995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3b4061980 0x7fb3b41373d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:42.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.853+0000 7fb3937fe700 1 -- 172.21.15.23:0/1252771995 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fb3a4071720 con 0x7fb3b4061380 2020-11-20T11:38:42.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.854+0000 7fb3b3fff700 1 --2- 172.21.15.23:0/1252771995 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb38c056d60 0x7fb38c059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:42.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.854+0000 7fb3b2ffd700 1 -- 172.21.15.23:0/1252771995 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb3b4069c70 con 0x7fb3b4061980 2020-11-20T11:38:42.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.854+0000 7fb3b2ffd700 1 --2- 172.21.15.23:0/1252771995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3b4061980 0x7fb3b41373d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fb3b409fb40 tx=0x7fb3a8009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:42.870 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.854+0000 7fb3937fe700 1 -- 172.21.15.23:0/1252771995 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a8010520 con 0x7fb3b4061980 2020-11-20T11:38:42.870 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.854+0000 7fb3b3fff700 1 --2- 172.21.15.23:0/1252771995 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb38c056d60 0x7fb38c059220 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fb3ac0013f0 tx=0x7fb3ac009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:42.870 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.854+0000 7fb3937fe700 1 -- 172.21.15.23:0/1252771995 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fb3a8010680 con 0x7fb3b4061980 2020-11-20T11:38:42.870 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.854+0000 7fb3937fe700 1 -- 172.21.15.23:0/1252771995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3b4061980 msgr2=0x7fb3b41373d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:42.870 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.854+0000 7fb3937fe700 1 --2- 172.21.15.23:0/1252771995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3b4061980 0x7fb3b41373d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fb3b409fb40 tx=0x7fb3a8009040).stop 2020-11-20T11:38:42.871 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.854+0000 7fb3937fe700 1 -- 172.21.15.23:0/1252771995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3b4061980 msgr2=0x7fb3b41373d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:38:42.871 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.854+0000 7fb3937fe700 1 --2- 172.21.15.23:0/1252771995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3b4061980 0x7fb3b41373d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:42.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.861+0000 7fb3b8b1b700 1 --2- 172.21.15.23:0/1252771995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3b41378c0 0x7fb3b40655d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:42.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.861+0000 7fb3b8b1b700 1 -- 172.21.15.23:0/1252771995 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b4067f20 con 0x7fb3b41378c0 2020-11-20T11:38:42.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.862+0000 7fb3b2ffd700 1 --2- 172.21.15.23:0/1252771995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3b41378c0 0x7fb3b40655d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:42.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.862+0000 7fb3b2ffd700 1 -- 172.21.15.23:0/1252771995 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb3b4057dc0 con 0x7fb3b41378c0 2020-11-20T11:38:42.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.862+0000 7fb3b2ffd700 1 --2- 172.21.15.23:0/1252771995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3b41378c0 0x7fb3b40655d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fb3a8015000 tx=0x7fb3a8018000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:42.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.862+0000 7fb3937fe700 1 -- 172.21.15.23:0/1252771995 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a8022420 con 0x7fb3b41378c0 2020-11-20T11:38:42.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.862+0000 7fb3937fe700 1 -- 172.21.15.23:0/1252771995 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4783 (secure 0 0 0) 0x7fb3a8010310 con 0x7fb3b41378c0 2020-11-20T11:38:42.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.862+0000 7fb3937fe700 1 -- 172.21.15.23:0/1252771995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3b41378c0 msgr2=0x7fb3b40655d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:42.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.862+0000 7fb3937fe700 1 --2- 172.21.15.23:0/1252771995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3b41378c0 0x7fb3b40655d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fb3a8015000 tx=0x7fb3a8018000).stop 2020-11-20T11:38:42.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.862+0000 7fb3937fe700 1 -- 172.21.15.23:0/1252771995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3b41378c0 msgr2=0x7fb3b40655d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:38:42.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.862+0000 7fb3937fe700 1 --2- 172.21.15.23:0/1252771995 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3b41378c0 0x7fb3b40655d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:42.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.863+0000 7fb3b8b1b700 1 -- 172.21.15.23:0/1252771995 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb38c056d60 msgr2=0x7fb38c059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:42.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.863+0000 7fb3b8b1b700 1 --2- 172.21.15.23:0/1252771995 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb38c056d60 0x7fb38c059220 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fb3ac0013f0 tx=0x7fb3ac009040).stop 2020-11-20T11:38:42.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.863+0000 7fb3b8b1b700 1 -- 172.21.15.23:0/1252771995 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3b4061380 msgr2=0x7fb3b409f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:42.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.863+0000 7fb3b8b1b700 1 --2- 172.21.15.23:0/1252771995 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3b4061380 0x7fb3b409f000 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fb3b4062300 tx=0x7fb3a4015000).stop 2020-11-20T11:38:42.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.863+0000 7fb3b3fff700 1 -- 172.21.15.23:0/1252771995 reap_dead start 2020-11-20T11:38:42.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.863+0000 7fb3b8b1b700 1 -- 172.21.15.23:0/1252771995 shutdown_connections 2020-11-20T11:38:42.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.863+0000 7fb3b8b1b700 1 --2- 172.21.15.23:0/1252771995 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3b4061380 0x7fb3b409f000 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:42.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.863+0000 7fb3b8b1b700 1 -- 172.21.15.23:0/1252771995 >> 172.21.15.23:0/1252771995 conn(0x7fb3b40ae0a0 msgr2=0x7fb3b40a5240 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:42.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.863+0000 7fb3b8b1b700 1 -- 172.21.15.23:0/1252771995 shutdown_connections 2020-11-20T11:38:42.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:42.863+0000 7fb3b8b1b700 1 -- 172.21.15.23:0/1252771995 wait complete. 2020-11-20T11:38:42.877 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:38:42.878 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:38:42.878 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:38:42.878 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:38:42.878 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 66, 2020-11-20T11:38:42.878 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:38:42.879 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:38:42.879 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:38:42.879 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:38:42.879 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:42.879 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 3, 2020-11-20T11:38:42.880 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:38:42.880 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:38:42.880 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:38:42.880 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:38:42.880 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:38:42.881 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:38:42.881 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:38:42.881 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:38:42.881 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:38:42.881 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:38:42.882 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:38:42.882 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:42.882 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:38:42.882 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:38:42.882 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:38:42.883 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:38:42.883 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:38:42.883 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:38:42.883 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:38:42.884 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:38:42.884 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:38:42.884 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:38:42.884 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:42.884 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:42.885 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:38:42.885 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:38:42.885 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:38:42.885 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:38:42.885 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:38:42.886 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:38:42.886 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:38:42.886 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:38:42.886 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:38:42.887 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:38:42.887 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:38:42.887 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:38:42.887 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:38:42.887 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:38:42.888 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:38:42.888 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:38:42.888 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:38:42.888 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:38:42.888 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:38:42.889 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:38:42.889 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:38:42.889 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:38:42.889 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:38:42.890 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:42.890 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:38:42.890 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:42.890 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:38:42.890 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:42.890 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:38:42.891 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:38:42.891 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:38:42.891 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:38:42.891 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:42.892 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:38:42.892 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:38:42.892 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:42.892 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:42.892 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:42.893 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:38:42.893 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:38:42.893 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:42.893 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:42.893 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:42.894 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:42.894 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:38:42.894 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:38:42.894 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:38:42.894 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:38:42.895 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:38:42.895 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:42.895 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:42.895 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:38:42.895 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:38:42.896 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:38:42.896 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:38:42.896 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:42.896 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:38:42.897 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:38:42.897 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:42.897 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:42.897 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:42.897 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:38:42.898 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:38:42.898 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:42.898 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:42.898 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:42.898 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:42.899 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:38:42.899 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:38:42.899 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:38:42.899 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:38:42.899 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:38:42.900 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:42.900 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:42.900 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:38:42.900 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:38:42.900 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:38:42.900 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:38:42.901 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:42.901 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:38:42.901 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:38:42.901 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:42.902 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:42.902 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:42.902 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:38:42.902 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:38:42.902 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:42.903 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:42.903 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:42.903 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:42.903 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:38:42.903 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:38:42.904 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:38:42.904 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:38:42.904 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:38:42.904 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:42.904 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:42.905 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:42.905 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:38:42.905 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:38:42.905 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:42.906 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:38:42.906 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:38:42.906 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:38:42.906 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:42.906 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:42.907 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:38:42.907 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:42.907 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:38:42.907 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:38:42.908 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:38:42.908 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:42.908 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:42.908 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:38:42.908 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:42.909 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:38:42.909 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:38:42.909 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:38:42.909 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:42.909 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:42.910 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:38:42.910 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:42.910 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:38:42.910 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:38:42.911 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:38:42.911 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:42.911 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:42.911 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:38:42.911 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:42.912 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:38:42.912 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:38:42.912 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:38:42.912 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:42.912 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:42.913 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:42.913 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:38:42.913 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:38:42.914 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:38:43.040 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.036+0000 7f087e0e2700 1 Processor -- start 2020-11-20T11:38:43.041 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.037+0000 7f087e0e2700 1 -- start start 2020-11-20T11:38:43.044 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.037+0000 7f087e0e2700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780665f0 0x7f087813adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:43.045 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.037+0000 7f087e0e2700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f087813bd10 con 0x7f0878066210 2020-11-20T11:38:43.045 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.037+0000 7f087e0e2700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08780ab730 con 0x7f08780ab2a0 2020-11-20T11:38:43.045 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.037+0000 7f087e0e2700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08780a9700 con 0x7f08780665f0 2020-11-20T11:38:43.045 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.037+0000 7f087d0e0700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f0878066210 0x7f08780a8ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57874/0 (socket says 172.21.15.23:57874) 2020-11-20T11:38:43.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.037+0000 7f087d0e0700 1 -- 172.21.15.23:0/190035836 learned_addr learned my addr 172.21.15.23:0/190035836 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:38:43.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.038+0000 7f087c8df700 1 --2- 172.21.15.23:0/190035836 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780665f0 0x7f087813adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:43.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.038+0000 7f0867fff700 1 -- 172.21.15.23:0/190035836 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2015703241 0 0) 0x7f087813bd10 con 0x7f0878066210 2020-11-20T11:38:43.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.038+0000 7f087c8df700 1 -- 172.21.15.23:0/190035836 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780665f0 msgr2=0x7f087813adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:38:43.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.038+0000 7f087c8df700 1 -- 172.21.15.23:0/190035836 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780665f0 msgr2=0x7f087813adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:38:43.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.038+0000 7f087c8df700 1 --2- 172.21.15.23:0/190035836 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780665f0 0x7f087813adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:38:43.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.038+0000 7f0867fff700 1 -- 172.21.15.23:0/190035836 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0860003280 con 0x7f0878066210 2020-11-20T11:38:43.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.038+0000 7f087c8df700 1 --2- 172.21.15.23:0/190035836 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780665f0 0x7f087813adb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:38:43.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.038+0000 7f0867fff700 1 -- 172.21.15.23:0/190035836 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3226870965 0 0) 0x7f08780ab730 con 0x7f08780ab2a0 2020-11-20T11:38:43.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.038+0000 7f0867fff700 1 -- 172.21.15.23:0/190035836 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f087813bd10 con 0x7f08780ab2a0 2020-11-20T11:38:43.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.038+0000 7f0867fff700 1 -- 172.21.15.23:0/190035836 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 286568660 0 0) 0x7f0860003280 con 0x7f0878066210 2020-11-20T11:38:43.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.038+0000 7f0867fff700 1 -- 172.21.15.23:0/190035836 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780665f0 msgr2=0x7f087813adb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:38:43.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.038+0000 7f0867fff700 1 --2- 172.21.15.23:0/190035836 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780665f0 0x7f087813adb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:43.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.038+0000 7f0867fff700 1 -- 172.21.15.23:0/190035836 >> v1:172.21.15.155:6789/0 conn(0x7f08780ab2a0 legacy=0x7f0878137f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:43.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.038+0000 7f0867fff700 1 -- 172.21.15.23:0/190035836 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08780aa930 con 0x7f0878066210 2020-11-20T11:38:43.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.038+0000 7f0867fff700 1 -- 172.21.15.23:0/190035836 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f0874001520 con 0x7f0878066210 2020-11-20T11:38:43.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.038+0000 7f0867fff700 1 -- 172.21.15.23:0/190035836 >> v1:172.21.15.23:6789/0 conn(0x7f0878066210 legacy=0x7f08780a8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:43.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.038+0000 7f0867fff700 1 --2- 172.21.15.23:0/190035836 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780665f0 0x7f0860003930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:43.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.038+0000 7f0867fff700 1 --2- 172.21.15.23:0/190035836 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0860003e20 0x7f0860004520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:43.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.038+0000 7f0867fff700 1 --2- 172.21.15.23:0/190035836 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0860004a10 0x7f0860004e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:43.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.038+0000 7f0867fff700 1 -- 172.21.15.23:0/190035836 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08780a9700 con 0x7f0860003e20 2020-11-20T11:38:43.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.038+0000 7f0867fff700 1 -- 172.21.15.23:0/190035836 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08600053b0 con 0x7f0860004a10 2020-11-20T11:38:43.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.038+0000 7f0867fff700 1 -- 172.21.15.23:0/190035836 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f086000b7c0 con 0x7f08780665f0 2020-11-20T11:38:43.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.038+0000 7f087d0e0700 1 --2- 172.21.15.23:0/190035836 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780665f0 0x7f0860003930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:43.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.038+0000 7f087c8df700 1 --2- 172.21.15.23:0/190035836 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0860003e20 0x7f0860004520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:43.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.038+0000 7f087d8e1700 1 --2- 172.21.15.23:0/190035836 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0860004a10 0x7f0860004e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:43.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.039+0000 7f087d0e0700 1 -- 172.21.15.23:0/190035836 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0860004a10 msgr2=0x7f0860004e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:38:43.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.039+0000 7f087d0e0700 1 --2- 172.21.15.23:0/190035836 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0860004a10 0x7f0860004e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:43.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.039+0000 7f087d0e0700 1 -- 172.21.15.23:0/190035836 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0860003e20 msgr2=0x7f0860004520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:38:43.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.039+0000 7f087d0e0700 1 --2- 172.21.15.23:0/190035836 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0860003e20 0x7f0860004520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:43.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.039+0000 7f087d0e0700 1 -- 172.21.15.23:0/190035836 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0860003280 con 0x7f08780665f0 2020-11-20T11:38:43.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.039+0000 7f087d0e0700 1 --2- 172.21.15.23:0/190035836 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780665f0 0x7f0860003930 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f087400c450 tx=0x7f0878065fe0).ready entity=mon.2 client_cookie=146f5697617ee4e8 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:43.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.039+0000 7f0867fff700 1 -- 172.21.15.23:0/190035836 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087400cea0 con 0x7f08780665f0 2020-11-20T11:38:43.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.039+0000 7f0867fff700 1 -- 172.21.15.23:0/190035836 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08740070d0 con 0x7f08780665f0 2020-11-20T11:38:43.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.039+0000 7f0867fff700 1 -- 172.21.15.23:0/190035836 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0874007490 con 0x7f08780665f0 2020-11-20T11:38:43.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.040+0000 7f087e0e2700 1 -- 172.21.15.23:0/190035836 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780665f0 msgr2=0x7f0860003930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:43.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.040+0000 7f087e0e2700 1 --2- 172.21.15.23:0/190035836 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780665f0 0x7f0860003930 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f087400c450 tx=0x7f0878065fe0).stop 2020-11-20T11:38:43.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.040+0000 7f087d8e1700 1 -- 172.21.15.23:0/190035836 reap_dead start 2020-11-20T11:38:43.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.040+0000 7f087e0e2700 1 -- 172.21.15.23:0/190035836 shutdown_connections 2020-11-20T11:38:43.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.040+0000 7f087e0e2700 1 -- 172.21.15.23:0/190035836 >> 172.21.15.23:0/190035836 conn(0x7f08780612c0 msgr2=0x7f08780616d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:43.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.040+0000 7f087e0e2700 1 -- 172.21.15.23:0/190035836 shutdown_connections 2020-11-20T11:38:43.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.040+0000 7f087e0e2700 1 -- 172.21.15.23:0/190035836 wait complete. 2020-11-20T11:38:43.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.041+0000 7f087e0e2700 1 Processor -- start 2020-11-20T11:38:43.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.041+0000 7f087e0e2700 1 -- start start 2020-11-20T11:38:43.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.041+0000 7f087e0e2700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780665f0 0x7f087813e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:43.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.041+0000 7f087e0e2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08780a5ac0 0x7f087813aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:43.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.041+0000 7f087e0e2700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08780a72c0 0x7f087813c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:43.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.042+0000 7f087e0e2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08780a7bd0 con 0x7f08780a5ac0 2020-11-20T11:38:43.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.042+0000 7f087e0e2700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08780a5f20 con 0x7f08780a72c0 2020-11-20T11:38:43.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.042+0000 7f087e0e2700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08780a5380 con 0x7f08780665f0 2020-11-20T11:38:43.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.042+0000 7f087c8df700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08780a5ac0 0x7f087813aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:43.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.042+0000 7f087c8df700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08780a5ac0 0x7f087813aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43638/0 (socket says 172.21.15.23:43638) 2020-11-20T11:38:43.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.042+0000 7f087c8df700 1 -- 172.21.15.23:0/2206238127 learned_addr learned my addr 172.21.15.23:0/2206238127 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:38:43.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.042+0000 7f087d0e0700 1 --2- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780665f0 0x7f087813e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:43.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.042+0000 7f087d8e1700 1 --2- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08780a72c0 0x7f087813c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:43.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.042+0000 7f087d0e0700 1 -- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780665f0 msgr2=0x7f087813e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2020-11-20T11:38:43.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.042+0000 7f087d0e0700 1 -- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780665f0 msgr2=0x7f087813e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:38:43.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.042+0000 7f087d0e0700 1 --2- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780665f0 0x7f087813e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:38:43.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.042+0000 7f087d0e0700 1 --2- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780665f0 0x7f087813e4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:38:43.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.042+0000 7f087d8e1700 1 -- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780665f0 msgr2=0x7f087813e4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:38:43.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.042+0000 7f087d8e1700 1 --2- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780665f0 0x7f087813e4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:43.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.042+0000 7f087d8e1700 1 -- 172.21.15.23:0/2206238127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08780a5ac0 msgr2=0x7f087813aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:43.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.042+0000 7f087d8e1700 1 --2- 172.21.15.23:0/2206238127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08780a5ac0 0x7f087813aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:43.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.042+0000 7f087d8e1700 1 -- 172.21.15.23:0/2206238127 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08780a5620 con 0x7f08780a72c0 2020-11-20T11:38:43.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.042+0000 7f087c8df700 1 --2- 172.21.15.23:0/2206238127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08780a5ac0 0x7f087813aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:38:43.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.042+0000 7f087d8e1700 1 --2- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08780a72c0 0x7f087813c430 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f0860003900 tx=0x7f0870007f50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:43.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.042+0000 7f08657fa700 1 -- 172.21.15.23:0/2206238127 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0870006ac0 con 0x7f08780a72c0 2020-11-20T11:38:43.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.042+0000 7f08657fa700 1 -- 172.21.15.23:0/2206238127 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0870006c60 con 0x7f08780a72c0 2020-11-20T11:38:43.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.042+0000 7f087e0e2700 1 -- 172.21.15.23:0/2206238127 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f087813d930 con 0x7f08780a72c0 2020-11-20T11:38:43.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.042+0000 7f08657fa700 1 -- 172.21.15.23:0/2206238127 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0870002020 con 0x7f08780a72c0 2020-11-20T11:38:43.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.042+0000 7f087e0e2700 1 -- 172.21.15.23:0/2206238127 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08780a7960 con 0x7f08780a72c0 2020-11-20T11:38:43.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.043+0000 7f087e0e2700 1 --2- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780622d0 0x7f087813c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:43.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.043+0000 7f087e0e2700 1 -- 172.21.15.23:0/2206238127 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08780a5200 con 0x7f08780622d0 2020-11-20T11:38:43.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.043+0000 7f087d0e0700 1 --2- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780622d0 0x7f087813c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:43.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.043+0000 7f087d0e0700 1 -- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780622d0 msgr2=0x7f087813c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 20 2020-11-20T11:38:43.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.043+0000 7f087d0e0700 1 -- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780622d0 msgr2=0x7f087813c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:38:43.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.043+0000 7f087d0e0700 1 --2- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780622d0 0x7f087813c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:38:43.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.043+0000 7f087d0e0700 1 --2- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780622d0 0x7f087813c920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:38:43.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.044+0000 7f08657fa700 1 -- 172.21.15.23:0/2206238127 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f08700023d0 con 0x7f08780a72c0 2020-11-20T11:38:43.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.044+0000 7f08657fa700 1 --2- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0850056d60 0x7f0850059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:43.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.044+0000 7f08657fa700 1 -- 172.21.15.23:0/2206238127 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f087006f9f0 con 0x7f08780a72c0 2020-11-20T11:38:43.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.044+0000 7f087c8df700 1 --2- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0850056d60 0x7f0850059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:43.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.045+0000 7f087c8df700 1 --2- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0850056d60 0x7f0850059220 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f086800bfd0 tx=0x7f0868008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:43.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.244+0000 7f087d0e0700 1 --2- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780622d0 0x7f087813c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:43.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.244+0000 7f087d0e0700 1 -- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780622d0 msgr2=0x7f087813c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2020-11-20T11:38:43.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.244+0000 7f087d0e0700 1 -- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780622d0 msgr2=0x7f087813c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:38:43.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.244+0000 7f087d0e0700 1 --2- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780622d0 0x7f087813c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:38:43.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.244+0000 7f087d0e0700 1 --2- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780622d0 0x7f087813c920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:38:43.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.645+0000 7f087d0e0700 1 --2- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780622d0 0x7f087813c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:43.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.645+0000 7f087d0e0700 1 -- 172.21.15.23:0/2206238127 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0878061750 con 0x7f08780622d0 2020-11-20T11:38:43.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.646+0000 7f087d0e0700 1 --2- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780622d0 0x7f087813c920 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f087400c450 tx=0x7f0874015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:43.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.646+0000 7f08657fa700 1 -- 172.21.15.23:0/2206238127 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0874001690 con 0x7f08780622d0 2020-11-20T11:38:43.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.646+0000 7f08657fa700 1 -- 172.21.15.23:0/2206238127 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f08740017f0 con 0x7f08780622d0 2020-11-20T11:38:43.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.646+0000 7f08657fa700 1 -- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780622d0 msgr2=0x7f087813c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:43.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.646+0000 7f08657fa700 1 --2- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780622d0 0x7f087813c920 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f087400c450 tx=0x7f0874015000).stop 2020-11-20T11:38:43.652 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.646+0000 7f08657fa700 1 -- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780622d0 msgr2=0x7f087813c920 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:38:43.652 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.646+0000 7f08657fa700 1 --2- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08780622d0 0x7f087813c920 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:43.661 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.655+0000 7f087e0e2700 1 --2- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f087813ce10 0x7f0878135ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:43.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.656+0000 7f087e0e2700 1 -- 172.21.15.23:0/2206238127 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0878062760 con 0x7f087813ce10 2020-11-20T11:38:43.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.656+0000 7f087d0e0700 1 --2- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f087813ce10 0x7f0878135ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:43.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.656+0000 7f087d0e0700 1 -- 172.21.15.23:0/2206238127 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f08780a57b0 con 0x7f087813ce10 2020-11-20T11:38:43.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.656+0000 7f087d0e0700 1 --2- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f087813ce10 0x7f0878135ce0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f087400c370 tx=0x7f0874022000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:43.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.656+0000 7f08657fa700 1 -- 172.21.15.23:0/2206238127 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087402b5a0 con 0x7f087813ce10 2020-11-20T11:38:43.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.657+0000 7f08657fa700 1 -- 172.21.15.23:0/2206238127 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4273 (secure 0 0 0) 0x7f0874001480 con 0x7f087813ce10 2020-11-20T11:38:43.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.657+0000 7f08657fa700 1 -- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f087813ce10 msgr2=0x7f0878135ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:43.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.657+0000 7f08657fa700 1 --2- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f087813ce10 0x7f0878135ce0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f087400c370 tx=0x7f0874022000).stop 2020-11-20T11:38:43.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.657+0000 7f08657fa700 1 -- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f087813ce10 msgr2=0x7f0878135ce0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:38:43.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.657+0000 7f08657fa700 1 --2- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f087813ce10 0x7f0878135ce0 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0x7f087400c370 tx=0x7f0874022000).stop 2020-11-20T11:38:43.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.657+0000 7f087e0e2700 1 -- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0850056d60 msgr2=0x7f0850059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:43.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.657+0000 7f087e0e2700 1 --2- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0850056d60 0x7f0850059220 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f086800bfd0 tx=0x7f0868008040).stop 2020-11-20T11:38:43.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.657+0000 7f087e0e2700 1 -- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08780a72c0 msgr2=0x7f087813c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:43.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.657+0000 7f087e0e2700 1 --2- 172.21.15.23:0/2206238127 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08780a72c0 0x7f087813c430 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f0860003900 tx=0x7f0870007f50).stop 2020-11-20T11:38:43.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.657+0000 7f087d8e1700 1 -- 172.21.15.23:0/2206238127 reap_dead start 2020-11-20T11:38:43.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.657+0000 7f087d8e1700 1 -- 172.21.15.23:0/2206238127 reap_dead start 2020-11-20T11:38:43.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.657+0000 7f087e0e2700 1 -- 172.21.15.23:0/2206238127 shutdown_connections 2020-11-20T11:38:43.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.657+0000 7f087e0e2700 1 -- 172.21.15.23:0/2206238127 >> 172.21.15.23:0/2206238127 conn(0x7f08780612c0 msgr2=0x7f0878066a40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:43.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.657+0000 7f087e0e2700 1 -- 172.21.15.23:0/2206238127 shutdown_connections 2020-11-20T11:38:43.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.657+0000 7f087e0e2700 1 -- 172.21.15.23:0/2206238127 wait complete. 2020-11-20T11:38:43.666 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:38:43.667 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:38:43.667 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:38:43.667 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:38:43.667 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 66, 2020-11-20T11:38:43.668 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:38:43.668 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:38:43.668 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:38:43.668 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:38:43.668 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:43.669 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 3, 2020-11-20T11:38:43.669 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:38:43.669 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:38:43.669 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:38:43.669 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:38:43.670 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:38:43.670 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:38:43.670 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:38:43.670 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:38:43.670 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:38:43.671 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:38:43.671 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:38:43.671 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:43.671 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:38:43.671 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:38:43.672 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:38:43.672 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:38:43.672 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:38:43.672 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:38:43.672 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:38:43.673 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:38:43.673 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:38:43.673 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:38:43.673 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:43.674 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:43.674 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:38:43.674 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:38:43.674 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:38:43.674 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:38:43.675 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:38:43.675 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:38:43.675 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:38:43.675 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:38:43.675 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:38:43.676 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:38:43.676 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:38:43.676 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:38:43.676 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:38:43.676 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:38:43.677 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:38:43.677 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:38:43.677 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:38:43.677 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:38:43.677 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:38:43.678 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:38:43.678 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:38:43.678 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:38:43.678 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:38:43.678 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:43.679 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:38:43.679 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:43.679 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:38:43.679 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:43.679 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:38:43.680 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:38:43.680 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:38:43.680 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:38:43.680 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:43.680 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:38:43.680 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:38:43.681 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:43.681 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:43.681 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:43.681 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:38:43.682 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:38:43.682 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:43.682 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:43.682 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:43.682 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:43.683 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:38:43.683 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:38:43.683 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:38:43.683 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:38:43.683 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:38:43.684 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:43.684 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:43.684 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:38:43.684 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:38:43.684 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:38:43.685 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:38:43.685 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:43.685 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:38:43.685 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:38:43.685 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:43.686 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:43.686 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:43.686 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:38:43.686 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:38:43.686 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:43.687 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:43.687 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:43.687 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:43.687 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:38:43.687 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:38:43.688 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:38:43.688 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:38:43.688 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:38:43.688 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:43.688 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:43.689 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:38:43.689 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:38:43.689 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:38:43.689 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:38:43.689 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:43.690 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:38:43.690 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:38:43.690 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:43.690 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:43.690 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:43.691 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:38:43.691 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:38:43.691 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:43.691 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:43.691 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:43.692 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:43.692 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:38:43.692 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:38:43.692 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:38:43.692 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:38:43.693 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:38:43.693 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:43.693 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:43.693 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:43.693 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:38:43.694 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:38:43.694 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:43.694 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:38:43.694 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:38:43.694 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:38:43.695 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:43.695 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:43.695 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:38:43.695 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:43.695 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:38:43.696 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:38:43.696 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:38:43.696 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:43.696 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:43.696 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:43.697 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:38:43.697 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:38:43.698 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2020-11-20T11:38:43.698 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2020-11-20T11:38:43.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.827+0000 7fb067231700 1 Processor -- start 2020-11-20T11:38:43.832 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.827+0000 7fb067231700 1 -- start start 2020-11-20T11:38:43.840 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.828+0000 7fb067231700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb060061380 0x7fb0600a8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:43.840 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.828+0000 7fb067231700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb0600a8d70 0x7fb06013e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:43.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.828+0000 7fb067231700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb06013edc0 0x7fb06013f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:43.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.828+0000 7fb067231700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb06013f730 con 0x7fb06013edc0 2020-11-20T11:38:43.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.828+0000 7fb067231700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0600a5000 con 0x7fb0600a8d70 2020-11-20T11:38:43.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.828+0000 7fb067231700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0600a5d30 con 0x7fb060061380 2020-11-20T11:38:43.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.828+0000 7fb066a30700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb06013edc0 0x7fb06013f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:43.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.828+0000 7fb066a30700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb06013edc0 0x7fb06013f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43656/0 (socket says 172.21.15.23:43656) 2020-11-20T11:38:43.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.828+0000 7fb066a30700 1 -- 172.21.15.23:0/1193422910 learned_addr learned my addr 172.21.15.23:0/1193422910 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:38:43.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.828+0000 7fb06622f700 1 --2- 172.21.15.23:0/1193422910 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb060061380 0x7fb0600a8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:43.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.828+0000 7fb065a2e700 1 --2- 172.21.15.23:0/1193422910 >> v2:172.21.15.155:3300/0 conn(0x7fb0600a8d70 0x7fb06013e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:43.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.828+0000 7fb066a30700 1 -- 172.21.15.23:0/1193422910 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb060061380 msgr2=0x7fb0600a8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:38:43.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.828+0000 7fb066a30700 1 --2- 172.21.15.23:0/1193422910 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb060061380 0x7fb0600a8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:43.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.828+0000 7fb066a30700 1 -- 172.21.15.23:0/1193422910 >> v2:172.21.15.155:3300/0 conn(0x7fb0600a8d70 msgr2=0x7fb06013e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:38:43.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.828+0000 7fb066a30700 1 --2- 172.21.15.23:0/1193422910 >> v2:172.21.15.155:3300/0 conn(0x7fb0600a8d70 0x7fb06013e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:43.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.828+0000 7fb066a30700 1 -- 172.21.15.23:0/1193422910 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0600a5fd0 con 0x7fb06013edc0 2020-11-20T11:38:43.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.828+0000 7fb06622f700 1 --2- 172.21.15.23:0/1193422910 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb060061380 0x7fb0600a8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:38:43.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.828+0000 7fb066a30700 1 --2- 172.21.15.23:0/1193422910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb06013edc0 0x7fb06013f240 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fb060062e40 tx=0x7fb0580092d0).ready entity=mon.0 client_cookie=a5b8343529f86a4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:43.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.829+0000 7fb06522d700 1 -- 172.21.15.23:0/1193422910 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05800bbf0 con 0x7fb06013edc0 2020-11-20T11:38:43.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.829+0000 7fb06522d700 1 -- 172.21.15.23:0/1193422910 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb05800bd50 con 0x7fb06013edc0 2020-11-20T11:38:43.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.829+0000 7fb06522d700 1 -- 172.21.15.23:0/1193422910 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb058011610 con 0x7fb06013edc0 2020-11-20T11:38:43.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.829+0000 7fb067231700 1 -- 172.21.15.23:0/1193422910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb06013edc0 msgr2=0x7fb06013f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:43.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.829+0000 7fb067231700 1 --2- 172.21.15.23:0/1193422910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb06013edc0 0x7fb06013f240 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fb060062e40 tx=0x7fb0580092d0).stop 2020-11-20T11:38:43.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.829+0000 7fb067231700 1 -- 172.21.15.23:0/1193422910 shutdown_connections 2020-11-20T11:38:43.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.829+0000 7fb067231700 1 --2- 172.21.15.23:0/1193422910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb06013edc0 0x7fb06013f240 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:43.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.829+0000 7fb067231700 1 --2- 172.21.15.23:0/1193422910 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb060061380 0x7fb0600a8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:43.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.829+0000 7fb067231700 1 --2- 172.21.15.23:0/1193422910 >> v2:172.21.15.155:3300/0 conn(0x7fb0600a8d70 0x7fb06013e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:43.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.829+0000 7fb067231700 1 -- 172.21.15.23:0/1193422910 >> 172.21.15.23:0/1193422910 conn(0x7fb0600ae0a0 msgr2=0x7fb0600aa0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:43.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.829+0000 7fb067231700 1 -- 172.21.15.23:0/1193422910 shutdown_connections 2020-11-20T11:38:43.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.829+0000 7fb067231700 1 -- 172.21.15.23:0/1193422910 wait complete. 2020-11-20T11:38:43.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.830+0000 7fb067231700 1 Processor -- start 2020-11-20T11:38:43.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.830+0000 7fb067231700 1 -- start start 2020-11-20T11:38:43.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.831+0000 7fb067231700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb060061380 0x7fb060065d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:43.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.831+0000 7fb067231700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0600a8d70 0x7fb0600a7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:43.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.831+0000 7fb067231700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb06013edc0 0x7fb06009f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:43.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.831+0000 7fb067231700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0600683e0 con 0x7fb06013edc0 2020-11-20T11:38:43.850 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.831+0000 7fb067231700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb060068680 con 0x7fb060061380 2020-11-20T11:38:43.850 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.831+0000 7fb067231700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb060065880 con 0x7fb0600a8d70 2020-11-20T11:38:43.850 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.831+0000 7fb066a30700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb06013edc0 0x7fb06009f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:43.850 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.831+0000 7fb066a30700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb06013edc0 0x7fb06009f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43662/0 (socket says 172.21.15.23:43662) 2020-11-20T11:38:43.850 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.831+0000 7fb066a30700 1 -- 172.21.15.23:0/2440071609 learned_addr learned my addr 172.21.15.23:0/2440071609 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:38:43.851 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.831+0000 7fb06622f700 1 --2- 172.21.15.23:0/2440071609 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb060061380 0x7fb060065d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:43.851 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.831+0000 7fb065a2e700 1 --2- 172.21.15.23:0/2440071609 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0600a8d70 0x7fb0600a7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:43.851 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.831+0000 7fb066a30700 1 -- 172.21.15.23:0/2440071609 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0600a8d70 msgr2=0x7fb0600a7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:43.851 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.831+0000 7fb066a30700 1 --2- 172.21.15.23:0/2440071609 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0600a8d70 0x7fb0600a7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:43.852 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.831+0000 7fb066a30700 1 -- 172.21.15.23:0/2440071609 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb060061380 msgr2=0x7fb060065d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:43.852 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.831+0000 7fb066a30700 1 --2- 172.21.15.23:0/2440071609 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb060061380 0x7fb060065d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:43.852 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.831+0000 7fb066a30700 1 -- 172.21.15.23:0/2440071609 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb06009ea10 con 0x7fb06013edc0 2020-11-20T11:38:43.852 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.831+0000 7fb06622f700 1 --2- 172.21.15.23:0/2440071609 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb060061380 0x7fb060065d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:38:43.852 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.831+0000 7fb066a30700 1 --2- 172.21.15.23:0/2440071609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb06013edc0 0x7fb06009f560 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fb058009130 tx=0x7fb058016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:43.852 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.832+0000 7fb04e7fc700 1 -- 172.21.15.23:0/2440071609 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb058009ea0 con 0x7fb06013edc0 2020-11-20T11:38:43.853 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.832+0000 7fb04e7fc700 1 -- 172.21.15.23:0/2440071609 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb05800bec0 con 0x7fb06013edc0 2020-11-20T11:38:43.853 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.832+0000 7fb04e7fc700 1 -- 172.21.15.23:0/2440071609 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb058010e50 con 0x7fb06013edc0 2020-11-20T11:38:43.853 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.832+0000 7fb067231700 1 -- 172.21.15.23:0/2440071609 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb060067eb0 con 0x7fb06013edc0 2020-11-20T11:38:43.853 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.832+0000 7fb067231700 1 -- 172.21.15.23:0/2440071609 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0600679b0 con 0x7fb06013edc0 2020-11-20T11:38:43.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.833+0000 7fb067231700 1 -- 172.21.15.23:0/2440071609 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb060069c70 con 0x7fb06013edc0 2020-11-20T11:38:43.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.833+0000 7fb04e7fc700 1 -- 172.21.15.23:0/2440071609 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb058017070 con 0x7fb06013edc0 2020-11-20T11:38:43.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.834+0000 7fb04e7fc700 1 --2- 172.21.15.23:0/2440071609 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb038056d60 0x7fb038059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:43.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.834+0000 7fb04e7fc700 1 -- 172.21.15.23:0/2440071609 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fb0580720b0 con 0x7fb06013edc0 2020-11-20T11:38:43.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.834+0000 7fb06622f700 1 --2- 172.21.15.23:0/2440071609 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb038056d60 0x7fb038059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:43.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.834+0000 7fb06622f700 1 --2- 172.21.15.23:0/2440071609 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb038056d60 0x7fb038059220 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fb060066cd0 tx=0x7fb05c008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:43.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:43.839+0000 7fb04e7fc700 1 -- 172.21.15.23:0/2440071609 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb0580473a0 con 0x7fb06013edc0 2020-11-20T11:38:44.089 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:44.085+0000 7fb067231700 1 -- 172.21.15.23:0/2440071609 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fb060065a40 con 0x7fb06013edc0 2020-11-20T11:38:44.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:44.086+0000 7fb04e7fc700 1 -- 172.21.15.23:0/2440071609 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fb058015070 con 0x7fb06013edc0 2020-11-20T11:38:44.093 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:44.089+0000 7fb067231700 1 -- 172.21.15.23:0/2440071609 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb038056d60 msgr2=0x7fb038059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:44.093 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:44.089+0000 7fb067231700 1 --2- 172.21.15.23:0/2440071609 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb038056d60 0x7fb038059220 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fb060066cd0 tx=0x7fb05c008040).stop 2020-11-20T11:38:44.093 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:44.089+0000 7fb067231700 1 -- 172.21.15.23:0/2440071609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb06013edc0 msgr2=0x7fb06009f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:44.094 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:44.089+0000 7fb067231700 1 --2- 172.21.15.23:0/2440071609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb06013edc0 0x7fb06009f560 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fb058009130 tx=0x7fb058016000).stop 2020-11-20T11:38:44.094 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:44.089+0000 7fb067231700 1 -- 172.21.15.23:0/2440071609 shutdown_connections 2020-11-20T11:38:44.094 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:44.089+0000 7fb067231700 1 --2- 172.21.15.23:0/2440071609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb06013edc0 0x7fb06009f560 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:44.094 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:44.089+0000 7fb067231700 1 --2- 172.21.15.23:0/2440071609 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb060061380 0x7fb060065d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:44.094 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:44.089+0000 7fb067231700 1 --2- 172.21.15.23:0/2440071609 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb038056d60 0x7fb038059220 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:44.095 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:44.089+0000 7fb067231700 1 --2- 172.21.15.23:0/2440071609 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb0600a8d70 0x7fb0600a7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:44.095 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:44.089+0000 7fb067231700 1 -- 172.21.15.23:0/2440071609 >> 172.21.15.23:0/2440071609 conn(0x7fb0600ae0a0 msgr2=0x7fb0600ab6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:44.095 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:44.089+0000 7fb067231700 1 -- 172.21.15.23:0/2440071609 shutdown_connections 2020-11-20T11:38:44.095 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:44.089+0000 7fb067231700 1 -- 172.21.15.23:0/2440071609 wait complete. 2020-11-20T11:38:44.105 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2020-11-20T11:38:54.106 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:38:54.107 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:38:54.232 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.228+0000 7ff955610700 1 Processor -- start 2020-11-20T11:38:54.233 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.228+0000 7ff955610700 1 -- start start 2020-11-20T11:38:54.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.229+0000 7ff955610700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff950067e90 0x7ff9500a2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:54.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.229+0000 7ff955610700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff9501375a0 0x7ff9501379e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:54.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.229+0000 7ff955610700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9500aa8e0 con 0x7ff9500af090 2020-11-20T11:38:54.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.229+0000 7ff955610700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9500aa6f0 con 0x7ff9501375a0 2020-11-20T11:38:54.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.229+0000 7ff955610700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9500a2710 con 0x7ff950067e90 2020-11-20T11:38:54.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.229+0000 7ff94f7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff9501375a0 0x7ff9501379e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:54.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.229+0000 7ff94effd700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7ff9500af090 0x7ff950067780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57916/0 (socket says 172.21.15.23:57916) 2020-11-20T11:38:54.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.229+0000 7ff94effd700 1 -- 172.21.15.23:0/1556569128 learned_addr learned my addr 172.21.15.23:0/1556569128 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:38:54.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.230+0000 7ff94e7fc700 1 --2- 172.21.15.23:0/1556569128 >> v2:172.21.15.155:3300/0 conn(0x7ff950067e90 0x7ff9500a2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:54.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.230+0000 7ff94f7fe700 1 -- 172.21.15.23:0/1556569128 >> v2:172.21.15.155:3300/0 conn(0x7ff950067e90 msgr2=0x7ff9500a2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:38:54.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.230+0000 7ff94f7fe700 1 --2- 172.21.15.23:0/1556569128 >> v2:172.21.15.155:3300/0 conn(0x7ff950067e90 0x7ff9500a2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:54.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.230+0000 7ff94f7fe700 1 -- 172.21.15.23:0/1556569128 >> v1:172.21.15.23:6789/0 conn(0x7ff9500af090 legacy=0x7ff950067780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:54.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.230+0000 7ff94dffb700 1 -- 172.21.15.23:0/1556569128 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321766572 0 0) 0x7ff9500aa8e0 con 0x7ff9500af090 2020-11-20T11:38:54.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.230+0000 7ff94f7fe700 1 -- 172.21.15.23:0/1556569128 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9500aab80 con 0x7ff9501375a0 2020-11-20T11:38:54.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.230+0000 7ff94f7fe700 1 --2- 172.21.15.23:0/1556569128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9501375a0 0x7ff9501379e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7ff9500aca40 tx=0x7ff948009220).ready entity=mon.0 client_cookie=2227c7d2bac120ae server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:54.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.230+0000 7ff94dffb700 1 -- 172.21.15.23:0/1556569128 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94800bbb0 con 0x7ff9501375a0 2020-11-20T11:38:54.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.230+0000 7ff94dffb700 1 -- 172.21.15.23:0/1556569128 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff94800bd10 con 0x7ff9501375a0 2020-11-20T11:38:54.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.230+0000 7ff94dffb700 1 -- 172.21.15.23:0/1556569128 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff948011610 con 0x7ff9501375a0 2020-11-20T11:38:54.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.231+0000 7ff955610700 1 -- 172.21.15.23:0/1556569128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9501375a0 msgr2=0x7ff9501379e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:54.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.231+0000 7ff955610700 1 --2- 172.21.15.23:0/1556569128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9501375a0 0x7ff9501379e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7ff9500aca40 tx=0x7ff948009220).stop 2020-11-20T11:38:54.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.231+0000 7ff955610700 1 -- 172.21.15.23:0/1556569128 shutdown_connections 2020-11-20T11:38:54.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.231+0000 7ff955610700 1 --2- 172.21.15.23:0/1556569128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9501375a0 0x7ff9501379e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:54.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.231+0000 7ff955610700 1 --2- 172.21.15.23:0/1556569128 >> v2:172.21.15.155:3300/0 conn(0x7ff950067e90 0x7ff9500a2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:54.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.231+0000 7ff955610700 1 -- 172.21.15.23:0/1556569128 >> 172.21.15.23:0/1556569128 conn(0x7ff950063ac0 msgr2=0x7ff950063ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:54.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.231+0000 7ff955610700 1 -- 172.21.15.23:0/1556569128 shutdown_connections 2020-11-20T11:38:54.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.231+0000 7ff955610700 1 -- 172.21.15.23:0/1556569128 wait complete. 2020-11-20T11:38:54.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.232+0000 7ff955610700 1 Processor -- start 2020-11-20T11:38:54.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.232+0000 7ff955610700 1 -- start start 2020-11-20T11:38:54.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.233+0000 7ff955610700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff950067e90 0x7ff950060b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:54.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.233+0000 7ff955610700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff9500af090 0x7ff95014d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:54.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.233+0000 7ff955610700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9501375a0 0x7ff95014d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:54.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.233+0000 7ff955610700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95014dd70 con 0x7ff9501375a0 2020-11-20T11:38:54.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.233+0000 7ff955610700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9500a4860 con 0x7ff9500af090 2020-11-20T11:38:54.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.233+0000 7ff955610700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9500a6440 con 0x7ff950067e90 2020-11-20T11:38:54.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.233+0000 7ff94f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9501375a0 0x7ff95014d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:54.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.233+0000 7ff94f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9501375a0 0x7ff95014d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43676/0 (socket says 172.21.15.23:43676) 2020-11-20T11:38:54.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.233+0000 7ff94f7fe700 1 -- 172.21.15.23:0/4086247791 learned_addr learned my addr 172.21.15.23:0/4086247791 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:38:54.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.233+0000 7ff94e7fc700 1 --2- 172.21.15.23:0/4086247791 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff9500af090 0x7ff95014d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:54.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.233+0000 7ff94effd700 1 --2- 172.21.15.23:0/4086247791 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff950067e90 0x7ff950060b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:54.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.233+0000 7ff94f7fe700 1 -- 172.21.15.23:0/4086247791 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff950067e90 msgr2=0x7ff950060b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:54.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.233+0000 7ff94f7fe700 1 --2- 172.21.15.23:0/4086247791 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff950067e90 0x7ff950060b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:54.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.233+0000 7ff94f7fe700 1 -- 172.21.15.23:0/4086247791 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff9500af090 msgr2=0x7ff95014d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:54.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.233+0000 7ff94f7fe700 1 --2- 172.21.15.23:0/4086247791 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff9500af090 0x7ff95014d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:54.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.233+0000 7ff94f7fe700 1 -- 172.21.15.23:0/4086247791 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9500a66e0 con 0x7ff9501375a0 2020-11-20T11:38:54.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.233+0000 7ff94f7fe700 1 --2- 172.21.15.23:0/4086247791 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9501375a0 0x7ff95014d880 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7ff948009c60 tx=0x7ff948016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:54.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.233+0000 7ff936ffd700 1 -- 172.21.15.23:0/4086247791 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff948011e30 con 0x7ff9501375a0 2020-11-20T11:38:54.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.233+0000 7ff936ffd700 1 -- 172.21.15.23:0/4086247791 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff948009e20 con 0x7ff9501375a0 2020-11-20T11:38:54.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.233+0000 7ff955610700 1 -- 172.21.15.23:0/4086247791 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff950138780 con 0x7ff9501375a0 2020-11-20T11:38:54.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.234+0000 7ff936ffd700 1 -- 172.21.15.23:0/4086247791 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94801d480 con 0x7ff9501375a0 2020-11-20T11:38:54.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.234+0000 7ff955610700 1 -- 172.21.15.23:0/4086247791 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9500a4350 con 0x7ff9501375a0 2020-11-20T11:38:54.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.235+0000 7ff955610700 1 -- 172.21.15.23:0/4086247791 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff950057dc0 con 0x7ff9501375a0 2020-11-20T11:38:54.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.235+0000 7ff936ffd700 1 -- 172.21.15.23:0/4086247791 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff948017070 con 0x7ff9501375a0 2020-11-20T11:38:54.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.235+0000 7ff936ffd700 1 --2- 172.21.15.23:0/4086247791 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff92c056d60 0x7ff92c059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:54.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.235+0000 7ff936ffd700 1 -- 172.21.15.23:0/4086247791 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7ff9480729b0 con 0x7ff9501375a0 2020-11-20T11:38:54.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.236+0000 7ff94effd700 1 --2- 172.21.15.23:0/4086247791 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff92c056d60 0x7ff92c059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:54.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.236+0000 7ff94effd700 1 --2- 172.21.15.23:0/4086247791 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff92c056d60 0x7ff92c059220 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7ff9400011e0 tx=0x7ff940007a40).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:54.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.240+0000 7ff936ffd700 1 -- 172.21.15.23:0/4086247791 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff948047ca0 con 0x7ff9501375a0 2020-11-20T11:38:54.515 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.511+0000 7ff955610700 1 -- 172.21.15.23:0/4086247791 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff9500a5e00 con 0x7ff9501375a0 2020-11-20T11:38:54.515 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.511+0000 7ff936ffd700 1 -- 172.21.15.23:0/4086247791 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7ff948015070 con 0x7ff9501375a0 2020-11-20T11:38:54.516 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:38:54.521 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.513+0000 7ff955610700 1 -- 172.21.15.23:0/4086247791 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff92c056d60 msgr2=0x7ff92c059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:54.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.513+0000 7ff955610700 1 --2- 172.21.15.23:0/4086247791 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff92c056d60 0x7ff92c059220 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7ff9400011e0 tx=0x7ff940007a40).stop 2020-11-20T11:38:54.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.513+0000 7ff955610700 1 -- 172.21.15.23:0/4086247791 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9501375a0 msgr2=0x7ff95014d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:54.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.513+0000 7ff955610700 1 --2- 172.21.15.23:0/4086247791 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9501375a0 0x7ff95014d880 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7ff948009c60 tx=0x7ff948016000).stop 2020-11-20T11:38:54.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.513+0000 7ff955610700 1 -- 172.21.15.23:0/4086247791 shutdown_connections 2020-11-20T11:38:54.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.513+0000 7ff955610700 1 --2- 172.21.15.23:0/4086247791 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9501375a0 0x7ff95014d880 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:54.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.513+0000 7ff955610700 1 --2- 172.21.15.23:0/4086247791 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff92c056d60 0x7ff92c059220 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:54.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.513+0000 7ff955610700 1 --2- 172.21.15.23:0/4086247791 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff950067e90 0x7ff950060b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:54.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.513+0000 7ff955610700 1 --2- 172.21.15.23:0/4086247791 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff9500af090 0x7ff95014d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:54.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.513+0000 7ff955610700 1 -- 172.21.15.23:0/4086247791 >> 172.21.15.23:0/4086247791 conn(0x7ff950063ac0 msgr2=0x7ff9500ac280 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:54.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.513+0000 7ff955610700 1 -- 172.21.15.23:0/4086247791 shutdown_connections 2020-11-20T11:38:54.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.513+0000 7ff955610700 1 -- 172.21.15.23:0/4086247791 wait complete. 2020-11-20T11:38:54.530 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:38:54.530 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2020-11-20T11:38:54.530 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:38:54.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.655+0000 7f33519a1700 1 Processor -- start 2020-11-20T11:38:54.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.655+0000 7f33519a1700 1 -- start start 2020-11-20T11:38:54.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.656+0000 7f33519a1700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f334c0aaf00 0x7f334c13e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:54.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.656+0000 7f33519a1700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f334c0ab2e0 0x7f334c0a7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:54.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.656+0000 7f33519a1700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f334c13e010 con 0x7f334c0a85c0 2020-11-20T11:38:54.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.656+0000 7f33519a1700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f334c13e2b0 con 0x7f334c0ab2e0 2020-11-20T11:38:54.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.656+0000 7f33519a1700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f334c13e550 con 0x7f334c0aaf00 2020-11-20T11:38:54.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.656+0000 7f334b7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f334c0ab2e0 0x7f334c0a7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:54.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.656+0000 7f334b7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f334c0ab2e0 0x7f334c0a7fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43682/0 (socket says 172.21.15.23:43682) 2020-11-20T11:38:54.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.656+0000 7f334b7fe700 1 -- 172.21.15.23:0/175503299 learned_addr learned my addr 172.21.15.23:0/175503299 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:38:54.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.656+0000 7f334affd700 1 --2- 172.21.15.23:0/175503299 >> v2:172.21.15.155:3300/0 conn(0x7f334c0aaf00 0x7f334c13e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:54.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.656+0000 7f334b7fe700 1 -- 172.21.15.23:0/175503299 >> v2:172.21.15.155:3300/0 conn(0x7f334c0aaf00 msgr2=0x7f334c13e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:38:54.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.656+0000 7f334b7fe700 1 --2- 172.21.15.23:0/175503299 >> v2:172.21.15.155:3300/0 conn(0x7f334c0aaf00 0x7f334c13e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:54.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.656+0000 7f334b7fe700 1 -- 172.21.15.23:0/175503299 >> v1:172.21.15.155:6789/0 conn(0x7f334c0a85c0 legacy=0x7f334c0a78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:38:54.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.656+0000 7f334b7fe700 1 -- 172.21.15.23:0/175503299 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f334c0a9000 con 0x7f334c0ab2e0 2020-11-20T11:38:54.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.656+0000 7f334affd700 1 --2- 172.21.15.23:0/175503299 >> v2:172.21.15.155:3300/0 conn(0x7f334c0aaf00 0x7f334c13e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:38:54.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.657+0000 7f334b7fe700 1 --2- 172.21.15.23:0/175503299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f334c0ab2e0 0x7f334c0a7fc0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f334c0b10c0 tx=0x7f33440092d0).ready entity=mon.0 client_cookie=428509a04e62f314 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:54.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.657+0000 7f334a7fc700 1 -- 172.21.15.23:0/175503299 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334400bbb0 con 0x7f334c0ab2e0 2020-11-20T11:38:54.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.657+0000 7f334a7fc700 1 -- 172.21.15.23:0/175503299 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f334400bd10 con 0x7f334c0ab2e0 2020-11-20T11:38:54.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.657+0000 7f334a7fc700 1 -- 172.21.15.23:0/175503299 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3344011610 con 0x7f334c0ab2e0 2020-11-20T11:38:54.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.657+0000 7f33519a1700 1 -- 172.21.15.23:0/175503299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f334c0ab2e0 msgr2=0x7f334c0a7fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:54.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.657+0000 7f33519a1700 1 --2- 172.21.15.23:0/175503299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f334c0ab2e0 0x7f334c0a7fc0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f334c0b10c0 tx=0x7f33440092d0).stop 2020-11-20T11:38:54.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.657+0000 7f33519a1700 1 -- 172.21.15.23:0/175503299 shutdown_connections 2020-11-20T11:38:54.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.657+0000 7f33519a1700 1 --2- 172.21.15.23:0/175503299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f334c0ab2e0 0x7f334c0a7fc0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:54.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.657+0000 7f33519a1700 1 --2- 172.21.15.23:0/175503299 >> v2:172.21.15.155:3300/0 conn(0x7f334c0aaf00 0x7f334c13e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:54.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.657+0000 7f33519a1700 1 -- 172.21.15.23:0/175503299 >> 172.21.15.23:0/175503299 conn(0x7f334c0678e0 msgr2=0x7f334c067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:54.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.657+0000 7f33519a1700 1 -- 172.21.15.23:0/175503299 shutdown_connections 2020-11-20T11:38:54.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.658+0000 7f33519a1700 1 -- 172.21.15.23:0/175503299 wait complete. 2020-11-20T11:38:54.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.658+0000 7f33519a1700 1 Processor -- start 2020-11-20T11:38:54.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.659+0000 7f33519a1700 1 -- start start 2020-11-20T11:38:54.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.659+0000 7f33519a1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f334c0a85c0 0x7f334c0ac090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:54.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.659+0000 7f33519a1700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f334c0aaf00 0x7f334c139fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:54.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.659+0000 7f33519a1700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f334c0ab2e0 0x7f334c14d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:54.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.659+0000 7f33519a1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334c139550 con 0x7f334c0a85c0 2020-11-20T11:38:54.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.659+0000 7f33519a1700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334c1397f0 con 0x7f334c0aaf00 2020-11-20T11:38:54.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.659+0000 7f33519a1700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f334c139a90 con 0x7f334c0ab2e0 2020-11-20T11:38:54.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.659+0000 7f334affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f334c0a85c0 0x7f334c0ac090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:54.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.659+0000 7f334affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f334c0a85c0 0x7f334c0ac090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43684/0 (socket says 172.21.15.23:43684) 2020-11-20T11:38:54.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.659+0000 7f334affd700 1 -- 172.21.15.23:0/973038640 learned_addr learned my addr 172.21.15.23:0/973038640 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:38:54.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.659+0000 7f334b7fe700 1 --2- 172.21.15.23:0/973038640 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f334c0ab2e0 0x7f334c14d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:54.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.659+0000 7f3343fff700 1 --2- 172.21.15.23:0/973038640 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f334c0aaf00 0x7f334c139fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:54.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.660+0000 7f334affd700 1 -- 172.21.15.23:0/973038640 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f334c0ab2e0 msgr2=0x7f334c14d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:54.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.660+0000 7f334affd700 1 --2- 172.21.15.23:0/973038640 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f334c0ab2e0 0x7f334c14d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:54.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.660+0000 7f334affd700 1 -- 172.21.15.23:0/973038640 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f334c0aaf00 msgr2=0x7f334c139fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:54.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.660+0000 7f334affd700 1 --2- 172.21.15.23:0/973038640 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f334c0aaf00 0x7f334c139fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:54.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.660+0000 7f334affd700 1 -- 172.21.15.23:0/973038640 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f334c13a4a0 con 0x7f334c0a85c0 2020-11-20T11:38:54.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.660+0000 7f334affd700 1 --2- 172.21.15.23:0/973038640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f334c0a85c0 0x7f334c0ac090 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f333c00cb50 tx=0x7f333c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:54.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.660+0000 7f3342ffd700 1 -- 172.21.15.23:0/973038640 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c007740 con 0x7f334c0a85c0 2020-11-20T11:38:54.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.660+0000 7f3342ffd700 1 -- 172.21.15.23:0/973038640 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f333c0078a0 con 0x7f334c0a85c0 2020-11-20T11:38:54.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.660+0000 7f33519a1700 1 -- 172.21.15.23:0/973038640 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f334c1400d0 con 0x7f334c0a85c0 2020-11-20T11:38:54.675 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.660+0000 7f3342ffd700 1 -- 172.21.15.23:0/973038640 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c007c20 con 0x7f334c0a85c0 2020-11-20T11:38:54.675 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.660+0000 7f33519a1700 1 -- 172.21.15.23:0/973038640 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f334c14d820 con 0x7f334c0a85c0 2020-11-20T11:38:54.675 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.661+0000 7f33519a1700 1 --2- 172.21.15.23:0/973038640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f334c13ef50 0x7f334c13f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:54.675 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.661+0000 7f33519a1700 1 -- 172.21.15.23:0/973038640 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334c05cf20 con 0x7f334c13ef50 2020-11-20T11:38:54.675 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.661+0000 7f3343fff700 1 --2- 172.21.15.23:0/973038640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f334c13ef50 0x7f334c13f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:54.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.661+0000 7f3342ffd700 1 -- 172.21.15.23:0/973038640 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f333c01b460 con 0x7f334c0a85c0 2020-11-20T11:38:54.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.661+0000 7f3343fff700 1 -- 172.21.15.23:0/973038640 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f334c13a690 con 0x7f334c13ef50 2020-11-20T11:38:54.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.661+0000 7f3343fff700 1 --2- 172.21.15.23:0/973038640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f334c13ef50 0x7f334c13f330 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f333400e6e0 tx=0x7f3334009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:54.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.662+0000 7f3342ffd700 1 --2- 172.21.15.23:0/973038640 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3328056d60 0x7f3328059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:54.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.662+0000 7f3342ffd700 1 -- 172.21.15.23:0/973038640 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f333c0702e0 con 0x7f334c0a85c0 2020-11-20T11:38:54.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.662+0000 7f3342ffd700 1 -- 172.21.15.23:0/973038640 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3334007240 con 0x7f334c13ef50 2020-11-20T11:38:54.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.662+0000 7f334b7fe700 1 --2- 172.21.15.23:0/973038640 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3328056d60 0x7f3328059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:54.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.662+0000 7f3342ffd700 1 -- 172.21.15.23:0/973038640 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f3334000e90 con 0x7f334c13ef50 2020-11-20T11:38:54.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.662+0000 7f3342ffd700 1 -- 172.21.15.23:0/973038640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f334c13ef50 msgr2=0x7f334c13f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:54.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.662+0000 7f3342ffd700 1 --2- 172.21.15.23:0/973038640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f334c13ef50 0x7f334c13f330 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f333400e6e0 tx=0x7f3334009040).stop 2020-11-20T11:38:54.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.662+0000 7f3342ffd700 1 -- 172.21.15.23:0/973038640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f334c13ef50 msgr2=0x7f334c13f330 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:38:54.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.662+0000 7f3342ffd700 1 --2- 172.21.15.23:0/973038640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f334c13ef50 0x7f334c13f330 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:54.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.662+0000 7f334b7fe700 1 --2- 172.21.15.23:0/973038640 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3328056d60 0x7f3328059220 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f334c0b10c0 tx=0x7f3344016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:54.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.669+0000 7f33519a1700 1 --2- 172.21.15.23:0/973038640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f334c13f820 0x7f334c12ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:54.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.669+0000 7f33519a1700 1 -- 172.21.15.23:0/973038640 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334c139690 con 0x7f334c13f820 2020-11-20T11:38:54.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.670+0000 7f3343fff700 1 --2- 172.21.15.23:0/973038640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f334c13f820 0x7f334c12ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:54.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.670+0000 7f3343fff700 1 -- 172.21.15.23:0/973038640 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f334c057dc0 con 0x7f334c13f820 2020-11-20T11:38:54.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.670+0000 7f3343fff700 1 --2- 172.21.15.23:0/973038640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f334c13f820 0x7f334c12ff40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f3334015000 tx=0x7f3334018000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:54.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.670+0000 7f3342ffd700 1 -- 172.21.15.23:0/973038640 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3334008ea0 con 0x7f334c13f820 2020-11-20T11:38:54.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.670+0000 7f3342ffd700 1 -- 172.21.15.23:0/973038640 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4446 (secure 0 0 0) 0x7f3334007030 con 0x7f334c13f820 2020-11-20T11:38:54.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.670+0000 7f3342ffd700 1 -- 172.21.15.23:0/973038640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f334c13f820 msgr2=0x7f334c12ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:54.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.670+0000 7f3342ffd700 1 --2- 172.21.15.23:0/973038640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f334c13f820 0x7f334c12ff40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f3334015000 tx=0x7f3334018000).stop 2020-11-20T11:38:54.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.670+0000 7f3342ffd700 1 -- 172.21.15.23:0/973038640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f334c13f820 msgr2=0x7f334c12ff40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:38:54.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.670+0000 7f3342ffd700 1 --2- 172.21.15.23:0/973038640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f334c13f820 0x7f334c12ff40 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:54.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.671+0000 7f33519a1700 1 -- 172.21.15.23:0/973038640 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3328056d60 msgr2=0x7f3328059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:54.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.671+0000 7f33519a1700 1 --2- 172.21.15.23:0/973038640 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3328056d60 0x7f3328059220 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f334c0b10c0 tx=0x7f3344016000).stop 2020-11-20T11:38:54.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.671+0000 7f33519a1700 1 -- 172.21.15.23:0/973038640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f334c0a85c0 msgr2=0x7f334c0ac090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:54.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.671+0000 7f33519a1700 1 --2- 172.21.15.23:0/973038640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f334c0a85c0 0x7f334c0ac090 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f333c00cb50 tx=0x7f333c008040).stop 2020-11-20T11:38:54.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.671+0000 7f334b7fe700 1 -- 172.21.15.23:0/973038640 reap_dead start 2020-11-20T11:38:54.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.671+0000 7f334b7fe700 1 -- 172.21.15.23:0/973038640 reap_dead start 2020-11-20T11:38:54.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.671+0000 7f33519a1700 1 -- 172.21.15.23:0/973038640 shutdown_connections 2020-11-20T11:38:54.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.671+0000 7f33519a1700 1 -- 172.21.15.23:0/973038640 >> 172.21.15.23:0/973038640 conn(0x7f334c0678e0 msgr2=0x7f334c13ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:54.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.671+0000 7f33519a1700 1 -- 172.21.15.23:0/973038640 shutdown_connections 2020-11-20T11:38:54.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.671+0000 7f33519a1700 1 -- 172.21.15.23:0/973038640 wait complete. 2020-11-20T11:38:54.684 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:38:54.685 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:38:54.685 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:38:54.685 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:38:54.685 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 66, 2020-11-20T11:38:54.685 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:38:54.686 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:38:54.686 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:38:54.686 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:38:54.686 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:54.687 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 14, 2020-11-20T11:38:54.687 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:38:54.687 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:38:54.687 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:38:54.687 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:38:54.688 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:38:54.688 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:38:54.688 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:38:54.688 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:38:54.688 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:38:54.689 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:38:54.689 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:38:54.689 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:54.689 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:38:54.689 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:38:54.690 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:38:54.690 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:38:54.690 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:38:54.690 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:38:54.690 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:38:54.691 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:38:54.691 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:38:54.691 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:38:54.691 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:54.691 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:54.692 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:38:54.692 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:38:54.692 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:38:54.692 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:38:54.692 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:38:54.692 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:38:54.693 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:38:54.693 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:38:54.693 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:38:54.693 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:38:54.694 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:38:54.694 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:38:54.694 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:38:54.694 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:38:54.694 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:38:54.695 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:38:54.695 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:38:54.695 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:38:54.695 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:38:54.695 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:38:54.696 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:38:54.696 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:38:54.696 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:38:54.696 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:54.696 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:38:54.697 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:54.697 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:38:54.697 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:54.697 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:38:54.697 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:38:54.698 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:38:54.698 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:38:54.698 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:54.698 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:38:54.698 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:38:54.699 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:54.699 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:54.699 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:54.699 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:38:54.699 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:38:54.700 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:54.700 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:54.700 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:54.700 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:54.700 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:38:54.701 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:38:54.701 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:38:54.701 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:38:54.701 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:38:54.701 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:54.702 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:54.702 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:38:54.702 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:38:54.702 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:38:54.702 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:38:54.703 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:54.703 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:38:54.703 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:38:54.703 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:54.703 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:54.704 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:54.704 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:38:54.704 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:38:54.704 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:54.704 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:54.705 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:54.705 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:54.705 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:38:54.705 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:38:54.706 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:38:54.706 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:38:54.706 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:38:54.706 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:54.706 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:54.706 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:38:54.707 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:38:54.707 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:38:54.707 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:38:54.707 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:54.707 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:38:54.708 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:38:54.708 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:54.708 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:54.708 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:54.708 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:38:54.709 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:38:54.709 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:54.709 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:54.709 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:54.709 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:54.710 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:38:54.710 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:38:54.710 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:38:54.710 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:38:54.710 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:38:54.711 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:54.711 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:54.711 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:54.711 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:38:54.711 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:38:54.712 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:54.712 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:38:54.712 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:38:54.712 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:38:54.712 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:54.713 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:54.713 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:38:54.713 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:54.713 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:38:54.713 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:38:54.714 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:38:54.714 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:54.714 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:54.714 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:38:54.714 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:54.715 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:38:54.715 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:38:54.715 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:38:54.715 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:54.715 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:54.716 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:54.716 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:38:54.716 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:38:54.717 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:38:54.839 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.835+0000 7f0422272700 1 Processor -- start 2020-11-20T11:38:54.840 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.835+0000 7f0422272700 1 -- start start 2020-11-20T11:38:54.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.836+0000 7f0422272700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f041c0aaab0 0x7f041c13ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:54.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.836+0000 7f0422272700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f041c13aeb0 0x7f041c0a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:54.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.836+0000 7f0422272700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f041c1388b0 con 0x7f041c0aae90 2020-11-20T11:38:54.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.836+0000 7f0422272700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f041c13e600 con 0x7f041c0aaab0 2020-11-20T11:38:54.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.836+0000 7f0422272700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f041c13e8a0 con 0x7f041c13aeb0 2020-11-20T11:38:54.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.836+0000 7f041bfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f041c0aaab0 0x7f041c13ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:54.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.836+0000 7f041b7fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f041c0aae90 0x7f041c1381a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57946/0 (socket says 172.21.15.23:57946) 2020-11-20T11:38:54.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.836+0000 7f041b7fe700 1 -- 172.21.15.23:0/293955620 learned_addr learned my addr 172.21.15.23:0/293955620 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:38:54.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.836+0000 7f042080f700 1 --2- 172.21.15.23:0/293955620 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f041c13aeb0 0x7f041c0a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:54.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.836+0000 7f041bfff700 1 -- 172.21.15.23:0/293955620 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f041c13aeb0 msgr2=0x7f041c0a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:38:54.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.836+0000 7f041bfff700 1 --2- 172.21.15.23:0/293955620 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f041c13aeb0 0x7f041c0a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:54.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.836+0000 7f041bfff700 1 -- 172.21.15.23:0/293955620 >> v1:172.21.15.23:6789/0 conn(0x7f041c0aae90 legacy=0x7f041c1381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:54.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.836+0000 7f041affd700 1 -- 172.21.15.23:0/293955620 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 555303856 0 0) 0x7f041c1388b0 con 0x7f041c0aae90 2020-11-20T11:38:54.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.836+0000 7f041bfff700 1 -- 172.21.15.23:0/293955620 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f041c0ab6d0 con 0x7f041c0aaab0 2020-11-20T11:38:54.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.837+0000 7f041bfff700 1 --2- 172.21.15.23:0/293955620 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f041c0aaab0 0x7f041c13ed40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f040c006fd0 tx=0x7f040c006e30).ready entity=mon.0 client_cookie=5bfedbf8823d244b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:54.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.837+0000 7f041affd700 1 -- 172.21.15.23:0/293955620 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040c0097e0 con 0x7f041c0aaab0 2020-11-20T11:38:54.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.837+0000 7f041affd700 1 -- 172.21.15.23:0/293955620 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f040c000e90 con 0x7f041c0aaab0 2020-11-20T11:38:54.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.837+0000 7f041affd700 1 -- 172.21.15.23:0/293955620 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040c009b50 con 0x7f041c0aaab0 2020-11-20T11:38:54.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.837+0000 7f0422272700 1 -- 172.21.15.23:0/293955620 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f041c0aaab0 msgr2=0x7f041c13ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:54.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.837+0000 7f0422272700 1 --2- 172.21.15.23:0/293955620 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f041c0aaab0 0x7f041c13ed40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f040c006fd0 tx=0x7f040c006e30).stop 2020-11-20T11:38:54.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.837+0000 7f0422272700 1 -- 172.21.15.23:0/293955620 shutdown_connections 2020-11-20T11:38:54.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.837+0000 7f0422272700 1 --2- 172.21.15.23:0/293955620 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f041c13aeb0 0x7f041c0a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:54.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.837+0000 7f0422272700 1 --2- 172.21.15.23:0/293955620 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f041c0aaab0 0x7f041c13ed40 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:54.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.837+0000 7f0422272700 1 -- 172.21.15.23:0/293955620 >> 172.21.15.23:0/293955620 conn(0x7f041c0678e0 msgr2=0x7f041c067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:54.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.837+0000 7f0422272700 1 -- 172.21.15.23:0/293955620 shutdown_connections 2020-11-20T11:38:54.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.837+0000 7f0422272700 1 -- 172.21.15.23:0/293955620 wait complete. 2020-11-20T11:38:54.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.838+0000 7f0422272700 1 Processor -- start 2020-11-20T11:38:54.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.839+0000 7f0422272700 1 -- start start 2020-11-20T11:38:54.851 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.839+0000 7f0422272700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f041c0aaab0 0x7f041c140660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:54.851 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.839+0000 7f0422272700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f041c0aae90 0x7f041c14d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:54.851 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.839+0000 7f0422272700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f041c13ccb0 0x7f041c133610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:54.852 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.839+0000 7f0422272700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041c14a4f0 con 0x7f041c0aaab0 2020-11-20T11:38:54.852 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.839+0000 7f0422272700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041c14a790 con 0x7f041c0aae90 2020-11-20T11:38:54.852 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.839+0000 7f0422272700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f041c14aa30 con 0x7f041c13ccb0 2020-11-20T11:38:54.852 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.839+0000 7f041bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f041c0aaab0 0x7f041c140660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:54.852 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.839+0000 7f041bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f041c0aaab0 0x7f041c140660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43702/0 (socket says 172.21.15.23:43702) 2020-11-20T11:38:54.853 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.839+0000 7f041bfff700 1 -- 172.21.15.23:0/4134647372 learned_addr learned my addr 172.21.15.23:0/4134647372 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:38:54.853 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.839+0000 7f041b7fe700 1 --2- 172.21.15.23:0/4134647372 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f041c0aae90 0x7f041c14d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:54.853 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.839+0000 7f042080f700 1 --2- 172.21.15.23:0/4134647372 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f041c13ccb0 0x7f041c133610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:54.853 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.840+0000 7f041bfff700 1 -- 172.21.15.23:0/4134647372 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f041c13ccb0 msgr2=0x7f041c133610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:54.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.840+0000 7f041bfff700 1 --2- 172.21.15.23:0/4134647372 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f041c13ccb0 0x7f041c133610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:54.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.840+0000 7f041bfff700 1 -- 172.21.15.23:0/4134647372 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f041c0aae90 msgr2=0x7f041c14d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:54.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.840+0000 7f041bfff700 1 --2- 172.21.15.23:0/4134647372 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f041c0aae90 0x7f041c14d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:54.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.840+0000 7f041bfff700 1 -- 172.21.15.23:0/4134647372 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f041c13e3b0 con 0x7f041c0aaab0 2020-11-20T11:38:54.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.840+0000 7f041b7fe700 1 --2- 172.21.15.23:0/4134647372 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f041c0aae90 0x7f041c14d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:38:54.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.840+0000 7f041bfff700 1 --2- 172.21.15.23:0/4134647372 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f041c0aaab0 0x7f041c140660 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f041c062f90 tx=0x7f040c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:54.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.840+0000 7f03fbfff700 1 -- 172.21.15.23:0/4134647372 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040c011c60 con 0x7f041c0aaab0 2020-11-20T11:38:54.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.840+0000 7f03fbfff700 1 -- 172.21.15.23:0/4134647372 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f040c011dc0 con 0x7f041c0aaab0 2020-11-20T11:38:54.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.840+0000 7f03fbfff700 1 -- 172.21.15.23:0/4134647372 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040c01e690 con 0x7f041c0aaab0 2020-11-20T11:38:54.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.840+0000 7f0422272700 1 -- 172.21.15.23:0/4134647372 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f041c142420 con 0x7f041c0aaab0 2020-11-20T11:38:54.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.840+0000 7f0422272700 1 -- 172.21.15.23:0/4134647372 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f041c14a140 con 0x7f041c0aaab0 2020-11-20T11:38:54.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.841+0000 7f0422272700 1 --2- 172.21.15.23:0/4134647372 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f041c13aeb0 0x7f041c133b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:54.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.841+0000 7f0422272700 1 -- 172.21.15.23:0/4134647372 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041c14b090 con 0x7f041c13aeb0 2020-11-20T11:38:54.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.841+0000 7f041b7fe700 1 --2- 172.21.15.23:0/4134647372 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f041c13aeb0 0x7f041c133b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:54.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.841+0000 7f03fbfff700 1 -- 172.21.15.23:0/4134647372 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f040c025030 con 0x7f041c0aaab0 2020-11-20T11:38:54.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.841+0000 7f041b7fe700 1 -- 172.21.15.23:0/4134647372 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f041c133ff0 con 0x7f041c13aeb0 2020-11-20T11:38:54.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.842+0000 7f041b7fe700 1 --2- 172.21.15.23:0/4134647372 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f041c13aeb0 0x7f041c133b00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f041c142750 tx=0x7f0410009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:54.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.842+0000 7f03fbfff700 1 --2- 172.21.15.23:0/4134647372 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f03f4056d60 0x7f03f4059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:54.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.842+0000 7f03fbfff700 1 -- 172.21.15.23:0/4134647372 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f040c0715f0 con 0x7f041c0aaab0 2020-11-20T11:38:54.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.842+0000 7f03fbfff700 1 -- 172.21.15.23:0/4134647372 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0410007320 con 0x7f041c13aeb0 2020-11-20T11:38:54.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.842+0000 7f042080f700 1 --2- 172.21.15.23:0/4134647372 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f03f4056d60 0x7f03f4059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:54.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.842+0000 7f03fbfff700 1 -- 172.21.15.23:0/4134647372 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f0410007480 con 0x7f041c13aeb0 2020-11-20T11:38:54.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.842+0000 7f03fbfff700 1 -- 172.21.15.23:0/4134647372 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f041c13aeb0 msgr2=0x7f041c133b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:54.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.842+0000 7f03fbfff700 1 --2- 172.21.15.23:0/4134647372 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f041c13aeb0 0x7f041c133b00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f041c142750 tx=0x7f0410009040).stop 2020-11-20T11:38:54.859 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.842+0000 7f03fbfff700 1 -- 172.21.15.23:0/4134647372 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f041c13aeb0 msgr2=0x7f041c133b00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:38:54.859 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.842+0000 7f03fbfff700 1 --2- 172.21.15.23:0/4134647372 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f041c13aeb0 0x7f041c133b00 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:54.859 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.842+0000 7f042080f700 1 --2- 172.21.15.23:0/4134647372 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f03f4056d60 0x7f03f4059220 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f041c062f60 tx=0x7f0414008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:54.861 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.852+0000 7f0422272700 1 --2- 172.21.15.23:0/4134647372 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f041c0601b0 0x7f041c0b02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:54.861 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.852+0000 7f0422272700 1 -- 172.21.15.23:0/4134647372 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041c14a330 con 0x7f041c0601b0 2020-11-20T11:38:54.861 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.852+0000 7f041b7fe700 1 --2- 172.21.15.23:0/4134647372 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f041c0601b0 0x7f041c0b02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:54.862 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.852+0000 7f041b7fe700 1 -- 172.21.15.23:0/4134647372 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f041c05c7d0 con 0x7f041c0601b0 2020-11-20T11:38:54.862 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.853+0000 7f041b7fe700 1 --2- 172.21.15.23:0/4134647372 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f041c0601b0 0x7f041c0b02a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f0410015000 tx=0x7f0410009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:54.862 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.853+0000 7f03fbfff700 1 -- 172.21.15.23:0/4134647372 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0410014c60 con 0x7f041c0601b0 2020-11-20T11:38:54.862 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.853+0000 7f03fbfff700 1 -- 172.21.15.23:0/4134647372 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4784 (secure 0 0 0) 0x7f0410007110 con 0x7f041c0601b0 2020-11-20T11:38:54.862 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.853+0000 7f03fbfff700 1 -- 172.21.15.23:0/4134647372 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f041c0601b0 msgr2=0x7f041c0b02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:54.863 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.853+0000 7f03fbfff700 1 --2- 172.21.15.23:0/4134647372 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f041c0601b0 0x7f041c0b02a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f0410015000 tx=0x7f0410009040).stop 2020-11-20T11:38:54.863 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.853+0000 7f03fbfff700 1 -- 172.21.15.23:0/4134647372 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f041c0601b0 msgr2=0x7f041c0b02a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:38:54.863 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.853+0000 7f03fbfff700 1 --2- 172.21.15.23:0/4134647372 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f041c0601b0 0x7f041c0b02a0 secure :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0x7f0410015000 tx=0x7f0410009040).stop 2020-11-20T11:38:54.863 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.853+0000 7f0422272700 1 -- 172.21.15.23:0/4134647372 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f03f4056d60 msgr2=0x7f03f4059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:54.864 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.853+0000 7f0422272700 1 --2- 172.21.15.23:0/4134647372 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f03f4056d60 0x7f03f4059220 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f041c062f60 tx=0x7f0414008040).stop 2020-11-20T11:38:54.864 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.853+0000 7f0422272700 1 -- 172.21.15.23:0/4134647372 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f041c0aaab0 msgr2=0x7f041c140660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:54.864 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.853+0000 7f0422272700 1 --2- 172.21.15.23:0/4134647372 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f041c0aaab0 0x7f041c140660 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f041c062f90 tx=0x7f040c015000).stop 2020-11-20T11:38:54.864 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.853+0000 7f042080f700 1 -- 172.21.15.23:0/4134647372 reap_dead start 2020-11-20T11:38:54.864 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.853+0000 7f042080f700 1 -- 172.21.15.23:0/4134647372 reap_dead start 2020-11-20T11:38:54.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.853+0000 7f0422272700 1 -- 172.21.15.23:0/4134647372 shutdown_connections 2020-11-20T11:38:54.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.853+0000 7f0422272700 1 -- 172.21.15.23:0/4134647372 >> 172.21.15.23:0/4134647372 conn(0x7f041c0678e0 msgr2=0x7f041c0aba00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:54.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.854+0000 7f0422272700 1 -- 172.21.15.23:0/4134647372 shutdown_connections 2020-11-20T11:38:54.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:54.854+0000 7f0422272700 1 -- 172.21.15.23:0/4134647372 wait complete. 2020-11-20T11:38:54.866 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:38:54.866 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:38:54.866 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:38:54.866 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:38:54.867 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 66, 2020-11-20T11:38:54.867 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:38:54.867 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:38:54.867 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:38:54.867 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:38:54.868 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:54.868 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 15, 2020-11-20T11:38:54.868 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:38:54.868 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:38:54.868 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:38:54.869 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:38:54.869 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:38:54.869 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:38:54.869 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:38:54.869 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:38:54.870 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:38:54.870 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:38:54.870 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:38:54.870 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:54.870 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:38:54.871 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:38:54.871 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:38:54.871 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:38:54.871 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:38:54.871 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:38:54.872 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:38:54.872 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:38:54.872 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:38:54.872 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:38:54.872 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:54.873 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:54.873 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:38:54.873 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:38:54.873 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:38:54.874 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:38:54.874 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:38:54.874 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:38:54.874 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:38:54.874 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:38:54.875 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:38:54.875 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:38:54.875 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:38:54.875 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:38:54.875 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:38:54.876 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:38:54.876 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:38:54.876 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:38:54.876 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:38:54.877 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:38:54.877 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:38:54.877 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:38:54.877 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:38:54.878 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:38:54.878 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:38:54.878 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:54.878 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:38:54.878 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:54.879 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:38:54.879 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:54.879 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:38:54.879 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:38:54.879 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:38:54.880 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:38:54.880 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:54.880 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:38:54.880 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:38:54.880 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:54.881 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:54.881 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:54.881 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:38:54.881 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:38:54.882 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:54.882 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:54.882 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:54.882 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:54.883 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:38:54.883 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:38:54.883 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:38:54.883 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:38:54.883 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:38:54.884 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:54.884 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:54.884 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:38:54.884 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:38:54.884 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:38:54.885 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:38:54.885 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:54.885 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:38:54.885 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:38:54.886 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:54.886 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:54.886 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:54.886 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:38:54.886 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:38:54.887 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:54.887 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:54.887 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:54.887 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:54.887 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:38:54.887 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:38:54.888 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:38:54.888 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:38:54.888 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:38:54.888 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:54.889 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:54.889 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:38:54.889 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:38:54.889 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:38:54.889 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:38:54.890 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:54.890 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:38:54.890 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:38:54.890 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:54.890 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:54.891 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:54.891 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:38:54.891 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:38:54.891 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:54.892 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:54.892 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:54.892 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:54.892 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:38:54.892 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:38:54.893 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:38:54.893 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:38:54.893 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:38:54.893 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:54.893 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:54.894 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:54.894 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:38:54.894 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:38:54.894 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:54.895 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:38:54.895 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:38:54.895 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:38:54.895 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:54.895 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:54.896 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:38:54.896 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:54.896 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:38:54.896 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:38:54.896 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:38:54.897 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:54.897 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:54.897 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:38:54.897 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:54.897 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:38:54.898 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:38:54.898 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:38:54.898 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:54.898 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:54.898 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:38:54.899 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:54.899 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:38:54.899 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:38:54.899 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:38:54.899 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:54.900 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:54.900 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:38:54.900 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:54.900 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:38:54.900 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:38:54.901 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:38:54.901 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:54.901 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:54.901 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:54.901 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:38:54.902 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:38:54.903 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:38:55.031 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.027+0000 7f12af2ca700 1 Processor -- start 2020-11-20T11:38:55.032 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.027+0000 7f12af2ca700 1 -- start start 2020-11-20T11:38:55.040 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.028+0000 7f12af2ca700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f12a8067690 0x7f12a8067b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:55.040 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.028+0000 7f12af2ca700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12a8135820 0x7f12a8135cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:55.041 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.028+0000 7f12af2ca700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a80a9fd0 con 0x7f12a80af060 2020-11-20T11:38:55.041 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.028+0000 7f12af2ca700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12a8068000 con 0x7f12a8067690 2020-11-20T11:38:55.041 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.028+0000 7f12af2ca700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a80aa230 con 0x7f12a8135820 2020-11-20T11:38:55.041 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.028+0000 7f12adac7700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f12a8067690 0x7f12a8067b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:55.041 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.028+0000 7f12ae2c8700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f12a80af060 0x7f12a80a6590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57964/0 (socket says 172.21.15.23:57964) 2020-11-20T11:38:55.042 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.028+0000 7f12ae2c8700 1 -- 172.21.15.23:0/1206665065 learned_addr learned my addr 172.21.15.23:0/1206665065 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:38:55.042 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.029+0000 7f12aeac9700 1 --2- 172.21.15.23:0/1206665065 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12a8135820 0x7f12a8135cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:55.042 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.029+0000 7f12adac7700 1 -- 172.21.15.23:0/1206665065 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12a8135820 msgr2=0x7f12a8135cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:38:55.043 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.029+0000 7f12adac7700 1 --2- 172.21.15.23:0/1206665065 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12a8135820 0x7f12a8135cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:55.043 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.029+0000 7f12adac7700 1 -- 172.21.15.23:0/1206665065 >> v1:172.21.15.23:6789/0 conn(0x7f12a80af060 legacy=0x7f12a80a6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:55.043 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.029+0000 7f12ad2c6700 1 -- 172.21.15.23:0/1206665065 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756025433 0 0) 0x7f12a80a9fd0 con 0x7f12a80af060 2020-11-20T11:38:55.043 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.029+0000 7f12adac7700 1 -- 172.21.15.23:0/1206665065 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a80aa4d0 con 0x7f12a8067690 2020-11-20T11:38:55.044 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.029+0000 7f12adac7700 1 --2- 172.21.15.23:0/1206665065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12a8067690 0x7f12a8067b10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f1298006fd0 tx=0x7f1298006e30).ready entity=mon.0 client_cookie=2ac13ac03cebeaa7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:55.044 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.029+0000 7f12ad2c6700 1 -- 172.21.15.23:0/1206665065 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12980097e0 con 0x7f12a8067690 2020-11-20T11:38:55.044 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.029+0000 7f12ad2c6700 1 -- 172.21.15.23:0/1206665065 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1298000e90 con 0x7f12a8067690 2020-11-20T11:38:55.045 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.029+0000 7f12ad2c6700 1 -- 172.21.15.23:0/1206665065 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1298009b50 con 0x7f12a8067690 2020-11-20T11:38:55.045 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.030+0000 7f12af2ca700 1 -- 172.21.15.23:0/1206665065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12a8067690 msgr2=0x7f12a8067b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:55.045 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.030+0000 7f12af2ca700 1 --2- 172.21.15.23:0/1206665065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12a8067690 0x7f12a8067b10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f1298006fd0 tx=0x7f1298006e30).stop 2020-11-20T11:38:55.045 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.030+0000 7f12af2ca700 1 -- 172.21.15.23:0/1206665065 shutdown_connections 2020-11-20T11:38:55.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.030+0000 7f12af2ca700 1 --2- 172.21.15.23:0/1206665065 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12a8135820 0x7f12a8135cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:55.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.030+0000 7f12af2ca700 1 --2- 172.21.15.23:0/1206665065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12a8067690 0x7f12a8067b10 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:55.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.030+0000 7f12af2ca700 1 -- 172.21.15.23:0/1206665065 >> 172.21.15.23:0/1206665065 conn(0x7f12a8063980 msgr2=0x7f12a8063d90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:55.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.030+0000 7f12af2ca700 1 -- 172.21.15.23:0/1206665065 shutdown_connections 2020-11-20T11:38:55.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.030+0000 7f12af2ca700 1 -- 172.21.15.23:0/1206665065 wait complete. 2020-11-20T11:38:55.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.031+0000 7f12af2ca700 1 Processor -- start 2020-11-20T11:38:55.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.031+0000 7f12af2ca700 1 -- start start 2020-11-20T11:38:55.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.031+0000 7f12af2ca700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12a8067690 0x7f12a813ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:55.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.031+0000 7f12af2ca700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12a80af060 0x7f12a80a76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:55.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.031+0000 7f12af2ca700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12a8135820 0x7f12a80a7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:55.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.031+0000 7f12af2ca700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a80a8090 con 0x7f12a8135820 2020-11-20T11:38:55.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.031+0000 7f12af2ca700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a813ae80 con 0x7f12a80af060 2020-11-20T11:38:55.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.031+0000 7f12af2ca700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a813b120 con 0x7f12a8067690 2020-11-20T11:38:55.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.032+0000 7f12aeac9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12a8135820 0x7f12a80a7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:55.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.032+0000 7f12aeac9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12a8135820 0x7f12a80a7ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43724/0 (socket says 172.21.15.23:43724) 2020-11-20T11:38:55.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.032+0000 7f12aeac9700 1 -- 172.21.15.23:0/902733004 learned_addr learned my addr 172.21.15.23:0/902733004 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:38:55.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.032+0000 7f12adac7700 1 --2- 172.21.15.23:0/902733004 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12a80af060 0x7f12a80a76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:55.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.032+0000 7f12ae2c8700 1 --2- 172.21.15.23:0/902733004 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12a8067690 0x7f12a813ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:55.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.032+0000 7f12aeac9700 1 -- 172.21.15.23:0/902733004 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12a8067690 msgr2=0x7f12a813ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:55.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.032+0000 7f12aeac9700 1 --2- 172.21.15.23:0/902733004 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12a8067690 0x7f12a813ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:55.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.032+0000 7f12aeac9700 1 -- 172.21.15.23:0/902733004 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12a80af060 msgr2=0x7f12a80a76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:55.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.032+0000 7f12aeac9700 1 --2- 172.21.15.23:0/902733004 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12a80af060 0x7f12a80a76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:55.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.032+0000 7f12aeac9700 1 -- 172.21.15.23:0/902733004 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a813b3c0 con 0x7f12a8135820 2020-11-20T11:38:55.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.032+0000 7f12adac7700 1 --2- 172.21.15.23:0/902733004 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f12a80af060 0x7f12a80a76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:38:55.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.032+0000 7f12aeac9700 1 --2- 172.21.15.23:0/902733004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12a8135820 0x7f12a80a7ba0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f12a000ebf0 tx=0x7f12a0008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:55.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.032+0000 7f12967fc700 1 -- 172.21.15.23:0/902733004 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a000c7c0 con 0x7f12a8135820 2020-11-20T11:38:55.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.032+0000 7f12967fc700 1 -- 172.21.15.23:0/902733004 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a000c920 con 0x7f12a8135820 2020-11-20T11:38:55.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.032+0000 7f12967fc700 1 -- 172.21.15.23:0/902733004 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a000ccc0 con 0x7f12a8135820 2020-11-20T11:38:55.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.032+0000 7f12af2ca700 1 -- 172.21.15.23:0/902733004 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12a813f360 con 0x7f12a8135820 2020-11-20T11:38:55.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.032+0000 7f12af2ca700 1 -- 172.21.15.23:0/902733004 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a80a8390 con 0x7f12a8135820 2020-11-20T11:38:55.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.034+0000 7f12af2ca700 1 --2- 172.21.15.23:0/902733004 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12a81311e0 0x7f12a81315c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:55.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.034+0000 7f12af2ca700 1 -- 172.21.15.23:0/902733004 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a80a8a50 con 0x7f12a81311e0 2020-11-20T11:38:55.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.034+0000 7f12967fc700 1 -- 172.21.15.23:0/902733004 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f12a0013640 con 0x7f12a8135820 2020-11-20T11:38:55.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.034+0000 7f12ae2c8700 1 --2- 172.21.15.23:0/902733004 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12a81311e0 0x7f12a81315c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:55.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.034+0000 7f12967fc700 1 --2- 172.21.15.23:0/902733004 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1280056d60 0x7f1280059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:55.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.034+0000 7f12967fc700 1 -- 172.21.15.23:0/902733004 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f12a0070150 con 0x7f12a8135820 2020-11-20T11:38:55.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.034+0000 7f12ae2c8700 1 -- 172.21.15.23:0/902733004 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f12a813b5c0 con 0x7f12a81311e0 2020-11-20T11:38:55.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.034+0000 7f12adac7700 1 --2- 172.21.15.23:0/902733004 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1280056d60 0x7f1280059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:55.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.034+0000 7f12ae2c8700 1 --2- 172.21.15.23:0/902733004 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12a81311e0 0x7f12a81315c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f12a400e6e0 tx=0x7f12a4009040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:55.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.035+0000 7f12967fc700 1 -- 172.21.15.23:0/902733004 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a4007320 con 0x7f12a81311e0 2020-11-20T11:38:55.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.035+0000 7f12adac7700 1 --2- 172.21.15.23:0/902733004 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1280056d60 0x7f1280059220 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f1298006e70 tx=0x7f1298015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:55.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.035+0000 7f12967fc700 1 -- 172.21.15.23:0/902733004 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f12a4005500 con 0x7f12a81311e0 2020-11-20T11:38:55.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.035+0000 7f12967fc700 1 -- 172.21.15.23:0/902733004 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12a81311e0 msgr2=0x7f12a81315c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:55.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.035+0000 7f12967fc700 1 --2- 172.21.15.23:0/902733004 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12a81311e0 0x7f12a81315c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f12a400e6e0 tx=0x7f12a4009040).stop 2020-11-20T11:38:55.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.035+0000 7f12967fc700 1 -- 172.21.15.23:0/902733004 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12a81311e0 msgr2=0x7f12a81315c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:38:55.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.035+0000 7f12967fc700 1 --2- 172.21.15.23:0/902733004 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12a81311e0 0x7f12a81315c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:55.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.043+0000 7f12af2ca700 1 --2- 172.21.15.23:0/902733004 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12a8147a30 0x7f12a8147e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:55.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.043+0000 7f12af2ca700 1 -- 172.21.15.23:0/902733004 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a813f550 con 0x7f12a8147a30 2020-11-20T11:38:55.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.043+0000 7f12ae2c8700 1 --2- 172.21.15.23:0/902733004 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12a8147a30 0x7f12a8147e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:55.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.043+0000 7f12ae2c8700 1 -- 172.21.15.23:0/902733004 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f12a80a85e0 con 0x7f12a8147a30 2020-11-20T11:38:55.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.043+0000 7f12ae2c8700 1 --2- 172.21.15.23:0/902733004 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12a8147a30 0x7f12a8147e10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f12a400e6e0 tx=0x7f12a4009040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:55.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.044+0000 7f12967fc700 1 -- 172.21.15.23:0/902733004 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a4014c60 con 0x7f12a8147a30 2020-11-20T11:38:55.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.044+0000 7f12967fc700 1 -- 172.21.15.23:0/902733004 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4274 (secure 0 0 0) 0x7f12a4007110 con 0x7f12a8147a30 2020-11-20T11:38:55.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.044+0000 7f12967fc700 1 -- 172.21.15.23:0/902733004 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12a8147a30 msgr2=0x7f12a8147e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:55.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.044+0000 7f12967fc700 1 --2- 172.21.15.23:0/902733004 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12a8147a30 0x7f12a8147e10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f12a400e6e0 tx=0x7f12a4009040).stop 2020-11-20T11:38:55.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.044+0000 7f12967fc700 1 -- 172.21.15.23:0/902733004 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12a8147a30 msgr2=0x7f12a8147e10 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:38:55.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.044+0000 7f12967fc700 1 --2- 172.21.15.23:0/902733004 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f12a8147a30 0x7f12a8147e10 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0x7f12a400e6e0 tx=0x7f12a4009040).stop 2020-11-20T11:38:55.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.044+0000 7f12af2ca700 1 -- 172.21.15.23:0/902733004 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1280056d60 msgr2=0x7f1280059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:55.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.044+0000 7f12af2ca700 1 --2- 172.21.15.23:0/902733004 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1280056d60 0x7f1280059220 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f1298006e70 tx=0x7f1298015000).stop 2020-11-20T11:38:55.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.044+0000 7f12af2ca700 1 -- 172.21.15.23:0/902733004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12a8135820 msgr2=0x7f12a80a7ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:55.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.044+0000 7f12af2ca700 1 --2- 172.21.15.23:0/902733004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12a8135820 0x7f12a80a7ba0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f12a000ebf0 tx=0x7f12a0008040).stop 2020-11-20T11:38:55.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.044+0000 7f12aeac9700 1 -- 172.21.15.23:0/902733004 reap_dead start 2020-11-20T11:38:55.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.044+0000 7f12aeac9700 1 -- 172.21.15.23:0/902733004 reap_dead start 2020-11-20T11:38:55.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.045+0000 7f12af2ca700 1 -- 172.21.15.23:0/902733004 shutdown_connections 2020-11-20T11:38:55.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.045+0000 7f12af2ca700 1 -- 172.21.15.23:0/902733004 >> 172.21.15.23:0/902733004 conn(0x7f12a8063980 msgr2=0x7f12a8135cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:55.068 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.045+0000 7f12af2ca700 1 -- 172.21.15.23:0/902733004 shutdown_connections 2020-11-20T11:38:55.068 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.045+0000 7f12af2ca700 1 -- 172.21.15.23:0/902733004 wait complete. 2020-11-20T11:38:55.068 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:38:55.068 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:38:55.069 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:38:55.069 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:38:55.069 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 66, 2020-11-20T11:38:55.069 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:38:55.069 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:38:55.070 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:38:55.070 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:38:55.070 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:55.070 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 15, 2020-11-20T11:38:55.070 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:38:55.071 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:38:55.071 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:38:55.071 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:38:55.071 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:38:55.071 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:38:55.072 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:38:55.072 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:38:55.072 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:38:55.072 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:38:55.072 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:38:55.073 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:55.073 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:38:55.073 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:38:55.073 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:38:55.073 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:38:55.074 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:38:55.074 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:38:55.074 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:38:55.074 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:38:55.074 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:38:55.075 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:38:55.075 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:55.075 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:55.075 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:38:55.075 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:38:55.076 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:38:55.076 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:38:55.076 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:38:55.076 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:38:55.076 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:38:55.077 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:38:55.077 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:38:55.077 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:38:55.077 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:38:55.077 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:38:55.078 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:38:55.078 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:38:55.078 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:38:55.078 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:38:55.078 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:38:55.079 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:38:55.079 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:38:55.079 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:38:55.079 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:38:55.079 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:38:55.080 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:38:55.080 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:55.080 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:38:55.081 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:55.081 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:38:55.081 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:55.081 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:38:55.081 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:38:55.082 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:38:55.082 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:38:55.082 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:55.082 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:38:55.082 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:38:55.083 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:55.083 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:55.083 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:55.083 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:38:55.083 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:38:55.084 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:55.084 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:55.084 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:55.084 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:55.084 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:38:55.084 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:38:55.085 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:38:55.085 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:38:55.085 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:38:55.085 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:55.086 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:55.086 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:38:55.086 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:38:55.086 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:38:55.086 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:38:55.086 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:55.087 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:38:55.087 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:38:55.087 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:55.087 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:55.087 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:55.088 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:38:55.088 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:38:55.088 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:55.088 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:55.088 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:55.089 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:55.089 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:38:55.089 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:38:55.089 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:38:55.089 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:38:55.090 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:38:55.090 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:55.090 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:55.090 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:38:55.090 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:38:55.091 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:38:55.091 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:38:55.091 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:55.091 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:38:55.091 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:38:55.092 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:55.092 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:55.092 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:55.092 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:38:55.092 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:38:55.093 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:38:55.093 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:55.093 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:55.093 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:55.093 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:38:55.094 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:38:55.094 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:38:55.094 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:38:55.094 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:38:55.094 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:55.095 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:55.095 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:55.095 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:38:55.095 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:38:55.095 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:55.096 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:38:55.096 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:38:55.096 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:38:55.096 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:55.096 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:38:55.097 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:38:55.097 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:38:55.097 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:38:55.097 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:38:55.097 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:38:55.098 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:38:55.098 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:38:55.098 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:38:55.098 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:38:55.099 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:38:55.099 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2020-11-20T11:38:55.099 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2020-11-20T11:38:55.100 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2020-11-20T11:38:55.100 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2020-11-20T11:38:55.100 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2020-11-20T11:38:55.101 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T11:38:55.111 INFO:tasks.ceph.mon.c:Stopped 2020-11-20T11:38:55.112 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2020-11-20T11:38:55.112 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:38:55.234 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.230+0000 7fdc47489700 1 Processor -- start 2020-11-20T11:38:55.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.230+0000 7fdc47489700 1 -- start start 2020-11-20T11:38:55.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.231+0000 7fdc47489700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fdc400a9040 0x7fdc400aaf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:55.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.231+0000 7fdc47489700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fdc400ace70 0x7fdc4013de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:55.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.231+0000 7fdc47489700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc400ad860 con 0x7fdc400ad2f0 2020-11-20T11:38:55.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.231+0000 7fdc47489700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc400aa8a0 con 0x7fdc400a9040 2020-11-20T11:38:55.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.231+0000 7fdc47489700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc400aaa90 con 0x7fdc400ace70 2020-11-20T11:38:55.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.231+0000 7fdc46487700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fdc400a9040 0x7fdc400aaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:55.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.231+0000 7fdc46c88700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fdc400ad2f0 0x7fdc4013e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:57986/0 (socket says 172.21.15.23:57986) 2020-11-20T11:38:55.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.231+0000 7fdc46c88700 1 -- 172.21.15.23:0/931478162 learned_addr learned my addr 172.21.15.23:0/931478162 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:38:55.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.232+0000 7fdc45c86700 1 --2- 172.21.15.23:0/931478162 >> v2:172.21.15.155:3300/0 conn(0x7fdc400ace70 0x7fdc4013de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:55.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.232+0000 7fdc46487700 1 -- 172.21.15.23:0/931478162 >> v2:172.21.15.155:3300/0 conn(0x7fdc400ace70 msgr2=0x7fdc4013de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:38:55.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.232+0000 7fdc46487700 1 --2- 172.21.15.23:0/931478162 >> v2:172.21.15.155:3300/0 conn(0x7fdc400ace70 0x7fdc4013de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:55.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.232+0000 7fdc46487700 1 -- 172.21.15.23:0/931478162 >> v1:172.21.15.23:6789/0 conn(0x7fdc400ad2f0 legacy=0x7fdc4013e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:55.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.232+0000 7fdc46487700 1 -- 172.21.15.23:0/931478162 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc400a9420 con 0x7fdc400a9040 2020-11-20T11:38:55.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.232+0000 7fdc45485700 1 -- 172.21.15.23:0/931478162 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3774425706 0 0) 0x7fdc400ad860 con 0x7fdc400ad2f0 2020-11-20T11:38:55.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.232+0000 7fdc46487700 1 --2- 172.21.15.23:0/931478162 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc400a9040 0x7fdc400aaf30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fdc400a9610 tx=0x7fdc3c006dd0).ready entity=mon.0 client_cookie=a28b6351501fdc55 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:55.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.232+0000 7fdc45485700 1 -- 172.21.15.23:0/931478162 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c00be90 con 0x7fdc400a9040 2020-11-20T11:38:55.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.232+0000 7fdc45485700 1 -- 172.21.15.23:0/931478162 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc3c006030 con 0x7fdc400a9040 2020-11-20T11:38:55.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.232+0000 7fdc45485700 1 -- 172.21.15.23:0/931478162 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c0063b0 con 0x7fdc400a9040 2020-11-20T11:38:55.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.233+0000 7fdc47489700 1 -- 172.21.15.23:0/931478162 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc400a9040 msgr2=0x7fdc400aaf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:55.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.233+0000 7fdc47489700 1 --2- 172.21.15.23:0/931478162 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc400a9040 0x7fdc400aaf30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fdc400a9610 tx=0x7fdc3c006dd0).stop 2020-11-20T11:38:55.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.233+0000 7fdc47489700 1 -- 172.21.15.23:0/931478162 shutdown_connections 2020-11-20T11:38:55.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.233+0000 7fdc47489700 1 --2- 172.21.15.23:0/931478162 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc400a9040 0x7fdc400aaf30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:55.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.233+0000 7fdc47489700 1 --2- 172.21.15.23:0/931478162 >> v2:172.21.15.155:3300/0 conn(0x7fdc400ace70 0x7fdc4013de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:55.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.233+0000 7fdc47489700 1 -- 172.21.15.23:0/931478162 >> 172.21.15.23:0/931478162 conn(0x7fdc400683b0 msgr2=0x7fdc400b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:55.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.233+0000 7fdc47489700 1 -- 172.21.15.23:0/931478162 shutdown_connections 2020-11-20T11:38:55.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.233+0000 7fdc47489700 1 -- 172.21.15.23:0/931478162 wait complete. 2020-11-20T11:38:55.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.234+0000 7fdc47489700 1 Processor -- start 2020-11-20T11:38:55.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.234+0000 7fdc47489700 1 -- start start 2020-11-20T11:38:55.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.234+0000 7fdc47489700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc400a9040 0x7fdc4009f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:55.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.234+0000 7fdc47489700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc400ace70 0x7fdc4014a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:55.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.234+0000 7fdc47489700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc400ad2f0 0x7fdc4009e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:55.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.234+0000 7fdc47489700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4009ffa0 con 0x7fdc400a9040 2020-11-20T11:38:55.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.234+0000 7fdc47489700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4009ed10 con 0x7fdc400ace70 2020-11-20T11:38:55.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.234+0000 7fdc47489700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc40149b40 con 0x7fdc400ad2f0 2020-11-20T11:38:55.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.234+0000 7fdc46487700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc400a9040 0x7fdc4009f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:55.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.234+0000 7fdc46c88700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc400ad2f0 msgr2=0x7fdc4009e0c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:38:55.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.234+0000 7fdc46c88700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc400ad2f0 0x7fdc4009e0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:38:55.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.235+0000 7fdc46487700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc400a9040 0x7fdc4009f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43738/0 (socket says 172.21.15.23:43738) 2020-11-20T11:38:55.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.235+0000 7fdc46487700 1 -- 172.21.15.23:0/294576456 learned_addr learned my addr 172.21.15.23:0/294576456 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:38:55.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.235+0000 7fdc45c86700 1 --2- 172.21.15.23:0/294576456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc400ace70 0x7fdc4014a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:55.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.235+0000 7fdc46487700 1 -- 172.21.15.23:0/294576456 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc400ad2f0 msgr2=0x7fdc4009e0c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:38:55.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.235+0000 7fdc46487700 1 --2- 172.21.15.23:0/294576456 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc400ad2f0 0x7fdc4009e0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:38:55.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.235+0000 7fdc46487700 1 -- 172.21.15.23:0/294576456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc400ace70 msgr2=0x7fdc4014a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:55.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.235+0000 7fdc46487700 1 --2- 172.21.15.23:0/294576456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc400ace70 0x7fdc4014a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:55.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.235+0000 7fdc46487700 1 -- 172.21.15.23:0/294576456 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc40149de0 con 0x7fdc400a9040 2020-11-20T11:38:55.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.235+0000 7fdc45c86700 1 --2- 172.21.15.23:0/294576456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc400ace70 0x7fdc4014a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:38:55.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.235+0000 7fdc46487700 1 --2- 172.21.15.23:0/294576456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc400a9040 0x7fdc4009f100 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fdc3c006e10 tx=0x7fdc3c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:55.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.235+0000 7fdc2e7fc700 1 -- 172.21.15.23:0/294576456 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c005920 con 0x7fdc400a9040 2020-11-20T11:38:55.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.235+0000 7fdc2e7fc700 1 -- 172.21.15.23:0/294576456 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc3c005a80 con 0x7fdc400a9040 2020-11-20T11:38:55.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.235+0000 7fdc47489700 1 -- 172.21.15.23:0/294576456 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc4014a080 con 0x7fdc400a9040 2020-11-20T11:38:55.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.235+0000 7fdc2e7fc700 1 -- 172.21.15.23:0/294576456 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c005e20 con 0x7fdc400a9040 2020-11-20T11:38:55.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.235+0000 7fdc47489700 1 -- 172.21.15.23:0/294576456 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc4014aac0 con 0x7fdc400a9040 2020-11-20T11:38:55.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.236+0000 7fdc47489700 1 -- 172.21.15.23:0/294576456 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc40057dc0 con 0x7fdc400a9040 2020-11-20T11:38:55.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.236+0000 7fdc2e7fc700 1 -- 172.21.15.23:0/294576456 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fdc3c025030 con 0x7fdc400a9040 2020-11-20T11:38:55.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.237+0000 7fdc2e7fc700 1 --2- 172.21.15.23:0/294576456 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fdc18056d10 0x7fdc180591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:55.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.237+0000 7fdc2e7fc700 1 -- 172.21.15.23:0/294576456 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fdc3c071660 con 0x7fdc400a9040 2020-11-20T11:38:55.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.237+0000 7fdc45c86700 1 --2- 172.21.15.23:0/294576456 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fdc18056d10 0x7fdc180591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:55.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.237+0000 7fdc45c86700 1 --2- 172.21.15.23:0/294576456 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fdc18056d10 0x7fdc180591d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fdc400b0080 tx=0x7fdc30008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:55.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.241+0000 7fdc2e7fc700 1 -- 172.21.15.23:0/294576456 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fdc3c046a00 con 0x7fdc400a9040 2020-11-20T11:38:55.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.515+0000 7fdc47489700 1 -- 172.21.15.23:0/294576456 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fdc40044710 con 0x7fdc400a9040 2020-11-20T11:38:55.520 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.516+0000 7fdc2e7fc700 1 -- 172.21.15.23:0/294576456 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7fdc3c023070 con 0x7fdc400a9040 2020-11-20T11:38:55.520 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:38:55.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.518+0000 7fdc47489700 1 -- 172.21.15.23:0/294576456 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fdc18056d10 msgr2=0x7fdc180591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:55.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.518+0000 7fdc47489700 1 --2- 172.21.15.23:0/294576456 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fdc18056d10 0x7fdc180591d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fdc400b0080 tx=0x7fdc30008040).stop 2020-11-20T11:38:55.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.518+0000 7fdc47489700 1 -- 172.21.15.23:0/294576456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc400a9040 msgr2=0x7fdc4009f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:55.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.518+0000 7fdc47489700 1 --2- 172.21.15.23:0/294576456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc400a9040 0x7fdc4009f100 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fdc3c006e10 tx=0x7fdc3c015000).stop 2020-11-20T11:38:55.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.518+0000 7fdc47489700 1 -- 172.21.15.23:0/294576456 shutdown_connections 2020-11-20T11:38:55.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.518+0000 7fdc47489700 1 --2- 172.21.15.23:0/294576456 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fdc18056d10 0x7fdc180591d0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:55.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.518+0000 7fdc47489700 1 --2- 172.21.15.23:0/294576456 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdc400ad2f0 0x7fdc4009e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:38:55.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.518+0000 7fdc47489700 1 --2- 172.21.15.23:0/294576456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdc400a9040 0x7fdc4009f100 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:55.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.518+0000 7fdc47489700 1 --2- 172.21.15.23:0/294576456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdc400ace70 0x7fdc4014a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:55.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.518+0000 7fdc47489700 1 -- 172.21.15.23:0/294576456 >> 172.21.15.23:0/294576456 conn(0x7fdc400683b0 msgr2=0x7fdc401375a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:55.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.518+0000 7fdc47489700 1 -- 172.21.15.23:0/294576456 shutdown_connections 2020-11-20T11:38:55.525 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:55.518+0000 7fdc47489700 1 -- 172.21.15.23:0/294576456 wait complete. 2020-11-20T11:38:58.541 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:38:58.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.664+0000 7f15697ba700 1 Processor -- start 2020-11-20T11:38:58.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.665+0000 7f15697ba700 1 -- start start 2020-11-20T11:38:58.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.665+0000 7f15697ba700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1564065780 0x7f15640a9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:58.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.665+0000 7f15697ba700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1564067430 con 0x7f15640a9cf0 2020-11-20T11:38:58.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.665+0000 7f15697ba700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1564138800 con 0x7f156405bc80 2020-11-20T11:38:58.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.665+0000 7f15697ba700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15640a8ca0 con 0x7f1564065780 2020-11-20T11:38:58.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.665+0000 7f15627fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1564065780 0x7f15640a9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:58.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.665+0000 7f15637fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f15640a9cf0 0x7f15641380f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58008/0 (socket says 172.21.15.23:58008) 2020-11-20T11:38:58.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.665+0000 7f15637fe700 1 -- 172.21.15.23:0/807332491 learned_addr learned my addr 172.21.15.23:0/807332491 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:38:58.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.666+0000 7f15627fc700 1 -- 172.21.15.23:0/807332491 >> v1:172.21.15.155:6789/0 conn(0x7f156405bc80 legacy=0x7f15641379e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:58.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.666+0000 7f15627fc700 1 -- 172.21.15.23:0/807332491 >> v1:172.21.15.23:6789/0 conn(0x7f15640a9cf0 legacy=0x7f15641380f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:58.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.666+0000 7f15627fc700 1 -- 172.21.15.23:0/807332491 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f156405c060 con 0x7f1564065780 2020-11-20T11:38:58.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.666+0000 7f15627fc700 1 --2- 172.21.15.23:0/807332491 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1564065780 0x7f15640a9140 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f1558006fd0 tx=0x7f1558006e30).ready entity=mon.0 client_cookie=7ac52a1461886d9a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:58.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.666+0000 7f1561ffb700 1 -- 172.21.15.23:0/807332491 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15580097e0 con 0x7f1564065780 2020-11-20T11:38:58.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.666+0000 7f1561ffb700 1 -- 172.21.15.23:0/807332491 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1558000e90 con 0x7f1564065780 2020-11-20T11:38:58.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.666+0000 7f1561ffb700 1 -- 172.21.15.23:0/807332491 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1558009b50 con 0x7f1564065780 2020-11-20T11:38:58.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.667+0000 7f15697ba700 1 -- 172.21.15.23:0/807332491 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1564065780 msgr2=0x7f15640a9140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:58.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.667+0000 7f15697ba700 1 --2- 172.21.15.23:0/807332491 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1564065780 0x7f15640a9140 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f1558006fd0 tx=0x7f1558006e30).stop 2020-11-20T11:38:58.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.667+0000 7f15697ba700 1 -- 172.21.15.23:0/807332491 shutdown_connections 2020-11-20T11:38:58.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.667+0000 7f15697ba700 1 --2- 172.21.15.23:0/807332491 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1564065780 0x7f15640a9140 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:58.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.667+0000 7f15697ba700 1 -- 172.21.15.23:0/807332491 >> 172.21.15.23:0/807332491 conn(0x7f15640ad7a0 msgr2=0x7f15640adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:58.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.667+0000 7f15697ba700 1 -- 172.21.15.23:0/807332491 shutdown_connections 2020-11-20T11:38:58.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.667+0000 7f15697ba700 1 -- 172.21.15.23:0/807332491 wait complete. 2020-11-20T11:38:58.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.668+0000 7f15697ba700 1 Processor -- start 2020-11-20T11:38:58.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.668+0000 7f15697ba700 1 -- start start 2020-11-20T11:38:58.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.668+0000 7f15697ba700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f156405bc80 0x7f15640a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:58.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.668+0000 7f15697ba700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1564065780 0x7f15640a70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:58.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.668+0000 7f15697ba700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f15640a9cf0 0x7f15640a4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:58.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.668+0000 7f15697ba700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15640a2c70 con 0x7f156405bc80 2020-11-20T11:38:58.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.668+0000 7f15697ba700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15640a3f60 con 0x7f1564065780 2020-11-20T11:38:58.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.668+0000 7f15697ba700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15640a41c0 con 0x7f15640a9cf0 2020-11-20T11:38:58.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.669+0000 7f1562ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f156405bc80 0x7f15640a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:58.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.669+0000 7f15637fe700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f15640a9cf0 msgr2=0x7f15640a4660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:38:58.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.669+0000 7f15637fe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f15640a9cf0 0x7f15640a4660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:38:58.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.669+0000 7f1562ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f156405bc80 0x7f15640a5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43760/0 (socket says 172.21.15.23:43760) 2020-11-20T11:38:58.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.669+0000 7f1562ffd700 1 -- 172.21.15.23:0/954371053 learned_addr learned my addr 172.21.15.23:0/954371053 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:38:58.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.669+0000 7f15627fc700 1 --2- 172.21.15.23:0/954371053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1564065780 0x7f15640a70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:58.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.669+0000 7f1562ffd700 1 -- 172.21.15.23:0/954371053 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f15640a9cf0 msgr2=0x7f15640a4660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:38:58.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.669+0000 7f1562ffd700 1 --2- 172.21.15.23:0/954371053 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f15640a9cf0 0x7f15640a4660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:38:58.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.669+0000 7f1562ffd700 1 -- 172.21.15.23:0/954371053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1564065780 msgr2=0x7f15640a70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:58.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.669+0000 7f1562ffd700 1 --2- 172.21.15.23:0/954371053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1564065780 0x7f15640a70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:58.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.669+0000 7f1562ffd700 1 -- 172.21.15.23:0/954371053 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15640a4b50 con 0x7f156405bc80 2020-11-20T11:38:58.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.669+0000 7f15627fc700 1 --2- 172.21.15.23:0/954371053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1564065780 0x7f15640a70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:38:58.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.669+0000 7f1562ffd700 1 --2- 172.21.15.23:0/954371053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f156405bc80 0x7f15640a5430 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f155400c710 tx=0x7f1554007c90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:58.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.669+0000 7f154affd700 1 -- 172.21.15.23:0/954371053 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1554007020 con 0x7f156405bc80 2020-11-20T11:38:58.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.669+0000 7f154affd700 1 -- 172.21.15.23:0/954371053 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1554001e90 con 0x7f156405bc80 2020-11-20T11:38:58.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.669+0000 7f15697ba700 1 -- 172.21.15.23:0/954371053 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15640a3370 con 0x7f156405bc80 2020-11-20T11:38:58.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.669+0000 7f154affd700 1 -- 172.21.15.23:0/954371053 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15540073b0 con 0x7f156405bc80 2020-11-20T11:38:58.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.669+0000 7f15697ba700 1 -- 172.21.15.23:0/954371053 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15640a78b0 con 0x7f156405bc80 2020-11-20T11:38:58.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.670+0000 7f15697ba700 1 -- 172.21.15.23:0/954371053 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1564062fe0 con 0x7f156405bc80 2020-11-20T11:38:58.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.671+0000 7f154affd700 1 -- 172.21.15.23:0/954371053 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1554010070 con 0x7f156405bc80 2020-11-20T11:38:58.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.671+0000 7f154affd700 1 --2- 172.21.15.23:0/954371053 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1540056d10 0x7f15400591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:38:58.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.671+0000 7f154affd700 1 -- 172.21.15.23:0/954371053 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f155406f440 con 0x7f156405bc80 2020-11-20T11:38:58.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.672+0000 7f15627fc700 1 --2- 172.21.15.23:0/954371053 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1540056d10 0x7f15400591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:38:58.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.672+0000 7f15627fc700 1 --2- 172.21.15.23:0/954371053 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1540056d10 0x7f15400591d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f1558006bc0 tx=0x7f1558015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:38:58.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.676+0000 7f154affd700 1 -- 172.21.15.23:0/954371053 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1554044730 con 0x7f156405bc80 2020-11-20T11:38:58.917 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.914+0000 7f15697ba700 1 -- 172.21.15.23:0/954371053 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f15640a7a40 con 0x7f156405bc80 2020-11-20T11:38:59.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.915+0000 7f154affd700 1 -- 172.21.15.23:0/954371053 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7f1554047d50 con 0x7f156405bc80 2020-11-20T11:38:59.144 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:38:59.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.916+0000 7f15697ba700 1 -- 172.21.15.23:0/954371053 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1540056d10 msgr2=0x7f15400591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:59.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.916+0000 7f15697ba700 1 --2- 172.21.15.23:0/954371053 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1540056d10 0x7f15400591d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f1558006bc0 tx=0x7f1558015000).stop 2020-11-20T11:38:59.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.916+0000 7f15697ba700 1 -- 172.21.15.23:0/954371053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f156405bc80 msgr2=0x7f15640a5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:38:59.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.916+0000 7f15697ba700 1 --2- 172.21.15.23:0/954371053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f156405bc80 0x7f15640a5430 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f155400c710 tx=0x7f1554007c90).stop 2020-11-20T11:38:59.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.916+0000 7f15697ba700 1 -- 172.21.15.23:0/954371053 shutdown_connections 2020-11-20T11:38:59.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.916+0000 7f15697ba700 1 --2- 172.21.15.23:0/954371053 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1540056d10 0x7f15400591d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:59.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.916+0000 7f15697ba700 1 --2- 172.21.15.23:0/954371053 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f15640a9cf0 0x7f15640a4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:38:59.148 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.917+0000 7f15697ba700 1 --2- 172.21.15.23:0/954371053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f156405bc80 0x7f15640a5430 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:59.148 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.917+0000 7f15697ba700 1 --2- 172.21.15.23:0/954371053 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1564065780 0x7f15640a70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:38:59.149 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.917+0000 7f15697ba700 1 -- 172.21.15.23:0/954371053 >> 172.21.15.23:0/954371053 conn(0x7f15640ad7a0 msgr2=0x7f15640601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:38:59.149 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.917+0000 7f15697ba700 1 -- 172.21.15.23:0/954371053 shutdown_connections 2020-11-20T11:38:59.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:38:58.917+0000 7f15697ba700 1 -- 172.21.15.23:0/954371053 wait complete. 2020-11-20T11:39:00.788 INFO:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:39:00.791 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:39:02.151 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:39:02.288 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.285+0000 7f5e79c2f700 1 Processor -- start 2020-11-20T11:39:02.289 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.285+0000 7f5e79c2f700 1 -- start start 2020-11-20T11:39:02.290 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.286+0000 7f5e79c2f700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5e74066210 0x7f5e7413adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:02.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.286+0000 7f5e79c2f700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e740665f0 0x7f5e740ab2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:02.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.286+0000 7f5e79c2f700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e7413bd10 con 0x7f5e740a8ff0 2020-11-20T11:39:02.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.286+0000 7f5e79c2f700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e740aab60 con 0x7f5e74066210 2020-11-20T11:39:02.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.286+0000 7f5e79c2f700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e740aae00 con 0x7f5e740665f0 2020-11-20T11:39:02.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.286+0000 7f5e73fff700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e740665f0 msgr2=0x7f5e740ab2a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:39:02.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.286+0000 7f5e7942e700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f5e740a8ff0 0x7f5e74137f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58026/0 (socket says 172.21.15.23:58026) 2020-11-20T11:39:02.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.286+0000 7f5e78c2d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5e74066210 0x7f5e7413adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:02.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.286+0000 7f5e7942e700 1 -- 172.21.15.23:0/3849713180 learned_addr learned my addr 172.21.15.23:0/3849713180 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:39:02.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.286+0000 7f5e73fff700 1 --2- 172.21.15.23:0/3849713180 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e740665f0 0x7f5e740ab2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:39:02.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.286+0000 7f5e78c2d700 1 -- 172.21.15.23:0/3849713180 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e740665f0 msgr2=0x7f5e740ab2a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:39:02.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.286+0000 7f5e78c2d700 1 --2- 172.21.15.23:0/3849713180 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e740665f0 0x7f5e740ab2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:39:02.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.286+0000 7f5e78c2d700 1 -- 172.21.15.23:0/3849713180 >> v1:172.21.15.23:6789/0 conn(0x7f5e740a8ff0 legacy=0x7f5e74137f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:02.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.286+0000 7f5e78c2d700 1 -- 172.21.15.23:0/3849713180 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e740aa930 con 0x7f5e74066210 2020-11-20T11:39:02.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.287+0000 7f5e78c2d700 1 --2- 172.21.15.23:0/3849713180 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e74066210 0x7f5e7413adb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f5e68006fd0 tx=0x7f5e68006e30).ready entity=mon.0 client_cookie=ddae0fa92657b760 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:02.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.287+0000 7f5e737fe700 1 -- 172.21.15.23:0/3849713180 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e680097e0 con 0x7f5e74066210 2020-11-20T11:39:02.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.287+0000 7f5e737fe700 1 -- 172.21.15.23:0/3849713180 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e68000e90 con 0x7f5e74066210 2020-11-20T11:39:02.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.287+0000 7f5e737fe700 1 -- 172.21.15.23:0/3849713180 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e68009b50 con 0x7f5e74066210 2020-11-20T11:39:02.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.287+0000 7f5e79c2f700 1 -- 172.21.15.23:0/3849713180 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e74066210 msgr2=0x7f5e7413adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:02.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.287+0000 7f5e79c2f700 1 --2- 172.21.15.23:0/3849713180 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e74066210 0x7f5e7413adb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f5e68006fd0 tx=0x7f5e68006e30).stop 2020-11-20T11:39:02.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.287+0000 7f5e79c2f700 1 -- 172.21.15.23:0/3849713180 shutdown_connections 2020-11-20T11:39:02.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.287+0000 7f5e79c2f700 1 --2- 172.21.15.23:0/3849713180 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e74066210 0x7f5e7413adb0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:02.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.287+0000 7f5e79c2f700 1 --2- 172.21.15.23:0/3849713180 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e740665f0 0x7f5e740ab2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:39:02.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.287+0000 7f5e79c2f700 1 -- 172.21.15.23:0/3849713180 >> 172.21.15.23:0/3849713180 conn(0x7f5e740612c0 msgr2=0x7f5e740616d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:02.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.287+0000 7f5e79c2f700 1 -- 172.21.15.23:0/3849713180 shutdown_connections 2020-11-20T11:39:02.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.288+0000 7f5e79c2f700 1 -- 172.21.15.23:0/3849713180 wait complete. 2020-11-20T11:39:02.298 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.289+0000 7f5e79c2f700 1 Processor -- start 2020-11-20T11:39:02.298 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.289+0000 7f5e79c2f700 1 -- start start 2020-11-20T11:39:02.299 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.290+0000 7f5e79c2f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e74066210 0x7f5e7413f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:02.299 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.290+0000 7f5e79c2f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e740665f0 0x7f5e7413aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:02.299 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.290+0000 7f5e79c2f700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e740a8ff0 0x7f5e7414d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:02.299 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.290+0000 7f5e79c2f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e740a48f0 con 0x7f5e74066210 2020-11-20T11:39:02.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.290+0000 7f5e79c2f700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e740a4b90 con 0x7f5e740665f0 2020-11-20T11:39:02.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.290+0000 7f5e79c2f700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7413fcd0 con 0x7f5e740a8ff0 2020-11-20T11:39:02.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.290+0000 7f5e78c2d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e74066210 0x7f5e7413f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:02.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.290+0000 7f5e78c2d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e74066210 0x7f5e7413f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43778/0 (socket says 172.21.15.23:43778) 2020-11-20T11:39:02.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.290+0000 7f5e7942e700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e740a8ff0 msgr2=0x7f5e7414d9f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:39:02.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.290+0000 7f5e78c2d700 1 -- 172.21.15.23:0/855312916 learned_addr learned my addr 172.21.15.23:0/855312916 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:39:02.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.290+0000 7f5e7942e700 1 --2- 172.21.15.23:0/855312916 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e740a8ff0 0x7f5e7414d9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:39:02.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.290+0000 7f5e73fff700 1 --2- 172.21.15.23:0/855312916 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e740665f0 0x7f5e7413aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:02.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.290+0000 7f5e78c2d700 1 -- 172.21.15.23:0/855312916 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e740a8ff0 msgr2=0x7f5e7414d9f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:39:02.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.290+0000 7f5e78c2d700 1 --2- 172.21.15.23:0/855312916 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e740a8ff0 0x7f5e7414d9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:39:02.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.290+0000 7f5e78c2d700 1 -- 172.21.15.23:0/855312916 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e740665f0 msgr2=0x7f5e7413aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:02.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.290+0000 7f5e78c2d700 1 --2- 172.21.15.23:0/855312916 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e740665f0 0x7f5e7413aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:02.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.290+0000 7f5e78c2d700 1 -- 172.21.15.23:0/855312916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e7413afc0 con 0x7f5e74066210 2020-11-20T11:39:02.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.290+0000 7f5e78c2d700 1 --2- 172.21.15.23:0/855312916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e74066210 0x7f5e7413f4a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f5e68006e70 tx=0x7f5e68015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:02.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.290+0000 7f5e70ff9700 1 -- 172.21.15.23:0/855312916 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e68011d40 con 0x7f5e74066210 2020-11-20T11:39:02.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.290+0000 7f5e70ff9700 1 -- 172.21.15.23:0/855312916 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e6801e460 con 0x7f5e74066210 2020-11-20T11:39:02.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.290+0000 7f5e79c2f700 1 -- 172.21.15.23:0/855312916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e740a3d10 con 0x7f5e74066210 2020-11-20T11:39:02.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.290+0000 7f5e70ff9700 1 -- 172.21.15.23:0/855312916 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e68011ea0 con 0x7f5e74066210 2020-11-20T11:39:02.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.291+0000 7f5e79c2f700 1 -- 172.21.15.23:0/855312916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e740a1ee0 con 0x7f5e74066210 2020-11-20T11:39:02.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.291+0000 7f5e79c2f700 1 -- 172.21.15.23:0/855312916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e74062760 con 0x7f5e74066210 2020-11-20T11:39:02.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.292+0000 7f5e70ff9700 1 -- 172.21.15.23:0/855312916 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5e68025030 con 0x7f5e74066210 2020-11-20T11:39:02.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.292+0000 7f5e70ff9700 1 --2- 172.21.15.23:0/855312916 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5e4c056d10 0x7f5e4c0591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:02.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.292+0000 7f5e70ff9700 1 -- 172.21.15.23:0/855312916 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f5e68071630 con 0x7f5e74066210 2020-11-20T11:39:02.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.292+0000 7f5e73fff700 1 --2- 172.21.15.23:0/855312916 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5e4c056d10 0x7f5e4c0591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:02.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.293+0000 7f5e73fff700 1 --2- 172.21.15.23:0/855312916 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5e4c056d10 0x7f5e4c0591d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f5e60006fd0 tx=0x7f5e60006e10).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:02.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.298+0000 7f5e70ff9700 1 -- 172.21.15.23:0/855312916 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5e680469d0 con 0x7f5e74066210 2020-11-20T11:39:02.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.564+0000 7f5e79c2f700 1 -- 172.21.15.23:0/855312916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5e740a4720 con 0x7f5e74066210 2020-11-20T11:39:02.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.564+0000 7f5e70ff9700 1 -- 172.21.15.23:0/855312916 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7f5e68023070 con 0x7f5e74066210 2020-11-20T11:39:02.568 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:39:02.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.566+0000 7f5e79c2f700 1 -- 172.21.15.23:0/855312916 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5e4c056d10 msgr2=0x7f5e4c0591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:02.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.566+0000 7f5e79c2f700 1 --2- 172.21.15.23:0/855312916 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5e4c056d10 0x7f5e4c0591d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f5e60006fd0 tx=0x7f5e60006e10).stop 2020-11-20T11:39:02.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.566+0000 7f5e79c2f700 1 -- 172.21.15.23:0/855312916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e74066210 msgr2=0x7f5e7413f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:02.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.566+0000 7f5e79c2f700 1 --2- 172.21.15.23:0/855312916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e74066210 0x7f5e7413f4a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f5e68006e70 tx=0x7f5e68015000).stop 2020-11-20T11:39:02.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.566+0000 7f5e79c2f700 1 -- 172.21.15.23:0/855312916 shutdown_connections 2020-11-20T11:39:02.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.566+0000 7f5e79c2f700 1 --2- 172.21.15.23:0/855312916 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5e4c056d10 0x7f5e4c0591d0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:02.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.566+0000 7f5e79c2f700 1 --2- 172.21.15.23:0/855312916 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e740a8ff0 0x7f5e7414d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:39:02.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.566+0000 7f5e79c2f700 1 --2- 172.21.15.23:0/855312916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e74066210 0x7f5e7413f4a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:02.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.566+0000 7f5e79c2f700 1 --2- 172.21.15.23:0/855312916 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e740665f0 0x7f5e7413aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:02.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.566+0000 7f5e79c2f700 1 -- 172.21.15.23:0/855312916 >> 172.21.15.23:0/855312916 conn(0x7f5e740612c0 msgr2=0x7f5e740ade90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:02.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.566+0000 7f5e79c2f700 1 -- 172.21.15.23:0/855312916 shutdown_connections 2020-11-20T11:39:02.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:02.566+0000 7f5e79c2f700 1 -- 172.21.15.23:0/855312916 wait complete. 2020-11-20T11:39:05.583 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:39:05.700 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.697+0000 7f82eb59e700 1 Processor -- start 2020-11-20T11:39:05.700 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.697+0000 7f82eb59e700 1 -- start start 2020-11-20T11:39:05.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.698+0000 7f82eb59e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82ec0a9040 0x7f82ec0aaf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:05.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.698+0000 7f82eb59e700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f82ec0ace70 0x7f82ec13de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:05.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.698+0000 7f82eb59e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f82ec0ad2f0 0x7f82ec13e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:05.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.698+0000 7f82eb59e700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82ec0aa8a0 con 0x7f82ec0ace70 2020-11-20T11:39:05.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.698+0000 7f82eb59e700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82ec0ad860 con 0x7f82ec0ad2f0 2020-11-20T11:39:05.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.698+0000 7f82eb59e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec0aaa90 con 0x7f82ec0a9040 2020-11-20T11:39:05.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.698+0000 7f82e9d9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f82ec0ace70 0x7f82ec13de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:05.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.698+0000 7f82ea59c700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82ec0a9040 msgr2=0x7f82ec0aaf30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:39:05.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.698+0000 7f82e9d9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f82ec0ace70 0x7f82ec13de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43788/0 (socket says 172.21.15.23:43788) 2020-11-20T11:39:05.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.698+0000 7f82ea59c700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82ec0a9040 0x7f82ec0aaf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:39:05.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.698+0000 7f82e9d9b700 1 -- 172.21.15.23:0/1685927100 learned_addr learned my addr 172.21.15.23:0/1685927100 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:39:05.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.698+0000 7f82e9d9b700 1 -- 172.21.15.23:0/1685927100 >> v2:172.21.15.23:3300/0 conn(0x7f82ec0ace70 msgr2=0x7f82ec13de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:39:05.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.698+0000 7f82e9d9b700 1 -- 172.21.15.23:0/1685927100 >> v2:172.21.15.23:3300/0 conn(0x7f82ec0ace70 msgr2=0x7f82ec13de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:39:05.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.698+0000 7f82e9d9b700 1 --2- 172.21.15.23:0/1685927100 >> v2:172.21.15.23:3300/0 conn(0x7f82ec0ace70 0x7f82ec13de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:39:05.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.698+0000 7f82e9d9b700 1 --2- 172.21.15.23:0/1685927100 >> v2:172.21.15.23:3300/0 conn(0x7f82ec0ace70 0x7f82ec13de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:39:05.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.698+0000 7f82ead9d700 1 --2- 172.21.15.23:0/1685927100 >> v2:172.21.15.155:3300/0 conn(0x7f82ec0ad2f0 0x7f82ec13e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:05.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.698+0000 7f82ead9d700 1 -- 172.21.15.23:0/1685927100 >> v2:172.21.15.155:3300/0 conn(0x7f82ec0ad2f0 msgr2=0x7f82ec13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:39:05.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.698+0000 7f82ead9d700 1 -- 172.21.15.23:0/1685927100 >> v2:172.21.15.155:3300/0 conn(0x7f82ec0ad2f0 msgr2=0x7f82ec13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:39:05.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.698+0000 7f82ead9d700 1 --2- 172.21.15.23:0/1685927100 >> v2:172.21.15.155:3300/0 conn(0x7f82ec0ad2f0 0x7f82ec13e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:39:05.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.698+0000 7f82ead9d700 1 --2- 172.21.15.23:0/1685927100 >> v2:172.21.15.155:3300/0 conn(0x7f82ec0ad2f0 0x7f82ec13e2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:39:05.902 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.899+0000 7f82e9d9b700 1 --2- 172.21.15.23:0/1685927100 >> v2:172.21.15.23:3300/0 conn(0x7f82ec0ace70 0x7f82ec13de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:05.903 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.899+0000 7f82ea59c700 1 -- 172.21.15.23:0/1685927100 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82ec0a9040 msgr2=0x7f82ec0aaf30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:39:05.903 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.899+0000 7f82ea59c700 1 --2- 172.21.15.23:0/1685927100 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82ec0a9040 0x7f82ec0aaf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:39:05.903 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.899+0000 7f82e9d9b700 1 -- 172.21.15.23:0/1685927100 >> v2:172.21.15.23:3300/0 conn(0x7f82ec0ace70 msgr2=0x7f82ec13de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:39:05.903 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.899+0000 7f82e9d9b700 1 -- 172.21.15.23:0/1685927100 >> v2:172.21.15.23:3300/0 conn(0x7f82ec0ace70 msgr2=0x7f82ec13de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:39:05.904 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.899+0000 7f82e9d9b700 1 --2- 172.21.15.23:0/1685927100 >> v2:172.21.15.23:3300/0 conn(0x7f82ec0ace70 0x7f82ec13de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:39:05.904 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.899+0000 7f82ead9d700 1 --2- 172.21.15.23:0/1685927100 >> v2:172.21.15.155:3300/0 conn(0x7f82ec0ad2f0 0x7f82ec13e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:05.904 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.899+0000 7f82e9d9b700 1 --2- 172.21.15.23:0/1685927100 >> v2:172.21.15.23:3300/0 conn(0x7f82ec0ace70 0x7f82ec13de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:39:05.904 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.899+0000 7f82ead9d700 1 -- 172.21.15.23:0/1685927100 >> v2:172.21.15.155:3300/0 conn(0x7f82ec0ad2f0 msgr2=0x7f82ec13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:39:05.904 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.899+0000 7f82ead9d700 1 -- 172.21.15.23:0/1685927100 >> v2:172.21.15.155:3300/0 conn(0x7f82ec0ad2f0 msgr2=0x7f82ec13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:39:05.905 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.899+0000 7f82ead9d700 1 --2- 172.21.15.23:0/1685927100 >> v2:172.21.15.155:3300/0 conn(0x7f82ec0ad2f0 0x7f82ec13e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:39:05.905 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:05.899+0000 7f82ead9d700 1 --2- 172.21.15.23:0/1685927100 >> v2:172.21.15.155:3300/0 conn(0x7f82ec0ad2f0 0x7f82ec13e2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:39:06.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:06.300+0000 7f82e9d9b700 1 --2- 172.21.15.23:0/1685927100 >> v2:172.21.15.23:3300/0 conn(0x7f82ec0ace70 0x7f82ec13de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:06.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:06.300+0000 7f82ea59c700 1 -- 172.21.15.23:0/1685927100 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82ec0a9040 msgr2=0x7f82ec0aaf30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:39:06.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:06.300+0000 7f82ea59c700 1 --2- 172.21.15.23:0/1685927100 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82ec0a9040 0x7f82ec0aaf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:39:06.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:06.300+0000 7f82e9d9b700 1 -- 172.21.15.23:0/1685927100 >> v2:172.21.15.23:3300/0 conn(0x7f82ec0ace70 msgr2=0x7f82ec13de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:39:06.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:06.300+0000 7f82e9d9b700 1 -- 172.21.15.23:0/1685927100 >> v2:172.21.15.23:3300/0 conn(0x7f82ec0ace70 msgr2=0x7f82ec13de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:39:06.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:06.300+0000 7f82e9d9b700 1 --2- 172.21.15.23:0/1685927100 >> v2:172.21.15.23:3300/0 conn(0x7f82ec0ace70 0x7f82ec13de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:39:06.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:06.300+0000 7f82e9d9b700 1 --2- 172.21.15.23:0/1685927100 >> v2:172.21.15.23:3300/0 conn(0x7f82ec0ace70 0x7f82ec13de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:39:06.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:06.300+0000 7f82ead9d700 1 --2- 172.21.15.23:0/1685927100 >> v2:172.21.15.155:3300/0 conn(0x7f82ec0ad2f0 0x7f82ec13e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:06.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:06.300+0000 7f82ead9d700 1 -- 172.21.15.23:0/1685927100 >> v2:172.21.15.155:3300/0 conn(0x7f82ec0ad2f0 msgr2=0x7f82ec13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:39:06.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:06.300+0000 7f82ead9d700 1 -- 172.21.15.23:0/1685927100 >> v2:172.21.15.155:3300/0 conn(0x7f82ec0ad2f0 msgr2=0x7f82ec13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:39:06.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:06.300+0000 7f82ead9d700 1 --2- 172.21.15.23:0/1685927100 >> v2:172.21.15.155:3300/0 conn(0x7f82ec0ad2f0 0x7f82ec13e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:39:06.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:06.300+0000 7f82ead9d700 1 --2- 172.21.15.23:0/1685927100 >> v2:172.21.15.155:3300/0 conn(0x7f82ec0ad2f0 0x7f82ec13e2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:39:07.104 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:07.101+0000 7f82ea59c700 1 -- 172.21.15.23:0/1685927100 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82ec0a9040 msgr2=0x7f82ec0aaf30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:39:07.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:07.101+0000 7f82ea59c700 1 --2- 172.21.15.23:0/1685927100 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82ec0a9040 0x7f82ec0aaf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:39:07.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:07.101+0000 7f82e9d9b700 1 --2- 172.21.15.23:0/1685927100 >> v2:172.21.15.23:3300/0 conn(0x7f82ec0ace70 0x7f82ec13de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:07.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:07.101+0000 7f82ead9d700 1 --2- 172.21.15.23:0/1685927100 >> v2:172.21.15.155:3300/0 conn(0x7f82ec0ad2f0 0x7f82ec13e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:07.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:07.101+0000 7f82e9d9b700 1 -- 172.21.15.23:0/1685927100 >> v2:172.21.15.23:3300/0 conn(0x7f82ec0ace70 msgr2=0x7f82ec13de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:39:07.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:07.101+0000 7f82e9d9b700 1 -- 172.21.15.23:0/1685927100 >> v2:172.21.15.23:3300/0 conn(0x7f82ec0ace70 msgr2=0x7f82ec13de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:39:07.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:07.101+0000 7f82e9d9b700 1 --2- 172.21.15.23:0/1685927100 >> v2:172.21.15.23:3300/0 conn(0x7f82ec0ace70 0x7f82ec13de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:39:07.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:07.101+0000 7f82e9d9b700 1 --2- 172.21.15.23:0/1685927100 >> v2:172.21.15.23:3300/0 conn(0x7f82ec0ace70 0x7f82ec13de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:39:07.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:07.101+0000 7f82ead9d700 1 -- 172.21.15.23:0/1685927100 >> v2:172.21.15.155:3300/0 conn(0x7f82ec0ad2f0 msgr2=0x7f82ec13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:39:07.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:07.101+0000 7f82ead9d700 1 -- 172.21.15.23:0/1685927100 >> v2:172.21.15.155:3300/0 conn(0x7f82ec0ad2f0 msgr2=0x7f82ec13e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:39:07.107 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:07.101+0000 7f82ead9d700 1 --2- 172.21.15.23:0/1685927100 >> v2:172.21.15.155:3300/0 conn(0x7f82ec0ad2f0 0x7f82ec13e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:39:07.107 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:07.101+0000 7f82ead9d700 1 --2- 172.21.15.23:0/1685927100 >> v2:172.21.15.155:3300/0 conn(0x7f82ec0ad2f0 0x7f82ec13e2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:39:08.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:08.698+0000 7f82dbfff700 1 -- 172.21.15.23:0/1685927100 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82ec0a9040 msgr2=0x7f82ec0aaf30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:39:08.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:08.698+0000 7f82dbfff700 1 --2- 172.21.15.23:0/1685927100 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82ec0a9040 0x7f82ec0aaf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:39:08.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:08.698+0000 7f82dbfff700 1 -- 172.21.15.23:0/1685927100 >> v2:172.21.15.155:3300/0 conn(0x7f82ec0ad2f0 msgr2=0x7f82ec13e2f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:39:08.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:08.698+0000 7f82dbfff700 1 --2- 172.21.15.23:0/1685927100 >> v2:172.21.15.155:3300/0 conn(0x7f82ec0ad2f0 0x7f82ec13e2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:08.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:08.698+0000 7f82dbfff700 1 -- 172.21.15.23:0/1685927100 >> v2:172.21.15.23:3300/0 conn(0x7f82ec0ace70 msgr2=0x7f82ec13de00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:39:08.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:08.698+0000 7f82dbfff700 1 --2- 172.21.15.23:0/1685927100 >> v2:172.21.15.23:3300/0 conn(0x7f82ec0ace70 0x7f82ec13de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:08.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:08.698+0000 7f82dbfff700 1 -- 172.21.15.23:0/1685927100 --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82ec0a9420 con 0x7f82c8007050 2020-11-20T11:39:08.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:08.698+0000 7f82dbfff700 1 -- 172.21.15.23:0/1685927100 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c800bdc0 con 0x7f82c80033c0 2020-11-20T11:39:09.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.677+0000 7f82e959a700 1 -- 172.21.15.23:0/1685927100 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1783915056 0 0) 0x7f82ec0a9420 con 0x7f82c8007050 2020-11-20T11:39:09.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.677+0000 7f82e959a700 1 -- 172.21.15.23:0/1685927100 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82d0003280 con 0x7f82c8007050 2020-11-20T11:39:09.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.678+0000 7f82e959a700 1 -- 172.21.15.23:0/1685927100 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1507326676 0 0) 0x7f82d0003280 con 0x7f82c8007050 2020-11-20T11:39:09.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.678+0000 7f82e959a700 1 -- 172.21.15.23:0/1685927100 >> v1:172.21.15.155:6789/0 conn(0x7f82c80033c0 legacy=0x7f82c8005820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:09.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.678+0000 7f82e959a700 1 -- 172.21.15.23:0/1685927100 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c800cfc0 con 0x7f82c8007050 2020-11-20T11:39:09.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.678+0000 7f82e959a700 1 -- 172.21.15.23:0/1685927100 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f82e000d3b0 con 0x7f82c8007050 2020-11-20T11:39:09.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.678+0000 7f82e959a700 1 -- 172.21.15.23:0/1685927100 >> v1:172.21.15.23:6789/0 conn(0x7f82c8007050 legacy=0x7f82c8009510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:09.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.678+0000 7f82e959a700 1 --2- 172.21.15.23:0/1685927100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82d00037a0 0x7f82d0003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:09.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.678+0000 7f82e959a700 1 --2- 172.21.15.23:0/1685927100 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82d00040b0 0x7f82d0004580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:09.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.678+0000 7f82e959a700 1 --2- 172.21.15.23:0/1685927100 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82ec0a9040 0x7f82d0004ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:09.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.678+0000 7f82e959a700 1 -- 172.21.15.23:0/1685927100 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d00051b0 con 0x7f82d00037a0 2020-11-20T11:39:09.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.678+0000 7f82e959a700 1 -- 172.21.15.23:0/1685927100 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d0005320 con 0x7f82d00040b0 2020-11-20T11:39:09.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.678+0000 7f82e959a700 1 -- 172.21.15.23:0/1685927100 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82d000b770 con 0x7f82ec0a9040 2020-11-20T11:39:09.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.678+0000 7f82ead9d700 1 -- 172.21.15.23:0/1685927100 reap_dead start 2020-11-20T11:39:09.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.678+0000 7f82ead9d700 1 -- 172.21.15.23:0/1685927100 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82ec0a9040 msgr2=0x7f82d0004ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:39:09.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.678+0000 7f82ea59c700 1 --2- 172.21.15.23:0/1685927100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82d00037a0 0x7f82d0003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:09.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.678+0000 7f82ead9d700 1 --2- 172.21.15.23:0/1685927100 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82ec0a9040 0x7f82d0004ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:39:09.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.678+0000 7f82e9d9b700 1 --2- 172.21.15.23:0/1685927100 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82d00040b0 0x7f82d0004580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:09.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.678+0000 7f82ea59c700 1 -- 172.21.15.23:0/1685927100 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82ec0a9040 msgr2=0x7f82d0004ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:39:09.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.678+0000 7f82ea59c700 1 --2- 172.21.15.23:0/1685927100 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82ec0a9040 0x7f82d0004ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:39:09.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.678+0000 7f82ea59c700 1 -- 172.21.15.23:0/1685927100 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82d00040b0 msgr2=0x7f82d0004580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:39:09.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.678+0000 7f82ea59c700 1 --2- 172.21.15.23:0/1685927100 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82d00040b0 0x7f82d0004580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:09.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.678+0000 7f82ea59c700 1 -- 172.21.15.23:0/1685927100 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d0003280 con 0x7f82d00037a0 2020-11-20T11:39:09.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.678+0000 7f82e9d9b700 1 --2- 172.21.15.23:0/1685927100 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82d00040b0 0x7f82d0004580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:39:09.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.678+0000 7f82ea59c700 1 --2- 172.21.15.23:0/1685927100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82d00037a0 0x7f82d0003ba0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f82dc009cc0 tx=0x7f82dc0092b0).ready entity=mon.0 client_cookie=a9e63bceed2af24d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:09.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.679+0000 7f82e959a700 1 -- 172.21.15.23:0/1685927100 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82dc003d30 con 0x7f82d00037a0 2020-11-20T11:39:09.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.679+0000 7f82e959a700 1 -- 172.21.15.23:0/1685927100 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82dc00b9e0 con 0x7f82d00037a0 2020-11-20T11:39:09.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.679+0000 7f82e959a700 1 -- 172.21.15.23:0/1685927100 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82dc00bda0 con 0x7f82d00037a0 2020-11-20T11:39:09.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.679+0000 7f82eb59e700 1 -- 172.21.15.23:0/1685927100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82d00037a0 msgr2=0x7f82d0003ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:09.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.679+0000 7f82eb59e700 1 --2- 172.21.15.23:0/1685927100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82d00037a0 0x7f82d0003ba0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f82dc009cc0 tx=0x7f82dc0092b0).stop 2020-11-20T11:39:09.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.679+0000 7f82eb59e700 1 -- 172.21.15.23:0/1685927100 shutdown_connections 2020-11-20T11:39:09.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.679+0000 7f82eb59e700 1 --2- 172.21.15.23:0/1685927100 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82d00040b0 0x7f82d0004580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:09.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.679+0000 7f82eb59e700 1 --2- 172.21.15.23:0/1685927100 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82ec0a9040 0x7f82d0004ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:39:09.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.679+0000 7f82eb59e700 1 --2- 172.21.15.23:0/1685927100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82d00037a0 0x7f82d0003ba0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:09.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.679+0000 7f82eb59e700 1 -- 172.21.15.23:0/1685927100 >> 172.21.15.23:0/1685927100 conn(0x7f82ec0683b0 msgr2=0x7f82ec0b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:09.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.679+0000 7f82eb59e700 1 -- 172.21.15.23:0/1685927100 shutdown_connections 2020-11-20T11:39:09.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.680+0000 7f82eb59e700 1 -- 172.21.15.23:0/1685927100 wait complete. 2020-11-20T11:39:09.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.681+0000 7f82eb59e700 1 Processor -- start 2020-11-20T11:39:09.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.681+0000 7f82eb59e700 1 -- start start 2020-11-20T11:39:09.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.682+0000 7f82eb59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82d00037a0 0x7f82ec13b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:09.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.682+0000 7f82eb59e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82d00040b0 0x7f82ec0a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:09.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.682+0000 7f82eb59e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82ec0a9040 0x7f82ec09fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:09.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.682+0000 7f82eb59e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec0a0d10 con 0x7f82d00037a0 2020-11-20T11:39:09.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.682+0000 7f82eb59e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec09f680 con 0x7f82d00040b0 2020-11-20T11:39:09.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.682+0000 7f82eb59e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec0a0f30 con 0x7f82ec0a9040 2020-11-20T11:39:09.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.682+0000 7f82ead9d700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82ec0a9040 msgr2=0x7f82ec09fae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:39:09.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.682+0000 7f82ead9d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82ec0a9040 0x7f82ec09fae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:39:09.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.682+0000 7f82e9d9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82d00040b0 0x7f82ec0a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:09.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.682+0000 7f82e9d9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82d00040b0 0x7f82ec0a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:36716/0 (socket says 172.21.15.23:36716) 2020-11-20T11:39:09.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.682+0000 7f82e9d9b700 1 -- 172.21.15.23:0/2274627728 learned_addr learned my addr 172.21.15.23:0/2274627728 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:39:09.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.682+0000 7f82e9d9b700 1 -- 172.21.15.23:0/2274627728 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82ec0a9040 msgr2=0x7f82ec09fae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:39:09.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.682+0000 7f82e9d9b700 1 --2- 172.21.15.23:0/2274627728 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82ec0a9040 0x7f82ec09fae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:39:09.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.682+0000 7f82e9d9b700 1 -- 172.21.15.23:0/2274627728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82d00037a0 msgr2=0x7f82ec13b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:09.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.682+0000 7f82e9d9b700 1 --2- 172.21.15.23:0/2274627728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82d00037a0 0x7f82ec13b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:39:09.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.682+0000 7f82e9d9b700 1 -- 172.21.15.23:0/2274627728 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ec0a11d0 con 0x7f82d00040b0 2020-11-20T11:39:09.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.683+0000 7f82e9d9b700 1 --2- 172.21.15.23:0/2274627728 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82d00040b0 0x7f82ec0a0820 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f82e0014fd0 tx=0x7f82e0008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:09.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.683+0000 7f82da7fc700 1 -- 172.21.15.23:0/2274627728 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e0012dd0 con 0x7f82d00040b0 2020-11-20T11:39:09.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.683+0000 7f82da7fc700 1 -- 172.21.15.23:0/2274627728 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e0016430 con 0x7f82d00040b0 2020-11-20T11:39:09.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.683+0000 7f82eb59e700 1 -- 172.21.15.23:0/2274627728 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82ec0a2b10 con 0x7f82d00040b0 2020-11-20T11:39:09.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.683+0000 7f82da7fc700 1 -- 172.21.15.23:0/2274627728 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e00167f0 con 0x7f82d00040b0 2020-11-20T11:39:09.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.683+0000 7f82eb59e700 1 -- 172.21.15.23:0/2274627728 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82ec0a0190 con 0x7f82d00040b0 2020-11-20T11:39:09.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.684+0000 7f82da7fc700 1 -- 172.21.15.23:0/2274627728 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f82e0016b60 con 0x7f82d00040b0 2020-11-20T11:39:09.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.685+0000 7f82da7fc700 1 --2- 172.21.15.23:0/2274627728 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f82c0056d10 0x7f82c00591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:09.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.685+0000 7f82da7fc700 1 -- 172.21.15.23:0/2274627728 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f82e0071160 con 0x7f82d00040b0 2020-11-20T11:39:09.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.685+0000 7f82ea59c700 1 --2- 172.21.15.23:0/2274627728 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f82c0056d10 0x7f82c00591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:09.699 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.685+0000 7f82ea59c700 1 --2- 172.21.15.23:0/2274627728 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f82c0056d10 0x7f82c00591d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f82dc009cc0 tx=0x7f82dc015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:09.699 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.685+0000 7f82eb59e700 1 -- 172.21.15.23:0/2274627728 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82ec0a0440 con 0x7f82d00040b0 2020-11-20T11:39:09.699 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.691+0000 7f82da7fc700 1 -- 172.21.15.23:0/2274627728 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f82e00463f0 con 0x7f82d00040b0 2020-11-20T11:39:09.958 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.948+0000 7f82eb59e700 1 -- 172.21.15.23:0/2274627728 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f82ec09f7f0 con 0x7f82d00040b0 2020-11-20T11:39:09.959 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.949+0000 7f82da7fc700 1 -- 172.21.15.23:0/2274627728 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1471 (secure 0 0 0) 0x7f82e0049a10 con 0x7f82d00040b0 2020-11-20T11:39:09.960 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.950+0000 7f82eb59e700 1 -- 172.21.15.23:0/2274627728 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f82c0056d10 msgr2=0x7f82c00591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:09.960 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.950+0000 7f82eb59e700 1 --2- 172.21.15.23:0/2274627728 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f82c0056d10 0x7f82c00591d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f82dc009cc0 tx=0x7f82dc015000).stop 2020-11-20T11:39:09.962 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.950+0000 7f82eb59e700 1 -- 172.21.15.23:0/2274627728 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82d00040b0 msgr2=0x7f82ec0a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:09.963 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.950+0000 7f82eb59e700 1 --2- 172.21.15.23:0/2274627728 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82d00040b0 0x7f82ec0a0820 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f82e0014fd0 tx=0x7f82e0008040).stop 2020-11-20T11:39:09.963 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.950+0000 7f82eb59e700 1 -- 172.21.15.23:0/2274627728 shutdown_connections 2020-11-20T11:39:09.963 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.950+0000 7f82eb59e700 1 --2- 172.21.15.23:0/2274627728 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f82c0056d10 0x7f82c00591d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:09.963 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.950+0000 7f82eb59e700 1 --2- 172.21.15.23:0/2274627728 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f82ec0a9040 0x7f82ec09fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:39:09.963 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.950+0000 7f82eb59e700 1 --2- 172.21.15.23:0/2274627728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82d00037a0 0x7f82ec13b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:39:09.964 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.950+0000 7f82eb59e700 1 --2- 172.21.15.23:0/2274627728 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82d00040b0 0x7f82ec0a0820 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:09.964 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.950+0000 7f82eb59e700 1 -- 172.21.15.23:0/2274627728 >> 172.21.15.23:0/2274627728 conn(0x7f82ec0683b0 msgr2=0x7f82ec0aba40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:09.964 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.950+0000 7f82eb59e700 1 -- 172.21.15.23:0/2274627728 shutdown_connections 2020-11-20T11:39:09.964 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:09.950+0000 7f82eb59e700 1 -- 172.21.15.23:0/2274627728 wait complete. 2020-11-20T11:39:09.965 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":70,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:39:09.969 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2020-11-20T11:39:09.970 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:39:10.113 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.096+0000 7ff1bbb54700 1 Processor -- start 2020-11-20T11:39:10.113 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.096+0000 7ff1bbb54700 1 -- start start 2020-11-20T11:39:10.113 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.097+0000 7ff1bbb54700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff1b4060ca0 0x7ff1b413eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:10.114 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.097+0000 7ff1bbb54700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1b41386f0 con 0x7ff1b40a1ac0 2020-11-20T11:39:10.114 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.097+0000 7ff1bbb54700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1b413e1b0 con 0x7ff1b40608c0 2020-11-20T11:39:10.114 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.097+0000 7ff1bbb54700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1b413e450 con 0x7ff1b4060ca0 2020-11-20T11:39:10.114 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.097+0000 7ff1ba0f1700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7ff1b40a1ac0 0x7ff1b40aabc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58080/0 (socket says 172.21.15.23:58080) 2020-11-20T11:39:10.114 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.097+0000 7ff1ba0f1700 1 -- 172.21.15.23:0/2166641735 learned_addr learned my addr 172.21.15.23:0/2166641735 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:39:10.115 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.097+0000 7ff1b90ef700 1 --2- 172.21.15.23:0/2166641735 >> v2:172.21.15.155:3300/0 conn(0x7ff1b4060ca0 0x7ff1b413eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:10.115 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.098+0000 7ff1b88ee700 1 -- 172.21.15.23:0/2166641735 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2115269638 0 0) 0x7ff1b41386f0 con 0x7ff1b40a1ac0 2020-11-20T11:39:10.115 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.098+0000 7ff1b88ee700 1 -- 172.21.15.23:0/2166641735 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff19c003280 con 0x7ff1b40a1ac0 2020-11-20T11:39:10.115 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.098+0000 7ff1b90ef700 1 -- 172.21.15.23:0/2166641735 >> v1:172.21.15.155:6789/0 conn(0x7ff1b40608c0 legacy=0x7ff1b4137fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:39:10.115 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.098+0000 7ff1b90ef700 1 -- 172.21.15.23:0/2166641735 >> v1:172.21.15.23:6789/0 conn(0x7ff1b40a1ac0 legacy=0x7ff1b40aabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:10.116 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.098+0000 7ff1b90ef700 1 -- 172.21.15.23:0/2166641735 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b413e6f0 con 0x7ff1b4060ca0 2020-11-20T11:39:10.116 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.098+0000 7ff1b90ef700 1 --2- 172.21.15.23:0/2166641735 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1b4060ca0 0x7ff1b413eb90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7ff1a4006fd0 tx=0x7ff1a4006e30).ready entity=mon.1 client_cookie=90bb17881321c550 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:10.116 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.098+0000 7ff1b88ee700 1 -- 172.21.15.23:0/2166641735 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a40097e0 con 0x7ff1b4060ca0 2020-11-20T11:39:10.116 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.098+0000 7ff1b88ee700 1 -- 172.21.15.23:0/2166641735 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1a4000e90 con 0x7ff1b4060ca0 2020-11-20T11:39:10.117 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.098+0000 7ff1b88ee700 1 -- 172.21.15.23:0/2166641735 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a4009b50 con 0x7ff1b4060ca0 2020-11-20T11:39:10.117 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.099+0000 7ff1bbb54700 1 -- 172.21.15.23:0/2166641735 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1b4060ca0 msgr2=0x7ff1b413eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:10.117 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.099+0000 7ff1bbb54700 1 --2- 172.21.15.23:0/2166641735 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1b4060ca0 0x7ff1b413eb90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7ff1a4006fd0 tx=0x7ff1a4006e30).stop 2020-11-20T11:39:10.117 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.099+0000 7ff1bbb54700 1 -- 172.21.15.23:0/2166641735 shutdown_connections 2020-11-20T11:39:10.117 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.099+0000 7ff1bbb54700 1 --2- 172.21.15.23:0/2166641735 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1b4060ca0 0x7ff1b413eb90 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:10.118 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.099+0000 7ff1bbb54700 1 -- 172.21.15.23:0/2166641735 >> 172.21.15.23:0/2166641735 conn(0x7ff1b4066750 msgr2=0x7ff1b40601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:10.118 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.099+0000 7ff1bbb54700 1 -- 172.21.15.23:0/2166641735 shutdown_connections 2020-11-20T11:39:10.118 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.099+0000 7ff1bbb54700 1 -- 172.21.15.23:0/2166641735 wait complete. 2020-11-20T11:39:10.118 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.100+0000 7ff1bbb54700 1 Processor -- start 2020-11-20T11:39:10.118 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.100+0000 7ff1bbb54700 1 -- start start 2020-11-20T11:39:10.119 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.101+0000 7ff1bbb54700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1b40608c0 0x7ff1b413a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:10.119 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.101+0000 7ff1bbb54700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff1b4060ca0 0x7ff1b4141ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:10.119 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.101+0000 7ff1bbb54700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1b40a1ac0 0x7ff1b414d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:10.119 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.101+0000 7ff1bbb54700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b41406b0 con 0x7ff1b40608c0 2020-11-20T11:39:10.119 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.101+0000 7ff1bbb54700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b41412c0 con 0x7ff1b40a1ac0 2020-11-20T11:39:10.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.101+0000 7ff1bbb54700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b41408e0 con 0x7ff1b4060ca0 2020-11-20T11:39:10.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.101+0000 7ff1b90ef700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff1b4060ca0 msgr2=0x7ff1b4141ca0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:39:10.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.101+0000 7ff1b98f0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1b40608c0 0x7ff1b413a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:10.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.101+0000 7ff1b90ef700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff1b4060ca0 0x7ff1b4141ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:39:10.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.101+0000 7ff1b98f0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1b40608c0 0x7ff1b413a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43832/0 (socket says 172.21.15.23:43832) 2020-11-20T11:39:10.121 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.101+0000 7ff1b98f0700 1 -- 172.21.15.23:0/1551017633 learned_addr learned my addr 172.21.15.23:0/1551017633 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:39:10.121 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.101+0000 7ff1ba0f1700 1 --2- 172.21.15.23:0/1551017633 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1b40a1ac0 0x7ff1b414d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:10.121 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.101+0000 7ff1b98f0700 1 -- 172.21.15.23:0/1551017633 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff1b4060ca0 msgr2=0x7ff1b4141ca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:39:10.122 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.101+0000 7ff1b98f0700 1 --2- 172.21.15.23:0/1551017633 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff1b4060ca0 0x7ff1b4141ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:39:10.122 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.101+0000 7ff1b98f0700 1 -- 172.21.15.23:0/1551017633 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1b40a1ac0 msgr2=0x7ff1b414d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:10.122 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.101+0000 7ff1b98f0700 1 --2- 172.21.15.23:0/1551017633 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1b40a1ac0 0x7ff1b414d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:10.122 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.101+0000 7ff1b98f0700 1 -- 172.21.15.23:0/1551017633 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b4140b80 con 0x7ff1b40608c0 2020-11-20T11:39:10.123 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.102+0000 7ff1b98f0700 1 --2- 172.21.15.23:0/1551017633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1b40608c0 0x7ff1b413a530 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7ff1b000c730 tx=0x7ff1b0007c70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:10.123 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.102+0000 7ff1a1ffb700 1 -- 172.21.15.23:0/1551017633 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b000e4a0 con 0x7ff1b40608c0 2020-11-20T11:39:10.123 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.102+0000 7ff1a1ffb700 1 -- 172.21.15.23:0/1551017633 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1b000e620 con 0x7ff1b40608c0 2020-11-20T11:39:10.123 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.102+0000 7ff1a1ffb700 1 -- 172.21.15.23:0/1551017633 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b000e9e0 con 0x7ff1b40608c0 2020-11-20T11:39:10.123 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.102+0000 7ff1bbb54700 1 -- 172.21.15.23:0/1551017633 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1b4140e20 con 0x7ff1b40608c0 2020-11-20T11:39:10.124 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.102+0000 7ff1bbb54700 1 -- 172.21.15.23:0/1551017633 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1b4142190 con 0x7ff1b40608c0 2020-11-20T11:39:10.124 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.103+0000 7ff1bbb54700 1 --2- 172.21.15.23:0/1551017633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1b414d9d0 0x7ff1b4136480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:10.124 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.103+0000 7ff1bbb54700 1 -- 172.21.15.23:0/1551017633 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b4044710 con 0x7ff1b414d9d0 2020-11-20T11:39:10.124 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.103+0000 7ff1b90ef700 1 --2- 172.21.15.23:0/1551017633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1b414d9d0 0x7ff1b4136480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:10.124 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.103+0000 7ff1a1ffb700 1 -- 172.21.15.23:0/1551017633 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff1b0019030 con 0x7ff1b40608c0 2020-11-20T11:39:10.125 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.103+0000 7ff1b90ef700 1 -- 172.21.15.23:0/1551017633 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1b405c280 con 0x7ff1b414d9d0 2020-11-20T11:39:10.125 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.103+0000 7ff1b90ef700 1 --2- 172.21.15.23:0/1551017633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1b414d9d0 0x7ff1b4136480 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7ff1a4006fd0 tx=0x7ff1a4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:10.125 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.103+0000 7ff1a1ffb700 1 --2- 172.21.15.23:0/1551017633 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff18c056d10 0x7ff18c0591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:10.125 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.103+0000 7ff1a1ffb700 1 -- 172.21.15.23:0/1551017633 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7ff1b006f460 con 0x7ff1b40608c0 2020-11-20T11:39:10.125 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.104+0000 7ff1a1ffb700 1 -- 172.21.15.23:0/1551017633 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a4011b20 con 0x7ff1b414d9d0 2020-11-20T11:39:10.126 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.104+0000 7ff1ba0f1700 1 --2- 172.21.15.23:0/1551017633 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff18c056d10 0x7ff18c0591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:10.126 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.104+0000 7ff1a1ffb700 1 -- 172.21.15.23:0/1551017633 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7ff1a4011c80 con 0x7ff1b414d9d0 2020-11-20T11:39:10.126 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.104+0000 7ff1a1ffb700 1 -- 172.21.15.23:0/1551017633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1b414d9d0 msgr2=0x7ff1b4136480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:10.126 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.104+0000 7ff1a1ffb700 1 --2- 172.21.15.23:0/1551017633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1b414d9d0 0x7ff1b4136480 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7ff1a4006fd0 tx=0x7ff1a4015000).stop 2020-11-20T11:39:10.126 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.104+0000 7ff1a1ffb700 1 -- 172.21.15.23:0/1551017633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1b414d9d0 msgr2=0x7ff1b4136480 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:39:10.127 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.104+0000 7ff1a1ffb700 1 --2- 172.21.15.23:0/1551017633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1b414d9d0 0x7ff1b4136480 secure :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0x7ff1a4006fd0 tx=0x7ff1a4015000).stop 2020-11-20T11:39:10.127 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.104+0000 7ff1ba0f1700 1 --2- 172.21.15.23:0/1551017633 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff18c056d10 0x7ff18c0591d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7ff1ac006fd0 tx=0x7ff1ac000f40).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:10.129 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.111+0000 7ff1bbb54700 1 --2- 172.21.15.23:0/1551017633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1b4136970 0x7ff1b4149f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:10.129 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.111+0000 7ff1bbb54700 1 -- 172.21.15.23:0/1551017633 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b4136d80 con 0x7ff1b4136970 2020-11-20T11:39:10.130 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.111+0000 7ff1b90ef700 1 --2- 172.21.15.23:0/1551017633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1b4136970 0x7ff1b4149f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:10.130 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.111+0000 7ff1b90ef700 1 -- 172.21.15.23:0/1551017633 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff1b40634d0 con 0x7ff1b4136970 2020-11-20T11:39:10.130 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.111+0000 7ff1b90ef700 1 --2- 172.21.15.23:0/1551017633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1b4136970 0x7ff1b4149f70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7ff1a4006e70 tx=0x7ff1a401e000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:10.130 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.111+0000 7ff1a1ffb700 1 -- 172.21.15.23:0/1551017633 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a401d720 con 0x7ff1b4136970 2020-11-20T11:39:10.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.112+0000 7ff1a1ffb700 1 -- 172.21.15.23:0/1551017633 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4434 (secure 0 0 0) 0x7ff1a4011810 con 0x7ff1b4136970 2020-11-20T11:39:10.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.112+0000 7ff1a1ffb700 1 -- 172.21.15.23:0/1551017633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1b4136970 msgr2=0x7ff1b4149f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:10.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.112+0000 7ff1a1ffb700 1 --2- 172.21.15.23:0/1551017633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1b4136970 0x7ff1b4149f70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7ff1a4006e70 tx=0x7ff1a401e000).stop 2020-11-20T11:39:10.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.112+0000 7ff1a1ffb700 1 -- 172.21.15.23:0/1551017633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1b4136970 msgr2=0x7ff1b4149f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:39:10.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.112+0000 7ff1a1ffb700 1 --2- 172.21.15.23:0/1551017633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1b4136970 0x7ff1b4149f70 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:10.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.112+0000 7ff1bbb54700 1 -- 172.21.15.23:0/1551017633 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff18c056d10 msgr2=0x7ff18c0591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:10.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.112+0000 7ff1bbb54700 1 --2- 172.21.15.23:0/1551017633 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff18c056d10 0x7ff18c0591d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7ff1ac006fd0 tx=0x7ff1ac000f40).stop 2020-11-20T11:39:10.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.112+0000 7ff1bbb54700 1 -- 172.21.15.23:0/1551017633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1b40608c0 msgr2=0x7ff1b413a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:10.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.112+0000 7ff1bbb54700 1 --2- 172.21.15.23:0/1551017633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1b40608c0 0x7ff1b413a530 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7ff1b000c730 tx=0x7ff1b0007c70).stop 2020-11-20T11:39:10.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.112+0000 7ff1ba0f1700 1 -- 172.21.15.23:0/1551017633 reap_dead start 2020-11-20T11:39:10.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.112+0000 7ff1ba0f1700 1 -- 172.21.15.23:0/1551017633 reap_dead start 2020-11-20T11:39:10.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.112+0000 7ff1bbb54700 1 -- 172.21.15.23:0/1551017633 shutdown_connections 2020-11-20T11:39:10.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.112+0000 7ff1bbb54700 1 -- 172.21.15.23:0/1551017633 >> 172.21.15.23:0/1551017633 conn(0x7ff1b4066750 msgr2=0x7ff1b40ab600 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:10.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.112+0000 7ff1bbb54700 1 -- 172.21.15.23:0/1551017633 shutdown_connections 2020-11-20T11:39:10.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.112+0000 7ff1bbb54700 1 -- 172.21.15.23:0/1551017633 wait complete. 2020-11-20T11:39:10.134 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:39:10.134 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:39:10.134 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:39:10.134 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:39:10.135 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 70, 2020-11-20T11:39:10.135 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:39:10.135 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:39:10.135 INFO:teuthology.orchestra.run.smithi023.stdout: 1 2020-11-20T11:39:10.135 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:10.136 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 0, 2020-11-20T11:39:10.136 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:39:10.136 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:39:10.136 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:39:10.136 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:39:10.137 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:39:10.137 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:39:10.137 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:39:10.137 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:39:10.137 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:39:10.138 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:39:10.138 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:39:10.138 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:10.138 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:39:10.138 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:39:10.139 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:39:10.139 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:39:10.139 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:39:10.139 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:39:10.139 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:39:10.140 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:39:10.140 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:39:10.140 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:39:10.140 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:10.140 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:10.141 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:39:10.141 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:39:10.141 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:39:10.141 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:39:10.142 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:39:10.142 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:39:10.142 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:39:10.142 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:39:10.142 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:39:10.143 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:39:10.143 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:39:10.143 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:39:10.143 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:39:10.143 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:39:10.144 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:39:10.144 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:39:10.144 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:39:10.144 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:39:10.144 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:39:10.145 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:39:10.145 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:39:10.145 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:39:10.145 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:39:10.145 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:10.146 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:39:10.146 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:10.146 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:39:10.146 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:10.146 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:39:10.147 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:39:10.147 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:39:10.147 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:39:10.147 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:10.147 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:39:10.148 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:39:10.148 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:10.148 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:10.148 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:10.149 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:39:10.149 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:39:10.149 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:10.149 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:10.149 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:10.150 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:10.150 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:39:10.150 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:39:10.150 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:39:10.150 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:39:10.151 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:39:10.151 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:10.151 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:10.151 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:39:10.151 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:39:10.152 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:39:10.152 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:39:10.152 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:10.152 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:39:10.152 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:39:10.153 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:10.153 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:10.153 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:10.153 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:39:10.153 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:39:10.154 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:10.154 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:10.154 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:10.154 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:10.154 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:39:10.155 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:39:10.155 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:39:10.155 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:39:10.155 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:39:10.155 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:10.156 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:10.156 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:39:10.156 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:39:10.156 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:39:10.156 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:39:10.157 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:10.157 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:39:10.157 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:39:10.157 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:10.157 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:10.158 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:10.158 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:39:10.158 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:39:10.158 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:10.158 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:10.159 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:10.159 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:10.159 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:39:10.159 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:39:10.159 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:39:10.160 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:39:10.160 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:39:10.160 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:10.160 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:10.160 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:10.161 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:39:10.161 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:39:10.161 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:10.161 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:39:10.162 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:39:10.162 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:39:10.162 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:10.162 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:10.162 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:39:10.163 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:10.163 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:39:10.163 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:39:10.163 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:39:10.163 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:10.164 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:10.164 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:39:10.164 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:10.164 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:39:10.165 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:39:10.165 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:39:10.165 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:10.165 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:10.165 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:10.166 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:39:10.166 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:39:10.166 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:39:10.299 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.296+0000 7f0227ec8700 1 Processor -- start 2020-11-20T11:39:10.299 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.296+0000 7f0227ec8700 1 -- start start 2020-11-20T11:39:10.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.296+0000 7f0227ec8700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f022013aeb0 0x7f02200a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:10.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.296+0000 7f0227ec8700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02201388b0 con 0x7f02200aae90 2020-11-20T11:39:10.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.296+0000 7f0227ec8700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f022013e600 con 0x7f02200aaab0 2020-11-20T11:39:10.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.296+0000 7f0227ec8700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f022013e8a0 con 0x7f022013aeb0 2020-11-20T11:39:10.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.297+0000 7f0225463700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f02200aae90 0x7f02201381a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58096/0 (socket says 172.21.15.23:58096) 2020-11-20T11:39:10.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.297+0000 7f0225463700 1 -- 172.21.15.23:0/807171032 learned_addr learned my addr 172.21.15.23:0/807171032 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:39:10.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.297+0000 7f0226465700 1 --2- 172.21.15.23:0/807171032 >> v2:172.21.15.155:3300/0 conn(0x7f022013aeb0 0x7f02200a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:10.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.297+0000 7f0224c62700 1 -- 172.21.15.23:0/807171032 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3373001715 0 0) 0x7f02201388b0 con 0x7f02200aae90 2020-11-20T11:39:10.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.297+0000 7f0224c62700 1 -- 172.21.15.23:0/807171032 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0208003280 con 0x7f02200aae90 2020-11-20T11:39:10.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.297+0000 7f0226465700 1 -- 172.21.15.23:0/807171032 >> v1:172.21.15.155:6789/0 conn(0x7f02200aaab0 legacy=0x7f022013ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:10.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.297+0000 7f0226465700 1 -- 172.21.15.23:0/807171032 >> v1:172.21.15.23:6789/0 conn(0x7f02200aae90 legacy=0x7f02201381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:10.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.297+0000 7f0226465700 1 -- 172.21.15.23:0/807171032 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02200ab6d0 con 0x7f022013aeb0 2020-11-20T11:39:10.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.297+0000 7f0226465700 1 --2- 172.21.15.23:0/807171032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f022013aeb0 0x7f02200a8c60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f0218009ce0 tx=0x7f0218009220).ready entity=mon.1 client_cookie=3ce5359de4b21b6f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:10.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.297+0000 7f0224c62700 1 -- 172.21.15.23:0/807171032 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0210001520 con 0x7f022013aeb0 2020-11-20T11:39:10.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.298+0000 7f0224c62700 1 -- 172.21.15.23:0/807171032 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f021800b730 con 0x7f022013aeb0 2020-11-20T11:39:10.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.298+0000 7f0224c62700 1 -- 172.21.15.23:0/807171032 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021800bab0 con 0x7f022013aeb0 2020-11-20T11:39:10.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.298+0000 7f0227ec8700 1 -- 172.21.15.23:0/807171032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f022013aeb0 msgr2=0x7f02200a8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:10.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.298+0000 7f0227ec8700 1 --2- 172.21.15.23:0/807171032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f022013aeb0 0x7f02200a8c60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f0218009ce0 tx=0x7f0218009220).stop 2020-11-20T11:39:10.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.298+0000 7f0227ec8700 1 -- 172.21.15.23:0/807171032 shutdown_connections 2020-11-20T11:39:10.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.298+0000 7f0227ec8700 1 --2- 172.21.15.23:0/807171032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f022013aeb0 0x7f02200a8c60 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:10.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.298+0000 7f0227ec8700 1 -- 172.21.15.23:0/807171032 >> 172.21.15.23:0/807171032 conn(0x7f02200678e0 msgr2=0x7f0220067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:10.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.298+0000 7f0227ec8700 1 -- 172.21.15.23:0/807171032 shutdown_connections 2020-11-20T11:39:10.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.298+0000 7f0227ec8700 1 -- 172.21.15.23:0/807171032 wait complete. 2020-11-20T11:39:10.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.299+0000 7f0227ec8700 1 Processor -- start 2020-11-20T11:39:10.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.299+0000 7f0227ec8700 1 -- start start 2020-11-20T11:39:10.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.300+0000 7f0227ec8700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f02200aaab0 0x7f022014a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:10.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.300+0000 7f0227ec8700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f02200aae90 0x7f0220136240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:10.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.300+0000 7f0227ec8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0220134ff0 0x7f02200a6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:10.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.300+0000 7f0227ec8700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02200a83c0 con 0x7f0220134ff0 2020-11-20T11:39:10.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.300+0000 7f0227ec8700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0220135b00 con 0x7f02200aaab0 2020-11-20T11:39:10.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.300+0000 7f0227ec8700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0220135da0 con 0x7f02200aae90 2020-11-20T11:39:10.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.300+0000 7f0226465700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0220134ff0 0x7f02200a6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:10.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.300+0000 7f0225463700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f02200aae90 msgr2=0x7f0220136240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:39:10.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.300+0000 7f0225463700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f02200aae90 0x7f0220136240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:39:10.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.300+0000 7f0226465700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0220134ff0 0x7f02200a6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43854/0 (socket says 172.21.15.23:43854) 2020-11-20T11:39:10.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.300+0000 7f0226465700 1 -- 172.21.15.23:0/4177397191 learned_addr learned my addr 172.21.15.23:0/4177397191 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:39:10.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.300+0000 7f0225c64700 1 --2- 172.21.15.23:0/4177397191 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f02200aaab0 0x7f022014a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:10.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.300+0000 7f0226465700 1 -- 172.21.15.23:0/4177397191 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f02200aae90 msgr2=0x7f0220136240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:39:10.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.300+0000 7f0226465700 1 --2- 172.21.15.23:0/4177397191 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f02200aae90 0x7f0220136240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:39:10.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.300+0000 7f0226465700 1 -- 172.21.15.23:0/4177397191 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f02200aaab0 msgr2=0x7f022014a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:10.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.300+0000 7f0226465700 1 --2- 172.21.15.23:0/4177397191 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f02200aaab0 0x7f022014a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:10.313 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.300+0000 7f0226465700 1 -- 172.21.15.23:0/4177397191 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0220135450 con 0x7f0220134ff0 2020-11-20T11:39:10.313 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.301+0000 7f0226465700 1 --2- 172.21.15.23:0/4177397191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0220134ff0 0x7f02200a6630 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f0220062f90 tx=0x7f0218009220).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:10.313 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.301+0000 7f020dffb700 1 -- 172.21.15.23:0/4177397191 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0218010640 con 0x7f0220134ff0 2020-11-20T11:39:10.313 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.301+0000 7f020dffb700 1 -- 172.21.15.23:0/4177397191 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02180107a0 con 0x7f0220134ff0 2020-11-20T11:39:10.314 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.301+0000 7f020dffb700 1 -- 172.21.15.23:0/4177397191 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0218010b20 con 0x7f0220134ff0 2020-11-20T11:39:10.314 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.301+0000 7f0227ec8700 1 -- 172.21.15.23:0/4177397191 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02200a6b20 con 0x7f0220134ff0 2020-11-20T11:39:10.314 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.301+0000 7f0227ec8700 1 -- 172.21.15.23:0/4177397191 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f022014dd80 con 0x7f0220134ff0 2020-11-20T11:39:10.314 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.302+0000 7f0227ec8700 1 --2- 172.21.15.23:0/4177397191 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f022013aeb0 0x7f022014e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:10.314 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.302+0000 7f0227ec8700 1 -- 172.21.15.23:0/4177397191 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02200a9100 con 0x7f022013aeb0 2020-11-20T11:39:10.315 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.302+0000 7f0225c64700 1 --2- 172.21.15.23:0/4177397191 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f022013aeb0 0x7f022014e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:10.315 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.303+0000 7f020dffb700 1 -- 172.21.15.23:0/4177397191 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0218024030 con 0x7f0220134ff0 2020-11-20T11:39:10.315 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.303+0000 7f020dffb700 1 --2- 172.21.15.23:0/4177397191 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f01f8056d10 0x7f01f80591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:10.315 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.303+0000 7f0225c64700 1 -- 172.21.15.23:0/4177397191 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0220135980 con 0x7f022013aeb0 2020-11-20T11:39:10.315 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.303+0000 7f020dffb700 1 -- 172.21.15.23:0/4177397191 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f0218071a70 con 0x7f0220134ff0 2020-11-20T11:39:10.316 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.303+0000 7f0225c64700 1 --2- 172.21.15.23:0/4177397191 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f022013aeb0 0x7f022014e060 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f021c00f850 tx=0x7f021c007bd0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:10.316 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.304+0000 7f0225463700 1 --2- 172.21.15.23:0/4177397191 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f01f8056d10 0x7f01f80591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:10.316 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.304+0000 7f020dffb700 1 -- 172.21.15.23:0/4177397191 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021c003040 con 0x7f022013aeb0 2020-11-20T11:39:10.316 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.304+0000 7f020dffb700 1 -- 172.21.15.23:0/4177397191 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f021c007030 con 0x7f022013aeb0 2020-11-20T11:39:10.316 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.304+0000 7f0225463700 1 --2- 172.21.15.23:0/4177397191 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f01f8056d10 0x7f01f80591d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f02200a61a0 tx=0x7f02100019f0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:10.317 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.304+0000 7f020dffb700 1 -- 172.21.15.23:0/4177397191 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f022013aeb0 msgr2=0x7f022014e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:10.317 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.304+0000 7f020dffb700 1 --2- 172.21.15.23:0/4177397191 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f022013aeb0 0x7f022014e060 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f021c00f850 tx=0x7f021c007bd0).stop 2020-11-20T11:39:10.317 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.304+0000 7f020dffb700 1 -- 172.21.15.23:0/4177397191 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f022013aeb0 msgr2=0x7f022014e060 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:39:10.317 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.304+0000 7f020dffb700 1 --2- 172.21.15.23:0/4177397191 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f022013aeb0 0x7f022014e060 secure :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0x7f021c00f850 tx=0x7f021c007bd0).stop 2020-11-20T11:39:10.319 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.314+0000 7f0227ec8700 1 --2- 172.21.15.23:0/4177397191 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f02200601b0 0x7f022014e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:10.319 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.314+0000 7f0227ec8700 1 -- 172.21.15.23:0/4177397191 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02200a6d10 con 0x7f02200601b0 2020-11-20T11:39:10.320 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.314+0000 7f0225c64700 1 --2- 172.21.15.23:0/4177397191 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f02200601b0 0x7f022014e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:10.320 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.314+0000 7f0225c64700 1 -- 172.21.15.23:0/4177397191 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f022014ea40 con 0x7f02200601b0 2020-11-20T11:39:10.320 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.314+0000 7f0225c64700 1 --2- 172.21.15.23:0/4177397191 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f02200601b0 0x7f022014e550 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f021c00f850 tx=0x7f021c018000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:10.320 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.315+0000 7f020dffb700 1 -- 172.21.15.23:0/4177397191 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f021c006b40 con 0x7f02200601b0 2020-11-20T11:39:10.321 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.315+0000 7f020dffb700 1 -- 172.21.15.23:0/4177397191 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4772 (secure 0 0 0) 0x7f021c00d4f0 con 0x7f02200601b0 2020-11-20T11:39:10.321 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.315+0000 7f020dffb700 1 -- 172.21.15.23:0/4177397191 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f02200601b0 msgr2=0x7f022014e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:10.321 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.315+0000 7f020dffb700 1 --2- 172.21.15.23:0/4177397191 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f02200601b0 0x7f022014e550 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f021c00f850 tx=0x7f021c018000).stop 2020-11-20T11:39:10.321 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.315+0000 7f020dffb700 1 -- 172.21.15.23:0/4177397191 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f02200601b0 msgr2=0x7f022014e550 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:39:10.321 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.315+0000 7f020dffb700 1 --2- 172.21.15.23:0/4177397191 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f02200601b0 0x7f022014e550 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:10.322 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.315+0000 7f0227ec8700 1 -- 172.21.15.23:0/4177397191 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f01f8056d10 msgr2=0x7f01f80591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:10.322 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.315+0000 7f0227ec8700 1 --2- 172.21.15.23:0/4177397191 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f01f8056d10 0x7f01f80591d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f02200a61a0 tx=0x7f02100019f0).stop 2020-11-20T11:39:10.322 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.315+0000 7f0227ec8700 1 -- 172.21.15.23:0/4177397191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0220134ff0 msgr2=0x7f02200a6630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:10.322 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.315+0000 7f0227ec8700 1 --2- 172.21.15.23:0/4177397191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0220134ff0 0x7f02200a6630 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f0220062f90 tx=0x7f0218009220).stop 2020-11-20T11:39:10.322 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.315+0000 7f0226465700 1 -- 172.21.15.23:0/4177397191 reap_dead start 2020-11-20T11:39:10.323 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.315+0000 7f0226465700 1 -- 172.21.15.23:0/4177397191 reap_dead start 2020-11-20T11:39:10.323 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.315+0000 7f0227ec8700 1 -- 172.21.15.23:0/4177397191 shutdown_connections 2020-11-20T11:39:10.323 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.315+0000 7f0227ec8700 1 -- 172.21.15.23:0/4177397191 >> 172.21.15.23:0/4177397191 conn(0x7f02200678e0 msgr2=0x7f0220068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:10.323 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.316+0000 7f0227ec8700 1 -- 172.21.15.23:0/4177397191 shutdown_connections 2020-11-20T11:39:10.323 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:10.316+0000 7f0227ec8700 1 -- 172.21.15.23:0/4177397191 wait complete. 2020-11-20T11:39:10.324 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:39:10.324 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:39:10.324 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:39:10.324 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:39:10.325 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 70, 2020-11-20T11:39:10.325 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:39:10.325 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:39:10.325 INFO:teuthology.orchestra.run.smithi023.stdout: 1 2020-11-20T11:39:10.325 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:10.326 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 0, 2020-11-20T11:39:10.326 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:39:10.326 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:39:10.326 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:39:10.326 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:39:10.327 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:39:10.327 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:39:10.327 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:39:10.327 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:39:10.327 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:39:10.328 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:39:10.328 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:39:10.328 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:10.328 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:39:10.328 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:39:10.329 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:39:10.329 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:39:10.329 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:39:10.329 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:39:10.329 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:39:10.330 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:39:10.330 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:39:10.330 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:39:10.330 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:10.330 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:10.331 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:39:10.331 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:39:10.331 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:39:10.331 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:39:10.332 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:39:10.332 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:39:10.332 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:39:10.332 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:39:10.332 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:39:10.333 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:39:10.333 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:39:10.333 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:39:10.333 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:39:10.333 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:39:10.334 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:39:10.334 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:39:10.334 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:39:10.335 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:39:10.335 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:39:10.335 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:39:10.335 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:39:10.335 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:39:10.336 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:39:10.336 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:10.336 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:39:10.336 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:10.337 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:39:10.337 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:10.337 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:39:10.337 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:39:10.337 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:39:10.338 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:39:10.338 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:10.338 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:39:10.338 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:39:10.339 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:10.339 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:10.339 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:10.339 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:39:10.339 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:39:10.340 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:10.340 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:10.340 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:10.340 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:10.340 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:39:10.341 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:39:10.341 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:39:10.341 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:39:10.341 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:39:10.341 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:10.342 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:10.342 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:39:10.342 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:39:10.342 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:39:10.342 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:39:10.343 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:10.343 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:39:10.343 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:39:10.343 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:10.343 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:10.344 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:10.344 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:39:10.344 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:39:10.344 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:10.344 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:10.345 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:10.345 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:10.345 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:39:10.345 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:39:10.345 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:39:10.346 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:39:10.346 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:39:10.346 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:10.346 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:10.346 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:39:10.347 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:39:10.347 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:39:10.347 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:39:10.347 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:10.347 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:39:10.348 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:39:10.348 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:10.348 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:10.348 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:10.348 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:39:10.349 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:39:10.349 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:10.349 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:10.349 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:10.349 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:10.350 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:39:10.350 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:39:10.350 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:39:10.350 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:39:10.350 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:39:10.351 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:10.351 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:10.351 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:10.351 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:39:10.351 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:39:10.352 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:10.352 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:39:10.352 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:39:10.352 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:39:10.352 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:10.353 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:10.353 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:39:10.353 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:10.353 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:39:10.353 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:39:10.354 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:39:10.354 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:10.354 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:10.354 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:39:10.354 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:10.355 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:39:10.355 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:39:10.355 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:39:10.355 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:10.355 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:10.356 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:39:10.356 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:10.356 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:39:10.356 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:39:10.356 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:39:10.357 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:10.357 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:10.357 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:39:10.357 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:10.357 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:39:10.358 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:39:10.358 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:39:10.358 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:10.358 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:10.358 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:10.359 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:39:10.359 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:39:10.360 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2020-11-20T11:39:30.361 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2020-11-20T11:39:30.361 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2020-11-20T11:39:30.362 INFO:tasks.ceph.mon.c:Restarting daemon 2020-11-20T11:39:30.362 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2020-11-20T11:39:30.365 INFO:tasks.ceph.mon.c:Started 2020-11-20T11:39:30.365 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:39:30.365 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:39:30.497 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.494+0000 7f990c37e700 1 Processor -- start 2020-11-20T11:39:30.498 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.494+0000 7f990c37e700 1 -- start start 2020-11-20T11:39:30.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.495+0000 7f990c37e700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f9904061380 0x7f99040aaac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:30.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.495+0000 7f990c37e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f99040a9dd0 0x7f99040a3a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:30.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.495+0000 7f990c37e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f99040a3f80 0x7f990413e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:30.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.495+0000 7f990c37e700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99040aafb0 con 0x7f9904061380 2020-11-20T11:39:30.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.495+0000 7f990c37e700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99040a3050 con 0x7f99040a3f80 2020-11-20T11:39:30.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.495+0000 7f990c37e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99040a32f0 con 0x7f99040a9dd0 2020-11-20T11:39:30.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.496+0000 7f990a11a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f9904061380 0x7f99040aaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:30.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.496+0000 7f990a11a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f9904061380 0x7f99040aaac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43864/0 (socket says 172.21.15.23:43864) 2020-11-20T11:39:30.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.496+0000 7f990a11a700 1 -- 172.21.15.23:0/2056766883 learned_addr learned my addr 172.21.15.23:0/2056766883 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:39:30.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.496+0000 7f990a91b700 1 --2- 172.21.15.23:0/2056766883 >> v2:172.21.15.155:3300/0 conn(0x7f99040a3f80 0x7f990413e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:30.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.496+0000 7f9909919700 1 --2- 172.21.15.23:0/2056766883 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f99040a9dd0 0x7f99040a3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:30.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.496+0000 7f9909919700 1 -- 172.21.15.23:0/2056766883 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f99040a9dd0 msgr2=0x7f99040a3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:39:30.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.496+0000 7f9909919700 1 -- 172.21.15.23:0/2056766883 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f99040a9dd0 msgr2=0x7f99040a3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:39:30.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.496+0000 7f9909919700 1 --2- 172.21.15.23:0/2056766883 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f99040a9dd0 0x7f99040a3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:39:30.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.496+0000 7f9909919700 1 --2- 172.21.15.23:0/2056766883 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f99040a9dd0 0x7f99040a3a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:39:30.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.496+0000 7f990a11a700 1 -- 172.21.15.23:0/2056766883 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f99040a9dd0 msgr2=0x7f99040a3a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:39:30.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.496+0000 7f990a11a700 1 --2- 172.21.15.23:0/2056766883 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f99040a9dd0 0x7f99040a3a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:30.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.496+0000 7f990a11a700 1 -- 172.21.15.23:0/2056766883 >> v2:172.21.15.155:3300/0 conn(0x7f99040a3f80 msgr2=0x7f990413e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:39:30.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.496+0000 7f990a11a700 1 --2- 172.21.15.23:0/2056766883 >> v2:172.21.15.155:3300/0 conn(0x7f99040a3f80 0x7f990413e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:30.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.496+0000 7f990a11a700 1 -- 172.21.15.23:0/2056766883 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99040a3590 con 0x7f9904061380 2020-11-20T11:39:30.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.496+0000 7f990a11a700 1 --2- 172.21.15.23:0/2056766883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9904061380 0x7f99040aaac0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f99040a2f30 tx=0x7f9900006ef0).ready entity=mon.0 client_cookie=6bb906c8d1f687d8 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:30.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.496+0000 7f9909118700 1 -- 172.21.15.23:0/2056766883 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990000b860 con 0x7f9904061380 2020-11-20T11:39:30.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.496+0000 7f9909118700 1 -- 172.21.15.23:0/2056766883 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9900000e90 con 0x7f9904061380 2020-11-20T11:39:30.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.496+0000 7f9909118700 1 -- 172.21.15.23:0/2056766883 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990000bbd0 con 0x7f9904061380 2020-11-20T11:39:30.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.497+0000 7f990c37e700 1 -- 172.21.15.23:0/2056766883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9904061380 msgr2=0x7f99040aaac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:30.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.497+0000 7f990c37e700 1 --2- 172.21.15.23:0/2056766883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9904061380 0x7f99040aaac0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f99040a2f30 tx=0x7f9900006ef0).stop 2020-11-20T11:39:30.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.497+0000 7f990c37e700 1 -- 172.21.15.23:0/2056766883 shutdown_connections 2020-11-20T11:39:30.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.497+0000 7f990c37e700 1 --2- 172.21.15.23:0/2056766883 >> v2:172.21.15.155:3300/0 conn(0x7f99040a3f80 0x7f990413e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:30.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.497+0000 7f990c37e700 1 --2- 172.21.15.23:0/2056766883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9904061380 0x7f99040aaac0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:30.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.497+0000 7f990c37e700 1 --2- 172.21.15.23:0/2056766883 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f99040a9dd0 0x7f99040a3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:30.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.497+0000 7f990c37e700 1 -- 172.21.15.23:0/2056766883 >> 172.21.15.23:0/2056766883 conn(0x7f99040ae280 msgr2=0x7f99040aa290 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:30.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.497+0000 7f990c37e700 1 -- 172.21.15.23:0/2056766883 shutdown_connections 2020-11-20T11:39:30.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.497+0000 7f990c37e700 1 -- 172.21.15.23:0/2056766883 wait complete. 2020-11-20T11:39:30.509 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.498+0000 7f990c37e700 1 Processor -- start 2020-11-20T11:39:30.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.498+0000 7f990c37e700 1 -- start start 2020-11-20T11:39:30.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.498+0000 7f990c37e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9904061380 0x7f99040acd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:30.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.498+0000 7f990c37e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99040a3f80 0x7f990412fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:30.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.498+0000 7f990c37e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99040a9dd0 0x7f9904130000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:30.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.498+0000 7f990c37e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99040a4ec0 con 0x7f99040a3f80 2020-11-20T11:39:30.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.498+0000 7f990c37e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99040a5160 con 0x7f99040a9dd0 2020-11-20T11:39:30.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.499+0000 7f990c37e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99040a5400 con 0x7f9904061380 2020-11-20T11:39:30.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.499+0000 7f9909919700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99040a3f80 0x7f990412fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:30.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.499+0000 7f9909919700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99040a3f80 0x7f990412fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43872/0 (socket says 172.21.15.23:43872) 2020-11-20T11:39:30.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.499+0000 7f9909919700 1 -- 172.21.15.23:0/3861344329 learned_addr learned my addr 172.21.15.23:0/3861344329 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:39:30.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.499+0000 7f990a11a700 1 --2- 172.21.15.23:0/3861344329 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9904061380 0x7f99040acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:30.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.499+0000 7f990a91b700 1 --2- 172.21.15.23:0/3861344329 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99040a9dd0 0x7f9904130000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:30.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.499+0000 7f990a11a700 1 -- 172.21.15.23:0/3861344329 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9904061380 msgr2=0x7f99040acd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2020-11-20T11:39:30.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.499+0000 7f990a11a700 1 -- 172.21.15.23:0/3861344329 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9904061380 msgr2=0x7f99040acd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:39:30.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.499+0000 7f990a11a700 1 --2- 172.21.15.23:0/3861344329 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9904061380 0x7f99040acd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:39:30.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.499+0000 7f990a11a700 1 --2- 172.21.15.23:0/3861344329 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9904061380 0x7f99040acd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:39:30.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.499+0000 7f9909919700 1 -- 172.21.15.23:0/3861344329 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9904061380 msgr2=0x7f99040acd00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:39:30.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.499+0000 7f9909919700 1 --2- 172.21.15.23:0/3861344329 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9904061380 0x7f99040acd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:30.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.499+0000 7f9909919700 1 -- 172.21.15.23:0/3861344329 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99040a9dd0 msgr2=0x7f9904130000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:30.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.499+0000 7f9909919700 1 --2- 172.21.15.23:0/3861344329 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99040a9dd0 0x7f9904130000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:30.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.499+0000 7f9909919700 1 -- 172.21.15.23:0/3861344329 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99040a4b40 con 0x7f99040a3f80 2020-11-20T11:39:30.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.499+0000 7f9909919700 1 --2- 172.21.15.23:0/3861344329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99040a3f80 0x7f990412fb10 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f98f400db30 tx=0x7f98f4008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:30.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.499+0000 7f98f27fc700 1 -- 172.21.15.23:0/3861344329 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f4009760 con 0x7f99040a3f80 2020-11-20T11:39:30.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.499+0000 7f98f27fc700 1 -- 172.21.15.23:0/3861344329 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98f40098e0 con 0x7f99040a3f80 2020-11-20T11:39:30.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.499+0000 7f98f27fc700 1 -- 172.21.15.23:0/3861344329 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f4009ca0 con 0x7f99040a3f80 2020-11-20T11:39:30.515 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.499+0000 7f990c37e700 1 -- 172.21.15.23:0/3861344329 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99040a7460 con 0x7f99040a3f80 2020-11-20T11:39:30.515 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.499+0000 7f990c37e700 1 -- 172.21.15.23:0/3861344329 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99040a7150 con 0x7f99040a3f80 2020-11-20T11:39:30.515 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.500+0000 7f990c37e700 1 -- 172.21.15.23:0/3861344329 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99040a4d30 con 0x7f99040a3f80 2020-11-20T11:39:30.515 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.501+0000 7f98f27fc700 1 -- 172.21.15.23:0/3861344329 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f98f4013480 con 0x7f99040a3f80 2020-11-20T11:39:30.515 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.501+0000 7f98f27fc700 1 --2- 172.21.15.23:0/3861344329 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f98dc056d60 0x7f98dc059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:30.516 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.501+0000 7f98f27fc700 1 -- 172.21.15.23:0/3861344329 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f98f406ffd0 con 0x7f99040a3f80 2020-11-20T11:39:30.516 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.501+0000 7f990a11a700 1 --2- 172.21.15.23:0/3861344329 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f98dc056d60 0x7f98dc059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:30.516 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.502+0000 7f990a11a700 1 --2- 172.21.15.23:0/3861344329 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f98dc056d60 0x7f98dc059220 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f9900006f30 tx=0x7f9900016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:30.516 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.505+0000 7f98f27fc700 1 -- 172.21.15.23:0/3861344329 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f98f4045260 con 0x7f99040a3f80 2020-11-20T11:39:30.780 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.776+0000 7f990c37e700 1 -- 172.21.15.23:0/3861344329 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9904061e10 con 0x7f99040a3f80 2020-11-20T11:39:30.780 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.776+0000 7f98f27fc700 1 -- 172.21.15.23:0/3861344329 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1472 (secure 0 0 0) 0x7f98f4048880 con 0x7f99040a3f80 2020-11-20T11:39:30.781 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":70,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:39:30.782 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.778+0000 7f990c37e700 1 -- 172.21.15.23:0/3861344329 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f98dc056d60 msgr2=0x7f98dc059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:30.782 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.778+0000 7f990c37e700 1 --2- 172.21.15.23:0/3861344329 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f98dc056d60 0x7f98dc059220 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f9900006f30 tx=0x7f9900016000).stop 2020-11-20T11:39:30.782 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.778+0000 7f990c37e700 1 -- 172.21.15.23:0/3861344329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99040a3f80 msgr2=0x7f990412fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:30.782 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.778+0000 7f990c37e700 1 --2- 172.21.15.23:0/3861344329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99040a3f80 0x7f990412fb10 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f98f400db30 tx=0x7f98f4008040).stop 2020-11-20T11:39:30.783 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.778+0000 7f990c37e700 1 -- 172.21.15.23:0/3861344329 shutdown_connections 2020-11-20T11:39:30.783 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.778+0000 7f990c37e700 1 --2- 172.21.15.23:0/3861344329 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99040a9dd0 0x7f9904130000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:30.783 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.778+0000 7f990c37e700 1 --2- 172.21.15.23:0/3861344329 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f98dc056d60 0x7f98dc059220 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:30.783 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.778+0000 7f990c37e700 1 --2- 172.21.15.23:0/3861344329 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9904061380 0x7f99040acd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:30.783 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.778+0000 7f990c37e700 1 --2- 172.21.15.23:0/3861344329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99040a3f80 0x7f990412fb10 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:30.784 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.778+0000 7f990c37e700 1 -- 172.21.15.23:0/3861344329 >> 172.21.15.23:0/3861344329 conn(0x7f99040ae280 msgr2=0x7f99041347a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:30.784 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.778+0000 7f990c37e700 1 -- 172.21.15.23:0/3861344329 shutdown_connections 2020-11-20T11:39:30.784 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:30.778+0000 7f990c37e700 1 -- 172.21.15.23:0/3861344329 wait complete. 2020-11-20T11:39:30.850 INFO:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:39:30.853 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:39:33.796 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:39:33.931 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.928+0000 7f328e540700 1 Processor -- start 2020-11-20T11:39:33.932 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.928+0000 7f328e540700 1 -- start start 2020-11-20T11:39:33.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.929+0000 7f328e540700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3288065780 0x7f32880a9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:33.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.929+0000 7f328e540700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32880a9cf0 0x7f32880601b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:33.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.929+0000 7f328e540700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3288067430 con 0x7f328805bc80 2020-11-20T11:39:33.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.929+0000 7f328e540700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32880a8ca0 con 0x7f3288065780 2020-11-20T11:39:33.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.929+0000 7f328e540700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f328805c060 con 0x7f32880a9cf0 2020-11-20T11:39:33.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.929+0000 7f3287fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f328805bc80 0x7f32881379e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58128/0 (socket says 172.21.15.23:58128) 2020-11-20T11:39:33.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.929+0000 7f3287fff700 1 -- 172.21.15.23:0/2889807283 learned_addr learned my addr 172.21.15.23:0/2889807283 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:39:33.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.930+0000 7f328cadd700 1 --2- 172.21.15.23:0/2889807283 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32880a9cf0 0x7f32880601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:33.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.930+0000 7f32877fe700 1 --2- 172.21.15.23:0/2889807283 >> v2:172.21.15.155:3300/0 conn(0x7f3288065780 0x7f32880a9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:33.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.930+0000 7f3286ffd700 1 -- 172.21.15.23:0/2889807283 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 758162770 0 0) 0x7f3288067430 con 0x7f328805bc80 2020-11-20T11:39:33.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.930+0000 7f3286ffd700 1 -- 172.21.15.23:0/2889807283 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3274003280 con 0x7f328805bc80 2020-11-20T11:39:33.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.930+0000 7f328cadd700 1 -- 172.21.15.23:0/2889807283 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32880a9cf0 msgr2=0x7f32880601b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:39:33.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.930+0000 7f328cadd700 1 -- 172.21.15.23:0/2889807283 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32880a9cf0 msgr2=0x7f32880601b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:39:33.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.930+0000 7f328cadd700 1 --2- 172.21.15.23:0/2889807283 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32880a9cf0 0x7f32880601b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:39:33.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.930+0000 7f328cadd700 1 --2- 172.21.15.23:0/2889807283 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32880a9cf0 0x7f32880601b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:39:33.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.930+0000 7f3286ffd700 1 -- 172.21.15.23:0/2889807283 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3233607242 0 0) 0x7f3274003280 con 0x7f328805bc80 2020-11-20T11:39:33.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.930+0000 7f3286ffd700 1 -- 172.21.15.23:0/2889807283 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32880a9cf0 msgr2=0x7f32880601b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:39:33.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.930+0000 7f3286ffd700 1 --2- 172.21.15.23:0/2889807283 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32880a9cf0 0x7f32880601b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:33.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.930+0000 7f3286ffd700 1 -- 172.21.15.23:0/2889807283 >> v2:172.21.15.155:3300/0 conn(0x7f3288065780 msgr2=0x7f32880a9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:39:33.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.930+0000 7f3286ffd700 1 --2- 172.21.15.23:0/2889807283 >> v2:172.21.15.155:3300/0 conn(0x7f3288065780 0x7f32880a9140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:33.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.930+0000 7f3286ffd700 1 -- 172.21.15.23:0/2889807283 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32880a8810 con 0x7f328805bc80 2020-11-20T11:39:33.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.930+0000 7f3286ffd700 1 -- 172.21.15.23:0/2889807283 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f3278001520 con 0x7f328805bc80 2020-11-20T11:39:33.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.930+0000 7f3286ffd700 1 -- 172.21.15.23:0/2889807283 >> v1:172.21.15.23:6789/0 conn(0x7f328805bc80 legacy=0x7f32881379e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:33.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.930+0000 7f3286ffd700 1 --2- 172.21.15.23:0/2889807283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3274003770 0x7f3274003b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:33.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.930+0000 7f3286ffd700 1 --2- 172.21.15.23:0/2889807283 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32880a9cf0 0x7f3274004520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:33.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.930+0000 7f3286ffd700 1 --2- 172.21.15.23:0/2889807283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3274004a10 0x7f3274004e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:33.942 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.930+0000 7f3286ffd700 1 -- 172.21.15.23:0/2889807283 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32880a8ca0 con 0x7f3274003770 2020-11-20T11:39:33.942 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.930+0000 7f3286ffd700 1 -- 172.21.15.23:0/2889807283 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328805c060 con 0x7f3274004a10 2020-11-20T11:39:33.942 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.930+0000 7f3286ffd700 1 -- 172.21.15.23:0/2889807283 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3274004210 con 0x7f32880a9cf0 2020-11-20T11:39:33.942 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.930+0000 7f32877fe700 1 --2- 172.21.15.23:0/2889807283 >> v2:172.21.15.155:3300/0 conn(0x7f3288065780 0x7f32880a9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:39:33.942 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.930+0000 7f3287fff700 1 --2- 172.21.15.23:0/2889807283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3274003770 0x7f3274003b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:33.943 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.930+0000 7f328cadd700 1 --2- 172.21.15.23:0/2889807283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3274004a10 0x7f3274004e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:33.943 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.930+0000 7f32877fe700 1 --2- 172.21.15.23:0/2889807283 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32880a9cf0 0x7f3274004520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:33.943 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.931+0000 7f3287fff700 1 -- 172.21.15.23:0/2889807283 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32880a9cf0 msgr2=0x7f3274004520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:39:33.943 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.931+0000 7f3287fff700 1 --2- 172.21.15.23:0/2889807283 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32880a9cf0 0x7f3274004520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:33.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.931+0000 7f3287fff700 1 -- 172.21.15.23:0/2889807283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3274004a10 msgr2=0x7f3274004e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:39:33.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.931+0000 7f3287fff700 1 --2- 172.21.15.23:0/2889807283 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3274004a10 0x7f3274004e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:33.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.931+0000 7f3287fff700 1 -- 172.21.15.23:0/2889807283 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3274003280 con 0x7f3274003770 2020-11-20T11:39:33.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.931+0000 7f3287fff700 1 --2- 172.21.15.23:0/2889807283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3274003770 0x7f3274003b90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f327800c440 tx=0x7f3278007840).ready entity=mon.0 client_cookie=dcbd08a40a13de51 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:33.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.931+0000 7f3286ffd700 1 -- 172.21.15.23:0/2889807283 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32780070d0 con 0x7f3274003770 2020-11-20T11:39:33.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.931+0000 7f3286ffd700 1 -- 172.21.15.23:0/2889807283 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3278007230 con 0x7f3274003770 2020-11-20T11:39:33.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.931+0000 7f3286ffd700 1 -- 172.21.15.23:0/2889807283 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3278005630 con 0x7f3274003770 2020-11-20T11:39:33.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.932+0000 7f328e540700 1 -- 172.21.15.23:0/2889807283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3274003770 msgr2=0x7f3274003b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:33.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.932+0000 7f328e540700 1 --2- 172.21.15.23:0/2889807283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3274003770 0x7f3274003b90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f327800c440 tx=0x7f3278007840).stop 2020-11-20T11:39:33.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.932+0000 7f328cadd700 1 -- 172.21.15.23:0/2889807283 reap_dead start 2020-11-20T11:39:33.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.932+0000 7f328e540700 1 -- 172.21.15.23:0/2889807283 shutdown_connections 2020-11-20T11:39:33.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.932+0000 7f328e540700 1 -- 172.21.15.23:0/2889807283 >> 172.21.15.23:0/2889807283 conn(0x7f32880ad7a0 msgr2=0x7f32880adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:33.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.932+0000 7f328e540700 1 -- 172.21.15.23:0/2889807283 shutdown_connections 2020-11-20T11:39:33.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.932+0000 7f328e540700 1 -- 172.21.15.23:0/2889807283 wait complete. 2020-11-20T11:39:33.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.933+0000 7f328e540700 1 Processor -- start 2020-11-20T11:39:33.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.933+0000 7f328e540700 1 -- start start 2020-11-20T11:39:33.949 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.933+0000 7f328e540700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3274003770 0x7f328814d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:33.949 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.933+0000 7f328e540700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32880a2220 0x7f32880a47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:33.950 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.933+0000 7f328e540700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32880a2d90 0x7f32880a7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:33.950 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.933+0000 7f328e540700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328814d910 con 0x7f3274003770 2020-11-20T11:39:33.950 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.933+0000 7f328e540700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32880a1840 con 0x7f32880a2d90 2020-11-20T11:39:33.950 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.933+0000 7f328e540700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32880a1ae0 con 0x7f32880a2220 2020-11-20T11:39:33.951 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.934+0000 7f3287fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3274003770 0x7f328814d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:33.951 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.934+0000 7f3287fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3274003770 0x7f328814d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43890/0 (socket says 172.21.15.23:43890) 2020-11-20T11:39:33.951 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.934+0000 7f3287fff700 1 -- 172.21.15.23:0/4001903032 learned_addr learned my addr 172.21.15.23:0/4001903032 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:39:33.951 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.934+0000 7f32877fe700 1 --2- 172.21.15.23:0/4001903032 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32880a2220 0x7f32880a47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:33.952 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.934+0000 7f328cadd700 1 --2- 172.21.15.23:0/4001903032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32880a2d90 0x7f32880a7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:33.952 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.934+0000 7f3287fff700 1 -- 172.21.15.23:0/4001903032 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32880a2220 msgr2=0x7f32880a47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:33.952 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.934+0000 7f3287fff700 1 --2- 172.21.15.23:0/4001903032 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32880a2220 0x7f32880a47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:33.952 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.934+0000 7f3287fff700 1 -- 172.21.15.23:0/4001903032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32880a2d90 msgr2=0x7f32880a7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:33.952 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.934+0000 7f3287fff700 1 --2- 172.21.15.23:0/4001903032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32880a2d90 0x7f32880a7260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:33.953 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.934+0000 7f3287fff700 1 -- 172.21.15.23:0/4001903032 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32880a1d80 con 0x7f3274003770 2020-11-20T11:39:33.953 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.934+0000 7f328cadd700 1 --2- 172.21.15.23:0/4001903032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32880a2d90 0x7f32880a7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:39:33.953 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.934+0000 7f3287fff700 1 --2- 172.21.15.23:0/4001903032 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3274003770 0x7f328814d420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f327800ca30 tx=0x7f3278015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:33.953 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.934+0000 7f3267fff700 1 -- 172.21.15.23:0/4001903032 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3278007570 con 0x7f3274003770 2020-11-20T11:39:33.953 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.934+0000 7f3267fff700 1 -- 172.21.15.23:0/4001903032 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3278001630 con 0x7f3274003770 2020-11-20T11:39:33.954 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.934+0000 7f328e540700 1 -- 172.21.15.23:0/4001903032 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f328814e2b0 con 0x7f3274003770 2020-11-20T11:39:33.954 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.934+0000 7f3267fff700 1 -- 172.21.15.23:0/4001903032 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3278013660 con 0x7f3274003770 2020-11-20T11:39:33.954 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.934+0000 7f328e540700 1 -- 172.21.15.23:0/4001903032 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32880a0be0 con 0x7f3274003770 2020-11-20T11:39:33.954 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.935+0000 7f328e540700 1 -- 172.21.15.23:0/4001903032 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32880a1f10 con 0x7f3274003770 2020-11-20T11:39:33.954 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.936+0000 7f3267fff700 1 -- 172.21.15.23:0/4001903032 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f327801d070 con 0x7f3274003770 2020-11-20T11:39:33.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.936+0000 7f3267fff700 1 --2- 172.21.15.23:0/4001903032 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3260056d60 0x7f3260059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:33.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.936+0000 7f3267fff700 1 -- 172.21.15.23:0/4001903032 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f3278071120 con 0x7f3274003770 2020-11-20T11:39:33.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.936+0000 7f32877fe700 1 --2- 172.21.15.23:0/4001903032 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3260056d60 0x7f3260059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:33.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.937+0000 7f32877fe700 1 --2- 172.21.15.23:0/4001903032 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3260056d60 0x7f3260059220 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f32880a12c0 tx=0x7f327c008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:33.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:33.940+0000 7f3267fff700 1 -- 172.21.15.23:0/4001903032 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3278046410 con 0x7f3274003770 2020-11-20T11:39:34.197 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:34.193+0000 7f328e540700 1 -- 172.21.15.23:0/4001903032 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3288062fe0 con 0x7f3274003770 2020-11-20T11:39:34.197 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:34.194+0000 7f3267fff700 1 -- 172.21.15.23:0/4001903032 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1472 (secure 0 0 0) 0x7f327801a080 con 0x7f3274003770 2020-11-20T11:39:34.197 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":70,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:39:34.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:34.196+0000 7f328e540700 1 -- 172.21.15.23:0/4001903032 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3260056d60 msgr2=0x7f3260059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:34.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:34.196+0000 7f328e540700 1 --2- 172.21.15.23:0/4001903032 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3260056d60 0x7f3260059220 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f32880a12c0 tx=0x7f327c008040).stop 2020-11-20T11:39:34.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:34.196+0000 7f328e540700 1 -- 172.21.15.23:0/4001903032 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3274003770 msgr2=0x7f328814d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:34.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:34.196+0000 7f328e540700 1 --2- 172.21.15.23:0/4001903032 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3274003770 0x7f328814d420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f327800ca30 tx=0x7f3278015000).stop 2020-11-20T11:39:34.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:34.196+0000 7f328e540700 1 -- 172.21.15.23:0/4001903032 shutdown_connections 2020-11-20T11:39:34.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:34.196+0000 7f328e540700 1 --2- 172.21.15.23:0/4001903032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32880a2d90 0x7f32880a7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:34.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:34.196+0000 7f328e540700 1 --2- 172.21.15.23:0/4001903032 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3274003770 0x7f328814d420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:34.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:34.196+0000 7f328e540700 1 --2- 172.21.15.23:0/4001903032 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3260056d60 0x7f3260059220 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:34.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:34.196+0000 7f328e540700 1 --2- 172.21.15.23:0/4001903032 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32880a2220 0x7f32880a47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:34.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:34.196+0000 7f328e540700 1 -- 172.21.15.23:0/4001903032 >> 172.21.15.23:0/4001903032 conn(0x7f32880ad7a0 msgr2=0x7f3288060230 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:34.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:34.196+0000 7f328e540700 1 -- 172.21.15.23:0/4001903032 shutdown_connections 2020-11-20T11:39:34.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:34.196+0000 7f328e540700 1 -- 172.21.15.23:0/4001903032 wait complete. 2020-11-20T11:39:37.212 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:39:37.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.334+0000 7f5815f85700 1 Processor -- start 2020-11-20T11:39:37.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.334+0000 7f5815f85700 1 -- start start 2020-11-20T11:39:37.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.335+0000 7f5815f85700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5810065780 0x7f58100a9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:37.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.335+0000 7f5815f85700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5810067430 con 0x7f581005bc80 2020-11-20T11:39:37.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.335+0000 7f5815f85700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5810138800 con 0x7f58100a9cf0 2020-11-20T11:39:37.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.335+0000 7f5815f85700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58100a8ca0 con 0x7f5810065780 2020-11-20T11:39:37.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.335+0000 7f580effd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5810065780 0x7f58100a9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:37.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.335+0000 7f580f7fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f581005bc80 0x7f58101379e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58150/0 (socket says 172.21.15.23:58150) 2020-11-20T11:39:37.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.335+0000 7f580f7fe700 1 -- 172.21.15.23:0/246944346 learned_addr learned my addr 172.21.15.23:0/246944346 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:39:37.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.335+0000 7f580effd700 1 -- 172.21.15.23:0/246944346 >> v1:172.21.15.155:6789/0 conn(0x7f58100a9cf0 legacy=0x7f58101380f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:39:37.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.335+0000 7f580effd700 1 -- 172.21.15.23:0/246944346 >> v1:172.21.15.23:6789/0 conn(0x7f581005bc80 legacy=0x7f58101379e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:37.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.335+0000 7f580effd700 1 -- 172.21.15.23:0/246944346 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f581005c060 con 0x7f5810065780 2020-11-20T11:39:37.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.335+0000 7f580e7fc700 1 -- 172.21.15.23:0/246944346 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2091897024 0 0) 0x7f5810067430 con 0x7f581005bc80 2020-11-20T11:39:37.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.335+0000 7f580effd700 1 --2- 172.21.15.23:0/246944346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5810065780 0x7f58100a9140 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f5800006fd0 tx=0x7f5800006e30).ready entity=mon.0 client_cookie=4ee5a7ed53526aae server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:37.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.336+0000 7f580e7fc700 1 -- 172.21.15.23:0/246944346 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58000097a0 con 0x7f5810065780 2020-11-20T11:39:37.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.336+0000 7f580e7fc700 1 -- 172.21.15.23:0/246944346 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5800000e90 con 0x7f5810065780 2020-11-20T11:39:37.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.336+0000 7f580e7fc700 1 -- 172.21.15.23:0/246944346 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5800009b10 con 0x7f5810065780 2020-11-20T11:39:37.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.336+0000 7f5815f85700 1 -- 172.21.15.23:0/246944346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5810065780 msgr2=0x7f58100a9140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:37.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.336+0000 7f5815f85700 1 --2- 172.21.15.23:0/246944346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5810065780 0x7f58100a9140 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f5800006fd0 tx=0x7f5800006e30).stop 2020-11-20T11:39:37.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.336+0000 7f5815f85700 1 -- 172.21.15.23:0/246944346 shutdown_connections 2020-11-20T11:39:37.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.336+0000 7f5815f85700 1 --2- 172.21.15.23:0/246944346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5810065780 0x7f58100a9140 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:37.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.336+0000 7f5815f85700 1 -- 172.21.15.23:0/246944346 >> 172.21.15.23:0/246944346 conn(0x7f58100ad7a0 msgr2=0x7f58100adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:37.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.336+0000 7f5815f85700 1 -- 172.21.15.23:0/246944346 shutdown_connections 2020-11-20T11:39:37.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.336+0000 7f5815f85700 1 -- 172.21.15.23:0/246944346 wait complete. 2020-11-20T11:39:37.346 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.338+0000 7f5815f85700 1 Processor -- start 2020-11-20T11:39:37.346 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.338+0000 7f5815f85700 1 -- start start 2020-11-20T11:39:37.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.338+0000 7f5815f85700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f581005bc80 0x7f58100a4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:37.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.338+0000 7f5815f85700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5810065780 0x7f58100a03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:37.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.338+0000 7f5815f85700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58100a9cf0 0x7f581009e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:37.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.338+0000 7f5815f85700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5810138b40 con 0x7f58100a9cf0 2020-11-20T11:39:37.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.339+0000 7f5815f85700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f581009d4a0 con 0x7f5810065780 2020-11-20T11:39:37.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.339+0000 7f5815f85700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f581009d740 con 0x7f581005bc80 2020-11-20T11:39:37.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.339+0000 7f580ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58100a9cf0 0x7f581009e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:37.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.339+0000 7f580ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58100a9cf0 0x7f581009e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43910/0 (socket says 172.21.15.23:43910) 2020-11-20T11:39:37.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.339+0000 7f580ffff700 1 -- 172.21.15.23:0/3755270712 learned_addr learned my addr 172.21.15.23:0/3755270712 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:39:37.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.339+0000 7f580effd700 1 --2- 172.21.15.23:0/3755270712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5810065780 0x7f58100a03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:37.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.339+0000 7f580f7fe700 1 --2- 172.21.15.23:0/3755270712 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f581005bc80 0x7f58100a4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:37.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.339+0000 7f580f7fe700 1 -- 172.21.15.23:0/3755270712 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f581005bc80 msgr2=0x7f58100a4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2020-11-20T11:39:37.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.339+0000 7f580f7fe700 1 -- 172.21.15.23:0/3755270712 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f581005bc80 msgr2=0x7f58100a4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:39:37.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.339+0000 7f580f7fe700 1 --2- 172.21.15.23:0/3755270712 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f581005bc80 0x7f58100a4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:39:37.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.339+0000 7f580ffff700 1 -- 172.21.15.23:0/3755270712 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f581005bc80 msgr2=0x7f58100a4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:37.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.339+0000 7f580f7fe700 1 --2- 172.21.15.23:0/3755270712 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f581005bc80 0x7f58100a4330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:39:37.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.339+0000 7f580ffff700 1 --2- 172.21.15.23:0/3755270712 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f581005bc80 0x7f58100a4330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:37.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.339+0000 7f580ffff700 1 -- 172.21.15.23:0/3755270712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5810065780 msgr2=0x7f58100a03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:37.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.339+0000 7f580ffff700 1 --2- 172.21.15.23:0/3755270712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5810065780 0x7f58100a03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:37.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.339+0000 7f580ffff700 1 -- 172.21.15.23:0/3755270712 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f581009d9e0 con 0x7f58100a9cf0 2020-11-20T11:39:37.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.339+0000 7f580ffff700 1 --2- 172.21.15.23:0/3755270712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58100a9cf0 0x7f581009e430 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f580800bae0 tx=0x7f5808009120).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:37.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.339+0000 7f57ef7fe700 1 -- 172.21.15.23:0/3755270712 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580800c740 con 0x7f58100a9cf0 2020-11-20T11:39:37.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.339+0000 7f57ef7fe700 1 -- 172.21.15.23:0/3755270712 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f580800c8c0 con 0x7f58100a9cf0 2020-11-20T11:39:37.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.339+0000 7f57ef7fe700 1 -- 172.21.15.23:0/3755270712 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580800ca70 con 0x7f58100a9cf0 2020-11-20T11:39:37.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.339+0000 7f5815f85700 1 -- 172.21.15.23:0/3755270712 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58100a7a80 con 0x7f58100a9cf0 2020-11-20T11:39:37.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.340+0000 7f5815f85700 1 -- 172.21.15.23:0/3755270712 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f581009eae0 con 0x7f58100a9cf0 2020-11-20T11:39:37.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.340+0000 7f5815f85700 1 -- 172.21.15.23:0/3755270712 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f581009ed90 con 0x7f58100a9cf0 2020-11-20T11:39:37.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.341+0000 7f57ef7fe700 1 -- 172.21.15.23:0/3755270712 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f580801c070 con 0x7f58100a9cf0 2020-11-20T11:39:37.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.341+0000 7f57ef7fe700 1 --2- 172.21.15.23:0/3755270712 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f57e8056d60 0x7f57e8059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:37.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.341+0000 7f57ef7fe700 1 -- 172.21.15.23:0/3755270712 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f580806f770 con 0x7f58100a9cf0 2020-11-20T11:39:37.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.342+0000 7f580f7fe700 1 --2- 172.21.15.23:0/3755270712 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f57e8056d60 0x7f57e8059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:37.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.342+0000 7f580f7fe700 1 --2- 172.21.15.23:0/3755270712 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f57e8056d60 0x7f57e8059220 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f58040011e0 tx=0x7f5804009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:37.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.347+0000 7f57ef7fe700 1 -- 172.21.15.23:0/3755270712 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5808044a60 con 0x7f58100a9cf0 2020-11-20T11:39:37.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.585+0000 7f5815f85700 1 -- 172.21.15.23:0/3755270712 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5810062fe0 con 0x7f58100a9cf0 2020-11-20T11:39:37.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.585+0000 7f57ef7fe700 1 -- 172.21.15.23:0/3755270712 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1477 (secure 0 0 0) 0x7f5808009240 con 0x7f58100a9cf0 2020-11-20T11:39:37.589 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":72,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:39:37.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.587+0000 7f5815f85700 1 -- 172.21.15.23:0/3755270712 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f57e8056d60 msgr2=0x7f57e8059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:37.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.587+0000 7f5815f85700 1 --2- 172.21.15.23:0/3755270712 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f57e8056d60 0x7f57e8059220 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f58040011e0 tx=0x7f5804009040).stop 2020-11-20T11:39:37.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.587+0000 7f5815f85700 1 -- 172.21.15.23:0/3755270712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58100a9cf0 msgr2=0x7f581009e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:37.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.587+0000 7f5815f85700 1 --2- 172.21.15.23:0/3755270712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58100a9cf0 0x7f581009e430 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f580800bae0 tx=0x7f5808009120).stop 2020-11-20T11:39:37.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.587+0000 7f5815f85700 1 -- 172.21.15.23:0/3755270712 shutdown_connections 2020-11-20T11:39:37.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.587+0000 7f5815f85700 1 --2- 172.21.15.23:0/3755270712 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f58100a9cf0 0x7f581009e430 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:37.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.587+0000 7f5815f85700 1 --2- 172.21.15.23:0/3755270712 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f57e8056d60 0x7f57e8059220 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:37.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.587+0000 7f5815f85700 1 --2- 172.21.15.23:0/3755270712 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f581005bc80 0x7f58100a4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:37.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.587+0000 7f5815f85700 1 --2- 172.21.15.23:0/3755270712 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5810065780 0x7f58100a03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:37.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.587+0000 7f5815f85700 1 -- 172.21.15.23:0/3755270712 >> 172.21.15.23:0/3755270712 conn(0x7f58100ad7a0 msgr2=0x7f5810138cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:37.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.587+0000 7f5815f85700 1 -- 172.21.15.23:0/3755270712 shutdown_connections 2020-11-20T11:39:37.593 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.587+0000 7f5815f85700 1 -- 172.21.15.23:0/3755270712 wait complete. 2020-11-20T11:39:37.608 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:39:37.608 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:39:37.732 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.729+0000 7f90963ae700 1 Processor -- start 2020-11-20T11:39:37.732 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.729+0000 7f90963ae700 1 -- start start 2020-11-20T11:39:37.734 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.730+0000 7f90963ae700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f90900a8f90 0x7f90900a6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:37.735 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.730+0000 7f90963ae700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90900a68f0 0x7f90900a6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:37.735 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.730+0000 7f90963ae700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90900a5000 con 0x7f9090061380 2020-11-20T11:39:37.735 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.730+0000 7f90963ae700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90900a5d30 con 0x7f90900a8f90 2020-11-20T11:39:37.735 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.730+0000 7f90963ae700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90900a5fd0 con 0x7f90900a68f0 2020-11-20T11:39:37.735 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.730+0000 7f9094bab700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f90900a8f90 0x7f90900a6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:37.736 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.730+0000 7f90953ac700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f9090061380 0x7f90900a8880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58164/0 (socket says 172.21.15.23:58164) 2020-11-20T11:39:37.736 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.730+0000 7f90953ac700 1 -- 172.21.15.23:0/1301804472 learned_addr learned my addr 172.21.15.23:0/1301804472 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:39:37.736 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.730+0000 7f9094bab700 1 -- 172.21.15.23:0/1301804472 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90900a68f0 msgr2=0x7f90900a6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:39:37.736 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.730+0000 7f9094bab700 1 --2- 172.21.15.23:0/1301804472 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90900a68f0 0x7f90900a6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:39:37.736 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.730+0000 7f9094bab700 1 -- 172.21.15.23:0/1301804472 >> v1:172.21.15.23:6789/0 conn(0x7f9090061380 legacy=0x7f90900a8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:37.737 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.730+0000 7f9094bab700 1 -- 172.21.15.23:0/1301804472 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90900a8140 con 0x7f90900a8f90 2020-11-20T11:39:37.737 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.730+0000 7f9094bab700 1 --2- 172.21.15.23:0/1301804472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90900a8f90 0x7f90900a6400 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f9084006fd0 tx=0x7f9084006e30).ready entity=mon.0 client_cookie=71eb1f5b3f04a00c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:37.737 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.730+0000 7f908ffff700 1 -- 172.21.15.23:0/1301804472 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90840097e0 con 0x7f90900a8f90 2020-11-20T11:39:37.737 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.730+0000 7f908ffff700 1 -- 172.21.15.23:0/1301804472 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9084000e90 con 0x7f90900a8f90 2020-11-20T11:39:37.738 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.731+0000 7f908ffff700 1 -- 172.21.15.23:0/1301804472 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084009b50 con 0x7f90900a8f90 2020-11-20T11:39:37.738 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.731+0000 7f90963ae700 1 -- 172.21.15.23:0/1301804472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90900a8f90 msgr2=0x7f90900a6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:37.738 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.731+0000 7f90963ae700 1 --2- 172.21.15.23:0/1301804472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90900a8f90 0x7f90900a6400 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f9084006fd0 tx=0x7f9084006e30).stop 2020-11-20T11:39:37.738 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.731+0000 7f90963ae700 1 -- 172.21.15.23:0/1301804472 shutdown_connections 2020-11-20T11:39:37.738 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.731+0000 7f90963ae700 1 --2- 172.21.15.23:0/1301804472 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90900a68f0 0x7f90900a6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:39:37.739 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.731+0000 7f90963ae700 1 --2- 172.21.15.23:0/1301804472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90900a8f90 0x7f90900a6400 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:37.739 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.731+0000 7f90963ae700 1 -- 172.21.15.23:0/1301804472 >> 172.21.15.23:0/1301804472 conn(0x7f90900ae0a0 msgr2=0x7f90900aa0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:37.739 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.731+0000 7f90963ae700 1 -- 172.21.15.23:0/1301804472 shutdown_connections 2020-11-20T11:39:37.739 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.731+0000 7f90963ae700 1 -- 172.21.15.23:0/1301804472 wait complete. 2020-11-20T11:39:37.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.733+0000 7f90963ae700 1 Processor -- start 2020-11-20T11:39:37.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.733+0000 7f90963ae700 1 -- start start 2020-11-20T11:39:37.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.734+0000 7f90963ae700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9090061380 0x7f909013f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:37.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.734+0000 7f90963ae700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90900a68f0 0x7f909013c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:37.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.734+0000 7f90963ae700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90900a8f90 0x7f909014aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:37.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.734+0000 7f90963ae700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909013ca30 con 0x7f9090061380 2020-11-20T11:39:37.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.734+0000 7f90963ae700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909014a330 con 0x7f90900a68f0 2020-11-20T11:39:37.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.734+0000 7f90963ae700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f909014a5d0 con 0x7f90900a8f90 2020-11-20T11:39:37.744 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.734+0000 7f90953ac700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9090061380 0x7f909013f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:37.744 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.734+0000 7f90953ac700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9090061380 0x7f909013f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43920/0 (socket says 172.21.15.23:43920) 2020-11-20T11:39:37.744 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.734+0000 7f90953ac700 1 -- 172.21.15.23:0/2028173519 learned_addr learned my addr 172.21.15.23:0/2028173519 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:39:37.744 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.734+0000 7f9094bab700 1 --2- 172.21.15.23:0/2028173519 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90900a68f0 0x7f909013c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:37.744 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.734+0000 7f9095bad700 1 --2- 172.21.15.23:0/2028173519 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90900a8f90 0x7f909014aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:37.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.734+0000 7f90953ac700 1 -- 172.21.15.23:0/2028173519 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90900a8f90 msgr2=0x7f909014aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:37.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.734+0000 7f90953ac700 1 --2- 172.21.15.23:0/2028173519 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f90900a8f90 0x7f909014aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:37.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.734+0000 7f90953ac700 1 -- 172.21.15.23:0/2028173519 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90900a68f0 msgr2=0x7f909013c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:37.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.734+0000 7f90953ac700 1 --2- 172.21.15.23:0/2028173519 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90900a68f0 0x7f909013c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:37.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.734+0000 7f90953ac700 1 -- 172.21.15.23:0/2028173519 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f909013ac60 con 0x7f9090061380 2020-11-20T11:39:37.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.734+0000 7f9094bab700 1 --2- 172.21.15.23:0/2028173519 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f90900a68f0 0x7f909013c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:39:37.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.734+0000 7f90953ac700 1 --2- 172.21.15.23:0/2028173519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9090061380 0x7f909013f350 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f908000c710 tx=0x7f9080007c80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:37.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.735+0000 7f908d7fa700 1 -- 172.21.15.23:0/2028173519 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9080007320 con 0x7f9090061380 2020-11-20T11:39:37.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.735+0000 7f908d7fa700 1 -- 172.21.15.23:0/2028173519 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9080007480 con 0x7f9090061380 2020-11-20T11:39:37.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.735+0000 7f90963ae700 1 -- 172.21.15.23:0/2028173519 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f909013c680 con 0x7f9090061380 2020-11-20T11:39:37.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.735+0000 7f908d7fa700 1 -- 172.21.15.23:0/2028173519 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9080005840 con 0x7f9090061380 2020-11-20T11:39:37.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.735+0000 7f90963ae700 1 -- 172.21.15.23:0/2028173519 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f909014b1a0 con 0x7f9090061380 2020-11-20T11:39:37.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.736+0000 7f90963ae700 1 --2- 172.21.15.23:0/2028173519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f909009db10 0x7f909013b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:37.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.736+0000 7f90963ae700 1 -- 172.21.15.23:0/2028173519 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9090069c70 con 0x7f909009db10 2020-11-20T11:39:37.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.736+0000 7f9094bab700 1 --2- 172.21.15.23:0/2028173519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f909009db10 0x7f909013b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:37.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.736+0000 7f908d7fa700 1 -- 172.21.15.23:0/2028173519 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9080010030 con 0x7f9090061380 2020-11-20T11:39:37.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.736+0000 7f9094bab700 1 -- 172.21.15.23:0/2028173519 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f909014a790 con 0x7f909009db10 2020-11-20T11:39:37.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.736+0000 7f9094bab700 1 --2- 172.21.15.23:0/2028173519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f909009db10 0x7f909013b130 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f9084006fd0 tx=0x7f9084015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:37.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.736+0000 7f908d7fa700 1 --2- 172.21.15.23:0/2028173519 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9068056d90 0x7f9068059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:37.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.736+0000 7f908d7fa700 1 -- 172.21.15.23:0/2028173519 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f908006fd70 con 0x7f9090061380 2020-11-20T11:39:37.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.736+0000 7f908d7fa700 1 -- 172.21.15.23:0/2028173519 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084011bf0 con 0x7f909009db10 2020-11-20T11:39:37.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.737+0000 7f9095bad700 1 --2- 172.21.15.23:0/2028173519 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9068056d90 0x7f9068059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:37.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.737+0000 7f908d7fa700 1 -- 172.21.15.23:0/2028173519 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f9084011d50 con 0x7f909009db10 2020-11-20T11:39:37.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.737+0000 7f908d7fa700 1 -- 172.21.15.23:0/2028173519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f909009db10 msgr2=0x7f909013b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:37.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.737+0000 7f908d7fa700 1 --2- 172.21.15.23:0/2028173519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f909009db10 0x7f909013b130 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f9084006fd0 tx=0x7f9084015000).stop 2020-11-20T11:39:37.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.737+0000 7f908d7fa700 1 -- 172.21.15.23:0/2028173519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f909009db10 msgr2=0x7f909013b130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:39:37.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.737+0000 7f908d7fa700 1 --2- 172.21.15.23:0/2028173519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f909009db10 0x7f909013b130 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:37.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.737+0000 7f9095bad700 1 --2- 172.21.15.23:0/2028173519 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9068056d90 0x7f9068059250 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f909014a260 tx=0x7f9088008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:37.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.747+0000 7f90963ae700 1 --2- 172.21.15.23:0/2028173519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9090061980 0x7f909013b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:37.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.747+0000 7f90963ae700 1 -- 172.21.15.23:0/2028173519 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9090057dc0 con 0x7f9090061980 2020-11-20T11:39:37.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.747+0000 7f9094bab700 1 --2- 172.21.15.23:0/2028173519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9090061980 0x7f909013b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:37.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.748+0000 7f9094bab700 1 -- 172.21.15.23:0/2028173519 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f909013bb10 con 0x7f9090061980 2020-11-20T11:39:37.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.748+0000 7f9094bab700 1 --2- 172.21.15.23:0/2028173519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9090061980 0x7f909013b620 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f908400b190 tx=0x7f9084020000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:37.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.748+0000 7f908d7fa700 1 -- 172.21.15.23:0/2028173519 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9084028a10 con 0x7f9090061980 2020-11-20T11:39:37.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.749+0000 7f908d7fa700 1 -- 172.21.15.23:0/2028173519 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4445 (secure 0 0 0) 0x7f90840119e0 con 0x7f9090061980 2020-11-20T11:39:37.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.749+0000 7f908d7fa700 1 -- 172.21.15.23:0/2028173519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9090061980 msgr2=0x7f909013b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:37.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.749+0000 7f908d7fa700 1 --2- 172.21.15.23:0/2028173519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9090061980 0x7f909013b620 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f908400b190 tx=0x7f9084020000).stop 2020-11-20T11:39:37.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.749+0000 7f908d7fa700 1 -- 172.21.15.23:0/2028173519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9090061980 msgr2=0x7f909013b620 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:39:37.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.749+0000 7f908d7fa700 1 --2- 172.21.15.23:0/2028173519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9090061980 0x7f909013b620 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:37.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.749+0000 7f90963ae700 1 -- 172.21.15.23:0/2028173519 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9068056d90 msgr2=0x7f9068059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:37.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.749+0000 7f90963ae700 1 --2- 172.21.15.23:0/2028173519 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9068056d90 0x7f9068059250 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f909014a260 tx=0x7f9088008040).stop 2020-11-20T11:39:37.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.749+0000 7f90963ae700 1 -- 172.21.15.23:0/2028173519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9090061380 msgr2=0x7f909013f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:37.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.749+0000 7f90963ae700 1 --2- 172.21.15.23:0/2028173519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9090061380 0x7f909013f350 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f908000c710 tx=0x7f9080007c80).stop 2020-11-20T11:39:37.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.749+0000 7f9095bad700 1 -- 172.21.15.23:0/2028173519 reap_dead start 2020-11-20T11:39:37.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.749+0000 7f9095bad700 1 -- 172.21.15.23:0/2028173519 reap_dead start 2020-11-20T11:39:37.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.749+0000 7f90963ae700 1 -- 172.21.15.23:0/2028173519 shutdown_connections 2020-11-20T11:39:37.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.749+0000 7f90963ae700 1 -- 172.21.15.23:0/2028173519 >> 172.21.15.23:0/2028173519 conn(0x7f90900ae0a0 msgr2=0x7f90901402c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:37.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.749+0000 7f90963ae700 1 -- 172.21.15.23:0/2028173519 shutdown_connections 2020-11-20T11:39:37.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.749+0000 7f90963ae700 1 -- 172.21.15.23:0/2028173519 wait complete. 2020-11-20T11:39:37.757 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:39:37.757 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:39:37.757 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:39:37.757 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:39:37.758 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 72, 2020-11-20T11:39:37.758 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:39:37.758 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:39:37.758 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:39:37.758 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:39:37.759 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:37.759 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 1, 2020-11-20T11:39:37.759 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:39:37.759 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:39:37.759 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:39:37.760 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:39:37.760 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:39:37.760 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:39:37.760 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:39:37.760 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:39:37.761 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:39:37.761 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:39:37.761 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:39:37.761 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:37.761 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:39:37.762 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:39:37.762 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:39:37.762 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:39:37.762 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:39:37.762 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:39:37.763 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:39:37.763 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:39:37.763 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:39:37.763 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:39:37.763 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:37.764 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:37.764 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:39:37.764 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:39:37.764 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:39:37.764 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:39:37.765 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:39:37.765 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:39:37.765 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:39:37.765 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:39:37.766 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:39:37.766 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:39:37.766 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:39:37.766 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:39:37.766 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:39:37.766 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:39:37.767 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:39:37.767 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:39:37.767 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:39:37.767 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:39:37.767 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:39:37.768 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:39:37.768 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:39:37.768 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:39:37.768 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:39:37.768 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:37.769 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:39:37.769 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:37.769 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:39:37.769 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:37.769 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:39:37.770 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:39:37.770 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:39:37.770 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:39:37.770 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:37.770 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:39:37.771 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:39:37.771 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:37.771 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:37.771 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:37.771 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:39:37.772 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:39:37.772 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:37.772 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:37.772 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:37.772 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:37.773 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:39:37.773 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:39:37.773 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:39:37.773 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:39:37.773 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:39:37.774 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:37.774 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:37.774 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:39:37.774 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:39:37.774 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:39:37.775 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:39:37.775 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:37.775 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:39:37.775 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:39:37.775 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:37.775 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:37.776 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:37.776 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:39:37.776 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:39:37.776 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:37.776 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:37.777 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:37.777 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:37.777 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:39:37.777 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:39:37.777 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:39:37.778 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:39:37.778 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:39:37.778 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:37.778 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:37.778 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:39:37.779 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:39:37.779 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:39:37.779 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:39:37.779 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:37.780 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:39:37.780 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:39:37.780 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:37.780 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:37.780 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:37.780 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:39:37.781 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:39:37.781 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:37.781 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:37.781 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:37.782 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:37.782 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:39:37.782 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:39:37.782 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:39:37.782 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:39:37.783 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:39:37.783 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:37.783 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:37.783 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:37.783 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:39:37.784 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:39:37.784 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:37.784 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:39:37.784 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:39:37.784 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:39:37.785 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:37.785 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:37.785 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:39:37.785 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:37.785 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:39:37.786 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:39:37.786 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:39:37.786 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:37.786 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:37.786 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:39:37.787 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:37.787 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:39:37.787 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:39:37.787 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:39:37.787 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:37.788 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:37.788 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:37.788 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:39:37.788 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:39:37.789 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:39:37.917 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.914+0000 7fc1c1ebf700 1 Processor -- start 2020-11-20T11:39:37.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.914+0000 7fc1c1ebf700 1 -- start start 2020-11-20T11:39:37.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.915+0000 7fc1c1ebf700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc1bc0aaab0 0x7fc1bc13ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:37.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.915+0000 7fc1c1ebf700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1bc0aae90 0x7fc1bc1381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:37.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.915+0000 7fc1c1ebf700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc1bc13aeb0 0x7fc1bc0a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:37.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.915+0000 7fc1c1ebf700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1bc13e600 con 0x7fc1bc0aaab0 2020-11-20T11:39:37.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.915+0000 7fc1c1ebf700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1bc13e8a0 con 0x7fc1bc13aeb0 2020-11-20T11:39:37.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.915+0000 7fc1c1ebf700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1bc0ab6d0 con 0x7fc1bc0aae90 2020-11-20T11:39:37.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.915+0000 7fc1bb7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc1bc0aaab0 0x7fc1bc13ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:37.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.915+0000 7fc1bb7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc1bc0aaab0 0x7fc1bc13ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43932/0 (socket says 172.21.15.23:43932) 2020-11-20T11:39:37.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.915+0000 7fc1bb7fe700 1 -- 172.21.15.23:0/1189961500 learned_addr learned my addr 172.21.15.23:0/1189961500 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:39:37.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.915+0000 7fc1bbfff700 1 --2- 172.21.15.23:0/1189961500 >> v2:172.21.15.155:3300/0 conn(0x7fc1bc13aeb0 0x7fc1bc0a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:37.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.915+0000 7fc1baffd700 1 --2- 172.21.15.23:0/1189961500 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1bc0aae90 0x7fc1bc1381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:37.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.916+0000 7fc1baffd700 1 -- 172.21.15.23:0/1189961500 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1bc0aae90 msgr2=0x7fc1bc1381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:39:37.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.916+0000 7fc1baffd700 1 -- 172.21.15.23:0/1189961500 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1bc0aae90 msgr2=0x7fc1bc1381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:39:37.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.916+0000 7fc1bb7fe700 1 -- 172.21.15.23:0/1189961500 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1bc0aae90 msgr2=0x7fc1bc1381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:39:37.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.916+0000 7fc1baffd700 1 --2- 172.21.15.23:0/1189961500 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1bc0aae90 0x7fc1bc1381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:39:37.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.916+0000 7fc1baffd700 1 --2- 172.21.15.23:0/1189961500 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1bc0aae90 0x7fc1bc1381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:39:37.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.916+0000 7fc1bb7fe700 1 --2- 172.21.15.23:0/1189961500 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1bc0aae90 0x7fc1bc1381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:37.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.916+0000 7fc1bb7fe700 1 -- 172.21.15.23:0/1189961500 >> v2:172.21.15.155:3300/0 conn(0x7fc1bc13aeb0 msgr2=0x7fc1bc0a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:39:37.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.916+0000 7fc1bb7fe700 1 --2- 172.21.15.23:0/1189961500 >> v2:172.21.15.155:3300/0 conn(0x7fc1bc13aeb0 0x7fc1bc0a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:37.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.916+0000 7fc1bb7fe700 1 -- 172.21.15.23:0/1189961500 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1bc13f230 con 0x7fc1bc0aaab0 2020-11-20T11:39:37.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.916+0000 7fc1bb7fe700 1 --2- 172.21.15.23:0/1189961500 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1bc0aaab0 0x7fc1bc13ed40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fc1bc0acf30 tx=0x7fc1ac006ef0).ready entity=mon.0 client_cookie=65d347208c7c722 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:37.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.916+0000 7fc1ba7fc700 1 -- 172.21.15.23:0/1189961500 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac00b860 con 0x7fc1bc0aaab0 2020-11-20T11:39:37.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.916+0000 7fc1ba7fc700 1 -- 172.21.15.23:0/1189961500 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1ac000e90 con 0x7fc1bc0aaab0 2020-11-20T11:39:37.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.916+0000 7fc1ba7fc700 1 -- 172.21.15.23:0/1189961500 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac00bbd0 con 0x7fc1bc0aaab0 2020-11-20T11:39:37.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.916+0000 7fc1c1ebf700 1 -- 172.21.15.23:0/1189961500 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1bc0aaab0 msgr2=0x7fc1bc13ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:37.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.916+0000 7fc1c1ebf700 1 --2- 172.21.15.23:0/1189961500 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1bc0aaab0 0x7fc1bc13ed40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fc1bc0acf30 tx=0x7fc1ac006ef0).stop 2020-11-20T11:39:37.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.916+0000 7fc1c1ebf700 1 -- 172.21.15.23:0/1189961500 shutdown_connections 2020-11-20T11:39:37.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.916+0000 7fc1c1ebf700 1 --2- 172.21.15.23:0/1189961500 >> v2:172.21.15.155:3300/0 conn(0x7fc1bc13aeb0 0x7fc1bc0a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:37.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.917+0000 7fc1c1ebf700 1 --2- 172.21.15.23:0/1189961500 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1bc0aaab0 0x7fc1bc13ed40 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:37.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.917+0000 7fc1c1ebf700 1 --2- 172.21.15.23:0/1189961500 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1bc0aae90 0x7fc1bc1381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:37.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.917+0000 7fc1c1ebf700 1 -- 172.21.15.23:0/1189961500 >> 172.21.15.23:0/1189961500 conn(0x7fc1bc0678e0 msgr2=0x7fc1bc067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:37.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.917+0000 7fc1c1ebf700 1 -- 172.21.15.23:0/1189961500 shutdown_connections 2020-11-20T11:39:37.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.917+0000 7fc1c1ebf700 1 -- 172.21.15.23:0/1189961500 wait complete. 2020-11-20T11:39:37.930 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.918+0000 7fc1c1ebf700 1 Processor -- start 2020-11-20T11:39:37.931 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.918+0000 7fc1c1ebf700 1 -- start start 2020-11-20T11:39:37.931 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.918+0000 7fc1c1ebf700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1bc0aaab0 0x7fc1bc0a5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:37.931 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.918+0000 7fc1c1ebf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1bc0aae90 0x7fc1bc0a7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:37.931 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.918+0000 7fc1c1ebf700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1bc0a6010 0x7fc1bc0a6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:37.931 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.918+0000 7fc1c1ebf700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1bc1377b0 con 0x7fc1bc0aae90 2020-11-20T11:39:37.932 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.918+0000 7fc1c1ebf700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1bc0a80a0 con 0x7fc1bc0aaab0 2020-11-20T11:39:37.932 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.918+0000 7fc1c1ebf700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1bc1379e0 con 0x7fc1bc0a6010 2020-11-20T11:39:37.932 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.918+0000 7fc1baffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1bc0aae90 0x7fc1bc0a7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:37.932 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.919+0000 7fc1baffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1bc0aae90 0x7fc1bc0a7bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43940/0 (socket says 172.21.15.23:43940) 2020-11-20T11:39:37.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.919+0000 7fc1baffd700 1 -- 172.21.15.23:0/2878420656 learned_addr learned my addr 172.21.15.23:0/2878420656 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:39:37.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.919+0000 7fc1bbfff700 1 --2- 172.21.15.23:0/2878420656 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1bc0a6010 0x7fc1bc0a6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:37.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.919+0000 7fc1bb7fe700 1 --2- 172.21.15.23:0/2878420656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1bc0aaab0 0x7fc1bc0a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:37.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.919+0000 7fc1bbfff700 1 -- 172.21.15.23:0/2878420656 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1bc0a6010 msgr2=0x7fc1bc0a6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2020-11-20T11:39:37.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.919+0000 7fc1bbfff700 1 -- 172.21.15.23:0/2878420656 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1bc0a6010 msgr2=0x7fc1bc0a6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:39:37.934 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.919+0000 7fc1bbfff700 1 --2- 172.21.15.23:0/2878420656 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1bc0a6010 0x7fc1bc0a6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:39:37.934 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.919+0000 7fc1bbfff700 1 --2- 172.21.15.23:0/2878420656 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1bc0a6010 0x7fc1bc0a6470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:39:37.934 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.919+0000 7fc1baffd700 1 -- 172.21.15.23:0/2878420656 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1bc0a6010 msgr2=0x7fc1bc0a6470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:39:37.934 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.919+0000 7fc1baffd700 1 --2- 172.21.15.23:0/2878420656 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc1bc0a6010 0x7fc1bc0a6470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:37.934 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.919+0000 7fc1baffd700 1 -- 172.21.15.23:0/2878420656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1bc0aaab0 msgr2=0x7fc1bc0a5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:37.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.919+0000 7fc1baffd700 1 --2- 172.21.15.23:0/2878420656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1bc0aaab0 0x7fc1bc0a5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:37.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.919+0000 7fc1baffd700 1 -- 172.21.15.23:0/2878420656 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1bc137c80 con 0x7fc1bc0aae90 2020-11-20T11:39:37.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.919+0000 7fc1bb7fe700 1 --2- 172.21.15.23:0/2878420656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1bc0aaab0 0x7fc1bc0a5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:39:37.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.919+0000 7fc1baffd700 1 --2- 172.21.15.23:0/2878420656 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1bc0aae90 0x7fc1bc0a7bb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fc1b000db30 tx=0x7fc1b0008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:37.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.919+0000 7fc19b7fe700 1 -- 172.21.15.23:0/2878420656 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b00097f0 con 0x7fc1bc0aae90 2020-11-20T11:39:37.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.919+0000 7fc19b7fe700 1 -- 172.21.15.23:0/2878420656 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b0009970 con 0x7fc1bc0aae90 2020-11-20T11:39:37.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.919+0000 7fc19b7fe700 1 -- 172.21.15.23:0/2878420656 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b0009d30 con 0x7fc1bc0aae90 2020-11-20T11:39:37.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.919+0000 7fc1c1ebf700 1 -- 172.21.15.23:0/2878420656 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1bc14a2c0 con 0x7fc1bc0aae90 2020-11-20T11:39:37.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.919+0000 7fc1c1ebf700 1 -- 172.21.15.23:0/2878420656 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1bc13d5f0 con 0x7fc1bc0aae90 2020-11-20T11:39:37.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.920+0000 7fc1c1ebf700 1 --2- 172.21.15.23:0/2878420656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1bc13aeb0 0x7fc1bc14a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:37.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.920+0000 7fc1c1ebf700 1 -- 172.21.15.23:0/2878420656 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1bc05c7d0 con 0x7fc1bc13aeb0 2020-11-20T11:39:37.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.920+0000 7fc1bb7fe700 1 --2- 172.21.15.23:0/2878420656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1bc13aeb0 0x7fc1bc14a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:37.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.921+0000 7fc19b7fe700 1 -- 172.21.15.23:0/2878420656 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc1b0013690 con 0x7fc1bc0aae90 2020-11-20T11:39:37.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.921+0000 7fc1bb7fe700 1 -- 172.21.15.23:0/2878420656 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc1bc0a5cf0 con 0x7fc1bc13aeb0 2020-11-20T11:39:37.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.921+0000 7fc1bb7fe700 1 --2- 172.21.15.23:0/2878420656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1bc13aeb0 0x7fc1bc14a760 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fc1bc0acf30 tx=0x7fc1ac016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:37.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.921+0000 7fc19b7fe700 1 --2- 172.21.15.23:0/2878420656 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc194056d60 0x7fc194059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:37.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.921+0000 7fc19b7fe700 1 -- 172.21.15.23:0/2878420656 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fc1b0070470 con 0x7fc1bc0aae90 2020-11-20T11:39:37.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.921+0000 7fc19b7fe700 1 -- 172.21.15.23:0/2878420656 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac011aa0 con 0x7fc1bc13aeb0 2020-11-20T11:39:37.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.921+0000 7fc1bbfff700 1 --2- 172.21.15.23:0/2878420656 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc194056d60 0x7fc194059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:37.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.921+0000 7fc19b7fe700 1 -- 172.21.15.23:0/2878420656 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fc1ac021070 con 0x7fc1bc13aeb0 2020-11-20T11:39:37.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.921+0000 7fc19b7fe700 1 -- 172.21.15.23:0/2878420656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1bc13aeb0 msgr2=0x7fc1bc14a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:37.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.921+0000 7fc19b7fe700 1 --2- 172.21.15.23:0/2878420656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1bc13aeb0 0x7fc1bc14a760 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fc1bc0acf30 tx=0x7fc1ac016000).stop 2020-11-20T11:39:37.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.921+0000 7fc19b7fe700 1 -- 172.21.15.23:0/2878420656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1bc13aeb0 msgr2=0x7fc1bc14a760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:39:37.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.921+0000 7fc19b7fe700 1 --2- 172.21.15.23:0/2878420656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1bc13aeb0 0x7fc1bc14a760 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:37.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.922+0000 7fc1bbfff700 1 --2- 172.21.15.23:0/2878420656 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc194056d60 0x7fc194059220 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fc1bc0ae590 tx=0x7fc1b4008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:37.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.930+0000 7fc1c1ebf700 1 --2- 172.21.15.23:0/2878420656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1bc0601b0 0x7fc1bc14ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:37.942 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.930+0000 7fc1c1ebf700 1 -- 172.21.15.23:0/2878420656 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1bc044710 con 0x7fc1bc0601b0 2020-11-20T11:39:37.942 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.930+0000 7fc1bb7fe700 1 --2- 172.21.15.23:0/2878420656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1bc0601b0 0x7fc1bc14ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:37.942 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.930+0000 7fc1bb7fe700 1 -- 172.21.15.23:0/2878420656 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc1bc05c280 con 0x7fc1bc0601b0 2020-11-20T11:39:37.942 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.930+0000 7fc1bb7fe700 1 --2- 172.21.15.23:0/2878420656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1bc0601b0 0x7fc1bc14ac50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fc1bc05c960 tx=0x7fc1ac01f070).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:37.943 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.930+0000 7fc19b7fe700 1 -- 172.21.15.23:0/2878420656 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac02c070 con 0x7fc1bc0601b0 2020-11-20T11:39:37.943 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.931+0000 7fc19b7fe700 1 -- 172.21.15.23:0/2878420656 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4783 (secure 0 0 0) 0x7fc1ac027e30 con 0x7fc1bc0601b0 2020-11-20T11:39:37.943 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.931+0000 7fc19b7fe700 1 -- 172.21.15.23:0/2878420656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1bc0601b0 msgr2=0x7fc1bc14ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:37.943 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.931+0000 7fc19b7fe700 1 --2- 172.21.15.23:0/2878420656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1bc0601b0 0x7fc1bc14ac50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fc1bc05c960 tx=0x7fc1ac01f070).stop 2020-11-20T11:39:37.943 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.931+0000 7fc19b7fe700 1 -- 172.21.15.23:0/2878420656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1bc0601b0 msgr2=0x7fc1bc14ac50 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:39:37.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.931+0000 7fc19b7fe700 1 --2- 172.21.15.23:0/2878420656 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc1bc0601b0 0x7fc1bc14ac50 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:37.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.931+0000 7fc1c1ebf700 1 -- 172.21.15.23:0/2878420656 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc194056d60 msgr2=0x7fc194059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:37.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.931+0000 7fc1c1ebf700 1 --2- 172.21.15.23:0/2878420656 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc194056d60 0x7fc194059220 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fc1bc0ae590 tx=0x7fc1b4008040).stop 2020-11-20T11:39:37.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.931+0000 7fc1c1ebf700 1 -- 172.21.15.23:0/2878420656 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1bc0aae90 msgr2=0x7fc1bc0a7bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:37.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.931+0000 7fc1c1ebf700 1 --2- 172.21.15.23:0/2878420656 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1bc0aae90 0x7fc1bc0a7bb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fc1b000db30 tx=0x7fc1b0008040).stop 2020-11-20T11:39:37.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.931+0000 7fc1bbfff700 1 -- 172.21.15.23:0/2878420656 reap_dead start 2020-11-20T11:39:37.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.931+0000 7fc1bbfff700 1 -- 172.21.15.23:0/2878420656 reap_dead start 2020-11-20T11:39:37.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.931+0000 7fc1c1ebf700 1 -- 172.21.15.23:0/2878420656 shutdown_connections 2020-11-20T11:39:37.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.931+0000 7fc1c1ebf700 1 -- 172.21.15.23:0/2878420656 >> 172.21.15.23:0/2878420656 conn(0x7fc1bc0678e0 msgr2=0x7fc1bc13eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:37.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.931+0000 7fc1c1ebf700 1 -- 172.21.15.23:0/2878420656 shutdown_connections 2020-11-20T11:39:37.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:37.931+0000 7fc1c1ebf700 1 -- 172.21.15.23:0/2878420656 wait complete. 2020-11-20T11:39:37.946 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:39:37.946 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:39:37.946 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:39:37.946 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:39:37.947 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 72, 2020-11-20T11:39:37.947 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:39:37.947 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:39:37.947 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:39:37.947 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:39:37.948 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:37.948 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 1, 2020-11-20T11:39:37.948 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:39:37.948 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:39:37.948 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:39:37.949 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:39:37.949 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:39:37.949 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:39:37.949 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:39:37.950 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:39:37.950 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:39:37.950 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:39:37.950 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:39:37.950 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:37.951 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:39:37.951 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:39:37.951 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:39:37.951 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:39:37.951 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:39:37.952 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:39:37.952 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:39:37.952 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:39:37.952 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:39:37.952 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:39:37.953 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:37.953 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:37.953 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:39:37.953 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:39:37.953 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:39:37.954 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:39:37.954 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:39:37.954 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:39:37.954 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:39:37.955 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:39:37.955 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:39:37.955 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:39:37.955 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:39:37.955 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:39:37.955 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:39:37.956 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:39:37.956 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:39:37.956 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:39:37.956 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:39:37.956 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:39:37.957 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:39:37.957 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:39:37.957 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:39:37.957 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:39:37.957 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:39:37.958 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:37.958 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:39:37.958 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:37.958 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:39:37.958 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:37.959 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:39:37.959 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:39:37.959 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:39:37.959 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:39:37.959 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:37.960 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:39:37.960 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:39:37.960 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:37.960 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:37.960 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:37.961 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:39:37.961 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:39:37.961 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:37.961 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:37.961 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:37.962 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:37.962 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:39:37.962 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:39:37.962 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:39:37.962 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:39:37.963 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:39:37.963 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:37.963 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:37.963 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:39:37.963 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:39:37.964 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:39:37.964 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:39:37.964 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:37.964 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:39:37.964 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:39:37.965 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:37.965 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:37.965 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:37.965 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:39:37.965 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:39:37.966 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:37.966 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:37.966 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:37.966 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:37.966 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:39:37.967 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:39:37.967 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:39:37.967 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:39:37.967 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:39:37.967 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:37.968 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:37.968 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:39:37.968 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:39:37.968 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:39:37.968 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:39:37.969 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:37.969 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:39:37.969 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:39:37.969 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:37.969 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:37.970 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:37.970 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:39:37.970 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:39:37.970 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:37.970 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:37.971 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:37.971 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:37.971 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:39:37.971 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:39:37.971 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:39:37.972 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:39:37.972 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:39:37.972 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:37.972 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:37.972 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:37.973 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:39:37.973 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:39:37.973 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:37.973 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:39:37.973 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:39:37.974 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:39:37.974 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:37.974 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:37.974 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:39:37.974 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:37.975 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:39:37.975 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:39:37.975 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:39:37.975 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:37.975 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:37.976 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:39:37.976 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:37.976 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:39:37.976 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:39:37.976 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:39:37.977 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:37.977 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:37.977 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:39:37.977 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:37.977 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:39:37.978 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:39:37.978 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:39:37.978 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:37.978 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:37.978 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:39:37.979 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:37.979 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:39:37.979 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:39:37.979 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:39:37.979 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:37.980 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:37.980 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:37.980 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:39:37.980 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:39:37.981 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:39:38.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.120+0000 7fd0782ff700 1 Processor -- start 2020-11-20T11:39:38.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.120+0000 7fd0782ff700 1 -- start start 2020-11-20T11:39:38.152 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.121+0000 7fd0782ff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd070067eb0 0x7fd070066700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:38.152 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.121+0000 7fd0782ff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd070068430 con 0x7fd07013ae70 2020-11-20T11:39:38.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.121+0000 7fd0782ff700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0700686d0 con 0x7fd07005bc80 2020-11-20T11:39:38.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.121+0000 7fd0782ff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd070067470 con 0x7fd070067eb0 2020-11-20T11:39:38.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.121+0000 7fd07589a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd070067eb0 0x7fd070066700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:38.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.121+0000 7fd07689c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fd07013ae70 0x7fd070134cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58202/0 (socket says 172.21.15.23:58202) 2020-11-20T11:39:38.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.121+0000 7fd07689c700 1 -- 172.21.15.23:0/3109945000 learned_addr learned my addr 172.21.15.23:0/3109945000 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:39:38.155 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.122+0000 7fd07589a700 1 -- 172.21.15.23:0/3109945000 >> v1:172.21.15.155:6789/0 conn(0x7fd07005bc80 legacy=0x7fd0701345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:38.155 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.122+0000 7fd07589a700 1 -- 172.21.15.23:0/3109945000 >> v1:172.21.15.23:6789/0 conn(0x7fd07013ae70 legacy=0x7fd070134cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:38.155 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.122+0000 7fd07589a700 1 -- 172.21.15.23:0/3109945000 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd070066260 con 0x7fd070067eb0 2020-11-20T11:39:38.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.122+0000 7fd07589a700 1 --2- 172.21.15.23:0/3109945000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd070067eb0 0x7fd070066700 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fd060006fd0 tx=0x7fd060006e30).ready entity=mon.0 client_cookie=c3953e6f681a3231 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:38.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.122+0000 7fd075099700 1 -- 172.21.15.23:0/3109945000 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0600097e0 con 0x7fd070067eb0 2020-11-20T11:39:38.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.122+0000 7fd075099700 1 -- 172.21.15.23:0/3109945000 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd060000e90 con 0x7fd070067eb0 2020-11-20T11:39:38.157 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.122+0000 7fd075099700 1 -- 172.21.15.23:0/3109945000 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd060009b50 con 0x7fd070067eb0 2020-11-20T11:39:38.157 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.122+0000 7fd0782ff700 1 -- 172.21.15.23:0/3109945000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd070067eb0 msgr2=0x7fd070066700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:38.157 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.122+0000 7fd0782ff700 1 --2- 172.21.15.23:0/3109945000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd070067eb0 0x7fd070066700 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fd060006fd0 tx=0x7fd060006e30).stop 2020-11-20T11:39:38.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.123+0000 7fd0782ff700 1 -- 172.21.15.23:0/3109945000 shutdown_connections 2020-11-20T11:39:38.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.123+0000 7fd0782ff700 1 --2- 172.21.15.23:0/3109945000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd070067eb0 0x7fd070066700 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:38.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.123+0000 7fd0782ff700 1 -- 172.21.15.23:0/3109945000 >> 172.21.15.23:0/3109945000 conn(0x7fd0700ad7a0 msgr2=0x7fd0700adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:38.159 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.123+0000 7fd0782ff700 1 -- 172.21.15.23:0/3109945000 shutdown_connections 2020-11-20T11:39:38.159 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.123+0000 7fd0782ff700 1 -- 172.21.15.23:0/3109945000 wait complete. 2020-11-20T11:39:38.159 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.124+0000 7fd0782ff700 1 Processor -- start 2020-11-20T11:39:38.160 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.124+0000 7fd0782ff700 1 -- start start 2020-11-20T11:39:38.160 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.124+0000 7fd0782ff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd07005bc80 0x7fd070060680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:38.160 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.124+0000 7fd0782ff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd070067eb0 0x7fd0700ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:38.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.124+0000 7fd0782ff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07013ae70 0x7fd0700a1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:38.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.124+0000 7fd0782ff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07013b2e0 con 0x7fd070067eb0 2020-11-20T11:39:38.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.124+0000 7fd0782ff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0700abe60 con 0x7fd07005bc80 2020-11-20T11:39:38.162 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.124+0000 7fd0782ff700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0700ac100 con 0x7fd07013ae70 2020-11-20T11:39:38.163 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.125+0000 7fd07589a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd070067eb0 0x7fd0700ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:38.163 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.125+0000 7fd07589a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd070067eb0 0x7fd0700ab7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43958/0 (socket says 172.21.15.23:43958) 2020-11-20T11:39:38.163 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.125+0000 7fd07589a700 1 -- 172.21.15.23:0/3340382676 learned_addr learned my addr 172.21.15.23:0/3340382676 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:39:38.164 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.125+0000 7fd07689c700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07013ae70 0x7fd0700a1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:38.164 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.125+0000 7fd07609b700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd07005bc80 0x7fd070060680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:38.164 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.125+0000 7fd07589a700 1 -- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07013ae70 msgr2=0x7fd0700a1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:38.165 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.125+0000 7fd07589a700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07013ae70 0x7fd0700a1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:38.165 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.125+0000 7fd07589a700 1 -- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd07005bc80 msgr2=0x7fd070060680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:38.165 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.125+0000 7fd07589a700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd07005bc80 0x7fd070060680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:38.165 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.125+0000 7fd07589a700 1 -- 172.21.15.23:0/3340382676 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd070140a60 con 0x7fd070067eb0 2020-11-20T11:39:38.166 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.125+0000 7fd07609b700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd07005bc80 0x7fd070060680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:39:38.166 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.125+0000 7fd07589a700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd070067eb0 0x7fd0700ab7c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fd060003000 tx=0x7fd060015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:38.166 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.125+0000 7fd05e7fc700 1 -- 172.21.15.23:0/3340382676 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd060011df0 con 0x7fd070067eb0 2020-11-20T11:39:38.166 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.125+0000 7fd05e7fc700 1 -- 172.21.15.23:0/3340382676 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd06001e460 con 0x7fd070067eb0 2020-11-20T11:39:38.167 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.125+0000 7fd0782ff700 1 -- 172.21.15.23:0/3340382676 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd070140d10 con 0x7fd070067eb0 2020-11-20T11:39:38.167 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.125+0000 7fd05e7fc700 1 -- 172.21.15.23:0/3340382676 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06001e7e0 con 0x7fd070067eb0 2020-11-20T11:39:38.167 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.125+0000 7fd0782ff700 1 -- 172.21.15.23:0/3340382676 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd070141250 con 0x7fd070067eb0 2020-11-20T11:39:38.167 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.126+0000 7fd0782ff700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014de50 0x7fd07014e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:38.167 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.126+0000 7fd0782ff700 1 -- 172.21.15.23:0/3340382676 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0700abcb0 con 0x7fd07014de50 2020-11-20T11:39:38.167 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.126+0000 7fd07609b700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014de50 0x7fd07014e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:38.168 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.126+0000 7fd07609b700 1 -- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014de50 msgr2=0x7fd07014e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 20 2020-11-20T11:39:38.168 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.126+0000 7fd07609b700 1 -- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014de50 msgr2=0x7fd07014e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:39:38.168 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.126+0000 7fd07609b700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014de50 0x7fd07014e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:39:38.168 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.126+0000 7fd07609b700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014de50 0x7fd07014e230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:39:38.168 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.127+0000 7fd05e7fc700 1 -- 172.21.15.23:0/3340382676 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd060025030 con 0x7fd070067eb0 2020-11-20T11:39:38.169 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.127+0000 7fd05e7fc700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd048056d60 0x7fd048059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:38.169 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.127+0000 7fd05e7fc700 1 -- 172.21.15.23:0/3340382676 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fd060071970 con 0x7fd070067eb0 2020-11-20T11:39:38.169 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.127+0000 7fd07689c700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd048056d60 0x7fd048059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:38.169 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.128+0000 7fd07689c700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd048056d60 0x7fd048059220 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fd068007fd0 tx=0x7fd068009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:38.332 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.327+0000 7fd07609b700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014de50 0x7fd07014e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:38.334 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.327+0000 7fd07609b700 1 -- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014de50 msgr2=0x7fd07014e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2020-11-20T11:39:38.334 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.327+0000 7fd07609b700 1 -- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014de50 msgr2=0x7fd07014e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:39:38.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.327+0000 7fd07609b700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014de50 0x7fd07014e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:39:38.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.327+0000 7fd07609b700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014de50 0x7fd07014e230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:39:38.731 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.728+0000 7fd07609b700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014de50 0x7fd07014e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:38.732 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.728+0000 7fd07609b700 1 -- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014de50 msgr2=0x7fd07014e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2020-11-20T11:39:38.733 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.728+0000 7fd07609b700 1 -- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014de50 msgr2=0x7fd07014e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:39:38.733 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.728+0000 7fd07609b700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014de50 0x7fd07014e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:39:38.733 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:38.728+0000 7fd07609b700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014de50 0x7fd07014e230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:39:39.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:39.529+0000 7fd07609b700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014de50 0x7fd07014e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:39.533 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:39.529+0000 7fd07609b700 1 -- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014de50 msgr2=0x7fd07014e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2020-11-20T11:39:39.533 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:39.529+0000 7fd07609b700 1 -- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014de50 msgr2=0x7fd07014e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:39:39.533 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:39.529+0000 7fd07609b700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014de50 0x7fd07014e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:39:39.534 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:39.529+0000 7fd07609b700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014de50 0x7fd07014e230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:39:41.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.131+0000 7fd07609b700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014de50 0x7fd07014e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:41.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.131+0000 7fd07609b700 1 -- 172.21.15.23:0/3340382676 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd0700aaf60 con 0x7fd07014de50 2020-11-20T11:39:41.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.132+0000 7fd07609b700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014de50 0x7fd07014e230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fd06c013b50 tx=0x7fd06c009040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:41.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.132+0000 7fd05e7fc700 1 -- 172.21.15.23:0/3340382676 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06c007020 con 0x7fd07014de50 2020-11-20T11:39:41.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.132+0000 7fd05e7fc700 1 -- 172.21.15.23:0/3340382676 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fd06c00d070 con 0x7fd07014de50 2020-11-20T11:39:41.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.132+0000 7fd05e7fc700 1 -- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014de50 msgr2=0x7fd07014e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:41.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.132+0000 7fd05e7fc700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014de50 0x7fd07014e230 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fd06c013b50 tx=0x7fd06c009040).stop 2020-11-20T11:39:41.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.132+0000 7fd05e7fc700 1 -- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014de50 msgr2=0x7fd07014e230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:39:41.137 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.132+0000 7fd05e7fc700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014de50 0x7fd07014e230 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:41.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.140+0000 7fd0782ff700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014a320 0x7fd07014a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:41.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.140+0000 7fd0782ff700 1 -- 172.21.15.23:0/3340382676 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0700ac2c0 con 0x7fd07014a320 2020-11-20T11:39:41.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.141+0000 7fd07609b700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014a320 0x7fd07014a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:41.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.141+0000 7fd07609b700 1 -- 172.21.15.23:0/3340382676 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd070062fe0 con 0x7fd07014a320 2020-11-20T11:39:41.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.141+0000 7fd07609b700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014a320 0x7fd07014a730 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fd06c01c000 tx=0x7fd06c009040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:41.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.141+0000 7fd05e7fc700 1 -- 172.21.15.23:0/3340382676 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06c006cb0 con 0x7fd07014a320 2020-11-20T11:39:41.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.142+0000 7fd05e7fc700 1 -- 172.21.15.23:0/3340382676 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4273 (secure 0 0 0) 0x7fd06c006e10 con 0x7fd07014a320 2020-11-20T11:39:41.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.142+0000 7fd05e7fc700 1 -- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014a320 msgr2=0x7fd07014a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:41.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.142+0000 7fd05e7fc700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014a320 0x7fd07014a730 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fd06c01c000 tx=0x7fd06c009040).stop 2020-11-20T11:39:41.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.142+0000 7fd05e7fc700 1 -- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014a320 msgr2=0x7fd07014a730 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:39:41.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.142+0000 7fd05e7fc700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd07014a320 0x7fd07014a730 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0x7fd06c01c000 tx=0x7fd06c009040).stop 2020-11-20T11:39:41.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.142+0000 7fd0782ff700 1 -- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd048056d60 msgr2=0x7fd048059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:41.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.142+0000 7fd0782ff700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd048056d60 0x7fd048059220 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fd068007fd0 tx=0x7fd068009040).stop 2020-11-20T11:39:41.148 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.142+0000 7fd0782ff700 1 -- 172.21.15.23:0/3340382676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd070067eb0 msgr2=0x7fd0700ab7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:41.148 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.142+0000 7fd0782ff700 1 --2- 172.21.15.23:0/3340382676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd070067eb0 0x7fd0700ab7c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fd060003000 tx=0x7fd060015000).stop 2020-11-20T11:39:41.148 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.142+0000 7fd07689c700 1 -- 172.21.15.23:0/3340382676 reap_dead start 2020-11-20T11:39:41.148 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.142+0000 7fd07689c700 1 -- 172.21.15.23:0/3340382676 reap_dead start 2020-11-20T11:39:41.148 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.142+0000 7fd0782ff700 1 -- 172.21.15.23:0/3340382676 shutdown_connections 2020-11-20T11:39:41.149 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.142+0000 7fd0782ff700 1 -- 172.21.15.23:0/3340382676 >> 172.21.15.23:0/3340382676 conn(0x7fd0700ad7a0 msgr2=0x7fd0701347f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:41.149 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.142+0000 7fd0782ff700 1 -- 172.21.15.23:0/3340382676 shutdown_connections 2020-11-20T11:39:41.149 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.142+0000 7fd0782ff700 1 -- 172.21.15.23:0/3340382676 wait complete. 2020-11-20T11:39:41.149 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:39:41.149 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:39:41.150 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:39:41.150 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:39:41.150 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 72, 2020-11-20T11:39:41.150 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:39:41.150 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:39:41.151 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:39:41.151 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:39:41.151 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:41.151 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 5, 2020-11-20T11:39:41.151 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:39:41.152 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:39:41.152 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:39:41.152 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:39:41.152 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:39:41.152 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:39:41.153 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:39:41.153 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:39:41.153 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:39:41.153 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:39:41.153 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:39:41.154 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:41.154 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:39:41.154 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:39:41.154 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:39:41.154 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:39:41.155 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:39:41.155 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:39:41.155 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:39:41.155 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:39:41.155 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:39:41.156 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:39:41.156 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:41.156 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:41.156 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:39:41.156 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:39:41.157 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:39:41.157 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:39:41.157 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:39:41.157 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:39:41.157 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:39:41.158 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:39:41.158 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:39:41.158 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:39:41.158 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:39:41.158 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:39:41.159 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:39:41.159 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:39:41.159 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:39:41.159 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:39:41.159 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:39:41.160 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:39:41.160 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:39:41.160 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:39:41.160 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:39:41.160 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:39:41.160 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:39:41.161 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:41.161 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:39:41.161 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:41.161 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:39:41.161 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:41.162 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:39:41.162 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:39:41.162 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:39:41.162 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:39:41.162 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:41.163 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:39:41.163 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:39:41.163 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:41.163 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:41.163 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:41.164 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:39:41.164 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:39:41.164 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:41.164 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:41.164 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:41.165 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:41.165 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:39:41.165 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:39:41.165 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:39:41.165 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:39:41.166 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:39:41.166 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:41.166 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:41.166 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:39:41.166 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:39:41.167 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:39:41.167 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:39:41.167 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:41.167 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:39:41.167 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:39:41.168 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:41.168 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:41.168 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:41.168 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:39:41.168 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:39:41.169 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:41.169 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:41.169 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:41.169 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:41.169 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:39:41.170 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:39:41.170 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:39:41.170 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:39:41.170 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:39:41.170 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:41.171 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:41.171 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:39:41.171 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:39:41.171 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:39:41.171 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:39:41.171 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:41.172 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:39:41.172 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:39:41.172 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:41.172 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:41.172 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:41.173 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:39:41.173 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:39:41.173 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:41.173 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:41.173 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:41.174 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:41.174 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:39:41.174 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:39:41.174 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:39:41.174 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:39:41.175 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:39:41.175 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:41.175 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:41.175 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:41.175 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:39:41.176 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:39:41.176 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:41.176 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:39:41.176 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:39:41.176 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:39:41.177 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:41.177 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:41.177 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:39:41.177 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:41.177 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:39:41.178 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:39:41.178 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:39:41.178 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:41.178 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:41.178 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:41.179 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:39:41.179 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:39:41.180 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2020-11-20T11:39:41.180 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2020-11-20T11:39:41.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.300+0000 7f05777db700 1 Processor -- start 2020-11-20T11:39:41.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.300+0000 7f05777db700 1 -- start start 2020-11-20T11:39:41.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.301+0000 7f05777db700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f05700aaab0 0x7f057013ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:41.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.301+0000 7f05777db700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05700aae90 0x7f05701381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:41.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.301+0000 7f05777db700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057013e600 con 0x7f057013aeb0 2020-11-20T11:39:41.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.301+0000 7f05777db700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f057013e8a0 con 0x7f05700aaab0 2020-11-20T11:39:41.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.301+0000 7f05777db700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05700ab6d0 con 0x7f05700aae90 2020-11-20T11:39:41.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.301+0000 7f0574d76700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05700aae90 0x7f05701381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:41.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.301+0000 7f0575577700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f05700aaab0 0x7f057013ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:41.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.301+0000 7f0574d76700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05700aae90 0x7f05701381a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:54218/0 (socket says 172.21.15.23:54218) 2020-11-20T11:39:41.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.301+0000 7f0575577700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f05700aaab0 0x7f057013ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:36872/0 (socket says 172.21.15.23:36872) 2020-11-20T11:39:41.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.301+0000 7f0574d76700 1 -- 172.21.15.23:0/3321816337 learned_addr learned my addr 172.21.15.23:0/3321816337 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:39:41.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.301+0000 7f0575d78700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f057013aeb0 0x7f05700a8c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:36204/0 (socket says 172.21.15.23:36204) 2020-11-20T11:39:41.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.302+0000 7f055ffff700 1 -- 172.21.15.23:0/3321816337 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1444554066 0 0) 0x7f057013e600 con 0x7f057013aeb0 2020-11-20T11:39:41.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.302+0000 7f055ffff700 1 -- 172.21.15.23:0/3321816337 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0558003280 con 0x7f057013aeb0 2020-11-20T11:39:41.308 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.302+0000 7f0574d76700 1 -- 172.21.15.23:0/3321816337 >> v2:172.21.15.155:3300/0 conn(0x7f05700aaab0 msgr2=0x7f057013ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:39:41.308 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.302+0000 7f0574d76700 1 --2- 172.21.15.23:0/3321816337 >> v2:172.21.15.155:3300/0 conn(0x7f05700aaab0 0x7f057013ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:41.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.302+0000 7f0574d76700 1 -- 172.21.15.23:0/3321816337 >> v1:172.21.15.155:6789/0 conn(0x7f057013aeb0 legacy=0x7f05700a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:41.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.302+0000 7f0574d76700 1 -- 172.21.15.23:0/3321816337 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05700a9370 con 0x7f05700aae90 2020-11-20T11:39:41.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.302+0000 7f0574d76700 1 --2- 172.21.15.23:0/3321816337 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05700aae90 0x7f05701381a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f0560009a10 tx=0x7f0560006ef0).ready entity=mon.2 client_cookie=5e081636d1b5f7f3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:41.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.302+0000 7f055ffff700 1 -- 172.21.15.23:0/3321816337 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056000b6b0 con 0x7f05700aae90 2020-11-20T11:39:41.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.302+0000 7f055ffff700 1 -- 172.21.15.23:0/3321816337 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0560000e90 con 0x7f05700aae90 2020-11-20T11:39:41.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.302+0000 7f055ffff700 1 -- 172.21.15.23:0/3321816337 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056000ba20 con 0x7f05700aae90 2020-11-20T11:39:41.313 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.303+0000 7f05777db700 1 -- 172.21.15.23:0/3321816337 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05700aae90 msgr2=0x7f05701381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:41.313 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.303+0000 7f05777db700 1 --2- 172.21.15.23:0/3321816337 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05700aae90 0x7f05701381a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f0560009a10 tx=0x7f0560006ef0).stop 2020-11-20T11:39:41.314 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.303+0000 7f05777db700 1 -- 172.21.15.23:0/3321816337 shutdown_connections 2020-11-20T11:39:41.314 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.303+0000 7f05777db700 1 --2- 172.21.15.23:0/3321816337 >> v2:172.21.15.155:3300/0 conn(0x7f05700aaab0 0x7f057013ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:41.314 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.303+0000 7f05777db700 1 --2- 172.21.15.23:0/3321816337 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05700aae90 0x7f05701381a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:41.314 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.303+0000 7f05777db700 1 -- 172.21.15.23:0/3321816337 >> 172.21.15.23:0/3321816337 conn(0x7f05700678e0 msgr2=0x7f0570067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:41.315 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.303+0000 7f05777db700 1 -- 172.21.15.23:0/3321816337 shutdown_connections 2020-11-20T11:39:41.315 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.304+0000 7f05777db700 1 -- 172.21.15.23:0/3321816337 wait complete. 2020-11-20T11:39:41.315 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.305+0000 7f05777db700 1 Processor -- start 2020-11-20T11:39:41.315 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.305+0000 7f05777db700 1 -- start start 2020-11-20T11:39:41.316 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.305+0000 7f05777db700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05700aaab0 0x7f057013a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:41.316 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.305+0000 7f05777db700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05700aae90 0x7f0570135780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:41.316 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.306+0000 7f05777db700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f057013ccc0 0x7f057014a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:41.316 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.306+0000 7f05777db700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057013c5d0 con 0x7f05700aaab0 2020-11-20T11:39:41.317 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.306+0000 7f05777db700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0570136090 con 0x7f05700aae90 2020-11-20T11:39:41.317 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.306+0000 7f05777db700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f057013c820 con 0x7f057013ccc0 2020-11-20T11:39:41.317 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.306+0000 7f0575577700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05700aaab0 0x7f057013a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:41.317 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.306+0000 7f0575577700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05700aaab0 0x7f057013a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43982/0 (socket says 172.21.15.23:43982) 2020-11-20T11:39:41.318 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.306+0000 7f0575577700 1 -- 172.21.15.23:0/1126024921 learned_addr learned my addr 172.21.15.23:0/1126024921 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:39:41.318 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.306+0000 7f0574d76700 1 --2- 172.21.15.23:0/1126024921 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05700aae90 0x7f0570135780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:41.318 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.306+0000 7f0575d78700 1 --2- 172.21.15.23:0/1126024921 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f057013ccc0 0x7f057014a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:41.318 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.306+0000 7f0575577700 1 -- 172.21.15.23:0/1126024921 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f057013ccc0 msgr2=0x7f057014a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:41.318 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.306+0000 7f0575577700 1 --2- 172.21.15.23:0/1126024921 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f057013ccc0 0x7f057014a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:41.319 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.306+0000 7f0575577700 1 -- 172.21.15.23:0/1126024921 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05700aae90 msgr2=0x7f0570135780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:41.319 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.306+0000 7f0575577700 1 --2- 172.21.15.23:0/1126024921 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05700aae90 0x7f0570135780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:41.319 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.306+0000 7f0575577700 1 -- 172.21.15.23:0/1126024921 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f057014a180 con 0x7f05700aaab0 2020-11-20T11:39:41.319 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.306+0000 7f0574d76700 1 --2- 172.21.15.23:0/1126024921 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05700aae90 0x7f0570135780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:39:41.320 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.306+0000 7f0575577700 1 --2- 172.21.15.23:0/1126024921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05700aaab0 0x7f057013a010 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f056c00eba0 tx=0x7f056c006e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:41.320 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.306+0000 7f055d7fa700 1 -- 172.21.15.23:0/1126024921 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c00c8e0 con 0x7f05700aaab0 2020-11-20T11:39:41.320 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.306+0000 7f055d7fa700 1 -- 172.21.15.23:0/1126024921 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f056c00ca40 con 0x7f05700aaab0 2020-11-20T11:39:41.320 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.306+0000 7f05777db700 1 -- 172.21.15.23:0/1126024921 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05700a5a70 con 0x7f05700aaab0 2020-11-20T11:39:41.320 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.307+0000 7f055d7fa700 1 -- 172.21.15.23:0/1126024921 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c00cdc0 con 0x7f05700aaab0 2020-11-20T11:39:41.321 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.307+0000 7f05777db700 1 -- 172.21.15.23:0/1126024921 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0570135e30 con 0x7f05700aaab0 2020-11-20T11:39:41.321 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.308+0000 7f05777db700 1 -- 172.21.15.23:0/1126024921 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f057005c7d0 con 0x7f05700aaab0 2020-11-20T11:39:41.321 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.308+0000 7f055d7fa700 1 -- 172.21.15.23:0/1126024921 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f056c012600 con 0x7f05700aaab0 2020-11-20T11:39:41.321 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.308+0000 7f055d7fa700 1 --2- 172.21.15.23:0/1126024921 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0548056d60 0x7f0548059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:41.322 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.308+0000 7f055d7fa700 1 -- 172.21.15.23:0/1126024921 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f056c0703f0 con 0x7f05700aaab0 2020-11-20T11:39:41.322 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.309+0000 7f0574d76700 1 --2- 172.21.15.23:0/1126024921 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0548056d60 0x7f0548059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:41.322 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.309+0000 7f0574d76700 1 --2- 172.21.15.23:0/1126024921 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0548056d60 0x7f0548059220 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f0560006f30 tx=0x7f0560016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:41.323 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.313+0000 7f055d7fa700 1 -- 172.21.15.23:0/1126024921 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f056c0456e0 con 0x7f05700aaab0 2020-11-20T11:39:41.541 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.538+0000 7f05777db700 1 -- 172.21.15.23:0/1126024921 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f057013c9e0 con 0x7f05700aaab0 2020-11-20T11:39:41.542 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.539+0000 7f055d7fa700 1 -- 172.21.15.23:0/1126024921 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f056c048d00 con 0x7f05700aaab0 2020-11-20T11:39:41.544 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.541+0000 7f05777db700 1 -- 172.21.15.23:0/1126024921 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0548056d60 msgr2=0x7f0548059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:41.544 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.541+0000 7f05777db700 1 --2- 172.21.15.23:0/1126024921 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0548056d60 0x7f0548059220 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f0560006f30 tx=0x7f0560016000).stop 2020-11-20T11:39:41.544 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.541+0000 7f05777db700 1 -- 172.21.15.23:0/1126024921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05700aaab0 msgr2=0x7f057013a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:41.545 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.541+0000 7f05777db700 1 --2- 172.21.15.23:0/1126024921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05700aaab0 0x7f057013a010 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f056c00eba0 tx=0x7f056c006e30).stop 2020-11-20T11:39:41.545 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.541+0000 7f05777db700 1 -- 172.21.15.23:0/1126024921 shutdown_connections 2020-11-20T11:39:41.545 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.541+0000 7f05777db700 1 --2- 172.21.15.23:0/1126024921 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0548056d60 0x7f0548059220 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:41.545 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.541+0000 7f05777db700 1 --2- 172.21.15.23:0/1126024921 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f057013ccc0 0x7f057014a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:41.546 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.541+0000 7f05777db700 1 --2- 172.21.15.23:0/1126024921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05700aaab0 0x7f057013a010 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:41.546 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.541+0000 7f05777db700 1 --2- 172.21.15.23:0/1126024921 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05700aae90 0x7f0570135780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:41.546 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.541+0000 7f05777db700 1 -- 172.21.15.23:0/1126024921 >> 172.21.15.23:0/1126024921 conn(0x7f05700678e0 msgr2=0x7f05700adda0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:41.546 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.541+0000 7f05777db700 1 -- 172.21.15.23:0/1126024921 shutdown_connections 2020-11-20T11:39:41.546 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:41.541+0000 7f05777db700 1 -- 172.21.15.23:0/1126024921 wait complete. 2020-11-20T11:39:41.559 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2020-11-20T11:39:51.560 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:39:51.561 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:39:51.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.682+0000 7f04627fb700 1 Processor -- start 2020-11-20T11:39:51.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.682+0000 7f04627fb700 1 -- start start 2020-11-20T11:39:51.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.683+0000 7f04627fb700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f045c0ad600 0x7f045c13af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:51.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.683+0000 7f04627fb700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f045c0a6a70 0x7f045c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:51.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.683+0000 7f04627fb700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f045c0a6f10 con 0x7f045c0ad9e0 2020-11-20T11:39:51.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.683+0000 7f04627fb700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f045c0a71b0 con 0x7f045c0ad600 2020-11-20T11:39:51.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.683+0000 7f04627fb700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f045c0a4da0 con 0x7f045c0a6a70 2020-11-20T11:39:51.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.683+0000 7f045bfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f045c0ad600 0x7f045c13af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:51.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.683+0000 7f045bfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f045c0ad600 0x7f045c13af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43990/0 (socket says 172.21.15.23:43990) 2020-11-20T11:39:51.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.683+0000 7f045bfff700 1 -- 172.21.15.23:0/1542626329 learned_addr learned my addr 172.21.15.23:0/1542626329 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:39:51.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.683+0000 7f0460d98700 1 --2- 172.21.15.23:0/1542626329 >> v2:172.21.15.155:3300/0 conn(0x7f045c0a6a70 0x7f045c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:51.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.683+0000 7f045bfff700 1 -- 172.21.15.23:0/1542626329 >> v2:172.21.15.155:3300/0 conn(0x7f045c0a6a70 msgr2=0x7f045c0a5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:39:51.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.683+0000 7f045bfff700 1 --2- 172.21.15.23:0/1542626329 >> v2:172.21.15.155:3300/0 conn(0x7f045c0a6a70 0x7f045c0a5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:51.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.683+0000 7f045bfff700 1 -- 172.21.15.23:0/1542626329 >> v1:172.21.15.155:6789/0 conn(0x7f045c0ad9e0 legacy=0x7f045c0a90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:39:51.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.683+0000 7f045bfff700 1 -- 172.21.15.23:0/1542626329 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f045c1352f0 con 0x7f045c0ad600 2020-11-20T11:39:51.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.683+0000 7f045bfff700 1 --2- 172.21.15.23:0/1542626329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f045c0ad600 0x7f045c13af90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f045c0a8fc0 tx=0x7f044c006ef0).ready entity=mon.0 client_cookie=65f0488d6c971236 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:51.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.683+0000 7f045affd700 1 -- 172.21.15.23:0/1542626329 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f044c00b820 con 0x7f045c0ad600 2020-11-20T11:39:51.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.684+0000 7f045affd700 1 -- 172.21.15.23:0/1542626329 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f044c000e90 con 0x7f045c0ad600 2020-11-20T11:39:51.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.684+0000 7f045affd700 1 -- 172.21.15.23:0/1542626329 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f044c00bb90 con 0x7f045c0ad600 2020-11-20T11:39:51.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.684+0000 7f04627fb700 1 -- 172.21.15.23:0/1542626329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f045c0ad600 msgr2=0x7f045c13af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:51.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.684+0000 7f04627fb700 1 --2- 172.21.15.23:0/1542626329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f045c0ad600 0x7f045c13af90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f045c0a8fc0 tx=0x7f044c006ef0).stop 2020-11-20T11:39:51.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.684+0000 7f04627fb700 1 -- 172.21.15.23:0/1542626329 shutdown_connections 2020-11-20T11:39:51.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.684+0000 7f04627fb700 1 --2- 172.21.15.23:0/1542626329 >> v2:172.21.15.155:3300/0 conn(0x7f045c0a6a70 0x7f045c0a5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:51.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.684+0000 7f04627fb700 1 --2- 172.21.15.23:0/1542626329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f045c0ad600 0x7f045c13af90 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:51.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.684+0000 7f04627fb700 1 -- 172.21.15.23:0/1542626329 >> 172.21.15.23:0/1542626329 conn(0x7f045c060c80 msgr2=0x7f045c0acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:51.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.684+0000 7f04627fb700 1 -- 172.21.15.23:0/1542626329 shutdown_connections 2020-11-20T11:39:51.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.684+0000 7f04627fb700 1 -- 172.21.15.23:0/1542626329 wait complete. 2020-11-20T11:39:51.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.685+0000 7f04627fb700 1 Processor -- start 2020-11-20T11:39:51.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.686+0000 7f04627fb700 1 -- start start 2020-11-20T11:39:51.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.686+0000 7f04627fb700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f045c0a6a70 0x7f045c13a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:51.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.686+0000 7f04627fb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f045c0ad600 0x7f045c067760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:51.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.686+0000 7f04627fb700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f045c0ad9e0 0x7f045c064ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:51.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.686+0000 7f04627fb700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045c137990 con 0x7f045c0ad600 2020-11-20T11:39:51.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.686+0000 7f04627fb700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045c137bc0 con 0x7f045c0ad9e0 2020-11-20T11:39:51.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.686+0000 7f04627fb700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f045c067c50 con 0x7f045c0a6a70 2020-11-20T11:39:51.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.686+0000 7f045b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f045c0ad600 0x7f045c067760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:51.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.686+0000 7f045b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f045c0ad600 0x7f045c067760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43998/0 (socket says 172.21.15.23:43998) 2020-11-20T11:39:51.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.686+0000 7f045b7fe700 1 -- 172.21.15.23:0/2580984884 learned_addr learned my addr 172.21.15.23:0/2580984884 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:39:51.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.686+0000 7f0460d98700 1 --2- 172.21.15.23:0/2580984884 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f045c0ad9e0 0x7f045c064ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:51.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.686+0000 7f045bfff700 1 --2- 172.21.15.23:0/2580984884 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f045c0a6a70 0x7f045c13a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:51.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.687+0000 7f045b7fe700 1 -- 172.21.15.23:0/2580984884 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f045c0a6a70 msgr2=0x7f045c13a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:51.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.687+0000 7f045b7fe700 1 --2- 172.21.15.23:0/2580984884 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f045c0a6a70 0x7f045c13a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:51.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.687+0000 7f045b7fe700 1 -- 172.21.15.23:0/2580984884 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f045c0ad9e0 msgr2=0x7f045c064ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:51.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.687+0000 7f045b7fe700 1 --2- 172.21.15.23:0/2580984884 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f045c0ad9e0 0x7f045c064ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:51.699 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.687+0000 7f045b7fe700 1 -- 172.21.15.23:0/2580984884 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f045c067f00 con 0x7f045c0ad600 2020-11-20T11:39:51.699 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.687+0000 7f045b7fe700 1 --2- 172.21.15.23:0/2580984884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f045c0ad600 0x7f045c067760 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f0450009690 tx=0x7f0450006db0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:51.699 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.687+0000 7f043bfff700 1 -- 172.21.15.23:0/2580984884 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0450013460 con 0x7f045c0ad600 2020-11-20T11:39:51.699 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.687+0000 7f043bfff700 1 -- 172.21.15.23:0/2580984884 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04500135e0 con 0x7f045c0ad600 2020-11-20T11:39:51.699 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.687+0000 7f04627fb700 1 -- 172.21.15.23:0/2580984884 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f045c138060 con 0x7f045c0ad600 2020-11-20T11:39:51.700 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.687+0000 7f043bfff700 1 -- 172.21.15.23:0/2580984884 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04500139a0 con 0x7f045c0ad600 2020-11-20T11:39:51.700 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.687+0000 7f04627fb700 1 -- 172.21.15.23:0/2580984884 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f045c065620 con 0x7f045c0ad600 2020-11-20T11:39:51.700 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.688+0000 7f04627fb700 1 -- 172.21.15.23:0/2580984884 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f045c05c520 con 0x7f045c0ad600 2020-11-20T11:39:51.700 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.688+0000 7f043bfff700 1 -- 172.21.15.23:0/2580984884 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f045001a060 con 0x7f045c0ad600 2020-11-20T11:39:51.700 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.689+0000 7f043bfff700 1 --2- 172.21.15.23:0/2580984884 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0434056d60 0x7f0434059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:51.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.689+0000 7f043bfff700 1 -- 172.21.15.23:0/2580984884 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f045006f8e0 con 0x7f045c0ad600 2020-11-20T11:39:51.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.689+0000 7f045bfff700 1 --2- 172.21.15.23:0/2580984884 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0434056d60 0x7f0434059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:51.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.689+0000 7f045bfff700 1 --2- 172.21.15.23:0/2580984884 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0434056d60 0x7f0434059220 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f044c006f30 tx=0x7f044c016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:51.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.693+0000 7f043bfff700 1 -- 172.21.15.23:0/2580984884 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0450044bd0 con 0x7f045c0ad600 2020-11-20T11:39:51.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.915+0000 7f04627fb700 1 -- 172.21.15.23:0/2580984884 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f045c0a8580 con 0x7f045c0ad600 2020-11-20T11:39:51.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.915+0000 7f043bfff700 1 -- 172.21.15.23:0/2580984884 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7f0450018030 con 0x7f045c0ad600 2020-11-20T11:39:51.919 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":72,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:39:51.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.918+0000 7f04627fb700 1 -- 172.21.15.23:0/2580984884 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0434056d60 msgr2=0x7f0434059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:51.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.918+0000 7f04627fb700 1 --2- 172.21.15.23:0/2580984884 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0434056d60 0x7f0434059220 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f044c006f30 tx=0x7f044c016000).stop 2020-11-20T11:39:51.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.918+0000 7f04627fb700 1 -- 172.21.15.23:0/2580984884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f045c0ad600 msgr2=0x7f045c067760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:51.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.918+0000 7f04627fb700 1 --2- 172.21.15.23:0/2580984884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f045c0ad600 0x7f045c067760 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f0450009690 tx=0x7f0450006db0).stop 2020-11-20T11:39:51.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.918+0000 7f04627fb700 1 -- 172.21.15.23:0/2580984884 shutdown_connections 2020-11-20T11:39:51.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.918+0000 7f04627fb700 1 --2- 172.21.15.23:0/2580984884 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f045c0ad9e0 0x7f045c064ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:51.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.918+0000 7f04627fb700 1 --2- 172.21.15.23:0/2580984884 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0434056d60 0x7f0434059220 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:51.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.918+0000 7f04627fb700 1 --2- 172.21.15.23:0/2580984884 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f045c0a6a70 0x7f045c13a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:51.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.918+0000 7f04627fb700 1 --2- 172.21.15.23:0/2580984884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f045c0ad600 0x7f045c067760 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:51.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.918+0000 7f04627fb700 1 -- 172.21.15.23:0/2580984884 >> 172.21.15.23:0/2580984884 conn(0x7f045c060c80 msgr2=0x7f045c0aced0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:51.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.918+0000 7f04627fb700 1 -- 172.21.15.23:0/2580984884 shutdown_connections 2020-11-20T11:39:51.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:51.918+0000 7f04627fb700 1 -- 172.21.15.23:0/2580984884 wait complete. 2020-11-20T11:39:51.933 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:39:51.933 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2020-11-20T11:39:51.934 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:39:52.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.056+0000 7f868fdf2700 1 Processor -- start 2020-11-20T11:39:52.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.056+0000 7f868fdf2700 1 -- start start 2020-11-20T11:39:52.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.056+0000 7f868fdf2700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f868805bc80 0x7f86880a9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:52.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.056+0000 7f868fdf2700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8688065780 0x7f86880601b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:52.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.056+0000 7f868fdf2700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8688067430 con 0x7f86880a9cf0 2020-11-20T11:39:52.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.056+0000 7f868fdf2700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86880a8ca0 con 0x7f868805bc80 2020-11-20T11:39:52.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.056+0000 7f868fdf2700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f868805c060 con 0x7f8688065780 2020-11-20T11:39:52.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.057+0000 7f868d38d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8688065780 0x7f86880601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:52.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.057+0000 7f868db8e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f868805bc80 0x7f86880a9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:52.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.057+0000 7f868db8e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f868805bc80 0x7f86880a9140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:36900/0 (socket says 172.21.15.23:36900) 2020-11-20T11:39:52.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.057+0000 7f868d38d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8688065780 0x7f86880601b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:54246/0 (socket says 172.21.15.23:54246) 2020-11-20T11:39:52.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.057+0000 7f868db8e700 1 -- 172.21.15.23:0/848061064 learned_addr learned my addr 172.21.15.23:0/848061064 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:39:52.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.057+0000 7f868e38f700 1 --1- 172.21.15.23:0/848061064 >> v1:172.21.15.155:6789/0 conn(0x7f86880a9cf0 0x7f86881379e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:36230/0 (socket says 172.21.15.23:36230) 2020-11-20T11:39:52.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.057+0000 7f868db8e700 1 -- 172.21.15.23:0/848061064 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8688065780 msgr2=0x7f86880601b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:39:52.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.057+0000 7f868db8e700 1 --2- 172.21.15.23:0/848061064 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8688065780 0x7f86880601b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:52.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.057+0000 7f868db8e700 1 -- 172.21.15.23:0/848061064 >> v1:172.21.15.155:6789/0 conn(0x7f86880a9cf0 legacy=0x7f86881379e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:52.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.057+0000 7f868db8e700 1 -- 172.21.15.23:0/848061064 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86880a8810 con 0x7f868805bc80 2020-11-20T11:39:52.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.057+0000 7f868cb8c700 1 -- 172.21.15.23:0/848061064 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 552765225 0 0) 0x7f8688067430 con 0x7f86880a9cf0 2020-11-20T11:39:52.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.057+0000 7f868d38d700 1 --2- 172.21.15.23:0/848061064 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8688065780 0x7f86880601b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:39:52.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.057+0000 7f868db8e700 1 --2- 172.21.15.23:0/848061064 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f868805bc80 0x7f86880a9140 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f8684009a10 tx=0x7f8684006ef0).ready entity=mon.1 client_cookie=eeacf000fcd1aa3f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:52.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.058+0000 7f868cb8c700 1 -- 172.21.15.23:0/848061064 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868400b860 con 0x7f868805bc80 2020-11-20T11:39:52.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.058+0000 7f868cb8c700 1 -- 172.21.15.23:0/848061064 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8684000e90 con 0x7f868805bc80 2020-11-20T11:39:52.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.058+0000 7f868cb8c700 1 -- 172.21.15.23:0/848061064 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868400bbd0 con 0x7f868805bc80 2020-11-20T11:39:52.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.058+0000 7f868fdf2700 1 -- 172.21.15.23:0/848061064 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f868805bc80 msgr2=0x7f86880a9140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:52.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.058+0000 7f868fdf2700 1 --2- 172.21.15.23:0/848061064 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f868805bc80 0x7f86880a9140 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f8684009a10 tx=0x7f8684006ef0).stop 2020-11-20T11:39:52.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.058+0000 7f868fdf2700 1 -- 172.21.15.23:0/848061064 shutdown_connections 2020-11-20T11:39:52.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.058+0000 7f868fdf2700 1 --2- 172.21.15.23:0/848061064 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f868805bc80 0x7f86880a9140 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:52.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.058+0000 7f868fdf2700 1 --2- 172.21.15.23:0/848061064 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8688065780 0x7f86880601b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:52.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.058+0000 7f868fdf2700 1 -- 172.21.15.23:0/848061064 >> 172.21.15.23:0/848061064 conn(0x7f86880ad7a0 msgr2=0x7f86880adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:52.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.058+0000 7f868fdf2700 1 -- 172.21.15.23:0/848061064 shutdown_connections 2020-11-20T11:39:52.068 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.058+0000 7f868fdf2700 1 -- 172.21.15.23:0/848061064 wait complete. 2020-11-20T11:39:52.071 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.059+0000 7f868fdf2700 1 Processor -- start 2020-11-20T11:39:52.071 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.059+0000 7f868fdf2700 1 -- start start 2020-11-20T11:39:52.071 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.060+0000 7f868fdf2700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f868805bc80 0x7f86880a17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:52.071 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.060+0000 7f868fdf2700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8688065780 0x7f868809fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:52.072 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.060+0000 7f868fdf2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86880a9cf0 0x7f86880a72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:52.072 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.060+0000 7f868fdf2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8688138b90 con 0x7f86880a9cf0 2020-11-20T11:39:52.072 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.060+0000 7f868fdf2700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86880a01c0 con 0x7f868805bc80 2020-11-20T11:39:52.072 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.060+0000 7f868fdf2700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f868809ead0 con 0x7f8688065780 2020-11-20T11:39:52.072 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.060+0000 7f868e38f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86880a9cf0 0x7f86880a72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:52.073 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.060+0000 7f868e38f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86880a9cf0 0x7f86880a72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44014/0 (socket says 172.21.15.23:44014) 2020-11-20T11:39:52.073 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.060+0000 7f868e38f700 1 -- 172.21.15.23:0/1608846165 learned_addr learned my addr 172.21.15.23:0/1608846165 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:39:52.073 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.060+0000 7f868db8e700 1 --2- 172.21.15.23:0/1608846165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f868805bc80 0x7f86880a17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:52.073 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.061+0000 7f868d38d700 1 --2- 172.21.15.23:0/1608846165 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8688065780 0x7f868809fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:52.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.063+0000 7f868e38f700 1 -- 172.21.15.23:0/1608846165 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8688065780 msgr2=0x7f868809fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:52.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.063+0000 7f868e38f700 1 --2- 172.21.15.23:0/1608846165 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8688065780 0x7f868809fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:52.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.063+0000 7f868e38f700 1 -- 172.21.15.23:0/1608846165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f868805bc80 msgr2=0x7f86880a17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:52.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.063+0000 7f868e38f700 1 --2- 172.21.15.23:0/1608846165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f868805bc80 0x7f86880a17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:52.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.063+0000 7f868e38f700 1 -- 172.21.15.23:0/1608846165 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f868809ed70 con 0x7f86880a9cf0 2020-11-20T11:39:52.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.063+0000 7f868e38f700 1 --2- 172.21.15.23:0/1608846165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86880a9cf0 0x7f86880a72e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f868000cae0 tx=0x7f8680007f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:52.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.063+0000 7f8675ffb700 1 -- 172.21.15.23:0/1608846165 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86800058f0 con 0x7f86880a9cf0 2020-11-20T11:39:52.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.063+0000 7f8675ffb700 1 -- 172.21.15.23:0/1608846165 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8680005a50 con 0x7f86880a9cf0 2020-11-20T11:39:52.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.063+0000 7f868fdf2700 1 -- 172.21.15.23:0/1608846165 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f868809f010 con 0x7f86880a9cf0 2020-11-20T11:39:52.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.063+0000 7f8675ffb700 1 -- 172.21.15.23:0/1608846165 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8680005dd0 con 0x7f86880a9cf0 2020-11-20T11:39:52.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.064+0000 7f868fdf2700 1 -- 172.21.15.23:0/1608846165 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f868809e840 con 0x7f86880a9cf0 2020-11-20T11:39:52.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.065+0000 7f868fdf2700 1 --2- 172.21.15.23:0/1608846165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86880a7830 0x7f8688149f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:52.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.065+0000 7f868fdf2700 1 -- 172.21.15.23:0/1608846165 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86880a7c40 con 0x7f86880a7830 2020-11-20T11:39:52.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.065+0000 7f8675ffb700 1 -- 172.21.15.23:0/1608846165 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8680010030 con 0x7f86880a9cf0 2020-11-20T11:39:52.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.065+0000 7f868db8e700 1 --2- 172.21.15.23:0/1608846165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86880a7830 0x7f8688149f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:52.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.066+0000 7f8675ffb700 1 --2- 172.21.15.23:0/1608846165 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8660056d60 0x7f8660059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:52.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.066+0000 7f8675ffb700 1 -- 172.21.15.23:0/1608846165 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f8680070090 con 0x7f86880a9cf0 2020-11-20T11:39:52.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.066+0000 7f868db8e700 1 -- 172.21.15.23:0/1608846165 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8688062fe0 con 0x7f86880a7830 2020-11-20T11:39:52.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.066+0000 7f868d38d700 1 --2- 172.21.15.23:0/1608846165 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8660056d60 0x7f8660059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:52.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.066+0000 7f868db8e700 1 --2- 172.21.15.23:0/1608846165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86880a7830 0x7f8688149f00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f8684009a10 tx=0x7f8684016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:52.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.066+0000 7f8675ffb700 1 -- 172.21.15.23:0/1608846165 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86840119c0 con 0x7f86880a7830 2020-11-20T11:39:52.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.067+0000 7f868d38d700 1 --2- 172.21.15.23:0/1608846165 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8660056d60 0x7f8660059220 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f868809dd20 tx=0x7f8678006e30).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:52.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.067+0000 7f8675ffb700 1 -- 172.21.15.23:0/1608846165 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f8684011b20 con 0x7f86880a7830 2020-11-20T11:39:52.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.067+0000 7f8675ffb700 1 -- 172.21.15.23:0/1608846165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86880a7830 msgr2=0x7f8688149f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:52.079 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.067+0000 7f8675ffb700 1 --2- 172.21.15.23:0/1608846165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86880a7830 0x7f8688149f00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f8684009a10 tx=0x7f8684016000).stop 2020-11-20T11:39:52.079 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.067+0000 7f8675ffb700 1 -- 172.21.15.23:0/1608846165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86880a7830 msgr2=0x7f8688149f00 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:39:52.079 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.067+0000 7f8675ffb700 1 --2- 172.21.15.23:0/1608846165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86880a7830 0x7f8688149f00 secure :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0x7f8684009a10 tx=0x7f8684016000).stop 2020-11-20T11:39:52.080 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.076+0000 7f868fdf2700 1 --2- 172.21.15.23:0/1608846165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f868814a3f0 0x7f868814d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:52.080 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.076+0000 7f868fdf2700 1 -- 172.21.15.23:0/1608846165 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8688044710 con 0x7f868814a3f0 2020-11-20T11:39:52.080 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.077+0000 7f868db8e700 1 --2- 172.21.15.23:0/1608846165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f868814a3f0 0x7f868814d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:52.082 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.077+0000 7f868db8e700 1 -- 172.21.15.23:0/1608846165 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8688069770 con 0x7f868814a3f0 2020-11-20T11:39:52.082 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.077+0000 7f868db8e700 1 --2- 172.21.15.23:0/1608846165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f868814a3f0 0x7f868814d460 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f8688147d30 tx=0x7f868401f040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:52.082 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.078+0000 7f8675ffb700 1 -- 172.21.15.23:0/1608846165 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868401d7a0 con 0x7f868814a3f0 2020-11-20T11:39:52.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.078+0000 7f8675ffb700 1 -- 172.21.15.23:0/1608846165 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4446 (secure 0 0 0) 0x7f86840117b0 con 0x7f868814a3f0 2020-11-20T11:39:52.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.078+0000 7f8675ffb700 1 -- 172.21.15.23:0/1608846165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f868814a3f0 msgr2=0x7f868814d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:52.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.078+0000 7f8675ffb700 1 --2- 172.21.15.23:0/1608846165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f868814a3f0 0x7f868814d460 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f8688147d30 tx=0x7f868401f040).stop 2020-11-20T11:39:52.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.078+0000 7f8675ffb700 1 -- 172.21.15.23:0/1608846165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f868814a3f0 msgr2=0x7f868814d460 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:39:52.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.078+0000 7f8675ffb700 1 --2- 172.21.15.23:0/1608846165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f868814a3f0 0x7f868814d460 secure :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0x7f8688147d30 tx=0x7f868401f040).stop 2020-11-20T11:39:52.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.078+0000 7f868fdf2700 1 -- 172.21.15.23:0/1608846165 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8660056d60 msgr2=0x7f8660059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:52.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.078+0000 7f868fdf2700 1 --2- 172.21.15.23:0/1608846165 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8660056d60 0x7f8660059220 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f868809dd20 tx=0x7f8678006e30).stop 2020-11-20T11:39:52.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.078+0000 7f868fdf2700 1 -- 172.21.15.23:0/1608846165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86880a9cf0 msgr2=0x7f86880a72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:52.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.078+0000 7f868fdf2700 1 --2- 172.21.15.23:0/1608846165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86880a9cf0 0x7f86880a72e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f868000cae0 tx=0x7f8680007f90).stop 2020-11-20T11:39:52.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.078+0000 7f868e38f700 1 -- 172.21.15.23:0/1608846165 reap_dead start 2020-11-20T11:39:52.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.078+0000 7f868fdf2700 1 -- 172.21.15.23:0/1608846165 shutdown_connections 2020-11-20T11:39:52.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.078+0000 7f868fdf2700 1 --2- 172.21.15.23:0/1608846165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f86880a9cf0 0x7f86880a72e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:52.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.078+0000 7f868fdf2700 1 -- 172.21.15.23:0/1608846165 >> 172.21.15.23:0/1608846165 conn(0x7f86880ad7a0 msgr2=0x7f86880aa9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:52.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.078+0000 7f868fdf2700 1 -- 172.21.15.23:0/1608846165 shutdown_connections 2020-11-20T11:39:52.086 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.079+0000 7f868fdf2700 1 -- 172.21.15.23:0/1608846165 wait complete. 2020-11-20T11:39:52.086 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:39:52.086 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:39:52.086 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:39:52.087 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:39:52.087 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 72, 2020-11-20T11:39:52.087 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:39:52.087 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:39:52.087 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:39:52.088 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:39:52.088 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:52.088 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 15, 2020-11-20T11:39:52.088 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:39:52.088 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:39:52.089 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:39:52.089 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:39:52.089 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:39:52.089 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:39:52.089 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:39:52.090 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:39:52.090 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:39:52.090 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:39:52.090 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:39:52.090 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:52.091 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:39:52.091 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:39:52.091 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:39:52.091 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:39:52.091 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:39:52.092 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:39:52.092 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:39:52.092 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:39:52.092 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:39:52.092 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:39:52.093 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:52.093 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.093 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:39:52.093 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:39:52.094 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:39:52.094 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:39:52.094 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:39:52.094 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:39:52.094 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:39:52.095 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:39:52.095 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:39:52.095 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:39:52.095 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:39:52.096 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:39:52.096 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:39:52.096 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:39:52.096 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:39:52.096 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:39:52.097 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:39:52.097 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:39:52.097 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:39:52.097 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:39:52.097 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:39:52.098 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:39:52.098 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:39:52.098 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:52.098 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:39:52.098 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.099 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:39:52.099 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.099 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:39:52.099 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:39:52.099 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:39:52.100 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:39:52.100 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.100 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:39:52.100 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:39:52.101 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:52.101 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.101 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.101 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:39:52.101 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:39:52.102 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:52.102 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:52.102 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:52.102 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.102 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:39:52.103 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:39:52.103 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:39:52.103 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:39:52.103 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:39:52.103 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.104 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.104 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:39:52.104 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:39:52.104 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:39:52.104 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:39:52.104 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.105 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:39:52.105 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:39:52.105 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:52.105 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.105 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.106 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:39:52.106 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:39:52.106 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:52.106 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:52.107 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:52.107 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.107 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:39:52.107 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:39:52.107 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:39:52.108 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:39:52.108 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:39:52.108 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.108 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.108 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:39:52.108 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:39:52.109 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:39:52.109 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:39:52.109 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.109 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:39:52.109 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:39:52.110 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:52.110 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.110 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.110 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:39:52.110 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:39:52.111 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:52.111 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:52.111 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:52.111 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.111 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:39:52.112 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:39:52.112 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:39:52.112 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:39:52.112 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:39:52.113 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:52.113 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:52.113 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.113 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:39:52.113 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:39:52.114 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.114 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:39:52.114 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:39:52.114 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:39:52.115 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:52.115 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:52.115 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:39:52.115 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.115 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:39:52.116 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:39:52.116 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:39:52.116 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:52.116 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:52.116 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:39:52.116 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.117 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:39:52.117 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:39:52.117 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:39:52.117 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:52.117 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:52.118 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.118 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:39:52.118 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:39:52.119 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:39:52.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.242+0000 7fea25f99700 1 Processor -- start 2020-11-20T11:39:52.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.242+0000 7fea25f99700 1 -- start start 2020-11-20T11:39:52.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.242+0000 7fea25f99700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fea200665f0 0x7fea2013adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:52.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.242+0000 7fea25f99700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fea200ab2a0 0x7fea20137f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:52.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.242+0000 7fea25f99700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea2013bd10 con 0x7fea20066210 2020-11-20T11:39:52.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.242+0000 7fea25f99700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea200ab700 con 0x7fea200ab2a0 2020-11-20T11:39:52.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.242+0000 7fea25f99700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea200a9700 con 0x7fea200665f0 2020-11-20T11:39:52.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.242+0000 7fea25798700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fea200ab2a0 0x7fea20137f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:52.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.243+0000 7fea24f97700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fea20066210 0x7fea200a8ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58272/0 (socket says 172.21.15.23:58272) 2020-11-20T11:39:52.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.243+0000 7fea24f97700 1 -- 172.21.15.23:0/1360797487 learned_addr learned my addr 172.21.15.23:0/1360797487 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:39:52.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.243+0000 7fea1ffff700 1 --2- 172.21.15.23:0/1360797487 >> v2:172.21.15.155:3300/0 conn(0x7fea200665f0 0x7fea2013adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:52.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.243+0000 7fea25798700 1 -- 172.21.15.23:0/1360797487 >> v2:172.21.15.155:3300/0 conn(0x7fea200665f0 msgr2=0x7fea2013adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:39:52.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.243+0000 7fea25798700 1 --2- 172.21.15.23:0/1360797487 >> v2:172.21.15.155:3300/0 conn(0x7fea200665f0 0x7fea2013adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:52.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.243+0000 7fea25798700 1 -- 172.21.15.23:0/1360797487 >> v1:172.21.15.23:6789/0 conn(0x7fea20066210 legacy=0x7fea200a8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:52.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.243+0000 7fea25798700 1 -- 172.21.15.23:0/1360797487 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea200aa930 con 0x7fea200ab2a0 2020-11-20T11:39:52.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.243+0000 7fea25798700 1 --2- 172.21.15.23:0/1360797487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea200ab2a0 0x7fea20137f30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fea200a98a0 tx=0x7fea18009220).ready entity=mon.0 client_cookie=f766972483032df5 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:52.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.243+0000 7fea1f7fe700 1 -- 172.21.15.23:0/1360797487 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1800bbc0 con 0x7fea200ab2a0 2020-11-20T11:39:52.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.243+0000 7fea1f7fe700 1 -- 172.21.15.23:0/1360797487 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea1800bd20 con 0x7fea200ab2a0 2020-11-20T11:39:52.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.243+0000 7fea1f7fe700 1 -- 172.21.15.23:0/1360797487 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea18011610 con 0x7fea200ab2a0 2020-11-20T11:39:52.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.244+0000 7fea25f99700 1 -- 172.21.15.23:0/1360797487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea200ab2a0 msgr2=0x7fea20137f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:52.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.244+0000 7fea25f99700 1 --2- 172.21.15.23:0/1360797487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea200ab2a0 0x7fea20137f30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fea200a98a0 tx=0x7fea18009220).stop 2020-11-20T11:39:52.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.244+0000 7fea25f99700 1 -- 172.21.15.23:0/1360797487 shutdown_connections 2020-11-20T11:39:52.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.244+0000 7fea25f99700 1 --2- 172.21.15.23:0/1360797487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea200ab2a0 0x7fea20137f30 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:52.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.244+0000 7fea25f99700 1 --2- 172.21.15.23:0/1360797487 >> v2:172.21.15.155:3300/0 conn(0x7fea200665f0 0x7fea2013adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:52.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.244+0000 7fea25f99700 1 -- 172.21.15.23:0/1360797487 >> 172.21.15.23:0/1360797487 conn(0x7fea200612c0 msgr2=0x7fea200616d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:52.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.244+0000 7fea25f99700 1 -- 172.21.15.23:0/1360797487 shutdown_connections 2020-11-20T11:39:52.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.244+0000 7fea25f99700 1 -- 172.21.15.23:0/1360797487 wait complete. 2020-11-20T11:39:52.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.245+0000 7fea25f99700 1 Processor -- start 2020-11-20T11:39:52.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.245+0000 7fea25f99700 1 -- start start 2020-11-20T11:39:52.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.246+0000 7fea25f99700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea20066210 0x7fea200ac2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:52.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.246+0000 7fea25f99700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea200665f0 0x7fea200a2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:52.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.246+0000 7fea25f99700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea200ab2a0 0x7fea200a2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:52.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.246+0000 7fea25f99700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea200a31f0 con 0x7fea200ab2a0 2020-11-20T11:39:52.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.246+0000 7fea25f99700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea200a3490 con 0x7fea20066210 2020-11-20T11:39:52.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.246+0000 7fea25f99700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea200a7640 con 0x7fea200665f0 2020-11-20T11:39:52.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.246+0000 7fea25798700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea200ab2a0 0x7fea200a2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:52.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.246+0000 7fea25798700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea200ab2a0 0x7fea200a2510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44032/0 (socket says 172.21.15.23:44032) 2020-11-20T11:39:52.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.246+0000 7fea25798700 1 -- 172.21.15.23:0/2256518526 learned_addr learned my addr 172.21.15.23:0/2256518526 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:39:52.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.246+0000 7fea24f97700 1 --2- 172.21.15.23:0/2256518526 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea20066210 0x7fea200ac2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:52.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.246+0000 7fea1ffff700 1 --2- 172.21.15.23:0/2256518526 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea200665f0 0x7fea200a2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:52.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.246+0000 7fea25798700 1 -- 172.21.15.23:0/2256518526 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea200665f0 msgr2=0x7fea200a2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:52.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.246+0000 7fea25798700 1 --2- 172.21.15.23:0/2256518526 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fea200665f0 0x7fea200a2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:52.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.246+0000 7fea25798700 1 -- 172.21.15.23:0/2256518526 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea20066210 msgr2=0x7fea200ac2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:52.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.246+0000 7fea25798700 1 --2- 172.21.15.23:0/2256518526 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea20066210 0x7fea200ac2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:52.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.246+0000 7fea25798700 1 -- 172.21.15.23:0/2256518526 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea200a7800 con 0x7fea200ab2a0 2020-11-20T11:39:52.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.246+0000 7fea25798700 1 --2- 172.21.15.23:0/2256518526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea200ab2a0 0x7fea200a2510 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fea18009260 tx=0x7fea18016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:52.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.246+0000 7fea1cff9700 1 -- 172.21.15.23:0/2256518526 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1801c6c0 con 0x7fea200ab2a0 2020-11-20T11:39:52.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.246+0000 7fea25f99700 1 -- 172.21.15.23:0/2256518526 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea200a4aa0 con 0x7fea200ab2a0 2020-11-20T11:39:52.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.247+0000 7fea1cff9700 1 -- 172.21.15.23:0/2256518526 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea1801c820 con 0x7fea200ab2a0 2020-11-20T11:39:52.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.247+0000 7fea25f99700 1 -- 172.21.15.23:0/2256518526 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea200a2f20 con 0x7fea200ab2a0 2020-11-20T11:39:52.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.247+0000 7fea1cff9700 1 -- 172.21.15.23:0/2256518526 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1801cba0 con 0x7fea200ab2a0 2020-11-20T11:39:52.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.247+0000 7fea25f99700 1 --2- 172.21.15.23:0/2256518526 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea200622d0 0x7fea2014d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:52.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.247+0000 7fea25f99700 1 -- 172.21.15.23:0/2256518526 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea200a30b0 con 0x7fea200622d0 2020-11-20T11:39:52.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.248+0000 7fea24f97700 1 --2- 172.21.15.23:0/2256518526 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea200622d0 0x7fea2014d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:52.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.248+0000 7fea1cff9700 1 -- 172.21.15.23:0/2256518526 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fea18017070 con 0x7fea200ab2a0 2020-11-20T11:39:52.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.248+0000 7fea24f97700 1 -- 172.21.15.23:0/2256518526 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fea200a7210 con 0x7fea200622d0 2020-11-20T11:39:52.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.248+0000 7fea24f97700 1 --2- 172.21.15.23:0/2256518526 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea200622d0 0x7fea2014d8f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fea14001620 tx=0x7fea14007bc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:52.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.248+0000 7fea1cff9700 1 --2- 172.21.15.23:0/2256518526 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe9f8056d60 0x7fe9f8059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:52.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.248+0000 7fea1cff9700 1 -- 172.21.15.23:0/2256518526 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fea18072690 con 0x7fea200ab2a0 2020-11-20T11:39:52.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.248+0000 7fea1cff9700 1 -- 172.21.15.23:0/2256518526 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea14011590 con 0x7fea200622d0 2020-11-20T11:39:52.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.248+0000 7fea1ffff700 1 --2- 172.21.15.23:0/2256518526 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe9f8056d60 0x7fe9f8059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:52.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.249+0000 7fea1cff9700 1 -- 172.21.15.23:0/2256518526 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fea140116f0 con 0x7fea200622d0 2020-11-20T11:39:52.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.249+0000 7fea1cff9700 1 -- 172.21.15.23:0/2256518526 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea200622d0 msgr2=0x7fea2014d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:52.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.249+0000 7fea1cff9700 1 --2- 172.21.15.23:0/2256518526 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea200622d0 0x7fea2014d8f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fea14001620 tx=0x7fea14007bc0).stop 2020-11-20T11:39:52.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.249+0000 7fea1cff9700 1 -- 172.21.15.23:0/2256518526 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea200622d0 msgr2=0x7fea2014d8f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:39:52.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.249+0000 7fea1cff9700 1 --2- 172.21.15.23:0/2256518526 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea200622d0 0x7fea2014d8f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:52.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.249+0000 7fea1ffff700 1 --2- 172.21.15.23:0/2256518526 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe9f8056d60 0x7fe9f8059220 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fea200a45a0 tx=0x7fea0c008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:52.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.257+0000 7fea25f99700 1 --2- 172.21.15.23:0/2256518526 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea2014dde0 0x7fea201567f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:52.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.257+0000 7fea25f99700 1 -- 172.21.15.23:0/2256518526 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea2014d440 con 0x7fea2014dde0 2020-11-20T11:39:52.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.257+0000 7fea24f97700 1 --2- 172.21.15.23:0/2256518526 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea2014dde0 0x7fea201567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:52.266 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.257+0000 7fea24f97700 1 -- 172.21.15.23:0/2256518526 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fea20062760 con 0x7fea2014dde0 2020-11-20T11:39:52.266 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.257+0000 7fea24f97700 1 --2- 172.21.15.23:0/2256518526 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea2014dde0 0x7fea201567f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fea14001620 tx=0x7fea14017000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:52.266 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.258+0000 7fea1cff9700 1 -- 172.21.15.23:0/2256518526 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea14015de0 con 0x7fea2014dde0 2020-11-20T11:39:52.266 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.258+0000 7fea1cff9700 1 -- 172.21.15.23:0/2256518526 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4784 (secure 0 0 0) 0x7fea14011380 con 0x7fea2014dde0 2020-11-20T11:39:52.266 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.258+0000 7fea1cff9700 1 -- 172.21.15.23:0/2256518526 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea2014dde0 msgr2=0x7fea201567f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:52.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.258+0000 7fea1cff9700 1 --2- 172.21.15.23:0/2256518526 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea2014dde0 0x7fea201567f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fea14001620 tx=0x7fea14017000).stop 2020-11-20T11:39:52.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.258+0000 7fea1cff9700 1 -- 172.21.15.23:0/2256518526 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea2014dde0 msgr2=0x7fea201567f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:39:52.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.258+0000 7fea1cff9700 1 --2- 172.21.15.23:0/2256518526 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fea2014dde0 0x7fea201567f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:52.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.258+0000 7fea25f99700 1 -- 172.21.15.23:0/2256518526 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe9f8056d60 msgr2=0x7fe9f8059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:52.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.258+0000 7fea25f99700 1 --2- 172.21.15.23:0/2256518526 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe9f8056d60 0x7fe9f8059220 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fea200a45a0 tx=0x7fea0c008040).stop 2020-11-20T11:39:52.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.258+0000 7fea25f99700 1 -- 172.21.15.23:0/2256518526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea200ab2a0 msgr2=0x7fea200a2510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:52.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.258+0000 7fea25f99700 1 --2- 172.21.15.23:0/2256518526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea200ab2a0 0x7fea200a2510 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fea18009260 tx=0x7fea18016000).stop 2020-11-20T11:39:52.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.258+0000 7fea25798700 1 -- 172.21.15.23:0/2256518526 reap_dead start 2020-11-20T11:39:52.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.258+0000 7fea25798700 1 -- 172.21.15.23:0/2256518526 reap_dead start 2020-11-20T11:39:52.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.258+0000 7fea25f99700 1 -- 172.21.15.23:0/2256518526 shutdown_connections 2020-11-20T11:39:52.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.258+0000 7fea25f99700 1 -- 172.21.15.23:0/2256518526 >> 172.21.15.23:0/2256518526 conn(0x7fea200612c0 msgr2=0x7fea200a8e90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:52.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.258+0000 7fea25f99700 1 -- 172.21.15.23:0/2256518526 shutdown_connections 2020-11-20T11:39:52.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.258+0000 7fea25f99700 1 -- 172.21.15.23:0/2256518526 wait complete. 2020-11-20T11:39:52.269 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:39:52.270 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:39:52.270 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:39:52.270 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:39:52.270 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 72, 2020-11-20T11:39:52.270 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:39:52.271 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:39:52.271 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:39:52.271 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:39:52.271 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:52.271 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 16, 2020-11-20T11:39:52.272 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:39:52.272 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:39:52.272 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:39:52.272 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:39:52.273 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:39:52.273 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:39:52.273 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:39:52.273 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:39:52.273 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:39:52.274 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:39:52.274 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:39:52.274 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:52.274 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:39:52.274 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:39:52.275 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:39:52.275 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:39:52.275 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:39:52.275 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:39:52.275 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:39:52.276 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:39:52.276 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:39:52.276 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:39:52.276 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:52.276 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.277 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:39:52.277 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:39:52.277 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:39:52.277 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:39:52.277 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:39:52.278 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:39:52.278 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:39:52.278 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:39:52.278 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:39:52.278 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:39:52.279 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:39:52.279 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:39:52.279 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:39:52.279 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:39:52.279 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:39:52.280 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:39:52.280 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:39:52.280 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:39:52.280 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:39:52.280 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:39:52.281 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:39:52.281 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:39:52.281 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:39:52.281 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:52.281 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:39:52.282 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.282 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:39:52.282 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.282 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:39:52.283 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:39:52.283 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:39:52.283 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:39:52.283 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.283 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:39:52.284 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:39:52.284 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:52.284 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.284 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.284 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:39:52.285 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:39:52.285 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:52.285 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:52.285 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:52.285 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.286 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:39:52.286 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:39:52.286 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:39:52.286 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:39:52.286 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:39:52.286 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.287 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.287 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:39:52.287 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:39:52.287 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:39:52.288 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:39:52.288 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.288 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:39:52.288 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:39:52.288 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:52.289 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.289 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.289 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:39:52.289 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:39:52.289 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:52.290 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:52.290 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:52.290 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.290 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:39:52.290 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:39:52.291 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:39:52.291 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:39:52.291 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:39:52.291 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.291 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.292 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:39:52.292 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:39:52.292 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:39:52.292 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:39:52.292 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.293 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:39:52.293 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:39:52.293 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:52.293 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.293 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.294 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:39:52.294 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:39:52.294 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:52.294 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:52.294 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:52.295 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.295 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:39:52.295 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:39:52.295 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:39:52.295 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:39:52.296 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:39:52.296 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:52.296 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:52.296 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.296 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:39:52.297 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:39:52.297 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.297 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:39:52.297 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:39:52.297 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:39:52.298 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:52.298 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:52.298 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:39:52.298 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.298 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:39:52.299 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:39:52.299 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:39:52.299 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:52.299 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:52.300 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:39:52.300 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.300 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:39:52.300 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:39:52.300 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:39:52.301 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:52.301 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:52.301 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:39:52.301 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.301 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:39:52.302 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:39:52.302 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:39:52.302 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:52.302 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:52.302 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:39:52.303 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.303 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:39:52.303 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:39:52.303 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:39:52.304 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:52.304 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:52.304 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.304 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:39:52.304 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:39:52.306 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:39:52.451 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.448+0000 7f5265b55700 1 Processor -- start 2020-11-20T11:39:52.451 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.448+0000 7f5265b55700 1 -- start start 2020-11-20T11:39:52.453 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.448+0000 7f5265b55700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5260061db0 0x7f52600a9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:52.454 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.449+0000 7f5265b55700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5260066170 0x7f5260137e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:52.454 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.449+0000 7f5265b55700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52600610b0 con 0x7f52600619d0 2020-11-20T11:39:52.454 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.449+0000 7f5265b55700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52600a9750 con 0x7f5260066170 2020-11-20T11:39:52.454 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.449+0000 7f5265b55700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5260066680 con 0x7f5260061db0 2020-11-20T11:39:52.454 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.449+0000 7f525ffff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5260066170 0x7f5260137e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:52.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.449+0000 7f525effd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5260061db0 0x7f52600a9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:52.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.449+0000 7f525ffff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5260066170 0x7f5260137e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:36940/0 (socket says 172.21.15.23:36940) 2020-11-20T11:39:52.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.449+0000 7f525ffff700 1 -- 172.21.15.23:0/487372260 learned_addr learned my addr 172.21.15.23:0/487372260 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:39:52.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.449+0000 7f525ffff700 1 -- 172.21.15.23:0/487372260 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5260061db0 msgr2=0x7f52600a9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:39:52.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.449+0000 7f525ffff700 1 --2- 172.21.15.23:0/487372260 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5260061db0 0x7f52600a9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:52.456 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.449+0000 7f525ffff700 1 -- 172.21.15.23:0/487372260 >> v1:172.21.15.155:6789/0 conn(0x7f52600619d0 legacy=0x7f52600609a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:52.456 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.449+0000 7f525ffff700 1 -- 172.21.15.23:0/487372260 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5260066920 con 0x7f5260066170 2020-11-20T11:39:52.456 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.449+0000 7f525e7fc700 1 -- 172.21.15.23:0/487372260 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3253259589 0 0) 0x7f52600610b0 con 0x7f52600619d0 2020-11-20T11:39:52.456 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.449+0000 7f525effd700 1 --2- 172.21.15.23:0/487372260 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5260061db0 0x7f52600a9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:39:52.457 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.450+0000 7f525ffff700 1 --2- 172.21.15.23:0/487372260 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5260066170 0x7f5260137e90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f5258009d90 tx=0x7f52580092d0).ready entity=mon.1 client_cookie=5c9d8f7765f704a4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:52.457 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.450+0000 7f525e7fc700 1 -- 172.21.15.23:0/487372260 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525800bbf0 con 0x7f5260066170 2020-11-20T11:39:52.457 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.450+0000 7f525e7fc700 1 -- 172.21.15.23:0/487372260 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f525800bd50 con 0x7f5260066170 2020-11-20T11:39:52.457 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.450+0000 7f525e7fc700 1 -- 172.21.15.23:0/487372260 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5258011610 con 0x7f5260066170 2020-11-20T11:39:52.458 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.450+0000 7f5265b55700 1 -- 172.21.15.23:0/487372260 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5260066170 msgr2=0x7f5260137e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:52.458 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.450+0000 7f5265b55700 1 --2- 172.21.15.23:0/487372260 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5260066170 0x7f5260137e90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f5258009d90 tx=0x7f52580092d0).stop 2020-11-20T11:39:52.458 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.450+0000 7f5265b55700 1 -- 172.21.15.23:0/487372260 shutdown_connections 2020-11-20T11:39:52.458 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.450+0000 7f5265b55700 1 --2- 172.21.15.23:0/487372260 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5260066170 0x7f5260137e90 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:52.459 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.450+0000 7f5265b55700 1 --2- 172.21.15.23:0/487372260 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5260061db0 0x7f52600a9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:52.459 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.450+0000 7f5265b55700 1 -- 172.21.15.23:0/487372260 >> 172.21.15.23:0/487372260 conn(0x7f5260063640 msgr2=0x7f52600612c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:52.459 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.450+0000 7f5265b55700 1 -- 172.21.15.23:0/487372260 shutdown_connections 2020-11-20T11:39:52.459 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.450+0000 7f5265b55700 1 -- 172.21.15.23:0/487372260 wait complete. 2020-11-20T11:39:52.462 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.452+0000 7f5265b55700 1 Processor -- start 2020-11-20T11:39:52.462 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.452+0000 7f5265b55700 1 -- start start 2020-11-20T11:39:52.462 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.452+0000 7f5265b55700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f52600619d0 0x7f5260136620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:52.462 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.452+0000 7f5265b55700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5260061db0 0x7f526014a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:52.462 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.452+0000 7f5265b55700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5260066170 0x7f526014a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:52.463 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.452+0000 7f5265b55700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52600a8490 con 0x7f5260066170 2020-11-20T11:39:52.463 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.452+0000 7f5265b55700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52600aeb60 con 0x7f5260061db0 2020-11-20T11:39:52.463 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.452+0000 7f5265b55700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52600ae180 con 0x7f52600619d0 2020-11-20T11:39:52.463 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.453+0000 7f525effd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5260061db0 0x7f526014a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:52.463 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.453+0000 7f525f7fe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f52600619d0 0x7f5260136620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:52.464 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.453+0000 7f525effd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5260061db0 0x7f526014a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:36942/0 (socket says 172.21.15.23:36942) 2020-11-20T11:39:52.464 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.453+0000 7f525effd700 1 -- 172.21.15.23:0/1043657705 learned_addr learned my addr 172.21.15.23:0/1043657705 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:39:52.464 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.453+0000 7f525ffff700 1 --2- 172.21.15.23:0/1043657705 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5260066170 0x7f526014a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:52.464 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.453+0000 7f525f7fe700 1 -- 172.21.15.23:0/1043657705 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5260061db0 msgr2=0x7f526014a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:52.465 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.453+0000 7f525f7fe700 1 --2- 172.21.15.23:0/1043657705 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5260061db0 0x7f526014a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:52.465 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.453+0000 7f525f7fe700 1 -- 172.21.15.23:0/1043657705 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5260066170 msgr2=0x7f526014a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:52.465 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.453+0000 7f525f7fe700 1 --2- 172.21.15.23:0/1043657705 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5260066170 0x7f526014a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:52.465 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.453+0000 7f525f7fe700 1 -- 172.21.15.23:0/1043657705 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52600ae420 con 0x7f52600619d0 2020-11-20T11:39:52.465 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.453+0000 7f525ffff700 1 --2- 172.21.15.23:0/1043657705 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5260066170 0x7f526014a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:39:52.466 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.453+0000 7f525f7fe700 1 --2- 172.21.15.23:0/1043657705 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f52600619d0 0x7f5260136620 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f5250007fd0 tx=0x7f5250007a40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:52.466 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.454+0000 7f523f7fe700 1 -- 172.21.15.23:0/1043657705 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525000e4a0 con 0x7f52600619d0 2020-11-20T11:39:52.466 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.454+0000 7f523f7fe700 1 -- 172.21.15.23:0/1043657705 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f525000e620 con 0x7f52600619d0 2020-11-20T11:39:52.466 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.454+0000 7f5265b55700 1 -- 172.21.15.23:0/1043657705 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52600ae6c0 con 0x7f52600619d0 2020-11-20T11:39:52.466 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.454+0000 7f523f7fe700 1 -- 172.21.15.23:0/1043657705 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525000e9e0 con 0x7f52600619d0 2020-11-20T11:39:52.467 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.454+0000 7f5265b55700 1 -- 172.21.15.23:0/1043657705 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52600adcf0 con 0x7f52600619d0 2020-11-20T11:39:52.467 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.455+0000 7f5265b55700 1 --2- 172.21.15.23:0/1043657705 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f526005c070 0x7f52600ad100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:52.467 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.455+0000 7f5265b55700 1 -- 172.21.15.23:0/1043657705 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52600adfa0 con 0x7f526005c070 2020-11-20T11:39:52.467 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.455+0000 7f525effd700 1 --2- 172.21.15.23:0/1043657705 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f526005c070 0x7f52600ad100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:52.468 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.455+0000 7f525effd700 1 -- 172.21.15.23:0/1043657705 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f52600ae850 con 0x7f526005c070 2020-11-20T11:39:52.468 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.455+0000 7f523f7fe700 1 -- 172.21.15.23:0/1043657705 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5250010030 con 0x7f52600619d0 2020-11-20T11:39:52.468 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.455+0000 7f525effd700 1 --2- 172.21.15.23:0/1043657705 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f526005c070 0x7f52600ad100 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f526005c9b0 tx=0x7f5254006e30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:52.468 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.456+0000 7f523f7fe700 1 --2- 172.21.15.23:0/1043657705 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f523805f570 0x7f5238061a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:52.468 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.456+0000 7f523f7fe700 1 -- 172.21.15.23:0/1043657705 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f525006fe00 con 0x7f52600619d0 2020-11-20T11:39:52.469 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.456+0000 7f523f7fe700 1 -- 172.21.15.23:0/1043657705 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5254018070 con 0x7f526005c070 2020-11-20T11:39:52.469 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.456+0000 7f525ffff700 1 --2- 172.21.15.23:0/1043657705 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f523805f570 0x7f5238061a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:52.469 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.456+0000 7f523f7fe700 1 -- 172.21.15.23:0/1043657705 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f5254000e90 con 0x7f526005c070 2020-11-20T11:39:52.469 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.456+0000 7f523f7fe700 1 -- 172.21.15.23:0/1043657705 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f526005c070 msgr2=0x7f52600ad100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:52.470 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.456+0000 7f523f7fe700 1 --2- 172.21.15.23:0/1043657705 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f526005c070 0x7f52600ad100 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f526005c9b0 tx=0x7f5254006e30).stop 2020-11-20T11:39:52.470 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.456+0000 7f523f7fe700 1 -- 172.21.15.23:0/1043657705 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f526005c070 msgr2=0x7f52600ad100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:39:52.470 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.456+0000 7f523f7fe700 1 --2- 172.21.15.23:0/1043657705 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f526005c070 0x7f52600ad100 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:52.470 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.456+0000 7f525ffff700 1 --2- 172.21.15.23:0/1043657705 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f523805f570 0x7f5238061a30 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f52600665b0 tx=0x7f5258016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:52.472 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.466+0000 7f5265b55700 1 --2- 172.21.15.23:0/1043657705 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f52600622d0 0x7f52600a13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:52.472 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.466+0000 7f5265b55700 1 -- 172.21.15.23:0/1043657705 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52600aea20 con 0x7f52600622d0 2020-11-20T11:39:52.473 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.467+0000 7f525effd700 1 --2- 172.21.15.23:0/1043657705 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f52600622d0 0x7f52600a13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:52.473 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.467+0000 7f525effd700 1 -- 172.21.15.23:0/1043657705 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f52600652b0 con 0x7f52600622d0 2020-11-20T11:39:52.473 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.467+0000 7f525effd700 1 --2- 172.21.15.23:0/1043657705 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f52600622d0 0x7f52600a13c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f526005c9b0 tx=0x7f525401a000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:52.473 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.468+0000 7f523f7fe700 1 -- 172.21.15.23:0/1043657705 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5254006240 con 0x7f52600622d0 2020-11-20T11:39:52.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.468+0000 7f523f7fe700 1 -- 172.21.15.23:0/1043657705 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4274 (secure 0 0 0) 0x7f5254027070 con 0x7f52600622d0 2020-11-20T11:39:52.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.468+0000 7f523f7fe700 1 -- 172.21.15.23:0/1043657705 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f52600622d0 msgr2=0x7f52600a13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:52.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.468+0000 7f523f7fe700 1 --2- 172.21.15.23:0/1043657705 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f52600622d0 0x7f52600a13c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f526005c9b0 tx=0x7f525401a000).stop 2020-11-20T11:39:52.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.468+0000 7f523f7fe700 1 -- 172.21.15.23:0/1043657705 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f52600622d0 msgr2=0x7f52600a13c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:39:52.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.468+0000 7f523f7fe700 1 --2- 172.21.15.23:0/1043657705 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f52600622d0 0x7f52600a13c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:52.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.468+0000 7f5265b55700 1 -- 172.21.15.23:0/1043657705 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f523805f570 msgr2=0x7f5238061a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:52.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.468+0000 7f5265b55700 1 --2- 172.21.15.23:0/1043657705 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f523805f570 0x7f5238061a30 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f52600665b0 tx=0x7f5258016000).stop 2020-11-20T11:39:52.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.468+0000 7f5265b55700 1 -- 172.21.15.23:0/1043657705 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f52600619d0 msgr2=0x7f5260136620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:52.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.468+0000 7f5265b55700 1 --2- 172.21.15.23:0/1043657705 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f52600619d0 0x7f5260136620 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f5250007fd0 tx=0x7f5250007a40).stop 2020-11-20T11:39:52.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.468+0000 7f525ffff700 1 -- 172.21.15.23:0/1043657705 reap_dead start 2020-11-20T11:39:52.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.468+0000 7f5265b55700 1 -- 172.21.15.23:0/1043657705 shutdown_connections 2020-11-20T11:39:52.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.468+0000 7f5265b55700 1 --2- 172.21.15.23:0/1043657705 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f52600619d0 0x7f5260136620 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:52.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.469+0000 7f5265b55700 1 -- 172.21.15.23:0/1043657705 >> 172.21.15.23:0/1043657705 conn(0x7f5260063640 msgr2=0x7f5260067800 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:52.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.469+0000 7f5265b55700 1 -- 172.21.15.23:0/1043657705 shutdown_connections 2020-11-20T11:39:52.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.469+0000 7f5265b55700 1 -- 172.21.15.23:0/1043657705 wait complete. 2020-11-20T11:39:52.477 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:39:52.477 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:39:52.477 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:39:52.478 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:39:52.478 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 72, 2020-11-20T11:39:52.478 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:39:52.478 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:39:52.478 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:39:52.479 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:39:52.479 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:52.479 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 16, 2020-11-20T11:39:52.479 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:39:52.479 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:39:52.479 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:39:52.480 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:39:52.480 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:39:52.480 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:39:52.480 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:39:52.481 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:39:52.481 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:39:52.481 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:39:52.481 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:39:52.481 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:52.482 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:39:52.482 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:39:52.482 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:39:52.482 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:39:52.482 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:39:52.483 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:39:52.483 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:39:52.483 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:39:52.483 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:39:52.484 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:39:52.484 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:52.484 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.484 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:39:52.484 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:39:52.485 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:39:52.485 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:39:52.485 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:39:52.485 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:39:52.485 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:39:52.486 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:39:52.486 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:39:52.486 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:39:52.486 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:39:52.487 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:39:52.487 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:39:52.487 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:39:52.487 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:39:52.487 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:39:52.488 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:39:52.488 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:39:52.488 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:39:52.488 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:39:52.488 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:39:52.489 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:39:52.489 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:39:52.489 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:52.489 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:39:52.490 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.490 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:39:52.490 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.490 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:39:52.490 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:39:52.491 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:39:52.491 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:39:52.491 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.491 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:39:52.491 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:39:52.492 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:52.492 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.492 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.492 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:39:52.492 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:39:52.493 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:52.493 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:52.493 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:52.493 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.494 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:39:52.494 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:39:52.494 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:39:52.494 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:39:52.494 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:39:52.495 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.495 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.495 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:39:52.495 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:39:52.495 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:39:52.496 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:39:52.496 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.496 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:39:52.496 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:39:52.496 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:52.497 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.497 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.497 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:39:52.497 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:39:52.497 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:52.498 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:52.498 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:52.498 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.498 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:39:52.499 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:39:52.499 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:39:52.499 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:39:52.499 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:39:52.499 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.500 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.500 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:39:52.500 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:39:52.500 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:39:52.501 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:39:52.501 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.501 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:39:52.501 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:39:52.501 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:52.502 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.502 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.502 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:39:52.502 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:39:52.502 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:39:52.503 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:52.503 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:52.503 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.503 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:39:52.503 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:39:52.504 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:39:52.504 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:39:52.504 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:39:52.504 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:52.504 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:52.505 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.505 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:39:52.505 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:39:52.505 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.505 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:39:52.506 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:39:52.506 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:39:52.506 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:52.506 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:39:52.507 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:39:52.507 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:39:52.507 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:39:52.507 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:39:52.507 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:39:52.508 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:39:52.508 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:39:52.508 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:39:52.508 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:39:52.508 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:39:52.510 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2020-11-20T11:39:52.510 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2020-11-20T11:39:52.510 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2020-11-20T11:39:52.510 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2020-11-20T11:39:52.511 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2020-11-20T11:39:52.511 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T11:39:52.526 INFO:tasks.ceph.mon.b:Stopped 2020-11-20T11:39:52.526 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2020-11-20T11:39:52.526 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:39:52.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.656+0000 7f41c9473700 1 Processor -- start 2020-11-20T11:39:52.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.657+0000 7f41c9473700 1 -- start start 2020-11-20T11:39:52.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.657+0000 7f41c9473700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41c40ace70 0x7f41c413a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:52.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.657+0000 7f41c9473700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f41c40ad2f0 0x7f41c413ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:52.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.657+0000 7f41c9473700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41c40ad860 con 0x7f41c40a9040 2020-11-20T11:39:52.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.657+0000 7f41c9473700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41c40aa8a0 con 0x7f41c40ad2f0 2020-11-20T11:39:52.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.657+0000 7f41c9473700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41c40aaa90 con 0x7f41c40ace70 2020-11-20T11:39:52.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.657+0000 7f41c3fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f41c40a9040 0x7f41c40aaf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58332/0 (socket says 172.21.15.23:58332) 2020-11-20T11:39:52.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.657+0000 7f41c3fff700 1 -- 172.21.15.23:0/2050118504 learned_addr learned my addr 172.21.15.23:0/2050118504 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:39:52.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.657+0000 7f41c8c72700 1 -- 172.21.15.23:0/2050118504 >> v2:172.21.15.155:3300/0 conn(0x7f41c40ad2f0 msgr2=0x7f41c413ad20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:39:52.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.657+0000 7f41c8c72700 1 --2- 172.21.15.23:0/2050118504 >> v2:172.21.15.155:3300/0 conn(0x7f41c40ad2f0 0x7f41c413ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:39:52.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.658+0000 7f41c37fe700 1 --2- 172.21.15.23:0/2050118504 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41c40ace70 0x7f41c413a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:52.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.658+0000 7f41c2ffd700 1 -- 172.21.15.23:0/2050118504 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1086494030 0 0) 0x7f41c40ad860 con 0x7f41c40a9040 2020-11-20T11:39:52.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.658+0000 7f41c2ffd700 1 -- 172.21.15.23:0/2050118504 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41b0003280 con 0x7f41c40a9040 2020-11-20T11:39:52.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.658+0000 7f41c37fe700 1 -- 172.21.15.23:0/2050118504 >> v2:172.21.15.155:3300/0 conn(0x7f41c40ad2f0 msgr2=0x7f41c413ad20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:39:52.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.658+0000 7f41c37fe700 1 --2- 172.21.15.23:0/2050118504 >> v2:172.21.15.155:3300/0 conn(0x7f41c40ad2f0 0x7f41c413ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:39:52.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.658+0000 7f41c37fe700 1 -- 172.21.15.23:0/2050118504 >> v1:172.21.15.23:6789/0 conn(0x7f41c40a9040 legacy=0x7f41c40aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:52.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.658+0000 7f41c37fe700 1 -- 172.21.15.23:0/2050118504 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c40a9420 con 0x7f41c40ace70 2020-11-20T11:39:52.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.658+0000 7f41c37fe700 1 --2- 172.21.15.23:0/2050118504 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41c40ace70 0x7f41c413a830 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f41b8006fd0 tx=0x7f41b8006e30).ready entity=mon.2 client_cookie=3dc66595e155e206 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:52.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.658+0000 7f41c2ffd700 1 -- 172.21.15.23:0/2050118504 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b80097e0 con 0x7f41c40ace70 2020-11-20T11:39:52.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.658+0000 7f41c2ffd700 1 -- 172.21.15.23:0/2050118504 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41b8000e90 con 0x7f41c40ace70 2020-11-20T11:39:52.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.658+0000 7f41c2ffd700 1 -- 172.21.15.23:0/2050118504 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b8009b50 con 0x7f41c40ace70 2020-11-20T11:39:52.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.659+0000 7f41c9473700 1 -- 172.21.15.23:0/2050118504 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41c40ace70 msgr2=0x7f41c413a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:52.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.659+0000 7f41c9473700 1 --2- 172.21.15.23:0/2050118504 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41c40ace70 0x7f41c413a830 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f41b8006fd0 tx=0x7f41b8006e30).stop 2020-11-20T11:39:52.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.659+0000 7f41c9473700 1 -- 172.21.15.23:0/2050118504 shutdown_connections 2020-11-20T11:39:52.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.659+0000 7f41c9473700 1 --2- 172.21.15.23:0/2050118504 >> v2:172.21.15.155:3300/0 conn(0x7f41c40ad2f0 0x7f41c413ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:39:52.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.659+0000 7f41c9473700 1 --2- 172.21.15.23:0/2050118504 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41c40ace70 0x7f41c413a830 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:52.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.659+0000 7f41c9473700 1 -- 172.21.15.23:0/2050118504 >> 172.21.15.23:0/2050118504 conn(0x7f41c40683b0 msgr2=0x7f41c40b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:52.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.659+0000 7f41c9473700 1 -- 172.21.15.23:0/2050118504 shutdown_connections 2020-11-20T11:39:52.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.659+0000 7f41c9473700 1 -- 172.21.15.23:0/2050118504 wait complete. 2020-11-20T11:39:52.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.660+0000 7f41c9473700 1 Processor -- start 2020-11-20T11:39:52.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.660+0000 7f41c9473700 1 -- start start 2020-11-20T11:39:52.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.661+0000 7f41c9473700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41c40a9040 0x7f41c409ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:52.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.661+0000 7f41c9473700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41c40ace70 0x7f41c40a04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:52.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.661+0000 7f41c9473700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41c40ad2f0 0x7f41c4149ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:52.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.661+0000 7f41c9473700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c409d620 con 0x7f41c40a9040 2020-11-20T11:39:52.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.661+0000 7f41c9473700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c409d8c0 con 0x7f41c40ad2f0 2020-11-20T11:39:52.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.661+0000 7f41c9473700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41c409db60 con 0x7f41c40ace70 2020-11-20T11:39:52.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.661+0000 7f41c3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41c40a9040 0x7f41c409ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:52.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.661+0000 7f41c3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41c40a9040 0x7f41c409ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44088/0 (socket says 172.21.15.23:44088) 2020-11-20T11:39:52.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.661+0000 7f41c3fff700 1 -- 172.21.15.23:0/3095084375 learned_addr learned my addr 172.21.15.23:0/3095084375 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:39:52.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.661+0000 7f41c37fe700 1 --2- 172.21.15.23:0/3095084375 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41c40ace70 0x7f41c40a04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:52.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.661+0000 7f41c8c72700 1 -- 172.21.15.23:0/3095084375 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41c40ad2f0 msgr2=0x7f41c4149ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:39:52.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.661+0000 7f41c8c72700 1 --2- 172.21.15.23:0/3095084375 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41c40ad2f0 0x7f41c4149ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:39:52.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.661+0000 7f41c3fff700 1 -- 172.21.15.23:0/3095084375 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41c40ace70 msgr2=0x7f41c40a04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:52.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.661+0000 7f41c3fff700 1 --2- 172.21.15.23:0/3095084375 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41c40ace70 0x7f41c40a04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:52.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.661+0000 7f41c3fff700 1 -- 172.21.15.23:0/3095084375 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41c40ad2f0 msgr2=0x7f41c4149ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:39:52.675 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.661+0000 7f41c3fff700 1 --2- 172.21.15.23:0/3095084375 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41c40ad2f0 0x7f41c4149ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:39:52.675 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.661+0000 7f41c3fff700 1 -- 172.21.15.23:0/3095084375 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c413dae0 con 0x7f41c40a9040 2020-11-20T11:39:52.675 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.661+0000 7f41c3fff700 1 --2- 172.21.15.23:0/3095084375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41c40a9040 0x7f41c409ea90 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f41b400c6e0 tx=0x7f41b40028d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:52.675 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.661+0000 7f41a3fff700 1 -- 172.21.15.23:0/3095084375 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b40072e0 con 0x7f41c40a9040 2020-11-20T11:39:52.675 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.661+0000 7f41a3fff700 1 -- 172.21.15.23:0/3095084375 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41b4005500 con 0x7f41c40a9040 2020-11-20T11:39:52.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.661+0000 7f41c9473700 1 -- 172.21.15.23:0/3095084375 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41c413e290 con 0x7f41c40a9040 2020-11-20T11:39:52.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.661+0000 7f41a3fff700 1 -- 172.21.15.23:0/3095084375 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b4005880 con 0x7f41c40a9040 2020-11-20T11:39:52.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.661+0000 7f41c9473700 1 -- 172.21.15.23:0/3095084375 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41c40a09d0 con 0x7f41c40a9040 2020-11-20T11:39:52.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.662+0000 7f41c9473700 1 -- 172.21.15.23:0/3095084375 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41c409dd20 con 0x7f41c40a9040 2020-11-20T11:39:52.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.663+0000 7f41a3fff700 1 -- 172.21.15.23:0/3095084375 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f41b4010070 con 0x7f41c40a9040 2020-11-20T11:39:52.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.663+0000 7f41a3fff700 1 --2- 172.21.15.23:0/3095084375 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f419c056d10 0x7f419c0591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:52.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.663+0000 7f41a3fff700 1 -- 172.21.15.23:0/3095084375 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f41b406f7c0 con 0x7f41c40a9040 2020-11-20T11:39:52.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.663+0000 7f41c37fe700 1 --2- 172.21.15.23:0/3095084375 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f419c056d10 0x7f419c0591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:52.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.664+0000 7f41c37fe700 1 --2- 172.21.15.23:0/3095084375 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f419c056d10 0x7f419c0591d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f41b8006e70 tx=0x7f41b8015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:52.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.668+0000 7f41a3fff700 1 -- 172.21.15.23:0/3095084375 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f41b4044ab0 con 0x7f41c40a9040 2020-11-20T11:39:52.906 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.903+0000 7f41c9473700 1 -- 172.21.15.23:0/3095084375 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f41c4057dc0 con 0x7f41c40a9040 2020-11-20T11:39:52.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.904+0000 7f41a3fff700 1 -- 172.21.15.23:0/3095084375 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7f41b40480d0 con 0x7f41c40a9040 2020-11-20T11:39:52.907 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":72,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:39:52.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.905+0000 7f41c9473700 1 -- 172.21.15.23:0/3095084375 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f419c056d10 msgr2=0x7f419c0591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:52.909 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.905+0000 7f41c9473700 1 --2- 172.21.15.23:0/3095084375 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f419c056d10 0x7f419c0591d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f41b8006e70 tx=0x7f41b8015000).stop 2020-11-20T11:39:52.909 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.905+0000 7f41c9473700 1 -- 172.21.15.23:0/3095084375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41c40a9040 msgr2=0x7f41c409ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:52.909 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.905+0000 7f41c9473700 1 --2- 172.21.15.23:0/3095084375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41c40a9040 0x7f41c409ea90 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f41b400c6e0 tx=0x7f41b40028d0).stop 2020-11-20T11:39:52.909 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.905+0000 7f41c9473700 1 -- 172.21.15.23:0/3095084375 shutdown_connections 2020-11-20T11:39:52.910 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.905+0000 7f41c9473700 1 --2- 172.21.15.23:0/3095084375 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41c40ad2f0 0x7f41c4149ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:39:52.910 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.905+0000 7f41c9473700 1 --2- 172.21.15.23:0/3095084375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41c40a9040 0x7f41c409ea90 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:52.910 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.905+0000 7f41c9473700 1 --2- 172.21.15.23:0/3095084375 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f419c056d10 0x7f419c0591d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:52.910 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.905+0000 7f41c9473700 1 --2- 172.21.15.23:0/3095084375 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41c40ace70 0x7f41c40a04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:52.911 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.905+0000 7f41c9473700 1 -- 172.21.15.23:0/3095084375 >> 172.21.15.23:0/3095084375 conn(0x7f41c40683b0 msgr2=0x7f41c40aae50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:52.911 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.905+0000 7f41c9473700 1 -- 172.21.15.23:0/3095084375 shutdown_connections 2020-11-20T11:39:52.911 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:52.906+0000 7f41c9473700 1 -- 172.21.15.23:0/3095084375 wait complete. 2020-11-20T11:39:55.921 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:39:56.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.059+0000 7f215e778700 1 Processor -- start 2020-11-20T11:39:56.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.060+0000 7f215e778700 1 -- start start 2020-11-20T11:39:56.073 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.060+0000 7f215e778700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f215813bfa0 0x7f21580a9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:56.073 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.060+0000 7f215e778700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21580a8c20 con 0x7f2158067fd0 2020-11-20T11:39:56.073 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.060+0000 7f215e778700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21580684e0 con 0x7f215805bc80 2020-11-20T11:39:56.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.060+0000 7f215e778700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f215813c410 con 0x7f215813bfa0 2020-11-20T11:39:56.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.061+0000 7f214ffff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f2158067fd0 0x7f21581412f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58356/0 (socket says 172.21.15.23:58356) 2020-11-20T11:39:56.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.061+0000 7f214ffff700 1 -- 172.21.15.23:0/3038122851 learned_addr learned my addr 172.21.15.23:0/3038122851 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:39:56.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.061+0000 7f215cd15700 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f215813bfa0 msgr2=0x7f21580a9080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:39:56.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.061+0000 7f2157fff700 1 -- 172.21.15.23:0/3038122851 >> v1:172.21.15.155:6789/0 conn(0x7f215805bc80 legacy=0x7f21580a6ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:39:56.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.061+0000 7f215cd15700 1 --2- 172.21.15.23:0/3038122851 >> v2:172.21.15.155:3300/0 conn(0x7f215813bfa0 0x7f21580a9080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:39:56.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.061+0000 7f21577fe700 1 -- 172.21.15.23:0/3038122851 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1065597538 0 0) 0x7f21580a8c20 con 0x7f2158067fd0 2020-11-20T11:39:56.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.061+0000 7f21577fe700 1 -- 172.21.15.23:0/3038122851 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2138003280 con 0x7f2158067fd0 2020-11-20T11:39:56.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.061+0000 7f21577fe700 1 -- 172.21.15.23:0/3038122851 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2441294625 0 0) 0x7f2138003280 con 0x7f2158067fd0 2020-11-20T11:39:56.077 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.061+0000 7f21577fe700 1 -- 172.21.15.23:0/3038122851 >> v2:172.21.15.155:3300/0 conn(0x7f215813bfa0 msgr2=0x7f21580a9080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:39:56.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.061+0000 7f21577fe700 1 --2- 172.21.15.23:0/3038122851 >> v2:172.21.15.155:3300/0 conn(0x7f215813bfa0 0x7f21580a9080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:39:56.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.061+0000 7f21577fe700 1 -- 172.21.15.23:0/3038122851 >> v1:172.21.15.155:6789/0 conn(0x7f215805bc80 legacy=0x7f21580a6ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:39:56.078 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.061+0000 7f21577fe700 1 -- 172.21.15.23:0/3038122851 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f215805c060 con 0x7f2158067fd0 2020-11-20T11:39:56.079 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.061+0000 7f21577fe700 1 -- 172.21.15.23:0/3038122851 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f2140001520 con 0x7f2158067fd0 2020-11-20T11:39:56.079 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.061+0000 7f21577fe700 1 -- 172.21.15.23:0/3038122851 >> v1:172.21.15.23:6789/0 conn(0x7f2158067fd0 legacy=0x7f21581412f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:56.079 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.061+0000 7f21577fe700 1 --2- 172.21.15.23:0/3038122851 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2138003770 0x7f2138003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:56.080 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.061+0000 7f21577fe700 1 --2- 172.21.15.23:0/3038122851 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2138004520 0x7f21380049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:56.080 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.061+0000 7f21577fe700 1 --2- 172.21.15.23:0/3038122851 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2138004e90 0x7f213800b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:56.080 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.061+0000 7f21577fe700 1 -- 172.21.15.23:0/3038122851 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215813c410 con 0x7f2138004520 2020-11-20T11:39:56.080 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.061+0000 7f21577fe700 1 -- 172.21.15.23:0/3038122851 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2138004230 con 0x7f2138004e90 2020-11-20T11:39:56.081 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.061+0000 7f21577fe700 1 -- 172.21.15.23:0/3038122851 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f213800bc60 con 0x7f2138003770 2020-11-20T11:39:56.081 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.061+0000 7f214ffff700 1 --2- 172.21.15.23:0/3038122851 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2138004520 0x7f21380049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:56.081 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.061+0000 7f215cd15700 1 -- 172.21.15.23:0/3038122851 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2138004e90 msgr2=0x7f213800b770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:39:56.082 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.061+0000 7f215cd15700 1 --2- 172.21.15.23:0/3038122851 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2138004e90 0x7f213800b770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:39:56.082 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.062+0000 7f2157fff700 1 --2- 172.21.15.23:0/3038122851 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2138003770 0x7f2138003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:56.082 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.062+0000 7f214ffff700 1 -- 172.21.15.23:0/3038122851 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2138003770 msgr2=0x7f2138003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:39:56.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.062+0000 7f214ffff700 1 --2- 172.21.15.23:0/3038122851 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2138003770 0x7f2138003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:56.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.062+0000 7f214ffff700 1 -- 172.21.15.23:0/3038122851 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2138004e90 msgr2=0x7f213800b770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:39:56.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.062+0000 7f214ffff700 1 --2- 172.21.15.23:0/3038122851 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2138004e90 0x7f213800b770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:39:56.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.062+0000 7f215cd15700 1 -- 172.21.15.23:0/3038122851 reap_dead start 2020-11-20T11:39:56.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.062+0000 7f214ffff700 1 -- 172.21.15.23:0/3038122851 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2138003280 con 0x7f2138004520 2020-11-20T11:39:56.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.062+0000 7f2157fff700 1 --2- 172.21.15.23:0/3038122851 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2138003770 0x7f2138003b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:39:56.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.062+0000 7f214ffff700 1 --2- 172.21.15.23:0/3038122851 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2138004520 0x7f21380049a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f214000c410 tx=0x7f2158068670).ready entity=mon.0 client_cookie=b8ad03236aa0dd62 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:56.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.062+0000 7f21577fe700 1 -- 172.21.15.23:0/3038122851 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21400070d0 con 0x7f2138004520 2020-11-20T11:39:56.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.062+0000 7f21577fe700 1 -- 172.21.15.23:0/3038122851 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2140007230 con 0x7f2138004520 2020-11-20T11:39:56.086 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.062+0000 7f21577fe700 1 -- 172.21.15.23:0/3038122851 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2140005630 con 0x7f2138004520 2020-11-20T11:39:56.086 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.063+0000 7f215e778700 1 -- 172.21.15.23:0/3038122851 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2138004520 msgr2=0x7f21380049a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:56.087 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.063+0000 7f215e778700 1 --2- 172.21.15.23:0/3038122851 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2138004520 0x7f21380049a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f214000c410 tx=0x7f2158068670).stop 2020-11-20T11:39:56.087 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.063+0000 7f215e778700 1 -- 172.21.15.23:0/3038122851 shutdown_connections 2020-11-20T11:39:56.087 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.063+0000 7f215e778700 1 --2- 172.21.15.23:0/3038122851 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2138004520 0x7f21380049a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:56.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.063+0000 7f215e778700 1 -- 172.21.15.23:0/3038122851 >> 172.21.15.23:0/3038122851 conn(0x7f21580ad7a0 msgr2=0x7f21580adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:56.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.063+0000 7f215e778700 1 -- 172.21.15.23:0/3038122851 shutdown_connections 2020-11-20T11:39:56.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.063+0000 7f215e778700 1 -- 172.21.15.23:0/3038122851 wait complete. 2020-11-20T11:39:56.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.064+0000 7f215e778700 1 Processor -- start 2020-11-20T11:39:56.089 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.064+0000 7f215e778700 1 -- start start 2020-11-20T11:39:56.089 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.064+0000 7f215e778700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2138004520 0x7f215813fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:56.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.064+0000 7f215e778700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f21580a46a0 0x7f21580a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:56.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.064+0000 7f215e778700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21580a1ca0 0x7f21580ab5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:56.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.064+0000 7f215e778700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21580a2c60 con 0x7f21580a1ca0 2020-11-20T11:39:56.091 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.064+0000 7f215e778700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21580a2120 con 0x7f2138004520 2020-11-20T11:39:56.091 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.064+0000 7f215e778700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21580a2ec0 con 0x7f21580a46a0 2020-11-20T11:39:56.091 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.064+0000 7f2157fff700 1 -- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2138004520 msgr2=0x7f215813fcc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:39:56.091 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.064+0000 7f2157fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2138004520 0x7f215813fcc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:39:56.092 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.064+0000 7f215cd15700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21580a1ca0 0x7f21580ab5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:56.092 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.064+0000 7f214ffff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f21580a46a0 0x7f21580a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:56.092 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.064+0000 7f215cd15700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21580a1ca0 0x7f21580ab5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44122/0 (socket says 172.21.15.23:44122) 2020-11-20T11:39:56.093 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.064+0000 7f214ffff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f21580a46a0 0x7f21580a17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:54360/0 (socket says 172.21.15.23:54360) 2020-11-20T11:39:56.093 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.064+0000 7f215cd15700 1 -- 172.21.15.23:0/2112942493 learned_addr learned my addr 172.21.15.23:0/2112942493 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:39:56.094 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.065+0000 7f215cd15700 1 -- 172.21.15.23:0/2112942493 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f21580a46a0 msgr2=0x7f21580a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:56.095 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.065+0000 7f215cd15700 1 --2- 172.21.15.23:0/2112942493 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f21580a46a0 0x7f21580a17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:56.095 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.065+0000 7f215cd15700 1 -- 172.21.15.23:0/2112942493 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2138004520 msgr2=0x7f215813fcc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:39:56.095 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.065+0000 7f215cd15700 1 --2- 172.21.15.23:0/2112942493 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2138004520 0x7f215813fcc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:39:56.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.065+0000 7f215cd15700 1 -- 172.21.15.23:0/2112942493 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21580a3160 con 0x7f21580a1ca0 2020-11-20T11:39:56.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.065+0000 7f215cd15700 1 --2- 172.21.15.23:0/2112942493 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21580a1ca0 0x7f21580ab5b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f2158064d40 tx=0x7f2150006f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:56.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.065+0000 7f2154ff9700 1 -- 172.21.15.23:0/2112942493 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2150003ca0 con 0x7f21580a1ca0 2020-11-20T11:39:56.097 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.065+0000 7f2154ff9700 1 -- 172.21.15.23:0/2112942493 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2150003e20 con 0x7f21580a1ca0 2020-11-20T11:39:56.097 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.065+0000 7f215e778700 1 -- 172.21.15.23:0/2112942493 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21580acb70 con 0x7f21580a1ca0 2020-11-20T11:39:56.097 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.065+0000 7f2154ff9700 1 -- 172.21.15.23:0/2112942493 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2150006170 con 0x7f21580a1ca0 2020-11-20T11:39:56.097 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.065+0000 7f215e778700 1 -- 172.21.15.23:0/2112942493 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f215809d660 con 0x7f21580a1ca0 2020-11-20T11:39:56.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.066+0000 7f215e778700 1 -- 172.21.15.23:0/2112942493 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2158062fe0 con 0x7f21580a1ca0 2020-11-20T11:39:56.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.067+0000 7f2154ff9700 1 -- 172.21.15.23:0/2112942493 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f21500065a0 con 0x7f21580a1ca0 2020-11-20T11:39:56.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.067+0000 7f2154ff9700 1 --2- 172.21.15.23:0/2112942493 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2134056d10 0x7f21340591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:56.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.067+0000 7f2154ff9700 1 -- 172.21.15.23:0/2112942493 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f2150011020 con 0x7f21580a1ca0 2020-11-20T11:39:56.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.067+0000 7f2157fff700 1 --2- 172.21.15.23:0/2112942493 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2134056d10 0x7f21340591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:56.099 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.068+0000 7f2157fff700 1 --2- 172.21.15.23:0/2112942493 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2134056d10 0x7f21340591d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f215805d210 tx=0x7f2148008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:56.099 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.071+0000 7f2154ff9700 1 -- 172.21.15.23:0/2112942493 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2150044b10 con 0x7f21580a1ca0 2020-11-20T11:39:56.346 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.343+0000 7f215e778700 1 -- 172.21.15.23:0/2112942493 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2158044710 con 0x7f21580a1ca0 2020-11-20T11:39:56.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.343+0000 7f2154ff9700 1 -- 172.21.15.23:0/2112942493 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7f2150048130 con 0x7f21580a1ca0 2020-11-20T11:39:56.347 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":72,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:39:56.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.345+0000 7f215e778700 1 -- 172.21.15.23:0/2112942493 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2134056d10 msgr2=0x7f21340591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:56.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.345+0000 7f215e778700 1 --2- 172.21.15.23:0/2112942493 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2134056d10 0x7f21340591d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f215805d210 tx=0x7f2148008040).stop 2020-11-20T11:39:56.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.345+0000 7f215e778700 1 -- 172.21.15.23:0/2112942493 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21580a1ca0 msgr2=0x7f21580ab5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:56.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.345+0000 7f215e778700 1 --2- 172.21.15.23:0/2112942493 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21580a1ca0 0x7f21580ab5b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f2158064d40 tx=0x7f2150006f90).stop 2020-11-20T11:39:56.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.345+0000 7f215e778700 1 -- 172.21.15.23:0/2112942493 shutdown_connections 2020-11-20T11:39:56.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.345+0000 7f215e778700 1 --2- 172.21.15.23:0/2112942493 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21580a1ca0 0x7f21580ab5b0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:56.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.345+0000 7f215e778700 1 --2- 172.21.15.23:0/2112942493 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2138004520 0x7f215813fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:39:56.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.345+0000 7f215e778700 1 --2- 172.21.15.23:0/2112942493 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2134056d10 0x7f21340591d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:56.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.345+0000 7f215e778700 1 --2- 172.21.15.23:0/2112942493 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f21580a46a0 0x7f21580a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:56.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.345+0000 7f215e778700 1 -- 172.21.15.23:0/2112942493 >> 172.21.15.23:0/2112942493 conn(0x7f21580ad7a0 msgr2=0x7f21580a7620 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:56.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.345+0000 7f215e778700 1 -- 172.21.15.23:0/2112942493 shutdown_connections 2020-11-20T11:39:56.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:56.345+0000 7f215e778700 1 -- 172.21.15.23:0/2112942493 wait complete. 2020-11-20T11:39:59.362 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:39:59.498 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.495+0000 7fe6e359e700 1 Processor -- start 2020-11-20T11:39:59.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.495+0000 7fe6e359e700 1 -- start start 2020-11-20T11:39:59.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.496+0000 7fe6e359e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe6e40a9040 0x7fe6e40aaf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:59.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.496+0000 7fe6e359e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6e40ace70 0x7fe6e413de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:59.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.496+0000 7fe6e359e700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e40ad860 con 0x7fe6e40ad2f0 2020-11-20T11:39:59.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.496+0000 7fe6e359e700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6e40aa8a0 con 0x7fe6e40a9040 2020-11-20T11:39:59.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.496+0000 7fe6e359e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e40aaa90 con 0x7fe6e40ace70 2020-11-20T11:39:59.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.496+0000 7fe6e2d9d700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fe6e40ad2f0 0x7fe6e413e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58382/0 (socket says 172.21.15.23:58382) 2020-11-20T11:39:59.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.496+0000 7fe6e2d9d700 1 -- 172.21.15.23:0/2902545934 learned_addr learned my addr 172.21.15.23:0/2902545934 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:39:59.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.496+0000 7fe6e259c700 1 -- >> v2:172.21.15.155:3300/0 conn(0x7fe6e40a9040 msgr2=0x7fe6e40aaf30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:39:59.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.496+0000 7fe6e259c700 1 --2- 172.21.15.23:0/2902545934 >> v2:172.21.15.155:3300/0 conn(0x7fe6e40a9040 0x7fe6e40aaf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:39:59.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.496+0000 7fe6e1d9b700 1 --2- 172.21.15.23:0/2902545934 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6e40ace70 0x7fe6e413de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:59.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.497+0000 7fe6e1d9b700 1 -- 172.21.15.23:0/2902545934 >> v2:172.21.15.155:3300/0 conn(0x7fe6e40a9040 msgr2=0x7fe6e40aaf30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:39:59.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.497+0000 7fe6e1d9b700 1 --2- 172.21.15.23:0/2902545934 >> v2:172.21.15.155:3300/0 conn(0x7fe6e40a9040 0x7fe6e40aaf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:39:59.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.497+0000 7fe6e1d9b700 1 -- 172.21.15.23:0/2902545934 >> v1:172.21.15.23:6789/0 conn(0x7fe6e40ad2f0 legacy=0x7fe6e413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:59.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.497+0000 7fe6e1d9b700 1 -- 172.21.15.23:0/2902545934 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e40a9420 con 0x7fe6e40ace70 2020-11-20T11:39:59.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.497+0000 7fe6e1d9b700 1 --2- 172.21.15.23:0/2902545934 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6e40ace70 0x7fe6e413de00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fe6d8006fd0 tx=0x7fe6d8006e30).ready entity=mon.2 client_cookie=7834c22abe9de4fa server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:59.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.497+0000 7fe6e159a700 1 -- 172.21.15.23:0/2902545934 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d80097e0 con 0x7fe6e40ace70 2020-11-20T11:39:59.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.497+0000 7fe6e159a700 1 -- 172.21.15.23:0/2902545934 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d8000e90 con 0x7fe6e40ace70 2020-11-20T11:39:59.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.497+0000 7fe6e159a700 1 -- 172.21.15.23:0/2902545934 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d8009b50 con 0x7fe6e40ace70 2020-11-20T11:39:59.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.498+0000 7fe6e359e700 1 -- 172.21.15.23:0/2902545934 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6e40ace70 msgr2=0x7fe6e413de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:59.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.498+0000 7fe6e359e700 1 --2- 172.21.15.23:0/2902545934 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6e40ace70 0x7fe6e413de00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fe6d8006fd0 tx=0x7fe6d8006e30).stop 2020-11-20T11:39:59.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.498+0000 7fe6e359e700 1 -- 172.21.15.23:0/2902545934 shutdown_connections 2020-11-20T11:39:59.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.498+0000 7fe6e359e700 1 --2- 172.21.15.23:0/2902545934 >> v2:172.21.15.155:3300/0 conn(0x7fe6e40a9040 0x7fe6e40aaf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:39:59.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.498+0000 7fe6e359e700 1 --2- 172.21.15.23:0/2902545934 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6e40ace70 0x7fe6e413de00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:59.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.498+0000 7fe6e359e700 1 -- 172.21.15.23:0/2902545934 >> 172.21.15.23:0/2902545934 conn(0x7fe6e40683b0 msgr2=0x7fe6e40b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:59.508 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.498+0000 7fe6e359e700 1 -- 172.21.15.23:0/2902545934 shutdown_connections 2020-11-20T11:39:59.508 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.498+0000 7fe6e359e700 1 -- 172.21.15.23:0/2902545934 wait complete. 2020-11-20T11:39:59.508 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.499+0000 7fe6e359e700 1 Processor -- start 2020-11-20T11:39:59.508 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.499+0000 7fe6e359e700 1 -- start start 2020-11-20T11:39:59.508 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.499+0000 7fe6e359e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6e40a9040 0x7fe6e4137930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:59.509 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.499+0000 7fe6e359e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6e40ace70 0x7fe6e4135440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:59.509 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.499+0000 7fe6e359e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6e40ad2f0 0x7fe6e4135930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:59.509 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.499+0000 7fe6e359e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e4135e20 con 0x7fe6e40a9040 2020-11-20T11:39:59.509 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.499+0000 7fe6e359e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e409d9b0 con 0x7fe6e40ace70 2020-11-20T11:39:59.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.499+0000 7fe6e359e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e409dc50 con 0x7fe6e40ad2f0 2020-11-20T11:39:59.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.500+0000 7fe6e259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6e40a9040 0x7fe6e4137930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:59.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.500+0000 7fe6e259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6e40a9040 0x7fe6e4137930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44134/0 (socket says 172.21.15.23:44134) 2020-11-20T11:39:59.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.500+0000 7fe6e1d9b700 1 -- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6e40ace70 msgr2=0x7fe6e4135440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:39:59.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.500+0000 7fe6e259c700 1 -- 172.21.15.23:0/144910347 learned_addr learned my addr 172.21.15.23:0/144910347 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:39:59.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.500+0000 7fe6e1d9b700 1 --2- 172.21.15.23:0/144910347 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6e40ace70 0x7fe6e4135440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:39:59.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.500+0000 7fe6e2d9d700 1 --2- 172.21.15.23:0/144910347 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6e40ad2f0 0x7fe6e4135930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:59.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.500+0000 7fe6e259c700 1 -- 172.21.15.23:0/144910347 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6e40ad2f0 msgr2=0x7fe6e4135930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:59.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.500+0000 7fe6e259c700 1 --2- 172.21.15.23:0/144910347 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6e40ad2f0 0x7fe6e4135930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:59.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.500+0000 7fe6e259c700 1 -- 172.21.15.23:0/144910347 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6e40ace70 msgr2=0x7fe6e4135440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:39:59.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.500+0000 7fe6e259c700 1 --2- 172.21.15.23:0/144910347 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6e40ace70 0x7fe6e4135440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:39:59.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.500+0000 7fe6e259c700 1 -- 172.21.15.23:0/144910347 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e409def0 con 0x7fe6e40a9040 2020-11-20T11:39:59.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.500+0000 7fe6e2d9d700 1 --2- 172.21.15.23:0/144910347 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6e40ad2f0 0x7fe6e4135930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:39:59.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.500+0000 7fe6e259c700 1 --2- 172.21.15.23:0/144910347 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6e40a9040 0x7fe6e4137930 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fe6d400bfd0 tx=0x7fe6d4006e90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:59.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.500+0000 7fe6d27fc700 1 -- 172.21.15.23:0/144910347 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d4006170 con 0x7fe6e40a9040 2020-11-20T11:39:59.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.500+0000 7fe6d27fc700 1 -- 172.21.15.23:0/144910347 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d4006310 con 0x7fe6e40a9040 2020-11-20T11:39:59.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.500+0000 7fe6e359e700 1 -- 172.21.15.23:0/144910347 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e413e330 con 0x7fe6e40a9040 2020-11-20T11:39:59.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.500+0000 7fe6d27fc700 1 -- 172.21.15.23:0/144910347 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d4003970 con 0x7fe6e40a9040 2020-11-20T11:39:59.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.500+0000 7fe6e359e700 1 -- 172.21.15.23:0/144910347 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e409d6e0 con 0x7fe6e40a9040 2020-11-20T11:39:59.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.501+0000 7fe6e359e700 1 -- 172.21.15.23:0/144910347 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6e409e080 con 0x7fe6e40a9040 2020-11-20T11:39:59.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.502+0000 7fe6d27fc700 1 -- 172.21.15.23:0/144910347 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe6d4017030 con 0x7fe6e40a9040 2020-11-20T11:39:59.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.502+0000 7fe6d27fc700 1 --2- 172.21.15.23:0/144910347 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe6b8056d10 0x7fe6b80591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:39:59.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.502+0000 7fe6d27fc700 1 -- 172.21.15.23:0/144910347 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fe6d406f220 con 0x7fe6e40a9040 2020-11-20T11:39:59.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.502+0000 7fe6e1d9b700 1 --2- 172.21.15.23:0/144910347 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe6b8056d10 0x7fe6b80591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:39:59.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.503+0000 7fe6e1d9b700 1 --2- 172.21.15.23:0/144910347 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe6b8056d10 0x7fe6b80591d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fe6e40a01f0 tx=0x7fe6d8015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:39:59.515 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.506+0000 7fe6d27fc700 1 -- 172.21.15.23:0/144910347 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe6d40445c0 con 0x7fe6e40a9040 2020-11-20T11:39:59.732 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.729+0000 7fe6e359e700 1 -- 172.21.15.23:0/144910347 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe6e4057dc0 con 0x7fe6e40a9040 2020-11-20T11:39:59.733 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.730+0000 7fe6d27fc700 1 -- 172.21.15.23:0/144910347 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7fe6d4047be0 con 0x7fe6e40a9040 2020-11-20T11:39:59.733 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":72,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":23,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:39:59.735 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.731+0000 7fe6e359e700 1 -- 172.21.15.23:0/144910347 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe6b8056d10 msgr2=0x7fe6b80591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:59.735 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.731+0000 7fe6e359e700 1 --2- 172.21.15.23:0/144910347 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe6b8056d10 0x7fe6b80591d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fe6e40a01f0 tx=0x7fe6d8015000).stop 2020-11-20T11:39:59.735 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.731+0000 7fe6e359e700 1 -- 172.21.15.23:0/144910347 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6e40a9040 msgr2=0x7fe6e4137930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:39:59.735 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.731+0000 7fe6e359e700 1 --2- 172.21.15.23:0/144910347 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6e40a9040 0x7fe6e4137930 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fe6d400bfd0 tx=0x7fe6d4006e90).stop 2020-11-20T11:39:59.736 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.731+0000 7fe6e359e700 1 -- 172.21.15.23:0/144910347 shutdown_connections 2020-11-20T11:39:59.736 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.731+0000 7fe6e359e700 1 --2- 172.21.15.23:0/144910347 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe6b8056d10 0x7fe6b80591d0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:59.736 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.731+0000 7fe6e359e700 1 --2- 172.21.15.23:0/144910347 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe6e40ad2f0 0x7fe6e4135930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:59.736 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.731+0000 7fe6e359e700 1 --2- 172.21.15.23:0/144910347 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6e40a9040 0x7fe6e4137930 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:39:59.736 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.731+0000 7fe6e359e700 1 --2- 172.21.15.23:0/144910347 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe6e40ace70 0x7fe6e4135440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:39:59.737 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.731+0000 7fe6e359e700 1 -- 172.21.15.23:0/144910347 >> 172.21.15.23:0/144910347 conn(0x7fe6e40683b0 msgr2=0x7fe6e40aba40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:39:59.737 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.732+0000 7fe6e359e700 1 -- 172.21.15.23:0/144910347 shutdown_connections 2020-11-20T11:39:59.737 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:39:59.732+0000 7fe6e359e700 1 -- 172.21.15.23:0/144910347 wait complete. 2020-11-20T11:40:00.940 INFO:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:40:00.942 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:40:02.747 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:40:02.871 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:02.868+0000 7f3fe8d24700 1 Processor -- start 2020-11-20T11:40:02.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:02.868+0000 7f3fe8d24700 1 -- start start 2020-11-20T11:40:02.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:02.869+0000 7f3fe8d24700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3fe40aaab0 0x7f3fe413ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:02.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:02.869+0000 7f3fe8d24700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3fe40aae90 0x7f3fe41381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:02.869+0000 7f3fe8d24700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fe413aeb0 0x7f3fe40a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:02.869+0000 7f3fe8d24700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fe413e600 con 0x7f3fe40aaab0 2020-11-20T11:40:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:02.869+0000 7f3fe8d24700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fe413e8a0 con 0x7f3fe40aae90 2020-11-20T11:40:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:02.869+0000 7f3fe8d24700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe40ab6d0 con 0x7f3fe413aeb0 2020-11-20T11:40:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:02.869+0000 7f3fe259c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3fe40aaab0 0x7f3fe413ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:02.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:02.869+0000 7f3fdbfff700 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f3fe40aae90 msgr2=0x7f3fe41381a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:40:02.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:02.869+0000 7f3fdbfff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3fe40aae90 0x7f3fe41381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:40:02.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:02.869+0000 7f3fe259c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3fe40aaab0 0x7f3fe413ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44142/0 (socket says 172.21.15.23:44142) 2020-11-20T11:40:02.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:02.869+0000 7f3fe259c700 1 -- 172.21.15.23:0/1069458099 learned_addr learned my addr 172.21.15.23:0/1069458099 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:40:02.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:02.869+0000 7f3fe2d9d700 1 --2- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fe413aeb0 0x7f3fe40a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:02.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:02.870+0000 7f3fe259c700 1 -- 172.21.15.23:0/1069458099 >> v2:172.21.15.23:3300/0 conn(0x7f3fe40aaab0 msgr2=0x7f3fe413ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:40:02.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:02.870+0000 7f3fe259c700 1 -- 172.21.15.23:0/1069458099 >> v2:172.21.15.23:3300/0 conn(0x7f3fe40aaab0 msgr2=0x7f3fe413ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:40:02.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:02.870+0000 7f3fe259c700 1 --2- 172.21.15.23:0/1069458099 >> v2:172.21.15.23:3300/0 conn(0x7f3fe40aaab0 0x7f3fe413ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:40:02.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:02.870+0000 7f3fe259c700 1 --2- 172.21.15.23:0/1069458099 >> v2:172.21.15.23:3300/0 conn(0x7f3fe40aaab0 0x7f3fe413ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:40:02.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:02.870+0000 7f3fe2d9d700 1 -- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fe413aeb0 msgr2=0x7f3fe40a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:40:02.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:02.870+0000 7f3fe2d9d700 1 -- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fe413aeb0 msgr2=0x7f3fe40a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:40:02.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:02.870+0000 7f3fe2d9d700 1 --2- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fe413aeb0 0x7f3fe40a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:40:02.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:02.870+0000 7f3fe2d9d700 1 --2- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fe413aeb0 0x7f3fe40a8c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:40:03.073 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:03.070+0000 7f3fdbfff700 1 -- 172.21.15.23:0/1069458099 >> v2:172.21.15.155:3300/0 conn(0x7f3fe40aae90 msgr2=0x7f3fe41381a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:40:03.073 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:03.070+0000 7f3fdbfff700 1 --2- 172.21.15.23:0/1069458099 >> v2:172.21.15.155:3300/0 conn(0x7f3fe40aae90 0x7f3fe41381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:40:03.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:03.071+0000 7f3fe259c700 1 --2- 172.21.15.23:0/1069458099 >> v2:172.21.15.23:3300/0 conn(0x7f3fe40aaab0 0x7f3fe413ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:03.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:03.071+0000 7f3fe259c700 1 -- 172.21.15.23:0/1069458099 >> v2:172.21.15.23:3300/0 conn(0x7f3fe40aaab0 msgr2=0x7f3fe413ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:40:03.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:03.071+0000 7f3fe259c700 1 -- 172.21.15.23:0/1069458099 >> v2:172.21.15.23:3300/0 conn(0x7f3fe40aaab0 msgr2=0x7f3fe413ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:40:03.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:03.071+0000 7f3fe259c700 1 --2- 172.21.15.23:0/1069458099 >> v2:172.21.15.23:3300/0 conn(0x7f3fe40aaab0 0x7f3fe413ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:40:03.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:03.071+0000 7f3fe259c700 1 --2- 172.21.15.23:0/1069458099 >> v2:172.21.15.23:3300/0 conn(0x7f3fe40aaab0 0x7f3fe413ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:40:03.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:03.071+0000 7f3fe2d9d700 1 --2- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fe413aeb0 0x7f3fe40a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:03.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:03.071+0000 7f3fe2d9d700 1 -- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fe413aeb0 msgr2=0x7f3fe40a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:40:03.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:03.071+0000 7f3fe2d9d700 1 -- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fe413aeb0 msgr2=0x7f3fe40a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:40:03.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:03.071+0000 7f3fe2d9d700 1 --2- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fe413aeb0 0x7f3fe40a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:40:03.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:03.071+0000 7f3fe2d9d700 1 --2- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fe413aeb0 0x7f3fe40a8c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:40:03.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:03.471+0000 7f3fdbfff700 1 -- 172.21.15.23:0/1069458099 >> v2:172.21.15.155:3300/0 conn(0x7f3fe40aae90 msgr2=0x7f3fe41381a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:40:03.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:03.471+0000 7f3fdbfff700 1 --2- 172.21.15.23:0/1069458099 >> v2:172.21.15.155:3300/0 conn(0x7f3fe40aae90 0x7f3fe41381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:40:03.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:03.472+0000 7f3fe259c700 1 --2- 172.21.15.23:0/1069458099 >> v2:172.21.15.23:3300/0 conn(0x7f3fe40aaab0 0x7f3fe413ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:03.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:03.472+0000 7f3fe259c700 1 -- 172.21.15.23:0/1069458099 >> v2:172.21.15.23:3300/0 conn(0x7f3fe40aaab0 msgr2=0x7f3fe413ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:40:03.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:03.472+0000 7f3fe259c700 1 -- 172.21.15.23:0/1069458099 >> v2:172.21.15.23:3300/0 conn(0x7f3fe40aaab0 msgr2=0x7f3fe413ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:40:03.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:03.472+0000 7f3fe259c700 1 --2- 172.21.15.23:0/1069458099 >> v2:172.21.15.23:3300/0 conn(0x7f3fe40aaab0 0x7f3fe413ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:40:03.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:03.472+0000 7f3fe2d9d700 1 --2- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fe413aeb0 0x7f3fe40a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:03.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:03.472+0000 7f3fe259c700 1 --2- 172.21.15.23:0/1069458099 >> v2:172.21.15.23:3300/0 conn(0x7f3fe40aaab0 0x7f3fe413ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:40:03.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:03.472+0000 7f3fe2d9d700 1 -- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fe413aeb0 msgr2=0x7f3fe40a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:40:03.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:03.472+0000 7f3fe2d9d700 1 -- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fe413aeb0 msgr2=0x7f3fe40a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:40:03.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:03.472+0000 7f3fe2d9d700 1 --2- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fe413aeb0 0x7f3fe40a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:40:03.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:03.472+0000 7f3fe2d9d700 1 --2- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fe413aeb0 0x7f3fe40a8c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:40:04.275 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:04.272+0000 7f3fdbfff700 1 -- 172.21.15.23:0/1069458099 >> v2:172.21.15.155:3300/0 conn(0x7f3fe40aae90 msgr2=0x7f3fe41381a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:40:04.275 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:04.272+0000 7f3fdbfff700 1 --2- 172.21.15.23:0/1069458099 >> v2:172.21.15.155:3300/0 conn(0x7f3fe40aae90 0x7f3fe41381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:40:04.276 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:04.273+0000 7f3fe259c700 1 --2- 172.21.15.23:0/1069458099 >> v2:172.21.15.23:3300/0 conn(0x7f3fe40aaab0 0x7f3fe413ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:04.276 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:04.273+0000 7f3fe259c700 1 -- 172.21.15.23:0/1069458099 >> v2:172.21.15.23:3300/0 conn(0x7f3fe40aaab0 msgr2=0x7f3fe413ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:40:04.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:04.273+0000 7f3fe259c700 1 -- 172.21.15.23:0/1069458099 >> v2:172.21.15.23:3300/0 conn(0x7f3fe40aaab0 msgr2=0x7f3fe413ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:40:04.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:04.273+0000 7f3fe259c700 1 --2- 172.21.15.23:0/1069458099 >> v2:172.21.15.23:3300/0 conn(0x7f3fe40aaab0 0x7f3fe413ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:40:04.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:04.273+0000 7f3fe2d9d700 1 --2- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fe413aeb0 0x7f3fe40a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:04.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:04.273+0000 7f3fe259c700 1 --2- 172.21.15.23:0/1069458099 >> v2:172.21.15.23:3300/0 conn(0x7f3fe40aaab0 0x7f3fe413ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:40:04.278 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:04.273+0000 7f3fe2d9d700 1 -- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fe413aeb0 msgr2=0x7f3fe40a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:40:04.278 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:04.273+0000 7f3fe2d9d700 1 -- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fe413aeb0 msgr2=0x7f3fe40a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:40:04.278 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:04.273+0000 7f3fe2d9d700 1 --2- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fe413aeb0 0x7f3fe40a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:40:04.278 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:04.273+0000 7f3fe2d9d700 1 --2- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fe413aeb0 0x7f3fe40a8c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:40:05.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:05.869+0000 7f3fe0d99700 1 -- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fe413aeb0 msgr2=0x7f3fe40a8c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:40:05.949 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:05.869+0000 7f3fe0d99700 1 --2- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fe413aeb0 0x7f3fe40a8c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:05.950 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:05.869+0000 7f3fe0d99700 1 -- 172.21.15.23:0/1069458099 >> v2:172.21.15.155:3300/0 conn(0x7f3fe40aae90 msgr2=0x7f3fe41381a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:40:05.950 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:05.869+0000 7f3fe0d99700 1 --2- 172.21.15.23:0/1069458099 >> v2:172.21.15.155:3300/0 conn(0x7f3fe40aae90 0x7f3fe41381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:40:05.950 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:05.869+0000 7f3fe0d99700 1 -- 172.21.15.23:0/1069458099 >> v2:172.21.15.23:3300/0 conn(0x7f3fe40aaab0 msgr2=0x7f3fe413ed40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:40:05.950 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:05.869+0000 7f3fe0d99700 1 --2- 172.21.15.23:0/1069458099 >> v2:172.21.15.23:3300/0 conn(0x7f3fe40aaab0 0x7f3fe413ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:05.950 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:05.869+0000 7f3fe0d99700 1 -- 172.21.15.23:0/1069458099 --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe413f230 con 0x7f3fc8007050 2020-11-20T11:40:05.951 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:05.869+0000 7f3fe0d99700 1 -- 172.21.15.23:0/1069458099 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fc800bdd0 con 0x7f3fc80033a0 2020-11-20T11:40:05.951 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:05.869+0000 7f3fe259c700 1 -- 172.21.15.23:0/1069458099 >> v1:172.21.15.155:6789/0 conn(0x7f3fc80033a0 legacy=0x7f3fc8005820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:40:06.073 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.070+0000 7f3fe259c700 1 -- 172.21.15.23:0/1069458099 >> v1:172.21.15.155:6789/0 conn(0x7f3fc80033a0 legacy=0x7f3fc8005820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:40:06.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.471+0000 7f3fe259c700 1 -- 172.21.15.23:0/1069458099 >> v1:172.21.15.155:6789/0 conn(0x7f3fc80033a0 legacy=0x7f3fc8005820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:40:06.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.698+0000 7f3fe1d9b700 1 -- 172.21.15.23:0/1069458099 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1580930852 0 0) 0x7f3fe413f230 con 0x7f3fc8007050 2020-11-20T11:40:06.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.698+0000 7f3fe1d9b700 1 -- 172.21.15.23:0/1069458099 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fc4003280 con 0x7f3fc8007050 2020-11-20T11:40:06.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.699+0000 7f3fe1d9b700 1 -- 172.21.15.23:0/1069458099 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3991672160 0 0) 0x7f3fc4003280 con 0x7f3fc8007050 2020-11-20T11:40:06.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.699+0000 7f3fe1d9b700 1 -- 172.21.15.23:0/1069458099 >> v1:172.21.15.155:6789/0 conn(0x7f3fc80033a0 legacy=0x7f3fc8005820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:40:06.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.699+0000 7f3fe1d9b700 1 -- 172.21.15.23:0/1069458099 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc800cfd0 con 0x7f3fc8007050 2020-11-20T11:40:06.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.699+0000 7f3fe1d9b700 1 -- 172.21.15.23:0/1069458099 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f3fcc003e50 con 0x7f3fc8007050 2020-11-20T11:40:06.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.699+0000 7f3fe1d9b700 1 -- 172.21.15.23:0/1069458099 >> v1:172.21.15.23:6789/0 conn(0x7f3fc8007050 legacy=0x7f3fc8009510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:06.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.699+0000 7f3fe2d9d700 1 -- 172.21.15.23:0/1069458099 reap_dead start 2020-11-20T11:40:06.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.699+0000 7f3fe1d9b700 1 --2- 172.21.15.23:0/1069458099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3fc40037a0 0x7f3fc4003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:06.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.699+0000 7f3fe1d9b700 1 --2- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fc4004070 0x7f3fc4004540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:06.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.699+0000 7f3fe1d9b700 1 --2- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fc4004b90 0x7f3fc4005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:06.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.699+0000 7f3fe1d9b700 1 -- 172.21.15.23:0/1069458099 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc400b930 con 0x7f3fc40037a0 2020-11-20T11:40:06.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.699+0000 7f3fe1d9b700 1 -- 172.21.15.23:0/1069458099 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc400baa0 con 0x7f3fc4004070 2020-11-20T11:40:06.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.699+0000 7f3fe1d9b700 1 -- 172.21.15.23:0/1069458099 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc400bc60 con 0x7f3fc4004b90 2020-11-20T11:40:06.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.699+0000 7f3fdbfff700 1 -- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fc4004070 msgr2=0x7f3fc4004540 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:40:06.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.699+0000 7f3fdbfff700 1 --2- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fc4004070 0x7f3fc4004540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:40:06.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.700+0000 7f3fe2d9d700 1 --2- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fc4004b90 0x7f3fc4005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:06.709 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.700+0000 7f3fe2d9d700 1 -- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fc4004070 msgr2=0x7f3fc4004540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:40:06.709 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.700+0000 7f3fe2d9d700 1 --2- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fc4004070 0x7f3fc4004540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:40:06.709 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.700+0000 7f3fe2d9d700 1 -- 172.21.15.23:0/1069458099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3fc40037a0 msgr2=0x7f3fc4003b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:40:06.709 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.700+0000 7f3fe2d9d700 1 --2- 172.21.15.23:0/1069458099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3fc40037a0 0x7f3fc4003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:40:06.710 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.700+0000 7f3fe2d9d700 1 -- 172.21.15.23:0/1069458099 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc800cfd0 con 0x7f3fc4004b90 2020-11-20T11:40:06.710 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.700+0000 7f3fe2d9d700 1 --2- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fc4004b90 0x7f3fc4005040 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f3fe4068590 tx=0x7f3fdc008040).ready entity=mon.2 client_cookie=36d76ca716df98d3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:06.710 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.700+0000 7f3fe1d9b700 1 -- 172.21.15.23:0/1069458099 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fdc014a90 con 0x7f3fc4004b90 2020-11-20T11:40:06.710 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.700+0000 7f3fe1d9b700 1 -- 172.21.15.23:0/1069458099 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fdc00c070 con 0x7f3fc4004b90 2020-11-20T11:40:06.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.700+0000 7f3fe1d9b700 1 -- 172.21.15.23:0/1069458099 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fdc014e00 con 0x7f3fc4004b90 2020-11-20T11:40:06.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.701+0000 7f3fe8d24700 1 -- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fc4004b90 msgr2=0x7f3fc4005040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:06.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.701+0000 7f3fe8d24700 1 --2- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fc4004b90 0x7f3fc4005040 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f3fe4068590 tx=0x7f3fdc008040).stop 2020-11-20T11:40:06.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.701+0000 7f3fe8d24700 1 -- 172.21.15.23:0/1069458099 shutdown_connections 2020-11-20T11:40:06.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.701+0000 7f3fe8d24700 1 --2- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fc4004b90 0x7f3fc4005040 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:06.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.701+0000 7f3fe8d24700 1 --2- 172.21.15.23:0/1069458099 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fc4004070 0x7f3fc4004540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:40:06.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.701+0000 7f3fe8d24700 1 --2- 172.21.15.23:0/1069458099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3fc40037a0 0x7f3fc4003b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:40:06.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.701+0000 7f3fe8d24700 1 -- 172.21.15.23:0/1069458099 >> 172.21.15.23:0/1069458099 conn(0x7f3fe40678e0 msgr2=0x7f3fe4067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:40:06.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.701+0000 7f3fe8d24700 1 -- 172.21.15.23:0/1069458099 shutdown_connections 2020-11-20T11:40:06.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.701+0000 7f3fe8d24700 1 -- 172.21.15.23:0/1069458099 wait complete. 2020-11-20T11:40:06.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.703+0000 7f3fe8d24700 1 Processor -- start 2020-11-20T11:40:06.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.703+0000 7f3fe8d24700 1 -- start start 2020-11-20T11:40:06.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.703+0000 7f3fe8d24700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fc40037a0 0x7f3fe4063c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:06.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.703+0000 7f3fe8d24700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3fc4004070 0x7f3fe40a62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:06.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.703+0000 7f3fe8d24700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fc4004b90 0x7f3fe41515f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:06.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.703+0000 7f3fe8d24700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe4064160 con 0x7f3fc4004070 2020-11-20T11:40:06.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.703+0000 7f3fe8d24700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe413c0f0 con 0x7f3fc4004b90 2020-11-20T11:40:06.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.703+0000 7f3fe8d24700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe4063850 con 0x7f3fc40037a0 2020-11-20T11:40:06.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.703+0000 7f3fdbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3fc4004070 0x7f3fe40a62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:06.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.703+0000 7f3fe259c700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fc40037a0 0x7f3fe4063c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:06.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.704+0000 7f3fe259c700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fc40037a0 0x7f3fe4063c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:54422/0 (socket says 172.21.15.23:54422) 2020-11-20T11:40:06.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.704+0000 7f3fdbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3fc4004070 0x7f3fe40a62d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44184/0 (socket says 172.21.15.23:44184) 2020-11-20T11:40:06.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.704+0000 7f3fe259c700 1 -- 172.21.15.23:0/386000500 learned_addr learned my addr 172.21.15.23:0/386000500 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:40:06.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.704+0000 7f3fe2d9d700 1 -- 172.21.15.23:0/386000500 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fc4004b90 msgr2=0x7f3fe41515f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:40:06.716 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.704+0000 7f3fe2d9d700 1 --2- 172.21.15.23:0/386000500 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fc4004b90 0x7f3fe41515f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:40:06.716 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.704+0000 7f3fdbfff700 1 -- 172.21.15.23:0/386000500 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fc40037a0 msgr2=0x7f3fe4063c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:06.716 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.704+0000 7f3fdbfff700 1 --2- 172.21.15.23:0/386000500 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fc40037a0 0x7f3fe4063c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:06.716 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.704+0000 7f3fdbfff700 1 -- 172.21.15.23:0/386000500 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fc4004b90 msgr2=0x7f3fe41515f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:40:06.716 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.704+0000 7f3fdbfff700 1 --2- 172.21.15.23:0/386000500 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fc4004b90 0x7f3fe41515f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:40:06.717 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.704+0000 7f3fdbfff700 1 -- 172.21.15.23:0/386000500 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe413d440 con 0x7f3fc4004070 2020-11-20T11:40:06.717 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.704+0000 7f3fe259c700 1 --2- 172.21.15.23:0/386000500 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fc40037a0 0x7f3fe4063c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:40:06.717 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.704+0000 7f3fdbfff700 1 --2- 172.21.15.23:0/386000500 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3fc4004070 0x7f3fe40a62d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f3fcc000e60 tx=0x7f3fcc00a3e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:06.717 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.704+0000 7f3fda7fc700 1 -- 172.21.15.23:0/386000500 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fcc007d90 con 0x7f3fc4004070 2020-11-20T11:40:06.718 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.705+0000 7f3fe8d24700 1 -- 172.21.15.23:0/386000500 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe40a81d0 con 0x7f3fc4004070 2020-11-20T11:40:06.718 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.705+0000 7f3fda7fc700 1 -- 172.21.15.23:0/386000500 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fcc011070 con 0x7f3fc4004070 2020-11-20T11:40:06.718 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.705+0000 7f3fda7fc700 1 -- 172.21.15.23:0/386000500 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fcc00d690 con 0x7f3fc4004070 2020-11-20T11:40:06.718 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.705+0000 7f3fe8d24700 1 -- 172.21.15.23:0/386000500 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe40634d0 con 0x7f3fc4004070 2020-11-20T11:40:06.718 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.705+0000 7f3fe8d24700 1 -- 172.21.15.23:0/386000500 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe405c7d0 con 0x7f3fc4004070 2020-11-20T11:40:06.719 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.706+0000 7f3fda7fc700 1 -- 172.21.15.23:0/386000500 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3fcc00da00 con 0x7f3fc4004070 2020-11-20T11:40:06.720 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.706+0000 7f3fda7fc700 1 --2- 172.21.15.23:0/386000500 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3fc0056d40 0x7f3fc0059200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:06.720 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.706+0000 7f3fda7fc700 1 -- 172.21.15.23:0/386000500 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f3fcc0709b0 con 0x7f3fc4004070 2020-11-20T11:40:06.720 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.707+0000 7f3fe259c700 1 --2- 172.21.15.23:0/386000500 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3fc0056d40 0x7f3fc0059200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:06.720 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.707+0000 7f3fe259c700 1 --2- 172.21.15.23:0/386000500 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3fc0056d40 0x7f3fc0059200 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f3fd4006e10 tx=0x7f3fd4008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:06.720 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.712+0000 7f3fda7fc700 1 -- 172.21.15.23:0/386000500 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3fcc045ca0 con 0x7f3fc4004070 2020-11-20T11:40:06.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.933+0000 7f3fe8d24700 1 -- 172.21.15.23:0/386000500 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3fe413d5d0 con 0x7f3fc4004070 2020-11-20T11:40:06.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.934+0000 7f3fda7fc700 1 -- 172.21.15.23:0/386000500 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1471 (secure 0 0 0) 0x7f3fcc0492c0 con 0x7f3fc4004070 2020-11-20T11:40:06.937 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":76,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:40:06.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.935+0000 7f3fe8d24700 1 -- 172.21.15.23:0/386000500 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3fc0056d40 msgr2=0x7f3fc0059200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:06.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.935+0000 7f3fe8d24700 1 --2- 172.21.15.23:0/386000500 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3fc0056d40 0x7f3fc0059200 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f3fd4006e10 tx=0x7f3fd4008040).stop 2020-11-20T11:40:06.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.935+0000 7f3fe8d24700 1 -- 172.21.15.23:0/386000500 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3fc4004070 msgr2=0x7f3fe40a62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:06.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.935+0000 7f3fe8d24700 1 --2- 172.21.15.23:0/386000500 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3fc4004070 0x7f3fe40a62d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f3fcc000e60 tx=0x7f3fcc00a3e0).stop 2020-11-20T11:40:06.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.935+0000 7f3fe8d24700 1 -- 172.21.15.23:0/386000500 shutdown_connections 2020-11-20T11:40:06.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.935+0000 7f3fe8d24700 1 --2- 172.21.15.23:0/386000500 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fc4004b90 0x7f3fe41515f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:40:06.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.935+0000 7f3fe8d24700 1 --2- 172.21.15.23:0/386000500 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3fc0056d40 0x7f3fc0059200 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:06.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.935+0000 7f3fe8d24700 1 --2- 172.21.15.23:0/386000500 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fc40037a0 0x7f3fe4063c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:06.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.935+0000 7f3fe8d24700 1 --2- 172.21.15.23:0/386000500 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3fc4004070 0x7f3fe40a62d0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:06.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.935+0000 7f3fe8d24700 1 -- 172.21.15.23:0/386000500 >> 172.21.15.23:0/386000500 conn(0x7f3fe40678e0 msgr2=0x7f3fe40b1790 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:40:06.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.935+0000 7f3fe8d24700 1 -- 172.21.15.23:0/386000500 shutdown_connections 2020-11-20T11:40:06.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:06.935+0000 7f3fe8d24700 1 -- 172.21.15.23:0/386000500 wait complete. 2020-11-20T11:40:06.951 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2020-11-20T11:40:06.951 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:40:07.081 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.078+0000 7fe811531700 1 Processor -- start 2020-11-20T11:40:07.082 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.078+0000 7fe811531700 1 -- start start 2020-11-20T11:40:07.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.079+0000 7fe811531700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe80c066210 0x7fe80c13adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:07.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.079+0000 7fe811531700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe80c0665f0 0x7fe80c0ab2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:07.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.079+0000 7fe811531700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe80c13bd10 con 0x7fe80c0a8ff0 2020-11-20T11:40:07.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.079+0000 7fe811531700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe80c0aab60 con 0x7fe80c0665f0 2020-11-20T11:40:07.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.079+0000 7fe811531700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c0aae00 con 0x7fe80c066210 2020-11-20T11:40:07.086 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.079+0000 7fe803fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe80c0665f0 0x7fe80c0ab2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:07.086 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.079+0000 7fe803fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe80c0665f0 0x7fe80c0ab2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44190/0 (socket says 172.21.15.23:44190) 2020-11-20T11:40:07.086 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.079+0000 7fe803fff700 1 -- 172.21.15.23:0/1924767989 learned_addr learned my addr 172.21.15.23:0/1924767989 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:40:07.086 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.079+0000 7fe810d30700 1 --1- 172.21.15.23:0/1924767989 >> v1:172.21.15.23:6789/0 conn(0x7fe80c0a8ff0 0x7fe80c137f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58440/0 (socket says 172.21.15.23:58440) 2020-11-20T11:40:07.087 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.080+0000 7fe80bfff700 1 --2- 172.21.15.23:0/1924767989 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe80c066210 0x7fe80c13adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:07.087 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.080+0000 7fe803fff700 1 -- 172.21.15.23:0/1924767989 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe80c066210 msgr2=0x7fe80c13adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:40:07.087 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.080+0000 7fe803fff700 1 --2- 172.21.15.23:0/1924767989 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe80c066210 0x7fe80c13adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:07.087 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.080+0000 7fe803fff700 1 -- 172.21.15.23:0/1924767989 >> v1:172.21.15.23:6789/0 conn(0x7fe80c0a8ff0 legacy=0x7fe80c137f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:07.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.080+0000 7fe80b7fe700 1 -- 172.21.15.23:0/1924767989 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2706873894 0 0) 0x7fe80c13bd10 con 0x7fe80c0a8ff0 2020-11-20T11:40:07.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.080+0000 7fe803fff700 1 -- 172.21.15.23:0/1924767989 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe80c0aa930 con 0x7fe80c0665f0 2020-11-20T11:40:07.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.080+0000 7fe803fff700 1 --2- 172.21.15.23:0/1924767989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe80c0665f0 0x7fe80c0ab2a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7fe7f4009a10 tx=0x7fe7f4006ef0).ready entity=mon.0 client_cookie=e953f31b67af19ec server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:07.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.080+0000 7fe80b7fe700 1 -- 172.21.15.23:0/1924767989 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f400b860 con 0x7fe80c0665f0 2020-11-20T11:40:07.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.080+0000 7fe80b7fe700 1 -- 172.21.15.23:0/1924767989 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f4000e90 con 0x7fe80c0665f0 2020-11-20T11:40:07.089 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.080+0000 7fe80b7fe700 1 -- 172.21.15.23:0/1924767989 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f400bbd0 con 0x7fe80c0665f0 2020-11-20T11:40:07.089 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.081+0000 7fe811531700 1 -- 172.21.15.23:0/1924767989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe80c0665f0 msgr2=0x7fe80c0ab2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:07.089 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.081+0000 7fe811531700 1 --2- 172.21.15.23:0/1924767989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe80c0665f0 0x7fe80c0ab2a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7fe7f4009a10 tx=0x7fe7f4006ef0).stop 2020-11-20T11:40:07.089 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.081+0000 7fe811531700 1 -- 172.21.15.23:0/1924767989 shutdown_connections 2020-11-20T11:40:07.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.081+0000 7fe811531700 1 --2- 172.21.15.23:0/1924767989 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe80c066210 0x7fe80c13adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:07.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.081+0000 7fe811531700 1 --2- 172.21.15.23:0/1924767989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe80c0665f0 0x7fe80c0ab2a0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:07.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.081+0000 7fe811531700 1 -- 172.21.15.23:0/1924767989 >> 172.21.15.23:0/1924767989 conn(0x7fe80c0612c0 msgr2=0x7fe80c0616d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:40:07.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.081+0000 7fe811531700 1 -- 172.21.15.23:0/1924767989 shutdown_connections 2020-11-20T11:40:07.091 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.081+0000 7fe811531700 1 -- 172.21.15.23:0/1924767989 wait complete. 2020-11-20T11:40:07.094 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.082+0000 7fe811531700 1 Processor -- start 2020-11-20T11:40:07.094 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.082+0000 7fe811531700 1 -- start start 2020-11-20T11:40:07.094 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.083+0000 7fe811531700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe80c066210 0x7fe80c0a6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:07.094 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.083+0000 7fe811531700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe80c0665f0 0x7fe80c0a5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:07.095 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.083+0000 7fe811531700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe80c0a8ff0 0x7fe80c14a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:07.095 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.083+0000 7fe811531700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c0a5d10 con 0x7fe80c0a8ff0 2020-11-20T11:40:07.095 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.083+0000 7fe811531700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c14a2d0 con 0x7fe80c066210 2020-11-20T11:40:07.095 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.083+0000 7fe811531700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c136ab0 con 0x7fe80c0665f0 2020-11-20T11:40:07.095 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.083+0000 7fe80bfff700 1 -- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe80c066210 msgr2=0x7fe80c0a6520 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:40:07.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.083+0000 7fe80bfff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe80c066210 0x7fe80c0a6520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:40:07.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.083+0000 7fe810d30700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe80c0a8ff0 0x7fe80c14a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:07.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.083+0000 7fe810d30700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe80c0a8ff0 0x7fe80c14a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44200/0 (socket says 172.21.15.23:44200) 2020-11-20T11:40:07.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.083+0000 7fe810d30700 1 -- 172.21.15.23:0/4214013813 learned_addr learned my addr 172.21.15.23:0/4214013813 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:40:07.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.083+0000 7fe803fff700 1 --2- 172.21.15.23:0/4214013813 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe80c0665f0 0x7fe80c0a5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:07.097 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.083+0000 7fe810d30700 1 -- 172.21.15.23:0/4214013813 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe80c0665f0 msgr2=0x7fe80c0a5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:07.097 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.083+0000 7fe810d30700 1 --2- 172.21.15.23:0/4214013813 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe80c0665f0 0x7fe80c0a5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:07.097 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.083+0000 7fe810d30700 1 -- 172.21.15.23:0/4214013813 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe80c066210 msgr2=0x7fe80c0a6520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:40:07.097 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.083+0000 7fe810d30700 1 --2- 172.21.15.23:0/4214013813 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe80c066210 0x7fe80c0a6520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:40:07.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.083+0000 7fe810d30700 1 -- 172.21.15.23:0/4214013813 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe80c0a5810 con 0x7fe80c0a8ff0 2020-11-20T11:40:07.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.083+0000 7fe810d30700 1 --2- 172.21.15.23:0/4214013813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe80c0a8ff0 0x7fe80c14a770 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fe80400cab0 tx=0x7fe804001360).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:07.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.083+0000 7fe808ff9700 1 -- 172.21.15.23:0/4214013813 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe804005b00 con 0x7fe80c0a8ff0 2020-11-20T11:40:07.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.083+0000 7fe808ff9700 1 -- 172.21.15.23:0/4214013813 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe804005c60 con 0x7fe80c0a8ff0 2020-11-20T11:40:07.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.083+0000 7fe808ff9700 1 -- 172.21.15.23:0/4214013813 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe804007110 con 0x7fe80c0a8ff0 2020-11-20T11:40:07.099 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.083+0000 7fe811531700 1 -- 172.21.15.23:0/4214013813 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe80c0a5a60 con 0x7fe80c0a8ff0 2020-11-20T11:40:07.099 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.083+0000 7fe811531700 1 -- 172.21.15.23:0/4214013813 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe80c14ae20 con 0x7fe80c0a8ff0 2020-11-20T11:40:07.099 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.084+0000 7fe811531700 1 --2- 172.21.15.23:0/4214013813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe80c0622d0 0x7fe80c1340b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:07.099 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.084+0000 7fe811531700 1 -- 172.21.15.23:0/4214013813 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c062760 con 0x7fe80c0622d0 2020-11-20T11:40:07.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.084+0000 7fe80bfff700 1 --2- 172.21.15.23:0/4214013813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe80c0622d0 0x7fe80c1340b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:07.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.085+0000 7fe80bfff700 1 -- 172.21.15.23:0/4214013813 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe80c14a490 con 0x7fe80c0622d0 2020-11-20T11:40:07.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.085+0000 7fe80bfff700 1 --2- 172.21.15.23:0/4214013813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe80c0622d0 0x7fe80c1340b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fe7fc00d9e0 tx=0x7fe7fc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:07.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.085+0000 7fe808ff9700 1 -- 172.21.15.23:0/4214013813 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe804010030 con 0x7fe80c0a8ff0 2020-11-20T11:40:07.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.085+0000 7fe808ff9700 1 --2- 172.21.15.23:0/4214013813 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe7e8056d10 0x7fe7e80591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:07.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.085+0000 7fe808ff9700 1 -- 172.21.15.23:0/4214013813 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc00be10 con 0x7fe80c0622d0 2020-11-20T11:40:07.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.085+0000 7fe808ff9700 1 -- 172.21.15.23:0/4214013813 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fe80406ff70 con 0x7fe80c0a8ff0 2020-11-20T11:40:07.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.085+0000 7fe808ff9700 1 -- 172.21.15.23:0/4214013813 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fe7fc009850 con 0x7fe80c0622d0 2020-11-20T11:40:07.102 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.086+0000 7fe808ff9700 1 -- 172.21.15.23:0/4214013813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe80c0622d0 msgr2=0x7fe80c1340b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:07.102 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.086+0000 7fe808ff9700 1 --2- 172.21.15.23:0/4214013813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe80c0622d0 0x7fe80c1340b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fe7fc00d9e0 tx=0x7fe7fc008040).stop 2020-11-20T11:40:07.102 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.086+0000 7fe808ff9700 1 -- 172.21.15.23:0/4214013813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe80c0622d0 msgr2=0x7fe80c1340b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:40:07.103 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.086+0000 7fe808ff9700 1 --2- 172.21.15.23:0/4214013813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe80c0622d0 0x7fe80c1340b0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:07.103 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.086+0000 7fe803fff700 1 --2- 172.21.15.23:0/4214013813 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe7e8056d10 0x7fe7e80591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:07.103 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.086+0000 7fe803fff700 1 --2- 172.21.15.23:0/4214013813 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe7e8056d10 0x7fe7e80591d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fe7f4009a10 tx=0x7fe7f4016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:07.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.096+0000 7fe811531700 1 --2- 172.21.15.23:0/4214013813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe80c1345a0 0x7fe80c14d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:07.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.096+0000 7fe811531700 1 -- 172.21.15.23:0/4214013813 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c14b070 con 0x7fe80c1345a0 2020-11-20T11:40:07.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.096+0000 7fe80bfff700 1 --2- 172.21.15.23:0/4214013813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe80c1345a0 0x7fe80c14d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:07.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.097+0000 7fe80bfff700 1 -- 172.21.15.23:0/4214013813 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe80c057dc0 con 0x7fe80c1345a0 2020-11-20T11:40:07.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.097+0000 7fe80bfff700 1 --2- 172.21.15.23:0/4214013813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe80c1345a0 0x7fe80c14d720 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fe7fc00d9e0 tx=0x7fe7fc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:07.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.097+0000 7fe808ff9700 1 -- 172.21.15.23:0/4214013813 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc01c5f0 con 0x7fe80c1345a0 2020-11-20T11:40:07.107 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.097+0000 7fe808ff9700 1 -- 172.21.15.23:0/4214013813 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4774 (secure 0 0 0) 0x7fe7fc009640 con 0x7fe80c1345a0 2020-11-20T11:40:07.107 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.097+0000 7fe808ff9700 1 -- 172.21.15.23:0/4214013813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe80c1345a0 msgr2=0x7fe80c14d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:07.107 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.097+0000 7fe808ff9700 1 --2- 172.21.15.23:0/4214013813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe80c1345a0 0x7fe80c14d720 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fe7fc00d9e0 tx=0x7fe7fc008040).stop 2020-11-20T11:40:07.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.097+0000 7fe808ff9700 1 -- 172.21.15.23:0/4214013813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe80c1345a0 msgr2=0x7fe80c14d720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:40:07.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.097+0000 7fe808ff9700 1 --2- 172.21.15.23:0/4214013813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe80c1345a0 0x7fe80c14d720 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:07.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.098+0000 7fe811531700 1 -- 172.21.15.23:0/4214013813 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe7e8056d10 msgr2=0x7fe7e80591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:07.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.098+0000 7fe811531700 1 --2- 172.21.15.23:0/4214013813 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe7e8056d10 0x7fe7e80591d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fe7f4009a10 tx=0x7fe7f4016000).stop 2020-11-20T11:40:07.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.098+0000 7fe811531700 1 -- 172.21.15.23:0/4214013813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe80c0a8ff0 msgr2=0x7fe80c14a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:07.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.098+0000 7fe811531700 1 --2- 172.21.15.23:0/4214013813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe80c0a8ff0 0x7fe80c14a770 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fe80400cab0 tx=0x7fe804001360).stop 2020-11-20T11:40:07.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.098+0000 7fe810d30700 1 -- 172.21.15.23:0/4214013813 reap_dead start 2020-11-20T11:40:07.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.098+0000 7fe811531700 1 -- 172.21.15.23:0/4214013813 shutdown_connections 2020-11-20T11:40:07.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.098+0000 7fe811531700 1 --2- 172.21.15.23:0/4214013813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe80c0a8ff0 0x7fe80c14a770 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:07.110 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.098+0000 7fe811531700 1 -- 172.21.15.23:0/4214013813 >> 172.21.15.23:0/4214013813 conn(0x7fe80c0612c0 msgr2=0x7fe80c0ade50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:40:07.110 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.098+0000 7fe811531700 1 -- 172.21.15.23:0/4214013813 shutdown_connections 2020-11-20T11:40:07.110 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.098+0000 7fe811531700 1 -- 172.21.15.23:0/4214013813 wait complete. 2020-11-20T11:40:07.110 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:40:07.110 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:40:07.111 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:40:07.111 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:40:07.111 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 76, 2020-11-20T11:40:07.111 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:40:07.111 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:40:07.112 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:40:07.112 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:07.112 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 0, 2020-11-20T11:40:07.112 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:40:07.112 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:40:07.113 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:40:07.113 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:40:07.113 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:40:07.113 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:40:07.113 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:40:07.114 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:40:07.114 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:40:07.114 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:40:07.114 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:40:07.115 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:07.115 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:40:07.115 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:40:07.115 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:40:07.115 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:40:07.116 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:40:07.116 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:40:07.116 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:40:07.116 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:40:07.116 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:40:07.116 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:40:07.117 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:07.117 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:07.117 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:40:07.117 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:40:07.117 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:40:07.118 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:40:07.118 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:40:07.118 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:40:07.118 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:40:07.119 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:40:07.119 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:40:07.119 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:40:07.119 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:40:07.119 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:40:07.120 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:40:07.120 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:40:07.120 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:40:07.120 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:40:07.120 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:40:07.120 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:40:07.121 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:40:07.121 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:40:07.121 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:40:07.121 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:40:07.121 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:40:07.122 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:07.122 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:40:07.122 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:07.122 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:40:07.123 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:07.123 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:40:07.124 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:40:07.124 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:40:07.124 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:40:07.124 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:07.124 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:40:07.125 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:40:07.125 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:07.125 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:07.125 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:07.126 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:40:07.126 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:40:07.126 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:07.126 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:07.126 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:07.127 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:07.127 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:40:07.127 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:40:07.127 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:40:07.127 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:40:07.127 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:40:07.128 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:07.128 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:07.128 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:40:07.128 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:40:07.128 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:40:07.129 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:40:07.129 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:07.129 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:40:07.129 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:40:07.129 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:07.130 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:07.130 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:07.130 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:40:07.130 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:40:07.130 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:07.131 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:07.131 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:07.131 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:07.131 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:40:07.131 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:40:07.132 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:40:07.132 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:40:07.132 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:40:07.132 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:07.133 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:07.134 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:40:07.134 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:40:07.134 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:40:07.134 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:40:07.135 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:07.135 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:40:07.135 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:40:07.135 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:07.135 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:07.136 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:07.136 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:40:07.136 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:40:07.136 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:07.136 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:07.137 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:07.137 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:07.137 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:40:07.137 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:40:07.138 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:40:07.138 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:40:07.138 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:40:07.138 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:07.138 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:07.139 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:07.139 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:40:07.139 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:40:07.139 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:07.139 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:40:07.140 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:40:07.140 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:40:07.140 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:07.140 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:07.140 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:40:07.140 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:07.141 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:40:07.141 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:40:07.141 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:40:07.141 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:07.141 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:07.142 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:40:07.142 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:07.142 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:40:07.142 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:40:07.142 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 8 2020-11-20T11:40:07.143 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:07.143 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:07.143 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:40:07.143 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:07.144 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:40:07.144 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:40:07.144 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:40:07.144 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:07.144 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:07.145 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:40:07.145 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:07.145 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:40:07.145 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:40:07.145 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:40:07.146 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:07.146 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:07.146 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:07.146 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:40:07.146 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:40:07.148 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:40:07.278 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.275+0000 7fed01f14700 1 Processor -- start 2020-11-20T11:40:07.278 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.275+0000 7fed01f14700 1 -- start start 2020-11-20T11:40:07.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.275+0000 7fed01f14700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fecfc0ace70 0x7fecfc13a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:07.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.275+0000 7fed01f14700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecfc0ad2f0 0x7fecfc13ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:07.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.275+0000 7fed01f14700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecfc0ad860 con 0x7fecfc0a9040 2020-11-20T11:40:07.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.275+0000 7fed01f14700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecfc0aa8a0 con 0x7fecfc0ace70 2020-11-20T11:40:07.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.275+0000 7fed01f14700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc0aaa90 con 0x7fecfc0ad2f0 2020-11-20T11:40:07.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.275+0000 7fed00f12700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fecfc0a9040 0x7fecfc0aaf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58458/0 (socket says 172.21.15.23:58458) 2020-11-20T11:40:07.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.275+0000 7fed00f12700 1 -- 172.21.15.23:0/1772767483 learned_addr learned my addr 172.21.15.23:0/1772767483 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:40:07.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.275+0000 7fecfbfff700 1 -- 172.21.15.23:0/1772767483 >> v2:172.21.15.155:3300/0 conn(0x7fecfc0ace70 msgr2=0x7fecfc13a830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:40:07.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.275+0000 7fecfbfff700 1 --2- 172.21.15.23:0/1772767483 >> v2:172.21.15.155:3300/0 conn(0x7fecfc0ace70 0x7fecfc13a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:40:07.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.276+0000 7fed01713700 1 --2- 172.21.15.23:0/1772767483 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecfc0ad2f0 0x7fecfc13ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:07.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.276+0000 7fecfb7fe700 1 -- 172.21.15.23:0/1772767483 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3112879209 0 0) 0x7fecfc0ad860 con 0x7fecfc0a9040 2020-11-20T11:40:07.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.276+0000 7fecfb7fe700 1 -- 172.21.15.23:0/1772767483 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fece0003280 con 0x7fecfc0a9040 2020-11-20T11:40:07.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.276+0000 7fecfb7fe700 1 -- 172.21.15.23:0/1772767483 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 861634424 0 0) 0x7fece0003280 con 0x7fecfc0a9040 2020-11-20T11:40:07.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.276+0000 7fecfb7fe700 1 -- 172.21.15.23:0/1772767483 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecfc0ad2f0 msgr2=0x7fecfc13ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:40:07.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.276+0000 7fecfb7fe700 1 --2- 172.21.15.23:0/1772767483 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecfc0ad2f0 0x7fecfc13ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:07.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.276+0000 7fecfb7fe700 1 -- 172.21.15.23:0/1772767483 >> v2:172.21.15.155:3300/0 conn(0x7fecfc0ace70 msgr2=0x7fecfc13a830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:40:07.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.276+0000 7fecfb7fe700 1 --2- 172.21.15.23:0/1772767483 >> v2:172.21.15.155:3300/0 conn(0x7fecfc0ace70 0x7fecfc13a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:40:07.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.276+0000 7fecfb7fe700 1 -- 172.21.15.23:0/1772767483 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecfc0a9420 con 0x7fecfc0a9040 2020-11-20T11:40:07.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.276+0000 7fecfb7fe700 1 -- 172.21.15.23:0/1772767483 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7fecf00016a0 con 0x7fecfc0a9040 2020-11-20T11:40:07.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.276+0000 7fecfb7fe700 1 -- 172.21.15.23:0/1772767483 >> v1:172.21.15.23:6789/0 conn(0x7fecfc0a9040 legacy=0x7fecfc0aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:07.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.276+0000 7fecfb7fe700 1 --2- 172.21.15.23:0/1772767483 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecfc0ad2f0 0x7fece0003770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:07.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.276+0000 7fecfb7fe700 1 --2- 172.21.15.23:0/1772767483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fece0003c80 0x7fece0004520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:07.287 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.276+0000 7fecfb7fe700 1 --2- 172.21.15.23:0/1772767483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fece0004a10 0x7fece0004e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:07.287 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.276+0000 7fecfb7fe700 1 -- 172.21.15.23:0/1772767483 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc0aaa90 con 0x7fece0004a10 2020-11-20T11:40:07.287 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.276+0000 7fecfb7fe700 1 -- 172.21.15.23:0/1772767483 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece00041c0 con 0x7fece0003c80 2020-11-20T11:40:07.287 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.276+0000 7fecfb7fe700 1 -- 172.21.15.23:0/1772767483 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece0004380 con 0x7fecfc0ad2f0 2020-11-20T11:40:07.287 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.276+0000 7fecfbfff700 1 -- 172.21.15.23:0/1772767483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fece0003c80 msgr2=0x7fece0004520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:40:07.288 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.276+0000 7fecfbfff700 1 --2- 172.21.15.23:0/1772767483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fece0003c80 0x7fece0004520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:40:07.288 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.276+0000 7fed01713700 1 --2- 172.21.15.23:0/1772767483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fece0004a10 0x7fece0004e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:07.288 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.276+0000 7fed00f12700 1 --2- 172.21.15.23:0/1772767483 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecfc0ad2f0 0x7fece0003770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:07.288 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.277+0000 7fed01713700 1 -- 172.21.15.23:0/1772767483 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecfc0ad2f0 msgr2=0x7fece0003770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:40:07.288 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.277+0000 7fed01713700 1 --2- 172.21.15.23:0/1772767483 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecfc0ad2f0 0x7fece0003770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:07.289 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.277+0000 7fed01713700 1 -- 172.21.15.23:0/1772767483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fece0003c80 msgr2=0x7fece0004520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:40:07.289 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.277+0000 7fed01713700 1 --2- 172.21.15.23:0/1772767483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fece0003c80 0x7fece0004520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:40:07.289 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.277+0000 7fed01713700 1 -- 172.21.15.23:0/1772767483 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece0003280 con 0x7fece0004a10 2020-11-20T11:40:07.289 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.277+0000 7fed01713700 1 --2- 172.21.15.23:0/1772767483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fece0004a10 0x7fece0004e90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fecf4009ef0 tx=0x7fecf4009220).ready entity=mon.0 client_cookie=53c0d863e47c1c1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:07.289 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.277+0000 7fecfb7fe700 1 -- 172.21.15.23:0/1772767483 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf4009610 con 0x7fece0004a10 2020-11-20T11:40:07.290 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.277+0000 7fecfb7fe700 1 -- 172.21.15.23:0/1772767483 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf400ecd0 con 0x7fece0004a10 2020-11-20T11:40:07.290 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.277+0000 7fecfb7fe700 1 -- 172.21.15.23:0/1772767483 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf4012600 con 0x7fece0004a10 2020-11-20T11:40:07.290 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.277+0000 7fed01f14700 1 -- 172.21.15.23:0/1772767483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fece0004a10 msgr2=0x7fece0004e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:07.290 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.277+0000 7fed01f14700 1 --2- 172.21.15.23:0/1772767483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fece0004a10 0x7fece0004e90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fecf4009ef0 tx=0x7fecf4009220).stop 2020-11-20T11:40:07.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.277+0000 7fed01713700 1 -- 172.21.15.23:0/1772767483 reap_dead start 2020-11-20T11:40:07.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.278+0000 7fed01f14700 1 -- 172.21.15.23:0/1772767483 shutdown_connections 2020-11-20T11:40:07.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.278+0000 7fed01f14700 1 -- 172.21.15.23:0/1772767483 >> 172.21.15.23:0/1772767483 conn(0x7fecfc0683b0 msgr2=0x7fecfc0b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:40:07.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.278+0000 7fed01f14700 1 -- 172.21.15.23:0/1772767483 shutdown_connections 2020-11-20T11:40:07.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.278+0000 7fed01f14700 1 -- 172.21.15.23:0/1772767483 wait complete. 2020-11-20T11:40:07.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.279+0000 7fed01f14700 1 Processor -- start 2020-11-20T11:40:07.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.279+0000 7fed01f14700 1 -- start start 2020-11-20T11:40:07.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.280+0000 7fed01f14700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecfc140660 0x7fecfc140a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:07.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.280+0000 7fed01f14700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecfc13dda0 0x7fecfc13f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:07.298 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.280+0000 7fed01f14700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecfc13f890 0x7fecfc13abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:07.298 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.280+0000 7fed01f14700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc14e5f0 con 0x7fecfc13f890 2020-11-20T11:40:07.299 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.280+0000 7fed01f14700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc14e890 con 0x7fecfc13dda0 2020-11-20T11:40:07.299 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.280+0000 7fed01f14700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc14e340 con 0x7fecfc140660 2020-11-20T11:40:07.299 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.280+0000 7fecfbfff700 1 -- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecfc13dda0 msgr2=0x7fecfc13f3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:40:07.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.280+0000 7fecfbfff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecfc13dda0 0x7fecfc13f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:40:07.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.280+0000 7fed01713700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecfc13f890 0x7fecfc13abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:07.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.280+0000 7fed01713700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecfc13f890 0x7fecfc13abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44224/0 (socket says 172.21.15.23:44224) 2020-11-20T11:40:07.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.280+0000 7fed01713700 1 -- 172.21.15.23:0/3940660966 learned_addr learned my addr 172.21.15.23:0/3940660966 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:40:07.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.280+0000 7fed00f12700 1 --2- 172.21.15.23:0/3940660966 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecfc140660 0x7fecfc140a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:07.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.280+0000 7fed01713700 1 -- 172.21.15.23:0/3940660966 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecfc140660 msgr2=0x7fecfc140a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:07.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.280+0000 7fed01713700 1 --2- 172.21.15.23:0/3940660966 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecfc140660 0x7fecfc140a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:07.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.280+0000 7fed01713700 1 -- 172.21.15.23:0/3940660966 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecfc13dda0 msgr2=0x7fecfc13f3a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:40:07.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.280+0000 7fed01713700 1 --2- 172.21.15.23:0/3940660966 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fecfc13dda0 0x7fecfc13f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:40:07.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.280+0000 7fed01713700 1 -- 172.21.15.23:0/3940660966 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecfc13fcf0 con 0x7fecfc13f890 2020-11-20T11:40:07.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.280+0000 7fed00f12700 1 --2- 172.21.15.23:0/3940660966 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecfc140660 0x7fecfc140a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:40:07.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.280+0000 7fed01713700 1 --2- 172.21.15.23:0/3940660966 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecfc13f890 0x7fecfc13abf0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fece00054a0 tx=0x7fecf401a000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:07.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.280+0000 7fecf8ff9700 1 -- 172.21.15.23:0/3940660966 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf401d6d0 con 0x7fecfc13f890 2020-11-20T11:40:07.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.281+0000 7fecf8ff9700 1 -- 172.21.15.23:0/3940660966 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf401d830 con 0x7fecfc13f890 2020-11-20T11:40:07.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.281+0000 7fed01f14700 1 -- 172.21.15.23:0/3940660966 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecfc140f30 con 0x7fecfc13f890 2020-11-20T11:40:07.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.281+0000 7fecf8ff9700 1 -- 172.21.15.23:0/3940660966 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf401dbb0 con 0x7fecfc13f890 2020-11-20T11:40:07.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.281+0000 7fed01f14700 1 -- 172.21.15.23:0/3940660966 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecfc13b2a0 con 0x7fecfc13f890 2020-11-20T11:40:07.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.281+0000 7fed01f14700 1 --2- 172.21.15.23:0/3940660966 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecfc13b4c0 0x7fecfc1513e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:07.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.281+0000 7fed01f14700 1 -- 172.21.15.23:0/3940660966 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc05cf20 con 0x7fecfc13b4c0 2020-11-20T11:40:07.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.282+0000 7fed00f12700 1 --2- 172.21.15.23:0/3940660966 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecfc13b4c0 0x7fecfc1513e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:07.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.282+0000 7fecf8ff9700 1 -- 172.21.15.23:0/3940660966 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fecf401b450 con 0x7fecfc13f890 2020-11-20T11:40:07.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.282+0000 7fed00f12700 1 -- 172.21.15.23:0/3940660966 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fecfc0a7370 con 0x7fecfc13b4c0 2020-11-20T11:40:07.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.282+0000 7fecf8ff9700 1 --2- 172.21.15.23:0/3940660966 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fecd4056d10 0x7fecd40591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:07.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.282+0000 7fecf8ff9700 1 -- 172.21.15.23:0/3940660966 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fecf4071df0 con 0x7fecfc13f890 2020-11-20T11:40:07.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.282+0000 7fed00f12700 1 --2- 172.21.15.23:0/3940660966 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecfc13b4c0 0x7fecfc1513e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fecf0000ef0 tx=0x7fecf0009040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:07.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.283+0000 7fecfbfff700 1 --2- 172.21.15.23:0/3940660966 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fecd4056d10 0x7fecd40591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:07.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.283+0000 7fecf8ff9700 1 -- 172.21.15.23:0/3940660966 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf0007e20 con 0x7fecfc13b4c0 2020-11-20T11:40:07.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.283+0000 7fecf8ff9700 1 -- 172.21.15.23:0/3940660966 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fecf00119e0 con 0x7fecfc13b4c0 2020-11-20T11:40:07.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.283+0000 7fecf8ff9700 1 -- 172.21.15.23:0/3940660966 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecfc13b4c0 msgr2=0x7fecfc1513e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:07.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.283+0000 7fecf8ff9700 1 --2- 172.21.15.23:0/3940660966 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecfc13b4c0 0x7fecfc1513e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fecf0000ef0 tx=0x7fecf0009040).stop 2020-11-20T11:40:07.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.283+0000 7fecf8ff9700 1 -- 172.21.15.23:0/3940660966 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecfc13b4c0 msgr2=0x7fecfc1513e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:40:07.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.283+0000 7fecf8ff9700 1 --2- 172.21.15.23:0/3940660966 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecfc13b4c0 0x7fecfc1513e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:07.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.283+0000 7fecfbfff700 1 --2- 172.21.15.23:0/3940660966 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fecd4056d10 0x7fecd40591d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fecfc0a7720 tx=0x7fece8006b70).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:07.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.291+0000 7fed01f14700 1 --2- 172.21.15.23:0/3940660966 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecfc14ea50 0x7fecfc1518d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:07.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.291+0000 7fed01f14700 1 -- 172.21.15.23:0/3940660966 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc14e730 con 0x7fecfc14ea50 2020-11-20T11:40:07.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.291+0000 7fed00f12700 1 --2- 172.21.15.23:0/3940660966 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecfc14ea50 0x7fecfc1518d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:07.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.291+0000 7fed00f12700 1 -- 172.21.15.23:0/3940660966 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fecfc057dc0 con 0x7fecfc14ea50 2020-11-20T11:40:07.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.292+0000 7fed00f12700 1 --2- 172.21.15.23:0/3940660966 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecfc14ea50 0x7fecfc1518d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fecf0018000 tx=0x7fecf001b000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:07.308 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.292+0000 7fecf8ff9700 1 -- 172.21.15.23:0/3940660966 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf0027070 con 0x7fecfc14ea50 2020-11-20T11:40:07.308 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.292+0000 7fecf8ff9700 1 -- 172.21.15.23:0/3940660966 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4602 (secure 0 0 0) 0x7fecf00117d0 con 0x7fecfc14ea50 2020-11-20T11:40:07.308 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.292+0000 7fecf8ff9700 1 -- 172.21.15.23:0/3940660966 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecfc14ea50 msgr2=0x7fecfc1518d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:07.308 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.292+0000 7fecf8ff9700 1 --2- 172.21.15.23:0/3940660966 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecfc14ea50 0x7fecfc1518d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fecf0018000 tx=0x7fecf001b000).stop 2020-11-20T11:40:07.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.292+0000 7fecf8ff9700 1 -- 172.21.15.23:0/3940660966 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecfc14ea50 msgr2=0x7fecfc1518d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:40:07.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.292+0000 7fecf8ff9700 1 --2- 172.21.15.23:0/3940660966 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fecfc14ea50 0x7fecfc1518d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:07.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.292+0000 7fed01f14700 1 -- 172.21.15.23:0/3940660966 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fecd4056d10 msgr2=0x7fecd40591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:07.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.292+0000 7fed01f14700 1 --2- 172.21.15.23:0/3940660966 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fecd4056d10 0x7fecd40591d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fecfc0a7720 tx=0x7fece8006b70).stop 2020-11-20T11:40:07.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.292+0000 7fed01f14700 1 -- 172.21.15.23:0/3940660966 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecfc13f890 msgr2=0x7fecfc13abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:07.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.292+0000 7fed01f14700 1 --2- 172.21.15.23:0/3940660966 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fecfc13f890 0x7fecfc13abf0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fece00054a0 tx=0x7fecf401a000).stop 2020-11-20T11:40:07.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.292+0000 7fed01713700 1 -- 172.21.15.23:0/3940660966 reap_dead start 2020-11-20T11:40:07.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.292+0000 7fed01713700 1 -- 172.21.15.23:0/3940660966 reap_dead start 2020-11-20T11:40:07.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.292+0000 7fed01f14700 1 -- 172.21.15.23:0/3940660966 shutdown_connections 2020-11-20T11:40:07.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.292+0000 7fed01f14700 1 -- 172.21.15.23:0/3940660966 >> 172.21.15.23:0/3940660966 conn(0x7fecfc0683b0 msgr2=0x7fecfc0aba40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:40:07.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.293+0000 7fed01f14700 1 -- 172.21.15.23:0/3940660966 shutdown_connections 2020-11-20T11:40:07.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:07.293+0000 7fed01f14700 1 -- 172.21.15.23:0/3940660966 wait complete. 2020-11-20T11:40:07.311 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:40:07.311 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:40:07.312 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:40:07.312 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:40:07.312 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 76, 2020-11-20T11:40:07.312 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:40:07.312 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:40:07.313 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:40:07.313 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:07.313 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 0, 2020-11-20T11:40:07.313 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:40:07.314 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:40:07.314 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:40:07.314 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:40:07.314 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:40:07.314 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:40:07.315 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:40:07.315 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:40:07.315 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:40:07.315 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:40:07.315 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:40:07.316 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:07.316 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:40:07.316 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:40:07.316 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:40:07.316 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:40:07.317 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:40:07.317 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:40:07.317 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:40:07.317 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:40:07.317 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:40:07.318 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:40:07.318 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:07.318 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:07.318 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:40:07.318 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:40:07.319 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:40:07.319 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:40:07.319 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:40:07.319 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:40:07.320 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:40:07.320 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:40:07.320 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:40:07.320 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:40:07.320 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:40:07.321 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:40:07.321 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:40:07.321 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:40:07.321 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:40:07.321 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:40:07.322 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:40:07.322 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:40:07.322 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:40:07.322 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:40:07.322 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:40:07.323 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:40:07.323 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:40:07.323 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:07.323 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:40:07.323 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:07.324 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:40:07.324 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:07.324 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:40:07.324 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:40:07.324 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:40:07.325 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:40:07.325 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:07.325 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:40:07.325 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:40:07.325 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:07.326 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:07.326 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:07.326 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:40:07.326 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:40:07.326 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:07.327 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:07.327 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:07.327 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:07.327 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:40:07.327 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:40:07.328 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:40:07.328 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:40:07.328 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:40:07.328 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:07.329 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:07.329 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:40:07.329 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:40:07.329 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:40:07.329 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:40:07.330 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:07.330 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:40:07.330 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:40:07.330 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:07.331 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:07.331 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:07.331 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:40:07.331 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:40:07.331 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:07.332 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:07.332 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:07.332 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:07.332 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:40:07.332 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:40:07.333 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:40:07.333 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:40:07.333 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:40:07.333 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:07.334 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:07.334 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:40:07.334 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:40:07.334 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:40:07.334 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:40:07.335 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:07.335 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:40:07.335 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:40:07.335 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:07.335 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:07.336 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:07.336 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:40:07.336 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:40:07.336 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:07.337 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:07.337 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:07.337 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:07.337 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:40:07.337 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:40:07.338 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:40:07.338 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:40:07.338 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:40:07.338 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:07.338 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:07.339 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:07.339 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:40:07.339 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:40:07.339 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:07.339 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:40:07.340 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:40:07.340 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:40:07.340 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:07.340 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:07.340 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:40:07.341 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:07.341 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:40:07.341 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:40:07.341 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:40:07.341 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:07.342 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:07.342 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:40:07.342 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:07.342 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:40:07.342 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:40:07.343 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:40:07.343 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:07.343 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:07.343 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:40:07.343 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:40:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:40:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:40:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:07.345 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:07.345 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:40:07.345 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:40:07.346 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2020-11-20T11:40:27.348 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2020-11-20T11:40:27.348 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2020-11-20T11:40:27.349 INFO:tasks.ceph.mon.b:Restarting daemon 2020-11-20T11:40:27.350 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2020-11-20T11:40:27.354 INFO:tasks.ceph.mon.b:Started 2020-11-20T11:40:27.355 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:40:27.355 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:40:27.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.472+0000 7f214a55d700 1 Processor -- start 2020-11-20T11:40:27.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.472+0000 7f214a55d700 1 -- start start 2020-11-20T11:40:27.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.473+0000 7f214a55d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f21440aaab0 0x7f214413ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:27.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.473+0000 7f214a55d700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f21440aae90 0x7f21441381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:27.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.473+0000 7f214a55d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f214413aeb0 0x7f21440a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:27.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.473+0000 7f214a55d700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f214413e600 con 0x7f21440aaab0 2020-11-20T11:40:27.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.473+0000 7f214a55d700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f214413e8a0 con 0x7f21440aae90 2020-11-20T11:40:27.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.473+0000 7f214a55d700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21440ab6d0 con 0x7f214413aeb0 2020-11-20T11:40:27.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.473+0000 7f2143fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f21440aaab0 0x7f214413ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:27.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.473+0000 7f2143fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f21440aaab0 0x7f214413ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44234/0 (socket says 172.21.15.23:44234) 2020-11-20T11:40:27.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.473+0000 7f2143fff700 1 -- 172.21.15.23:0/697265844 learned_addr learned my addr 172.21.15.23:0/697265844 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:40:27.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.473+0000 7f21437fe700 1 --2- 172.21.15.23:0/697265844 >> v2:172.21.15.155:3300/0 conn(0x7f21440aae90 0x7f21441381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:27.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.473+0000 7f2148afa700 1 --2- 172.21.15.23:0/697265844 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f214413aeb0 0x7f21440a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:27.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.473+0000 7f21437fe700 1 -- 172.21.15.23:0/697265844 >> v2:172.21.15.155:3300/0 conn(0x7f21440aae90 msgr2=0x7f21441381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:40:27.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.473+0000 7f21437fe700 1 -- 172.21.15.23:0/697265844 >> v2:172.21.15.155:3300/0 conn(0x7f21440aae90 msgr2=0x7f21441381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:40:27.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.473+0000 7f21437fe700 1 --2- 172.21.15.23:0/697265844 >> v2:172.21.15.155:3300/0 conn(0x7f21440aae90 0x7f21441381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:40:27.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.473+0000 7f2143fff700 1 -- 172.21.15.23:0/697265844 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f214413aeb0 msgr2=0x7f21440a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:40:27.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.473+0000 7f2143fff700 1 --2- 172.21.15.23:0/697265844 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f214413aeb0 0x7f21440a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:27.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.473+0000 7f2143fff700 1 -- 172.21.15.23:0/697265844 >> v2:172.21.15.155:3300/0 conn(0x7f21440aae90 msgr2=0x7f21441381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:40:27.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.473+0000 7f21437fe700 1 --2- 172.21.15.23:0/697265844 >> v2:172.21.15.155:3300/0 conn(0x7f21440aae90 0x7f21441381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:40:27.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.473+0000 7f2143fff700 1 --2- 172.21.15.23:0/697265844 >> v2:172.21.15.155:3300/0 conn(0x7f21440aae90 0x7f21441381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:27.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.473+0000 7f2143fff700 1 -- 172.21.15.23:0/697265844 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f214413f230 con 0x7f21440aaab0 2020-11-20T11:40:27.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.473+0000 7f2143fff700 1 --2- 172.21.15.23:0/697265844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21440aaab0 0x7f214413ed40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f21440ab930 tx=0x7f2134006ef0).ready entity=mon.0 client_cookie=f575b64ab052e099 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:27.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.474+0000 7f2142ffd700 1 -- 172.21.15.23:0/697265844 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213400b860 con 0x7f21440aaab0 2020-11-20T11:40:27.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.474+0000 7f2142ffd700 1 -- 172.21.15.23:0/697265844 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2134000e90 con 0x7f21440aaab0 2020-11-20T11:40:27.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.474+0000 7f2142ffd700 1 -- 172.21.15.23:0/697265844 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213400bbd0 con 0x7f21440aaab0 2020-11-20T11:40:27.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.474+0000 7f214a55d700 1 -- 172.21.15.23:0/697265844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21440aaab0 msgr2=0x7f214413ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:27.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.474+0000 7f214a55d700 1 --2- 172.21.15.23:0/697265844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21440aaab0 0x7f214413ed40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f21440ab930 tx=0x7f2134006ef0).stop 2020-11-20T11:40:27.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.474+0000 7f214a55d700 1 -- 172.21.15.23:0/697265844 shutdown_connections 2020-11-20T11:40:27.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.474+0000 7f214a55d700 1 --2- 172.21.15.23:0/697265844 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f214413aeb0 0x7f21440a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:27.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.474+0000 7f214a55d700 1 --2- 172.21.15.23:0/697265844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21440aaab0 0x7f214413ed40 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:27.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.474+0000 7f214a55d700 1 --2- 172.21.15.23:0/697265844 >> v2:172.21.15.155:3300/0 conn(0x7f21440aae90 0x7f21441381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:27.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.474+0000 7f214a55d700 1 -- 172.21.15.23:0/697265844 >> 172.21.15.23:0/697265844 conn(0x7f21440678e0 msgr2=0x7f2144067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:40:27.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.474+0000 7f214a55d700 1 -- 172.21.15.23:0/697265844 shutdown_connections 2020-11-20T11:40:27.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.474+0000 7f214a55d700 1 -- 172.21.15.23:0/697265844 wait complete. 2020-11-20T11:40:27.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.475+0000 7f214a55d700 1 Processor -- start 2020-11-20T11:40:27.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.475+0000 7f214a55d700 1 -- start start 2020-11-20T11:40:27.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.476+0000 7f214a55d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f21440aae90 0x7f21440ad140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:27.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.476+0000 7f214a55d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21440a7a30 0x7f214413d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:27.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.476+0000 7f214a55d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21440a5170 0x7f21441344c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:27.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.476+0000 7f214a55d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21440a5d60 con 0x7f21440a5170 2020-11-20T11:40:27.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.476+0000 7f214a55d700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21440a7590 con 0x7f21440a7a30 2020-11-20T11:40:27.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.476+0000 7f214a55d700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21440a6000 con 0x7f21440aae90 2020-11-20T11:40:27.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.476+0000 7f2148afa700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21440a5170 0x7f21441344c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:27.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.476+0000 7f2148afa700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21440a5170 0x7f21441344c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44244/0 (socket says 172.21.15.23:44244) 2020-11-20T11:40:27.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.476+0000 7f2148afa700 1 -- 172.21.15.23:0/1687725366 learned_addr learned my addr 172.21.15.23:0/1687725366 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:40:27.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.476+0000 7f21437fe700 1 --2- 172.21.15.23:0/1687725366 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21440a7a30 0x7f214413d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:27.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.476+0000 7f2143fff700 1 --2- 172.21.15.23:0/1687725366 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f21440aae90 0x7f21440ad140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:27.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.476+0000 7f21437fe700 1 -- 172.21.15.23:0/1687725366 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21440a7a30 msgr2=0x7f214413d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2020-11-20T11:40:27.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.476+0000 7f21437fe700 1 -- 172.21.15.23:0/1687725366 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21440a7a30 msgr2=0x7f214413d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:40:27.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.476+0000 7f21437fe700 1 --2- 172.21.15.23:0/1687725366 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21440a7a30 0x7f214413d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:40:27.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.476+0000 7f21437fe700 1 --2- 172.21.15.23:0/1687725366 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21440a7a30 0x7f214413d700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:40:27.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.477+0000 7f2143fff700 1 -- 172.21.15.23:0/1687725366 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21440a7a30 msgr2=0x7f214413d700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:40:27.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.477+0000 7f2143fff700 1 --2- 172.21.15.23:0/1687725366 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21440a7a30 0x7f214413d700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:27.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.477+0000 7f2143fff700 1 -- 172.21.15.23:0/1687725366 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21440a5170 msgr2=0x7f21441344c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:27.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.477+0000 7f2143fff700 1 --2- 172.21.15.23:0/1687725366 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21440a5170 0x7f21441344c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:27.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.477+0000 7f2143fff700 1 -- 172.21.15.23:0/1687725366 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21440a59e0 con 0x7f21440aae90 2020-11-20T11:40:27.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.477+0000 7f2148afa700 1 --2- 172.21.15.23:0/1687725366 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21440a5170 0x7f21441344c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:40:27.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.477+0000 7f2143fff700 1 --2- 172.21.15.23:0/1687725366 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f21440aae90 0x7f21440ad140 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f2134013000 tx=0x7f2134016000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:27.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.477+0000 7f2123fff700 1 -- 172.21.15.23:0/1687725366 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2134011ad0 con 0x7f21440aae90 2020-11-20T11:40:27.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.477+0000 7f2123fff700 1 -- 172.21.15.23:0/1687725366 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2134011c30 con 0x7f21440aae90 2020-11-20T11:40:27.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.477+0000 7f2123fff700 1 -- 172.21.15.23:0/1687725366 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213401d480 con 0x7f21440aae90 2020-11-20T11:40:27.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.477+0000 7f214a55d700 1 -- 172.21.15.23:0/1687725366 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21440a55b0 con 0x7f21440aae90 2020-11-20T11:40:27.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.477+0000 7f214a55d700 1 -- 172.21.15.23:0/1687725366 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2144134bf0 con 0x7f21440aae90 2020-11-20T11:40:27.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.478+0000 7f214a55d700 1 -- 172.21.15.23:0/1687725366 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21440a5bd0 con 0x7f21440aae90 2020-11-20T11:40:27.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.479+0000 7f2123fff700 1 -- 172.21.15.23:0/1687725366 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2134017070 con 0x7f21440aae90 2020-11-20T11:40:27.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.479+0000 7f2123fff700 1 --2- 172.21.15.23:0/1687725366 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f211c056d60 0x7f211c059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:27.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.479+0000 7f2123fff700 1 -- 172.21.15.23:0/1687725366 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f2134071d40 con 0x7f21440aae90 2020-11-20T11:40:27.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.479+0000 7f21437fe700 1 --2- 172.21.15.23:0/1687725366 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f211c056d60 0x7f211c059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:27.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.480+0000 7f21437fe700 1 --2- 172.21.15.23:0/1687725366 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f211c056d60 0x7f211c059220 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f2138006fd0 tx=0x7f2138008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:27.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.483+0000 7f2123fff700 1 -- 172.21.15.23:0/1687725366 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2134047010 con 0x7f21440aae90 2020-11-20T11:40:27.718 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:27.715+0000 7f214a55d700 1 -- 172.21.15.23:0/1687725366 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f214405c7d0 con 0x7f21440aae90 2020-11-20T11:40:31.021 INFO:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:40:31.026 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:40:33.093 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.090+0000 7f2123fff700 1 -- 172.21.15.23:0/1687725366 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1477 (secure 0 0 0) 0x7f2134015070 con 0x7f21440aae90 2020-11-20T11:40:33.094 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:40:33.095 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.092+0000 7f214a55d700 1 -- 172.21.15.23:0/1687725366 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f211c056d60 msgr2=0x7f211c059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:33.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.092+0000 7f214a55d700 1 --2- 172.21.15.23:0/1687725366 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f211c056d60 0x7f211c059220 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f2138006fd0 tx=0x7f2138008040).stop 2020-11-20T11:40:33.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.092+0000 7f214a55d700 1 -- 172.21.15.23:0/1687725366 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f21440aae90 msgr2=0x7f21440ad140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:33.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.092+0000 7f214a55d700 1 --2- 172.21.15.23:0/1687725366 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f21440aae90 0x7f21440ad140 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f2134013000 tx=0x7f2134016000).stop 2020-11-20T11:40:33.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.092+0000 7f214a55d700 1 -- 172.21.15.23:0/1687725366 shutdown_connections 2020-11-20T11:40:33.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.092+0000 7f214a55d700 1 --2- 172.21.15.23:0/1687725366 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f21440a5170 0x7f21441344c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:33.097 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.092+0000 7f214a55d700 1 --2- 172.21.15.23:0/1687725366 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f211c056d60 0x7f211c059220 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:33.097 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.092+0000 7f214a55d700 1 --2- 172.21.15.23:0/1687725366 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f21440aae90 0x7f21440ad140 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:33.097 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.092+0000 7f214a55d700 1 --2- 172.21.15.23:0/1687725366 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f21440a7a30 0x7f214413d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:33.097 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.092+0000 7f214a55d700 1 -- 172.21.15.23:0/1687725366 >> 172.21.15.23:0/1687725366 conn(0x7f21440678e0 msgr2=0x7f2144139d40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:40:33.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.092+0000 7f214a55d700 1 -- 172.21.15.23:0/1687725366 shutdown_connections 2020-11-20T11:40:33.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.092+0000 7f214a55d700 1 -- 172.21.15.23:0/1687725366 wait complete. 2020-11-20T11:40:33.108 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:40:33.108 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:40:33.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.238+0000 7f1bd20f4700 1 Processor -- start 2020-11-20T11:40:33.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.238+0000 7f1bd20f4700 1 -- start start 2020-11-20T11:40:33.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.239+0000 7f1bd20f4700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1bcc05bc80 0x7f1bcc13bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:33.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.239+0000 7f1bd20f4700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bcc0a9080 0x7f1bcc0a7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:33.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.239+0000 7f1bd20f4700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bcc0a8c20 con 0x7f1bcc067fd0 2020-11-20T11:40:33.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.239+0000 7f1bd20f4700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bcc0a94f0 con 0x7f1bcc05bc80 2020-11-20T11:40:33.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.239+0000 7f1bd20f4700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc0a9e20 con 0x7f1bcc0a9080 2020-11-20T11:40:33.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.239+0000 7f1bcb7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1bcc05bc80 0x7f1bcc13bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:33.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.239+0000 7f1bcb7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1bcc05bc80 0x7f1bcc13bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44250/0 (socket says 172.21.15.23:44250) 2020-11-20T11:40:33.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.239+0000 7f1bcb7fe700 1 -- 172.21.15.23:0/56064814 learned_addr learned my addr 172.21.15.23:0/56064814 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:40:33.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.239+0000 7f1bcbfff700 1 --2- 172.21.15.23:0/56064814 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bcc0a9080 0x7f1bcc0a7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:33.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.240+0000 7f1bcb7fe700 1 -- 172.21.15.23:0/56064814 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bcc0a9080 msgr2=0x7f1bcc0a7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:40:33.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.240+0000 7f1bcb7fe700 1 --2- 172.21.15.23:0/56064814 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bcc0a9080 0x7f1bcc0a7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:33.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.240+0000 7f1bcb7fe700 1 -- 172.21.15.23:0/56064814 >> v1:172.21.15.155:6789/0 conn(0x7f1bcc067fd0 legacy=0x7f1bcc0a6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:33.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.240+0000 7f1bcb7fe700 1 -- 172.21.15.23:0/56064814 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bcc0684e0 con 0x7f1bcc05bc80 2020-11-20T11:40:33.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.240+0000 7f1bcbfff700 1 --2- 172.21.15.23:0/56064814 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bcc0a9080 0x7f1bcc0a7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:40:33.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.240+0000 7f1bcb7fe700 1 --2- 172.21.15.23:0/56064814 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bcc05bc80 0x7f1bcc13bfa0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f1bcc0a8db0 tx=0x7f1bbc006ef0).ready entity=mon.0 client_cookie=1a1b23bc6f22266b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:33.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.241+0000 7f1bca7fc700 1 -- 172.21.15.23:0/56064814 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc00b860 con 0x7f1bcc05bc80 2020-11-20T11:40:33.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.241+0000 7f1bca7fc700 1 -- 172.21.15.23:0/56064814 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bbc000e90 con 0x7f1bcc05bc80 2020-11-20T11:40:33.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.241+0000 7f1bca7fc700 1 -- 172.21.15.23:0/56064814 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc00bbd0 con 0x7f1bcc05bc80 2020-11-20T11:40:33.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.241+0000 7f1bd20f4700 1 -- 172.21.15.23:0/56064814 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bcc05bc80 msgr2=0x7f1bcc13bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:33.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.241+0000 7f1bd20f4700 1 --2- 172.21.15.23:0/56064814 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bcc05bc80 0x7f1bcc13bfa0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f1bcc0a8db0 tx=0x7f1bbc006ef0).stop 2020-11-20T11:40:33.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.241+0000 7f1bd20f4700 1 -- 172.21.15.23:0/56064814 shutdown_connections 2020-11-20T11:40:33.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.241+0000 7f1bd20f4700 1 --2- 172.21.15.23:0/56064814 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bcc05bc80 0x7f1bcc13bfa0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:33.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.241+0000 7f1bd20f4700 1 --2- 172.21.15.23:0/56064814 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bcc0a9080 0x7f1bcc0a7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:33.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.241+0000 7f1bd20f4700 1 -- 172.21.15.23:0/56064814 >> 172.21.15.23:0/56064814 conn(0x7f1bcc0ad7a0 msgr2=0x7f1bcc0adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:40:33.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.241+0000 7f1bd20f4700 1 -- 172.21.15.23:0/56064814 shutdown_connections 2020-11-20T11:40:33.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.241+0000 7f1bd20f4700 1 -- 172.21.15.23:0/56064814 wait complete. 2020-11-20T11:40:33.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.243+0000 7f1bd20f4700 1 Processor -- start 2020-11-20T11:40:33.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.243+0000 7f1bd20f4700 1 -- start start 2020-11-20T11:40:33.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.244+0000 7f1bd20f4700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1bcc05bc80 0x7f1bcc0a4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:33.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.244+0000 7f1bd20f4700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bcc067fd0 0x7f1bcc13b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:33.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.244+0000 7f1bd20f4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bcc0a9080 0x7f1bcc0664e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:33.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.244+0000 7f1bd20f4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc0a4960 con 0x7f1bcc0a9080 2020-11-20T11:40:33.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.244+0000 7f1bd20f4700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc065b00 con 0x7f1bcc05bc80 2020-11-20T11:40:33.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.244+0000 7f1bd20f4700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc065da0 con 0x7f1bcc067fd0 2020-11-20T11:40:33.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.244+0000 7f1bcbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bcc0a9080 0x7f1bcc0664e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:33.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.244+0000 7f1bcbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bcc0a9080 0x7f1bcc0664e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44260/0 (socket says 172.21.15.23:44260) 2020-11-20T11:40:33.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.244+0000 7f1bcaffd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bcc067fd0 0x7f1bcc13b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:33.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.244+0000 7f1bcbfff700 1 -- 172.21.15.23:0/98412124 learned_addr learned my addr 172.21.15.23:0/98412124 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:40:33.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.244+0000 7f1bcb7fe700 1 --2- 172.21.15.23:0/98412124 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1bcc05bc80 0x7f1bcc0a4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:33.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.245+0000 7f1bcbfff700 1 -- 172.21.15.23:0/98412124 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bcc067fd0 msgr2=0x7f1bcc13b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:33.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.245+0000 7f1bcbfff700 1 --2- 172.21.15.23:0/98412124 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bcc067fd0 0x7f1bcc13b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:33.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.245+0000 7f1bcbfff700 1 -- 172.21.15.23:0/98412124 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1bcc05bc80 msgr2=0x7f1bcc0a4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:33.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.245+0000 7f1bcbfff700 1 --2- 172.21.15.23:0/98412124 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1bcc05bc80 0x7f1bcc0a4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:33.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.245+0000 7f1bcbfff700 1 -- 172.21.15.23:0/98412124 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bcc066040 con 0x7f1bcc0a9080 2020-11-20T11:40:33.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.245+0000 7f1bcbfff700 1 --2- 172.21.15.23:0/98412124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bcc0a9080 0x7f1bcc0664e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f1bc400ebf0 tx=0x7f1bc4008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:33.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.245+0000 7f1bab7fe700 1 -- 172.21.15.23:0/98412124 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc400c650 con 0x7f1bcc0a9080 2020-11-20T11:40:33.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.245+0000 7f1bab7fe700 1 -- 172.21.15.23:0/98412124 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bc400c7b0 con 0x7f1bcc0a9080 2020-11-20T11:40:33.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.245+0000 7f1bd20f4700 1 -- 172.21.15.23:0/98412124 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bcc139f90 con 0x7f1bcc0a9080 2020-11-20T11:40:33.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.245+0000 7f1bab7fe700 1 -- 172.21.15.23:0/98412124 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc400cb30 con 0x7f1bcc0a9080 2020-11-20T11:40:33.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.245+0000 7f1bd20f4700 1 -- 172.21.15.23:0/98412124 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bcc13be50 con 0x7f1bcc0a9080 2020-11-20T11:40:33.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.246+0000 7f1bd20f4700 1 --2- 172.21.15.23:0/98412124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bcc0669d0 0x7f1bcc135110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:33.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.246+0000 7f1bd20f4700 1 -- 172.21.15.23:0/98412124 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc066db0 con 0x7f1bcc0669d0 2020-11-20T11:40:33.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.246+0000 7f1bcb7fe700 1 --2- 172.21.15.23:0/98412124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bcc0669d0 0x7f1bcc135110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:33.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.246+0000 7f1bab7fe700 1 -- 172.21.15.23:0/98412124 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1bc401d460 con 0x7f1bcc0a9080 2020-11-20T11:40:33.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.246+0000 7f1bcb7fe700 1 -- 172.21.15.23:0/98412124 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1bcc0661d0 con 0x7f1bcc0669d0 2020-11-20T11:40:33.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.247+0000 7f1bcb7fe700 1 --2- 172.21.15.23:0/98412124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bcc0669d0 0x7f1bcc135110 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f1bcc0a8db0 tx=0x7f1bbc016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:33.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.247+0000 7f1bab7fe700 1 --2- 172.21.15.23:0/98412124 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1ba4056d60 0x7f1ba4059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:33.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.247+0000 7f1bab7fe700 1 -- 172.21.15.23:0/98412124 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f1bc4070000 con 0x7f1bcc0a9080 2020-11-20T11:40:33.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.247+0000 7f1bcaffd700 1 --2- 172.21.15.23:0/98412124 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1ba4056d60 0x7f1ba4059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:33.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.247+0000 7f1bab7fe700 1 -- 172.21.15.23:0/98412124 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc011dc0 con 0x7f1bcc0669d0 2020-11-20T11:40:33.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.247+0000 7f1bab7fe700 1 -- 172.21.15.23:0/98412124 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f1bbc01d440 con 0x7f1bcc0669d0 2020-11-20T11:40:33.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.247+0000 7f1bab7fe700 1 -- 172.21.15.23:0/98412124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bcc0669d0 msgr2=0x7f1bcc135110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:33.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.247+0000 7f1bab7fe700 1 --2- 172.21.15.23:0/98412124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bcc0669d0 0x7f1bcc135110 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f1bcc0a8db0 tx=0x7f1bbc016000).stop 2020-11-20T11:40:33.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.247+0000 7f1bab7fe700 1 -- 172.21.15.23:0/98412124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bcc0669d0 msgr2=0x7f1bcc135110 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:40:33.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.247+0000 7f1bab7fe700 1 --2- 172.21.15.23:0/98412124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bcc0669d0 0x7f1bcc135110 secure :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0x7f1bcc0a8db0 tx=0x7f1bbc016000).stop 2020-11-20T11:40:33.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.247+0000 7f1bcaffd700 1 --2- 172.21.15.23:0/98412124 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1ba4056d60 0x7f1ba4059220 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f1bc0007fd0 tx=0x7f1bc0007bd0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:33.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.257+0000 7f1bd20f4700 1 --2- 172.21.15.23:0/98412124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bcc135600 0x7f1bcc13a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:33.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.257+0000 7f1bd20f4700 1 -- 172.21.15.23:0/98412124 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc062fe0 con 0x7f1bcc135600 2020-11-20T11:40:33.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.257+0000 7f1bcb7fe700 1 --2- 172.21.15.23:0/98412124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bcc135600 0x7f1bcc13a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:33.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.257+0000 7f1bcb7fe700 1 -- 172.21.15.23:0/98412124 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1bcc044710 con 0x7f1bcc135600 2020-11-20T11:40:33.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.257+0000 7f1bcb7fe700 1 --2- 172.21.15.23:0/98412124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bcc135600 0x7f1bcc13a260 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f1bcc066ef0 tx=0x7f1bbc01f040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:33.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.257+0000 7f1bab7fe700 1 -- 172.21.15.23:0/98412124 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc027b30 con 0x7f1bcc135600 2020-11-20T11:40:33.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.258+0000 7f1bab7fe700 1 -- 172.21.15.23:0/98412124 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4785 (secure 0 0 0) 0x7f1bbc011bb0 con 0x7f1bcc135600 2020-11-20T11:40:33.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.258+0000 7f1bab7fe700 1 -- 172.21.15.23:0/98412124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bcc135600 msgr2=0x7f1bcc13a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:33.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.258+0000 7f1bab7fe700 1 --2- 172.21.15.23:0/98412124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bcc135600 0x7f1bcc13a260 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f1bcc066ef0 tx=0x7f1bbc01f040).stop 2020-11-20T11:40:33.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.258+0000 7f1bab7fe700 1 -- 172.21.15.23:0/98412124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bcc135600 msgr2=0x7f1bcc13a260 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:40:33.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.258+0000 7f1bab7fe700 1 --2- 172.21.15.23:0/98412124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bcc135600 0x7f1bcc13a260 secure :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0x7f1bcc066ef0 tx=0x7f1bbc01f040).stop 2020-11-20T11:40:33.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.258+0000 7f1bd20f4700 1 -- 172.21.15.23:0/98412124 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1ba4056d60 msgr2=0x7f1ba4059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:33.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.258+0000 7f1bd20f4700 1 --2- 172.21.15.23:0/98412124 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1ba4056d60 0x7f1ba4059220 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f1bc0007fd0 tx=0x7f1bc0007bd0).stop 2020-11-20T11:40:33.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.258+0000 7f1bd20f4700 1 -- 172.21.15.23:0/98412124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bcc0a9080 msgr2=0x7f1bcc0664e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:33.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.258+0000 7f1bd20f4700 1 --2- 172.21.15.23:0/98412124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bcc0a9080 0x7f1bcc0664e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f1bc400ebf0 tx=0x7f1bc4008040).stop 2020-11-20T11:40:33.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.258+0000 7f1bcbfff700 1 -- 172.21.15.23:0/98412124 reap_dead start 2020-11-20T11:40:33.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.258+0000 7f1bcbfff700 1 -- 172.21.15.23:0/98412124 reap_dead start 2020-11-20T11:40:33.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.258+0000 7f1bd20f4700 1 -- 172.21.15.23:0/98412124 shutdown_connections 2020-11-20T11:40:33.266 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.258+0000 7f1bd20f4700 1 -- 172.21.15.23:0/98412124 >> 172.21.15.23:0/98412124 conn(0x7f1bcc0ad7a0 msgr2=0x7f1bcc0aa310 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:40:33.266 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.258+0000 7f1bd20f4700 1 -- 172.21.15.23:0/98412124 shutdown_connections 2020-11-20T11:40:33.266 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.258+0000 7f1bd20f4700 1 -- 172.21.15.23:0/98412124 wait complete. 2020-11-20T11:40:33.266 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:40:33.266 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:40:33.267 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:40:33.267 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:40:33.267 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 82, 2020-11-20T11:40:33.267 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:40:33.267 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:40:33.267 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:40:33.268 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:40:33.268 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:33.268 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 0, 2020-11-20T11:40:33.268 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:40:33.268 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:40:33.269 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:40:33.269 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:40:33.269 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:40:33.269 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:40:33.269 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:40:33.270 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:40:33.270 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:40:33.270 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:40:33.270 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:40:33.270 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:33.270 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:40:33.271 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:40:33.271 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:40:33.271 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:40:33.271 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:40:33.271 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:40:33.272 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:40:33.272 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:40:33.272 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:40:33.272 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:40:33.272 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:33.272 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.273 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:40:33.273 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:40:33.273 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:40:33.273 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:40:33.273 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:40:33.274 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:40:33.274 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:40:33.274 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:40:33.274 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:40:33.274 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:40:33.275 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:40:33.275 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:40:33.275 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:40:33.275 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:40:33.275 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:40:33.275 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:40:33.276 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:40:33.276 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:40:33.276 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:40:33.276 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:40:33.276 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:40:33.276 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:40:33.277 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:40:33.277 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:33.277 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:40:33.277 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.277 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:40:33.278 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.278 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:40:33.278 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:40:33.278 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:40:33.278 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:40:33.278 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.279 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:40:33.279 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:40:33.279 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:33.279 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.279 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.279 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:40:33.280 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:40:33.280 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:33.280 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:33.280 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:33.280 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.281 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:40:33.281 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:40:33.281 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:40:33.281 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:40:33.281 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:40:33.281 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.282 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.282 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:40:33.282 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:40:33.282 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:40:33.282 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:40:33.283 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.283 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:40:33.283 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:40:33.283 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:33.283 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.284 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.284 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:40:33.284 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:40:33.284 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:33.284 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:33.284 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:33.285 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.285 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:40:33.285 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:40:33.285 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:40:33.285 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:40:33.286 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:40:33.286 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.286 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.286 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:40:33.286 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:40:33.287 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:40:33.287 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:40:33.287 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.287 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:40:33.288 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:40:33.288 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:33.288 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.288 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.288 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:40:33.288 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:40:33.289 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:33.289 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:33.289 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:33.289 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.289 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:40:33.290 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:40:33.290 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:40:33.290 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:40:33.290 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:40:33.290 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:33.290 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:33.291 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.291 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:40:33.291 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:40:33.291 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.291 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:40:33.291 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:40:33.292 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:40:33.292 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:33.292 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:33.292 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:40:33.292 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.293 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:40:33.293 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:40:33.293 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:40:33.293 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:33.294 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:33.294 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:40:33.294 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.294 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:40:33.294 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:40:33.295 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 8 2020-11-20T11:40:33.295 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:33.295 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:33.295 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:40:33.295 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.296 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:40:33.296 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:40:33.296 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:40:33.296 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:33.296 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:33.297 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:40:33.297 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.297 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:40:33.297 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:40:33.297 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:40:33.298 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:33.298 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:33.298 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.298 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:40:33.298 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:40:33.300 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:40:33.428 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.425+0000 7f6ad1640700 1 Processor -- start 2020-11-20T11:40:33.429 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.425+0000 7f6ad1640700 1 -- start start 2020-11-20T11:40:33.431 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.426+0000 7f6ad1640700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6acc0aaf00 0x7f6acc13e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:33.431 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.426+0000 7f6ad1640700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6acc0ab2e0 0x7f6acc0a7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:33.431 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.426+0000 7f6ad1640700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6acc13e010 con 0x7f6acc0a85c0 2020-11-20T11:40:33.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.426+0000 7f6ad1640700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6acc13e2b0 con 0x7f6acc0ab2e0 2020-11-20T11:40:33.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.426+0000 7f6ad1640700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc13e550 con 0x7f6acc0aaf00 2020-11-20T11:40:33.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.426+0000 7f6acaffd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6acc0aaf00 0x7f6acc13e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:33.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.426+0000 7f6acb7fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6acc0ab2e0 0x7f6acc0a7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:33.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.426+0000 7f6acaffd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6acc0aaf00 0x7f6acc13e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:54508/0 (socket says 172.21.15.23:54508) 2020-11-20T11:40:33.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.426+0000 7f6acaffd700 1 -- 172.21.15.23:0/2332546178 learned_addr learned my addr 172.21.15.23:0/2332546178 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:40:33.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.427+0000 7f6acb7fe700 1 -- 172.21.15.23:0/2332546178 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6acc0aaf00 msgr2=0x7f6acc13e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:40:33.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.427+0000 7f6acb7fe700 1 --2- 172.21.15.23:0/2332546178 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6acc0aaf00 0x7f6acc13e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:33.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.427+0000 7f6acb7fe700 1 -- 172.21.15.23:0/2332546178 >> v1:172.21.15.155:6789/0 conn(0x7f6acc0a85c0 legacy=0x7f6acc0a78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:33.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.427+0000 7f6acb7fe700 1 -- 172.21.15.23:0/2332546178 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6acc0a9000 con 0x7f6acc0ab2e0 2020-11-20T11:40:33.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.427+0000 7f6acaffd700 1 --2- 172.21.15.23:0/2332546178 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6acc0aaf00 0x7f6acc13e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:40:33.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.427+0000 7f6acb7fe700 1 --2- 172.21.15.23:0/2332546178 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6acc0ab2e0 0x7f6acc0a7fc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f6ac4009ce0 tx=0x7f6ac4009220).ready entity=mon.1 client_cookie=24e4401976c83d03 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:33.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.427+0000 7f6ac9ffb700 1 -- 172.21.15.23:0/2332546178 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac400ba40 con 0x7f6acc0ab2e0 2020-11-20T11:40:33.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.427+0000 7f6ac9ffb700 1 -- 172.21.15.23:0/2332546178 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac400bba0 con 0x7f6acc0ab2e0 2020-11-20T11:40:33.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.427+0000 7f6ac9ffb700 1 -- 172.21.15.23:0/2332546178 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac4011400 con 0x7f6acc0ab2e0 2020-11-20T11:40:33.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.428+0000 7f6ad1640700 1 -- 172.21.15.23:0/2332546178 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6acc0ab2e0 msgr2=0x7f6acc0a7fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:33.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.428+0000 7f6ad1640700 1 --2- 172.21.15.23:0/2332546178 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6acc0ab2e0 0x7f6acc0a7fc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f6ac4009ce0 tx=0x7f6ac4009220).stop 2020-11-20T11:40:33.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.428+0000 7f6ad1640700 1 -- 172.21.15.23:0/2332546178 shutdown_connections 2020-11-20T11:40:33.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.428+0000 7f6ad1640700 1 --2- 172.21.15.23:0/2332546178 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6acc0ab2e0 0x7f6acc0a7fc0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:33.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.428+0000 7f6ad1640700 1 --2- 172.21.15.23:0/2332546178 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6acc0aaf00 0x7f6acc13e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:33.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.428+0000 7f6ad1640700 1 -- 172.21.15.23:0/2332546178 >> 172.21.15.23:0/2332546178 conn(0x7f6acc0678e0 msgr2=0x7f6acc067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:40:33.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.428+0000 7f6ad1640700 1 -- 172.21.15.23:0/2332546178 shutdown_connections 2020-11-20T11:40:33.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.428+0000 7f6ad1640700 1 -- 172.21.15.23:0/2332546178 wait complete. 2020-11-20T11:40:33.438 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.429+0000 7f6ad1640700 1 Processor -- start 2020-11-20T11:40:33.438 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.429+0000 7f6ad1640700 1 -- start start 2020-11-20T11:40:33.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.430+0000 7f6ad1640700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6acc0a85c0 0x7f6acc0aff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:33.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.430+0000 7f6ad1640700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6acc0aaf00 0x7f6acc09f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:33.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.430+0000 7f6ad1640700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6acc0ab2e0 0x7f6acc14a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:33.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.430+0000 7f6ad1640700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc0b0830 con 0x7f6acc0ab2e0 2020-11-20T11:40:33.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.430+0000 7f6ad1640700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc0b0590 con 0x7f6acc0aaf00 2020-11-20T11:40:33.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.430+0000 7f6ad1640700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc149e10 con 0x7f6acc0a85c0 2020-11-20T11:40:33.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.430+0000 7f6acb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6acc0ab2e0 0x7f6acc14a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:33.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.430+0000 7f6acb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6acc0ab2e0 0x7f6acc14a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44274/0 (socket says 172.21.15.23:44274) 2020-11-20T11:40:33.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.430+0000 7f6acb7fe700 1 -- 172.21.15.23:0/205444097 learned_addr learned my addr 172.21.15.23:0/205444097 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:40:33.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.430+0000 7f6acaffd700 1 --2- 172.21.15.23:0/205444097 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6acc0a85c0 0x7f6acc0aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:33.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.430+0000 7f6aca7fc700 1 --2- 172.21.15.23:0/205444097 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6acc0aaf00 0x7f6acc09f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:33.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.430+0000 7f6acb7fe700 1 -- 172.21.15.23:0/205444097 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6acc0a85c0 msgr2=0x7f6acc0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:33.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.430+0000 7f6acb7fe700 1 --2- 172.21.15.23:0/205444097 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6acc0a85c0 0x7f6acc0aff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:33.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.430+0000 7f6acb7fe700 1 -- 172.21.15.23:0/205444097 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6acc0aaf00 msgr2=0x7f6acc09f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:33.442 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.430+0000 7f6acb7fe700 1 --2- 172.21.15.23:0/205444097 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6acc0aaf00 0x7f6acc09f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:33.442 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.430+0000 7f6acb7fe700 1 -- 172.21.15.23:0/205444097 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6acc09fd20 con 0x7f6acc0ab2e0 2020-11-20T11:40:33.442 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.430+0000 7f6acb7fe700 1 --2- 172.21.15.23:0/205444097 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6acc0ab2e0 0x7f6acc14a2b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f6ac4009260 tx=0x7f6ac4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:33.442 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.430+0000 7f6ab2ffd700 1 -- 172.21.15.23:0/205444097 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac4011400 con 0x7f6acc0ab2e0 2020-11-20T11:40:33.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.431+0000 7f6ab2ffd700 1 -- 172.21.15.23:0/205444097 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac4010a50 con 0x7f6acc0ab2e0 2020-11-20T11:40:33.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.431+0000 7f6ab2ffd700 1 -- 172.21.15.23:0/205444097 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac4010df0 con 0x7f6acc0ab2e0 2020-11-20T11:40:33.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.431+0000 7f6ad1640700 1 -- 172.21.15.23:0/205444097 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6acc09d860 con 0x7f6acc0ab2e0 2020-11-20T11:40:33.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.431+0000 7f6ad1640700 1 -- 172.21.15.23:0/205444097 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6acc14a9e0 con 0x7f6acc0ab2e0 2020-11-20T11:40:33.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.431+0000 7f6ad1640700 1 --2- 172.21.15.23:0/205444097 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6acc1323c0 0x7f6acc1327a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:33.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.431+0000 7f6ad1640700 1 -- 172.21.15.23:0/205444097 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc0b0450 con 0x7f6acc1323c0 2020-11-20T11:40:33.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.432+0000 7f6acaffd700 1 --2- 172.21.15.23:0/205444097 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6acc1323c0 0x7f6acc1327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:33.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.432+0000 7f6acaffd700 1 -- 172.21.15.23:0/205444097 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6acc149c90 con 0x7f6acc1323c0 2020-11-20T11:40:33.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.432+0000 7f6acaffd700 1 --2- 172.21.15.23:0/205444097 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6acc1323c0 0x7f6acc1327a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f6abc011ec0 tx=0x7f6abc006ef0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:33.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.432+0000 7f6ab2ffd700 1 -- 172.21.15.23:0/205444097 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6abc00fd00 con 0x7f6acc1323c0 2020-11-20T11:40:33.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.433+0000 7f6ab2ffd700 1 -- 172.21.15.23:0/205444097 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6ac4022030 con 0x7f6acc0ab2e0 2020-11-20T11:40:33.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.433+0000 7f6ab2ffd700 1 --2- 172.21.15.23:0/205444097 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6aa8056e00 0x7f6aa80592c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:33.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.433+0000 7f6ab2ffd700 1 -- 172.21.15.23:0/205444097 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f6abc00fe60 con 0x7f6acc1323c0 2020-11-20T11:40:33.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.433+0000 7f6ab2ffd700 1 -- 172.21.15.23:0/205444097 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6acc1323c0 msgr2=0x7f6acc1327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:33.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.433+0000 7f6ab2ffd700 1 --2- 172.21.15.23:0/205444097 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6acc1323c0 0x7f6acc1327a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f6abc011ec0 tx=0x7f6abc006ef0).stop 2020-11-20T11:40:33.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.433+0000 7f6ab2ffd700 1 -- 172.21.15.23:0/205444097 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6acc1323c0 msgr2=0x7f6acc1327a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:40:33.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.433+0000 7f6ab2ffd700 1 --2- 172.21.15.23:0/205444097 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6acc1323c0 0x7f6acc1327a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:33.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.433+0000 7f6ab2ffd700 1 -- 172.21.15.23:0/205444097 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f6ac4072670 con 0x7f6acc0ab2e0 2020-11-20T11:40:33.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.434+0000 7f6aca7fc700 1 --2- 172.21.15.23:0/205444097 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6aa8056e00 0x7f6aa80592c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:33.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.434+0000 7f6aca7fc700 1 --2- 172.21.15.23:0/205444097 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6aa8056e00 0x7f6aa80592c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f6acc136ff0 tx=0x7f6ac0007c60).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:33.448 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.445+0000 7f6ad1640700 1 --2- 172.21.15.23:0/205444097 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6acc14d260 0x7f6acc14d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:33.448 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.445+0000 7f6ad1640700 1 -- 172.21.15.23:0/205444097 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc09feb0 con 0x7f6acc14d260 2020-11-20T11:40:33.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.445+0000 7f6acaffd700 1 --2- 172.21.15.23:0/205444097 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6acc14d260 0x7f6acc14d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:33.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.446+0000 7f6acaffd700 1 -- 172.21.15.23:0/205444097 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6acc149fd0 con 0x7f6acc14d260 2020-11-20T11:40:33.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.446+0000 7f6acaffd700 1 --2- 172.21.15.23:0/205444097 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6acc14d260 0x7f6acc14d670 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f6abc011ec0 tx=0x7f6abc01c000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:33.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.446+0000 7f6ab2ffd700 1 -- 172.21.15.23:0/205444097 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6abc01eaf0 con 0x7f6acc14d260 2020-11-20T11:40:33.451 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.446+0000 7f6ab2ffd700 1 -- 172.21.15.23:0/205444097 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4273 (secure 0 0 0) 0x7f6abc00faf0 con 0x7f6acc14d260 2020-11-20T11:40:33.451 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.446+0000 7f6ab2ffd700 1 -- 172.21.15.23:0/205444097 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6acc14d260 msgr2=0x7f6acc14d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:33.451 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.446+0000 7f6ab2ffd700 1 --2- 172.21.15.23:0/205444097 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6acc14d260 0x7f6acc14d670 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f6abc011ec0 tx=0x7f6abc01c000).stop 2020-11-20T11:40:33.451 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.446+0000 7f6ab2ffd700 1 -- 172.21.15.23:0/205444097 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6acc14d260 msgr2=0x7f6acc14d670 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:40:33.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.446+0000 7f6ab2ffd700 1 --2- 172.21.15.23:0/205444097 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6acc14d260 0x7f6acc14d670 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:33.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.447+0000 7f6ad1640700 1 -- 172.21.15.23:0/205444097 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6aa8056e00 msgr2=0x7f6aa80592c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:33.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.447+0000 7f6ad1640700 1 --2- 172.21.15.23:0/205444097 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6aa8056e00 0x7f6aa80592c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f6acc136ff0 tx=0x7f6ac0007c60).stop 2020-11-20T11:40:33.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.447+0000 7f6ad1640700 1 -- 172.21.15.23:0/205444097 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6acc0ab2e0 msgr2=0x7f6acc14a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:33.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.447+0000 7f6ad1640700 1 --2- 172.21.15.23:0/205444097 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6acc0ab2e0 0x7f6acc14a2b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f6ac4009260 tx=0x7f6ac4016000).stop 2020-11-20T11:40:33.453 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.447+0000 7f6acb7fe700 1 -- 172.21.15.23:0/205444097 reap_dead start 2020-11-20T11:40:33.453 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.447+0000 7f6ad1640700 1 -- 172.21.15.23:0/205444097 shutdown_connections 2020-11-20T11:40:33.453 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.447+0000 7f6ad1640700 1 --2- 172.21.15.23:0/205444097 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6acc0ab2e0 0x7f6acc14a2b0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:33.453 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.447+0000 7f6ad1640700 1 -- 172.21.15.23:0/205444097 >> 172.21.15.23:0/205444097 conn(0x7f6acc0678e0 msgr2=0x7f6acc067cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:40:33.453 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.447+0000 7f6ad1640700 1 -- 172.21.15.23:0/205444097 shutdown_connections 2020-11-20T11:40:33.454 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.447+0000 7f6ad1640700 1 -- 172.21.15.23:0/205444097 wait complete. 2020-11-20T11:40:33.454 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:40:33.454 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:40:33.454 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:40:33.455 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:40:33.455 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 82, 2020-11-20T11:40:33.455 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:40:33.455 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:40:33.455 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:40:33.456 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:40:33.456 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:33.456 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 0, 2020-11-20T11:40:33.456 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:40:33.456 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:40:33.457 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:40:33.457 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:40:33.457 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:40:33.457 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:40:33.458 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:40:33.458 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:40:33.458 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:40:33.458 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:40:33.458 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:40:33.459 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:33.459 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:40:33.459 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:40:33.459 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:40:33.460 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:40:33.460 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:40:33.460 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:40:33.460 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:40:33.461 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:40:33.461 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:40:33.461 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:40:33.461 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:33.461 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.462 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:40:33.462 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:40:33.462 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:40:33.462 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:40:33.462 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:40:33.463 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:40:33.463 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:40:33.463 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:40:33.463 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:40:33.463 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:40:33.464 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:40:33.464 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:40:33.464 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:40:33.464 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:40:33.464 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:40:33.465 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:40:33.465 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:40:33.465 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:40:33.465 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:40:33.465 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:40:33.466 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:40:33.466 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:40:33.466 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:40:33.466 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:33.467 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:40:33.467 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.467 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:40:33.467 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.467 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:40:33.468 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:40:33.468 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:40:33.468 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:40:33.468 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.468 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:40:33.469 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:40:33.469 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:33.469 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.469 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.469 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:40:33.470 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:40:33.470 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:33.470 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:33.470 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:33.470 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.471 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:40:33.471 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:40:33.471 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:40:33.471 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:40:33.471 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:40:33.472 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.472 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.472 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:40:33.472 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:40:33.472 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:40:33.473 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:40:33.473 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.473 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:40:33.473 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:40:33.473 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:33.474 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.474 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.474 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:40:33.474 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:40:33.474 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:33.475 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:33.475 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:33.475 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.475 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:40:33.475 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:40:33.476 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:40:33.476 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:40:33.476 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:40:33.476 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.476 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.477 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:40:33.477 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:40:33.477 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:40:33.477 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:40:33.477 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.478 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:40:33.478 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:40:33.478 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:33.478 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.478 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.479 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:40:33.479 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:40:33.479 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:33.479 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:33.479 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:33.480 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.480 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:40:33.480 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:40:33.480 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:40:33.480 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:40:33.481 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:40:33.481 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:33.481 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:33.481 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.481 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:40:33.482 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:40:33.482 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.482 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:40:33.482 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:40:33.482 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:40:33.483 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:33.483 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:33.483 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:40:33.483 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.483 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:40:33.484 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:40:33.484 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:40:33.484 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:33.484 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:33.484 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.485 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:40:33.485 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:40:33.486 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:40:33.618 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.615+0000 7f7efe852700 1 Processor -- start 2020-11-20T11:40:33.618 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.615+0000 7f7efe852700 1 -- start start 2020-11-20T11:40:33.621 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.616+0000 7f7efe852700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f7ef80ace70 0x7f7ef813a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:33.621 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.616+0000 7f7efe852700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ef80ad860 con 0x7f7ef80a9040 2020-11-20T11:40:33.621 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.616+0000 7f7efe852700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ef80aaa90 con 0x7f7ef80ad2f0 2020-11-20T11:40:33.621 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.616+0000 7f7efe852700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ef80aa8a0 con 0x7f7ef80ace70 2020-11-20T11:40:33.622 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.616+0000 7f7efd850700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f7ef80a9040 0x7f7ef80aaf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58536/0 (socket says 172.21.15.23:58536) 2020-11-20T11:40:33.622 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.616+0000 7f7efd850700 1 -- 172.21.15.23:0/4226126486 learned_addr learned my addr 172.21.15.23:0/4226126486 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:40:33.622 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.616+0000 7f7efd04f700 1 --2- 172.21.15.23:0/4226126486 >> v2:172.21.15.155:3300/0 conn(0x7f7ef80ace70 0x7f7ef813a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:33.622 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.616+0000 7f7efc84e700 1 -- 172.21.15.23:0/4226126486 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2109900409 0 0) 0x7f7ef80ad860 con 0x7f7ef80a9040 2020-11-20T11:40:33.623 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.616+0000 7f7efc84e700 1 -- 172.21.15.23:0/4226126486 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ee0003280 con 0x7f7ef80a9040 2020-11-20T11:40:33.623 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.616+0000 7f7efc84e700 1 -- 172.21.15.23:0/4226126486 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1101457791 0 0) 0x7f7ee0003280 con 0x7f7ef80a9040 2020-11-20T11:40:33.623 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.617+0000 7f7efd04f700 1 -- 172.21.15.23:0/4226126486 >> v1:172.21.15.155:6789/0 conn(0x7f7ef80ad2f0 legacy=0x7f7ef813ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:33.623 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.617+0000 7f7efd04f700 1 -- 172.21.15.23:0/4226126486 >> v1:172.21.15.23:6789/0 conn(0x7f7ef80a9040 legacy=0x7f7ef80aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:33.623 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.617+0000 7f7efd04f700 1 -- 172.21.15.23:0/4226126486 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef80a9420 con 0x7f7ef80ace70 2020-11-20T11:40:33.624 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.617+0000 7f7efd04f700 1 --2- 172.21.15.23:0/4226126486 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ef80ace70 0x7f7ef813a830 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f7ee8006fd0 tx=0x7f7ee8006e30).ready entity=mon.1 client_cookie=10cf28a3ddadfe4d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:33.624 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.617+0000 7f7efc84e700 1 -- 172.21.15.23:0/4226126486 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef40016a0 con 0x7f7ef80ace70 2020-11-20T11:40:33.624 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.617+0000 7f7efc84e700 1 -- 172.21.15.23:0/4226126486 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ee8000e90 con 0x7f7ef80ace70 2020-11-20T11:40:33.624 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.617+0000 7f7efc84e700 1 -- 172.21.15.23:0/4226126486 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee80096e0 con 0x7f7ef80ace70 2020-11-20T11:40:33.625 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.617+0000 7f7efe852700 1 -- 172.21.15.23:0/4226126486 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ef80ace70 msgr2=0x7f7ef813a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:33.625 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.617+0000 7f7efe852700 1 --2- 172.21.15.23:0/4226126486 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ef80ace70 0x7f7ef813a830 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f7ee8006fd0 tx=0x7f7ee8006e30).stop 2020-11-20T11:40:33.625 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.617+0000 7f7efe852700 1 -- 172.21.15.23:0/4226126486 shutdown_connections 2020-11-20T11:40:33.625 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.617+0000 7f7efe852700 1 --2- 172.21.15.23:0/4226126486 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ef80ace70 0x7f7ef813a830 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:33.625 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.617+0000 7f7efe852700 1 -- 172.21.15.23:0/4226126486 >> 172.21.15.23:0/4226126486 conn(0x7f7ef80683b0 msgr2=0x7f7ef80b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:40:33.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.618+0000 7f7efe852700 1 -- 172.21.15.23:0/4226126486 shutdown_connections 2020-11-20T11:40:33.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.618+0000 7f7efe852700 1 -- 172.21.15.23:0/4226126486 wait complete. 2020-11-20T11:40:33.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.619+0000 7f7efe852700 1 Processor -- start 2020-11-20T11:40:33.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.619+0000 7f7efe852700 1 -- start start 2020-11-20T11:40:33.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.619+0000 7f7efe852700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ef80a9040 0x7f7ef809fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:33.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.619+0000 7f7efe852700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ef80ace70 0x7f7ef809e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:33.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.619+0000 7f7efe852700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ef80ad2f0 0x7f7ef809e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:33.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.619+0000 7f7efe852700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef814e100 con 0x7f7ef80ace70 2020-11-20T11:40:33.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.619+0000 7f7efe852700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef814d720 con 0x7f7ef80a9040 2020-11-20T11:40:33.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.619+0000 7f7efe852700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef814d9c0 con 0x7f7ef80ad2f0 2020-11-20T11:40:33.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.619+0000 7f7efd04f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ef80ace70 0x7f7ef809e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:33.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.620+0000 7f7efd04f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ef80ace70 0x7f7ef809e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44294/0 (socket says 172.21.15.23:44294) 2020-11-20T11:40:33.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.620+0000 7f7efd04f700 1 -- 172.21.15.23:0/1769907959 learned_addr learned my addr 172.21.15.23:0/1769907959 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:40:33.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.620+0000 7f7efd850700 1 --2- 172.21.15.23:0/1769907959 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ef80a9040 0x7f7ef809fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:33.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.620+0000 7f7efe051700 1 --2- 172.21.15.23:0/1769907959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ef80ad2f0 0x7f7ef809e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:33.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.620+0000 7f7efd04f700 1 -- 172.21.15.23:0/1769907959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ef80ad2f0 msgr2=0x7f7ef809e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:33.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.620+0000 7f7efd04f700 1 --2- 172.21.15.23:0/1769907959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ef80ad2f0 0x7f7ef809e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:33.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.620+0000 7f7efd04f700 1 -- 172.21.15.23:0/1769907959 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ef80a9040 msgr2=0x7f7ef809fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:33.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.620+0000 7f7efd04f700 1 --2- 172.21.15.23:0/1769907959 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7ef80a9040 0x7f7ef809fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:33.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.620+0000 7f7efd04f700 1 -- 172.21.15.23:0/1769907959 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef814dc60 con 0x7f7ef80ace70 2020-11-20T11:40:33.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.620+0000 7f7efd04f700 1 --2- 172.21.15.23:0/1769907959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ef80ace70 0x7f7ef809e340 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f7ef80ad790 tx=0x7f7ef80aa360).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:33.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.620+0000 7f7ee5ffb700 1 -- 172.21.15.23:0/1769907959 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee8012670 con 0x7f7ef80ace70 2020-11-20T11:40:33.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.620+0000 7f7ee5ffb700 1 -- 172.21.15.23:0/1769907959 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ee8017030 con 0x7f7ef80ace70 2020-11-20T11:40:33.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.620+0000 7f7efe852700 1 -- 172.21.15.23:0/1769907959 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ef809d7f0 con 0x7f7ef80ace70 2020-11-20T11:40:33.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.620+0000 7f7ee5ffb700 1 -- 172.21.15.23:0/1769907959 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee80129e0 con 0x7f7ef80ace70 2020-11-20T11:40:33.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.621+0000 7f7efe852700 1 -- 172.21.15.23:0/1769907959 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ef814d450 con 0x7f7ef80ace70 2020-11-20T11:40:33.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.621+0000 7f7efe852700 1 --2- 172.21.15.23:0/1769907959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ef80a18a0 0x7f7ef80a6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:33.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.621+0000 7f7efe852700 1 -- 172.21.15.23:0/1769907959 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef805cf20 con 0x7f7ef80a18a0 2020-11-20T11:40:33.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.622+0000 7f7efd850700 1 --2- 172.21.15.23:0/1769907959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ef80a18a0 0x7f7ef80a6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:33.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.622+0000 7f7ee5ffb700 1 -- 172.21.15.23:0/1769907959 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7ee8025020 con 0x7f7ef80ace70 2020-11-20T11:40:33.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.622+0000 7f7efd850700 1 -- 172.21.15.23:0/1769907959 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7ef8057dc0 con 0x7f7ef80a18a0 2020-11-20T11:40:33.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.622+0000 7f7efd850700 1 --2- 172.21.15.23:0/1769907959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ef80a18a0 0x7f7ef80a6e10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f7ef809db10 tx=0x7f7ef40012c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:33.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.622+0000 7f7ee5ffb700 1 --2- 172.21.15.23:0/1769907959 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7ed0056d60 0x7f7ed0059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:33.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.622+0000 7f7ee5ffb700 1 -- 172.21.15.23:0/1769907959 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f7ee8070ee0 con 0x7f7ef80ace70 2020-11-20T11:40:33.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.622+0000 7f7ee5ffb700 1 -- 172.21.15.23:0/1769907959 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef4006240 con 0x7f7ef80a18a0 2020-11-20T11:40:33.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.622+0000 7f7efe051700 1 --2- 172.21.15.23:0/1769907959 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7ed0056d60 0x7f7ed0059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:33.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.623+0000 7f7ee5ffb700 1 -- 172.21.15.23:0/1769907959 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f7ef40063a0 con 0x7f7ef80a18a0 2020-11-20T11:40:33.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.623+0000 7f7ee5ffb700 1 -- 172.21.15.23:0/1769907959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ef80a18a0 msgr2=0x7f7ef80a6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:33.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.623+0000 7f7ee5ffb700 1 --2- 172.21.15.23:0/1769907959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ef80a18a0 0x7f7ef80a6e10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f7ef809db10 tx=0x7f7ef40012c0).stop 2020-11-20T11:40:33.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.623+0000 7f7ee5ffb700 1 -- 172.21.15.23:0/1769907959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ef80a18a0 msgr2=0x7f7ef80a6e10 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:40:33.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.623+0000 7f7ee5ffb700 1 --2- 172.21.15.23:0/1769907959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ef80a18a0 0x7f7ef80a6e10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:33.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.623+0000 7f7efe051700 1 --2- 172.21.15.23:0/1769907959 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7ed0056d60 0x7f7ed0059220 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f7ef0007fd0 tx=0x7f7ef0007e70).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:33.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.630+0000 7f7efe852700 1 --2- 172.21.15.23:0/1769907959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ef80a7300 0x7f7ef8149c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:33.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.630+0000 7f7efe852700 1 -- 172.21.15.23:0/1769907959 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef80a7710 con 0x7f7ef80a7300 2020-11-20T11:40:33.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.631+0000 7f7efd850700 1 --2- 172.21.15.23:0/1769907959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ef80a7300 0x7f7ef8149c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:33.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.631+0000 7f7efd850700 1 -- 172.21.15.23:0/1769907959 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7ef8044710 con 0x7f7ef80a7300 2020-11-20T11:40:33.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.631+0000 7f7efd850700 1 --2- 172.21.15.23:0/1769907959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ef80a7300 0x7f7ef8149c80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f7ef809db10 tx=0x7f7ef4018000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:33.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.631+0000 7f7ee5ffb700 1 -- 172.21.15.23:0/1769907959 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef4005cf0 con 0x7f7ef80a7300 2020-11-20T11:40:33.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.632+0000 7f7ee5ffb700 1 -- 172.21.15.23:0/1769907959 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4613 (secure 0 0 0) 0x7f7ef4005e50 con 0x7f7ef80a7300 2020-11-20T11:40:33.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.632+0000 7f7ee5ffb700 1 -- 172.21.15.23:0/1769907959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ef80a7300 msgr2=0x7f7ef8149c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:33.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.632+0000 7f7ee5ffb700 1 --2- 172.21.15.23:0/1769907959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ef80a7300 0x7f7ef8149c80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f7ef809db10 tx=0x7f7ef4018000).stop 2020-11-20T11:40:33.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.632+0000 7f7ee5ffb700 1 -- 172.21.15.23:0/1769907959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ef80a7300 msgr2=0x7f7ef8149c80 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:40:33.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.632+0000 7f7ee5ffb700 1 --2- 172.21.15.23:0/1769907959 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7ef80a7300 0x7f7ef8149c80 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:33.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.632+0000 7f7efe852700 1 -- 172.21.15.23:0/1769907959 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7ed0056d60 msgr2=0x7f7ed0059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:33.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.632+0000 7f7efe852700 1 --2- 172.21.15.23:0/1769907959 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7ed0056d60 0x7f7ed0059220 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f7ef0007fd0 tx=0x7f7ef0007e70).stop 2020-11-20T11:40:33.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.632+0000 7f7efe852700 1 -- 172.21.15.23:0/1769907959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ef80ace70 msgr2=0x7f7ef809e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:33.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.632+0000 7f7efe852700 1 --2- 172.21.15.23:0/1769907959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ef80ace70 0x7f7ef809e340 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f7ef80ad790 tx=0x7f7ef80aa360).stop 2020-11-20T11:40:33.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.632+0000 7f7efe051700 1 -- 172.21.15.23:0/1769907959 reap_dead start 2020-11-20T11:40:33.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.632+0000 7f7efe051700 1 -- 172.21.15.23:0/1769907959 reap_dead start 2020-11-20T11:40:33.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.632+0000 7f7efe852700 1 -- 172.21.15.23:0/1769907959 shutdown_connections 2020-11-20T11:40:33.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.632+0000 7f7efe852700 1 -- 172.21.15.23:0/1769907959 >> 172.21.15.23:0/1769907959 conn(0x7f7ef80683b0 msgr2=0x7f7ef80ac6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:40:33.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.632+0000 7f7efe852700 1 -- 172.21.15.23:0/1769907959 shutdown_connections 2020-11-20T11:40:33.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.632+0000 7f7efe852700 1 -- 172.21.15.23:0/1769907959 wait complete. 2020-11-20T11:40:33.643 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:40:33.644 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:40:33.644 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:40:33.644 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:40:33.644 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 82, 2020-11-20T11:40:33.644 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:40:33.645 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:40:33.645 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:40:33.645 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:40:33.645 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:33.646 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 0, 2020-11-20T11:40:33.646 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:40:33.647 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:40:33.647 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:40:33.647 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:40:33.647 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:40:33.648 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:40:33.648 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:40:33.648 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:40:33.648 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:40:33.648 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:40:33.649 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:40:33.649 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:33.649 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:40:33.649 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:40:33.649 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:40:33.650 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:40:33.650 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:40:33.650 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:40:33.650 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:40:33.650 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:40:33.651 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:40:33.651 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:40:33.651 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:33.651 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.651 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:40:33.652 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:40:33.652 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:40:33.652 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:40:33.652 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:40:33.653 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:40:33.653 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:40:33.653 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:40:33.653 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:40:33.653 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:40:33.654 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:40:33.654 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:40:33.654 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:40:33.654 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:40:33.654 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:40:33.655 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:40:33.655 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:40:33.655 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:40:33.655 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:40:33.655 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:40:33.656 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:40:33.656 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:40:33.656 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:40:33.656 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:33.656 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:40:33.657 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.657 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:40:33.657 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.657 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:40:33.657 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:40:33.658 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:40:33.658 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:40:33.658 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.658 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:40:33.658 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:40:33.659 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:33.659 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.659 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.659 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:40:33.659 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:40:33.660 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:33.660 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:33.660 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:33.660 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.660 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:40:33.661 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:40:33.661 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:40:33.661 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:40:33.661 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:40:33.662 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.662 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.662 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:40:33.662 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:40:33.662 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:40:33.663 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:40:33.663 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.663 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:40:33.663 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:40:33.663 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:33.664 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.664 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.664 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:40:33.664 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:40:33.664 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:33.665 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:33.665 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:33.665 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.665 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:40:33.665 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:40:33.666 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:40:33.666 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:40:33.666 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:40:33.666 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.666 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.667 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:40:33.667 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:40:33.667 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:40:33.667 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:40:33.667 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.668 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:40:33.668 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:40:33.668 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:33.668 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.668 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.669 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:40:33.669 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:40:33.669 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:33.669 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:33.669 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:33.670 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.670 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:40:33.670 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:40:33.670 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:40:33.671 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:40:33.671 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:40:33.671 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:33.671 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:33.671 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.672 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:40:33.672 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:40:33.672 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.672 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:40:33.672 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:40:33.673 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:40:33.673 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:33.673 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:33.673 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:40:33.673 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.674 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:40:33.674 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:40:33.674 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:40:33.674 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:33.674 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:33.675 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:40:33.675 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.675 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:40:33.675 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:40:33.675 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:40:33.676 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:33.676 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:33.676 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:40:33.676 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:33.676 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:40:33.677 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:40:33.677 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:40:33.677 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:33.677 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:33.678 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:33.678 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:40:33.678 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:40:33.679 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2020-11-20T11:40:33.679 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2020-11-20T11:40:33.804 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.801+0000 7feb8b59e700 1 Processor -- start 2020-11-20T11:40:33.805 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.801+0000 7feb8b59e700 1 -- start start 2020-11-20T11:40:33.807 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.802+0000 7feb8b59e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb8c0ace70 0x7feb8c13a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:33.807 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.802+0000 7feb8b59e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7feb8c0ad2f0 0x7feb8c13ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:33.807 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.802+0000 7feb8b59e700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb8c0ad860 con 0x7feb8c0a9040 2020-11-20T11:40:33.808 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.802+0000 7feb8b59e700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb8c0aa8a0 con 0x7feb8c0ad2f0 2020-11-20T11:40:33.808 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.802+0000 7feb8b59e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c0aaa90 con 0x7feb8c0ace70 2020-11-20T11:40:33.808 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.802+0000 7feb8a59c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7feb8c0a9040 0x7feb8c0aaf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58554/0 (socket says 172.21.15.23:58554) 2020-11-20T11:40:33.808 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.802+0000 7feb8a59c700 1 -- 172.21.15.23:0/3148321741 learned_addr learned my addr 172.21.15.23:0/3148321741 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:40:33.809 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.802+0000 7feb8ad9d700 1 --2- 172.21.15.23:0/3148321741 >> v2:172.21.15.155:3300/0 conn(0x7feb8c0ad2f0 0x7feb8c13ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:33.809 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.802+0000 7feb89d9b700 1 --2- 172.21.15.23:0/3148321741 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb8c0ace70 0x7feb8c13a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:33.809 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.802+0000 7feb8959a700 1 -- 172.21.15.23:0/3148321741 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 928208939 0 0) 0x7feb8c0ad860 con 0x7feb8c0a9040 2020-11-20T11:40:33.809 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.802+0000 7feb8959a700 1 -- 172.21.15.23:0/3148321741 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb70003280 con 0x7feb8c0a9040 2020-11-20T11:40:33.809 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.803+0000 7feb8959a700 1 -- 172.21.15.23:0/3148321741 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1641879664 0 0) 0x7feb70003280 con 0x7feb8c0a9040 2020-11-20T11:40:33.810 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.803+0000 7feb8ad9d700 1 -- 172.21.15.23:0/3148321741 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb8c0ace70 msgr2=0x7feb8c13a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:40:33.810 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.803+0000 7feb8ad9d700 1 --2- 172.21.15.23:0/3148321741 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb8c0ace70 0x7feb8c13a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:33.810 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.803+0000 7feb8ad9d700 1 -- 172.21.15.23:0/3148321741 >> v1:172.21.15.23:6789/0 conn(0x7feb8c0a9040 legacy=0x7feb8c0aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:33.810 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.803+0000 7feb8ad9d700 1 -- 172.21.15.23:0/3148321741 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8c0a9420 con 0x7feb8c0ad2f0 2020-11-20T11:40:33.810 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.803+0000 7feb89d9b700 1 --2- 172.21.15.23:0/3148321741 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb8c0ace70 0x7feb8c13a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:40:33.811 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.803+0000 7feb8ad9d700 1 --2- 172.21.15.23:0/3148321741 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb8c0ad2f0 0x7feb8c13ad20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7feb84009ce0 tx=0x7feb84009220).ready entity=mon.1 client_cookie=e6240ae57802550b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:33.811 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.803+0000 7feb8959a700 1 -- 172.21.15.23:0/3148321741 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7c0016a0 con 0x7feb8c0ad2f0 2020-11-20T11:40:33.811 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.803+0000 7feb8959a700 1 -- 172.21.15.23:0/3148321741 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb8400b680 con 0x7feb8c0ad2f0 2020-11-20T11:40:33.811 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.803+0000 7feb8959a700 1 -- 172.21.15.23:0/3148321741 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb8400ba00 con 0x7feb8c0ad2f0 2020-11-20T11:40:33.812 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.804+0000 7feb8b59e700 1 -- 172.21.15.23:0/3148321741 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb8c0ad2f0 msgr2=0x7feb8c13ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:33.812 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.804+0000 7feb8b59e700 1 --2- 172.21.15.23:0/3148321741 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb8c0ad2f0 0x7feb8c13ad20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7feb84009ce0 tx=0x7feb84009220).stop 2020-11-20T11:40:33.812 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.804+0000 7feb8b59e700 1 -- 172.21.15.23:0/3148321741 shutdown_connections 2020-11-20T11:40:33.812 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.804+0000 7feb8b59e700 1 --2- 172.21.15.23:0/3148321741 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb8c0ad2f0 0x7feb8c13ad20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:33.812 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.804+0000 7feb8b59e700 1 --2- 172.21.15.23:0/3148321741 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb8c0ace70 0x7feb8c13a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:33.813 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.804+0000 7feb8b59e700 1 -- 172.21.15.23:0/3148321741 >> 172.21.15.23:0/3148321741 conn(0x7feb8c0683b0 msgr2=0x7feb8c0b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:40:33.813 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.804+0000 7feb8b59e700 1 -- 172.21.15.23:0/3148321741 shutdown_connections 2020-11-20T11:40:33.813 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.804+0000 7feb8b59e700 1 -- 172.21.15.23:0/3148321741 wait complete. 2020-11-20T11:40:33.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.805+0000 7feb8b59e700 1 Processor -- start 2020-11-20T11:40:33.816 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.805+0000 7feb8b59e700 1 -- start start 2020-11-20T11:40:33.816 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.805+0000 7feb8b59e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb8c0a9040 0x7feb8c13dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:33.816 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.805+0000 7feb8b59e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb8c0ace70 0x7feb8c09e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:33.816 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.806+0000 7feb8b59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb8c0ad2f0 0x7feb8c09ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:33.817 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.806+0000 7feb8b59e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c09f330 con 0x7feb8c0ad2f0 2020-11-20T11:40:33.817 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.806+0000 7feb8b59e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c09fe30 con 0x7feb8c0ace70 2020-11-20T11:40:33.817 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.806+0000 7feb8b59e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb8c0a00d0 con 0x7feb8c0a9040 2020-11-20T11:40:33.817 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.806+0000 7feb8ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb8c0ad2f0 0x7feb8c09ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:33.817 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.806+0000 7feb8ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb8c0ad2f0 0x7feb8c09ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44314/0 (socket says 172.21.15.23:44314) 2020-11-20T11:40:33.818 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.806+0000 7feb8a59c700 1 --2- 172.21.15.23:0/3086128085 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb8c0a9040 0x7feb8c13dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:33.818 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.806+0000 7feb8ad9d700 1 -- 172.21.15.23:0/3086128085 learned_addr learned my addr 172.21.15.23:0/3086128085 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:40:33.818 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.806+0000 7feb89d9b700 1 --2- 172.21.15.23:0/3086128085 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb8c0ace70 0x7feb8c09e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:33.818 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.806+0000 7feb8ad9d700 1 -- 172.21.15.23:0/3086128085 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb8c0a9040 msgr2=0x7feb8c13dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:33.818 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.806+0000 7feb8ad9d700 1 --2- 172.21.15.23:0/3086128085 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb8c0a9040 0x7feb8c13dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:33.819 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.806+0000 7feb8ad9d700 1 -- 172.21.15.23:0/3086128085 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb8c0ace70 msgr2=0x7feb8c09e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:33.819 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.806+0000 7feb8ad9d700 1 --2- 172.21.15.23:0/3086128085 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb8c0ace70 0x7feb8c09e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:33.819 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.806+0000 7feb8ad9d700 1 -- 172.21.15.23:0/3086128085 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8c0a0370 con 0x7feb8c0ad2f0 2020-11-20T11:40:33.819 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.806+0000 7feb8ad9d700 1 --2- 172.21.15.23:0/3086128085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb8c0ad2f0 0x7feb8c09ee40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7feb840090b0 tx=0x7feb84016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:33.819 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.806+0000 7feb7a7fc700 1 -- 172.21.15.23:0/3086128085 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb8401c650 con 0x7feb8c0ad2f0 2020-11-20T11:40:33.820 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.806+0000 7feb7a7fc700 1 -- 172.21.15.23:0/3086128085 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb8401c7b0 con 0x7feb8c0ad2f0 2020-11-20T11:40:33.820 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.806+0000 7feb8b59e700 1 -- 172.21.15.23:0/3086128085 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb8c13e610 con 0x7feb8c0ad2f0 2020-11-20T11:40:33.820 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.806+0000 7feb7a7fc700 1 -- 172.21.15.23:0/3086128085 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb8401cb30 con 0x7feb8c0ad2f0 2020-11-20T11:40:33.820 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.806+0000 7feb8b59e700 1 -- 172.21.15.23:0/3086128085 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb8c09fbd0 con 0x7feb8c0ad2f0 2020-11-20T11:40:33.820 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.807+0000 7feb8b59e700 1 -- 172.21.15.23:0/3086128085 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb8c13b3b0 con 0x7feb8c0ad2f0 2020-11-20T11:40:33.820 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.808+0000 7feb7a7fc700 1 -- 172.21.15.23:0/3086128085 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7feb8402c070 con 0x7feb8c0ad2f0 2020-11-20T11:40:33.821 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.809+0000 7feb7a7fc700 1 --2- 172.21.15.23:0/3086128085 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7feb60056d60 0x7feb60059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:33.821 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.809+0000 7feb7a7fc700 1 -- 172.21.15.23:0/3086128085 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7feb84072470 con 0x7feb8c0ad2f0 2020-11-20T11:40:33.821 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.809+0000 7feb8a59c700 1 --2- 172.21.15.23:0/3086128085 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7feb60056d60 0x7feb60059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:33.821 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.809+0000 7feb8a59c700 1 --2- 172.21.15.23:0/3086128085 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7feb60056d60 0x7feb60059220 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7feb7c000ef0 tx=0x7feb7c001440).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:33.822 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:33.813+0000 7feb7a7fc700 1 -- 172.21.15.23:0/3086128085 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7feb840477e0 con 0x7feb8c0ad2f0 2020-11-20T11:40:34.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:34.063+0000 7feb8b59e700 1 -- 172.21.15.23:0/3086128085 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7feb8c057dc0 con 0x7feb8c0ad2f0 2020-11-20T11:40:34.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:34.064+0000 7feb7a7fc700 1 -- 172.21.15.23:0/3086128085 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7feb8402a080 con 0x7feb8c0ad2f0 2020-11-20T11:40:34.069 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:34.065+0000 7feb8b59e700 1 -- 172.21.15.23:0/3086128085 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7feb60056d60 msgr2=0x7feb60059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:34.069 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:34.065+0000 7feb8b59e700 1 --2- 172.21.15.23:0/3086128085 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7feb60056d60 0x7feb60059220 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7feb7c000ef0 tx=0x7feb7c001440).stop 2020-11-20T11:40:34.069 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:34.065+0000 7feb8b59e700 1 -- 172.21.15.23:0/3086128085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb8c0ad2f0 msgr2=0x7feb8c09ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:34.069 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:34.065+0000 7feb8b59e700 1 --2- 172.21.15.23:0/3086128085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb8c0ad2f0 0x7feb8c09ee40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7feb840090b0 tx=0x7feb84016000).stop 2020-11-20T11:40:34.070 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:34.065+0000 7feb8b59e700 1 -- 172.21.15.23:0/3086128085 shutdown_connections 2020-11-20T11:40:34.070 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:34.065+0000 7feb8b59e700 1 --2- 172.21.15.23:0/3086128085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feb8c0ad2f0 0x7feb8c09ee40 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:34.070 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:34.065+0000 7feb8b59e700 1 --2- 172.21.15.23:0/3086128085 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7feb60056d60 0x7feb60059220 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:34.070 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:34.065+0000 7feb8b59e700 1 --2- 172.21.15.23:0/3086128085 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feb8c0a9040 0x7feb8c13dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:34.070 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:34.065+0000 7feb8b59e700 1 --2- 172.21.15.23:0/3086128085 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feb8c0ace70 0x7feb8c09e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:34.071 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:34.065+0000 7feb8b59e700 1 -- 172.21.15.23:0/3086128085 >> 172.21.15.23:0/3086128085 conn(0x7feb8c0683b0 msgr2=0x7feb8c0aae50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:40:34.071 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:34.066+0000 7feb8b59e700 1 -- 172.21.15.23:0/3086128085 shutdown_connections 2020-11-20T11:40:34.071 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:34.066+0000 7feb8b59e700 1 -- 172.21.15.23:0/3086128085 wait complete. 2020-11-20T11:40:34.081 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2020-11-20T11:40:44.082 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:40:44.082 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:40:44.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.212+0000 7f70e2b00700 1 Processor -- start 2020-11-20T11:40:44.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.212+0000 7f70e2b00700 1 -- start start 2020-11-20T11:40:44.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.213+0000 7f70e2b00700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f70dc060a80 0x7f70dc09d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:44.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.213+0000 7f70e2b00700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f70dc09ded0 0x7f70dc0678b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:44.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.213+0000 7f70e2b00700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70dc0a9b40 con 0x7f70dc067da0 2020-11-20T11:40:44.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.213+0000 7f70e2b00700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70dc068230 con 0x7f70dc060a80 2020-11-20T11:40:44.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.213+0000 7f70e2b00700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70dc0aacb0 con 0x7f70dc09ded0 2020-11-20T11:40:44.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.213+0000 7f70e089c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f70dc060a80 0x7f70dc09d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:44.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.213+0000 7f70e089c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f70dc060a80 0x7f70dc09d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44318/0 (socket says 172.21.15.23:44318) 2020-11-20T11:40:44.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.213+0000 7f70e089c700 1 -- 172.21.15.23:0/1806764152 learned_addr learned my addr 172.21.15.23:0/1806764152 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:40:44.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.213+0000 7f70dbfff700 1 --2- 172.21.15.23:0/1806764152 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f70dc09ded0 0x7f70dc0678b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:44.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.213+0000 7f70e089c700 1 -- 172.21.15.23:0/1806764152 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f70dc09ded0 msgr2=0x7f70dc0678b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:40:44.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.213+0000 7f70e089c700 1 --2- 172.21.15.23:0/1806764152 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f70dc09ded0 0x7f70dc0678b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:44.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.213+0000 7f70e089c700 1 -- 172.21.15.23:0/1806764152 >> v1:172.21.15.155:6789/0 conn(0x7f70dc067da0 legacy=0x7f70dc135650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:40:44.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.213+0000 7f70e089c700 1 -- 172.21.15.23:0/1806764152 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70dc0a9da0 con 0x7f70dc060a80 2020-11-20T11:40:44.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.213+0000 7f70dbfff700 1 --2- 172.21.15.23:0/1806764152 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f70dc09ded0 0x7f70dc0678b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:40:44.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.213+0000 7f70e089c700 1 --2- 172.21.15.23:0/1806764152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70dc060a80 0x7f70dc09d9e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f70dc068370 tx=0x7f70d0006ef0).ready entity=mon.0 client_cookie=9f7a867ccc3f4643 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:44.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.213+0000 7f70db7fe700 1 -- 172.21.15.23:0/1806764152 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d000b820 con 0x7f70dc060a80 2020-11-20T11:40:44.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.213+0000 7f70db7fe700 1 -- 172.21.15.23:0/1806764152 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70d0000e90 con 0x7f70dc060a80 2020-11-20T11:40:44.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.213+0000 7f70db7fe700 1 -- 172.21.15.23:0/1806764152 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d000bb90 con 0x7f70dc060a80 2020-11-20T11:40:44.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.214+0000 7f70e2b00700 1 -- 172.21.15.23:0/1806764152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70dc060a80 msgr2=0x7f70dc09d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:44.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.214+0000 7f70e2b00700 1 --2- 172.21.15.23:0/1806764152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70dc060a80 0x7f70dc09d9e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f70dc068370 tx=0x7f70d0006ef0).stop 2020-11-20T11:40:44.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.214+0000 7f70e2b00700 1 -- 172.21.15.23:0/1806764152 shutdown_connections 2020-11-20T11:40:44.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.214+0000 7f70e2b00700 1 --2- 172.21.15.23:0/1806764152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70dc060a80 0x7f70dc09d9e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:44.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.214+0000 7f70e2b00700 1 --2- 172.21.15.23:0/1806764152 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f70dc09ded0 0x7f70dc0678b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:44.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.214+0000 7f70e2b00700 1 -- 172.21.15.23:0/1806764152 >> 172.21.15.23:0/1806764152 conn(0x7f70dc066730 msgr2=0x7f70dc0601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:40:44.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.214+0000 7f70e2b00700 1 -- 172.21.15.23:0/1806764152 shutdown_connections 2020-11-20T11:40:44.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.214+0000 7f70e2b00700 1 -- 172.21.15.23:0/1806764152 wait complete. 2020-11-20T11:40:44.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.215+0000 7f70e2b00700 1 Processor -- start 2020-11-20T11:40:44.225 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.215+0000 7f70e2b00700 1 -- start start 2020-11-20T11:40:44.226 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.216+0000 7f70e2b00700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f70dc060a80 0x7f70dc1339e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:44.226 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.216+0000 7f70e2b00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70dc067da0 0x7f70dc1341c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:44.226 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.216+0000 7f70e2b00700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f70dc09ded0 0x7f70dc14b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:44.226 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.216+0000 7f70e2b00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70dc13ec60 con 0x7f70dc067da0 2020-11-20T11:40:44.227 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.216+0000 7f70e2b00700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70dc13ef00 con 0x7f70dc09ded0 2020-11-20T11:40:44.227 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.216+0000 7f70e2b00700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70dc133540 con 0x7f70dc060a80 2020-11-20T11:40:44.227 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.216+0000 7f70dbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70dc067da0 0x7f70dc1341c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:44.227 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.216+0000 7f70dbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70dc067da0 0x7f70dc1341c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44326/0 (socket says 172.21.15.23:44326) 2020-11-20T11:40:44.227 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.216+0000 7f70dbfff700 1 -- 172.21.15.23:0/4251583036 learned_addr learned my addr 172.21.15.23:0/4251583036 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:40:44.228 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.216+0000 7f70e109d700 1 --2- 172.21.15.23:0/4251583036 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f70dc09ded0 0x7f70dc14b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:44.228 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.216+0000 7f70e089c700 1 --2- 172.21.15.23:0/4251583036 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f70dc060a80 0x7f70dc1339e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:44.228 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.217+0000 7f70dbfff700 1 -- 172.21.15.23:0/4251583036 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f70dc060a80 msgr2=0x7f70dc1339e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:44.228 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.217+0000 7f70dbfff700 1 --2- 172.21.15.23:0/4251583036 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f70dc060a80 0x7f70dc1339e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:44.228 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.217+0000 7f70dbfff700 1 -- 172.21.15.23:0/4251583036 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f70dc09ded0 msgr2=0x7f70dc14b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:44.229 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.217+0000 7f70dbfff700 1 --2- 172.21.15.23:0/4251583036 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f70dc09ded0 0x7f70dc14b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:44.229 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.217+0000 7f70dbfff700 1 -- 172.21.15.23:0/4251583036 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70dc13dc70 con 0x7f70dc067da0 2020-11-20T11:40:44.229 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.217+0000 7f70dbfff700 1 --2- 172.21.15.23:0/4251583036 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70dc067da0 0x7f70dc1341c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f70c800cb20 tx=0x7f70c8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:44.229 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.217+0000 7f70d8ff9700 1 -- 172.21.15.23:0/4251583036 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c800ae10 con 0x7f70dc067da0 2020-11-20T11:40:44.230 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.217+0000 7f70d8ff9700 1 -- 172.21.15.23:0/4251583036 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70c8007740 con 0x7f70dc067da0 2020-11-20T11:40:44.230 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.217+0000 7f70e2b00700 1 -- 172.21.15.23:0/4251583036 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70dc13e750 con 0x7f70dc067da0 2020-11-20T11:40:44.230 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.217+0000 7f70d8ff9700 1 -- 172.21.15.23:0/4251583036 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c8007ac0 con 0x7f70dc067da0 2020-11-20T11:40:44.230 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.217+0000 7f70e2b00700 1 -- 172.21.15.23:0/4251583036 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70dc136250 con 0x7f70dc067da0 2020-11-20T11:40:44.230 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.218+0000 7f70e2b00700 1 -- 172.21.15.23:0/4251583036 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70dc13ead0 con 0x7f70dc067da0 2020-11-20T11:40:44.231 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.218+0000 7f70d8ff9700 1 -- 172.21.15.23:0/4251583036 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f70c801b460 con 0x7f70dc067da0 2020-11-20T11:40:44.231 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.219+0000 7f70d8ff9700 1 --2- 172.21.15.23:0/4251583036 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f70b4056d60 0x7f70b4059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:44.231 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.219+0000 7f70d8ff9700 1 -- 172.21.15.23:0/4251583036 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f70c8070120 con 0x7f70dc067da0 2020-11-20T11:40:44.231 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.219+0000 7f70e089c700 1 --2- 172.21.15.23:0/4251583036 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f70b4056d60 0x7f70b4059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:44.231 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.220+0000 7f70e089c700 1 --2- 172.21.15.23:0/4251583036 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f70b4056d60 0x7f70b4059220 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f70d000c010 tx=0x7f70d0016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:44.232 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.223+0000 7f70d8ff9700 1 -- 172.21.15.23:0/4251583036 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f70c80453b0 con 0x7f70dc067da0 2020-11-20T11:40:44.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.471+0000 7f70e2b00700 1 -- 172.21.15.23:0/4251583036 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f70dc0a6310 con 0x7f70dc067da0 2020-11-20T11:40:44.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.471+0000 7f70d8ff9700 1 -- 172.21.15.23:0/4251583036 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7f70c80489d0 con 0x7f70dc067da0 2020-11-20T11:40:44.475 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:40:44.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.473+0000 7f70e2b00700 1 -- 172.21.15.23:0/4251583036 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f70b4056d60 msgr2=0x7f70b4059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:44.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.473+0000 7f70e2b00700 1 --2- 172.21.15.23:0/4251583036 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f70b4056d60 0x7f70b4059220 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f70d000c010 tx=0x7f70d0016000).stop 2020-11-20T11:40:44.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.473+0000 7f70e2b00700 1 -- 172.21.15.23:0/4251583036 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70dc067da0 msgr2=0x7f70dc1341c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:44.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.473+0000 7f70e2b00700 1 --2- 172.21.15.23:0/4251583036 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70dc067da0 0x7f70dc1341c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f70c800cb20 tx=0x7f70c8008040).stop 2020-11-20T11:40:44.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.473+0000 7f70e2b00700 1 -- 172.21.15.23:0/4251583036 shutdown_connections 2020-11-20T11:40:44.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.473+0000 7f70e2b00700 1 --2- 172.21.15.23:0/4251583036 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f70dc09ded0 0x7f70dc14b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:44.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.473+0000 7f70e2b00700 1 --2- 172.21.15.23:0/4251583036 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f70b4056d60 0x7f70b4059220 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:44.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.473+0000 7f70e2b00700 1 --2- 172.21.15.23:0/4251583036 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f70dc060a80 0x7f70dc1339e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:44.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.473+0000 7f70e2b00700 1 --2- 172.21.15.23:0/4251583036 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70dc067da0 0x7f70dc1341c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:44.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.473+0000 7f70e2b00700 1 -- 172.21.15.23:0/4251583036 >> 172.21.15.23:0/4251583036 conn(0x7f70dc066730 msgr2=0x7f70dc1325d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:40:44.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.473+0000 7f70e2b00700 1 -- 172.21.15.23:0/4251583036 shutdown_connections 2020-11-20T11:40:44.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.473+0000 7f70e2b00700 1 -- 172.21.15.23:0/4251583036 wait complete. 2020-11-20T11:40:44.488 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:40:44.489 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2020-11-20T11:40:44.489 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:40:44.619 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.616+0000 7f5be1d80700 1 Processor -- start 2020-11-20T11:40:44.619 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.616+0000 7f5be1d80700 1 -- start start 2020-11-20T11:40:44.622 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.616+0000 7f5be1d80700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bdc0ace70 0x7f5bdc13a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:44.622 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.616+0000 7f5be1d80700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5bdc0ad2f0 0x7f5bdc13ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:44.623 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.616+0000 7f5be1d80700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bdc0ad860 con 0x7f5bdc0a9040 2020-11-20T11:40:44.623 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.616+0000 7f5be1d80700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bdc0aa8a0 con 0x7f5bdc0ad2f0 2020-11-20T11:40:44.623 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.616+0000 7f5be1d80700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc0aaa90 con 0x7f5bdc0ace70 2020-11-20T11:40:44.623 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.616+0000 7f5be0d7e700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f5bdc0a9040 0x7f5bdc0aaf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58582/0 (socket says 172.21.15.23:58582) 2020-11-20T11:40:44.624 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.616+0000 7f5be0d7e700 1 -- 172.21.15.23:0/4107629588 learned_addr learned my addr 172.21.15.23:0/4107629588 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:40:44.624 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.617+0000 7f5bdbfff700 1 --2- 172.21.15.23:0/4107629588 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bdc0ace70 0x7f5bdc13a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:44.624 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.617+0000 7f5be157f700 1 --2- 172.21.15.23:0/4107629588 >> v2:172.21.15.155:3300/0 conn(0x7f5bdc0ad2f0 0x7f5bdc13ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:44.624 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.617+0000 7f5bdb7fe700 1 -- 172.21.15.23:0/4107629588 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3109530251 0 0) 0x7f5bdc0ad860 con 0x7f5bdc0a9040 2020-11-20T11:40:44.624 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.617+0000 7f5bdb7fe700 1 -- 172.21.15.23:0/4107629588 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bc0003280 con 0x7f5bdc0a9040 2020-11-20T11:40:44.625 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.617+0000 7f5bdb7fe700 1 -- 172.21.15.23:0/4107629588 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 394684185 0 0) 0x7f5bc0003280 con 0x7f5bdc0a9040 2020-11-20T11:40:44.625 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.617+0000 7f5bdb7fe700 1 -- 172.21.15.23:0/4107629588 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bdc0ace70 msgr2=0x7f5bdc13a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:40:44.625 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.617+0000 7f5bdb7fe700 1 --2- 172.21.15.23:0/4107629588 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bdc0ace70 0x7f5bdc13a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:44.625 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.617+0000 7f5bdb7fe700 1 -- 172.21.15.23:0/4107629588 >> v2:172.21.15.155:3300/0 conn(0x7f5bdc0ad2f0 msgr2=0x7f5bdc13ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:40:44.625 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.617+0000 7f5bdb7fe700 1 --2- 172.21.15.23:0/4107629588 >> v2:172.21.15.155:3300/0 conn(0x7f5bdc0ad2f0 0x7f5bdc13ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:44.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.617+0000 7f5bdb7fe700 1 -- 172.21.15.23:0/4107629588 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bdc0a9420 con 0x7f5bdc0a9040 2020-11-20T11:40:44.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.617+0000 7f5bdb7fe700 1 -- 172.21.15.23:0/4107629588 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f5bd00016a0 con 0x7f5bdc0a9040 2020-11-20T11:40:44.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.617+0000 7f5bdb7fe700 1 -- 172.21.15.23:0/4107629588 >> v1:172.21.15.23:6789/0 conn(0x7f5bdc0a9040 legacy=0x7f5bdc0aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:44.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.617+0000 7f5bdb7fe700 1 --2- 172.21.15.23:0/4107629588 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bdc0ace70 0x7f5bc0003770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:44.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.617+0000 7f5bdb7fe700 1 --2- 172.21.15.23:0/4107629588 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bc0003c80 0x7f5bc0004520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:44.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.617+0000 7f5bdb7fe700 1 --2- 172.21.15.23:0/4107629588 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bc0004a10 0x7f5bc0004e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:44.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.617+0000 7f5bdb7fe700 1 -- 172.21.15.23:0/4107629588 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc0aaa90 con 0x7f5bc0003c80 2020-11-20T11:40:44.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.617+0000 7f5bdb7fe700 1 -- 172.21.15.23:0/4107629588 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc0004210 con 0x7f5bc0004a10 2020-11-20T11:40:44.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.617+0000 7f5bdb7fe700 1 -- 172.21.15.23:0/4107629588 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc000b770 con 0x7f5bdc0ace70 2020-11-20T11:40:44.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.617+0000 7f5be0d7e700 1 --2- 172.21.15.23:0/4107629588 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bdc0ace70 0x7f5bc0003770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:44.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.618+0000 7f5be157f700 1 --2- 172.21.15.23:0/4107629588 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bc0004a10 0x7f5bc0004e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:44.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.618+0000 7f5bdbfff700 1 --2- 172.21.15.23:0/4107629588 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bc0003c80 0x7f5bc0004520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:44.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.618+0000 7f5be0d7e700 1 -- 172.21.15.23:0/4107629588 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bc0004a10 msgr2=0x7f5bc0004e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:40:44.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.618+0000 7f5be0d7e700 1 --2- 172.21.15.23:0/4107629588 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bc0004a10 0x7f5bc0004e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:44.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.618+0000 7f5be0d7e700 1 -- 172.21.15.23:0/4107629588 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bc0003c80 msgr2=0x7f5bc0004520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:40:44.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.618+0000 7f5be0d7e700 1 --2- 172.21.15.23:0/4107629588 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bc0003c80 0x7f5bc0004520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:44.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.618+0000 7f5be0d7e700 1 -- 172.21.15.23:0/4107629588 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc0003280 con 0x7f5bdc0ace70 2020-11-20T11:40:44.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.618+0000 7f5be0d7e700 1 --2- 172.21.15.23:0/4107629588 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bdc0ace70 0x7f5bc0003770 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f5bd000c440 tx=0x7f5bd0007870).ready entity=mon.2 client_cookie=4aa3e13f6fa7438 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:44.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.618+0000 7f5bdb7fe700 1 -- 172.21.15.23:0/4107629588 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd00078e0 con 0x7f5bdc0ace70 2020-11-20T11:40:44.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.618+0000 7f5bdb7fe700 1 -- 172.21.15.23:0/4107629588 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bd00070d0 con 0x7f5bdc0ace70 2020-11-20T11:40:44.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.618+0000 7f5bdb7fe700 1 -- 172.21.15.23:0/4107629588 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd0005500 con 0x7f5bdc0ace70 2020-11-20T11:40:44.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.619+0000 7f5be1d80700 1 -- 172.21.15.23:0/4107629588 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bdc0ace70 msgr2=0x7f5bc0003770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:44.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.619+0000 7f5be1d80700 1 --2- 172.21.15.23:0/4107629588 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bdc0ace70 0x7f5bc0003770 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f5bd000c440 tx=0x7f5bd0007870).stop 2020-11-20T11:40:44.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.619+0000 7f5be157f700 1 -- 172.21.15.23:0/4107629588 reap_dead start 2020-11-20T11:40:44.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.619+0000 7f5be1d80700 1 -- 172.21.15.23:0/4107629588 shutdown_connections 2020-11-20T11:40:44.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.619+0000 7f5be1d80700 1 -- 172.21.15.23:0/4107629588 >> 172.21.15.23:0/4107629588 conn(0x7f5bdc0683b0 msgr2=0x7f5bdc0b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:40:44.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.619+0000 7f5be1d80700 1 -- 172.21.15.23:0/4107629588 shutdown_connections 2020-11-20T11:40:44.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.619+0000 7f5be1d80700 1 -- 172.21.15.23:0/4107629588 wait complete. 2020-11-20T11:40:44.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.620+0000 7f5be1d80700 1 Processor -- start 2020-11-20T11:40:44.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.620+0000 7f5be1d80700 1 -- start start 2020-11-20T11:40:44.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.621+0000 7f5be1d80700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bdc0ace70 0x7f5bdc13d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:44.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.621+0000 7f5be1d80700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bdc09f960 0x7f5bdc09ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:44.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.621+0000 7f5be1d80700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bdc0a1c10 0x7f5bdc09ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:44.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.621+0000 7f5be1d80700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc149c40 con 0x7f5bdc0a1c10 2020-11-20T11:40:44.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.621+0000 7f5be1d80700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc09fe70 con 0x7f5bdc09f960 2020-11-20T11:40:44.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.621+0000 7f5be1d80700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc0a0110 con 0x7f5bdc0ace70 2020-11-20T11:40:44.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.621+0000 7f5be157f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bdc0a1c10 0x7f5bdc09ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:44.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.621+0000 7f5be157f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bdc0a1c10 0x7f5bdc09ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44348/0 (socket says 172.21.15.23:44348) 2020-11-20T11:40:44.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.621+0000 7f5be157f700 1 -- 172.21.15.23:0/1112482414 learned_addr learned my addr 172.21.15.23:0/1112482414 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:40:44.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.621+0000 7f5bdbfff700 1 --2- 172.21.15.23:0/1112482414 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bdc09f960 0x7f5bdc09ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:44.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.621+0000 7f5be0d7e700 1 --2- 172.21.15.23:0/1112482414 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bdc0ace70 0x7f5bdc13d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:44.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.621+0000 7f5be157f700 1 -- 172.21.15.23:0/1112482414 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bdc0ace70 msgr2=0x7f5bdc13d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:44.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.621+0000 7f5be157f700 1 --2- 172.21.15.23:0/1112482414 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bdc0ace70 0x7f5bdc13d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:44.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.621+0000 7f5be157f700 1 -- 172.21.15.23:0/1112482414 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bdc09f960 msgr2=0x7f5bdc09ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:44.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.621+0000 7f5be157f700 1 --2- 172.21.15.23:0/1112482414 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bdc09f960 0x7f5bdc09ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:44.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.621+0000 7f5be157f700 1 -- 172.21.15.23:0/1112482414 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bdc13e3c0 con 0x7f5bdc0a1c10 2020-11-20T11:40:44.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.621+0000 7f5be0d7e700 1 --2- 172.21.15.23:0/1112482414 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bdc0ace70 0x7f5bdc13d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:40:44.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.621+0000 7f5bdbfff700 1 --2- 172.21.15.23:0/1112482414 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bdc09f960 0x7f5bdc09ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:40:44.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.621+0000 7f5be157f700 1 --2- 172.21.15.23:0/1112482414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bdc0a1c10 0x7f5bdc09ef20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f5bd4009160 tx=0x7f5bd4009220).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:44.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.621+0000 7f5bd8ff9700 1 -- 172.21.15.23:0/1112482414 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd400faf0 con 0x7f5bdc0a1c10 2020-11-20T11:40:44.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.622+0000 7f5bd8ff9700 1 -- 172.21.15.23:0/1112482414 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bd400fc50 con 0x7f5bdc0a1c10 2020-11-20T11:40:44.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.622+0000 7f5bd8ff9700 1 -- 172.21.15.23:0/1112482414 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4014440 con 0x7f5bdc0a1c10 2020-11-20T11:40:44.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.622+0000 7f5be1d80700 1 -- 172.21.15.23:0/1112482414 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bdc149fc0 con 0x7f5bdc0a1c10 2020-11-20T11:40:44.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.622+0000 7f5be1d80700 1 -- 172.21.15.23:0/1112482414 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bdc14a4e0 con 0x7f5bdc0a1c10 2020-11-20T11:40:44.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.622+0000 7f5be1d80700 1 --2- 172.21.15.23:0/1112482414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bdc14a7c0 0x7f5bdc14d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:44.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.622+0000 7f5be1d80700 1 -- 172.21.15.23:0/1112482414 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc05cf20 con 0x7f5bdc14a7c0 2020-11-20T11:40:44.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.623+0000 7f5be0d7e700 1 --2- 172.21.15.23:0/1112482414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bdc14a7c0 0x7f5bdc14d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:44.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.623+0000 7f5be0d7e700 1 -- 172.21.15.23:0/1112482414 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5bdc0a02d0 con 0x7f5bdc14a7c0 2020-11-20T11:40:44.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.623+0000 7f5be0d7e700 1 --2- 172.21.15.23:0/1112482414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bdc14a7c0 0x7f5bdc14d1a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f5bd000c440 tx=0x7f5bd0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:44.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.623+0000 7f5bd8ff9700 1 -- 172.21.15.23:0/1112482414 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5bd4014830 con 0x7f5bdc0a1c10 2020-11-20T11:40:44.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.623+0000 7f5bd8ff9700 1 --2- 172.21.15.23:0/1112482414 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5bb4056d60 0x7f5bb4059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:44.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.623+0000 7f5bd8ff9700 1 -- 172.21.15.23:0/1112482414 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f5bd4070960 con 0x7f5bdc0a1c10 2020-11-20T11:40:44.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.624+0000 7f5bd8ff9700 1 -- 172.21.15.23:0/1112482414 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd0001350 con 0x7f5bdc14a7c0 2020-11-20T11:40:44.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.624+0000 7f5bd8ff9700 1 -- 172.21.15.23:0/1112482414 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f5bd00014b0 con 0x7f5bdc14a7c0 2020-11-20T11:40:44.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.624+0000 7f5bd8ff9700 1 -- 172.21.15.23:0/1112482414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bdc14a7c0 msgr2=0x7f5bdc14d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:44.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.624+0000 7f5bd8ff9700 1 --2- 172.21.15.23:0/1112482414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bdc14a7c0 0x7f5bdc14d1a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f5bd000c440 tx=0x7f5bd0015000).stop 2020-11-20T11:40:44.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.624+0000 7f5bd8ff9700 1 -- 172.21.15.23:0/1112482414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bdc14a7c0 msgr2=0x7f5bdc14d1a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:40:44.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.624+0000 7f5bd8ff9700 1 --2- 172.21.15.23:0/1112482414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bdc14a7c0 0x7f5bdc14d1a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:44.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.624+0000 7f5bdbfff700 1 --2- 172.21.15.23:0/1112482414 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5bb4056d60 0x7f5bb4059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:44.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.624+0000 7f5bdbfff700 1 --2- 172.21.15.23:0/1112482414 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5bb4056d60 0x7f5bb4059220 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f5bc8006d40 tx=0x7f5bc8008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:44.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.630+0000 7f5be1d80700 1 --2- 172.21.15.23:0/1112482414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bdc14d690 0x7f5bdc14da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:44.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.630+0000 7f5be1d80700 1 -- 172.21.15.23:0/1112482414 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bdc09ffb0 con 0x7f5bdc14d690 2020-11-20T11:40:44.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.631+0000 7f5be0d7e700 1 --2- 172.21.15.23:0/1112482414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bdc14d690 0x7f5bdc14da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:44.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.631+0000 7f5be0d7e700 1 -- 172.21.15.23:0/1112482414 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5bdc14aba0 con 0x7f5bdc14d690 2020-11-20T11:40:44.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.631+0000 7f5be0d7e700 1 --2- 172.21.15.23:0/1112482414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bdc14d690 0x7f5bdc14da70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f5bdc14a1e0 tx=0x7f5bd001e000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:44.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.631+0000 7f5bd8ff9700 1 -- 172.21.15.23:0/1112482414 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd0014dd0 con 0x7f5bdc14d690 2020-11-20T11:40:44.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.632+0000 7f5bd8ff9700 1 -- 172.21.15.23:0/1112482414 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4786 (secure 0 0 0) 0x7f5bd0005da0 con 0x7f5bdc14d690 2020-11-20T11:40:44.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.632+0000 7f5bd8ff9700 1 -- 172.21.15.23:0/1112482414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bdc14d690 msgr2=0x7f5bdc14da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:44.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.632+0000 7f5bd8ff9700 1 --2- 172.21.15.23:0/1112482414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bdc14d690 0x7f5bdc14da70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f5bdc14a1e0 tx=0x7f5bd001e000).stop 2020-11-20T11:40:44.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.632+0000 7f5bd8ff9700 1 -- 172.21.15.23:0/1112482414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bdc14d690 msgr2=0x7f5bdc14da70 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:40:44.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.632+0000 7f5bd8ff9700 1 --2- 172.21.15.23:0/1112482414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bdc14d690 0x7f5bdc14da70 secure :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0x7f5bdc14a1e0 tx=0x7f5bd001e000).stop 2020-11-20T11:40:44.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.632+0000 7f5be1d80700 1 -- 172.21.15.23:0/1112482414 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5bb4056d60 msgr2=0x7f5bb4059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:44.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.632+0000 7f5be1d80700 1 --2- 172.21.15.23:0/1112482414 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5bb4056d60 0x7f5bb4059220 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f5bc8006d40 tx=0x7f5bc8008040).stop 2020-11-20T11:40:44.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.632+0000 7f5be1d80700 1 -- 172.21.15.23:0/1112482414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bdc0a1c10 msgr2=0x7f5bdc09ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:44.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.632+0000 7f5be1d80700 1 --2- 172.21.15.23:0/1112482414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bdc0a1c10 0x7f5bdc09ef20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f5bd4009160 tx=0x7f5bd4009220).stop 2020-11-20T11:40:44.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.632+0000 7f5be157f700 1 -- 172.21.15.23:0/1112482414 reap_dead start 2020-11-20T11:40:44.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.632+0000 7f5be157f700 1 -- 172.21.15.23:0/1112482414 reap_dead start 2020-11-20T11:40:44.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.632+0000 7f5be1d80700 1 -- 172.21.15.23:0/1112482414 shutdown_connections 2020-11-20T11:40:44.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.632+0000 7f5be1d80700 1 -- 172.21.15.23:0/1112482414 >> 172.21.15.23:0/1112482414 conn(0x7f5bdc0683b0 msgr2=0x7f5bdc0aae50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:40:44.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.632+0000 7f5be1d80700 1 -- 172.21.15.23:0/1112482414 shutdown_connections 2020-11-20T11:40:44.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.632+0000 7f5be1d80700 1 -- 172.21.15.23:0/1112482414 wait complete. 2020-11-20T11:40:44.650 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:40:44.650 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:40:44.650 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:40:44.651 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:40:44.651 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 82, 2020-11-20T11:40:44.651 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:40:44.651 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:40:44.652 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:40:44.652 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:40:44.652 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:44.652 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 11, 2020-11-20T11:40:44.652 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:40:44.653 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:40:44.653 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:40:44.653 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:40:44.653 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:40:44.653 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:40:44.654 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:40:44.654 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:40:44.654 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:40:44.654 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:40:44.654 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:40:44.655 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:44.655 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:40:44.655 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:40:44.655 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:40:44.655 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:40:44.656 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:40:44.656 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:40:44.656 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:40:44.656 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:40:44.657 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:40:44.657 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:40:44.657 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:44.657 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:44.657 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:40:44.658 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:40:44.658 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:40:44.658 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:40:44.658 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:40:44.658 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:40:44.659 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:40:44.659 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:40:44.659 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:40:44.659 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:40:44.659 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:40:44.660 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:40:44.660 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:40:44.660 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:40:44.660 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:40:44.660 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:40:44.661 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:40:44.661 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:40:44.661 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:40:44.661 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:40:44.661 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:40:44.662 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:40:44.662 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:40:44.662 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:44.662 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:40:44.662 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:44.663 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:40:44.663 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:44.663 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:40:44.663 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:40:44.663 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:40:44.664 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:40:44.664 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:44.664 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:40:44.664 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:40:44.664 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:44.665 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:44.665 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:44.665 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:40:44.665 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:40:44.665 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:44.666 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:44.666 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:44.666 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:44.666 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:40:44.666 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:40:44.667 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:40:44.667 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:40:44.667 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:40:44.667 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:44.667 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:44.668 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:40:44.668 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:40:44.668 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:40:44.668 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:40:44.668 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:44.669 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:40:44.669 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:40:44.669 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:44.669 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:44.669 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:44.670 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:40:44.670 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:40:44.670 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:44.670 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:44.670 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:44.671 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:44.671 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:40:44.671 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:40:44.671 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:40:44.671 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:40:44.672 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:40:44.672 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:44.672 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:44.672 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:40:44.672 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:40:44.673 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:40:44.673 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:40:44.673 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:44.673 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:40:44.673 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:40:44.674 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:44.674 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:44.674 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:44.674 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:40:44.675 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:40:44.675 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:44.675 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:44.675 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:44.675 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:44.676 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:40:44.676 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:40:44.676 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:40:44.676 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:40:44.676 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:40:44.677 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:44.677 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:44.677 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:44.677 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:40:44.678 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:40:44.678 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:44.678 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:40:44.678 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:40:44.678 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:40:44.679 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:44.679 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:44.679 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:40:44.679 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:44.680 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:40:44.680 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:40:44.680 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:40:44.680 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:44.681 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:44.681 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:40:44.681 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:44.681 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:40:44.681 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:40:44.682 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 8 2020-11-20T11:40:44.682 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:44.682 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:44.682 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:40:44.682 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:44.683 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:40:44.683 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:40:44.683 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:40:44.683 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:44.683 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:44.684 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:40:44.684 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:44.684 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:40:44.684 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:40:44.684 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:40:44.685 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:44.685 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:44.685 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:44.685 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:40:44.685 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:40:44.687 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:40:44.812 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.809+0000 7f195079c700 1 Processor -- start 2020-11-20T11:40:44.812 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.809+0000 7f195079c700 1 -- start start 2020-11-20T11:40:44.814 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.810+0000 7f195079c700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f194813aeb0 0x7f19480a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:44.814 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.810+0000 7f195079c700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19481388b0 con 0x7f19480aae90 2020-11-20T11:40:44.814 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.810+0000 7f195079c700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f194813e600 con 0x7f19480aaab0 2020-11-20T11:40:44.814 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.810+0000 7f195079c700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f194813e8a0 con 0x7f194813aeb0 2020-11-20T11:40:44.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.810+0000 7f194dd37700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f19480aae90 0x7f19481381a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58608/0 (socket says 172.21.15.23:58608) 2020-11-20T11:40:44.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.810+0000 7f194dd37700 1 -- 172.21.15.23:0/2398486142 learned_addr learned my addr 172.21.15.23:0/2398486142 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:40:44.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.810+0000 7f194ed39700 1 --2- 172.21.15.23:0/2398486142 >> v2:172.21.15.155:3300/0 conn(0x7f194813aeb0 0x7f19480a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:44.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.810+0000 7f194d536700 1 -- 172.21.15.23:0/2398486142 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1755705514 0 0) 0x7f19481388b0 con 0x7f19480aae90 2020-11-20T11:40:44.816 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.810+0000 7f194d536700 1 -- 172.21.15.23:0/2398486142 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1930003280 con 0x7f19480aae90 2020-11-20T11:40:44.816 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.811+0000 7f194ed39700 1 -- 172.21.15.23:0/2398486142 >> v1:172.21.15.155:6789/0 conn(0x7f19480aaab0 legacy=0x7f194813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:44.816 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.811+0000 7f194d536700 1 -- 172.21.15.23:0/2398486142 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1072782150 0 0) 0x7f194813e600 con 0x7f19480aaab0 2020-11-20T11:40:44.816 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.811+0000 7f194ed39700 1 -- 172.21.15.23:0/2398486142 >> v1:172.21.15.23:6789/0 conn(0x7f19480aae90 legacy=0x7f19481381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:44.816 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.811+0000 7f194ed39700 1 -- 172.21.15.23:0/2398486142 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19480ab6d0 con 0x7f194813aeb0 2020-11-20T11:40:44.817 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.811+0000 7f194ed39700 1 --2- 172.21.15.23:0/2398486142 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f194813aeb0 0x7f19480a8c60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f1940009ce0 tx=0x7f1940009220).ready entity=mon.1 client_cookie=5996330dc18c4e8a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:44.817 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.811+0000 7f194d536700 1 -- 172.21.15.23:0/2398486142 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194000ba40 con 0x7f194813aeb0 2020-11-20T11:40:44.817 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.811+0000 7f194d536700 1 -- 172.21.15.23:0/2398486142 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f194000bba0 con 0x7f194813aeb0 2020-11-20T11:40:44.817 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.811+0000 7f194d536700 1 -- 172.21.15.23:0/2398486142 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1940011400 con 0x7f194813aeb0 2020-11-20T11:40:44.821 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.811+0000 7f195079c700 1 -- 172.21.15.23:0/2398486142 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f194813aeb0 msgr2=0x7f19480a8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:44.821 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.811+0000 7f195079c700 1 --2- 172.21.15.23:0/2398486142 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f194813aeb0 0x7f19480a8c60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f1940009ce0 tx=0x7f1940009220).stop 2020-11-20T11:40:44.821 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.812+0000 7f195079c700 1 -- 172.21.15.23:0/2398486142 shutdown_connections 2020-11-20T11:40:44.821 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.812+0000 7f195079c700 1 --2- 172.21.15.23:0/2398486142 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f194813aeb0 0x7f19480a8c60 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:44.822 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.812+0000 7f195079c700 1 -- 172.21.15.23:0/2398486142 >> 172.21.15.23:0/2398486142 conn(0x7f19480678e0 msgr2=0x7f1948067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:40:44.822 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.812+0000 7f195079c700 1 -- 172.21.15.23:0/2398486142 shutdown_connections 2020-11-20T11:40:44.822 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.812+0000 7f195079c700 1 -- 172.21.15.23:0/2398486142 wait complete. 2020-11-20T11:40:44.822 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.813+0000 7f195079c700 1 Processor -- start 2020-11-20T11:40:44.822 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.813+0000 7f195079c700 1 -- start start 2020-11-20T11:40:44.823 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.813+0000 7f195079c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19480aaab0 0x7f1948135b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:44.823 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.813+0000 7f195079c700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19480aae90 0x7f194813d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:44.823 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.813+0000 7f195079c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19480a6170 0x7f194813d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:44.823 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.813+0000 7f195079c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194813da90 con 0x7f19480a6170 2020-11-20T11:40:44.823 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.813+0000 7f195079c700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194813c6e0 con 0x7f19480aaab0 2020-11-20T11:40:44.824 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.813+0000 7f195079c700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f194813c980 con 0x7f19480aae90 2020-11-20T11:40:44.824 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.813+0000 7f194ed39700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19480a6170 0x7f194813d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:44.824 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.813+0000 7f194e538700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19480aaab0 0x7f1948135b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:44.824 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.813+0000 7f194ed39700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19480a6170 0x7f194813d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44366/0 (socket says 172.21.15.23:44366) 2020-11-20T11:40:44.825 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.813+0000 7f194ed39700 1 -- 172.21.15.23:0/3950116321 learned_addr learned my addr 172.21.15.23:0/3950116321 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:40:44.825 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.813+0000 7f194dd37700 1 --2- 172.21.15.23:0/3950116321 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19480aae90 0x7f194813d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:44.825 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.814+0000 7f194ed39700 1 -- 172.21.15.23:0/3950116321 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19480aae90 msgr2=0x7f194813d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:44.825 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.814+0000 7f194ed39700 1 --2- 172.21.15.23:0/3950116321 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19480aae90 0x7f194813d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:44.825 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.814+0000 7f194ed39700 1 -- 172.21.15.23:0/3950116321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19480aaab0 msgr2=0x7f1948135b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:44.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.814+0000 7f194ed39700 1 --2- 172.21.15.23:0/3950116321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19480aaab0 0x7f1948135b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:44.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.814+0000 7f194ed39700 1 -- 172.21.15.23:0/3950116321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1948133270 con 0x7f19480a6170 2020-11-20T11:40:44.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.814+0000 7f194ed39700 1 --2- 172.21.15.23:0/3950116321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19480a6170 0x7f194813d5a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f1940006f90 tx=0x7f1940016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:44.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.814+0000 7f19367fc700 1 -- 172.21.15.23:0/3950116321 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1940010a50 con 0x7f19480a6170 2020-11-20T11:40:44.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.814+0000 7f195079c700 1 -- 172.21.15.23:0/3950116321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f194813c2b0 con 0x7f19480a6170 2020-11-20T11:40:44.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.814+0000 7f19367fc700 1 -- 172.21.15.23:0/3950116321 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1940010bb0 con 0x7f19480a6170 2020-11-20T11:40:44.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.814+0000 7f19367fc700 1 -- 172.21.15.23:0/3950116321 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194001c470 con 0x7f19480a6170 2020-11-20T11:40:44.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.815+0000 7f195079c700 1 -- 172.21.15.23:0/3950116321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1948131690 con 0x7f19480a6170 2020-11-20T11:40:44.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.815+0000 7f195079c700 1 --2- 172.21.15.23:0/3950116321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f194813aeb0 0x7f1948131880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:44.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.815+0000 7f195079c700 1 -- 172.21.15.23:0/3950116321 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1948133460 con 0x7f194813aeb0 2020-11-20T11:40:44.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.816+0000 7f19367fc700 1 -- 172.21.15.23:0/3950116321 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1940017070 con 0x7f19480a6170 2020-11-20T11:40:44.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.816+0000 7f194e538700 1 --2- 172.21.15.23:0/3950116321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f194813aeb0 0x7f1948131880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:44.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.816+0000 7f19367fc700 1 --2- 172.21.15.23:0/3950116321 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1920056d60 0x7f1920059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:44.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.816+0000 7f194e538700 1 -- 172.21.15.23:0/3950116321 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f194805c7d0 con 0x7f194813aeb0 2020-11-20T11:40:44.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.816+0000 7f19367fc700 1 -- 172.21.15.23:0/3950116321 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f1940072510 con 0x7f19480a6170 2020-11-20T11:40:44.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.816+0000 7f194e538700 1 --2- 172.21.15.23:0/3950116321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f194813aeb0 0x7f1948131880 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f1944002910 tx=0x7f1944007a40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:44.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.816+0000 7f194dd37700 1 --2- 172.21.15.23:0/3950116321 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1920056d60 0x7f1920059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:44.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.816+0000 7f19367fc700 1 -- 172.21.15.23:0/3950116321 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194400f960 con 0x7f194813aeb0 2020-11-20T11:40:44.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.816+0000 7f19367fc700 1 -- 172.21.15.23:0/3950116321 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f194400fac0 con 0x7f194813aeb0 2020-11-20T11:40:44.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.816+0000 7f194dd37700 1 --2- 172.21.15.23:0/3950116321 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1920056d60 0x7f1920059220 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f19480ae7f0 tx=0x7f1938009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:44.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.816+0000 7f19367fc700 1 -- 172.21.15.23:0/3950116321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f194813aeb0 msgr2=0x7f1948131880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:44.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.816+0000 7f19367fc700 1 --2- 172.21.15.23:0/3950116321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f194813aeb0 0x7f1948131880 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f1944002910 tx=0x7f1944007a40).stop 2020-11-20T11:40:44.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.816+0000 7f19367fc700 1 -- 172.21.15.23:0/3950116321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f194813aeb0 msgr2=0x7f1948131880 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:40:44.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.816+0000 7f19367fc700 1 --2- 172.21.15.23:0/3950116321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f194813aeb0 0x7f1948131880 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:44.832 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.824+0000 7f195079c700 1 --2- 172.21.15.23:0/3950116321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19480601b0 0x7f1948131d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:44.832 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.824+0000 7f195079c700 1 -- 172.21.15.23:0/3950116321 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1948044710 con 0x7f19480601b0 2020-11-20T11:40:44.833 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.824+0000 7f194e538700 1 --2- 172.21.15.23:0/3950116321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19480601b0 0x7f1948131d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:44.833 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.825+0000 7f194e538700 1 -- 172.21.15.23:0/3950116321 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f194805c280 con 0x7f19480601b0 2020-11-20T11:40:44.833 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.825+0000 7f194e538700 1 --2- 172.21.15.23:0/3950116321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19480601b0 0x7f1948131d70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f1944016000 tx=0x7f1944019000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:44.833 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.825+0000 7f19367fc700 1 -- 172.21.15.23:0/3950116321 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1944014e00 con 0x7f19480601b0 2020-11-20T11:40:44.833 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.825+0000 7f19367fc700 1 -- 172.21.15.23:0/3950116321 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4274 (secure 0 0 0) 0x7f194400f750 con 0x7f19480601b0 2020-11-20T11:40:44.834 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.825+0000 7f19367fc700 1 -- 172.21.15.23:0/3950116321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19480601b0 msgr2=0x7f1948131d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:44.834 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.825+0000 7f19367fc700 1 --2- 172.21.15.23:0/3950116321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19480601b0 0x7f1948131d70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f1944016000 tx=0x7f1944019000).stop 2020-11-20T11:40:44.834 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.825+0000 7f19367fc700 1 -- 172.21.15.23:0/3950116321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19480601b0 msgr2=0x7f1948131d70 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:40:44.834 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.825+0000 7f19367fc700 1 --2- 172.21.15.23:0/3950116321 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f19480601b0 0x7f1948131d70 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0x7f1944016000 tx=0x7f1944019000).stop 2020-11-20T11:40:44.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.826+0000 7f195079c700 1 -- 172.21.15.23:0/3950116321 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1920056d60 msgr2=0x7f1920059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:44.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.826+0000 7f195079c700 1 --2- 172.21.15.23:0/3950116321 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1920056d60 0x7f1920059220 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f19480ae7f0 tx=0x7f1938009040).stop 2020-11-20T11:40:44.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.826+0000 7f195079c700 1 -- 172.21.15.23:0/3950116321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19480a6170 msgr2=0x7f194813d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:44.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.826+0000 7f195079c700 1 --2- 172.21.15.23:0/3950116321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19480a6170 0x7f194813d5a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f1940006f90 tx=0x7f1940016000).stop 2020-11-20T11:40:44.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.826+0000 7f194ed39700 1 -- 172.21.15.23:0/3950116321 reap_dead start 2020-11-20T11:40:44.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.826+0000 7f195079c700 1 -- 172.21.15.23:0/3950116321 shutdown_connections 2020-11-20T11:40:44.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.826+0000 7f195079c700 1 --2- 172.21.15.23:0/3950116321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f19480a6170 0x7f194813d5a0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:44.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.826+0000 7f195079c700 1 -- 172.21.15.23:0/3950116321 >> 172.21.15.23:0/3950116321 conn(0x7f19480678e0 msgr2=0x7f1948068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:40:44.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.826+0000 7f195079c700 1 -- 172.21.15.23:0/3950116321 shutdown_connections 2020-11-20T11:40:44.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.826+0000 7f195079c700 1 -- 172.21.15.23:0/3950116321 wait complete. 2020-11-20T11:40:44.837 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:40:44.837 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:40:44.837 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:40:44.837 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:40:44.838 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 82, 2020-11-20T11:40:44.838 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:40:44.838 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:40:44.838 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:40:44.838 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:40:44.839 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:44.839 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 11, 2020-11-20T11:40:44.839 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:40:44.839 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:40:44.839 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:40:44.840 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:40:44.840 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:40:44.840 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:40:44.840 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:40:44.840 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:40:44.841 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:40:44.841 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:40:44.841 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:40:44.841 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:44.841 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:40:44.842 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:40:44.842 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:40:44.842 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:40:44.842 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:40:44.842 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:40:44.843 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:40:44.843 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:40:44.843 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:40:44.843 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:40:44.843 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:44.844 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:44.844 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:40:44.844 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:40:44.844 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:40:44.844 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:40:44.845 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:40:44.845 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:40:44.845 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:40:44.845 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:40:44.846 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:40:44.846 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:40:44.846 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:40:44.846 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:40:44.846 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:40:44.847 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:40:44.847 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:40:44.847 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:40:44.847 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:40:44.847 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:40:44.848 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:40:44.848 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:40:44.848 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:40:44.848 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:40:44.848 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:40:44.849 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:44.849 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:40:44.849 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:44.849 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:40:44.849 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:44.850 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:40:44.850 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:40:44.850 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:40:44.850 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:40:44.850 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:44.851 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:40:44.851 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:40:44.851 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:44.851 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:44.851 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:44.852 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:40:44.852 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:40:44.852 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:44.852 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:44.852 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:44.853 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:44.853 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:40:44.853 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:40:44.853 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:40:44.853 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:40:44.854 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:40:44.854 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:44.854 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:44.854 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:40:44.854 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:40:44.855 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:40:44.855 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:40:44.855 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:44.855 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:40:44.855 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:40:44.856 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:44.856 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:44.856 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:44.856 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:40:44.856 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:40:44.857 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:44.857 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:44.857 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:44.857 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:44.858 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:40:44.858 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:40:44.858 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:40:44.858 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:40:44.858 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:40:44.859 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:44.859 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:44.859 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:40:44.859 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:40:44.859 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:40:44.860 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:40:44.860 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:44.860 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:40:44.860 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:40:44.860 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:44.861 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:44.861 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:44.861 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:40:44.861 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:40:44.861 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:44.862 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:44.862 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:44.862 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:44.862 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:40:44.862 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:40:44.863 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:40:44.863 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:40:44.863 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:40:44.863 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:44.863 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:44.864 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:44.864 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:40:44.864 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:40:44.864 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:44.864 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:40:44.865 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:40:44.865 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:40:44.865 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:44.865 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:44.865 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:40:44.866 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:44.866 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:40:44.866 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:40:44.866 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:40:44.867 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:44.867 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:44.867 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:44.867 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:40:44.867 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:40:44.868 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:40:44.994 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.991+0000 7fbceb59e700 1 Processor -- start 2020-11-20T11:40:44.994 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.991+0000 7fbceb59e700 1 -- start start 2020-11-20T11:40:44.996 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.991+0000 7fbceb59e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbcec061380 0x7fbcec0a8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:44.997 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.992+0000 7fbceb59e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fbcec0a8d70 0x7fbcec13e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:44.997 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.992+0000 7fbceb59e700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcec0a5000 con 0x7fbcec13edc0 2020-11-20T11:40:44.997 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.992+0000 7fbceb59e700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcec0a5d30 con 0x7fbcec0a8d70 2020-11-20T11:40:44.997 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.992+0000 7fbceb59e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcec0a5fd0 con 0x7fbcec061380 2020-11-20T11:40:44.998 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.992+0000 7fbcead9d700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fbcec13edc0 0x7fbcec0a6400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58628/0 (socket says 172.21.15.23:58628) 2020-11-20T11:40:44.998 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.992+0000 7fbcead9d700 1 -- 172.21.15.23:0/1600506210 learned_addr learned my addr 172.21.15.23:0/1600506210 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:40:44.998 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.992+0000 7fbce9d9b700 1 --2- 172.21.15.23:0/1600506210 >> v2:172.21.15.155:3300/0 conn(0x7fbcec0a8d70 0x7fbcec13e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:44.998 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.992+0000 7fbcea59c700 1 --2- 172.21.15.23:0/1600506210 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbcec061380 0x7fbcec0a8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:44.998 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.992+0000 7fbce959a700 1 -- 172.21.15.23:0/1600506210 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744606535 0 0) 0x7fbcec0a5000 con 0x7fbcec13edc0 2020-11-20T11:40:44.999 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.992+0000 7fbce959a700 1 -- 172.21.15.23:0/1600506210 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcd0003280 con 0x7fbcec13edc0 2020-11-20T11:40:44.999 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.992+0000 7fbcea59c700 1 -- 172.21.15.23:0/1600506210 >> v2:172.21.15.155:3300/0 conn(0x7fbcec0a8d70 msgr2=0x7fbcec13e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:40:44.999 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.992+0000 7fbcea59c700 1 --2- 172.21.15.23:0/1600506210 >> v2:172.21.15.155:3300/0 conn(0x7fbcec0a8d70 0x7fbcec13e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:44.999 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.992+0000 7fbcea59c700 1 -- 172.21.15.23:0/1600506210 >> v1:172.21.15.23:6789/0 conn(0x7fbcec13edc0 legacy=0x7fbcec0a6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:45.000 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.992+0000 7fbcea59c700 1 -- 172.21.15.23:0/1600506210 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcec0a8140 con 0x7fbcec061380 2020-11-20T11:40:45.000 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.992+0000 7fbce9d9b700 1 --2- 172.21.15.23:0/1600506210 >> v2:172.21.15.155:3300/0 conn(0x7fbcec0a8d70 0x7fbcec13e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:40:45.000 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.993+0000 7fbcea59c700 1 --2- 172.21.15.23:0/1600506210 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbcec061380 0x7fbcec0a8880 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fbcdc006fd0 tx=0x7fbcdc006e30).ready entity=mon.2 client_cookie=f65b5496f2b3877c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:45.000 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.993+0000 7fbce959a700 1 -- 172.21.15.23:0/1600506210 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc009630 con 0x7fbcec061380 2020-11-20T11:40:45.000 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.993+0000 7fbce959a700 1 -- 172.21.15.23:0/1600506210 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcdc000e90 con 0x7fbcec061380 2020-11-20T11:40:45.001 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.993+0000 7fbce959a700 1 -- 172.21.15.23:0/1600506210 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc0099a0 con 0x7fbcec061380 2020-11-20T11:40:45.001 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.993+0000 7fbceb59e700 1 -- 172.21.15.23:0/1600506210 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbcec061380 msgr2=0x7fbcec0a8880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:45.001 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.993+0000 7fbceb59e700 1 --2- 172.21.15.23:0/1600506210 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbcec061380 0x7fbcec0a8880 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fbcdc006fd0 tx=0x7fbcdc006e30).stop 2020-11-20T11:40:45.001 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.993+0000 7fbceb59e700 1 -- 172.21.15.23:0/1600506210 shutdown_connections 2020-11-20T11:40:45.001 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.993+0000 7fbceb59e700 1 --2- 172.21.15.23:0/1600506210 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbcec061380 0x7fbcec0a8880 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:45.002 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.993+0000 7fbceb59e700 1 --2- 172.21.15.23:0/1600506210 >> v2:172.21.15.155:3300/0 conn(0x7fbcec0a8d70 0x7fbcec13e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:45.002 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.993+0000 7fbceb59e700 1 -- 172.21.15.23:0/1600506210 >> 172.21.15.23:0/1600506210 conn(0x7fbcec0ae0a0 msgr2=0x7fbcec0aa0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:40:45.002 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.993+0000 7fbceb59e700 1 -- 172.21.15.23:0/1600506210 shutdown_connections 2020-11-20T11:40:45.002 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.993+0000 7fbceb59e700 1 -- 172.21.15.23:0/1600506210 wait complete. 2020-11-20T11:40:45.005 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.995+0000 7fbceb59e700 1 Processor -- start 2020-11-20T11:40:45.005 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.995+0000 7fbceb59e700 1 -- start start 2020-11-20T11:40:45.006 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.995+0000 7fbceb59e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbcec061380 0x7fbcec0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:45.006 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.995+0000 7fbceb59e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcec0a8d70 0x7fbcec14a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:45.006 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.995+0000 7fbceb59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcec13edc0 0x7fbcec0a3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:45.006 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.996+0000 7fbceb59e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcec14b2c0 con 0x7fbcec13edc0 2020-11-20T11:40:45.006 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.996+0000 7fbceb59e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcec14dc20 con 0x7fbcec0a8d70 2020-11-20T11:40:45.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.996+0000 7fbceb59e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcec14dec0 con 0x7fbcec061380 2020-11-20T11:40:45.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.996+0000 7fbcead9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcec13edc0 0x7fbcec0a3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:45.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.996+0000 7fbcead9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcec13edc0 0x7fbcec0a3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44384/0 (socket says 172.21.15.23:44384) 2020-11-20T11:40:45.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.996+0000 7fbcead9d700 1 -- 172.21.15.23:0/462536949 learned_addr learned my addr 172.21.15.23:0/462536949 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:40:45.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.996+0000 7fbcea59c700 1 --2- 172.21.15.23:0/462536949 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbcec061380 0x7fbcec0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:45.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.996+0000 7fbce9d9b700 1 --2- 172.21.15.23:0/462536949 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcec0a8d70 0x7fbcec14a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:45.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.996+0000 7fbcead9d700 1 -- 172.21.15.23:0/462536949 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbcec061380 msgr2=0x7fbcec0a6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:45.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.996+0000 7fbcead9d700 1 --2- 172.21.15.23:0/462536949 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbcec061380 0x7fbcec0a6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:45.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.996+0000 7fbcead9d700 1 -- 172.21.15.23:0/462536949 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcec0a8d70 msgr2=0x7fbcec14a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:45.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.996+0000 7fbcead9d700 1 --2- 172.21.15.23:0/462536949 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbcec0a8d70 0x7fbcec14a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:45.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.996+0000 7fbcead9d700 1 -- 172.21.15.23:0/462536949 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcec14e160 con 0x7fbcec13edc0 2020-11-20T11:40:45.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.996+0000 7fbcead9d700 1 --2- 172.21.15.23:0/462536949 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcec13edc0 0x7fbcec0a3d30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fbce4001460 tx=0x7fbce4007e60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:45.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.996+0000 7fbcda7fc700 1 -- 172.21.15.23:0/462536949 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce400ce10 con 0x7fbcec13edc0 2020-11-20T11:40:45.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.996+0000 7fbcda7fc700 1 -- 172.21.15.23:0/462536949 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce4005bc0 con 0x7fbcec13edc0 2020-11-20T11:40:45.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.996+0000 7fbceb59e700 1 -- 172.21.15.23:0/462536949 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcec14d910 con 0x7fbcec13edc0 2020-11-20T11:40:45.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.996+0000 7fbcda7fc700 1 -- 172.21.15.23:0/462536949 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce4007110 con 0x7fbcec13edc0 2020-11-20T11:40:45.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.996+0000 7fbceb59e700 1 -- 172.21.15.23:0/462536949 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcec0a6f30 con 0x7fbcec13edc0 2020-11-20T11:40:45.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.998+0000 7fbceb59e700 1 --2- 172.21.15.23:0/462536949 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbcec061980 0x7fbcec066ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:45.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.998+0000 7fbceb59e700 1 -- 172.21.15.23:0/462536949 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcec14a9f0 con 0x7fbcec061980 2020-11-20T11:40:45.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.998+0000 7fbcea59c700 1 --2- 172.21.15.23:0/462536949 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbcec061980 0x7fbcec066ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:45.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.998+0000 7fbcda7fc700 1 -- 172.21.15.23:0/462536949 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbce4010030 con 0x7fbcec13edc0 2020-11-20T11:40:45.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.998+0000 7fbcea59c700 1 -- 172.21.15.23:0/462536949 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbcec0a8a10 con 0x7fbcec061980 2020-11-20T11:40:45.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.998+0000 7fbcda7fc700 1 --2- 172.21.15.23:0/462536949 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbcc0056d60 0x7fbcc0059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:45.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.998+0000 7fbcda7fc700 1 -- 172.21.15.23:0/462536949 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fbce406fe70 con 0x7fbcec13edc0 2020-11-20T11:40:45.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.998+0000 7fbcea59c700 1 --2- 172.21.15.23:0/462536949 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbcec061980 0x7fbcec066ef0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fbcdc006fd0 tx=0x7fbcdc015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:45.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.998+0000 7fbce9d9b700 1 --2- 172.21.15.23:0/462536949 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbcc0056d60 0x7fbcc0059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:45.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.998+0000 7fbcda7fc700 1 -- 172.21.15.23:0/462536949 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc0117e0 con 0x7fbcec061980 2020-11-20T11:40:45.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.999+0000 7fbcda7fc700 1 -- 172.21.15.23:0/462536949 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fbcdc011940 con 0x7fbcec061980 2020-11-20T11:40:45.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.999+0000 7fbcda7fc700 1 -- 172.21.15.23:0/462536949 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbcec061980 msgr2=0x7fbcec066ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:45.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.999+0000 7fbcda7fc700 1 --2- 172.21.15.23:0/462536949 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbcec061980 0x7fbcec066ef0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fbcdc006fd0 tx=0x7fbcdc015000).stop 2020-11-20T11:40:45.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.999+0000 7fbcda7fc700 1 -- 172.21.15.23:0/462536949 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbcec061980 msgr2=0x7fbcec066ef0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:40:45.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.999+0000 7fbcda7fc700 1 --2- 172.21.15.23:0/462536949 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbcec061980 0x7fbcec066ef0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:45.014 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:44.999+0000 7fbce9d9b700 1 --2- 172.21.15.23:0/462536949 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbcc0056d60 0x7fbcc0059220 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fbcec0a6a10 tx=0x7fbce0006e30).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:45.015 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.006+0000 7fbceb59e700 1 --2- 172.21.15.23:0/462536949 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbcec14e520 0x7fbcec0a4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:45.016 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.006+0000 7fbceb59e700 1 -- 172.21.15.23:0/462536949 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcec069c70 con 0x7fbcec14e520 2020-11-20T11:40:45.016 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.006+0000 7fbcea59c700 1 --2- 172.21.15.23:0/462536949 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbcec14e520 0x7fbcec0a4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:45.016 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.007+0000 7fbcea59c700 1 -- 172.21.15.23:0/462536949 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbcec14e2f0 con 0x7fbcec14e520 2020-11-20T11:40:45.016 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.007+0000 7fbcea59c700 1 --2- 172.21.15.23:0/462536949 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbcec14e520 0x7fbcec0a4220 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fbcdc0093c0 tx=0x7fbcdc021070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:45.016 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.007+0000 7fbcda7fc700 1 -- 172.21.15.23:0/462536949 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc02d760 con 0x7fbcec14e520 2020-11-20T11:40:45.017 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.007+0000 7fbcda7fc700 1 -- 172.21.15.23:0/462536949 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4614 (secure 0 0 0) 0x7fbcdc0115d0 con 0x7fbcec14e520 2020-11-20T11:40:45.017 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.007+0000 7fbcda7fc700 1 -- 172.21.15.23:0/462536949 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbcec14e520 msgr2=0x7fbcec0a4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:45.017 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.007+0000 7fbcda7fc700 1 --2- 172.21.15.23:0/462536949 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbcec14e520 0x7fbcec0a4220 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fbcdc0093c0 tx=0x7fbcdc021070).stop 2020-11-20T11:40:45.017 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.007+0000 7fbcda7fc700 1 -- 172.21.15.23:0/462536949 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbcec14e520 msgr2=0x7fbcec0a4220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:40:45.018 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.007+0000 7fbcda7fc700 1 --2- 172.21.15.23:0/462536949 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbcec14e520 0x7fbcec0a4220 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:45.018 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.007+0000 7fbceb59e700 1 -- 172.21.15.23:0/462536949 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbcc0056d60 msgr2=0x7fbcc0059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:45.018 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.007+0000 7fbceb59e700 1 --2- 172.21.15.23:0/462536949 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbcc0056d60 0x7fbcc0059220 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fbcec0a6a10 tx=0x7fbce0006e30).stop 2020-11-20T11:40:45.018 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.007+0000 7fbceb59e700 1 -- 172.21.15.23:0/462536949 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcec13edc0 msgr2=0x7fbcec0a3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:45.018 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.007+0000 7fbceb59e700 1 --2- 172.21.15.23:0/462536949 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbcec13edc0 0x7fbcec0a3d30 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fbce4001460 tx=0x7fbce4007e60).stop 2020-11-20T11:40:45.019 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.007+0000 7fbcead9d700 1 -- 172.21.15.23:0/462536949 reap_dead start 2020-11-20T11:40:45.019 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.007+0000 7fbcead9d700 1 -- 172.21.15.23:0/462536949 reap_dead start 2020-11-20T11:40:45.019 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.007+0000 7fbceb59e700 1 -- 172.21.15.23:0/462536949 shutdown_connections 2020-11-20T11:40:45.019 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.007+0000 7fbceb59e700 1 -- 172.21.15.23:0/462536949 >> 172.21.15.23:0/462536949 conn(0x7fbcec0ae0a0 msgr2=0x7fbcec13f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:40:45.019 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.008+0000 7fbceb59e700 1 -- 172.21.15.23:0/462536949 shutdown_connections 2020-11-20T11:40:45.020 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.008+0000 7fbceb59e700 1 -- 172.21.15.23:0/462536949 wait complete. 2020-11-20T11:40:45.020 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:40:45.020 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:40:45.020 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:40:45.020 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:40:45.021 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 82, 2020-11-20T11:40:45.021 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:40:45.021 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:40:45.021 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:40:45.021 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:40:45.022 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:45.022 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 11, 2020-11-20T11:40:45.022 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:40:45.022 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:40:45.022 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:40:45.023 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:40:45.023 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:40:45.023 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:40:45.023 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:40:45.023 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:40:45.024 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:40:45.024 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:40:45.024 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:40:45.024 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:45.024 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:40:45.025 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:40:45.025 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:40:45.025 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:40:45.025 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:40:45.026 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:40:45.026 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:40:45.026 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:40:45.026 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:40:45.026 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:40:45.026 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:45.027 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:45.027 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:40:45.027 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:40:45.027 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:40:45.027 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:40:45.028 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:40:45.028 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:40:45.028 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:40:45.028 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:40:45.028 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:40:45.029 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:40:45.029 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:40:45.029 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:40:45.029 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:40:45.029 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:40:45.030 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:40:45.030 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:40:45.030 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:40:45.030 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:40:45.030 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:40:45.031 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:40:45.031 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:40:45.031 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:40:45.031 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:40:45.031 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:45.032 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:40:45.032 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:45.032 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:40:45.032 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:45.032 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:40:45.033 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:40:45.033 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:40:45.033 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:40:45.033 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:45.033 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:40:45.034 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:40:45.034 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:45.034 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:45.034 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:45.035 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:40:45.035 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:40:45.035 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:45.035 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:45.035 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:45.035 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:45.036 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:40:45.036 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:40:45.036 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:40:45.036 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:40:45.036 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:40:45.037 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:45.037 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:45.037 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:40:45.037 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:40:45.037 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:40:45.038 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:40:45.038 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:45.038 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:40:45.038 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:40:45.038 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:45.038 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:45.039 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:45.039 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:40:45.039 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:40:45.039 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:45.039 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:45.040 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:45.040 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:45.040 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:40:45.040 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:40:45.040 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:40:45.041 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:40:45.041 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:40:45.041 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:45.041 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:45.041 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:40:45.042 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:40:45.042 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:40:45.042 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:40:45.042 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:45.042 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:40:45.043 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:40:45.043 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:45.043 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:45.043 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:45.043 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:40:45.044 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:40:45.044 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:40:45.044 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:45.044 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:45.044 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:45.045 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:40:45.045 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:40:45.045 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:40:45.045 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:40:45.045 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:40:45.046 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:45.046 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:45.046 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:45.046 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:40:45.046 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:40:45.047 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:45.047 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:40:45.047 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:40:45.047 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:40:45.047 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:45.048 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:45.048 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:40:45.048 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:45.048 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:40:45.048 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:40:45.049 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:40:45.049 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:45.049 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:45.049 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:40:45.049 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:45.050 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:40:45.050 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:40:45.050 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:40:45.050 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:45.050 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:40:45.050 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:40:45.051 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:40:45.051 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:40:45.051 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:40:45.051 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:40:45.051 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:40:45.052 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:40:45.052 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:40:45.052 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:40:45.052 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:40:45.053 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2020-11-20T11:40:45.054 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2020-11-20T11:40:45.054 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2020-11-20T11:40:45.054 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2020-11-20T11:40:45.054 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2020-11-20T11:40:45.055 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T11:40:45.065 INFO:tasks.ceph.mon.c:Stopped 2020-11-20T11:40:45.066 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2020-11-20T11:40:45.066 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:40:45.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.196+0000 7f69dffff700 1 Processor -- start 2020-11-20T11:40:45.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.196+0000 7f69dffff700 1 -- start start 2020-11-20T11:40:45.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.197+0000 7f69dffff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f69e0067a10 0x7f69e0067e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:45.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.197+0000 7f69dffff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69e00aaab0 con 0x7f69e00a2080 2020-11-20T11:40:45.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.197+0000 7f69dffff700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69e00aad50 con 0x7f69e00a2460 2020-11-20T11:40:45.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.197+0000 7f69dffff700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e013aa50 con 0x7f69e0067a10 2020-11-20T11:40:45.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.197+0000 7f69df7fe700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f69e0067a10 msgr2=0x7f69e0067e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:40:45.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.197+0000 7f69deffd700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f69e00a2080 0x7f69e00aef10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58644/0 (socket says 172.21.15.23:58644) 2020-11-20T11:40:45.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.197+0000 7f69deffd700 1 -- 172.21.15.23:0/1547810219 learned_addr learned my addr 172.21.15.23:0/1547810219 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:40:45.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.197+0000 7f69df7fe700 1 --2- 172.21.15.23:0/1547810219 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f69e0067a10 0x7f69e0067e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:40:45.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.197+0000 7f69de7fc700 1 -- 172.21.15.23:0/1547810219 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 727081334 0 0) 0x7f69e00aaab0 con 0x7f69e00a2080 2020-11-20T11:40:45.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.197+0000 7f69de7fc700 1 -- 172.21.15.23:0/1547810219 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69c0003280 con 0x7f69e00a2080 2020-11-20T11:40:45.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.197+0000 7f69de7fc700 1 -- 172.21.15.23:0/1547810219 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 244493214 0 0) 0x7f69e00aad50 con 0x7f69e00a2460 2020-11-20T11:40:45.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.197+0000 7f69de7fc700 1 -- 172.21.15.23:0/1547810219 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69e00aaab0 con 0x7f69e00a2460 2020-11-20T11:40:45.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.197+0000 7f69de7fc700 1 -- 172.21.15.23:0/1547810219 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 151677078 0 0) 0x7f69c0003280 con 0x7f69e00a2080 2020-11-20T11:40:45.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.197+0000 7f69de7fc700 1 -- 172.21.15.23:0/1547810219 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f69e0067a10 msgr2=0x7f69e0067e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:40:45.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.197+0000 7f69de7fc700 1 --2- 172.21.15.23:0/1547810219 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f69e0067a10 0x7f69e0067e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:40:45.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.197+0000 7f69de7fc700 1 -- 172.21.15.23:0/1547810219 >> v1:172.21.15.155:6789/0 conn(0x7f69e00a2460 legacy=0x7f69e0137df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:45.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.197+0000 7f69de7fc700 1 -- 172.21.15.23:0/1547810219 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e013b920 con 0x7f69e00a2080 2020-11-20T11:40:45.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.197+0000 7f69de7fc700 1 -- 172.21.15.23:0/1547810219 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f69d00016a0 con 0x7f69e00a2080 2020-11-20T11:40:45.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.197+0000 7f69de7fc700 1 -- 172.21.15.23:0/1547810219 >> v1:172.21.15.23:6789/0 conn(0x7f69e00a2080 legacy=0x7f69e00aef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:45.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.198+0000 7f69de7fc700 1 --2- 172.21.15.23:0/1547810219 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f69e0067a10 0x7f69c0003930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:45.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.198+0000 7f69de7fc700 1 --2- 172.21.15.23:0/1547810219 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69c0003e40 0x7f69c0004520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:45.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.198+0000 7f69de7fc700 1 --2- 172.21.15.23:0/1547810219 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69c0004a10 0x7f69c0004e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:45.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.198+0000 7f69de7fc700 1 -- 172.21.15.23:0/1547810219 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c0005380 con 0x7f69c0003e40 2020-11-20T11:40:45.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.198+0000 7f69de7fc700 1 -- 172.21.15.23:0/1547810219 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c000b770 con 0x7f69c0004a10 2020-11-20T11:40:45.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.198+0000 7f69de7fc700 1 -- 172.21.15.23:0/1547810219 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69c000b8e0 con 0x7f69e0067a10 2020-11-20T11:40:45.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.198+0000 7f69deffd700 1 -- 172.21.15.23:0/1547810219 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f69e0067a10 msgr2=0x7f69c0003930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:40:45.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.198+0000 7f69deffd700 1 --2- 172.21.15.23:0/1547810219 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f69e0067a10 0x7f69c0003930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:40:45.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.198+0000 7f69d7fff700 1 --2- 172.21.15.23:0/1547810219 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69c0003e40 0x7f69c0004520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:45.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.198+0000 7f69df7fe700 1 --2- 172.21.15.23:0/1547810219 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69c0004a10 0x7f69c0004e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:45.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.198+0000 7f69d7fff700 1 -- 172.21.15.23:0/1547810219 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f69e0067a10 msgr2=0x7f69c0003930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:40:45.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.198+0000 7f69d7fff700 1 --2- 172.21.15.23:0/1547810219 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f69e0067a10 0x7f69c0003930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:40:45.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.198+0000 7f69d7fff700 1 -- 172.21.15.23:0/1547810219 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69c0004a10 msgr2=0x7f69c0004e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:40:45.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.198+0000 7f69d7fff700 1 --2- 172.21.15.23:0/1547810219 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69c0004a10 0x7f69c0004e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:45.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.198+0000 7f69d7fff700 1 -- 172.21.15.23:0/1547810219 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c0003280 con 0x7f69c0003e40 2020-11-20T11:40:45.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.198+0000 7f69d7fff700 1 --2- 172.21.15.23:0/1547810219 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69c0003e40 0x7f69c0004520 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f69c8001b30 tx=0x7f69c8007b90).ready entity=mon.0 client_cookie=632d1c1ec06f6a4a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:45.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.198+0000 7f69de7fc700 1 -- 172.21.15.23:0/1547810219 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c800c860 con 0x7f69c0003e40 2020-11-20T11:40:45.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.198+0000 7f69de7fc700 1 -- 172.21.15.23:0/1547810219 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69c800c9c0 con 0x7f69c0003e40 2020-11-20T11:40:45.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.198+0000 7f69de7fc700 1 -- 172.21.15.23:0/1547810219 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c800cd80 con 0x7f69c0003e40 2020-11-20T11:40:45.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.199+0000 7f69dffff700 1 -- 172.21.15.23:0/1547810219 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69c0003e40 msgr2=0x7f69c0004520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:45.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.199+0000 7f69dffff700 1 --2- 172.21.15.23:0/1547810219 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69c0003e40 0x7f69c0004520 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f69c8001b30 tx=0x7f69c8007b90).stop 2020-11-20T11:40:45.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.199+0000 7f69df7fe700 1 -- 172.21.15.23:0/1547810219 reap_dead start 2020-11-20T11:40:45.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.199+0000 7f69dffff700 1 -- 172.21.15.23:0/1547810219 shutdown_connections 2020-11-20T11:40:45.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.199+0000 7f69dffff700 1 -- 172.21.15.23:0/1547810219 >> 172.21.15.23:0/1547810219 conn(0x7f69e0063980 msgr2=0x7f69e0063d90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:40:45.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.199+0000 7f69dffff700 1 -- 172.21.15.23:0/1547810219 shutdown_connections 2020-11-20T11:40:45.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.199+0000 7f69dffff700 1 -- 172.21.15.23:0/1547810219 wait complete. 2020-11-20T11:40:45.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.200+0000 7f69dffff700 1 Processor -- start 2020-11-20T11:40:45.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.200+0000 7f69dffff700 1 -- start start 2020-11-20T11:40:45.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.201+0000 7f69dffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69e0140b70 0x7f69e00a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:45.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.201+0000 7f69dffff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69e0140f50 0x7f69e00a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:45.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.201+0000 7f69dffff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f69e00a45a0 0x7f69e014d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:45.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.201+0000 7f69dffff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e00a3620 con 0x7f69e0140b70 2020-11-20T11:40:45.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.201+0000 7f69dffff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e013c160 con 0x7f69e0140f50 2020-11-20T11:40:45.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.201+0000 7f69dffff700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e013c400 con 0x7f69e00a45a0 2020-11-20T11:40:45.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.201+0000 7f69deffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69e0140b70 0x7f69e00a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:45.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.201+0000 7f69deffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69e0140b70 0x7f69e00a4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44406/0 (socket says 172.21.15.23:44406) 2020-11-20T11:40:45.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.201+0000 7f69deffd700 1 -- 172.21.15.23:0/1434963002 learned_addr learned my addr 172.21.15.23:0/1434963002 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:40:45.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.201+0000 7f69df7fe700 1 -- 172.21.15.23:0/1434963002 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f69e00a45a0 msgr2=0x7f69e014d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:40:45.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.201+0000 7f69d7fff700 1 --2- 172.21.15.23:0/1434963002 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69e0140f50 0x7f69e00a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:45.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.201+0000 7f69df7fe700 1 --2- 172.21.15.23:0/1434963002 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f69e00a45a0 0x7f69e014d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:40:45.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.201+0000 7f69deffd700 1 -- 172.21.15.23:0/1434963002 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f69e00a45a0 msgr2=0x7f69e014d510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:40:45.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.201+0000 7f69deffd700 1 --2- 172.21.15.23:0/1434963002 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f69e00a45a0 0x7f69e014d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:40:45.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.201+0000 7f69deffd700 1 -- 172.21.15.23:0/1434963002 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69e0140f50 msgr2=0x7f69e00a57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:45.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.201+0000 7f69deffd700 1 --2- 172.21.15.23:0/1434963002 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69e0140f50 0x7f69e00a57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:45.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.201+0000 7f69deffd700 1 -- 172.21.15.23:0/1434963002 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e013c6a0 con 0x7f69e0140b70 2020-11-20T11:40:45.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.201+0000 7f69d7fff700 1 --2- 172.21.15.23:0/1434963002 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69e0140f50 0x7f69e00a57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:40:45.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.202+0000 7f69deffd700 1 --2- 172.21.15.23:0/1434963002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69e0140b70 0x7f69e00a4ab0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f69d0001360 tx=0x7f69d0000d00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:45.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.202+0000 7f69d6ffd700 1 -- 172.21.15.23:0/1434963002 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d0007e30 con 0x7f69e0140b70 2020-11-20T11:40:45.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.202+0000 7f69d6ffd700 1 -- 172.21.15.23:0/1434963002 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d0007320 con 0x7f69e0140b70 2020-11-20T11:40:45.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.202+0000 7f69d6ffd700 1 -- 172.21.15.23:0/1434963002 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d000a7b0 con 0x7f69e0140b70 2020-11-20T11:40:45.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.202+0000 7f69dffff700 1 -- 172.21.15.23:0/1434963002 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69e013be50 con 0x7f69e0140b70 2020-11-20T11:40:45.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.202+0000 7f69dffff700 1 -- 172.21.15.23:0/1434963002 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e00a4fa0 con 0x7f69e0140b70 2020-11-20T11:40:45.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.203+0000 7f69dffff700 1 -- 172.21.15.23:0/1434963002 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69e0057dc0 con 0x7f69e0140b70 2020-11-20T11:40:45.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.203+0000 7f69d6ffd700 1 -- 172.21.15.23:0/1434963002 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f69d0010030 con 0x7f69e0140b70 2020-11-20T11:40:45.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.204+0000 7f69d6ffd700 1 --2- 172.21.15.23:0/1434963002 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f69bc056d10 0x7f69bc0591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:45.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.204+0000 7f69d6ffd700 1 -- 172.21.15.23:0/1434963002 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f69d006fcd0 con 0x7f69e0140b70 2020-11-20T11:40:45.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.204+0000 7f69d7fff700 1 --2- 172.21.15.23:0/1434963002 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f69bc056d10 0x7f69bc0591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:45.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.204+0000 7f69d7fff700 1 --2- 172.21.15.23:0/1434963002 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f69bc056d10 0x7f69bc0591d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f69e005ec90 tx=0x7f69c8017000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:45.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.207+0000 7f69d6ffd700 1 -- 172.21.15.23:0/1434963002 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f69d0044fc0 con 0x7f69e0140b70 2020-11-20T11:40:45.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.433+0000 7f69dffff700 1 -- 172.21.15.23:0/1434963002 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f69e0044710 con 0x7f69e0140b70 2020-11-20T11:40:45.437 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.434+0000 7f69d6ffd700 1 -- 172.21.15.23:0/1434963002 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7f69d00485e0 con 0x7f69e0140b70 2020-11-20T11:40:45.437 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:40:45.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.435+0000 7f69dffff700 1 -- 172.21.15.23:0/1434963002 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f69bc056d10 msgr2=0x7f69bc0591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:45.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.435+0000 7f69dffff700 1 --2- 172.21.15.23:0/1434963002 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f69bc056d10 0x7f69bc0591d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f69e005ec90 tx=0x7f69c8017000).stop 2020-11-20T11:40:45.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.435+0000 7f69dffff700 1 -- 172.21.15.23:0/1434963002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69e0140b70 msgr2=0x7f69e00a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:45.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.435+0000 7f69dffff700 1 --2- 172.21.15.23:0/1434963002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69e0140b70 0x7f69e00a4ab0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f69d0001360 tx=0x7f69d0000d00).stop 2020-11-20T11:40:45.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.435+0000 7f69dffff700 1 -- 172.21.15.23:0/1434963002 shutdown_connections 2020-11-20T11:40:45.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.435+0000 7f69dffff700 1 --2- 172.21.15.23:0/1434963002 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f69bc056d10 0x7f69bc0591d0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:45.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.435+0000 7f69dffff700 1 --2- 172.21.15.23:0/1434963002 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f69e00a45a0 0x7f69e014d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:40:45.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.435+0000 7f69dffff700 1 --2- 172.21.15.23:0/1434963002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69e0140b70 0x7f69e00a4ab0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:45.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.435+0000 7f69dffff700 1 --2- 172.21.15.23:0/1434963002 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f69e0140f50 0x7f69e00a57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:45.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.435+0000 7f69dffff700 1 -- 172.21.15.23:0/1434963002 >> 172.21.15.23:0/1434963002 conn(0x7f69e0063980 msgr2=0x7f69e0063d90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:40:45.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.435+0000 7f69dffff700 1 -- 172.21.15.23:0/1434963002 shutdown_connections 2020-11-20T11:40:45.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:45.435+0000 7f69dffff700 1 -- 172.21.15.23:0/1434963002 wait complete. 2020-11-20T11:40:48.451 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:40:48.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.575+0000 7f602d500700 1 Processor -- start 2020-11-20T11:40:48.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.575+0000 7f602d500700 1 -- start start 2020-11-20T11:40:48.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.576+0000 7f602d500700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f602813aeb0 0x7f60280a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:48.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.576+0000 7f602d500700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60281388b0 con 0x7f60280aae90 2020-11-20T11:40:48.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.576+0000 7f602d500700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f602813e600 con 0x7f60280aaab0 2020-11-20T11:40:48.584 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.576+0000 7f602d500700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f602813e8a0 con 0x7f602813aeb0 2020-11-20T11:40:48.584 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.576+0000 7f60267fc700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f60280aae90 0x7f60281381a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58674/0 (socket says 172.21.15.23:58674) 2020-11-20T11:40:48.584 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.576+0000 7f60267fc700 1 -- 172.21.15.23:0/3822232378 learned_addr learned my addr 172.21.15.23:0/3822232378 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:40:48.584 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.576+0000 7f60277fe700 1 --2- 172.21.15.23:0/3822232378 >> v2:172.21.15.155:3300/0 conn(0x7f602813aeb0 0x7f60280a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:48.584 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.576+0000 7f6025ffb700 1 -- 172.21.15.23:0/3822232378 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1127429943 0 0) 0x7f60281388b0 con 0x7f60280aae90 2020-11-20T11:40:48.585 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.576+0000 7f6025ffb700 1 -- 172.21.15.23:0/3822232378 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6014003280 con 0x7f60280aae90 2020-11-20T11:40:48.585 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.576+0000 7f6025ffb700 1 -- 172.21.15.23:0/3822232378 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2341090380 0 0) 0x7f6014003280 con 0x7f60280aae90 2020-11-20T11:40:48.585 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.576+0000 7f6025ffb700 1 -- 172.21.15.23:0/3822232378 >> v2:172.21.15.155:3300/0 conn(0x7f602813aeb0 msgr2=0x7f60280a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:40:48.585 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.576+0000 7f6025ffb700 1 --2- 172.21.15.23:0/3822232378 >> v2:172.21.15.155:3300/0 conn(0x7f602813aeb0 0x7f60280a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:48.585 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.576+0000 7f6025ffb700 1 -- 172.21.15.23:0/3822232378 >> v1:172.21.15.155:6789/0 conn(0x7f60280aaab0 legacy=0x7f602813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:48.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.576+0000 7f6025ffb700 1 -- 172.21.15.23:0/3822232378 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60280ab6d0 con 0x7f60280aae90 2020-11-20T11:40:48.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.576+0000 7f6025ffb700 1 -- 172.21.15.23:0/3822232378 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f601c001520 con 0x7f60280aae90 2020-11-20T11:40:48.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.576+0000 7f6025ffb700 1 -- 172.21.15.23:0/3822232378 >> v1:172.21.15.23:6789/0 conn(0x7f60280aae90 legacy=0x7f60281381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:48.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.577+0000 7f6025ffb700 1 --2- 172.21.15.23:0/3822232378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6014003770 0x7f6014003b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:48.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.577+0000 7f6025ffb700 1 --2- 172.21.15.23:0/3822232378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6014004520 0x7f60140049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:48.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.577+0000 7f6025ffb700 1 --2- 172.21.15.23:0/3822232378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6014004e90 0x7f601400b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:48.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.577+0000 7f6025ffb700 1 -- 172.21.15.23:0/3822232378 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602813e8a0 con 0x7f6014004e90 2020-11-20T11:40:48.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.577+0000 7f6025ffb700 1 -- 172.21.15.23:0/3822232378 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60140042a0 con 0x7f6014004520 2020-11-20T11:40:48.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.577+0000 7f6025ffb700 1 -- 172.21.15.23:0/3822232378 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f601400bc60 con 0x7f6014003770 2020-11-20T11:40:48.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.577+0000 7f6026ffd700 1 -- 172.21.15.23:0/3822232378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6014003770 msgr2=0x7f6014003b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:40:48.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.577+0000 7f6026ffd700 1 --2- 172.21.15.23:0/3822232378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6014003770 0x7f6014003b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:40:48.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.577+0000 7f60277fe700 1 --2- 172.21.15.23:0/3822232378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6014004e90 0x7f601400b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:48.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.577+0000 7f60267fc700 1 --2- 172.21.15.23:0/3822232378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6014004520 0x7f60140049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:48.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.577+0000 7f60277fe700 1 -- 172.21.15.23:0/3822232378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6014003770 msgr2=0x7f6014003b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:40:48.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.577+0000 7f60277fe700 1 --2- 172.21.15.23:0/3822232378 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6014003770 0x7f6014003b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:40:48.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.577+0000 7f60277fe700 1 -- 172.21.15.23:0/3822232378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6014004520 msgr2=0x7f60140049a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:40:48.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.577+0000 7f60277fe700 1 --2- 172.21.15.23:0/3822232378 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6014004520 0x7f60140049a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:48.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.577+0000 7f60277fe700 1 -- 172.21.15.23:0/3822232378 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6014003280 con 0x7f6014004e90 2020-11-20T11:40:48.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.577+0000 7f60277fe700 1 -- 172.21.15.23:0/3822232378 reap_dead start 2020-11-20T11:40:48.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.577+0000 7f60277fe700 1 --2- 172.21.15.23:0/3822232378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6014004e90 0x7f601400b770 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f6020009ec0 tx=0x7f6020009220).ready entity=mon.0 client_cookie=89f9f4984a6cd502 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:48.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.577+0000 7f6025ffb700 1 -- 172.21.15.23:0/3822232378 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6020009610 con 0x7f6014004e90 2020-11-20T11:40:48.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.577+0000 7f6025ffb700 1 -- 172.21.15.23:0/3822232378 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f602000ed30 con 0x7f6014004e90 2020-11-20T11:40:48.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.577+0000 7f6025ffb700 1 -- 172.21.15.23:0/3822232378 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6020012600 con 0x7f6014004e90 2020-11-20T11:40:48.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.578+0000 7f602d500700 1 -- 172.21.15.23:0/3822232378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6014004e90 msgr2=0x7f601400b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:48.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.578+0000 7f602d500700 1 --2- 172.21.15.23:0/3822232378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6014004e90 0x7f601400b770 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f6020009ec0 tx=0x7f6020009220).stop 2020-11-20T11:40:48.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.578+0000 7f602d500700 1 -- 172.21.15.23:0/3822232378 shutdown_connections 2020-11-20T11:40:48.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.578+0000 7f602d500700 1 --2- 172.21.15.23:0/3822232378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6014004e90 0x7f601400b770 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:48.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.578+0000 7f602d500700 1 -- 172.21.15.23:0/3822232378 >> 172.21.15.23:0/3822232378 conn(0x7f60280678e0 msgr2=0x7f6028067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:40:48.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.578+0000 7f602d500700 1 -- 172.21.15.23:0/3822232378 shutdown_connections 2020-11-20T11:40:48.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.578+0000 7f602d500700 1 -- 172.21.15.23:0/3822232378 wait complete. 2020-11-20T11:40:48.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.579+0000 7f602d500700 1 Processor -- start 2020-11-20T11:40:48.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.579+0000 7f602d500700 1 -- start start 2020-11-20T11:40:48.594 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.580+0000 7f602d500700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6014004e90 0x7f60280a7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:48.595 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.580+0000 7f602d500700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f602813d510 0x7f60280a64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:48.595 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.580+0000 7f602d500700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60280a6990 0x7f6028133250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:48.595 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.580+0000 7f602d500700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602809ede0 con 0x7f602813d510 2020-11-20T11:40:48.596 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.580+0000 7f602d500700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602813d990 con 0x7f6014004e90 2020-11-20T11:40:48.596 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.580+0000 7f602d500700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f602809efd0 con 0x7f60280a6990 2020-11-20T11:40:48.596 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.580+0000 7f60277fe700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60280a6990 msgr2=0x7f6028133250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:40:48.596 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.580+0000 7f60277fe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60280a6990 0x7f6028133250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:40:48.596 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.580+0000 7f60267fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f602813d510 0x7f60280a64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:48.597 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.580+0000 7f60267fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f602813d510 0x7f60280a64a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44436/0 (socket says 172.21.15.23:44436) 2020-11-20T11:40:48.597 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.580+0000 7f60267fc700 1 -- 172.21.15.23:0/1522290285 learned_addr learned my addr 172.21.15.23:0/1522290285 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:40:48.597 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.580+0000 7f6026ffd700 1 --2- 172.21.15.23:0/1522290285 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6014004e90 0x7f60280a7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:48.597 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.580+0000 7f60267fc700 1 -- 172.21.15.23:0/1522290285 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60280a6990 msgr2=0x7f6028133250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:40:48.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.580+0000 7f60267fc700 1 --2- 172.21.15.23:0/1522290285 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60280a6990 0x7f6028133250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:40:48.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.580+0000 7f60267fc700 1 -- 172.21.15.23:0/1522290285 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6014004e90 msgr2=0x7f60280a7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:48.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.580+0000 7f60267fc700 1 --2- 172.21.15.23:0/1522290285 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6014004e90 0x7f60280a7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:48.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.580+0000 7f60267fc700 1 -- 172.21.15.23:0/1522290285 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f602809f270 con 0x7f602813d510 2020-11-20T11:40:48.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.581+0000 7f60267fc700 1 --2- 172.21.15.23:0/1522290285 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f602813d510 0x7f60280a64a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f601c004b70 tx=0x7f601c007810).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:48.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.581+0000 7f600effd700 1 -- 172.21.15.23:0/1522290285 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601c00f900 con 0x7f602813d510 2020-11-20T11:40:48.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.581+0000 7f600effd700 1 -- 172.21.15.23:0/1522290285 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f601c00fa60 con 0x7f602813d510 2020-11-20T11:40:48.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.581+0000 7f602d500700 1 -- 172.21.15.23:0/1522290285 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f602813dd80 con 0x7f602813d510 2020-11-20T11:40:48.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.581+0000 7f600effd700 1 -- 172.21.15.23:0/1522290285 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601c00fe00 con 0x7f602813d510 2020-11-20T11:40:48.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.581+0000 7f602d500700 1 -- 172.21.15.23:0/1522290285 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6028133740 con 0x7f602813d510 2020-11-20T11:40:48.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.582+0000 7f602d500700 1 -- 172.21.15.23:0/1522290285 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f602805c7d0 con 0x7f602813d510 2020-11-20T11:40:48.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.582+0000 7f600effd700 1 -- 172.21.15.23:0/1522290285 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f601c01b030 con 0x7f602813d510 2020-11-20T11:40:48.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.582+0000 7f600effd700 1 --2- 172.21.15.23:0/1522290285 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6004056d10 0x7f60040591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:48.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.582+0000 7f600effd700 1 -- 172.21.15.23:0/1522290285 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f601c06f7f0 con 0x7f602813d510 2020-11-20T11:40:48.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.583+0000 7f6026ffd700 1 --2- 172.21.15.23:0/1522290285 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6004056d10 0x7f60040591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:48.601 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.583+0000 7f6026ffd700 1 --2- 172.21.15.23:0/1522290285 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6004056d10 0x7f60040591d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f602813a490 tx=0x7f6018009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:48.601 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.587+0000 7f600effd700 1 -- 172.21.15.23:0/1522290285 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f601c044ae0 con 0x7f602813d510 2020-11-20T11:40:48.839 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.835+0000 7f602d500700 1 -- 172.21.15.23:0/1522290285 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f602809f630 con 0x7f602813d510 2020-11-20T11:40:48.840 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.836+0000 7f600effd700 1 -- 172.21.15.23:0/1522290285 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7f601c06de60 con 0x7f602813d510 2020-11-20T11:40:48.841 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:40:48.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.838+0000 7f602d500700 1 -- 172.21.15.23:0/1522290285 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6004056d10 msgr2=0x7f60040591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:48.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.838+0000 7f602d500700 1 --2- 172.21.15.23:0/1522290285 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6004056d10 0x7f60040591d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f602813a490 tx=0x7f6018009040).stop 2020-11-20T11:40:48.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.838+0000 7f602d500700 1 -- 172.21.15.23:0/1522290285 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f602813d510 msgr2=0x7f60280a64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:48.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.838+0000 7f602d500700 1 --2- 172.21.15.23:0/1522290285 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f602813d510 0x7f60280a64a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f601c004b70 tx=0x7f601c007810).stop 2020-11-20T11:40:48.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.838+0000 7f602d500700 1 -- 172.21.15.23:0/1522290285 shutdown_connections 2020-11-20T11:40:48.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.838+0000 7f602d500700 1 --2- 172.21.15.23:0/1522290285 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6004056d10 0x7f60040591d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:48.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.838+0000 7f602d500700 1 --2- 172.21.15.23:0/1522290285 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f60280a6990 0x7f6028133250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:40:48.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.838+0000 7f602d500700 1 --2- 172.21.15.23:0/1522290285 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6014004e90 0x7f60280a7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:48.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.838+0000 7f602d500700 1 --2- 172.21.15.23:0/1522290285 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f602813d510 0x7f60280a64a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:48.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.838+0000 7f602d500700 1 -- 172.21.15.23:0/1522290285 >> 172.21.15.23:0/1522290285 conn(0x7f60280678e0 msgr2=0x7f60281399c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:40:48.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.838+0000 7f602d500700 1 -- 172.21.15.23:0/1522290285 shutdown_connections 2020-11-20T11:40:48.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:48.838+0000 7f602d500700 1 -- 172.21.15.23:0/1522290285 wait complete. 2020-11-20T11:40:51.854 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:40:51.976 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.972+0000 7fb31e92d700 1 Processor -- start 2020-11-20T11:40:51.977 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.972+0000 7fb31e92d700 1 -- start start 2020-11-20T11:40:51.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.973+0000 7fb31e92d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3180a2460 0x7fb318137df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:51.983 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.973+0000 7fb31e92d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb318067a10 0x7fb318067ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:51.983 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.973+0000 7fb31e92d700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3180aaab0 con 0x7fb3180a2080 2020-11-20T11:40:51.983 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.973+0000 7fb31e92d700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb31813aa50 con 0x7fb318067a10 2020-11-20T11:40:51.983 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.973+0000 7fb31e92d700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3180aad50 con 0x7fb3180a2460 2020-11-20T11:40:51.984 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.973+0000 7fb31e12c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb318067a10 0x7fb318067ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:51.984 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.973+0000 7fb31d12a700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3180a2460 msgr2=0x7fb318137df0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:40:51.984 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.973+0000 7fb31d12a700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3180a2460 0x7fb318137df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:40:51.984 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.973+0000 7fb31e12c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb318067a10 0x7fb318067ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44448/0 (socket says 172.21.15.23:44448) 2020-11-20T11:40:51.984 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.973+0000 7fb31e12c700 1 -- 172.21.15.23:0/844539658 learned_addr learned my addr 172.21.15.23:0/844539658 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:40:51.985 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.974+0000 7fb31e12c700 1 -- 172.21.15.23:0/844539658 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3180a2460 msgr2=0x7fb318137df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:40:51.985 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.974+0000 7fb31e12c700 1 --2- 172.21.15.23:0/844539658 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3180a2460 0x7fb318137df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:40:51.985 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.974+0000 7fb31e12c700 1 -- 172.21.15.23:0/844539658 >> v1:172.21.15.23:6789/0 conn(0x7fb3180a2080 legacy=0x7fb3180aef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:51.985 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.974+0000 7fb31e12c700 1 -- 172.21.15.23:0/844539658 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31813b920 con 0x7fb318067a10 2020-11-20T11:40:51.986 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.974+0000 7fb31e12c700 1 --2- 172.21.15.23:0/844539658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb318067a10 0x7fb318067ea0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fb310009d90 tx=0x7fb3100092d0).ready entity=mon.0 client_cookie=6023ee402d4f495c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:51.986 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.974+0000 7fb31c929700 1 -- 172.21.15.23:0/844539658 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31000bbf0 con 0x7fb318067a10 2020-11-20T11:40:51.986 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.974+0000 7fb31c929700 1 -- 172.21.15.23:0/844539658 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb31000bd50 con 0x7fb318067a10 2020-11-20T11:40:51.986 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.974+0000 7fb31c929700 1 -- 172.21.15.23:0/844539658 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb310011610 con 0x7fb318067a10 2020-11-20T11:40:51.986 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.974+0000 7fb31e92d700 1 -- 172.21.15.23:0/844539658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb318067a10 msgr2=0x7fb318067ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:51.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.974+0000 7fb31e92d700 1 --2- 172.21.15.23:0/844539658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb318067a10 0x7fb318067ea0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fb310009d90 tx=0x7fb3100092d0).stop 2020-11-20T11:40:51.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.975+0000 7fb31e92d700 1 -- 172.21.15.23:0/844539658 shutdown_connections 2020-11-20T11:40:51.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.975+0000 7fb31e92d700 1 --2- 172.21.15.23:0/844539658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb318067a10 0x7fb318067ea0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:51.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.975+0000 7fb31e92d700 1 --2- 172.21.15.23:0/844539658 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3180a2460 0x7fb318137df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:40:51.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.975+0000 7fb31e92d700 1 -- 172.21.15.23:0/844539658 >> 172.21.15.23:0/844539658 conn(0x7fb318063980 msgr2=0x7fb318063d90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:40:51.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.975+0000 7fb31e92d700 1 -- 172.21.15.23:0/844539658 shutdown_connections 2020-11-20T11:40:51.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.975+0000 7fb31e92d700 1 -- 172.21.15.23:0/844539658 wait complete. 2020-11-20T11:40:51.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.976+0000 7fb31e92d700 1 Processor -- start 2020-11-20T11:40:51.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.976+0000 7fb31e92d700 1 -- start start 2020-11-20T11:40:51.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.976+0000 7fb31e92d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb318067a10 0x7fb3180a7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:51.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.976+0000 7fb31e92d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3180a2080 0x7fb31813e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:51.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.977+0000 7fb31e92d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3180a2460 0x7fb31814d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:51.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.977+0000 7fb31e92d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31813d040 con 0x7fb3180a2080 2020-11-20T11:40:51.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.977+0000 7fb31e92d700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31813d2e0 con 0x7fb318067a10 2020-11-20T11:40:51.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.977+0000 7fb31d12a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3180a2080 0x7fb31813e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:51.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.977+0000 7fb31e92d700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb31813d580 con 0x7fb3180a2460 2020-11-20T11:40:51.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.977+0000 7fb31d12a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3180a2080 0x7fb31813e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44452/0 (socket says 172.21.15.23:44452) 2020-11-20T11:40:51.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.977+0000 7fb31d12a700 1 -- 172.21.15.23:0/1886269177 learned_addr learned my addr 172.21.15.23:0/1886269177 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:40:51.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.977+0000 7fb31e12c700 1 -- 172.21.15.23:0/1886269177 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3180a2460 msgr2=0x7fb31814d370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:40:51.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.977+0000 7fb31d92b700 1 --2- 172.21.15.23:0/1886269177 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb318067a10 0x7fb3180a7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:51.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.977+0000 7fb31e12c700 1 --2- 172.21.15.23:0/1886269177 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3180a2460 0x7fb31814d370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:40:51.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.977+0000 7fb31d12a700 1 -- 172.21.15.23:0/1886269177 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3180a2460 msgr2=0x7fb31814d370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:40:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.977+0000 7fb31d12a700 1 --2- 172.21.15.23:0/1886269177 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3180a2460 0x7fb31814d370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:40:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.977+0000 7fb31d12a700 1 -- 172.21.15.23:0/1886269177 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb318067a10 msgr2=0x7fb3180a7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.977+0000 7fb31d12a700 1 --2- 172.21.15.23:0/1886269177 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb318067a10 0x7fb3180a7250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.977+0000 7fb31d12a700 1 -- 172.21.15.23:0/1886269177 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3180a7740 con 0x7fb3180a2080 2020-11-20T11:40:51.992 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.977+0000 7fb31d12a700 1 --2- 172.21.15.23:0/1886269177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3180a2080 0x7fb31813e580 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fb308009790 tx=0x7fb308006ea0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:51.993 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.977+0000 7fb305ffb700 1 -- 172.21.15.23:0/1886269177 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb308006170 con 0x7fb3180a2080 2020-11-20T11:40:51.993 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.977+0000 7fb305ffb700 1 -- 172.21.15.23:0/1886269177 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3080062d0 con 0x7fb3180a2080 2020-11-20T11:40:51.993 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.977+0000 7fb305ffb700 1 -- 172.21.15.23:0/1886269177 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb308003920 con 0x7fb3180a2080 2020-11-20T11:40:51.993 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.977+0000 7fb31e92d700 1 -- 172.21.15.23:0/1886269177 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb31813c3e0 con 0x7fb3180a2080 2020-11-20T11:40:51.993 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.978+0000 7fb31e92d700 1 -- 172.21.15.23:0/1886269177 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb31813ecb0 con 0x7fb3180a2080 2020-11-20T11:40:51.994 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.979+0000 7fb305ffb700 1 -- 172.21.15.23:0/1886269177 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb308017030 con 0x7fb3180a2080 2020-11-20T11:40:51.995 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.979+0000 7fb31e92d700 1 -- 172.21.15.23:0/1886269177 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb318067150 con 0x7fb3180a2080 2020-11-20T11:40:51.995 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.979+0000 7fb305ffb700 1 --2- 172.21.15.23:0/1886269177 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb2f0056d10 0x7fb2f00591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:51.995 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.979+0000 7fb305ffb700 1 -- 172.21.15.23:0/1886269177 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fb30806f1e0 con 0x7fb3180a2080 2020-11-20T11:40:51.995 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.980+0000 7fb31d92b700 1 --2- 172.21.15.23:0/1886269177 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb2f0056d10 0x7fb2f00591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:51.995 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.980+0000 7fb31d92b700 1 --2- 172.21.15.23:0/1886269177 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb2f0056d10 0x7fb2f00591d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fb314007fd0 tx=0x7fb314009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:40:51.996 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:51.983+0000 7fb305ffb700 1 -- 172.21.15.23:0/1886269177 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb308074060 con 0x7fb3180a2080 2020-11-20T11:40:52.100 INFO:tasks.workunit.client.0.smithi023.stdout:fs setup took 407 secs 2020-11-20T11:40:52.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:52.236+0000 7fb31e92d700 1 -- 172.21.15.23:0/1886269177 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb318057dc0 con 0x7fb3180a2080 2020-11-20T11:40:52.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:52.236+0000 7fb305ffb700 1 -- 172.21.15.23:0/1886269177 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7fb308074080 con 0x7fb3180a2080 2020-11-20T11:40:52.240 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:40:52.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:52.238+0000 7fb31e92d700 1 -- 172.21.15.23:0/1886269177 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb2f0056d10 msgr2=0x7fb2f00591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:52.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:52.238+0000 7fb31e92d700 1 --2- 172.21.15.23:0/1886269177 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb2f0056d10 0x7fb2f00591d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fb314007fd0 tx=0x7fb314009040).stop 2020-11-20T11:40:52.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:52.238+0000 7fb31e92d700 1 -- 172.21.15.23:0/1886269177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3180a2080 msgr2=0x7fb31813e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:52.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:52.238+0000 7fb31e92d700 1 --2- 172.21.15.23:0/1886269177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3180a2080 0x7fb31813e580 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fb308009790 tx=0x7fb308006ea0).stop 2020-11-20T11:40:52.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:52.238+0000 7fb31e92d700 1 -- 172.21.15.23:0/1886269177 shutdown_connections 2020-11-20T11:40:52.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:52.238+0000 7fb31e92d700 1 --2- 172.21.15.23:0/1886269177 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb2f0056d10 0x7fb2f00591d0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:52.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:52.238+0000 7fb31e92d700 1 --2- 172.21.15.23:0/1886269177 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3180a2460 0x7fb31814d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:40:52.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:52.238+0000 7fb31e92d700 1 --2- 172.21.15.23:0/1886269177 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb318067a10 0x7fb3180a7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:52.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:52.238+0000 7fb31e92d700 1 --2- 172.21.15.23:0/1886269177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3180a2080 0x7fb31813e580 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:40:52.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:52.238+0000 7fb31e92d700 1 -- 172.21.15.23:0/1886269177 >> 172.21.15.23:0/1886269177 conn(0x7fb318063980 msgr2=0x7fb318068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:40:52.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:52.238+0000 7fb31e92d700 1 -- 172.21.15.23:0/1886269177 shutdown_connections 2020-11-20T11:40:52.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:52.238+0000 7fb31e92d700 1 -- 172.21.15.23:0/1886269177 wait complete. 2020-11-20T11:40:54.714 INFO:tasks.workunit.client.0.smithi023.stdout:Syncing()...2 sec 2020-11-20T11:40:54.715 INFO:tasks.workunit.client.0.smithi023.stdout:Starting Actual Benchmark At: Fri Nov 20 11:40:54 2020 2020-11-20T11:40:54.715 INFO:tasks.workunit.client.0.smithi023.stdout: 2020-11-20T11:40:55.254 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:40:55.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:55.628+0000 7f1bfddf6700 1 Processor -- start 2020-11-20T11:40:55.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:55.628+0000 7f1bfddf6700 1 -- start start 2020-11-20T11:40:55.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:55.629+0000 7f1bfddf6700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bf80ad600 0x7f1bf813f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:55.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:55.629+0000 7f1bfddf6700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf81424c0 con 0x7f1bf80a06e0 2020-11-20T11:40:55.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:55.629+0000 7f1bfddf6700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf80a5e90 con 0x7f1bf813fba0 2020-11-20T11:40:55.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:55.629+0000 7f1bfddf6700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf813ef70 con 0x7f1bf80ad600 2020-11-20T11:40:55.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:55.629+0000 7f1bf77fe700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bf80ad600 msgr2=0x7f1bf813f6b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:40:55.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:55.629+0000 7f1bf77fe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bf80ad600 0x7f1bf813f6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:40:55.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:55.629+0000 7f1bf6ffd700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f1bf80a06e0 0x7f1bf8142ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58708/0 (socket says 172.21.15.23:58708) 2020-11-20T11:40:55.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:55.629+0000 7f1bf6ffd700 1 -- 172.21.15.23:0/2119721854 learned_addr learned my addr 172.21.15.23:0/2119721854 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:40:55.834 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:55.830+0000 7f1bf77fe700 1 -- 172.21.15.23:0/2119721854 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bf80ad600 msgr2=0x7f1bf813f6b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:40:55.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:55.830+0000 7f1bf77fe700 1 --2- 172.21.15.23:0/2119721854 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bf80ad600 0x7f1bf813f6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:40:56.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:56.232+0000 7f1bf77fe700 1 -- 172.21.15.23:0/2119721854 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bf80ad600 msgr2=0x7f1bf813f6b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:40:56.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:56.232+0000 7f1bf77fe700 1 --2- 172.21.15.23:0/2119721854 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bf80ad600 0x7f1bf813f6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:40:57.037 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:57.033+0000 7f1bf77fe700 1 -- 172.21.15.23:0/2119721854 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bf80ad600 msgr2=0x7f1bf813f6b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:40:57.037 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:57.033+0000 7f1bf77fe700 1 --2- 172.21.15.23:0/2119721854 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bf80ad600 0x7f1bf813f6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:40:58.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:58.630+0000 7f1bf57fa700 1 -- 172.21.15.23:0/2119721854 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bf80ad600 msgr2=0x7f1bf813f6b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:40:58.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:58.630+0000 7f1bf57fa700 1 --2- 172.21.15.23:0/2119721854 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bf80ad600 0x7f1bf813f6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:40:58.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:58.630+0000 7f1bf57fa700 1 -- 172.21.15.23:0/2119721854 >> v1:172.21.15.155:6789/0 conn(0x7f1bf813fba0 legacy=0x7f1bf8149840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:58.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:58.630+0000 7f1bf57fa700 1 -- 172.21.15.23:0/2119721854 >> v1:172.21.15.23:6789/0 conn(0x7f1bf80a06e0 legacy=0x7f1bf8142ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:40:58.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:58.630+0000 7f1bf57fa700 1 --2- 172.21.15.23:0/2119721854 >> v2:172.21.15.23:3300/0 conn(0x7f1be0003440 0x7f1be00058c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:58.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:58.630+0000 7f1bf57fa700 1 --2- 172.21.15.23:0/2119721854 >> v2:172.21.15.155:3300/0 conn(0x7f1be0005ec0 0x7f1be0008360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:40:58.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:58.630+0000 7f1bf57fa700 1 -- 172.21.15.23:0/2119721854 --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bf813ef70 con 0x7f1be0003440 2020-11-20T11:40:58.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:58.630+0000 7f1bf57fa700 1 -- 172.21.15.23:0/2119721854 --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1be00089d0 con 0x7f1be0005ec0 2020-11-20T11:40:58.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:58.631+0000 7f1bf77fe700 1 --2- 172.21.15.23:0/2119721854 >> v2:172.21.15.23:3300/0 conn(0x7f1be0003440 0x7f1be00058c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:58.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:58.631+0000 7f1bf6ffd700 1 --2- 172.21.15.23:0/2119721854 >> v2:172.21.15.155:3300/0 conn(0x7f1be0005ec0 0x7f1be0008360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:58.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:58.631+0000 7f1bf77fe700 1 -- 172.21.15.23:0/2119721854 >> v2:172.21.15.23:3300/0 conn(0x7f1be0003440 msgr2=0x7f1be00058c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:40:58.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:58.631+0000 7f1bf77fe700 1 -- 172.21.15.23:0/2119721854 >> v2:172.21.15.23:3300/0 conn(0x7f1be0003440 msgr2=0x7f1be00058c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:40:58.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:58.631+0000 7f1bf77fe700 1 --2- 172.21.15.23:0/2119721854 >> v2:172.21.15.23:3300/0 conn(0x7f1be0003440 0x7f1be00058c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:40:58.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:58.631+0000 7f1bf6ffd700 1 -- 172.21.15.23:0/2119721854 >> v2:172.21.15.155:3300/0 conn(0x7f1be0005ec0 msgr2=0x7f1be0008360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:40:58.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:58.631+0000 7f1bf6ffd700 1 -- 172.21.15.23:0/2119721854 >> v2:172.21.15.155:3300/0 conn(0x7f1be0005ec0 msgr2=0x7f1be0008360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:40:58.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:58.631+0000 7f1bf6ffd700 1 --2- 172.21.15.23:0/2119721854 >> v2:172.21.15.155:3300/0 conn(0x7f1be0005ec0 0x7f1be0008360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:40:58.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:58.631+0000 7f1bf77fe700 1 --2- 172.21.15.23:0/2119721854 >> v2:172.21.15.23:3300/0 conn(0x7f1be0003440 0x7f1be00058c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:40:58.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:58.631+0000 7f1bf6ffd700 1 --2- 172.21.15.23:0/2119721854 >> v2:172.21.15.155:3300/0 conn(0x7f1be0005ec0 0x7f1be0008360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:40:58.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:58.832+0000 7f1bf77fe700 1 --2- 172.21.15.23:0/2119721854 >> v2:172.21.15.23:3300/0 conn(0x7f1be0003440 0x7f1be00058c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:58.837 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:58.832+0000 7f1bf77fe700 1 -- 172.21.15.23:0/2119721854 >> v2:172.21.15.23:3300/0 conn(0x7f1be0003440 msgr2=0x7f1be00058c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:40:58.837 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:58.832+0000 7f1bf77fe700 1 -- 172.21.15.23:0/2119721854 >> v2:172.21.15.23:3300/0 conn(0x7f1be0003440 msgr2=0x7f1be00058c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:40:58.837 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:58.832+0000 7f1bf77fe700 1 --2- 172.21.15.23:0/2119721854 >> v2:172.21.15.23:3300/0 conn(0x7f1be0003440 0x7f1be00058c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:40:58.837 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:58.832+0000 7f1bf77fe700 1 --2- 172.21.15.23:0/2119721854 >> v2:172.21.15.23:3300/0 conn(0x7f1be0003440 0x7f1be00058c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:40:58.838 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:58.832+0000 7f1bf6ffd700 1 --2- 172.21.15.23:0/2119721854 >> v2:172.21.15.155:3300/0 conn(0x7f1be0005ec0 0x7f1be0008360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:58.838 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:58.832+0000 7f1bf6ffd700 1 -- 172.21.15.23:0/2119721854 >> v2:172.21.15.155:3300/0 conn(0x7f1be0005ec0 msgr2=0x7f1be0008360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:40:58.838 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:58.832+0000 7f1bf6ffd700 1 -- 172.21.15.23:0/2119721854 >> v2:172.21.15.155:3300/0 conn(0x7f1be0005ec0 msgr2=0x7f1be0008360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:40:58.838 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:58.832+0000 7f1bf6ffd700 1 --2- 172.21.15.23:0/2119721854 >> v2:172.21.15.155:3300/0 conn(0x7f1be0005ec0 0x7f1be0008360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:40:58.838 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:58.832+0000 7f1bf6ffd700 1 --2- 172.21.15.23:0/2119721854 >> v2:172.21.15.155:3300/0 conn(0x7f1be0005ec0 0x7f1be0008360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:40:59.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:59.233+0000 7f1bf6ffd700 1 --2- 172.21.15.23:0/2119721854 >> v2:172.21.15.155:3300/0 conn(0x7f1be0005ec0 0x7f1be0008360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:59.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:59.233+0000 7f1bf77fe700 1 --2- 172.21.15.23:0/2119721854 >> v2:172.21.15.23:3300/0 conn(0x7f1be0003440 0x7f1be00058c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:40:59.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:59.233+0000 7f1bf77fe700 1 -- 172.21.15.23:0/2119721854 >> v2:172.21.15.23:3300/0 conn(0x7f1be0003440 msgr2=0x7f1be00058c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:40:59.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:59.233+0000 7f1bf77fe700 1 -- 172.21.15.23:0/2119721854 >> v2:172.21.15.23:3300/0 conn(0x7f1be0003440 msgr2=0x7f1be00058c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:40:59.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:59.233+0000 7f1bf77fe700 1 --2- 172.21.15.23:0/2119721854 >> v2:172.21.15.23:3300/0 conn(0x7f1be0003440 0x7f1be00058c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:40:59.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:59.234+0000 7f1bf77fe700 1 --2- 172.21.15.23:0/2119721854 >> v2:172.21.15.23:3300/0 conn(0x7f1be0003440 0x7f1be00058c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:40:59.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:59.234+0000 7f1bf6ffd700 1 -- 172.21.15.23:0/2119721854 >> v2:172.21.15.155:3300/0 conn(0x7f1be0005ec0 msgr2=0x7f1be0008360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:40:59.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:59.234+0000 7f1bf6ffd700 1 -- 172.21.15.23:0/2119721854 >> v2:172.21.15.155:3300/0 conn(0x7f1be0005ec0 msgr2=0x7f1be0008360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:40:59.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:59.234+0000 7f1bf6ffd700 1 --2- 172.21.15.23:0/2119721854 >> v2:172.21.15.155:3300/0 conn(0x7f1be0005ec0 0x7f1be0008360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:40:59.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:40:59.234+0000 7f1bf6ffd700 1 --2- 172.21.15.23:0/2119721854 >> v2:172.21.15.155:3300/0 conn(0x7f1be0005ec0 0x7f1be0008360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:41:00.039 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.035+0000 7f1bf77fe700 1 --2- 172.21.15.23:0/2119721854 >> v2:172.21.15.23:3300/0 conn(0x7f1be0003440 0x7f1be00058c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:00.040 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.035+0000 7f1bf77fe700 1 -- 172.21.15.23:0/2119721854 >> v2:172.21.15.155:3300/0 conn(0x7f1be0005ec0 msgr2=0x7f1be0008360 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2020-11-20T11:41:00.040 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.035+0000 7f1bf77fe700 1 --2- 172.21.15.23:0/2119721854 >> v2:172.21.15.155:3300/0 conn(0x7f1be0005ec0 0x7f1be0008360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:00.040 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.035+0000 7f1bf77fe700 1 -- 172.21.15.23:0/2119721854 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf813f210 con 0x7f1be0003440 2020-11-20T11:41:00.041 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.036+0000 7f1bf77fe700 1 --2- 172.21.15.23:0/2119721854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1be0003440 0x7f1be00058c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f1be0005e90 tx=0x7f1bec019040).ready entity=mon.0 client_cookie=cee2265807018223 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:00.042 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.036+0000 7f1bf67fc700 1 -- 172.21.15.23:0/2119721854 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bec014ca0 con 0x7f1be0003440 2020-11-20T11:41:00.042 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.036+0000 7f1bf67fc700 1 -- 172.21.15.23:0/2119721854 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bec014e00 con 0x7f1be0003440 2020-11-20T11:41:00.042 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.036+0000 7f1bf67fc700 1 -- 172.21.15.23:0/2119721854 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bec022ab0 con 0x7f1be0003440 2020-11-20T11:41:00.042 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.037+0000 7f1bfddf6700 1 -- 172.21.15.23:0/2119721854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1be0003440 msgr2=0x7f1be00058c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:00.043 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.037+0000 7f1bfddf6700 1 --2- 172.21.15.23:0/2119721854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1be0003440 0x7f1be00058c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f1be0005e90 tx=0x7f1bec019040).stop 2020-11-20T11:41:00.043 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.038+0000 7f1bf7fff700 1 -- 172.21.15.23:0/2119721854 reap_dead start 2020-11-20T11:41:00.043 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.038+0000 7f1bfddf6700 1 -- 172.21.15.23:0/2119721854 shutdown_connections 2020-11-20T11:41:00.043 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.038+0000 7f1bfddf6700 1 --2- 172.21.15.23:0/2119721854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1be0003440 0x7f1be00058c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:00.044 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.038+0000 7f1bfddf6700 1 -- 172.21.15.23:0/2119721854 >> 172.21.15.23:0/2119721854 conn(0x7f1bf8060c80 msgr2=0x7f1bf80acd30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:41:00.044 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.038+0000 7f1bfddf6700 1 -- 172.21.15.23:0/2119721854 shutdown_connections 2020-11-20T11:41:00.044 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.038+0000 7f1bfddf6700 1 -- 172.21.15.23:0/2119721854 wait complete. 2020-11-20T11:41:00.044 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.040+0000 7f1bfddf6700 1 Processor -- start 2020-11-20T11:41:00.045 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.040+0000 7f1bfddf6700 1 -- start start 2020-11-20T11:41:00.045 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.041+0000 7f1bfddf6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1be0003440 0x7f1bf80a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:00.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.041+0000 7f1bfddf6700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bf80a19b0 0x7f1bf8135390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:00.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.041+0000 7f1bfddf6700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1bf80a1e30 0x7f1bf8135880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:00.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.041+0000 7f1bfddf6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf8066600 con 0x7f1be0003440 2020-11-20T11:41:00.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.041+0000 7f1bfddf6700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf8066d00 con 0x7f1bf80a1e30 2020-11-20T11:41:00.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.041+0000 7f1bfddf6700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf8066860 con 0x7f1bf80a19b0 2020-11-20T11:41:00.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.041+0000 7f1bf7fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1bf80a1e30 0x7f1bf8135880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:00.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.041+0000 7f1bf77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1be0003440 0x7f1bf80a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:00.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.041+0000 7f1bf77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1be0003440 0x7f1bf80a11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44490/0 (socket says 172.21.15.23:44490) 2020-11-20T11:41:00.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.041+0000 7f1bf77fe700 1 -- 172.21.15.23:0/4175379391 learned_addr learned my addr 172.21.15.23:0/4175379391 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:41:00.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.041+0000 7f1bf6ffd700 1 -- 172.21.15.23:0/4175379391 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bf80a19b0 msgr2=0x7f1bf8135390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:41:00.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.041+0000 7f1bf6ffd700 1 --2- 172.21.15.23:0/4175379391 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bf80a19b0 0x7f1bf8135390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:41:00.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.042+0000 7f1bf77fe700 1 -- 172.21.15.23:0/4175379391 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bf80a19b0 msgr2=0x7f1bf8135390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:41:00.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.042+0000 7f1bf77fe700 1 --2- 172.21.15.23:0/4175379391 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bf80a19b0 0x7f1bf8135390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:00.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.042+0000 7f1bf77fe700 1 -- 172.21.15.23:0/4175379391 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1bf80a1e30 msgr2=0x7f1bf8135880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:00.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.042+0000 7f1bf77fe700 1 --2- 172.21.15.23:0/4175379391 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1bf80a1e30 0x7f1bf8135880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:00.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.042+0000 7f1bf77fe700 1 -- 172.21.15.23:0/4175379391 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf8065520 con 0x7f1be0003440 2020-11-20T11:41:00.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.042+0000 7f1bf77fe700 1 --2- 172.21.15.23:0/4175379391 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1be0003440 0x7f1bf80a11a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f1bec014fd0 tx=0x7f1bec019040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:00.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.042+0000 7f1bdb7fe700 1 -- 172.21.15.23:0/4175379391 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bec014ca0 con 0x7f1be0003440 2020-11-20T11:41:00.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.043+0000 7f1bfddf6700 1 -- 172.21.15.23:0/4175379391 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bf8135d70 con 0x7f1be0003440 2020-11-20T11:41:00.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.043+0000 7f1bfddf6700 1 -- 172.21.15.23:0/4175379391 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bf813cc00 con 0x7f1be0003440 2020-11-20T11:41:00.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.043+0000 7f1bdb7fe700 1 -- 172.21.15.23:0/4175379391 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bec022e60 con 0x7f1be0003440 2020-11-20T11:41:00.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.043+0000 7f1bdb7fe700 1 -- 172.21.15.23:0/4175379391 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bec026690 con 0x7f1be0003440 2020-11-20T11:41:00.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.044+0000 7f1bfddf6700 1 -- 172.21.15.23:0/4175379391 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bf80a7ea0 con 0x7f1be0003440 2020-11-20T11:41:00.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.048+0000 7f1bdb7fe700 1 -- 172.21.15.23:0/4175379391 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1bec038020 con 0x7f1be0003440 2020-11-20T11:41:00.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.048+0000 7f1bdb7fe700 1 --2- 172.21.15.23:0/4175379391 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1bcc056d60 0x7f1bcc059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:00.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.049+0000 7f1bdb7fe700 1 -- 172.21.15.23:0/4175379391 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f1bec083c50 con 0x7f1be0003440 2020-11-20T11:41:00.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.049+0000 7f1bf6ffd700 1 --2- 172.21.15.23:0/4175379391 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1bcc056d60 0x7f1bcc059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:00.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.050+0000 7f1bf6ffd700 1 --2- 172.21.15.23:0/4175379391 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1bcc056d60 0x7f1bcc059220 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f1bf0007fd0 tx=0x7f1bf0009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:00.071 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.067+0000 7f1bdb7fe700 1 -- 172.21.15.23:0/4175379391 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1bec05c8e0 con 0x7f1be0003440 2020-11-20T11:41:00.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.875+0000 7f1bfddf6700 1 -- 172.21.15.23:0/4175379391 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1bf8057dc0 con 0x7f1be0003440 2020-11-20T11:41:00.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.877+0000 7f1bdb7fe700 1 -- 172.21.15.23:0/4175379391 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1471 (secure 0 0 0) 0x7f1bec048080 con 0x7f1be0003440 2020-11-20T11:41:00.881 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":86,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:41:00.884 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.880+0000 7f1bd97fa700 1 -- 172.21.15.23:0/4175379391 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1bcc056d60 msgr2=0x7f1bcc059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:00.884 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.880+0000 7f1bd97fa700 1 --2- 172.21.15.23:0/4175379391 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1bcc056d60 0x7f1bcc059220 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f1bf0007fd0 tx=0x7f1bf0009040).stop 2020-11-20T11:41:00.885 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.880+0000 7f1bd97fa700 1 -- 172.21.15.23:0/4175379391 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1be0003440 msgr2=0x7f1bf80a11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:00.885 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.880+0000 7f1bd97fa700 1 --2- 172.21.15.23:0/4175379391 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1be0003440 0x7f1bf80a11a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f1bec014fd0 tx=0x7f1bec019040).stop 2020-11-20T11:41:00.885 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.881+0000 7f1bd97fa700 1 -- 172.21.15.23:0/4175379391 shutdown_connections 2020-11-20T11:41:00.886 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.881+0000 7f1bd97fa700 1 --2- 172.21.15.23:0/4175379391 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1bf80a1e30 0x7f1bf8135880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:00.886 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.881+0000 7f1bd97fa700 1 --2- 172.21.15.23:0/4175379391 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1be0003440 0x7f1bf80a11a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:00.886 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.881+0000 7f1bd97fa700 1 --2- 172.21.15.23:0/4175379391 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1bcc056d60 0x7f1bcc059220 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:00.886 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.881+0000 7f1bd97fa700 1 --2- 172.21.15.23:0/4175379391 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1bf80a19b0 0x7f1bf8135390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:00.887 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.881+0000 7f1bd97fa700 1 -- 172.21.15.23:0/4175379391 >> 172.21.15.23:0/4175379391 conn(0x7f1bf8060c80 msgr2=0x7f1bf80a69e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:41:00.887 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.881+0000 7f1bd97fa700 1 -- 172.21.15.23:0/4175379391 shutdown_connections 2020-11-20T11:41:00.887 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:00.881+0000 7f1bd97fa700 1 -- 172.21.15.23:0/4175379391 wait complete. 2020-11-20T11:41:00.917 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2020-11-20T11:41:00.917 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:41:01.093 INFO:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:41:01.096 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:41:01.316 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.310+0000 7f8cdbe1a700 1 Processor -- start 2020-11-20T11:41:01.316 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.311+0000 7f8cdbe1a700 1 -- start start 2020-11-20T11:41:01.316 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.312+0000 7f8cdbe1a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8cd40ad7c0 0x7f8cd40a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:01.317 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.312+0000 7f8cdbe1a700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8cd40a67d0 0x7f8cd4061f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:01.317 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.312+0000 7f8cdbe1a700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd40a3cc0 con 0x7f8cd4063fe0 2020-11-20T11:41:01.317 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.312+0000 7f8cdbe1a700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cd40a6c30 con 0x7f8cd40ad7c0 2020-11-20T11:41:01.317 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.312+0000 7f8cdbe1a700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd40a4640 con 0x7f8cd40a67d0 2020-11-20T11:41:01.318 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.313+0000 7f8cd9bb6700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8cd40ad7c0 0x7f8cd40a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:01.319 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.313+0000 7f8cd9bb6700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8cd40ad7c0 0x7f8cd40a62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44494/0 (socket says 172.21.15.23:44494) 2020-11-20T11:41:01.319 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.313+0000 7f8cd9bb6700 1 -- 172.21.15.23:0/1258638354 learned_addr learned my addr 172.21.15.23:0/1258638354 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:41:01.319 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.313+0000 7f8cd9bb6700 1 -- 172.21.15.23:0/1258638354 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8cd40a67d0 msgr2=0x7f8cd4061f60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2020-11-20T11:41:01.319 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.313+0000 7f8cd9bb6700 1 --2- 172.21.15.23:0/1258638354 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8cd40a67d0 0x7f8cd4061f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:01.319 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.313+0000 7f8cd9bb6700 1 -- 172.21.15.23:0/1258638354 >> v1:172.21.15.23:6789/0 conn(0x7f8cd4063fe0 legacy=0x7f8cd4061850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:41:01.320 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.313+0000 7f8cd9bb6700 1 -- 172.21.15.23:0/1258638354 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd40a5ba0 con 0x7f8cd40ad7c0 2020-11-20T11:41:01.320 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.314+0000 7f8cd9bb6700 1 --2- 172.21.15.23:0/1258638354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cd40ad7c0 0x7f8cd40a62e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f8cc4009a10 tx=0x7f8cc4006ef0).ready entity=mon.0 client_cookie=6dabf1ff308b2973 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:01.323 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.317+0000 7f8cd8bb4700 1 -- 172.21.15.23:0/1258638354 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc400b820 con 0x7f8cd40ad7c0 2020-11-20T11:41:01.324 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.317+0000 7f8cd8bb4700 1 -- 172.21.15.23:0/1258638354 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc4000e90 con 0x7f8cd40ad7c0 2020-11-20T11:41:01.324 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.317+0000 7f8cd8bb4700 1 -- 172.21.15.23:0/1258638354 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc400bb90 con 0x7f8cd40ad7c0 2020-11-20T11:41:01.324 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.318+0000 7f8cdbe1a700 1 -- 172.21.15.23:0/1258638354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cd40ad7c0 msgr2=0x7f8cd40a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:01.324 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.318+0000 7f8cdbe1a700 1 --2- 172.21.15.23:0/1258638354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cd40ad7c0 0x7f8cd40a62e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f8cc4009a10 tx=0x7f8cc4006ef0).stop 2020-11-20T11:41:01.325 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.318+0000 7f8cdbe1a700 1 -- 172.21.15.23:0/1258638354 shutdown_connections 2020-11-20T11:41:01.325 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.318+0000 7f8cdbe1a700 1 --2- 172.21.15.23:0/1258638354 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8cd40a67d0 0x7f8cd4061f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:01.325 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.318+0000 7f8cdbe1a700 1 --2- 172.21.15.23:0/1258638354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cd40ad7c0 0x7f8cd40a62e0 secure :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0x7f8cc4009a10 tx=0x7f8cc4006ef0).stop 2020-11-20T11:41:01.325 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.318+0000 7f8cdbe1a700 1 -- 172.21.15.23:0/1258638354 >> 172.21.15.23:0/1258638354 conn(0x7f8cd4060c80 msgr2=0x7f8cd40acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:41:01.326 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.321+0000 7f8cdbe1a700 1 -- 172.21.15.23:0/1258638354 shutdown_connections 2020-11-20T11:41:01.329 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.323+0000 7f8cdbe1a700 1 -- 172.21.15.23:0/1258638354 wait complete. 2020-11-20T11:41:01.329 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.325+0000 7f8cdbe1a700 1 Processor -- start 2020-11-20T11:41:01.332 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.327+0000 7f8cdbe1a700 1 -- start start 2020-11-20T11:41:01.333 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.328+0000 7f8cdbe1a700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8cd40a67d0 0x7f8cd40650b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:01.333 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.328+0000 7f8cdbe1a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8cd4067850 0x7f8cd413a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:01.333 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.328+0000 7f8cdbe1a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cd413a730 0x7f8cd4065a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:01.334 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.328+0000 7f8cdbe1a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd413d210 con 0x7f8cd413a730 2020-11-20T11:41:01.334 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.328+0000 7f8cdbe1a700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd413abb0 con 0x7f8cd4067850 2020-11-20T11:41:01.334 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.328+0000 7f8cdbe1a700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd41419b0 con 0x7f8cd40a67d0 2020-11-20T11:41:01.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.330+0000 7f8cda3b7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cd413a730 0x7f8cd4065a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:01.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.330+0000 7f8cda3b7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cd413a730 0x7f8cd4065a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44498/0 (socket says 172.21.15.23:44498) 2020-11-20T11:41:01.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.330+0000 7f8cda3b7700 1 -- 172.21.15.23:0/1480771040 learned_addr learned my addr 172.21.15.23:0/1480771040 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:41:01.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.330+0000 7f8cda3b7700 1 -- 172.21.15.23:0/1480771040 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8cd40a67d0 msgr2=0x7f8cd40650b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:41:01.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.331+0000 7f8cda3b7700 1 --2- 172.21.15.23:0/1480771040 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8cd40a67d0 0x7f8cd40650b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:01.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.331+0000 7f8cda3b7700 1 -- 172.21.15.23:0/1480771040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8cd4067850 msgr2=0x7f8cd413a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:01.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.331+0000 7f8cda3b7700 1 --2- 172.21.15.23:0/1480771040 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8cd4067850 0x7f8cd413a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:01.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.331+0000 7f8cda3b7700 1 -- 172.21.15.23:0/1480771040 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd413d440 con 0x7f8cd413a730 2020-11-20T11:41:01.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.331+0000 7f8cda3b7700 1 --2- 172.21.15.23:0/1480771040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cd413a730 0x7f8cd4065a10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f8ccc00bb10 tx=0x7f8ccc0090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:01.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.332+0000 7f8cc9ffb700 1 -- 172.21.15.23:0/1480771040 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc00c830 con 0x7f8cd413a730 2020-11-20T11:41:01.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.332+0000 7f8cc9ffb700 1 -- 172.21.15.23:0/1480771040 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ccc00c990 con 0x7f8cd413a730 2020-11-20T11:41:01.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.332+0000 7f8cdbe1a700 1 -- 172.21.15.23:0/1480771040 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd413d6e0 con 0x7f8cd413a730 2020-11-20T11:41:01.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.332+0000 7f8cc9ffb700 1 -- 172.21.15.23:0/1480771040 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc00cb40 con 0x7f8cd413a730 2020-11-20T11:41:01.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.333+0000 7f8cdbe1a700 1 -- 172.21.15.23:0/1480771040 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd4065f00 con 0x7f8cd413a730 2020-11-20T11:41:01.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.334+0000 7f8cdbe1a700 1 --2- 172.21.15.23:0/1480771040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cd413b0b0 0x7f8cd413b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:01.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.334+0000 7f8cdbe1a700 1 -- 172.21.15.23:0/1480771040 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd4061650 con 0x7f8cd413b0b0 2020-11-20T11:41:01.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.335+0000 7f8cd9bb6700 1 --2- 172.21.15.23:0/1480771040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cd413b0b0 0x7f8cd413b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:01.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.335+0000 7f8cd9bb6700 1 -- 172.21.15.23:0/1480771040 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8cd413d870 con 0x7f8cd413b0b0 2020-11-20T11:41:01.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.335+0000 7f8cd9bb6700 1 --2- 172.21.15.23:0/1480771040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cd413b0b0 0x7f8cd413b490 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f8cc4009a10 tx=0x7f8cc4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:01.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.336+0000 7f8cc9ffb700 1 -- 172.21.15.23:0/1480771040 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc4011b80 con 0x7f8cd413b0b0 2020-11-20T11:41:01.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.336+0000 7f8cc9ffb700 1 -- 172.21.15.23:0/1480771040 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f8cc4011ce0 con 0x7f8cd413b0b0 2020-11-20T11:41:01.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.336+0000 7f8cc9ffb700 1 -- 172.21.15.23:0/1480771040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cd413b0b0 msgr2=0x7f8cd413b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:01.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.336+0000 7f8cc9ffb700 1 --2- 172.21.15.23:0/1480771040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cd413b0b0 0x7f8cd413b490 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f8cc4009a10 tx=0x7f8cc4016000).stop 2020-11-20T11:41:01.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.336+0000 7f8cc9ffb700 1 -- 172.21.15.23:0/1480771040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cd413b0b0 msgr2=0x7f8cd413b490 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:41:01.345 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.336+0000 7f8cc9ffb700 1 --2- 172.21.15.23:0/1480771040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cd413b0b0 0x7f8cd413b490 secure :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0x7f8cc4009a10 tx=0x7f8cc4016000).stop 2020-11-20T11:41:01.345 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.340+0000 7f8cc9ffb700 1 -- 172.21.15.23:0/1480771040 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8ccc01c070 con 0x7f8cd413a730 2020-11-20T11:41:01.346 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.341+0000 7f8cc9ffb700 1 --2- 172.21.15.23:0/1480771040 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8cb40574d0 0x7f8cb4059990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:01.346 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.341+0000 7f8cc9ffb700 1 -- 172.21.15.23:0/1480771040 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f8ccc06f6c0 con 0x7f8cd413a730 2020-11-20T11:41:01.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.353+0000 7f8cd9bb6700 1 --2- 172.21.15.23:0/1480771040 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8cb40574d0 0x7f8cb4059990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:01.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.354+0000 7f8cdbe1a700 1 --2- 172.21.15.23:0/1480771040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cd413b980 0x7f8cd411ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:01.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.354+0000 7f8cd9bb6700 1 --2- 172.21.15.23:0/1480771040 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8cb40574d0 0x7f8cb4059990 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f8cd405cfa0 tx=0x7f8cc401d000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:01.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.354+0000 7f8cdbe1a700 1 -- 172.21.15.23:0/1480771040 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd405c520 con 0x7f8cd413b980 2020-11-20T11:41:01.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.355+0000 7f8cd93b5700 1 --2- 172.21.15.23:0/1480771040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cd413b980 0x7f8cd411ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:01.364 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.355+0000 7f8cd93b5700 1 -- 172.21.15.23:0/1480771040 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8cd4044710 con 0x7f8cd413b980 2020-11-20T11:41:01.364 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.355+0000 7f8cd93b5700 1 --2- 172.21.15.23:0/1480771040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cd413b980 0x7f8cd411ca80 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f8cd4067350 tx=0x7f8cd0006d30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:01.364 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.356+0000 7f8cc9ffb700 1 -- 172.21.15.23:0/1480771040 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd0012460 con 0x7f8cd413b980 2020-11-20T11:41:01.364 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.356+0000 7f8cc9ffb700 1 -- 172.21.15.23:0/1480771040 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4774 (secure 0 0 0) 0x7f8cd0000be0 con 0x7f8cd413b980 2020-11-20T11:41:01.365 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.356+0000 7f8cc9ffb700 1 -- 172.21.15.23:0/1480771040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cd413b980 msgr2=0x7f8cd411ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:01.365 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.356+0000 7f8cc9ffb700 1 --2- 172.21.15.23:0/1480771040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cd413b980 0x7f8cd411ca80 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f8cd4067350 tx=0x7f8cd0006d30).stop 2020-11-20T11:41:01.365 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.356+0000 7f8cc9ffb700 1 -- 172.21.15.23:0/1480771040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cd413b980 msgr2=0x7f8cd411ca80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:41:01.365 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.356+0000 7f8cc9ffb700 1 --2- 172.21.15.23:0/1480771040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cd413b980 0x7f8cd411ca80 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:01.366 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:41:01.366 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:41:01.366 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:41:01.366 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:41:01.367 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 86, 2020-11-20T11:41:01.367 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:41:01.367 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:41:01.367 INFO:teuthology.orchestra.run.smithi023.stdout: 1 2020-11-20T11:41:01.367 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:01.368 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 1, 2020-11-20T11:41:01.368 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:41:01.368 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:41:01.368 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:41:01.368 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:41:01.369 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:41:01.369 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:41:01.369 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:41:01.369 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:41:01.370 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:41:01.370 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:41:01.370 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:41:01.370 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:01.370 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:41:01.371 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:41:01.371 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:41:01.371 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:41:01.371 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:41:01.371 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:41:01.372 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:41:01.372 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:41:01.372 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:41:01.373 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:41:01.373 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:01.373 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:01.373 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:41:01.374 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:41:01.374 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:41:01.374 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:41:01.374 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:41:01.374 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:41:01.375 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:41:01.375 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:41:01.375 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:41:01.375 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:41:01.375 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:41:01.376 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:41:01.376 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:41:01.376 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:41:01.376 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:41:01.376 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:41:01.377 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:41:01.377 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:41:01.377 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:41:01.377 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:41:01.378 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:41:01.378 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:41:01.378 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:41:01.378 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:01.378 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:41:01.379 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:01.379 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:41:01.379 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:01.379 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:41:01.379 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:41:01.379 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:41:01.380 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:41:01.380 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:01.380 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:41:01.380 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:41:01.380 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:01.381 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:01.381 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:01.381 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:41:01.381 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:41:01.381 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:01.382 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:01.382 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:01.382 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:01.382 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:41:01.382 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:41:01.383 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:41:01.383 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:41:01.383 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:41:01.383 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:01.383 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:01.384 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:41:01.384 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:41:01.384 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:41:01.384 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:41:01.384 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:01.385 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:41:01.385 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:41:01.385 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:01.385 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:01.385 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:01.386 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:41:01.386 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:41:01.386 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:01.386 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:01.386 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:01.387 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:01.387 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:41:01.387 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:41:01.387 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:41:01.387 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:41:01.388 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:41:01.388 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:01.388 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:01.388 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:41:01.388 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:41:01.389 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:41:01.389 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:41:01.389 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:01.389 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:41:01.389 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:41:01.390 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:01.390 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:01.390 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:01.390 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:41:01.390 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:41:01.391 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:01.391 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:01.391 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:01.391 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:01.391 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:41:01.392 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:41:01.392 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:41:01.392 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:41:01.392 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:41:01.392 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:01.393 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:01.393 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:01.393 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:41:01.393 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:41:01.393 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:01.394 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:41:01.394 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:41:01.394 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:41:01.394 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:01.394 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:01.395 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:41:01.395 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:01.395 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:41:01.395 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:41:01.395 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:41:01.396 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:01.396 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:01.396 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:41:01.396 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:01.396 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:41:01.397 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:41:01.397 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 8 2020-11-20T11:41:01.397 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:01.397 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:01.397 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:41:01.398 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:01.398 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:41:01.398 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:41:01.398 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:41:01.399 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:01.399 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:01.399 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:41:01.399 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:01.399 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:41:01.400 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:41:01.400 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:41:01.400 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:01.400 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:01.401 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:01.401 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:41:01.401 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:41:01.402 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.366+0000 7f8caf7fe700 1 -- 172.21.15.23:0/1480771040 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8cb40574d0 msgr2=0x7f8cb4059990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:01.402 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.366+0000 7f8caf7fe700 1 --2- 172.21.15.23:0/1480771040 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8cb40574d0 0x7f8cb4059990 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f8cd405cfa0 tx=0x7f8cc401d000).stop 2020-11-20T11:41:01.402 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.366+0000 7f8caf7fe700 1 -- 172.21.15.23:0/1480771040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cd413a730 msgr2=0x7f8cd4065a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:01.402 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.366+0000 7f8caf7fe700 1 --2- 172.21.15.23:0/1480771040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8cd413a730 0x7f8cd4065a10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f8ccc00bb10 tx=0x7f8ccc0090f0).stop 2020-11-20T11:41:01.403 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.366+0000 7f8cda3b7700 1 -- 172.21.15.23:0/1480771040 reap_dead start 2020-11-20T11:41:01.403 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.366+0000 7f8cda3b7700 1 -- 172.21.15.23:0/1480771040 reap_dead start 2020-11-20T11:41:01.403 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.366+0000 7f8caf7fe700 1 -- 172.21.15.23:0/1480771040 shutdown_connections 2020-11-20T11:41:01.403 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.366+0000 7f8caf7fe700 1 -- 172.21.15.23:0/1480771040 >> 172.21.15.23:0/1480771040 conn(0x7f8cd4060c80 msgr2=0x7f8cd40aa0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:41:01.403 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.367+0000 7f8caf7fe700 1 -- 172.21.15.23:0/1480771040 shutdown_connections 2020-11-20T11:41:01.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.367+0000 7f8caf7fe700 1 -- 172.21.15.23:0/1480771040 wait complete. 2020-11-20T11:41:01.423 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:41:01.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.754+0000 7f4fcbca0700 1 Processor -- start 2020-11-20T11:41:01.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.755+0000 7f4fcbca0700 1 -- start start 2020-11-20T11:41:01.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.755+0000 7f4fcbca0700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4fc405bc80 0x7f4fc4067c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:01.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.755+0000 7f4fcbca0700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4fc40608c0 0x7f4fc4060cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:01.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.755+0000 7f4fcbca0700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fc40669e0 con 0x7f4fc4065780 2020-11-20T11:41:01.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.755+0000 7f4fcbca0700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fc40674a0 con 0x7f4fc405bc80 2020-11-20T11:41:01.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.755+0000 7f4fcbca0700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc405c060 con 0x7f4fc40608c0 2020-11-20T11:41:01.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.756+0000 7f4fc9a3c700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4fc405bc80 0x7f4fc4067c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:01.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.756+0000 7f4fc9a3c700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4fc405bc80 0x7f4fc4067c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:37404/0 (socket says 172.21.15.23:37404) 2020-11-20T11:41:01.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.756+0000 7f4fc9a3c700 1 -- 172.21.15.23:0/1904253963 learned_addr learned my addr 172.21.15.23:0/1904253963 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:41:01.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.756+0000 7f4fca23d700 1 -- 172.21.15.23:0/1904253963 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4fc40608c0 msgr2=0x7f4fc4060cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:41:01.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.756+0000 7f4fca23d700 1 --2- 172.21.15.23:0/1904253963 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4fc40608c0 0x7f4fc4060cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:41:01.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.756+0000 7f4fc9a3c700 1 -- 172.21.15.23:0/1904253963 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4fc40608c0 msgr2=0x7f4fc4060cd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:41:01.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.756+0000 7f4fc9a3c700 1 --2- 172.21.15.23:0/1904253963 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4fc40608c0 0x7f4fc4060cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:01.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.756+0000 7f4fc9a3c700 1 -- 172.21.15.23:0/1904253963 >> v1:172.21.15.23:6789/0 conn(0x7f4fc4065780 legacy=0x7f4fc40601b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:41:01.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.756+0000 7f4fc9a3c700 1 -- 172.21.15.23:0/1904253963 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc4067800 con 0x7f4fc405bc80 2020-11-20T11:41:01.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.757+0000 7f4fc9a3c700 1 --2- 172.21.15.23:0/1904253963 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4fc405bc80 0x7f4fc4067c30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f4fb8009a10 tx=0x7f4fb8006ef0).ready entity=mon.1 client_cookie=afd8cd81c5d7af25 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:01.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.757+0000 7f4fc8a3a700 1 -- 172.21.15.23:0/1904253963 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb800b820 con 0x7f4fc405bc80 2020-11-20T11:41:01.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.757+0000 7f4fc8a3a700 1 -- 172.21.15.23:0/1904253963 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fb8000e90 con 0x7f4fc405bc80 2020-11-20T11:41:01.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.757+0000 7f4fc8a3a700 1 -- 172.21.15.23:0/1904253963 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb800bb90 con 0x7f4fc405bc80 2020-11-20T11:41:01.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.758+0000 7f4fcbca0700 1 -- 172.21.15.23:0/1904253963 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4fc405bc80 msgr2=0x7f4fc4067c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:01.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.758+0000 7f4fcbca0700 1 --2- 172.21.15.23:0/1904253963 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4fc405bc80 0x7f4fc4067c30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f4fb8009a10 tx=0x7f4fb8006ef0).stop 2020-11-20T11:41:01.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.762+0000 7f4fcbca0700 1 -- 172.21.15.23:0/1904253963 shutdown_connections 2020-11-20T11:41:01.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.762+0000 7f4fcbca0700 1 --2- 172.21.15.23:0/1904253963 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4fc40608c0 0x7f4fc4060cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:01.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.762+0000 7f4fcbca0700 1 --2- 172.21.15.23:0/1904253963 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4fc405bc80 0x7f4fc4067c30 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:01.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.762+0000 7f4fcbca0700 1 -- 172.21.15.23:0/1904253963 >> 172.21.15.23:0/1904253963 conn(0x7f4fc40ad7a0 msgr2=0x7f4fc40adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:41:01.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.763+0000 7f4fcbca0700 1 -- 172.21.15.23:0/1904253963 shutdown_connections 2020-11-20T11:41:01.769 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.763+0000 7f4fcbca0700 1 -- 172.21.15.23:0/1904253963 wait complete. 2020-11-20T11:41:01.769 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.765+0000 7f4fcbca0700 1 Processor -- start 2020-11-20T11:41:01.770 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.765+0000 7f4fcbca0700 1 -- start start 2020-11-20T11:41:01.771 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.766+0000 7f4fcbca0700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4fc405bc80 0x7f4fc40a3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:01.771 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.766+0000 7f4fcbca0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4fc40608c0 0x7f4fc409f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:01.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.766+0000 7f4fcbca0700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4fc4065780 0x7f4fc409fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:01.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.766+0000 7f4fcbca0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc40a02d0 con 0x7f4fc40608c0 2020-11-20T11:41:01.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.766+0000 7f4fcbca0700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc409ea70 con 0x7f4fc405bc80 2020-11-20T11:41:01.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.766+0000 7f4fcbca0700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc409ed10 con 0x7f4fc4065780 2020-11-20T11:41:01.773 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.766+0000 7f4fc923b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4fc40608c0 0x7f4fc409f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:01.773 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.766+0000 7f4fc923b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4fc40608c0 0x7f4fc409f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44516/0 (socket says 172.21.15.23:44516) 2020-11-20T11:41:01.773 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.766+0000 7f4fc923b700 1 -- 172.21.15.23:0/4218831769 learned_addr learned my addr 172.21.15.23:0/4218831769 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:41:01.773 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.766+0000 7f4fc9a3c700 1 --2- 172.21.15.23:0/4218831769 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4fc405bc80 0x7f4fc40a3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:01.773 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.766+0000 7f4fca23d700 1 -- 172.21.15.23:0/4218831769 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4fc4065780 msgr2=0x7f4fc409fde0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:41:01.774 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.766+0000 7f4fca23d700 1 --2- 172.21.15.23:0/4218831769 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4fc4065780 0x7f4fc409fde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:41:01.774 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.767+0000 7f4fc9a3c700 1 -- 172.21.15.23:0/4218831769 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4fc4065780 msgr2=0x7f4fc409fde0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:41:01.774 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.767+0000 7f4fc9a3c700 1 --2- 172.21.15.23:0/4218831769 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4fc4065780 0x7f4fc409fde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:01.774 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.767+0000 7f4fc9a3c700 1 -- 172.21.15.23:0/4218831769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4fc40608c0 msgr2=0x7f4fc409f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:01.774 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.767+0000 7f4fc9a3c700 1 --2- 172.21.15.23:0/4218831769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4fc40608c0 0x7f4fc409f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:01.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.767+0000 7f4fc9a3c700 1 -- 172.21.15.23:0/4218831769 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc409efb0 con 0x7f4fc405bc80 2020-11-20T11:41:01.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.767+0000 7f4fc9a3c700 1 --2- 172.21.15.23:0/4218831769 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4fc405bc80 0x7f4fc40a3540 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f4fb8013000 tx=0x7f4fb8016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:01.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.767+0000 7f4fb5ffb700 1 -- 172.21.15.23:0/4218831769 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb8011d00 con 0x7f4fc405bc80 2020-11-20T11:41:01.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.767+0000 7f4fcbca0700 1 -- 172.21.15.23:0/4218831769 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fc409e6f0 con 0x7f4fc405bc80 2020-11-20T11:41:01.776 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.768+0000 7f4fcbca0700 1 -- 172.21.15.23:0/4218831769 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fc4140ef0 con 0x7f4fc405bc80 2020-11-20T11:41:01.777 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.769+0000 7f4fcbca0700 1 --2- 172.21.15.23:0/4218831769 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4fc41300c0 0x7f4fc4128520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:01.777 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.769+0000 7f4fb5ffb700 1 -- 172.21.15.23:0/4218831769 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fb8011e60 con 0x7f4fc405bc80 2020-11-20T11:41:01.778 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.769+0000 7f4fcbca0700 1 -- 172.21.15.23:0/4218831769 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc409e8e0 con 0x7f4fc41300c0 2020-11-20T11:41:01.778 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.769+0000 7f4fb5ffb700 1 -- 172.21.15.23:0/4218831769 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb801c670 con 0x7f4fc405bc80 2020-11-20T11:41:01.778 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.769+0000 7f4fc923b700 1 --2- 172.21.15.23:0/4218831769 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4fc41300c0 0x7f4fc4128520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:01.778 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.769+0000 7f4fc923b700 1 -- 172.21.15.23:0/4218831769 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4fc40603f0 con 0x7f4fc41300c0 2020-11-20T11:41:01.779 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.770+0000 7f4fc923b700 1 --2- 172.21.15.23:0/4218831769 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4fc41300c0 0x7f4fc4128520 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f4fc40a3d10 tx=0x7f4fc0008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:01.779 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.770+0000 7f4fb5ffb700 1 -- 172.21.15.23:0/4218831769 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4fb802c070 con 0x7f4fc405bc80 2020-11-20T11:41:01.779 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.771+0000 7f4fb5ffb700 1 --2- 172.21.15.23:0/4218831769 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4f9c056d10 0x7f4f9c0591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:01.779 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.771+0000 7f4fb5ffb700 1 -- 172.21.15.23:0/4218831769 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f4fb8071cb0 con 0x7f4fc405bc80 2020-11-20T11:41:01.779 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.771+0000 7f4fca23d700 1 --2- 172.21.15.23:0/4218831769 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4f9c056d10 0x7f4f9c0591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:01.780 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.772+0000 7f4fca23d700 1 --2- 172.21.15.23:0/4218831769 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4f9c056d10 0x7f4f9c0591d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f4fc4140630 tx=0x7f4fbc006d80).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:01.780 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.774+0000 7f4fb5ffb700 1 -- 172.21.15.23:0/4218831769 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc000ce20 con 0x7f4fc41300c0 2020-11-20T11:41:01.780 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.774+0000 7f4fb5ffb700 1 -- 172.21.15.23:0/4218831769 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f4fc000b8a0 con 0x7f4fc41300c0 2020-11-20T11:41:01.780 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.774+0000 7f4fb5ffb700 1 -- 172.21.15.23:0/4218831769 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4fc41300c0 msgr2=0x7f4fc4128520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:01.781 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.774+0000 7f4fb5ffb700 1 --2- 172.21.15.23:0/4218831769 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4fc41300c0 0x7f4fc4128520 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f4fc40a3d10 tx=0x7f4fc0008040).stop 2020-11-20T11:41:01.781 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.774+0000 7f4fb5ffb700 1 -- 172.21.15.23:0/4218831769 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4fc41300c0 msgr2=0x7f4fc4128520 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:41:01.781 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.774+0000 7f4fb5ffb700 1 --2- 172.21.15.23:0/4218831769 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4fc41300c0 0x7f4fc4128520 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0x7f4fc40a3d10 tx=0x7f4fc0008040).stop 2020-11-20T11:41:01.789 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.786+0000 7f4fcbca0700 1 --2- 172.21.15.23:0/4218831769 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4fc4128a10 0x7f4fc4128e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:01.790 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.786+0000 7f4fcbca0700 1 -- 172.21.15.23:0/4218831769 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc409ee50 con 0x7f4fc4128a10 2020-11-20T11:41:01.790 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.786+0000 7f4fc923b700 1 --2- 172.21.15.23:0/4218831769 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4fc4128a10 0x7f4fc4128e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:01.791 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.786+0000 7f4fc923b700 1 -- 172.21.15.23:0/4218831769 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4fc409f140 con 0x7f4fc4128a10 2020-11-20T11:41:01.791 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.786+0000 7f4fc923b700 1 --2- 172.21.15.23:0/4218831769 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4fc4128a10 0x7f4fc4128e20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f4fc40a3d10 tx=0x7f4fc0018000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:01.792 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.787+0000 7f4fb5ffb700 1 -- 172.21.15.23:0/4218831769 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc0013d00 con 0x7f4fc4128a10 2020-11-20T11:41:01.793 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.789+0000 7f4fb5ffb700 1 -- 172.21.15.23:0/4218831769 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4602 (secure 0 0 0) 0x7f4fc0013e60 con 0x7f4fc4128a10 2020-11-20T11:41:01.793 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.789+0000 7f4fb5ffb700 1 -- 172.21.15.23:0/4218831769 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4fc4128a10 msgr2=0x7f4fc4128e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:01.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.789+0000 7f4fb5ffb700 1 --2- 172.21.15.23:0/4218831769 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4fc4128a10 0x7f4fc4128e20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f4fc40a3d10 tx=0x7f4fc0018000).stop 2020-11-20T11:41:01.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.789+0000 7f4fb5ffb700 1 -- 172.21.15.23:0/4218831769 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4fc4128a10 msgr2=0x7f4fc4128e20 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:41:01.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.789+0000 7f4fb5ffb700 1 --2- 172.21.15.23:0/4218831769 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4fc4128a10 0x7f4fc4128e20 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0x7f4fc40a3d10 tx=0x7f4fc0018000).stop 2020-11-20T11:41:01.794 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:41:01.795 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:41:01.795 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:41:01.795 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:41:01.795 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 86, 2020-11-20T11:41:01.795 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:41:01.796 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:41:01.796 INFO:teuthology.orchestra.run.smithi023.stdout: 1 2020-11-20T11:41:01.796 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:01.796 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 2, 2020-11-20T11:41:01.796 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:41:01.797 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:41:01.797 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:41:01.797 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:41:01.797 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:41:01.797 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:41:01.798 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:41:01.798 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:41:01.798 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:41:01.798 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:41:01.798 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:41:01.799 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:01.799 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:41:01.799 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:41:01.799 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:41:01.799 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:41:01.800 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:41:01.800 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:41:01.800 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:41:01.800 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:41:01.800 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:41:01.801 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:41:01.801 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:01.801 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:01.801 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:41:01.801 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:41:01.802 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:41:01.802 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:41:01.802 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:41:01.802 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:41:01.802 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:41:01.803 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:41:01.803 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:41:01.803 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:41:01.803 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:41:01.803 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:41:01.803 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:41:01.804 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:41:01.804 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:41:01.804 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:41:01.804 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:41:01.804 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:41:01.805 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:41:01.805 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:41:01.805 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:41:01.805 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:41:01.805 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:41:01.806 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:01.806 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:41:01.806 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:01.806 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:41:01.806 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:01.807 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:41:01.807 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:41:01.807 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:41:01.807 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:41:01.807 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:01.808 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:41:01.808 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:41:01.808 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:01.808 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:01.808 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:01.809 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:41:01.809 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:41:01.809 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:01.809 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:01.809 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:01.810 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:01.810 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:41:01.810 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:41:01.810 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:41:01.810 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:41:01.811 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:41:01.811 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:01.811 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:01.811 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:41:01.812 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:41:01.812 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:41:01.812 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:41:01.812 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:01.812 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:41:01.813 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:41:01.813 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:01.813 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:01.813 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:01.813 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:41:01.814 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:41:01.814 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:01.814 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:01.814 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:01.814 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:01.815 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:41:01.815 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:41:01.815 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:41:01.815 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:41:01.815 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:41:01.816 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:01.816 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:01.816 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:41:01.816 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:41:01.816 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:41:01.817 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:41:01.817 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:01.817 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:41:01.817 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:41:01.817 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:01.818 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:01.818 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:01.818 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:41:01.818 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:41:01.818 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:01.819 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:01.819 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:01.819 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:01.819 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:41:01.819 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:41:01.820 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:41:01.820 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:41:01.820 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:41:01.820 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:01.820 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:01.821 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:01.821 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:41:01.821 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:41:01.821 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:01.821 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:41:01.822 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:41:01.822 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:41:01.822 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:01.822 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:01.822 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:41:01.823 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:01.823 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:41:01.823 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:41:01.823 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:41:01.823 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:01.824 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:01.824 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:41:01.824 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:01.824 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:41:01.824 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:41:01.825 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:41:01.825 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:01.825 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:01.825 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:41:01.825 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:01.826 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:41:01.826 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:41:01.826 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:41:01.826 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:01.826 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:01.827 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:01.827 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:41:01.827 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:41:01.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.792+0000 7f4fcbca0700 1 -- 172.21.15.23:0/4218831769 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4f9c056d10 msgr2=0x7f4f9c0591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:01.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.792+0000 7f4fcbca0700 1 --2- 172.21.15.23:0/4218831769 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4f9c056d10 0x7f4f9c0591d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f4fc4140630 tx=0x7f4fbc006d80).stop 2020-11-20T11:41:01.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.792+0000 7f4fcbca0700 1 -- 172.21.15.23:0/4218831769 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4fc405bc80 msgr2=0x7f4fc40a3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:01.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.792+0000 7f4fcbca0700 1 --2- 172.21.15.23:0/4218831769 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4fc405bc80 0x7f4fc40a3540 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f4fb8013000 tx=0x7f4fb8016000).stop 2020-11-20T11:41:01.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.792+0000 7f4fca23d700 1 -- 172.21.15.23:0/4218831769 reap_dead start 2020-11-20T11:41:01.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.792+0000 7f4fca23d700 1 -- 172.21.15.23:0/4218831769 reap_dead start 2020-11-20T11:41:01.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.793+0000 7f4fcbca0700 1 -- 172.21.15.23:0/4218831769 shutdown_connections 2020-11-20T11:41:01.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.793+0000 7f4fcbca0700 1 -- 172.21.15.23:0/4218831769 >> 172.21.15.23:0/4218831769 conn(0x7f4fc40ad7a0 msgr2=0x7f4fc4067b50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:41:01.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.794+0000 7f4fcbca0700 1 -- 172.21.15.23:0/4218831769 shutdown_connections 2020-11-20T11:41:01.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:01.794+0000 7f4fcbca0700 1 -- 172.21.15.23:0/4218831769 wait complete. 2020-11-20T11:41:01.846 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2020-11-20T11:41:21.847 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2020-11-20T11:41:21.847 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2020-11-20T11:41:21.848 INFO:tasks.ceph.mon.c:Restarting daemon 2020-11-20T11:41:21.848 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2020-11-20T11:41:21.852 INFO:tasks.ceph.mon.c:Started 2020-11-20T11:41:21.852 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:41:21.852 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:41:31.196 INFO:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:41:31.201 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:41:31.780 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.774+0000 7fa92e21b700 1 Processor -- start 2020-11-20T11:41:31.780 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.774+0000 7fa92e21b700 1 -- start start 2020-11-20T11:41:31.781 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.775+0000 7fa92e21b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9280acca0 0x7fa9280a7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:31.781 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.775+0000 7fa92e21b700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa9280a9740 0x7fa9280a9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:31.781 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.775+0000 7fa92e21b700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9280a5a30 con 0x7fa9280ad080 2020-11-20T11:41:31.781 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.775+0000 7fa92e21b700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9280a7880 con 0x7fa9280a9740 2020-11-20T11:41:31.781 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.775+0000 7fa92e21b700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9280a7b20 con 0x7fa9280acca0 2020-11-20T11:41:31.782 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.775+0000 7fa92ca18700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fa9280ad080 0x7fa9280a5240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:36754/0 (socket says 172.21.15.23:36754) 2020-11-20T11:41:31.782 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.775+0000 7fa92ca18700 1 -- 172.21.15.23:0/3569567843 learned_addr learned my addr 172.21.15.23:0/3569567843 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:41:31.782 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.775+0000 7fa92d219700 1 --2- 172.21.15.23:0/3569567843 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9280acca0 0x7fa9280a7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:31.782 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.776+0000 7fa92d219700 1 -- 172.21.15.23:0/3569567843 >> v2:172.21.15.155:3300/0 conn(0x7fa9280a9740 msgr2=0x7fa9280a9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:41:31.783 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.776+0000 7fa92d219700 1 --2- 172.21.15.23:0/3569567843 >> v2:172.21.15.155:3300/0 conn(0x7fa9280a9740 0x7fa9280a9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:31.783 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.776+0000 7fa92d219700 1 -- 172.21.15.23:0/3569567843 >> v1:172.21.15.155:6789/0 conn(0x7fa9280ad080 legacy=0x7fa9280a5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:31.783 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.776+0000 7fa92d219700 1 -- 172.21.15.23:0/3569567843 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9280a84b0 con 0x7fa9280acca0 2020-11-20T11:41:31.783 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.776+0000 7fa92d219700 1 --2- 172.21.15.23:0/3569567843 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9280acca0 0x7fa9280a7fc0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fa924009dd0 tx=0x7fa924006e30).ready entity=mon.2 client_cookie=952f7278e74d668c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:31.783 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.777+0000 7fa91ffff700 1 -- 172.21.15.23:0/3569567843 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa92400b760 con 0x7fa9280acca0 2020-11-20T11:41:31.784 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.777+0000 7fa91ffff700 1 -- 172.21.15.23:0/3569567843 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa924000e90 con 0x7fa9280acca0 2020-11-20T11:41:31.784 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.777+0000 7fa92e21b700 1 -- 172.21.15.23:0/3569567843 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9280acca0 msgr2=0x7fa9280a7fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:31.785 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.777+0000 7fa92e21b700 1 --2- 172.21.15.23:0/3569567843 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9280acca0 0x7fa9280a7fc0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fa924009dd0 tx=0x7fa924006e30).stop 2020-11-20T11:41:31.788 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.784+0000 7fa92e21b700 1 -- 172.21.15.23:0/3569567843 shutdown_connections 2020-11-20T11:41:31.788 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.784+0000 7fa92e21b700 1 --2- 172.21.15.23:0/3569567843 >> v2:172.21.15.155:3300/0 conn(0x7fa9280a9740 0x7fa9280a9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:31.788 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.784+0000 7fa92e21b700 1 --2- 172.21.15.23:0/3569567843 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9280acca0 0x7fa9280a7fc0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:31.788 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.784+0000 7fa92e21b700 1 -- 172.21.15.23:0/3569567843 >> 172.21.15.23:0/3569567843 conn(0x7fa928061850 msgr2=0x7fa928061c60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:41:31.789 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.785+0000 7fa92e21b700 1 -- 172.21.15.23:0/3569567843 shutdown_connections 2020-11-20T11:41:31.789 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.785+0000 7fa92e21b700 1 -- 172.21.15.23:0/3569567843 wait complete. 2020-11-20T11:41:31.790 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.787+0000 7fa92e21b700 1 Processor -- start 2020-11-20T11:41:31.790 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.787+0000 7fa92e21b700 1 -- start start 2020-11-20T11:41:31.797 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.788+0000 7fa92e21b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9280a9740 0x7fa92809d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:31.802 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.788+0000 7fa92e21b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9280acca0 0x7fa92815f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:31.803 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.788+0000 7fa92e21b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9280ad080 0x7fa928072730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:31.803 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.788+0000 7fa92e21b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa92809dfb0 con 0x7fa9280acca0 2020-11-20T11:41:31.803 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.788+0000 7fa92e21b700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa92809dc70 con 0x7fa9280a9740 2020-11-20T11:41:31.803 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.788+0000 7fa92e21b700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa92807e4e0 con 0x7fa9280ad080 2020-11-20T11:41:31.804 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.789+0000 7fa92da1a700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9280ad080 0x7fa928072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:31.804 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.789+0000 7fa92ca18700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9280acca0 0x7fa92815f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:31.804 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.789+0000 7fa92da1a700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9280ad080 0x7fa928072730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:54778/0 (socket says 172.21.15.23:54778) 2020-11-20T11:41:31.804 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.789+0000 7fa92da1a700 1 -- 172.21.15.23:0/181727569 learned_addr learned my addr 172.21.15.23:0/181727569 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:41:31.804 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.789+0000 7fa92d219700 1 --2- 172.21.15.23:0/181727569 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9280a9740 0x7fa92809d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:31.805 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.789+0000 7fa92da1a700 1 -- 172.21.15.23:0/181727569 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9280a9740 msgr2=0x7fa92809d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:31.805 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.789+0000 7fa92da1a700 1 --2- 172.21.15.23:0/181727569 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9280a9740 0x7fa92809d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:31.805 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.789+0000 7fa92da1a700 1 -- 172.21.15.23:0/181727569 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9280acca0 msgr2=0x7fa92815f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:31.805 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.789+0000 7fa92da1a700 1 --2- 172.21.15.23:0/181727569 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9280acca0 0x7fa92815f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:31.805 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.789+0000 7fa92da1a700 1 -- 172.21.15.23:0/181727569 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa92807e6a0 con 0x7fa9280ad080 2020-11-20T11:41:31.806 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.789+0000 7fa92ca18700 1 --2- 172.21.15.23:0/181727569 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9280acca0 0x7fa92815f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:41:31.806 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.790+0000 7fa92da1a700 1 --2- 172.21.15.23:0/181727569 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9280ad080 0x7fa928072730 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fa918009690 tx=0x7fa918006de0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:31.806 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.791+0000 7fa91d7fa700 1 -- 172.21.15.23:0/181727569 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa918013460 con 0x7fa9280ad080 2020-11-20T11:41:31.806 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.791+0000 7fa91d7fa700 1 -- 172.21.15.23:0/181727569 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9180135c0 con 0x7fa9280ad080 2020-11-20T11:41:31.806 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.791+0000 7fa92e21b700 1 -- 172.21.15.23:0/181727569 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa92806efb0 con 0x7fa9280ad080 2020-11-20T11:41:31.807 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.791+0000 7fa91d7fa700 1 -- 172.21.15.23:0/181727569 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa918013980 con 0x7fa9280ad080 2020-11-20T11:41:31.807 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.791+0000 7fa92e21b700 1 -- 172.21.15.23:0/181727569 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa92809ef70 con 0x7fa9280ad080 2020-11-20T11:41:31.807 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.794+0000 7fa91d7fa700 1 -- 172.21.15.23:0/181727569 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa91800f070 con 0x7fa9280ad080 2020-11-20T11:41:31.807 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.794+0000 7fa91d7fa700 1 --2- 172.21.15.23:0/181727569 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa904056d60 0x7fa904059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:31.807 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.795+0000 7fa91d7fa700 1 -- 172.21.15.23:0/181727569 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fa91806f910 con 0x7fa9280ad080 2020-11-20T11:41:31.808 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.795+0000 7fa92d219700 1 --2- 172.21.15.23:0/181727569 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa904056d60 0x7fa904059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:31.808 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.795+0000 7fa92d219700 1 --2- 172.21.15.23:0/181727569 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa904056d60 0x7fa904059220 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fa92400a010 tx=0x7fa924018000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:31.824 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.820+0000 7fa92e21b700 1 -- 172.21.15.23:0/181727569 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8fc004fd0 con 0x7fa9280ad080 2020-11-20T11:41:31.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:31.826+0000 7fa91d7fa700 1 -- 172.21.15.23:0/181727569 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa918044af0 con 0x7fa9280ad080 2020-11-20T11:41:32.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.367+0000 7fa92e21b700 1 -- 172.21.15.23:0/181727569 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa8fc001670 con 0x7fa9280ad080 2020-11-20T11:41:32.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.367+0000 7fa91d7fa700 1 -- 172.21.15.23:0/181727569 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1477 (secure 0 0 0) 0x7fa91804c070 con 0x7fa9280ad080 2020-11-20T11:41:32.371 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:41:32.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.383+0000 7fa92e21b700 1 -- 172.21.15.23:0/181727569 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa904056d60 msgr2=0x7fa904059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:32.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.384+0000 7fa92e21b700 1 --2- 172.21.15.23:0/181727569 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa904056d60 0x7fa904059220 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fa92400a010 tx=0x7fa924018000).stop 2020-11-20T11:41:32.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.384+0000 7fa92e21b700 1 -- 172.21.15.23:0/181727569 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9280ad080 msgr2=0x7fa928072730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:32.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.384+0000 7fa92e21b700 1 --2- 172.21.15.23:0/181727569 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9280ad080 0x7fa928072730 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fa918009690 tx=0x7fa918006de0).stop 2020-11-20T11:41:32.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.385+0000 7fa92e21b700 1 -- 172.21.15.23:0/181727569 shutdown_connections 2020-11-20T11:41:32.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.385+0000 7fa92e21b700 1 --2- 172.21.15.23:0/181727569 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa904056d60 0x7fa904059220 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:32.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.385+0000 7fa92e21b700 1 --2- 172.21.15.23:0/181727569 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa9280ad080 0x7fa928072730 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:32.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.385+0000 7fa92e21b700 1 --2- 172.21.15.23:0/181727569 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa9280a9740 0x7fa92809d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:32.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.385+0000 7fa92e21b700 1 --2- 172.21.15.23:0/181727569 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9280acca0 0x7fa92815f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:32.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.385+0000 7fa92e21b700 1 -- 172.21.15.23:0/181727569 >> 172.21.15.23:0/181727569 conn(0x7fa928061850 msgr2=0x7fa928061c30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:41:32.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.386+0000 7fa92e21b700 1 -- 172.21.15.23:0/181727569 shutdown_connections 2020-11-20T11:41:32.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.388+0000 7fa92e21b700 1 -- 172.21.15.23:0/181727569 wait complete. 2020-11-20T11:41:32.438 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:41:32.438 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:41:32.811 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.805+0000 7faba8848700 1 Processor -- start 2020-11-20T11:41:32.812 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.805+0000 7faba8848700 1 -- start start 2020-11-20T11:41:32.812 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.806+0000 7faba8848700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7faba4065140 0x7faba40a8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:32.812 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.806+0000 7faba8848700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faba40a8e50 0x7faba4151b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:32.812 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.806+0000 7faba8848700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba40a92e0 con 0x7faba4064d60 2020-11-20T11:41:32.812 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.806+0000 7faba8848700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faba40a7f80 con 0x7faba4065140 2020-11-20T11:41:32.813 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.806+0000 7faba8848700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faba40a8220 con 0x7faba40a8e50 2020-11-20T11:41:32.813 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.806+0000 7faba259c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7faba4064d60 0x7faba40a7750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58794/0 (socket says 172.21.15.23:58794) 2020-11-20T11:41:32.813 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.806+0000 7faba259c700 1 -- 172.21.15.23:0/3873804276 learned_addr learned my addr 172.21.15.23:0/3873804276 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:41:32.813 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.806+0000 7faba1d9b700 1 --2- 172.21.15.23:0/3873804276 >> v2:172.21.15.155:3300/0 conn(0x7faba4065140 0x7faba40a8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:32.814 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.806+0000 7faba2d9d700 1 --2- 172.21.15.23:0/3873804276 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faba40a8e50 0x7faba4151b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:32.814 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.807+0000 7faba159a700 1 -- 172.21.15.23:0/3873804276 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1865286986 0 0) 0x7faba40a92e0 con 0x7faba4064d60 2020-11-20T11:41:32.814 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.807+0000 7faba159a700 1 -- 172.21.15.23:0/3873804276 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab8c003280 con 0x7faba4064d60 2020-11-20T11:41:32.814 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.807+0000 7faba1d9b700 1 -- 172.21.15.23:0/3873804276 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faba40a8e50 msgr2=0x7faba4151b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:41:32.814 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.807+0000 7faba1d9b700 1 --2- 172.21.15.23:0/3873804276 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faba40a8e50 0x7faba4151b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:32.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.807+0000 7faba1d9b700 1 -- 172.21.15.23:0/3873804276 >> v1:172.21.15.23:6789/0 conn(0x7faba4064d60 legacy=0x7faba40a7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:32.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.807+0000 7faba1d9b700 1 -- 172.21.15.23:0/3873804276 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba40a84c0 con 0x7faba4065140 2020-11-20T11:41:32.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.807+0000 7faba1d9b700 1 --2- 172.21.15.23:0/3873804276 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faba4065140 0x7faba40a8960 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fab9c005eb0 tx=0x7fab9c00ac90).ready entity=mon.1 client_cookie=aa794b2cd88a7e10 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:32.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.807+0000 7faba159a700 1 -- 172.21.15.23:0/3873804276 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9c008a90 con 0x7faba4065140 2020-11-20T11:41:32.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.807+0000 7faba159a700 1 -- 172.21.15.23:0/3873804276 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab9c008bf0 con 0x7faba4065140 2020-11-20T11:41:32.816 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.807+0000 7faba159a700 1 -- 172.21.15.23:0/3873804276 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9c00a0d0 con 0x7faba4065140 2020-11-20T11:41:32.816 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.808+0000 7faba8848700 1 -- 172.21.15.23:0/3873804276 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faba4065140 msgr2=0x7faba40a8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:32.816 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.808+0000 7faba8848700 1 --2- 172.21.15.23:0/3873804276 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faba4065140 0x7faba40a8960 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fab9c005eb0 tx=0x7fab9c00ac90).stop 2020-11-20T11:41:32.818 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.811+0000 7faba8848700 1 -- 172.21.15.23:0/3873804276 shutdown_connections 2020-11-20T11:41:32.818 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.811+0000 7faba8848700 1 --2- 172.21.15.23:0/3873804276 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faba40a8e50 0x7faba4151b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:32.819 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.811+0000 7faba8848700 1 --2- 172.21.15.23:0/3873804276 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faba4065140 0x7faba40a8960 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:32.819 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.811+0000 7faba8848700 1 -- 172.21.15.23:0/3873804276 >> 172.21.15.23:0/3873804276 conn(0x7faba40601b0 msgr2=0x7faba40605c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:41:32.819 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.812+0000 7faba8848700 1 -- 172.21.15.23:0/3873804276 shutdown_connections 2020-11-20T11:41:32.819 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.812+0000 7faba8848700 1 -- 172.21.15.23:0/3873804276 wait complete. 2020-11-20T11:41:32.820 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.813+0000 7faba8848700 1 Processor -- start 2020-11-20T11:41:32.820 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.814+0000 7faba8848700 1 -- start start 2020-11-20T11:41:32.820 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.814+0000 7faba8848700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faba4064d60 0x7faba406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:32.820 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.814+0000 7faba8848700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faba4065140 0x7faba40a25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:32.820 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.814+0000 7faba8848700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faba40a8e50 0x7faba409f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:32.820 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.814+0000 7faba8848700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba40a0f80 con 0x7faba4065140 2020-11-20T11:41:32.821 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.814+0000 7faba8848700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba407efc0 con 0x7faba4064d60 2020-11-20T11:41:32.821 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.814+0000 7faba8848700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faba40a0160 con 0x7faba40a8e50 2020-11-20T11:41:32.821 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.815+0000 7faba1d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faba4065140 0x7faba40a25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:32.822 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.815+0000 7faba1d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faba4065140 0x7faba40a25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44552/0 (socket says 172.21.15.23:44552) 2020-11-20T11:41:32.822 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.815+0000 7faba1d9b700 1 -- 172.21.15.23:0/1762357068 learned_addr learned my addr 172.21.15.23:0/1762357068 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:41:32.822 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.815+0000 7faba259c700 1 --2- 172.21.15.23:0/1762357068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faba4064d60 0x7faba406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:32.822 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.815+0000 7faba259c700 1 -- 172.21.15.23:0/1762357068 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faba40a8e50 msgr2=0x7faba409f480 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2020-11-20T11:41:32.822 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.815+0000 7faba259c700 1 --2- 172.21.15.23:0/1762357068 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faba40a8e50 0x7faba409f480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:32.823 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.815+0000 7faba259c700 1 -- 172.21.15.23:0/1762357068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faba4065140 msgr2=0x7faba40a25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:32.823 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.815+0000 7faba259c700 1 --2- 172.21.15.23:0/1762357068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faba4065140 0x7faba40a25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:32.823 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.815+0000 7faba259c700 1 -- 172.21.15.23:0/1762357068 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba40a11b0 con 0x7faba4064d60 2020-11-20T11:41:32.823 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.816+0000 7faba259c700 1 --2- 172.21.15.23:0/1762357068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faba4064d60 0x7faba406efb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fab9400c710 tx=0x7fab94007a60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:32.825 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.816+0000 7fab927fc700 1 -- 172.21.15.23:0/1762357068 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab94007320 con 0x7faba4064d60 2020-11-20T11:41:32.825 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.816+0000 7fab927fc700 1 -- 172.21.15.23:0/1762357068 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab94007480 con 0x7faba4064d60 2020-11-20T11:41:32.825 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.816+0000 7fab927fc700 1 -- 172.21.15.23:0/1762357068 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab94005860 con 0x7faba4064d60 2020-11-20T11:41:32.825 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.816+0000 7faba8848700 1 -- 172.21.15.23:0/1762357068 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faba40a1450 con 0x7faba4064d60 2020-11-20T11:41:32.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.816+0000 7faba8848700 1 -- 172.21.15.23:0/1762357068 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faba406f4a0 con 0x7faba4064d60 2020-11-20T11:41:32.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.818+0000 7fab927fc700 1 -- 172.21.15.23:0/1762357068 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fab94010030 con 0x7faba4064d60 2020-11-20T11:41:32.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.818+0000 7fab927fc700 1 --2- 172.21.15.23:0/1762357068 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fab74056db0 0x7fab74059270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:32.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.819+0000 7fab927fc700 1 -- 172.21.15.23:0/1762357068 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fab9406fde0 con 0x7faba4064d60 2020-11-20T11:41:32.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.819+0000 7faba1d9b700 1 --2- 172.21.15.23:0/1762357068 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fab74056db0 0x7fab74059270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:32.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.819+0000 7faba1d9b700 1 --2- 172.21.15.23:0/1762357068 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fab74056db0 0x7fab74059270 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fab9c0070b0 tx=0x7fab9c019000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:32.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.820+0000 7faba8848700 1 --2- 172.21.15.23:0/1762357068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faba40611c0 0x7faba4156ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:32.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.820+0000 7faba8848700 1 -- 172.21.15.23:0/1762357068 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba4061650 con 0x7faba40611c0 2020-11-20T11:41:32.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.821+0000 7faba2d9d700 1 --2- 172.21.15.23:0/1762357068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faba40611c0 0x7faba4156ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:32.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.822+0000 7faba2d9d700 1 -- 172.21.15.23:0/1762357068 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faba4057dc0 con 0x7faba40611c0 2020-11-20T11:41:32.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.822+0000 7faba2d9d700 1 --2- 172.21.15.23:0/1762357068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faba40611c0 0x7faba4156ee0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7faba40a3ee0 tx=0x7fab98007870).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:32.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.822+0000 7fab927fc700 1 -- 172.21.15.23:0/1762357068 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab98007320 con 0x7faba40611c0 2020-11-20T11:41:32.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.822+0000 7fab927fc700 1 -- 172.21.15.23:0/1762357068 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fab98007480 con 0x7faba40611c0 2020-11-20T11:41:32.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.822+0000 7fab927fc700 1 -- 172.21.15.23:0/1762357068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faba40611c0 msgr2=0x7faba4156ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:32.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.823+0000 7fab927fc700 1 --2- 172.21.15.23:0/1762357068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faba40611c0 0x7faba4156ee0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7faba40a3ee0 tx=0x7fab98007870).stop 2020-11-20T11:41:32.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.823+0000 7fab927fc700 1 -- 172.21.15.23:0/1762357068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faba40611c0 msgr2=0x7faba4156ee0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:41:32.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.823+0000 7fab927fc700 1 --2- 172.21.15.23:0/1762357068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faba40611c0 0x7faba4156ee0 secure :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0x7faba40a3ee0 tx=0x7fab98007870).stop 2020-11-20T11:41:32.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.842+0000 7fab83fff700 1 --2- 172.21.15.23:0/1762357068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faba41573d0 0x7faba40a15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:32.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.842+0000 7fab83fff700 1 -- 172.21.15.23:0/1762357068 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba40aa100 con 0x7faba41573d0 2020-11-20T11:41:32.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.842+0000 7faba2d9d700 1 --2- 172.21.15.23:0/1762357068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faba41573d0 0x7faba40a15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:32.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.843+0000 7faba2d9d700 1 -- 172.21.15.23:0/1762357068 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7faba4044710 con 0x7faba41573d0 2020-11-20T11:41:32.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.843+0000 7faba2d9d700 1 --2- 172.21.15.23:0/1762357068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faba41573d0 0x7faba40a15e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7faba40a3ee0 tx=0x7fab98017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:32.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.843+0000 7fab927fc700 1 -- 172.21.15.23:0/1762357068 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9801d040 con 0x7faba41573d0 2020-11-20T11:41:32.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.844+0000 7fab927fc700 1 -- 172.21.15.23:0/1762357068 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4785 (secure 0 0 0) 0x7fab98014d40 con 0x7faba41573d0 2020-11-20T11:41:32.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.844+0000 7fab927fc700 1 -- 172.21.15.23:0/1762357068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faba41573d0 msgr2=0x7faba40a15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:32.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.844+0000 7fab927fc700 1 --2- 172.21.15.23:0/1762357068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faba41573d0 0x7faba40a15e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7faba40a3ee0 tx=0x7fab98017000).stop 2020-11-20T11:41:32.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.844+0000 7fab927fc700 1 -- 172.21.15.23:0/1762357068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faba41573d0 msgr2=0x7faba40a15e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:41:32.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.844+0000 7fab927fc700 1 --2- 172.21.15.23:0/1762357068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faba41573d0 0x7faba40a15e0 secure :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0x7faba40a3ee0 tx=0x7fab98017000).stop 2020-11-20T11:41:32.851 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:41:32.851 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:41:32.851 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:41:32.851 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:41:32.851 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 88, 2020-11-20T11:41:32.852 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:41:32.852 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:41:32.852 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:41:32.852 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:41:32.852 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:32.853 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 7, 2020-11-20T11:41:32.853 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:41:32.853 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:41:32.853 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:41:32.854 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:41:32.854 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:41:32.854 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:41:32.854 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:41:32.854 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:41:32.854 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:41:32.855 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:41:32.855 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:41:32.855 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:32.855 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:41:32.855 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:41:32.856 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:41:32.856 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:41:32.856 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:41:32.856 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:41:32.856 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:41:32.857 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:41:32.857 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:41:32.857 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:41:32.857 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:32.858 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:32.858 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:41:32.858 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:41:32.858 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:41:32.858 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:41:32.859 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:41:32.859 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:41:32.859 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:41:32.859 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:41:32.859 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:41:32.860 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:41:32.860 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:41:32.860 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:41:32.860 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:41:32.860 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:41:32.861 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:41:32.861 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:41:32.861 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:41:32.861 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:41:32.861 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:41:32.862 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:41:32.862 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:41:32.862 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:41:32.862 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:41:32.862 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:32.863 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:41:32.863 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:32.863 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:41:32.863 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:32.863 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:41:32.863 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:41:32.864 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:41:32.864 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:41:32.864 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:32.864 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:41:32.864 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:41:32.865 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:32.865 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:32.865 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:32.865 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:41:32.865 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:41:32.866 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:32.866 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:32.866 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:32.866 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:32.866 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:41:32.867 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:41:32.867 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:41:32.867 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:41:32.867 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:41:32.867 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:32.868 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:32.868 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:41:32.868 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:41:32.868 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:41:32.868 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:41:32.869 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:32.869 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:41:32.869 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:41:32.869 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:32.869 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:32.870 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:32.870 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:41:32.870 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:41:32.870 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:32.870 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:32.871 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:32.871 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:32.871 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:41:32.871 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:41:32.871 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:41:32.872 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:41:32.872 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:41:32.872 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:32.872 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:32.872 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:41:32.873 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:41:32.873 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:41:32.873 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:41:32.873 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:32.873 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:41:32.874 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:41:32.874 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:32.874 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:32.874 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:32.874 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:41:32.875 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:41:32.875 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:32.875 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:32.875 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:32.875 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:32.876 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:41:32.876 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:41:32.876 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:41:32.876 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:41:32.876 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:41:32.877 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:32.877 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:32.877 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:32.877 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:41:32.877 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:41:32.878 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:32.878 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:41:32.878 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:41:32.878 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:41:32.878 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:32.879 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:32.879 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:41:32.879 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:32.879 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:41:32.879 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:41:32.880 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:41:32.880 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:32.880 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:32.880 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:41:32.880 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:32.881 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:41:32.881 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:41:32.881 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 8 2020-11-20T11:41:32.881 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:32.881 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:32.882 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:41:32.882 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:32.882 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:41:32.882 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:41:32.882 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 4 2020-11-20T11:41:32.883 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:32.883 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:32.883 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:41:32.883 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:32.883 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:41:32.884 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:41:32.884 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:41:32.884 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:32.884 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:32.884 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:32.885 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:41:32.885 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:41:32.886 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.853+0000 7fab83fff700 1 -- 172.21.15.23:0/1762357068 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fab74056db0 msgr2=0x7fab74059270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:32.886 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.853+0000 7fab83fff700 1 --2- 172.21.15.23:0/1762357068 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fab74056db0 0x7fab74059270 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fab9c0070b0 tx=0x7fab9c019000).stop 2020-11-20T11:41:32.886 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.853+0000 7faba2d9d700 1 -- 172.21.15.23:0/1762357068 reap_dead start 2020-11-20T11:41:32.886 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.853+0000 7fab83fff700 1 -- 172.21.15.23:0/1762357068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faba4064d60 msgr2=0x7faba406efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:32.887 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.853+0000 7fab83fff700 1 --2- 172.21.15.23:0/1762357068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faba4064d60 0x7faba406efb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fab9400c710 tx=0x7fab94007a60).stop 2020-11-20T11:41:32.887 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.853+0000 7fab83fff700 1 -- 172.21.15.23:0/1762357068 shutdown_connections 2020-11-20T11:41:32.887 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.853+0000 7fab83fff700 1 --2- 172.21.15.23:0/1762357068 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faba4064d60 0x7faba406efb0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:32.887 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.853+0000 7fab83fff700 1 -- 172.21.15.23:0/1762357068 >> 172.21.15.23:0/1762357068 conn(0x7faba40601b0 msgr2=0x7faba4060590 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:41:32.887 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.853+0000 7fab83fff700 1 -- 172.21.15.23:0/1762357068 shutdown_connections 2020-11-20T11:41:32.888 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:32.853+0000 7fab83fff700 1 -- 172.21.15.23:0/1762357068 wait complete. 2020-11-20T11:41:32.927 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:41:33.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.133+0000 7ff1ab23f700 1 Processor -- start 2020-11-20T11:41:33.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.136+0000 7ff1ab23f700 1 -- start start 2020-11-20T11:41:33.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.137+0000 7ff1ab23f700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff1a40ad080 0x7ff1a40a8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:33.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.137+0000 7ff1ab23f700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a40a5a30 con 0x7ff1a413df30 2020-11-20T11:41:33.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.137+0000 7ff1ab23f700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a40a79d0 con 0x7ff1a40acca0 2020-11-20T11:41:33.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.137+0000 7ff1ab23f700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1a40a9560 con 0x7ff1a40ad080 2020-11-20T11:41:33.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.137+0000 7ff1a9a3c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff1a40ad080 0x7ff1a40a8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:33.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.137+0000 7ff1a9a3c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff1a40ad080 0x7ff1a40a8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44562/0 (socket says 172.21.15.23:44562) 2020-11-20T11:41:33.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.137+0000 7ff1a9a3c700 1 -- 172.21.15.23:0/1554758676 learned_addr learned my addr 172.21.15.23:0/1554758676 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:41:33.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.138+0000 7ff1a9a3c700 1 -- 172.21.15.23:0/1554758676 >> v1:172.21.15.155:6789/0 conn(0x7ff1a40acca0 legacy=0x7ff1a40a8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:41:33.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.138+0000 7ff1a9a3c700 1 -- 172.21.15.23:0/1554758676 >> v1:172.21.15.23:6789/0 conn(0x7ff1a413df30 legacy=0x7ff1a413e3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:41:33.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.138+0000 7ff1a9a3c700 1 -- 172.21.15.23:0/1554758676 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a40a7c70 con 0x7ff1a40ad080 2020-11-20T11:41:33.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.140+0000 7ff1a9a3c700 1 --2- 172.21.15.23:0/1554758676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1a40ad080 0x7ff1a40a8110 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7ff19c009d90 tx=0x7ff19c0092d0).ready entity=mon.0 client_cookie=bf6ba4fd6a8e3b51 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:33.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.140+0000 7ff1a923b700 1 -- 172.21.15.23:0/1554758676 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19c00bbb0 con 0x7ff1a40ad080 2020-11-20T11:41:33.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.141+0000 7ff1a923b700 1 -- 172.21.15.23:0/1554758676 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff19c00bd10 con 0x7ff1a40ad080 2020-11-20T11:41:33.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.142+0000 7ff1ab23f700 1 -- 172.21.15.23:0/1554758676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1a40ad080 msgr2=0x7ff1a40a8110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:33.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.142+0000 7ff1ab23f700 1 --2- 172.21.15.23:0/1554758676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1a40ad080 0x7ff1a40a8110 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7ff19c009d90 tx=0x7ff19c0092d0).stop 2020-11-20T11:41:33.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.142+0000 7ff1ab23f700 1 -- 172.21.15.23:0/1554758676 shutdown_connections 2020-11-20T11:41:33.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.142+0000 7ff1ab23f700 1 --2- 172.21.15.23:0/1554758676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1a40ad080 0x7ff1a40a8110 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:33.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.142+0000 7ff1ab23f700 1 -- 172.21.15.23:0/1554758676 >> 172.21.15.23:0/1554758676 conn(0x7ff1a4061850 msgr2=0x7ff1a4061c60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:41:33.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.143+0000 7ff1ab23f700 1 -- 172.21.15.23:0/1554758676 shutdown_connections 2020-11-20T11:41:33.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.143+0000 7ff1ab23f700 1 -- 172.21.15.23:0/1554758676 wait complete. 2020-11-20T11:41:33.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.144+0000 7ff1ab23f700 1 Processor -- start 2020-11-20T11:41:33.148 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.144+0000 7ff1ab23f700 1 -- start start 2020-11-20T11:41:33.148 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.145+0000 7ff1ab23f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1a40acca0 0x7ff1a4066220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:33.149 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.145+0000 7ff1aa23d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1a40acca0 0x7ff1a4066220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:33.149 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.145+0000 7ff1aa23d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1a40acca0 0x7ff1a4066220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44566/0 (socket says 172.21.15.23:44566) 2020-11-20T11:41:33.149 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.145+0000 7ff1aa23d700 1 -- 172.21.15.23:0/3776242743 learned_addr learned my addr 172.21.15.23:0/3776242743 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:41:33.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.145+0000 7ff1ab23f700 1 --2- 172.21.15.23:0/3776242743 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff1a40ad080 0x7ff1a4137760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:33.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.145+0000 7ff1ab23f700 1 --2- 172.21.15.23:0/3776242743 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1a413df30 0x7ff1a413c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:33.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.145+0000 7ff1ab23f700 1 -- 172.21.15.23:0/3776242743 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a413bb30 con 0x7ff1a40acca0 2020-11-20T11:41:33.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.146+0000 7ff1ab23f700 1 -- 172.21.15.23:0/3776242743 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a413bee0 con 0x7ff1a413df30 2020-11-20T11:41:33.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.146+0000 7ff1ab23f700 1 -- 172.21.15.23:0/3776242743 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a413c180 con 0x7ff1a40ad080 2020-11-20T11:41:33.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.146+0000 7ff1aa23d700 1 -- 172.21.15.23:0/3776242743 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff1a40ad080 msgr2=0x7ff1a4137760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:33.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.146+0000 7ff1aa23d700 1 --2- 172.21.15.23:0/3776242743 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff1a40ad080 0x7ff1a4137760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:33.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.146+0000 7ff1aa23d700 1 -- 172.21.15.23:0/3776242743 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1a413df30 msgr2=0x7ff1a413c620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:41:33.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.146+0000 7ff1aa23d700 1 --2- 172.21.15.23:0/3776242743 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff1a413df30 0x7ff1a413c620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:33.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.146+0000 7ff1aa23d700 1 -- 172.21.15.23:0/3776242743 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a40a7430 con 0x7ff1a40acca0 2020-11-20T11:41:33.152 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.146+0000 7ff1aa23d700 1 --2- 172.21.15.23:0/3776242743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1a40acca0 0x7ff1a4066220 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7ff19400c710 tx=0x7ff194007c90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:33.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.146+0000 7ff1927fc700 1 -- 172.21.15.23:0/3776242743 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff194007020 con 0x7ff1a40acca0 2020-11-20T11:41:33.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.147+0000 7ff1ab23f700 1 -- 172.21.15.23:0/3776242743 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1a40a01f0 con 0x7ff1a40acca0 2020-11-20T11:41:33.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.147+0000 7ff1ab23f700 1 -- 172.21.15.23:0/3776242743 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1a413d770 con 0x7ff1a40acca0 2020-11-20T11:41:33.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.147+0000 7ff1927fc700 1 -- 172.21.15.23:0/3776242743 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff194001e90 con 0x7ff1a40acca0 2020-11-20T11:41:33.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.149+0000 7ff1927fc700 1 -- 172.21.15.23:0/3776242743 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff194007390 con 0x7ff1a40acca0 2020-11-20T11:41:33.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.149+0000 7ff1927fc700 1 -- 172.21.15.23:0/3776242743 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff194010070 con 0x7ff1a40acca0 2020-11-20T11:41:33.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.149+0000 7ff1927fc700 1 --2- 172.21.15.23:0/3776242743 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff184056db0 0x7ff184059270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:33.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.149+0000 7ff1927fc700 1 -- 172.21.15.23:0/3776242743 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7ff19406f560 con 0x7ff1a40acca0 2020-11-20T11:41:33.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.149+0000 7ff1ab23f700 1 --2- 172.21.15.23:0/3776242743 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff17c002a50 0x7ff17c004f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:33.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.149+0000 7ff1ab23f700 1 -- 172.21.15.23:0/3776242743 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff17c007bb0 con 0x7ff17c002a50 2020-11-20T11:41:33.155 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.150+0000 7ff1aaa3e700 1 --2- 172.21.15.23:0/3776242743 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff17c002a50 0x7ff17c004f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:33.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.151+0000 7ff1aaa3e700 1 -- 172.21.15.23:0/3776242743 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff17c007d70 con 0x7ff17c002a50 2020-11-20T11:41:33.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.151+0000 7ff1aaa3e700 1 --2- 172.21.15.23:0/3776242743 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff17c002a50 0x7ff17c004f10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7ff1a0009320 tx=0x7ff1a0006be0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:33.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.151+0000 7ff1927fc700 1 -- 172.21.15.23:0/3776242743 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a0009e80 con 0x7ff17c002a50 2020-11-20T11:41:33.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.151+0000 7ff1927fc700 1 -- 172.21.15.23:0/3776242743 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7ff1a0002320 con 0x7ff17c002a50 2020-11-20T11:41:33.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.151+0000 7ff1927fc700 1 -- 172.21.15.23:0/3776242743 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff17c002a50 msgr2=0x7ff17c004f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:33.157 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.151+0000 7ff1927fc700 1 --2- 172.21.15.23:0/3776242743 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff17c002a50 0x7ff17c004f10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7ff1a0009320 tx=0x7ff1a0006be0).stop 2020-11-20T11:41:33.157 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.151+0000 7ff1927fc700 1 -- 172.21.15.23:0/3776242743 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff17c002a50 msgr2=0x7ff17c004f10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:41:33.157 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.151+0000 7ff1927fc700 1 --2- 172.21.15.23:0/3776242743 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff17c002a50 0x7ff17c004f10 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:33.157 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.153+0000 7ff1a9a3c700 1 --2- 172.21.15.23:0/3776242743 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff184056db0 0x7ff184059270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:33.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.153+0000 7ff1a9a3c700 1 --2- 172.21.15.23:0/3776242743 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff184056db0 0x7ff184059270 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7ff1a40aa7a0 tx=0x7ff19c016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:33.182 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.179+0000 7ff1ab23f700 1 --2- 172.21.15.23:0/3776242743 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff17c006c00 0x7ff17c007070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:33.183 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.179+0000 7ff1ab23f700 1 -- 172.21.15.23:0/3776242743 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff17c007680 con 0x7ff17c006c00 2020-11-20T11:41:33.183 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.180+0000 7ff1aaa3e700 1 --2- 172.21.15.23:0/3776242743 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff17c006c00 0x7ff17c007070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:33.184 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.180+0000 7ff1aaa3e700 1 -- 172.21.15.23:0/3776242743 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff17c007840 con 0x7ff17c006c00 2020-11-20T11:41:33.184 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.180+0000 7ff1aaa3e700 1 --2- 172.21.15.23:0/3776242743 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff17c006c00 0x7ff17c007070 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7ff1a0009320 tx=0x7ff1a0015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:33.184 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.181+0000 7ff1927fc700 1 -- 172.21.15.23:0/3776242743 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a0012e40 con 0x7ff17c006c00 2020-11-20T11:41:33.184 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.181+0000 7ff1927fc700 1 -- 172.21.15.23:0/3776242743 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4613 (secure 0 0 0) 0x7ff1a00113b0 con 0x7ff17c006c00 2020-11-20T11:41:33.185 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.181+0000 7ff1927fc700 1 -- 172.21.15.23:0/3776242743 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff17c006c00 msgr2=0x7ff17c007070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:33.185 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.181+0000 7ff1927fc700 1 --2- 172.21.15.23:0/3776242743 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff17c006c00 0x7ff17c007070 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7ff1a0009320 tx=0x7ff1a0015000).stop 2020-11-20T11:41:33.185 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.181+0000 7ff1927fc700 1 -- 172.21.15.23:0/3776242743 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff17c006c00 msgr2=0x7ff17c007070 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:41:33.185 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.181+0000 7ff1927fc700 1 --2- 172.21.15.23:0/3776242743 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff17c006c00 0x7ff17c007070 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0x7ff1a0009320 tx=0x7ff1a0015000).stop 2020-11-20T11:41:33.186 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:41:33.186 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:41:33.186 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:41:33.186 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:41:33.187 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 88, 2020-11-20T11:41:33.187 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:41:33.187 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:41:33.187 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:41:33.187 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:41:33.188 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:33.188 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 7, 2020-11-20T11:41:33.188 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:41:33.188 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:41:33.188 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:41:33.189 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:41:33.189 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:41:33.189 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:41:33.189 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:41:33.189 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:41:33.190 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:41:33.190 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:41:33.190 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:41:33.190 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:33.190 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:41:33.191 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:41:33.191 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:41:33.191 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:41:33.191 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:41:33.191 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:41:33.192 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:41:33.192 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:41:33.192 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:41:33.192 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:41:33.192 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:33.193 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:33.193 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:41:33.193 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:41:33.193 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:41:33.193 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:41:33.194 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:41:33.194 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:41:33.194 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:41:33.194 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:41:33.194 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:41:33.195 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:41:33.195 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:41:33.195 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:41:33.195 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:41:33.195 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:41:33.196 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:41:33.196 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:41:33.196 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:41:33.196 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:41:33.196 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:41:33.196 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:41:33.197 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:41:33.197 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:41:33.197 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:41:33.197 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:33.197 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:41:33.198 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:33.198 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:41:33.198 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:33.198 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:41:33.198 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:41:33.199 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:41:33.199 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:41:33.199 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:33.199 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:41:33.199 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:41:33.200 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:33.200 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:33.200 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:33.200 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:41:33.200 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:41:33.201 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:33.201 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:33.201 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:33.201 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:33.201 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:41:33.202 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:41:33.202 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:41:33.202 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:41:33.202 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:41:33.202 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:33.202 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:33.203 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:41:33.203 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:41:33.203 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:41:33.203 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:41:33.203 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:33.204 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:41:33.204 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:41:33.204 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:33.204 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:33.204 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:33.205 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:41:33.205 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:41:33.205 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:33.205 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:33.205 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:33.206 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:33.206 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:41:33.206 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:41:33.206 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:41:33.206 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:41:33.207 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:41:33.207 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:33.207 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:33.207 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:41:33.207 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:41:33.208 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:41:33.208 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:41:33.208 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:33.208 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:41:33.208 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:41:33.208 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:33.209 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:33.209 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:33.209 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:41:33.209 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:41:33.210 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:33.210 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:33.210 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:33.210 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:33.210 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:41:33.210 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:41:33.211 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:41:33.211 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:41:33.211 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:41:33.211 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:33.211 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:33.212 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:33.212 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:41:33.212 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:41:33.212 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:33.212 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:41:33.213 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:41:33.213 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:41:33.213 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:33.213 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:33.213 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:41:33.214 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:33.214 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:41:33.214 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:41:33.214 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:41:33.214 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:33.215 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:33.215 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:41:33.215 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:33.215 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:41:33.215 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:41:33.216 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:41:33.216 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:33.216 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:33.216 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:41:33.216 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:33.217 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:41:33.217 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:41:33.217 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:41:33.217 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:33.217 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:33.218 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:33.218 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:41:33.218 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:41:33.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.184+0000 7ff1ab23f700 1 -- 172.21.15.23:0/3776242743 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff184056db0 msgr2=0x7ff184059270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:33.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.184+0000 7ff1ab23f700 1 --2- 172.21.15.23:0/3776242743 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff184056db0 0x7ff184059270 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7ff1a40aa7a0 tx=0x7ff19c016000).stop 2020-11-20T11:41:33.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.184+0000 7ff1aaa3e700 1 -- 172.21.15.23:0/3776242743 reap_dead start 2020-11-20T11:41:33.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.184+0000 7ff1ab23f700 1 -- 172.21.15.23:0/3776242743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1a40acca0 msgr2=0x7ff1a4066220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:33.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.184+0000 7ff1ab23f700 1 --2- 172.21.15.23:0/3776242743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1a40acca0 0x7ff1a4066220 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7ff19400c710 tx=0x7ff194007c90).stop 2020-11-20T11:41:33.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.184+0000 7ff1ab23f700 1 -- 172.21.15.23:0/3776242743 shutdown_connections 2020-11-20T11:41:33.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.184+0000 7ff1ab23f700 1 --2- 172.21.15.23:0/3776242743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1a40acca0 0x7ff1a4066220 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:33.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.184+0000 7ff1ab23f700 1 -- 172.21.15.23:0/3776242743 >> 172.21.15.23:0/3776242743 conn(0x7ff1a4061850 msgr2=0x7ff1a413a990 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:41:33.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.184+0000 7ff1ab23f700 1 -- 172.21.15.23:0/3776242743 shutdown_connections 2020-11-20T11:41:33.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.184+0000 7ff1ab23f700 1 -- 172.21.15.23:0/3776242743 wait complete. 2020-11-20T11:41:33.227 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:41:33.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.560+0000 7fc0112ae700 1 Processor -- start 2020-11-20T11:41:33.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.560+0000 7fc0112ae700 1 -- start start 2020-11-20T11:41:33.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.561+0000 7fc0112ae700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc00c062120 0x7fc00c0a86f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:33.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.561+0000 7fc0112ae700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc00c14b8c0 0x7fc00c152480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:33.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.561+0000 7fc0112ae700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc00c0a9300 con 0x7fc00c0a8be0 2020-11-20T11:41:33.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.561+0000 7fc0112ae700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc00c0a5a30 con 0x7fc00c062120 2020-11-20T11:41:33.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.561+0000 7fc0112ae700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c0a5cd0 con 0x7fc00c14b8c0 2020-11-20T11:41:33.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.562+0000 7fc00affd700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc00c062120 0x7fc00c0a86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:33.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.562+0000 7fc00affd700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc00c062120 0x7fc00c0a86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:37472/0 (socket says 172.21.15.23:37472) 2020-11-20T11:41:33.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.562+0000 7fc00affd700 1 -- 172.21.15.23:0/3623089728 learned_addr learned my addr 172.21.15.23:0/3623089728 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:41:33.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.562+0000 7fc009ffb700 1 -- 172.21.15.23:0/3623089728 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 931010248 0 0) 0x7fc00c0a9300 con 0x7fc00c0a8be0 2020-11-20T11:41:33.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.563+0000 7fc009ffb700 1 -- 172.21.15.23:0/3623089728 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbff4003260 con 0x7fc00c0a8be0 2020-11-20T11:41:33.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.563+0000 7fc00affd700 1 -- 172.21.15.23:0/3623089728 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc00c14b8c0 msgr2=0x7fc00c152480 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2020-11-20T11:41:33.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.563+0000 7fc00affd700 1 --2- 172.21.15.23:0/3623089728 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc00c14b8c0 0x7fc00c152480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:33.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.563+0000 7fc00affd700 1 -- 172.21.15.23:0/3623089728 >> v1:172.21.15.155:6789/0 conn(0x7fc00c0a8be0 legacy=0x7fc00c14b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:33.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.563+0000 7fc00affd700 1 -- 172.21.15.23:0/3623089728 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc00c0a5f70 con 0x7fc00c062120 2020-11-20T11:41:33.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.563+0000 7fc00affd700 1 --2- 172.21.15.23:0/3623089728 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc00c062120 0x7fc00c0a86f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fc000009a40 tx=0x7fc000006ef0).ready entity=mon.1 client_cookie=32bf33e902d5f3b4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:33.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.564+0000 7fc009ffb700 1 -- 172.21.15.23:0/3623089728 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00000b820 con 0x7fc00c062120 2020-11-20T11:41:33.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.564+0000 7fc009ffb700 1 -- 172.21.15.23:0/3623089728 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc000000e90 con 0x7fc00c062120 2020-11-20T11:41:33.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.564+0000 7fc009ffb700 1 -- 172.21.15.23:0/3623089728 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00000bb90 con 0x7fc00c062120 2020-11-20T11:41:33.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.565+0000 7fc0112ae700 1 -- 172.21.15.23:0/3623089728 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc00c062120 msgr2=0x7fc00c0a86f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:33.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.565+0000 7fc0112ae700 1 --2- 172.21.15.23:0/3623089728 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc00c062120 0x7fc00c0a86f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fc000009a40 tx=0x7fc000006ef0).stop 2020-11-20T11:41:33.573 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.567+0000 7fc0112ae700 1 -- 172.21.15.23:0/3623089728 shutdown_connections 2020-11-20T11:41:33.573 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.567+0000 7fc0112ae700 1 --2- 172.21.15.23:0/3623089728 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc00c062120 0x7fc00c0a86f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:33.573 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.568+0000 7fc0112ae700 1 --2- 172.21.15.23:0/3623089728 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc00c14b8c0 0x7fc00c152480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:33.573 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.568+0000 7fc0112ae700 1 -- 172.21.15.23:0/3623089728 >> 172.21.15.23:0/3623089728 conn(0x7fc00c060c80 msgr2=0x7fc00c061850 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:41:33.574 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.568+0000 7fc0112ae700 1 -- 172.21.15.23:0/3623089728 shutdown_connections 2020-11-20T11:41:33.574 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.568+0000 7fc0112ae700 1 -- 172.21.15.23:0/3623089728 wait complete. 2020-11-20T11:41:33.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.569+0000 7fc0112ae700 1 Processor -- start 2020-11-20T11:41:33.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.569+0000 7fc0112ae700 1 -- start start 2020-11-20T11:41:33.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.570+0000 7fc0112ae700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc00c062120 0x7fc00c09f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:33.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.570+0000 7fc0112ae700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc00c0a8be0 0x7fc00c09ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:33.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.570+0000 7fc0112ae700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc00c14b8c0 0x7fc00c0a13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:33.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.570+0000 7fc0112ae700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c09d930 con 0x7fc00c14b8c0 2020-11-20T11:41:33.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.570+0000 7fc0112ae700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c09ff50 con 0x7fc00c0a8be0 2020-11-20T11:41:33.585 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.570+0000 7fc0112ae700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c09e330 con 0x7fc00c062120 2020-11-20T11:41:33.585 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.570+0000 7fc00affd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc00c062120 0x7fc00c09f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:33.585 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.570+0000 7fc00affd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc00c062120 0x7fc00c09f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:54822/0 (socket says 172.21.15.23:54822) 2020-11-20T11:41:33.585 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.570+0000 7fc00affd700 1 -- 172.21.15.23:0/3625984882 learned_addr learned my addr 172.21.15.23:0/3625984882 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:41:33.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.570+0000 7fc00a7fc700 1 --2- 172.21.15.23:0/3625984882 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc00c0a8be0 0x7fc00c09ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:33.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.570+0000 7fc00b7fe700 1 --2- 172.21.15.23:0/3625984882 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc00c14b8c0 0x7fc00c0a13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:33.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.571+0000 7fc00a7fc700 1 -- 172.21.15.23:0/3625984882 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc00c062120 msgr2=0x7fc00c09f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:33.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.571+0000 7fc00a7fc700 1 --2- 172.21.15.23:0/3625984882 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc00c062120 0x7fc00c09f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:33.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.571+0000 7fc00a7fc700 1 -- 172.21.15.23:0/3625984882 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc00c14b8c0 msgr2=0x7fc00c0a13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:33.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.571+0000 7fc00a7fc700 1 --2- 172.21.15.23:0/3625984882 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc00c14b8c0 0x7fc00c0a13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:33.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.571+0000 7fc00a7fc700 1 -- 172.21.15.23:0/3625984882 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc00c09e5d0 con 0x7fc00c0a8be0 2020-11-20T11:41:33.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.571+0000 7fc00a7fc700 1 --2- 172.21.15.23:0/3625984882 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc00c0a8be0 0x7fc00c09ea70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fc00400ca90 tx=0x7fc004007de0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:33.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.571+0000 7fbff2ffd700 1 -- 172.21.15.23:0/3625984882 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc004002240 con 0x7fc00c0a8be0 2020-11-20T11:41:33.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.571+0000 7fc0112ae700 1 -- 172.21.15.23:0/3625984882 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc00c09ef60 con 0x7fc00c0a8be0 2020-11-20T11:41:33.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.571+0000 7fc0112ae700 1 -- 172.21.15.23:0/3625984882 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc00c167d80 con 0x7fc00c0a8be0 2020-11-20T11:41:33.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.571+0000 7fbff2ffd700 1 -- 172.21.15.23:0/3625984882 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0040023a0 con 0x7fc00c0a8be0 2020-11-20T11:41:33.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.573+0000 7fbff2ffd700 1 -- 172.21.15.23:0/3625984882 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc004001860 con 0x7fc00c0a8be0 2020-11-20T11:41:33.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.573+0000 7fbff2ffd700 1 -- 172.21.15.23:0/3625984882 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc004019030 con 0x7fc00c0a8be0 2020-11-20T11:41:33.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.574+0000 7fbff2ffd700 1 --2- 172.21.15.23:0/3625984882 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbfe8056db0 0x7fbfe8059270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:33.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.574+0000 7fc0112ae700 1 --2- 172.21.15.23:0/3625984882 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc00c0a3250 0x7fc00c168580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:33.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.574+0000 7fc0112ae700 1 -- 172.21.15.23:0/3625984882 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c05c280 con 0x7fc00c0a3250 2020-11-20T11:41:33.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.574+0000 7fbff2ffd700 1 -- 172.21.15.23:0/3625984882 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fc00406f800 con 0x7fc00c0a8be0 2020-11-20T11:41:33.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.574+0000 7fc00affd700 1 --2- 172.21.15.23:0/3625984882 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbfe8056db0 0x7fbfe8059270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:33.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.574+0000 7fc00b7fe700 1 --2- 172.21.15.23:0/3625984882 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc00c0a3250 0x7fc00c168580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:33.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.574+0000 7fc00b7fe700 1 -- 172.21.15.23:0/3625984882 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc00c061650 con 0x7fc00c0a3250 2020-11-20T11:41:33.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.575+0000 7fc00b7fe700 1 --2- 172.21.15.23:0/3625984882 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc00c0a3250 0x7fc00c168580 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fbffc00e870 tx=0x7fbffc008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:33.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.575+0000 7fbff2ffd700 1 -- 172.21.15.23:0/3625984882 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbffc00b720 con 0x7fc00c0a3250 2020-11-20T11:41:33.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.576+0000 7fbff2ffd700 1 -- 172.21.15.23:0/3625984882 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fbffc00b880 con 0x7fc00c0a3250 2020-11-20T11:41:33.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.576+0000 7fc00affd700 1 --2- 172.21.15.23:0/3625984882 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbfe8056db0 0x7fbfe8059270 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fc000006f30 tx=0x7fc000018000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:33.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.576+0000 7fbff2ffd700 1 -- 172.21.15.23:0/3625984882 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc00c0a3250 msgr2=0x7fc00c168580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:33.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.576+0000 7fbff2ffd700 1 --2- 172.21.15.23:0/3625984882 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc00c0a3250 0x7fc00c168580 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fbffc00e870 tx=0x7fbffc008040).stop 2020-11-20T11:41:33.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.576+0000 7fbff2ffd700 1 -- 172.21.15.23:0/3625984882 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc00c0a3250 msgr2=0x7fc00c168580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:41:33.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.576+0000 7fbff2ffd700 1 --2- 172.21.15.23:0/3625984882 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc00c0a3250 0x7fc00c168580 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:33.595 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.592+0000 7fbff0ff9700 1 --2- 172.21.15.23:0/3625984882 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc00c168a70 0x7fc00c16bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:33.596 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.592+0000 7fbff0ff9700 1 -- 172.21.15.23:0/3625984882 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c09e760 con 0x7fc00c168a70 2020-11-20T11:41:33.596 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.592+0000 7fc00b7fe700 1 --2- 172.21.15.23:0/3625984882 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc00c168a70 0x7fc00c16bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:33.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.593+0000 7fc00b7fe700 1 -- 172.21.15.23:0/3625984882 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc00c0a4030 con 0x7fc00c168a70 2020-11-20T11:41:33.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.593+0000 7fc00b7fe700 1 --2- 172.21.15.23:0/3625984882 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc00c168a70 0x7fc00c16bb40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fbffc00e870 tx=0x7fbffc018000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:33.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.593+0000 7fbff2ffd700 1 -- 172.21.15.23:0/3625984882 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbffc022470 con 0x7fc00c168a70 2020-11-20T11:41:33.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.593+0000 7fbff2ffd700 1 -- 172.21.15.23:0/3625984882 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4273 (secure 0 0 0) 0x7fbffc0123b0 con 0x7fc00c168a70 2020-11-20T11:41:33.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.593+0000 7fbff2ffd700 1 -- 172.21.15.23:0/3625984882 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc00c168a70 msgr2=0x7fc00c16bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:33.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.593+0000 7fbff2ffd700 1 --2- 172.21.15.23:0/3625984882 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc00c168a70 0x7fc00c16bb40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fbffc00e870 tx=0x7fbffc018000).stop 2020-11-20T11:41:33.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.594+0000 7fbff2ffd700 1 -- 172.21.15.23:0/3625984882 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc00c168a70 msgr2=0x7fc00c16bb40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:41:33.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.594+0000 7fbff2ffd700 1 --2- 172.21.15.23:0/3625984882 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc00c168a70 0x7fc00c16bb40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:33.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.594+0000 7fbff0ff9700 1 -- 172.21.15.23:0/3625984882 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbfe8056db0 msgr2=0x7fbfe8059270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:33.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.594+0000 7fbff0ff9700 1 --2- 172.21.15.23:0/3625984882 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbfe8056db0 0x7fbfe8059270 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fc000006f30 tx=0x7fc000018000).stop 2020-11-20T11:41:33.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.594+0000 7fbff0ff9700 1 -- 172.21.15.23:0/3625984882 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc00c0a8be0 msgr2=0x7fc00c09ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:33.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.594+0000 7fbff0ff9700 1 --2- 172.21.15.23:0/3625984882 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc00c0a8be0 0x7fc00c09ea70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fc00400ca90 tx=0x7fc004007de0).stop 2020-11-20T11:41:33.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.594+0000 7fc00b7fe700 1 -- 172.21.15.23:0/3625984882 reap_dead start 2020-11-20T11:41:33.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.594+0000 7fbff0ff9700 1 -- 172.21.15.23:0/3625984882 shutdown_connections 2020-11-20T11:41:33.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.594+0000 7fbff0ff9700 1 -- 172.21.15.23:0/3625984882 >> 172.21.15.23:0/3625984882 conn(0x7fc00c060c80 msgr2=0x7fc00c149ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:41:33.601 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.595+0000 7fbff0ff9700 1 -- 172.21.15.23:0/3625984882 shutdown_connections 2020-11-20T11:41:33.601 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:33.595+0000 7fbff0ff9700 1 -- 172.21.15.23:0/3625984882 wait complete. 2020-11-20T11:41:33.601 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:41:33.601 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:41:33.602 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:41:33.602 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:41:33.602 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 88, 2020-11-20T11:41:33.602 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:41:33.602 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:41:33.603 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:41:33.603 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:41:33.603 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:33.603 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 7, 2020-11-20T11:41:33.603 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:41:33.604 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:41:33.604 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:41:33.604 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:41:33.604 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:41:33.604 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:41:33.605 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:41:33.605 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:41:33.605 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:41:33.605 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:41:33.606 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:41:33.606 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:33.606 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:41:33.606 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:41:33.606 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:41:33.606 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:41:33.607 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:41:33.607 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:41:33.607 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:41:33.607 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:41:33.607 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:41:33.608 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:41:33.608 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:33.608 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:33.608 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:41:33.608 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:41:33.609 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:41:33.609 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:41:33.609 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:41:33.609 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:41:33.609 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:41:33.610 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:41:33.610 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:41:33.610 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:41:33.610 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:41:33.611 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:41:33.611 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:41:33.611 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:41:33.611 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:41:33.611 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:41:33.611 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:41:33.612 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:41:33.612 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:41:33.612 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:41:33.612 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:41:33.612 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:41:33.613 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:41:33.613 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:33.613 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:41:33.613 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:33.613 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:41:33.614 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:33.614 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:41:33.614 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:41:33.614 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:41:33.614 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:41:33.615 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:33.615 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:41:33.615 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:41:33.615 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:33.615 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:33.616 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:33.616 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:41:33.616 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:41:33.616 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:33.616 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:33.617 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:33.617 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:33.617 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:41:33.617 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:41:33.617 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:41:33.618 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:41:33.618 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:41:33.618 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:33.618 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:33.619 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:41:33.619 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:41:33.619 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:41:33.619 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:41:33.619 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:33.620 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:41:33.620 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:41:33.620 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:33.620 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:33.620 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:33.621 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:41:33.621 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:41:33.621 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:33.621 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:33.621 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:33.622 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:33.622 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:41:33.622 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:41:33.622 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:41:33.622 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:41:33.623 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:41:33.623 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:33.623 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:33.623 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:41:33.623 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:41:33.624 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:41:33.624 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:41:33.624 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:33.624 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:41:33.624 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:41:33.625 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:33.625 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:33.625 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:33.625 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:41:33.625 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:41:33.626 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:33.626 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:33.626 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:33.626 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:33.626 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:41:33.627 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:41:33.627 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:41:33.627 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:41:33.627 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:41:33.627 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:33.628 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:33.628 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:33.628 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:41:33.628 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:41:33.628 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:33.629 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:41:33.629 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:41:33.629 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:41:33.629 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:33.630 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:33.630 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:41:33.630 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:33.630 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:41:33.630 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:41:33.631 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:41:33.631 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:33.631 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:33.631 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:33.631 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:41:33.632 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:41:33.633 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2020-11-20T11:41:33.633 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2020-11-20T11:41:34.081 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.078+0000 7fbeeea0a700 1 Processor -- start 2020-11-20T11:41:34.081 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.078+0000 7fbeeea0a700 1 -- start start 2020-11-20T11:41:34.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.079+0000 7fbeeea0a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fbee80acca0 0x7fbee80a8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:34.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.079+0000 7fbeeea0a700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbee80ad080 0x7fbee80a8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:34.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.079+0000 7fbeeea0a700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee80a5a30 con 0x7fbee80a9020 2020-11-20T11:41:34.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.079+0000 7fbeeea0a700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbee80a9560 con 0x7fbee80acca0 2020-11-20T11:41:34.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.079+0000 7fbeeea0a700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbee80a77e0 con 0x7fbee80ad080 2020-11-20T11:41:34.086 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.081+0000 7fbeee209700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fbee80a9020 0x7fbee80a9790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58844/0 (socket says 172.21.15.23:58844) 2020-11-20T11:41:34.086 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.082+0000 7fbeee209700 1 -- 172.21.15.23:0/3859834692 learned_addr learned my addr 172.21.15.23:0/3859834692 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:41:34.086 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.082+0000 7fbeeda08700 1 --2- 172.21.15.23:0/3859834692 >> v2:172.21.15.23:3300/0 conn(0x7fbee80acca0 0x7fbee80a8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:34.087 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.082+0000 7fbeeda08700 1 -- 172.21.15.23:0/3859834692 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbee80ad080 msgr2=0x7fbee80a8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:41:34.087 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.082+0000 7fbeeda08700 1 --2- 172.21.15.23:0/3859834692 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbee80ad080 0x7fbee80a8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:34.087 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.082+0000 7fbeeda08700 1 -- 172.21.15.23:0/3859834692 >> v1:172.21.15.23:6789/0 conn(0x7fbee80a9020 legacy=0x7fbee80a9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:34.087 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.082+0000 7fbeeda08700 1 -- 172.21.15.23:0/3859834692 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee80a79d0 con 0x7fbee80acca0 2020-11-20T11:41:34.087 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.082+0000 7fbeeda08700 1 --2- 172.21.15.23:0/3859834692 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbee80acca0 0x7fbee80a8110 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fbee4006fd0 tx=0x7fbee4006e30).ready entity=mon.0 client_cookie=8900efdb1504be61 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:34.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.085+0000 7fbeeca06700 1 -- 172.21.15.23:0/3859834692 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee40096e0 con 0x7fbee80acca0 2020-11-20T11:41:34.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.085+0000 7fbeeca06700 1 -- 172.21.15.23:0/3859834692 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbee4000e90 con 0x7fbee80acca0 2020-11-20T11:41:34.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.085+0000 7fbeeca06700 1 -- 172.21.15.23:0/3859834692 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee4009a50 con 0x7fbee80acca0 2020-11-20T11:41:34.089 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.086+0000 7fbeeea0a700 1 -- 172.21.15.23:0/3859834692 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbee80acca0 msgr2=0x7fbee80a8110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:34.089 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.086+0000 7fbeeea0a700 1 --2- 172.21.15.23:0/3859834692 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbee80acca0 0x7fbee80a8110 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fbee4006fd0 tx=0x7fbee4006e30).stop 2020-11-20T11:41:34.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.086+0000 7fbeeea0a700 1 -- 172.21.15.23:0/3859834692 shutdown_connections 2020-11-20T11:41:34.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.086+0000 7fbeeea0a700 1 --2- 172.21.15.23:0/3859834692 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbee80acca0 0x7fbee80a8110 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:34.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.086+0000 7fbeeea0a700 1 --2- 172.21.15.23:0/3859834692 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbee80ad080 0x7fbee80a8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:34.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.086+0000 7fbeeea0a700 1 -- 172.21.15.23:0/3859834692 >> 172.21.15.23:0/3859834692 conn(0x7fbee8061850 msgr2=0x7fbee8061c60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:41:34.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.086+0000 7fbeeea0a700 1 -- 172.21.15.23:0/3859834692 shutdown_connections 2020-11-20T11:41:34.095 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.086+0000 7fbeeea0a700 1 -- 172.21.15.23:0/3859834692 wait complete. 2020-11-20T11:41:34.095 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.088+0000 7fbeeea0a700 1 Processor -- start 2020-11-20T11:41:34.095 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.088+0000 7fbeeea0a700 1 -- start start 2020-11-20T11:41:34.095 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.089+0000 7fbeeea0a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbee80a9020 0x7fbee809da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:34.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.089+0000 7fbeeea0a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbee80acca0 0x7fbee813caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:34.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.089+0000 7fbeeea0a700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbee80ad080 0x7fbee80667c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:34.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.089+0000 7fbeeea0a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee80652a0 con 0x7fbee80acca0 2020-11-20T11:41:34.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.089+0000 7fbeeea0a700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee8065540 con 0x7fbee80a9020 2020-11-20T11:41:34.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.089+0000 7fbeeea0a700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbee80657e0 con 0x7fbee80ad080 2020-11-20T11:41:34.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.091+0000 7fbeee209700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbee80ad080 0x7fbee80667c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:34.097 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.091+0000 7fbeee209700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbee80ad080 0x7fbee80667c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:54840/0 (socket says 172.21.15.23:54840) 2020-11-20T11:41:34.097 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.091+0000 7fbeee209700 1 -- 172.21.15.23:0/1576568707 learned_addr learned my addr 172.21.15.23:0/1576568707 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:41:34.097 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.092+0000 7fbeee209700 1 -- 172.21.15.23:0/1576568707 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbee80a9020 msgr2=0x7fbee809da90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:41:34.097 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.092+0000 7fbeee209700 1 --2- 172.21.15.23:0/1576568707 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbee80a9020 0x7fbee809da90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:34.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.092+0000 7fbeee209700 1 -- 172.21.15.23:0/1576568707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbee80acca0 msgr2=0x7fbee813caa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:41:34.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.092+0000 7fbeee209700 1 --2- 172.21.15.23:0/1576568707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbee80acca0 0x7fbee813caa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:34.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.092+0000 7fbeee209700 1 -- 172.21.15.23:0/1576568707 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee813cf90 con 0x7fbee80ad080 2020-11-20T11:41:34.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.092+0000 7fbeee209700 1 --2- 172.21.15.23:0/1576568707 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbee80ad080 0x7fbee80667c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fbed800c7b0 tx=0x7fbed8007b90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:34.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.097+0000 7fbed5ffb700 1 -- 172.21.15.23:0/1576568707 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed8007320 con 0x7fbee80ad080 2020-11-20T11:41:34.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.097+0000 7fbeeea0a700 1 -- 172.21.15.23:0/1576568707 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbee8066cb0 con 0x7fbee80ad080 2020-11-20T11:41:34.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.097+0000 7fbeeea0a700 1 -- 172.21.15.23:0/1576568707 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbee8065c80 con 0x7fbee80ad080 2020-11-20T11:41:34.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.097+0000 7fbed5ffb700 1 -- 172.21.15.23:0/1576568707 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbed8007480 con 0x7fbee80ad080 2020-11-20T11:41:34.102 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.097+0000 7fbed5ffb700 1 -- 172.21.15.23:0/1576568707 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed8005840 con 0x7fbee80ad080 2020-11-20T11:41:34.104 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.101+0000 7fbeeea0a700 1 -- 172.21.15.23:0/1576568707 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbee8065f20 con 0x7fbee80ad080 2020-11-20T11:41:34.104 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.101+0000 7fbed5ffb700 1 -- 172.21.15.23:0/1576568707 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbed8010030 con 0x7fbee80ad080 2020-11-20T11:41:34.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.102+0000 7fbed5ffb700 1 --2- 172.21.15.23:0/1576568707 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbec0056db0 0x7fbec0059270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:34.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.102+0000 7fbed5ffb700 1 -- 172.21.15.23:0/1576568707 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fbed806fef0 con 0x7fbee80ad080 2020-11-20T11:41:34.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.102+0000 7fbeeda08700 1 --2- 172.21.15.23:0/1576568707 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbec0056db0 0x7fbec0059270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:34.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.103+0000 7fbeeda08700 1 --2- 172.21.15.23:0/1576568707 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbec0056db0 0x7fbec0059270 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fbee400b190 tx=0x7fbee4015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:34.111 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.108+0000 7fbed5ffb700 1 -- 172.21.15.23:0/1576568707 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fbee8065f20 con 0x7fbee80ad080 2020-11-20T11:41:34.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.692+0000 7fbeeea0a700 1 -- 172.21.15.23:0/1576568707 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fbee8065f20 con 0x7fbee80ad080 2020-11-20T11:41:34.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.699+0000 7fbed5ffb700 1 -- 172.21.15.23:0/1576568707 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fbee8065f20 con 0x7fbee80ad080 2020-11-20T11:41:34.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.709+0000 7fbeeea0a700 1 -- 172.21.15.23:0/1576568707 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbec0056db0 msgr2=0x7fbec0059270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:34.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.709+0000 7fbeeea0a700 1 --2- 172.21.15.23:0/1576568707 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbec0056db0 0x7fbec0059270 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fbee400b190 tx=0x7fbee4015000).stop 2020-11-20T11:41:34.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.709+0000 7fbeeea0a700 1 -- 172.21.15.23:0/1576568707 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbee80ad080 msgr2=0x7fbee80667c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:34.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.709+0000 7fbeeea0a700 1 --2- 172.21.15.23:0/1576568707 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbee80ad080 0x7fbee80667c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fbed800c7b0 tx=0x7fbed8007b90).stop 2020-11-20T11:41:34.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.711+0000 7fbeeea0a700 1 -- 172.21.15.23:0/1576568707 shutdown_connections 2020-11-20T11:41:34.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.711+0000 7fbeeea0a700 1 --2- 172.21.15.23:0/1576568707 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbec0056db0 0x7fbec0059270 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:34.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.711+0000 7fbeeea0a700 1 --2- 172.21.15.23:0/1576568707 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbee80ad080 0x7fbee80667c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:34.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.711+0000 7fbeeea0a700 1 --2- 172.21.15.23:0/1576568707 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbee80a9020 0x7fbee809da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:34.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.711+0000 7fbeeea0a700 1 --2- 172.21.15.23:0/1576568707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbee80acca0 0x7fbee813caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:34.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.711+0000 7fbeeea0a700 1 -- 172.21.15.23:0/1576568707 >> 172.21.15.23:0/1576568707 conn(0x7fbee8061850 msgr2=0x7fbee8061c30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:41:34.718 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.714+0000 7fbeeea0a700 1 -- 172.21.15.23:0/1576568707 shutdown_connections 2020-11-20T11:41:34.718 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:34.715+0000 7fbeeea0a700 1 -- 172.21.15.23:0/1576568707 wait complete. 2020-11-20T11:41:34.751 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2020-11-20T11:41:44.753 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:41:44.753 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:41:45.966 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.959+0000 7ff61ffff700 1 Processor -- start 2020-11-20T11:41:45.966 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.959+0000 7ff61ffff700 1 -- start start 2020-11-20T11:41:45.966 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.960+0000 7ff61ffff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff620061380 0x7ff6200a7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:45.966 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.960+0000 7ff61ffff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff6201429f0 0x7ff620142e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:45.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.960+0000 7ff61ffff700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6200a5d30 con 0x7ff620143370 2020-11-20T11:41:45.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.960+0000 7ff61ffff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6200a5fd0 con 0x7ff6201429f0 2020-11-20T11:41:45.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.960+0000 7ff61ffff700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6200a9a10 con 0x7ff620061380 2020-11-20T11:41:45.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.961+0000 7ff61e7fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff6201429f0 0x7ff620142e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:45.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.961+0000 7ff61e7fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff6201429f0 0x7ff620142e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44608/0 (socket says 172.21.15.23:44608) 2020-11-20T11:41:45.968 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.961+0000 7ff61effd700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff620061380 0x7ff6200a7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:45.968 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.961+0000 7ff61e7fc700 1 -- 172.21.15.23:0/1772922030 learned_addr learned my addr 172.21.15.23:0/1772922030 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:41:45.968 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.961+0000 7ff61effd700 1 -- 172.21.15.23:0/1772922030 >> v2:172.21.15.23:3300/0 conn(0x7ff6201429f0 msgr2=0x7ff620142e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:41:45.968 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.961+0000 7ff61effd700 1 --2- 172.21.15.23:0/1772922030 >> v2:172.21.15.23:3300/0 conn(0x7ff6201429f0 0x7ff620142e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:45.968 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.961+0000 7ff61effd700 1 -- 172.21.15.23:0/1772922030 >> v1:172.21.15.155:6789/0 conn(0x7ff620143370 legacy=0x7ff6200a6400 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2020-11-20T11:41:45.969 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.961+0000 7ff61effd700 1 -- 172.21.15.23:0/1772922030 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6200a9cb0 con 0x7ff620061380 2020-11-20T11:41:45.969 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.961+0000 7ff61e7fc700 1 --2- 172.21.15.23:0/1772922030 >> v2:172.21.15.23:3300/0 conn(0x7ff6201429f0 0x7ff620142e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:41:45.969 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.962+0000 7ff61effd700 1 --2- 172.21.15.23:0/1772922030 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff620061380 0x7ff6200a7fc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7ff608006fd0 tx=0x7ff608006e30).ready entity=mon.1 client_cookie=f974db576808ffcf server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:45.969 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.962+0000 7ff61dffb700 1 -- 172.21.15.23:0/1772922030 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff608009760 con 0x7ff620061380 2020-11-20T11:41:45.969 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.962+0000 7ff61dffb700 1 -- 172.21.15.23:0/1772922030 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff608000e90 con 0x7ff620061380 2020-11-20T11:41:45.970 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.962+0000 7ff61dffb700 1 -- 172.21.15.23:0/1772922030 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff608009ad0 con 0x7ff620061380 2020-11-20T11:41:45.970 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.967+0000 7ff61ffff700 1 -- 172.21.15.23:0/1772922030 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff620061380 msgr2=0x7ff6200a7fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:45.970 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.967+0000 7ff61ffff700 1 --2- 172.21.15.23:0/1772922030 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff620061380 0x7ff6200a7fc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7ff608006fd0 tx=0x7ff608006e30).stop 2020-11-20T11:41:45.974 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.971+0000 7ff61ffff700 1 -- 172.21.15.23:0/1772922030 shutdown_connections 2020-11-20T11:41:45.975 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.971+0000 7ff61ffff700 1 --2- 172.21.15.23:0/1772922030 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff620061380 0x7ff6200a7fc0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:45.975 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.971+0000 7ff61ffff700 1 --2- 172.21.15.23:0/1772922030 >> v2:172.21.15.23:3300/0 conn(0x7ff6201429f0 0x7ff620142e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:45.975 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.971+0000 7ff61ffff700 1 -- 172.21.15.23:0/1772922030 >> 172.21.15.23:0/1772922030 conn(0x7ff6200ae0a0 msgr2=0x7ff6200aa0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:41:45.975 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.971+0000 7ff61ffff700 1 -- 172.21.15.23:0/1772922030 shutdown_connections 2020-11-20T11:41:45.976 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.971+0000 7ff61ffff700 1 -- 172.21.15.23:0/1772922030 wait complete. 2020-11-20T11:41:45.978 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.973+0000 7ff61ffff700 1 Processor -- start 2020-11-20T11:41:45.978 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.973+0000 7ff61ffff700 1 -- start start 2020-11-20T11:41:45.979 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.974+0000 7ff61ffff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff620061380 0x7ff620127700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:45.979 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.974+0000 7ff61ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6201429f0 0x7ff620127bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:45.979 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.974+0000 7ff61ffff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff620143370 0x7ff6200a2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:45.979 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.974+0000 7ff61ffff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6200a1910 con 0x7ff6201429f0 2020-11-20T11:41:45.979 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.974+0000 7ff61ffff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6200a1bb0 con 0x7ff620061380 2020-11-20T11:41:45.980 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.974+0000 7ff61ffff700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6200a1e50 con 0x7ff620143370 2020-11-20T11:41:45.980 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.974+0000 7ff61f7fe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff620143370 0x7ff6200a2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:45.980 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.974+0000 7ff61e7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6201429f0 0x7ff620127bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:45.980 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.974+0000 7ff61f7fe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff620143370 0x7ff6200a2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:54854/0 (socket says 172.21.15.23:54854) 2020-11-20T11:41:45.980 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.974+0000 7ff61f7fe700 1 -- 172.21.15.23:0/3039945885 learned_addr learned my addr 172.21.15.23:0/3039945885 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:41:45.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.975+0000 7ff61f7fe700 1 -- 172.21.15.23:0/3039945885 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff620061380 msgr2=0x7ff620127700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:45.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.975+0000 7ff61f7fe700 1 --2- 172.21.15.23:0/3039945885 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff620061380 0x7ff620127700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:45.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.975+0000 7ff61f7fe700 1 -- 172.21.15.23:0/3039945885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6201429f0 msgr2=0x7ff620127bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:45.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.975+0000 7ff61f7fe700 1 --2- 172.21.15.23:0/3039945885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6201429f0 0x7ff620127bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:45.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.975+0000 7ff61f7fe700 1 -- 172.21.15.23:0/3039945885 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6200a1430 con 0x7ff620143370 2020-11-20T11:41:45.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.975+0000 7ff61f7fe700 1 --2- 172.21.15.23:0/3039945885 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff620143370 0x7ff6200a2620 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7ff61800bae0 tx=0x7ff6180090f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:45.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.975+0000 7ff61e7fc700 1 --2- 172.21.15.23:0/3039945885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6201429f0 0x7ff620127bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:41:45.983 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.977+0000 7ff616ffd700 1 -- 172.21.15.23:0/3039945885 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61800c740 con 0x7ff620143370 2020-11-20T11:41:45.984 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.978+0000 7ff616ffd700 1 -- 172.21.15.23:0/3039945885 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff61800c8a0 con 0x7ff620143370 2020-11-20T11:41:45.984 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.978+0000 7ff61ffff700 1 -- 172.21.15.23:0/3039945885 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6200ad200 con 0x7ff620143370 2020-11-20T11:41:45.984 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.979+0000 7ff61ffff700 1 -- 172.21.15.23:0/3039945885 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff62009fb60 con 0x7ff620143370 2020-11-20T11:41:45.984 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.980+0000 7ff616ffd700 1 -- 172.21.15.23:0/3039945885 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61800ca50 con 0x7ff620143370 2020-11-20T11:41:45.985 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.980+0000 7ff61ffff700 1 -- 172.21.15.23:0/3039945885 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff620069c70 con 0x7ff620143370 2020-11-20T11:41:45.985 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.980+0000 7ff616ffd700 1 -- 172.21.15.23:0/3039945885 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff61801c070 con 0x7ff620143370 2020-11-20T11:41:45.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.983+0000 7ff616ffd700 1 --2- 172.21.15.23:0/3039945885 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff5fc056db0 0x7ff5fc059270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:45.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.984+0000 7ff616ffd700 1 -- 172.21.15.23:0/3039945885 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7ff61806f5a0 con 0x7ff620143370 2020-11-20T11:41:45.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.984+0000 7ff61effd700 1 --2- 172.21.15.23:0/3039945885 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff5fc056db0 0x7ff5fc059270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:45.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.985+0000 7ff61effd700 1 --2- 172.21.15.23:0/3039945885 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff5fc056db0 0x7ff5fc059270 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7ff608006e70 tx=0x7ff608015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:45.994 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:45.988+0000 7ff616ffd700 1 -- 172.21.15.23:0/3039945885 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff6180449a0 con 0x7ff620143370 2020-11-20T11:41:46.557 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:46.553+0000 7ff61ffff700 1 -- 172.21.15.23:0/3039945885 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff620057dc0 con 0x7ff620143370 2020-11-20T11:41:46.557 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:46.553+0000 7ff616ffd700 1 -- 172.21.15.23:0/3039945885 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7ff61801a070 con 0x7ff620143370 2020-11-20T11:41:46.558 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:41:46.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:46.558+0000 7ff61ffff700 1 -- 172.21.15.23:0/3039945885 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff5fc056db0 msgr2=0x7ff5fc059270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:46.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:46.558+0000 7ff61ffff700 1 --2- 172.21.15.23:0/3039945885 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff5fc056db0 0x7ff5fc059270 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7ff608006e70 tx=0x7ff608015000).stop 2020-11-20T11:41:46.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:46.558+0000 7ff61ffff700 1 -- 172.21.15.23:0/3039945885 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff620143370 msgr2=0x7ff6200a2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:46.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:46.558+0000 7ff61ffff700 1 --2- 172.21.15.23:0/3039945885 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff620143370 0x7ff6200a2620 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7ff61800bae0 tx=0x7ff6180090f0).stop 2020-11-20T11:41:46.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:46.558+0000 7ff61ffff700 1 -- 172.21.15.23:0/3039945885 shutdown_connections 2020-11-20T11:41:46.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:46.559+0000 7ff61ffff700 1 --2- 172.21.15.23:0/3039945885 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff5fc056db0 0x7ff5fc059270 secure :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0x7ff608006e70 tx=0x7ff608015000).stop 2020-11-20T11:41:46.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:46.559+0000 7ff61ffff700 1 --2- 172.21.15.23:0/3039945885 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff620143370 0x7ff6200a2620 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:46.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:46.559+0000 7ff61ffff700 1 --2- 172.21.15.23:0/3039945885 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff620061380 0x7ff620127700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:46.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:46.559+0000 7ff61ffff700 1 --2- 172.21.15.23:0/3039945885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6201429f0 0x7ff620127bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:46.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:46.559+0000 7ff61ffff700 1 -- 172.21.15.23:0/3039945885 >> 172.21.15.23:0/3039945885 conn(0x7ff6200ae0a0 msgr2=0x7ff6200a7ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:41:46.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:46.559+0000 7ff61ffff700 1 -- 172.21.15.23:0/3039945885 shutdown_connections 2020-11-20T11:41:46.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:46.559+0000 7ff61ffff700 1 -- 172.21.15.23:0/3039945885 wait complete. 2020-11-20T11:41:46.631 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:41:46.631 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2020-11-20T11:41:46.632 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:41:47.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.200+0000 7f5ae963b700 1 Processor -- start 2020-11-20T11:41:47.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.200+0000 7f5ae963b700 1 -- start start 2020-11-20T11:41:47.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.201+0000 7f5ae963b700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5ae40608c0 0x7f5ae413eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.201+0000 7f5ae963b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5ae4060ca0 0x7f5ae4137fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.201+0000 7f5ae963b700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae413e1b0 con 0x7f5ae41384d0 2020-11-20T11:41:47.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.201+0000 7f5ae963b700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ae413e450 con 0x7f5ae40608c0 2020-11-20T11:41:47.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.201+0000 7f5ae963b700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae413e6f0 con 0x7f5ae4060ca0 2020-11-20T11:41:47.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.201+0000 7f5ae27fc700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5ae4060ca0 0x7f5ae4137fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:47.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.201+0000 7f5ae37fe700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f5ae41384d0 0x7f5ae40aabc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:36844/0 (socket says 172.21.15.23:36844) 2020-11-20T11:41:47.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.201+0000 7f5ae37fe700 1 -- 172.21.15.23:0/3842992827 learned_addr learned my addr 172.21.15.23:0/3842992827 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:41:47.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.201+0000 7f5ae2ffd700 1 --2- 172.21.15.23:0/3842992827 >> v2:172.21.15.155:3300/0 conn(0x7f5ae40608c0 0x7f5ae413eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:47.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.201+0000 7f5ae27fc700 1 -- 172.21.15.23:0/3842992827 >> v2:172.21.15.155:3300/0 conn(0x7f5ae40608c0 msgr2=0x7f5ae413eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:41:47.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.201+0000 7f5ae27fc700 1 --2- 172.21.15.23:0/3842992827 >> v2:172.21.15.155:3300/0 conn(0x7f5ae40608c0 0x7f5ae413eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:47.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.201+0000 7f5ae27fc700 1 -- 172.21.15.23:0/3842992827 >> v1:172.21.15.155:6789/0 conn(0x7f5ae41384d0 legacy=0x7f5ae40aabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:47.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.201+0000 7f5ae27fc700 1 -- 172.21.15.23:0/3842992827 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae413f080 con 0x7f5ae4060ca0 2020-11-20T11:41:47.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.201+0000 7f5ae2ffd700 1 --2- 172.21.15.23:0/3842992827 >> v2:172.21.15.155:3300/0 conn(0x7f5ae40608c0 0x7f5ae413eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:41:47.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.202+0000 7f5ae27fc700 1 --2- 172.21.15.23:0/3842992827 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5ae4060ca0 0x7f5ae4137fe0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f5ad8006fd0 tx=0x7f5ad8006e30).ready entity=mon.2 client_cookie=d4521d8de4df9f53 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:47.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.202+0000 7f5ae1ffb700 1 -- 172.21.15.23:0/3842992827 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad80097e0 con 0x7f5ae4060ca0 2020-11-20T11:41:47.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.202+0000 7f5ae1ffb700 1 -- 172.21.15.23:0/3842992827 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ad8000e90 con 0x7f5ae4060ca0 2020-11-20T11:41:47.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.202+0000 7f5ae1ffb700 1 -- 172.21.15.23:0/3842992827 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad8009b50 con 0x7f5ae4060ca0 2020-11-20T11:41:47.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.202+0000 7f5ae963b700 1 -- 172.21.15.23:0/3842992827 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5ae4060ca0 msgr2=0x7f5ae4137fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:47.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.202+0000 7f5ae963b700 1 --2- 172.21.15.23:0/3842992827 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5ae4060ca0 0x7f5ae4137fe0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f5ad8006fd0 tx=0x7f5ad8006e30).stop 2020-11-20T11:41:47.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.202+0000 7f5ae963b700 1 -- 172.21.15.23:0/3842992827 shutdown_connections 2020-11-20T11:41:47.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.202+0000 7f5ae963b700 1 --2- 172.21.15.23:0/3842992827 >> v2:172.21.15.155:3300/0 conn(0x7f5ae40608c0 0x7f5ae413eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:47.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.202+0000 7f5ae963b700 1 --2- 172.21.15.23:0/3842992827 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5ae4060ca0 0x7f5ae4137fe0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:47.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.202+0000 7f5ae963b700 1 -- 172.21.15.23:0/3842992827 >> 172.21.15.23:0/3842992827 conn(0x7f5ae4066750 msgr2=0x7f5ae40601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:41:47.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.202+0000 7f5ae963b700 1 -- 172.21.15.23:0/3842992827 shutdown_connections 2020-11-20T11:41:47.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.202+0000 7f5ae963b700 1 -- 172.21.15.23:0/3842992827 wait complete. 2020-11-20T11:41:47.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.204+0000 7f5ae963b700 1 Processor -- start 2020-11-20T11:41:47.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.204+0000 7f5ae963b700 1 -- start start 2020-11-20T11:41:47.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.205+0000 7f5ae963b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5ae40608c0 0x7f5ae413aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.205+0000 7f5ae963b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ae4060ca0 0x7f5ae414a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.205+0000 7f5ae963b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ae41384d0 0x7f5ae40a8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.205+0000 7f5ae963b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae414afc0 con 0x7f5ae41384d0 2020-11-20T11:41:47.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.205+0000 7f5ae963b700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae40af2c0 con 0x7f5ae4060ca0 2020-11-20T11:41:47.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.205+0000 7f5ae963b700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae40ae640 con 0x7f5ae40608c0 2020-11-20T11:41:47.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.206+0000 7f5ae37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ae41384d0 0x7f5ae40a8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:47.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.206+0000 7f5ae37fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ae41384d0 0x7f5ae40a8c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44628/0 (socket says 172.21.15.23:44628) 2020-11-20T11:41:47.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.206+0000 7f5ae37fe700 1 -- 172.21.15.23:0/4089789740 learned_addr learned my addr 172.21.15.23:0/4089789740 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:41:47.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.206+0000 7f5ae37fe700 1 -- 172.21.15.23:0/4089789740 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5ae40608c0 msgr2=0x7f5ae413aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:47.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.206+0000 7f5ae37fe700 1 --2- 172.21.15.23:0/4089789740 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5ae40608c0 0x7f5ae413aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:47.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.206+0000 7f5ae37fe700 1 -- 172.21.15.23:0/4089789740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ae4060ca0 msgr2=0x7f5ae414a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.206+0000 7f5ae37fe700 1 --2- 172.21.15.23:0/4089789740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5ae4060ca0 0x7f5ae414a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.206+0000 7f5ae37fe700 1 -- 172.21.15.23:0/4089789740 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae40ae8e0 con 0x7f5ae41384d0 2020-11-20T11:41:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.206+0000 7f5ae37fe700 1 --2- 172.21.15.23:0/4089789740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ae41384d0 0x7f5ae40a8c30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f5adc00ca90 tx=0x7f5adc0013a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.206+0000 7f5ac3fff700 1 -- 172.21.15.23:0/4089789740 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5adc005bc0 con 0x7f5ae41384d0 2020-11-20T11:41:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.207+0000 7f5ae963b700 1 -- 172.21.15.23:0/4089789740 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ae40aeb80 con 0x7f5ae41384d0 2020-11-20T11:41:47.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.207+0000 7f5ae963b700 1 -- 172.21.15.23:0/4089789740 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ae413b620 con 0x7f5ae41384d0 2020-11-20T11:41:47.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.207+0000 7f5ac3fff700 1 -- 172.21.15.23:0/4089789740 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5adc005d20 con 0x7f5ae41384d0 2020-11-20T11:41:47.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.207+0000 7f5ac3fff700 1 -- 172.21.15.23:0/4089789740 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5adc007320 con 0x7f5ae41384d0 2020-11-20T11:41:47.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.214+0000 7f5ac3fff700 1 -- 172.21.15.23:0/4089789740 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5adc010030 con 0x7f5ae41384d0 2020-11-20T11:41:47.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.214+0000 7f5ac3fff700 1 --2- 172.21.15.23:0/4089789740 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5ab4078d70 0x7f5ab407b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.215+0000 7f5ac3fff700 1 -- 172.21.15.23:0/4089789740 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f5adc0700c0 con 0x7f5ae41384d0 2020-11-20T11:41:47.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.215+0000 7f5ae2ffd700 1 --2- 172.21.15.23:0/4089789740 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5ab4078d70 0x7f5ab407b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:47.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.215+0000 7f5ae2ffd700 1 --2- 172.21.15.23:0/4089789740 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5ab4078d70 0x7f5ab407b230 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f5ad4006fd0 tx=0x7f5ad4006e30).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:47.226 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.221+0000 7f5ae963b700 1 --2- 172.21.15.23:0/4089789740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ae40af760 0x7f5ae40a9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.227 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.221+0000 7f5ae963b700 1 -- 172.21.15.23:0/4089789740 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae40a9610 con 0x7f5ae40af760 2020-11-20T11:41:47.227 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.221+0000 7f5ae27fc700 1 --2- 172.21.15.23:0/4089789740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ae40af760 0x7f5ae40a9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:47.227 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.221+0000 7f5ae27fc700 1 -- 172.21.15.23:0/4089789740 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5ae4044710 con 0x7f5ae40af760 2020-11-20T11:41:47.227 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.222+0000 7f5ae27fc700 1 --2- 172.21.15.23:0/4089789740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ae40af760 0x7f5ae40a9120 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f5ad8006fd0 tx=0x7f5ad8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:47.227 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.222+0000 7f5ac3fff700 1 -- 172.21.15.23:0/4089789740 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad8011760 con 0x7f5ae40af760 2020-11-20T11:41:47.228 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.222+0000 7f5ac3fff700 1 -- 172.21.15.23:0/4089789740 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f5ad80118c0 con 0x7f5ae40af760 2020-11-20T11:41:47.228 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.222+0000 7f5ac3fff700 1 -- 172.21.15.23:0/4089789740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ae40af760 msgr2=0x7f5ae40a9120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:47.228 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.222+0000 7f5ac3fff700 1 --2- 172.21.15.23:0/4089789740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ae40af760 0x7f5ae40a9120 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f5ad8006fd0 tx=0x7f5ad8015000).stop 2020-11-20T11:41:47.228 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.222+0000 7f5ac3fff700 1 -- 172.21.15.23:0/4089789740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ae40af760 msgr2=0x7f5ae40a9120 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:41:47.228 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.222+0000 7f5ac3fff700 1 --2- 172.21.15.23:0/4089789740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ae40af760 0x7f5ae40a9120 secure :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0x7f5ad8006fd0 tx=0x7f5ad8015000).stop 2020-11-20T11:41:47.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.231+0000 7f5ae963b700 1 --2- 172.21.15.23:0/4089789740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ae414a540 0x7f5ae40a13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.231+0000 7f5ae963b700 1 -- 172.21.15.23:0/4089789740 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae40aed10 con 0x7f5ae414a540 2020-11-20T11:41:47.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.232+0000 7f5ae27fc700 1 --2- 172.21.15.23:0/4089789740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ae414a540 0x7f5ae40a13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:47.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.232+0000 7f5ae27fc700 1 -- 172.21.15.23:0/4089789740 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5ae405c280 con 0x7f5ae414a540 2020-11-20T11:41:47.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.232+0000 7f5ae27fc700 1 --2- 172.21.15.23:0/4089789740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ae414a540 0x7f5ae40a13c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f5ad8006e70 tx=0x7f5ad801e000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:47.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.232+0000 7f5ac3fff700 1 -- 172.21.15.23:0/4089789740 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad801d6a0 con 0x7f5ae414a540 2020-11-20T11:41:47.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.233+0000 7f5ac3fff700 1 -- 172.21.15.23:0/4089789740 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4786 (secure 0 0 0) 0x7f5ad8011550 con 0x7f5ae414a540 2020-11-20T11:41:47.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.233+0000 7f5ac3fff700 1 -- 172.21.15.23:0/4089789740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ae414a540 msgr2=0x7f5ae40a13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:47.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.233+0000 7f5ac3fff700 1 --2- 172.21.15.23:0/4089789740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ae414a540 0x7f5ae40a13c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f5ad8006e70 tx=0x7f5ad801e000).stop 2020-11-20T11:41:47.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.233+0000 7f5ac3fff700 1 -- 172.21.15.23:0/4089789740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ae414a540 msgr2=0x7f5ae40a13c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:41:47.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.233+0000 7f5ac3fff700 1 --2- 172.21.15.23:0/4089789740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ae414a540 0x7f5ae40a13c0 secure :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0x7f5ad8006e70 tx=0x7f5ad801e000).stop 2020-11-20T11:41:47.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.233+0000 7f5ae963b700 1 -- 172.21.15.23:0/4089789740 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5ab4078d70 msgr2=0x7f5ab407b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:47.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.233+0000 7f5ae963b700 1 --2- 172.21.15.23:0/4089789740 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5ab4078d70 0x7f5ab407b230 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f5ad4006fd0 tx=0x7f5ad4006e30).stop 2020-11-20T11:41:47.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.233+0000 7f5ae963b700 1 -- 172.21.15.23:0/4089789740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ae41384d0 msgr2=0x7f5ae40a8c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:47.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.233+0000 7f5ae963b700 1 --2- 172.21.15.23:0/4089789740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ae41384d0 0x7f5ae40a8c30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f5adc00ca90 tx=0x7f5adc0013a0).stop 2020-11-20T11:41:47.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.233+0000 7f5ae37fe700 1 -- 172.21.15.23:0/4089789740 reap_dead start 2020-11-20T11:41:47.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.233+0000 7f5ae37fe700 1 -- 172.21.15.23:0/4089789740 reap_dead start 2020-11-20T11:41:47.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.234+0000 7f5ae963b700 1 -- 172.21.15.23:0/4089789740 shutdown_connections 2020-11-20T11:41:47.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.234+0000 7f5ae963b700 1 -- 172.21.15.23:0/4089789740 >> 172.21.15.23:0/4089789740 conn(0x7f5ae4066750 msgr2=0x7f5ae40a18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:41:47.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.234+0000 7f5ae963b700 1 -- 172.21.15.23:0/4089789740 shutdown_connections 2020-11-20T11:41:47.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.234+0000 7f5ae963b700 1 -- 172.21.15.23:0/4089789740 wait complete. 2020-11-20T11:41:47.241 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:41:47.241 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:41:47.242 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:41:47.242 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:41:47.242 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 88, 2020-11-20T11:41:47.242 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:41:47.242 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:41:47.242 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:41:47.243 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:41:47.243 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:47.243 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 21, 2020-11-20T11:41:47.243 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:41:47.243 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:41:47.244 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:41:47.244 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:41:47.244 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:41:47.244 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:41:47.244 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:41:47.245 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:41:47.245 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:41:47.245 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:41:47.245 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:41:47.245 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:47.246 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:41:47.247 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:41:47.247 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:41:47.247 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:41:47.248 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:41:47.249 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:41:47.249 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:41:47.249 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:41:47.249 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:41:47.250 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:41:47.250 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:47.250 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.250 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:41:47.251 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:41:47.251 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:41:47.251 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:41:47.251 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:41:47.251 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:41:47.252 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:41:47.252 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:41:47.252 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:41:47.252 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:41:47.252 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:41:47.253 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:41:47.253 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:41:47.253 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:41:47.253 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:41:47.253 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:41:47.254 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:41:47.254 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:41:47.254 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:41:47.254 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:41:47.254 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:41:47.255 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:41:47.255 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:41:47.255 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:47.255 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:41:47.256 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.256 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:41:47.256 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.256 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:41:47.257 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:41:47.257 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:41:47.257 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:41:47.257 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.257 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:41:47.258 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:41:47.258 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:47.258 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.258 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.258 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:41:47.259 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:41:47.259 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:47.259 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:47.259 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:47.259 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.260 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:41:47.260 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:41:47.260 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:41:47.260 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:41:47.260 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:41:47.260 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.261 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.261 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:41:47.261 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:41:47.261 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:41:47.262 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:41:47.262 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.262 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:41:47.262 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:41:47.262 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:47.263 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.263 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.263 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:41:47.263 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:41:47.263 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:47.264 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:47.264 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:47.264 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.264 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:41:47.265 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:41:47.265 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:41:47.265 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:41:47.265 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:41:47.265 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.266 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.266 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:41:47.266 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:41:47.266 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:41:47.266 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:41:47.267 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.267 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:41:47.268 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:41:47.268 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:47.268 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.268 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.268 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:41:47.269 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:41:47.269 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:47.269 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:47.269 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:47.269 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.270 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:41:47.270 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:41:47.270 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:41:47.270 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:41:47.271 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:41:47.271 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:47.271 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:47.271 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.272 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:41:47.272 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:41:47.272 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.272 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:41:47.272 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:41:47.273 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:41:47.273 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:47.273 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:47.273 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:41:47.274 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.274 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:41:47.274 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:41:47.274 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:41:47.274 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:47.275 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:47.275 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:41:47.275 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.275 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:41:47.275 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:41:47.276 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 8 2020-11-20T11:41:47.276 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:47.276 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:47.276 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:41:47.276 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.277 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:41:47.277 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:41:47.277 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:41:47.277 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:47.278 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:47.278 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:41:47.278 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.278 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:41:47.278 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:41:47.279 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:41:47.279 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:47.279 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:47.279 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.279 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:41:47.280 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:41:47.281 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:41:47.423 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.420+0000 7f9709191700 1 Processor -- start 2020-11-20T11:41:47.424 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.420+0000 7f9709191700 1 -- start start 2020-11-20T11:41:47.425 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.421+0000 7f9709191700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97040667e0 0x7f970413c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.425 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.421+0000 7f9709191700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97040a8830 con 0x7f9704066400 2020-11-20T11:41:47.425 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.421+0000 7f9709191700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9704142770 con 0x7f97040a93c0 2020-11-20T11:41:47.425 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.421+0000 7f9709191700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97040a98d0 con 0x7f97040667e0 2020-11-20T11:41:47.426 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.422+0000 7f97037fe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97040667e0 0x7f970413c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:47.426 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.422+0000 7f9708990700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f97040a93c0 0x7f9704149ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:36862/0 (socket says 172.21.15.23:36862) 2020-11-20T11:41:47.426 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.422+0000 7f9703fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f9704066400 0x7f970413f960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58890/0 (socket says 172.21.15.23:58890) 2020-11-20T11:41:47.426 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.422+0000 7f9708990700 1 -- 172.21.15.23:0/840267156 learned_addr learned my addr 172.21.15.23:0/840267156 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:41:47.428 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.423+0000 7f9702ffd700 1 -- 172.21.15.23:0/840267156 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2671577662 0 0) 0x7f9704142770 con 0x7f97040a93c0 2020-11-20T11:41:47.428 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.424+0000 7f9702ffd700 1 -- 172.21.15.23:0/840267156 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96e4003280 con 0x7f97040a93c0 2020-11-20T11:41:47.428 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.424+0000 7f9702ffd700 1 -- 172.21.15.23:0/840267156 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2071238671 0 0) 0x7f97040a8830 con 0x7f9704066400 2020-11-20T11:41:47.429 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.425+0000 7f9702ffd700 1 -- 172.21.15.23:0/840267156 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9704142770 con 0x7f9704066400 2020-11-20T11:41:47.429 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.425+0000 7f97037fe700 1 -- 172.21.15.23:0/840267156 >> v1:172.21.15.155:6789/0 conn(0x7f97040a93c0 legacy=0x7f9704149ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:47.429 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.425+0000 7f9702ffd700 1 -- 172.21.15.23:0/840267156 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3794716316 0 0) 0x7f96e4003280 con 0x7f97040a93c0 2020-11-20T11:41:47.429 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.425+0000 7f97037fe700 1 -- 172.21.15.23:0/840267156 >> v1:172.21.15.23:6789/0 conn(0x7f9704066400 legacy=0x7f970413f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:47.429 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.425+0000 7f97037fe700 1 -- 172.21.15.23:0/840267156 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f970413f220 con 0x7f97040667e0 2020-11-20T11:41:47.430 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.425+0000 7f97037fe700 1 --2- 172.21.15.23:0/840267156 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97040667e0 0x7f970413c4d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f96f400b980 tx=0x7f96f4006e30).ready entity=mon.2 client_cookie=ad12fc753eaf978a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:47.430 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.426+0000 7f9702ffd700 1 -- 172.21.15.23:0/840267156 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96fc0018e0 con 0x7f97040667e0 2020-11-20T11:41:47.431 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.426+0000 7f9702ffd700 1 -- 172.21.15.23:0/840267156 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f4000e90 con 0x7f97040667e0 2020-11-20T11:41:47.431 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.426+0000 7f9702ffd700 1 -- 172.21.15.23:0/840267156 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f4011610 con 0x7f97040667e0 2020-11-20T11:41:47.431 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.426+0000 7f9709191700 1 -- 172.21.15.23:0/840267156 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97040667e0 msgr2=0x7f970413c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:47.431 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.426+0000 7f9709191700 1 --2- 172.21.15.23:0/840267156 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97040667e0 0x7f970413c4d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f96f400b980 tx=0x7f96f4006e30).stop 2020-11-20T11:41:47.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.428+0000 7f9709191700 1 -- 172.21.15.23:0/840267156 shutdown_connections 2020-11-20T11:41:47.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.428+0000 7f9709191700 1 --2- 172.21.15.23:0/840267156 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97040667e0 0x7f970413c4d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:47.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.428+0000 7f9709191700 1 -- 172.21.15.23:0/840267156 >> 172.21.15.23:0/840267156 conn(0x7f97040612c0 msgr2=0x7f97040616d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:41:47.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.430+0000 7f9709191700 1 -- 172.21.15.23:0/840267156 shutdown_connections 2020-11-20T11:41:47.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.430+0000 7f9709191700 1 -- 172.21.15.23:0/840267156 wait complete. 2020-11-20T11:41:47.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.431+0000 7f9709191700 1 Processor -- start 2020-11-20T11:41:47.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.431+0000 7f9709191700 1 -- start start 2020-11-20T11:41:47.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.432+0000 7f9709191700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9704066400 0x7f97040a5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.432+0000 7f9709191700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97040667e0 0x7f970414f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.432+0000 7f9709191700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97040a93c0 0x7f9704141800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.432+0000 7f9709191700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f970414e650 con 0x7f9704066400 2020-11-20T11:41:47.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.432+0000 7f9709191700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f970414e8f0 con 0x7f97040667e0 2020-11-20T11:41:47.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.432+0000 7f9709191700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f970414eb90 con 0x7f97040a93c0 2020-11-20T11:41:47.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.432+0000 7f9703fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9704066400 0x7f97040a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:47.437 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.432+0000 7f9703fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9704066400 0x7f97040a5200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44644/0 (socket says 172.21.15.23:44644) 2020-11-20T11:41:47.437 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.432+0000 7f97037fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97040667e0 0x7f970414f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:47.437 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.432+0000 7f9703fff700 1 -- 172.21.15.23:0/1300628442 learned_addr learned my addr 172.21.15.23:0/1300628442 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:41:47.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.432+0000 7f9703fff700 1 -- 172.21.15.23:0/1300628442 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97040a93c0 msgr2=0x7f9704141800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:41:47.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.432+0000 7f9703fff700 1 --2- 172.21.15.23:0/1300628442 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f97040a93c0 0x7f9704141800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:47.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.433+0000 7f9703fff700 1 -- 172.21.15.23:0/1300628442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97040667e0 msgr2=0x7f970414f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:47.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.433+0000 7f9703fff700 1 --2- 172.21.15.23:0/1300628442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97040667e0 0x7f970414f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:47.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.433+0000 7f9703fff700 1 -- 172.21.15.23:0/1300628442 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9704142360 con 0x7f9704066400 2020-11-20T11:41:47.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.433+0000 7f9703fff700 1 --2- 172.21.15.23:0/1300628442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9704066400 0x7f97040a5200 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f96ec00c710 tx=0x7f96ec007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:47.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.433+0000 7f96fbfff700 1 -- 172.21.15.23:0/1300628442 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ec0072e0 con 0x7f9704066400 2020-11-20T11:41:47.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.433+0000 7f96fbfff700 1 -- 172.21.15.23:0/1300628442 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96ec005570 con 0x7f9704066400 2020-11-20T11:41:47.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.433+0000 7f9709191700 1 -- 172.21.15.23:0/1300628442 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97040a1c80 con 0x7f9704066400 2020-11-20T11:41:47.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.433+0000 7f96fbfff700 1 -- 172.21.15.23:0/1300628442 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ec005910 con 0x7f9704066400 2020-11-20T11:41:47.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.433+0000 7f9709191700 1 -- 172.21.15.23:0/1300628442 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97040a39b0 con 0x7f9704066400 2020-11-20T11:41:47.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.434+0000 7f9709191700 1 --2- 172.21.15.23:0/1300628442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97040622d0 0x7f97040a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.442 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.434+0000 7f9709191700 1 -- 172.21.15.23:0/1300628442 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9704141cf0 con 0x7f97040622d0 2020-11-20T11:41:47.442 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.434+0000 7f97037fe700 1 --2- 172.21.15.23:0/1300628442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97040622d0 0x7f97040a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:47.442 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.435+0000 7f97037fe700 1 -- 172.21.15.23:0/1300628442 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f970414e4d0 con 0x7f97040622d0 2020-11-20T11:41:47.442 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.435+0000 7f96fbfff700 1 -- 172.21.15.23:0/1300628442 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f96ec01a060 con 0x7f9704066400 2020-11-20T11:41:47.442 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.435+0000 7f96fbfff700 1 --2- 172.21.15.23:0/1300628442 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f96e0056d60 0x7f96e0059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.435+0000 7f96fbfff700 1 -- 172.21.15.23:0/1300628442 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f96ec06f840 con 0x7f9704066400 2020-11-20T11:41:47.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.435+0000 7f9708990700 1 --2- 172.21.15.23:0/1300628442 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f96e0056d60 0x7f96e0059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:47.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.435+0000 7f97037fe700 1 --2- 172.21.15.23:0/1300628442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97040622d0 0x7f97040a5a80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f96f400b980 tx=0x7f96f4015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:47.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.436+0000 7f96fbfff700 1 -- 172.21.15.23:0/1300628442 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f401e8e0 con 0x7f97040622d0 2020-11-20T11:41:47.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.436+0000 7f9708990700 1 --2- 172.21.15.23:0/1300628442 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f96e0056d60 0x7f96e0059220 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f96fc001a10 tx=0x7f96fc0015f0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:47.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.436+0000 7f96fbfff700 1 -- 172.21.15.23:0/1300628442 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f96f4023070 con 0x7f97040622d0 2020-11-20T11:41:47.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.436+0000 7f96fbfff700 1 -- 172.21.15.23:0/1300628442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97040622d0 msgr2=0x7f97040a5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:47.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.436+0000 7f96fbfff700 1 --2- 172.21.15.23:0/1300628442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97040622d0 0x7f97040a5a80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f96f400b980 tx=0x7f96f4015000).stop 2020-11-20T11:41:47.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.436+0000 7f96fbfff700 1 -- 172.21.15.23:0/1300628442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97040622d0 msgr2=0x7f97040a5a80 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:41:47.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.436+0000 7f96fbfff700 1 --2- 172.21.15.23:0/1300628442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97040622d0 0x7f97040a5a80 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0x7f96f400b980 tx=0x7f96f4015000).stop 2020-11-20T11:41:47.451 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.448+0000 7f9709191700 1 --2- 172.21.15.23:0/1300628442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97040a5f70 0x7f9704151aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.448+0000 7f9709191700 1 -- 172.21.15.23:0/1300628442 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97041424f0 con 0x7f97040a5f70 2020-11-20T11:41:47.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.449+0000 7f97037fe700 1 --2- 172.21.15.23:0/1300628442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97040a5f70 0x7f9704151aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:47.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.449+0000 7f97037fe700 1 -- 172.21.15.23:0/1300628442 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9704062760 con 0x7f97040a5f70 2020-11-20T11:41:47.453 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.449+0000 7f97037fe700 1 --2- 172.21.15.23:0/1300628442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97040a5f70 0x7f9704151aa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f96f4006e70 tx=0x7f96f4021070).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:47.454 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.450+0000 7f96fbfff700 1 -- 172.21.15.23:0/1300628442 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f401e8e0 con 0x7f97040a5f70 2020-11-20T11:41:47.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.450+0000 7f96fbfff700 1 -- 172.21.15.23:0/1300628442 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4614 (secure 0 0 0) 0x7f96f4028e50 con 0x7f97040a5f70 2020-11-20T11:41:47.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.450+0000 7f96fbfff700 1 -- 172.21.15.23:0/1300628442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97040a5f70 msgr2=0x7f9704151aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:47.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.450+0000 7f96fbfff700 1 --2- 172.21.15.23:0/1300628442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97040a5f70 0x7f9704151aa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f96f4006e70 tx=0x7f96f4021070).stop 2020-11-20T11:41:47.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.450+0000 7f96fbfff700 1 -- 172.21.15.23:0/1300628442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97040a5f70 msgr2=0x7f9704151aa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:41:47.456 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.450+0000 7f96fbfff700 1 --2- 172.21.15.23:0/1300628442 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f97040a5f70 0x7f9704151aa0 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0x7f96f4006e70 tx=0x7f96f4021070).stop 2020-11-20T11:41:47.456 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.450+0000 7f9709191700 1 -- 172.21.15.23:0/1300628442 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f96e0056d60 msgr2=0x7f96e0059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:47.456 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.450+0000 7f9709191700 1 --2- 172.21.15.23:0/1300628442 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f96e0056d60 0x7f96e0059220 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f96fc001a10 tx=0x7f96fc0015f0).stop 2020-11-20T11:41:47.456 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.450+0000 7f9709191700 1 -- 172.21.15.23:0/1300628442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9704066400 msgr2=0x7f97040a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:47.456 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.450+0000 7f9709191700 1 --2- 172.21.15.23:0/1300628442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9704066400 0x7f97040a5200 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f96ec00c710 tx=0x7f96ec007c10).stop 2020-11-20T11:41:47.457 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.450+0000 7f9708990700 1 -- 172.21.15.23:0/1300628442 reap_dead start 2020-11-20T11:41:47.457 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.451+0000 7f9708990700 1 -- 172.21.15.23:0/1300628442 reap_dead start 2020-11-20T11:41:47.457 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.451+0000 7f9709191700 1 -- 172.21.15.23:0/1300628442 shutdown_connections 2020-11-20T11:41:47.457 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.451+0000 7f9709191700 1 -- 172.21.15.23:0/1300628442 >> 172.21.15.23:0/1300628442 conn(0x7f97040612c0 msgr2=0x7f970413c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:41:47.457 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.451+0000 7f9709191700 1 -- 172.21.15.23:0/1300628442 shutdown_connections 2020-11-20T11:41:47.458 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.451+0000 7f9709191700 1 -- 172.21.15.23:0/1300628442 wait complete. 2020-11-20T11:41:47.458 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:41:47.458 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:41:47.458 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:41:47.458 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:41:47.459 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 88, 2020-11-20T11:41:47.459 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:41:47.459 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:41:47.459 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:41:47.459 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:41:47.460 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:47.460 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 21, 2020-11-20T11:41:47.460 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:41:47.460 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:41:47.460 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:41:47.461 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:41:47.461 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:41:47.461 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:41:47.461 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:41:47.461 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:41:47.462 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:41:47.462 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:41:47.462 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:41:47.462 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:47.462 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:41:47.463 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:41:47.463 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:41:47.463 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:41:47.463 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:41:47.463 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:41:47.463 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:41:47.464 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:41:47.464 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:41:47.464 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:41:47.464 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:47.464 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.465 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:41:47.465 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:41:47.465 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:41:47.465 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:41:47.465 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:41:47.466 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:41:47.466 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:41:47.466 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:41:47.466 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:41:47.466 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:41:47.467 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:41:47.467 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:41:47.467 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:41:47.467 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:41:47.467 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:41:47.468 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:41:47.468 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:41:47.468 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:41:47.468 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:41:47.468 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:41:47.469 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:41:47.469 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:41:47.469 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:41:47.469 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:47.469 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:41:47.469 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.470 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:41:47.470 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.470 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:41:47.470 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:41:47.470 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:41:47.471 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:41:47.471 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.471 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:41:47.471 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:41:47.471 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:47.472 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.472 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.472 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:41:47.472 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:41:47.472 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:47.473 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:47.473 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:47.473 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.473 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:41:47.473 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:41:47.474 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:41:47.474 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:41:47.474 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:41:47.474 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.474 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.475 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:41:47.475 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:41:47.475 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:41:47.475 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:41:47.475 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.476 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:41:47.476 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:41:47.476 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:47.476 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.476 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.477 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:41:47.477 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:41:47.477 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:47.477 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:47.477 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:47.478 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.478 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:41:47.478 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:41:47.478 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:41:47.478 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:41:47.479 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:41:47.479 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.479 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.479 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:41:47.479 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:41:47.480 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:41:47.480 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:41:47.480 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.480 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:41:47.480 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:41:47.480 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:47.481 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.481 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.481 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:41:47.481 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:41:47.481 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:47.482 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:47.482 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:47.482 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.482 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:41:47.482 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:41:47.483 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:41:47.483 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:41:47.483 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:41:47.483 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:47.483 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:47.484 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.484 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:41:47.484 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:41:47.484 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.484 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:41:47.485 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:41:47.485 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:41:47.485 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:47.485 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:47.485 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:41:47.486 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.486 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:41:47.486 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:41:47.486 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:41:47.486 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:47.487 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:47.487 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:41:47.487 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.487 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:41:47.487 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:41:47.488 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:41:47.488 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:47.488 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:47.488 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:41:47.488 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.488 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:41:47.489 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:41:47.489 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:41:47.489 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:47.489 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:47.489 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.490 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:41:47.490 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:41:47.491 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:41:47.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.644+0000 7f3f45760700 1 Processor -- start 2020-11-20T11:41:47.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.644+0000 7f3f45760700 1 -- start start 2020-11-20T11:41:47.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.645+0000 7f3f45760700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f400a6e10 0x7f3f40061850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.645+0000 7f3f45760700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3f40137410 0x7f3f40137870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.661 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.645+0000 7f3f45760700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f40062ad0 con 0x7f3f4013a760 2020-11-20T11:41:47.661 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.645+0000 7f3f45760700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f400a61f0 con 0x7f3f40137410 2020-11-20T11:41:47.661 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.645+0000 7f3f45760700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f400a63e0 con 0x7f3f400a6e10 2020-11-20T11:41:47.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.645+0000 7f3f3f7fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f3f4013a760 0x7f3f4013abf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58904/0 (socket says 172.21.15.23:58904) 2020-11-20T11:41:47.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.645+0000 7f3f3f7fe700 1 -- 172.21.15.23:0/2795022595 learned_addr learned my addr 172.21.15.23:0/2795022595 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:41:47.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.645+0000 7f3f367fc700 1 --2- 172.21.15.23:0/2795022595 >> v2:172.21.15.155:3300/0 conn(0x7f3f40137410 0x7f3f40137870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:47.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.645+0000 7f3f3e7fc700 1 -- 172.21.15.23:0/2795022595 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870081433 0 0) 0x7f3f40062ad0 con 0x7f3f4013a760 2020-11-20T11:41:47.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.645+0000 7f3f3e7fc700 1 -- 172.21.15.23:0/2795022595 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f20003280 con 0x7f3f4013a760 2020-11-20T11:41:47.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.646+0000 7f3f3e7fc700 1 -- 172.21.15.23:0/2795022595 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1215197311 0 0) 0x7f3f20003280 con 0x7f3f4013a760 2020-11-20T11:41:47.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.646+0000 7f3f3e7fc700 1 -- 172.21.15.23:0/2795022595 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f400a6e10 msgr2=0x7f3f40061850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:41:47.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.646+0000 7f3f3e7fc700 1 --2- 172.21.15.23:0/2795022595 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f400a6e10 0x7f3f40061850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:47.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.646+0000 7f3f3e7fc700 1 -- 172.21.15.23:0/2795022595 >> v2:172.21.15.155:3300/0 conn(0x7f3f40137410 msgr2=0x7f3f40137870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:41:47.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.646+0000 7f3f3e7fc700 1 --2- 172.21.15.23:0/2795022595 >> v2:172.21.15.155:3300/0 conn(0x7f3f40137410 0x7f3f40137870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:47.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.646+0000 7f3f3e7fc700 1 -- 172.21.15.23:0/2795022595 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f400a6680 con 0x7f3f4013a760 2020-11-20T11:41:47.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.646+0000 7f3f3e7fc700 1 -- 172.21.15.23:0/2795022595 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f3f380018a0 con 0x7f3f4013a760 2020-11-20T11:41:47.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.646+0000 7f3f3e7fc700 1 -- 172.21.15.23:0/2795022595 >> v1:172.21.15.23:6789/0 conn(0x7f3f4013a760 legacy=0x7f3f4013abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:47.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.646+0000 7f3f3e7fc700 1 --2- 172.21.15.23:0/2795022595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f20003770 0x7f3f20003b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.646+0000 7f3f3e7fc700 1 --2- 172.21.15.23:0/2795022595 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f400a6e10 0x7f3f20004520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.646+0000 7f3f3e7fc700 1 --2- 172.21.15.23:0/2795022595 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f20004a10 0x7f3f20004e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.646+0000 7f3f3e7fc700 1 -- 172.21.15.23:0/2795022595 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f400a63e0 con 0x7f3f20003770 2020-11-20T11:41:47.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.646+0000 7f3f3e7fc700 1 -- 172.21.15.23:0/2795022595 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f20004210 con 0x7f3f20004a10 2020-11-20T11:41:47.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.646+0000 7f3f3e7fc700 1 -- 172.21.15.23:0/2795022595 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2000d980 con 0x7f3f400a6e10 2020-11-20T11:41:47.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.646+0000 7f3f3effd700 1 --2- 172.21.15.23:0/2795022595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f20003770 0x7f3f20003b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:47.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.646+0000 7f3f3f7fe700 1 --2- 172.21.15.23:0/2795022595 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f20004a10 0x7f3f20004e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:47.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.646+0000 7f3f367fc700 1 --2- 172.21.15.23:0/2795022595 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f400a6e10 0x7f3f20004520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:47.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.646+0000 7f3f3effd700 1 -- 172.21.15.23:0/2795022595 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f400a6e10 msgr2=0x7f3f20004520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:41:47.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.646+0000 7f3f3effd700 1 --2- 172.21.15.23:0/2795022595 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f400a6e10 0x7f3f20004520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:47.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.646+0000 7f3f3effd700 1 -- 172.21.15.23:0/2795022595 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f20004a10 msgr2=0x7f3f20004e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:41:47.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.646+0000 7f3f3effd700 1 --2- 172.21.15.23:0/2795022595 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f20004a10 0x7f3f20004e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:47.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.646+0000 7f3f3effd700 1 -- 172.21.15.23:0/2795022595 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f20003280 con 0x7f3f20003770 2020-11-20T11:41:47.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.646+0000 7f3f3effd700 1 --2- 172.21.15.23:0/2795022595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f20003770 0x7f3f20003b90 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f3f2800b9b0 tx=0x7f3f28006e70).ready entity=mon.0 client_cookie=d09002dcfb1b514e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.647+0000 7f3f3e7fc700 1 -- 172.21.15.23:0/2795022595 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f28009600 con 0x7f3f20003770 2020-11-20T11:41:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.647+0000 7f3f3e7fc700 1 -- 172.21.15.23:0/2795022595 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f28009760 con 0x7f3f20003770 2020-11-20T11:41:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.647+0000 7f3f3e7fc700 1 -- 172.21.15.23:0/2795022595 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f28009b20 con 0x7f3f20003770 2020-11-20T11:41:47.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.647+0000 7f3f45760700 1 -- 172.21.15.23:0/2795022595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f20003770 msgr2=0x7f3f20003b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:47.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.647+0000 7f3f45760700 1 --2- 172.21.15.23:0/2795022595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f20003770 0x7f3f20003b90 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f3f2800b9b0 tx=0x7f3f28006e70).stop 2020-11-20T11:41:47.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.647+0000 7f3f3f7fe700 1 -- 172.21.15.23:0/2795022595 reap_dead start 2020-11-20T11:41:47.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.647+0000 7f3f45760700 1 -- 172.21.15.23:0/2795022595 shutdown_connections 2020-11-20T11:41:47.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.647+0000 7f3f45760700 1 -- 172.21.15.23:0/2795022595 >> 172.21.15.23:0/2795022595 conn(0x7f3f40060c80 msgr2=0x7f3f400acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:41:47.675 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.647+0000 7f3f45760700 1 -- 172.21.15.23:0/2795022595 shutdown_connections 2020-11-20T11:41:47.675 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.647+0000 7f3f45760700 1 -- 172.21.15.23:0/2795022595 wait complete. 2020-11-20T11:41:47.675 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.648+0000 7f3f45760700 1 Processor -- start 2020-11-20T11:41:47.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.648+0000 7f3f45760700 1 -- start start 2020-11-20T11:41:47.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.649+0000 7f3f45760700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f20003770 0x7f3f40146140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.649+0000 7f3f45760700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f4009dee0 0x7f3f400a90f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.649+0000 7f3f45760700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f400a95e0 0x7f3f40066ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.649+0000 7f3f45760700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4009e360 con 0x7f3f400a95e0 2020-11-20T11:41:47.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.649+0000 7f3f45760700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f400a8cc0 con 0x7f3f4009dee0 2020-11-20T11:41:47.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.649+0000 7f3f45760700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f401467e0 con 0x7f3f20003770 2020-11-20T11:41:47.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.649+0000 7f3f3f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f400a95e0 0x7f3f40066ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:47.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.649+0000 7f3f3f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f400a95e0 0x7f3f40066ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44668/0 (socket says 172.21.15.23:44668) 2020-11-20T11:41:47.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.649+0000 7f3f3f7fe700 1 -- 172.21.15.23:0/4040103188 learned_addr learned my addr 172.21.15.23:0/4040103188 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:41:47.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.649+0000 7f3f3effd700 1 --2- 172.21.15.23:0/4040103188 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f20003770 0x7f3f40146140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:47.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.649+0000 7f3f367fc700 1 --2- 172.21.15.23:0/4040103188 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f4009dee0 0x7f3f400a90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:47.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.649+0000 7f3f3f7fe700 1 -- 172.21.15.23:0/4040103188 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f20003770 msgr2=0x7f3f40146140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:47.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.649+0000 7f3f3f7fe700 1 --2- 172.21.15.23:0/4040103188 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f20003770 0x7f3f40146140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:47.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.649+0000 7f3f3f7fe700 1 -- 172.21.15.23:0/4040103188 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f4009dee0 msgr2=0x7f3f400a90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:47.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.649+0000 7f3f3f7fe700 1 --2- 172.21.15.23:0/4040103188 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3f4009dee0 0x7f3f400a90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:47.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.649+0000 7f3f3f7fe700 1 -- 172.21.15.23:0/4040103188 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f400a9a40 con 0x7f3f400a95e0 2020-11-20T11:41:47.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.649+0000 7f3f3f7fe700 1 --2- 172.21.15.23:0/4040103188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f400a95e0 0x7f3f40066ac0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f3f400a7480 tx=0x7f3f40137210).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:47.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.650+0000 7f3f377fe700 1 -- 172.21.15.23:0/4040103188 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3800fbb0 con 0x7f3f400a95e0 2020-11-20T11:41:47.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.650+0000 7f3f377fe700 1 -- 172.21.15.23:0/4040103188 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f3800fd10 con 0x7f3f400a95e0 2020-11-20T11:41:47.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.650+0000 7f3f45760700 1 -- 172.21.15.23:0/4040103188 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f401383f0 con 0x7f3f400a95e0 2020-11-20T11:41:47.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.650+0000 7f3f377fe700 1 -- 172.21.15.23:0/4040103188 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f38015660 con 0x7f3f400a95e0 2020-11-20T11:41:47.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.650+0000 7f3f45760700 1 -- 172.21.15.23:0/4040103188 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f40064ef0 con 0x7f3f400a95e0 2020-11-20T11:41:47.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.650+0000 7f3f45760700 1 --2- 172.21.15.23:0/4040103188 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f40066fb0 0x7f3f40067390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.650+0000 7f3f45760700 1 -- 172.21.15.23:0/4040103188 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f400a8e50 con 0x7f3f40066fb0 2020-11-20T11:41:47.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.651+0000 7f3f377fe700 1 -- 172.21.15.23:0/4040103188 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3f380143b0 con 0x7f3f400a95e0 2020-11-20T11:41:47.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.651+0000 7f3f3effd700 1 --2- 172.21.15.23:0/4040103188 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f40066fb0 0x7f3f40067390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:47.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.651+0000 7f3f377fe700 1 --2- 172.21.15.23:0/4040103188 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3f1c056d90 0x7f3f1c059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.651+0000 7f3f377fe700 1 -- 172.21.15.23:0/4040103188 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f3f3806fc70 con 0x7f3f400a95e0 2020-11-20T11:41:47.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.652+0000 7f3f367fc700 1 --2- 172.21.15.23:0/4040103188 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3f1c056d90 0x7f3f1c059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:47.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.652+0000 7f3f3effd700 1 -- 172.21.15.23:0/4040103188 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3f40146630 con 0x7f3f40066fb0 2020-11-20T11:41:47.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.652+0000 7f3f3effd700 1 --2- 172.21.15.23:0/4040103188 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f40066fb0 0x7f3f40067390 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f3f2800b9b0 tx=0x7f3f28013000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:47.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.652+0000 7f3f377fe700 1 -- 172.21.15.23:0/4040103188 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f28014d30 con 0x7f3f40066fb0 2020-11-20T11:41:47.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.652+0000 7f3f367fc700 1 --2- 172.21.15.23:0/4040103188 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3f1c056d90 0x7f3f1c059250 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f3f40134ce0 tx=0x7f3f30006e30).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:47.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.652+0000 7f3f377fe700 1 -- 172.21.15.23:0/4040103188 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f3f280123c0 con 0x7f3f40066fb0 2020-11-20T11:41:47.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.652+0000 7f3f377fe700 1 -- 172.21.15.23:0/4040103188 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f40066fb0 msgr2=0x7f3f40067390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:47.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.652+0000 7f3f377fe700 1 --2- 172.21.15.23:0/4040103188 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f40066fb0 0x7f3f40067390 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f3f2800b9b0 tx=0x7f3f28013000).stop 2020-11-20T11:41:47.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.652+0000 7f3f377fe700 1 -- 172.21.15.23:0/4040103188 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f40066fb0 msgr2=0x7f3f40067390 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:41:47.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.652+0000 7f3f377fe700 1 --2- 172.21.15.23:0/4040103188 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f40066fb0 0x7f3f40067390 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:47.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.661+0000 7f3f45760700 1 --2- 172.21.15.23:0/4040103188 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f40149190 0x7f3f401495a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.661+0000 7f3f45760700 1 -- 172.21.15.23:0/4040103188 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4005c520 con 0x7f3f40149190 2020-11-20T11:41:47.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.662+0000 7f3f3effd700 1 --2- 172.21.15.23:0/4040103188 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f40149190 0x7f3f401495a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:47.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.662+0000 7f3f3effd700 1 -- 172.21.15.23:0/4040103188 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3f40044710 con 0x7f3f40149190 2020-11-20T11:41:47.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.662+0000 7f3f3effd700 1 --2- 172.21.15.23:0/4040103188 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f40149190 0x7f3f401495a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f3f28006eb0 tx=0x7f3f2801f000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:47.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.663+0000 7f3f377fe700 1 -- 172.21.15.23:0/4040103188 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f280219c0 con 0x7f3f40149190 2020-11-20T11:41:47.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.663+0000 7f3f377fe700 1 -- 172.21.15.23:0/4040103188 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4274 (secure 0 0 0) 0x7f3f28014b20 con 0x7f3f40149190 2020-11-20T11:41:47.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.663+0000 7f3f377fe700 1 -- 172.21.15.23:0/4040103188 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f40149190 msgr2=0x7f3f401495a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:47.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.663+0000 7f3f377fe700 1 --2- 172.21.15.23:0/4040103188 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f40149190 0x7f3f401495a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f3f28006eb0 tx=0x7f3f2801f000).stop 2020-11-20T11:41:47.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.663+0000 7f3f377fe700 1 -- 172.21.15.23:0/4040103188 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f40149190 msgr2=0x7f3f401495a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:41:47.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.663+0000 7f3f377fe700 1 --2- 172.21.15.23:0/4040103188 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3f40149190 0x7f3f401495a0 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0x7f3f28006eb0 tx=0x7f3f2801f000).stop 2020-11-20T11:41:47.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.664+0000 7f3f45760700 1 -- 172.21.15.23:0/4040103188 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3f1c056d90 msgr2=0x7f3f1c059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:47.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.664+0000 7f3f45760700 1 --2- 172.21.15.23:0/4040103188 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3f1c056d90 0x7f3f1c059250 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f3f40134ce0 tx=0x7f3f30006e30).stop 2020-11-20T11:41:47.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.664+0000 7f3f45760700 1 -- 172.21.15.23:0/4040103188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f400a95e0 msgr2=0x7f3f40066ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:47.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.664+0000 7f3f45760700 1 --2- 172.21.15.23:0/4040103188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f400a95e0 0x7f3f40066ac0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f3f400a7480 tx=0x7f3f40137210).stop 2020-11-20T11:41:47.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.664+0000 7f3f3f7fe700 1 -- 172.21.15.23:0/4040103188 reap_dead start 2020-11-20T11:41:47.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.664+0000 7f3f3f7fe700 1 -- 172.21.15.23:0/4040103188 reap_dead start 2020-11-20T11:41:47.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.664+0000 7f3f45760700 1 -- 172.21.15.23:0/4040103188 shutdown_connections 2020-11-20T11:41:47.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.664+0000 7f3f45760700 1 -- 172.21.15.23:0/4040103188 >> 172.21.15.23:0/4040103188 conn(0x7f3f40060c80 msgr2=0x7f3f400ad770 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:41:47.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.664+0000 7f3f45760700 1 -- 172.21.15.23:0/4040103188 shutdown_connections 2020-11-20T11:41:47.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.664+0000 7f3f45760700 1 -- 172.21.15.23:0/4040103188 wait complete. 2020-11-20T11:41:47.697 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:41:47.698 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:41:47.698 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:41:47.698 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:41:47.698 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 88, 2020-11-20T11:41:47.699 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:41:47.699 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:41:47.699 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:41:47.699 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:41:47.699 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:47.700 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 22, 2020-11-20T11:41:47.700 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:41:47.700 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:41:47.700 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:41:47.700 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:41:47.701 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:41:47.701 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:41:47.701 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:41:47.701 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:41:47.701 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:41:47.702 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:41:47.702 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:41:47.702 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:47.702 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:41:47.702 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:41:47.703 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:41:47.703 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:41:47.703 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:41:47.703 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:41:47.703 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:41:47.704 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:41:47.704 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:41:47.704 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:41:47.704 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:47.704 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.705 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:41:47.705 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:41:47.705 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:41:47.705 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:41:47.705 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:41:47.706 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:41:47.706 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:41:47.706 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:41:47.706 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:41:47.706 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:41:47.707 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:41:47.707 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:41:47.707 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:41:47.707 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:41:47.707 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:41:47.708 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:41:47.708 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:41:47.708 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:41:47.708 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:41:47.708 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:41:47.709 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:41:47.709 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:41:47.709 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:41:47.709 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:47.709 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:41:47.710 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.710 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:41:47.710 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.710 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:41:47.710 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:41:47.711 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:41:47.711 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:41:47.711 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.711 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:41:47.711 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:41:47.712 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:47.712 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.712 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.712 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:41:47.712 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:41:47.713 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:47.713 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:47.713 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:47.713 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.713 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:41:47.714 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:41:47.714 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:41:47.714 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:41:47.714 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:41:47.714 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.715 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.715 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:41:47.715 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:41:47.715 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:41:47.715 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:41:47.716 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.716 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:41:47.716 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:41:47.716 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:47.716 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.716 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.717 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:41:47.717 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:41:47.717 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:47.717 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:47.717 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:47.718 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.718 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:41:47.718 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:41:47.718 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:41:47.718 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:41:47.719 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:41:47.719 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.719 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.719 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:41:47.719 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:41:47.720 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:41:47.720 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:41:47.720 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.720 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:41:47.720 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:41:47.721 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:47.721 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.721 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.721 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:41:47.721 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:41:47.722 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:41:47.722 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:47.722 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:47.722 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.722 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:41:47.723 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:41:47.723 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:41:47.723 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:41:47.723 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:41:47.723 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:47.724 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:47.724 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.724 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:41:47.724 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:41:47.724 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.725 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:41:47.725 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:41:47.725 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:41:47.725 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:47.725 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:41:47.726 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:41:47.726 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:41:47.726 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:41:47.726 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:41:47.726 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:41:47.727 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:41:47.727 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:41:47.727 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:41:47.727 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:41:47.728 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:41:47.728 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2020-11-20T11:41:47.728 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2020-11-20T11:41:47.729 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2020-11-20T11:41:47.729 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2020-11-20T11:41:47.729 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2020-11-20T11:41:47.729 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T11:41:47.741 INFO:tasks.ceph.mon.b:Stopped 2020-11-20T11:41:47.741 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2020-11-20T11:41:47.741 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:41:47.886 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.883+0000 7f651cf17700 1 Processor -- start 2020-11-20T11:41:47.886 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.883+0000 7f651cf17700 1 -- start start 2020-11-20T11:41:47.888 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.884+0000 7f651cf17700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f65180a99f0 0x7f6518137e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.888 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.884+0000 7f651cf17700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6518138380 0x7f6518138810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.888 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.884+0000 7f651cf17700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6518138d00 con 0x7f6518061b90 2020-11-20T11:41:47.888 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.884+0000 7f651cf17700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6518067190 con 0x7f65180a99f0 2020-11-20T11:41:47.889 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.884+0000 7f651cf17700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6518067430 con 0x7f6518138380 2020-11-20T11:41:47.889 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.884+0000 7f651659c700 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f6518061b90 legacy=0x7f65180a92e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:41:47.889 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.884+0000 7f6516d9d700 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f6518138380 msgr2=0x7f6518138810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:41:47.889 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.884+0000 7f6516d9d700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6518138380 0x7f6518138810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:41:47.889 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.884+0000 7f6515d9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f65180a99f0 0x7f6518137e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:47.890 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.884+0000 7f6515d9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f65180a99f0 0x7f6518137e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44680/0 (socket says 172.21.15.23:44680) 2020-11-20T11:41:47.890 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.884+0000 7f6515d9b700 1 -- 172.21.15.23:0/132298779 learned_addr learned my addr 172.21.15.23:0/132298779 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:41:47.890 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.884+0000 7f6515d9b700 1 -- 172.21.15.23:0/132298779 >> v2:172.21.15.155:3300/0 conn(0x7f6518138380 msgr2=0x7f6518138810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:41:47.890 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.884+0000 7f6515d9b700 1 --2- 172.21.15.23:0/132298779 >> v2:172.21.15.155:3300/0 conn(0x7f6518138380 0x7f6518138810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:47.890 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.885+0000 7f6515d9b700 1 -- 172.21.15.23:0/132298779 >> v1:172.21.15.155:6789/0 conn(0x7f6518061b90 legacy=0x7f65180a92e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:41:47.891 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.885+0000 7f6515d9b700 1 -- 172.21.15.23:0/132298779 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6518067860 con 0x7f65180a99f0 2020-11-20T11:41:47.891 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.885+0000 7f6515d9b700 1 --2- 172.21.15.23:0/132298779 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65180a99f0 0x7f6518137e90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f65180628b0 tx=0x7f650c006ef0).ready entity=mon.0 client_cookie=71d2f9895caf3b3f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:47.891 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.885+0000 7f651559a700 1 -- 172.21.15.23:0/132298779 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c00b820 con 0x7f65180a99f0 2020-11-20T11:41:47.891 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.885+0000 7f651559a700 1 -- 172.21.15.23:0/132298779 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f650c000e90 con 0x7f65180a99f0 2020-11-20T11:41:47.891 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.885+0000 7f651559a700 1 -- 172.21.15.23:0/132298779 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c00bb90 con 0x7f65180a99f0 2020-11-20T11:41:47.894 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.885+0000 7f651cf17700 1 -- 172.21.15.23:0/132298779 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65180a99f0 msgr2=0x7f6518137e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:47.894 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.886+0000 7f651cf17700 1 --2- 172.21.15.23:0/132298779 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65180a99f0 0x7f6518137e90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f65180628b0 tx=0x7f650c006ef0).stop 2020-11-20T11:41:47.895 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.886+0000 7f651cf17700 1 -- 172.21.15.23:0/132298779 shutdown_connections 2020-11-20T11:41:47.896 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.886+0000 7f651cf17700 1 --2- 172.21.15.23:0/132298779 >> v2:172.21.15.155:3300/0 conn(0x7f6518138380 0x7f6518138810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:47.896 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.886+0000 7f651cf17700 1 --2- 172.21.15.23:0/132298779 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65180a99f0 0x7f6518137e90 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:47.896 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.886+0000 7f651cf17700 1 -- 172.21.15.23:0/132298779 >> 172.21.15.23:0/132298779 conn(0x7f6518063640 msgr2=0x7f65180612c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:41:47.896 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.886+0000 7f651cf17700 1 -- 172.21.15.23:0/132298779 shutdown_connections 2020-11-20T11:41:47.897 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.886+0000 7f651cf17700 1 -- 172.21.15.23:0/132298779 wait complete. 2020-11-20T11:41:47.897 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.887+0000 7f651cf17700 1 Processor -- start 2020-11-20T11:41:47.897 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.887+0000 7f651cf17700 1 -- start start 2020-11-20T11:41:47.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.888+0000 7f651cf17700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6518061b90 0x7f651809d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.888+0000 7f651cf17700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65180a99f0 0x7f65180a3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.888+0000 7f651cf17700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6518138380 0x7f65180a23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.888+0000 7f651cf17700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65180a42b0 con 0x7f6518138380 2020-11-20T11:41:47.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.888+0000 7f651cf17700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65180a80a0 con 0x7f65180a99f0 2020-11-20T11:41:47.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.888+0000 7f651cf17700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65180a3c30 con 0x7f6518061b90 2020-11-20T11:41:47.900 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.888+0000 7f6515d9b700 1 -- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65180a99f0 msgr2=0x7f65180a3740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:41:47.900 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.888+0000 7f6515d9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65180a99f0 0x7f65180a3740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:41:47.901 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.888+0000 7f6516d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6518138380 0x7f65180a23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:47.902 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.888+0000 7f6516d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6518138380 0x7f65180a23b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44686/0 (socket says 172.21.15.23:44686) 2020-11-20T11:41:47.902 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.888+0000 7f6516d9d700 1 -- 172.21.15.23:0/4243967304 learned_addr learned my addr 172.21.15.23:0/4243967304 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:41:47.903 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.888+0000 7f651659c700 1 --2- 172.21.15.23:0/4243967304 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6518061b90 0x7f651809d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:47.904 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.888+0000 7f6516d9d700 1 -- 172.21.15.23:0/4243967304 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6518061b90 msgr2=0x7f651809d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:47.904 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.888+0000 7f6516d9d700 1 --2- 172.21.15.23:0/4243967304 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6518061b90 0x7f651809d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:47.904 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.888+0000 7f6516d9d700 1 -- 172.21.15.23:0/4243967304 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65180a99f0 msgr2=0x7f65180a3740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:41:47.904 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.888+0000 7f6516d9d700 1 --2- 172.21.15.23:0/4243967304 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65180a99f0 0x7f65180a3740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:47.905 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.888+0000 7f6516d9d700 1 -- 172.21.15.23:0/4243967304 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65180605e0 con 0x7f6518138380 2020-11-20T11:41:47.905 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.888+0000 7f6516d9d700 1 --2- 172.21.15.23:0/4243967304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6518138380 0x7f65180a23b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f651000bab0 tx=0x7f65100091a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:47.905 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.889+0000 7f65067fc700 1 -- 172.21.15.23:0/4243967304 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6510003c50 con 0x7f6518138380 2020-11-20T11:41:47.905 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.889+0000 7f65067fc700 1 -- 172.21.15.23:0/4243967304 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6510003dd0 con 0x7f6518138380 2020-11-20T11:41:47.906 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.889+0000 7f651cf17700 1 -- 172.21.15.23:0/4243967304 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6518060880 con 0x7f6518138380 2020-11-20T11:41:47.906 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.889+0000 7f65067fc700 1 -- 172.21.15.23:0/4243967304 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6510006170 con 0x7f6518138380 2020-11-20T11:41:47.906 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.889+0000 7f651cf17700 1 -- 172.21.15.23:0/4243967304 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6518060370 con 0x7f6518138380 2020-11-20T11:41:47.906 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.890+0000 7f65067fc700 1 -- 172.21.15.23:0/4243967304 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f65100064e0 con 0x7f6518138380 2020-11-20T11:41:47.906 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.890+0000 7f65067fc700 1 --2- 172.21.15.23:0/4243967304 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f64ec056d10 0x7f64ec0591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:47.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.890+0000 7f65067fc700 1 -- 172.21.15.23:0/4243967304 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f6510011020 con 0x7f6518138380 2020-11-20T11:41:47.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.890+0000 7f651cf17700 1 -- 172.21.15.23:0/4243967304 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65180ab540 con 0x7f6518138380 2020-11-20T11:41:47.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.890+0000 7f651659c700 1 --2- 172.21.15.23:0/4243967304 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f64ec056d10 0x7f64ec0591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:47.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.891+0000 7f651659c700 1 --2- 172.21.15.23:0/4243967304 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f64ec056d10 0x7f64ec0591d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f6508006fd0 tx=0x7f6508006e10).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:47.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:47.894+0000 7f65067fc700 1 -- 172.21.15.23:0/4243967304 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6510044d90 con 0x7f6518138380 2020-11-20T11:41:48.186 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:48.183+0000 7f651cf17700 1 -- 172.21.15.23:0/4243967304 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6518044710 con 0x7f6518138380 2020-11-20T11:41:48.187 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:48.183+0000 7f65067fc700 1 -- 172.21.15.23:0/4243967304 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7f65100483b0 con 0x7f6518138380 2020-11-20T11:41:48.187 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:41:48.189 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:48.185+0000 7f651cf17700 1 -- 172.21.15.23:0/4243967304 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f64ec056d10 msgr2=0x7f64ec0591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:48.190 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:48.185+0000 7f651cf17700 1 --2- 172.21.15.23:0/4243967304 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f64ec056d10 0x7f64ec0591d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f6508006fd0 tx=0x7f6508006e10).stop 2020-11-20T11:41:48.190 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:48.185+0000 7f651cf17700 1 -- 172.21.15.23:0/4243967304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6518138380 msgr2=0x7f65180a23b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:48.190 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:48.185+0000 7f651cf17700 1 --2- 172.21.15.23:0/4243967304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6518138380 0x7f65180a23b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f651000bab0 tx=0x7f65100091a0).stop 2020-11-20T11:41:48.190 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:48.186+0000 7f651cf17700 1 -- 172.21.15.23:0/4243967304 shutdown_connections 2020-11-20T11:41:48.191 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:48.186+0000 7f651cf17700 1 --2- 172.21.15.23:0/4243967304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6518138380 0x7f65180a23b0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:48.191 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:48.186+0000 7f651cf17700 1 --2- 172.21.15.23:0/4243967304 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f64ec056d10 0x7f64ec0591d0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:48.191 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:48.186+0000 7f651cf17700 1 --2- 172.21.15.23:0/4243967304 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6518061b90 0x7f651809d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:48.191 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:48.186+0000 7f651cf17700 1 --2- 172.21.15.23:0/4243967304 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65180a99f0 0x7f65180a3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:48.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:48.186+0000 7f651cf17700 1 -- 172.21.15.23:0/4243967304 >> 172.21.15.23:0/4243967304 conn(0x7f6518063640 msgr2=0x7f6518136f50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:41:48.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:48.186+0000 7f651cf17700 1 -- 172.21.15.23:0/4243967304 shutdown_connections 2020-11-20T11:41:48.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:48.186+0000 7f651cf17700 1 -- 172.21.15.23:0/4243967304 wait complete. 2020-11-20T11:41:51.201 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:41:52.431 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.428+0000 7ff889108700 1 Processor -- start 2020-11-20T11:41:52.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.428+0000 7ff889108700 1 -- start start 2020-11-20T11:41:52.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.429+0000 7ff889108700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff8840608c0 0x7ff884061a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:52.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.429+0000 7ff889108700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff8840aa0e0 0x7ff884142810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:52.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.429+0000 7ff889108700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8840620d0 con 0x7ff884060ca0 2020-11-20T11:41:52.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.429+0000 7ff889108700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff884067b00 con 0x7ff8840aa0e0 2020-11-20T11:41:52.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.429+0000 7ff889108700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff884067fa0 con 0x7ff8840608c0 2020-11-20T11:41:52.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.429+0000 7ff88359e700 1 -- >> v2:172.21.15.155:3300/0 conn(0x7ff8840aa0e0 msgr2=0x7ff884142810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:41:52.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.429+0000 7ff88359e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff8840aa0e0 0x7ff884142810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:41:52.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.430+0000 7ff88259c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7ff884060ca0 0x7ff8840aa5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58950/0 (socket says 172.21.15.23:58950) 2020-11-20T11:41:52.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.430+0000 7ff88259c700 1 -- 172.21.15.23:0/2819270244 learned_addr learned my addr 172.21.15.23:0/2819270244 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:41:52.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.430+0000 7ff882d9d700 1 --2- 172.21.15.23:0/2819270244 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff8840608c0 0x7ff884061a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:52.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.430+0000 7ff881d9b700 1 -- 172.21.15.23:0/2819270244 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816425906 0 0) 0x7ff8840620d0 con 0x7ff884060ca0 2020-11-20T11:41:52.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.430+0000 7ff881d9b700 1 -- 172.21.15.23:0/2819270244 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff864003280 con 0x7ff884060ca0 2020-11-20T11:41:52.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.431+0000 7ff881d9b700 1 -- 172.21.15.23:0/2819270244 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4033792595 0 0) 0x7ff864003280 con 0x7ff884060ca0 2020-11-20T11:41:52.437 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.431+0000 7ff882d9d700 1 -- 172.21.15.23:0/2819270244 >> v2:172.21.15.155:3300/0 conn(0x7ff8840aa0e0 msgr2=0x7ff884142810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:41:52.437 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.431+0000 7ff882d9d700 1 --2- 172.21.15.23:0/2819270244 >> v2:172.21.15.155:3300/0 conn(0x7ff8840aa0e0 0x7ff884142810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:52.437 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.431+0000 7ff882d9d700 1 -- 172.21.15.23:0/2819270244 >> v1:172.21.15.23:6789/0 conn(0x7ff884060ca0 legacy=0x7ff8840aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:52.437 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.431+0000 7ff882d9d700 1 -- 172.21.15.23:0/2819270244 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff884068110 con 0x7ff8840608c0 2020-11-20T11:41:52.438 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.431+0000 7ff882d9d700 1 --2- 172.21.15.23:0/2819270244 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff8840608c0 0x7ff884061a30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7ff86c006fd0 tx=0x7ff86c006e30).ready entity=mon.2 client_cookie=73f9ea48e922d34f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:52.438 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.431+0000 7ff881d9b700 1 -- 172.21.15.23:0/2819270244 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff874001520 con 0x7ff8840608c0 2020-11-20T11:41:52.438 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.431+0000 7ff881d9b700 1 -- 172.21.15.23:0/2819270244 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff86c000e90 con 0x7ff8840608c0 2020-11-20T11:41:52.438 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.431+0000 7ff881d9b700 1 -- 172.21.15.23:0/2819270244 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86c0096e0 con 0x7ff8840608c0 2020-11-20T11:41:52.438 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.432+0000 7ff889108700 1 -- 172.21.15.23:0/2819270244 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff8840608c0 msgr2=0x7ff884061a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:52.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.432+0000 7ff889108700 1 --2- 172.21.15.23:0/2819270244 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff8840608c0 0x7ff884061a30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7ff86c006fd0 tx=0x7ff86c006e30).stop 2020-11-20T11:41:52.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.432+0000 7ff889108700 1 -- 172.21.15.23:0/2819270244 shutdown_connections 2020-11-20T11:41:52.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.432+0000 7ff889108700 1 --2- 172.21.15.23:0/2819270244 >> v2:172.21.15.155:3300/0 conn(0x7ff8840aa0e0 0x7ff884142810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:52.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.432+0000 7ff889108700 1 --2- 172.21.15.23:0/2819270244 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff8840608c0 0x7ff884061a30 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:52.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.432+0000 7ff889108700 1 -- 172.21.15.23:0/2819270244 >> 172.21.15.23:0/2819270244 conn(0x7ff884066750 msgr2=0x7ff8840601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:41:52.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.432+0000 7ff889108700 1 -- 172.21.15.23:0/2819270244 shutdown_connections 2020-11-20T11:41:52.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.432+0000 7ff889108700 1 -- 172.21.15.23:0/2819270244 wait complete. 2020-11-20T11:41:52.442 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.433+0000 7ff889108700 1 Processor -- start 2020-11-20T11:41:52.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.433+0000 7ff889108700 1 -- start start 2020-11-20T11:41:52.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.434+0000 7ff889108700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8840608c0 0x7ff884142610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:52.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.434+0000 7ff889108700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff884060ca0 0x7ff884143ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:52.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.434+0000 7ff889108700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8840aa0e0 0x7ff8840a14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:52.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.434+0000 7ff889108700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88409f090 con 0x7ff8840608c0 2020-11-20T11:41:52.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.434+0000 7ff889108700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8840a0880 con 0x7ff8840aa0e0 2020-11-20T11:41:52.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.434+0000 7ff889108700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8840a0b20 con 0x7ff884060ca0 2020-11-20T11:41:52.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.434+0000 7ff88359e700 1 -- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8840aa0e0 msgr2=0x7ff8840a14c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:41:52.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.434+0000 7ff882d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8840608c0 0x7ff884142610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:52.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.434+0000 7ff88359e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8840aa0e0 0x7ff8840a14c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:41:52.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.435+0000 7ff882d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8840608c0 0x7ff884142610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44706/0 (socket says 172.21.15.23:44706) 2020-11-20T11:41:52.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.435+0000 7ff882d9d700 1 -- 172.21.15.23:0/3055232908 learned_addr learned my addr 172.21.15.23:0/3055232908 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:41:52.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.435+0000 7ff88259c700 1 --2- 172.21.15.23:0/3055232908 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff884060ca0 0x7ff884143ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:52.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.435+0000 7ff88259c700 1 -- 172.21.15.23:0/3055232908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8840aa0e0 msgr2=0x7ff8840a14c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:41:52.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.435+0000 7ff88259c700 1 --2- 172.21.15.23:0/3055232908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8840aa0e0 0x7ff8840a14c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:52.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.435+0000 7ff88259c700 1 -- 172.21.15.23:0/3055232908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8840608c0 msgr2=0x7ff884142610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:52.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.435+0000 7ff88259c700 1 --2- 172.21.15.23:0/3055232908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8840608c0 0x7ff884142610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:52.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.435+0000 7ff88259c700 1 -- 172.21.15.23:0/3055232908 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8840a0410 con 0x7ff884060ca0 2020-11-20T11:41:52.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.435+0000 7ff882d9d700 1 --2- 172.21.15.23:0/3055232908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8840608c0 0x7ff884142610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:41:52.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.435+0000 7ff88259c700 1 --2- 172.21.15.23:0/3055232908 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff884060ca0 0x7ff884143ea0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7ff8740011e0 tx=0x7ff874001230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:52.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.435+0000 7ff87affd700 1 -- 172.21.15.23:0/3055232908 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff874007320 con 0x7ff884060ca0 2020-11-20T11:41:52.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.435+0000 7ff87affd700 1 -- 172.21.15.23:0/3055232908 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8740074a0 con 0x7ff884060ca0 2020-11-20T11:41:52.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.436+0000 7ff889108700 1 -- 172.21.15.23:0/3055232908 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff88409f7e0 con 0x7ff884060ca0 2020-11-20T11:41:52.448 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.436+0000 7ff889108700 1 -- 172.21.15.23:0/3055232908 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8840a1b70 con 0x7ff884060ca0 2020-11-20T11:41:52.448 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.436+0000 7ff87affd700 1 -- 172.21.15.23:0/3055232908 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff874005860 con 0x7ff884060ca0 2020-11-20T11:41:52.448 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.437+0000 7ff87affd700 1 -- 172.21.15.23:0/3055232908 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff874010030 con 0x7ff884060ca0 2020-11-20T11:41:52.448 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.437+0000 7ff87affd700 1 --2- 172.21.15.23:0/3055232908 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff860056d40 0x7ff860059200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:52.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.438+0000 7ff87affd700 1 -- 172.21.15.23:0/3055232908 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7ff87406fec0 con 0x7ff884060ca0 2020-11-20T11:41:52.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.440+0000 7ff882d9d700 1 --2- 172.21.15.23:0/3055232908 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff860056d40 0x7ff860059200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:52.450 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.441+0000 7ff882d9d700 1 --2- 172.21.15.23:0/3055232908 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff860056d40 0x7ff860059200 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7ff86c006c90 tx=0x7ff86c015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:41:52.450 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.444+0000 7ff889108700 1 -- 172.21.15.23:0/3055232908 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8840a0ce0 con 0x7ff884060ca0 2020-11-20T11:41:52.454 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.451+0000 7ff87affd700 1 -- 172.21.15.23:0/3055232908 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff8740451b0 con 0x7ff884060ca0 2020-11-20T11:41:52.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.764+0000 7ff889108700 1 -- 172.21.15.23:0/3055232908 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff8840a2590 con 0x7ff884060ca0 2020-11-20T11:41:52.767 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":27,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:41:52.871 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.764+0000 7ff87affd700 1 -- 172.21.15.23:0/3055232908 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7ff8740487d0 con 0x7ff884060ca0 2020-11-20T11:41:52.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.766+0000 7ff889108700 1 -- 172.21.15.23:0/3055232908 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff860056d40 msgr2=0x7ff860059200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:52.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.766+0000 7ff889108700 1 --2- 172.21.15.23:0/3055232908 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff860056d40 0x7ff860059200 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7ff86c006c90 tx=0x7ff86c015000).stop 2020-11-20T11:41:52.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.766+0000 7ff889108700 1 -- 172.21.15.23:0/3055232908 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff884060ca0 msgr2=0x7ff884143ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:52.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.766+0000 7ff889108700 1 --2- 172.21.15.23:0/3055232908 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff884060ca0 0x7ff884143ea0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7ff8740011e0 tx=0x7ff874001230).stop 2020-11-20T11:41:52.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.766+0000 7ff889108700 1 -- 172.21.15.23:0/3055232908 shutdown_connections 2020-11-20T11:41:52.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.766+0000 7ff889108700 1 --2- 172.21.15.23:0/3055232908 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff8840aa0e0 0x7ff8840a14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:52.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.766+0000 7ff889108700 1 --2- 172.21.15.23:0/3055232908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8840608c0 0x7ff884142610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:52.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.766+0000 7ff889108700 1 --2- 172.21.15.23:0/3055232908 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff860056d40 0x7ff860059200 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:52.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.766+0000 7ff889108700 1 --2- 172.21.15.23:0/3055232908 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff884060ca0 0x7ff884143ea0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:41:52.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.766+0000 7ff889108700 1 -- 172.21.15.23:0/3055232908 >> 172.21.15.23:0/3055232908 conn(0x7ff884066750 msgr2=0x7ff8840601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:41:52.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.766+0000 7ff889108700 1 -- 172.21.15.23:0/3055232908 shutdown_connections 2020-11-20T11:41:52.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:52.766+0000 7ff889108700 1 -- 172.21.15.23:0/3055232908 wait complete. 2020-11-20T11:41:55.909 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:41:56.082 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:56.076+0000 7f9ca6490700 1 Processor -- start 2020-11-20T11:41:56.082 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:56.076+0000 7f9ca6490700 1 -- start start 2020-11-20T11:41:56.082 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:56.077+0000 7f9ca6490700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f9ca01432b0 0x7f9ca013c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:56.082 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:56.077+0000 7f9ca6490700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca00a9690 con 0x7f9ca00a8c70 2020-11-20T11:41:56.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:56.077+0000 7f9ca6490700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca00a8480 con 0x7f9ca00aac70 2020-11-20T11:41:56.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:56.077+0000 7f9ca6490700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ca00a9d60 con 0x7f9ca01432b0 2020-11-20T11:41:56.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:56.077+0000 7f9c9ffff700 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f9ca00aac70 legacy=0x7f9ca0142ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:41:56.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:56.077+0000 7f9c9f7fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f9ca00a8c70 0x7f9ca013c210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:58968/0 (socket says 172.21.15.23:58968) 2020-11-20T11:41:56.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:56.077+0000 7f9c9f7fe700 1 -- 172.21.15.23:0/1050788353 learned_addr learned my addr 172.21.15.23:0/1050788353 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:41:56.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:56.077+0000 7f9ca4a2d700 1 -- 172.21.15.23:0/1050788353 >> v2:172.21.15.155:3300/0 conn(0x7f9ca01432b0 msgr2=0x7f9ca013c920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:41:56.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:56.077+0000 7f9ca4a2d700 1 --2- 172.21.15.23:0/1050788353 >> v2:172.21.15.155:3300/0 conn(0x7f9ca01432b0 0x7f9ca013c920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:41:56.281 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:56.278+0000 7f9c9ffff700 1 -- 172.21.15.23:0/1050788353 >> v1:172.21.15.155:6789/0 conn(0x7f9ca00aac70 legacy=0x7f9ca0142ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:41:56.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:56.278+0000 7f9ca4a2d700 1 -- 172.21.15.23:0/1050788353 >> v2:172.21.15.155:3300/0 conn(0x7f9ca01432b0 msgr2=0x7f9ca013c920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:41:56.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:56.278+0000 7f9ca4a2d700 1 --2- 172.21.15.23:0/1050788353 >> v2:172.21.15.155:3300/0 conn(0x7f9ca01432b0 0x7f9ca013c920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:41:56.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:56.679+0000 7f9c9ffff700 1 -- 172.21.15.23:0/1050788353 >> v1:172.21.15.155:6789/0 conn(0x7f9ca00aac70 legacy=0x7f9ca0142ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:41:56.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:56.679+0000 7f9ca4a2d700 1 -- 172.21.15.23:0/1050788353 >> v2:172.21.15.155:3300/0 conn(0x7f9ca01432b0 msgr2=0x7f9ca013c920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:41:56.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:56.679+0000 7f9ca4a2d700 1 --2- 172.21.15.23:0/1050788353 >> v2:172.21.15.155:3300/0 conn(0x7f9ca01432b0 0x7f9ca013c920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:41:57.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:57.480+0000 7f9ca4a2d700 1 -- 172.21.15.23:0/1050788353 >> v2:172.21.15.155:3300/0 conn(0x7f9ca01432b0 msgr2=0x7f9ca013c920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:41:57.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:57.480+0000 7f9c9ffff700 1 -- 172.21.15.23:0/1050788353 >> v1:172.21.15.155:6789/0 conn(0x7f9ca00aac70 legacy=0x7f9ca0142ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:41:57.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:57.480+0000 7f9ca4a2d700 1 --2- 172.21.15.23:0/1050788353 >> v2:172.21.15.155:3300/0 conn(0x7f9ca01432b0 0x7f9ca013c920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:41:59.081 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.077+0000 7f9c9dffb700 1 -- 172.21.15.23:0/1050788353 >> v2:172.21.15.155:3300/0 conn(0x7f9ca01432b0 msgr2=0x7f9ca013c920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:41:59.082 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.077+0000 7f9c9dffb700 1 --2- 172.21.15.23:0/1050788353 >> v2:172.21.15.155:3300/0 conn(0x7f9ca01432b0 0x7f9ca013c920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:41:59.082 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.077+0000 7f9c9dffb700 1 -- 172.21.15.23:0/1050788353 >> v1:172.21.15.155:6789/0 conn(0x7f9ca00aac70 legacy=0x7f9ca0142ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:41:59.082 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.077+0000 7f9c9dffb700 1 -- 172.21.15.23:0/1050788353 >> v1:172.21.15.23:6789/0 conn(0x7f9ca00a8c70 legacy=0x7f9ca013c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:41:59.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.077+0000 7f9c9dffb700 1 --2- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c800033f0 0x7f9c80005870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:59.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.077+0000 7f9c9dffb700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c80005e10 0x7f9c800082d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:41:59.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.077+0000 7f9c9dffb700 1 -- 172.21.15.23:0/1050788353 --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ca00a9d60 con 0x7f9c800033f0 2020-11-20T11:41:59.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.077+0000 7f9c9dffb700 1 -- 172.21.15.23:0/1050788353 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c800089a0 con 0x7f9c80005e10 2020-11-20T11:41:59.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.077+0000 7f9c9ffff700 1 --2- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c800033f0 0x7f9c80005870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:59.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.078+0000 7f9c9f7fe700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c80005e10 0x7f9c800082d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:59.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.078+0000 7f9c9ffff700 1 -- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c800033f0 msgr2=0x7f9c80005870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:41:59.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.078+0000 7f9c9ffff700 1 -- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c800033f0 msgr2=0x7f9c80005870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:41:59.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.078+0000 7f9c9ffff700 1 --2- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c800033f0 0x7f9c80005870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:41:59.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.078+0000 7f9c9f7fe700 1 -- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c80005e10 msgr2=0x7f9c800082d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:41:59.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.078+0000 7f9c9f7fe700 1 -- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c80005e10 msgr2=0x7f9c800082d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:41:59.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.078+0000 7f9c9ffff700 1 --2- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c800033f0 0x7f9c80005870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:41:59.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.078+0000 7f9c9f7fe700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c80005e10 0x7f9c800082d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:41:59.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.078+0000 7f9c9f7fe700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c80005e10 0x7f9c800082d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:41:59.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.282+0000 7f9c9ffff700 1 --2- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c800033f0 0x7f9c80005870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:59.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.282+0000 7f9c9ffff700 1 -- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c800033f0 msgr2=0x7f9c80005870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:41:59.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.282+0000 7f9c9ffff700 1 -- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c800033f0 msgr2=0x7f9c80005870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:41:59.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.282+0000 7f9c9ffff700 1 --2- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c800033f0 0x7f9c80005870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:41:59.287 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.282+0000 7f9c9f7fe700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c80005e10 0x7f9c800082d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:59.287 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.282+0000 7f9c9ffff700 1 --2- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c800033f0 0x7f9c80005870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:41:59.287 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.282+0000 7f9c9f7fe700 1 -- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c80005e10 msgr2=0x7f9c800082d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:41:59.288 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.282+0000 7f9c9f7fe700 1 -- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c80005e10 msgr2=0x7f9c800082d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:41:59.288 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.282+0000 7f9c9f7fe700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c80005e10 0x7f9c800082d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:41:59.289 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.282+0000 7f9c9f7fe700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c80005e10 0x7f9c800082d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:41:59.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.683+0000 7f9c9ffff700 1 --2- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c800033f0 0x7f9c80005870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:59.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.683+0000 7f9c9f7fe700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c80005e10 0x7f9c800082d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:41:59.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.683+0000 7f9c9f7fe700 1 -- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c80005e10 msgr2=0x7f9c800082d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:41:59.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.683+0000 7f9c9f7fe700 1 -- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c80005e10 msgr2=0x7f9c800082d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:41:59.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.683+0000 7f9c9f7fe700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c80005e10 0x7f9c800082d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:41:59.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.683+0000 7f9c9ffff700 1 -- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c800033f0 msgr2=0x7f9c80005870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:41:59.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.683+0000 7f9c9f7fe700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c80005e10 0x7f9c800082d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:41:59.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.683+0000 7f9c9ffff700 1 -- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c800033f0 msgr2=0x7f9c80005870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:41:59.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.683+0000 7f9c9ffff700 1 --2- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c800033f0 0x7f9c80005870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:41:59.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:41:59.683+0000 7f9c9ffff700 1 --2- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c800033f0 0x7f9c80005870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:42:00.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:00.484+0000 7f9c9f7fe700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c80005e10 0x7f9c800082d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:00.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:00.484+0000 7f9c9f7fe700 1 -- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c80005e10 msgr2=0x7f9c800082d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:42:00.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:00.484+0000 7f9c9f7fe700 1 -- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c80005e10 msgr2=0x7f9c800082d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:42:00.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:00.484+0000 7f9c9f7fe700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c80005e10 0x7f9c800082d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:42:00.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:00.484+0000 7f9c9f7fe700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c80005e10 0x7f9c800082d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:42:00.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:00.484+0000 7f9c9ffff700 1 --2- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c800033f0 0x7f9c80005870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:00.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:00.484+0000 7f9c9ffff700 1 -- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c800033f0 msgr2=0x7f9c80005870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:42:00.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:00.484+0000 7f9c9ffff700 1 -- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c800033f0 msgr2=0x7f9c80005870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:42:00.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:00.484+0000 7f9c9ffff700 1 --2- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c800033f0 0x7f9c80005870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:42:00.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:00.485+0000 7f9c9ffff700 1 --2- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c800033f0 0x7f9c80005870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:42:02.081 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.077+0000 7f9c9dffb700 1 -- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c80005e10 msgr2=0x7f9c800082d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:42:02.082 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.077+0000 7f9c9dffb700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c80005e10 0x7f9c800082d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:02.082 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.077+0000 7f9c9dffb700 1 -- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c800033f0 msgr2=0x7f9c80005870 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:42:02.082 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.077+0000 7f9c9dffb700 1 --2- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c800033f0 0x7f9c80005870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:02.082 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.077+0000 7f9ca4a2d700 1 -- 172.21.15.23:0/1050788353 reap_dead start 2020-11-20T11:42:02.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.077+0000 7f9c9dffb700 1 --2- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c8000c5b0 0x7f9c8000ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:02.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.077+0000 7f9c9dffb700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c8000f020 0x7f9c800114b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:02.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.077+0000 7f9c9dffb700 1 -- 172.21.15.23:0/1050788353 --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca00a9150 con 0x7f9ca00a8c70 2020-11-20T11:42:02.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.077+0000 7f9c9dffb700 1 -- 172.21.15.23:0/1050788353 --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ca00a9d60 con 0x7f9c8000c5b0 2020-11-20T11:42:02.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.077+0000 7f9c9dffb700 1 -- 172.21.15.23:0/1050788353 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c800089a0 con 0x7f9c8000f020 2020-11-20T11:42:02.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.078+0000 7f9c9f7fe700 1 --2- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c8000c5b0 0x7f9c8000ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:02.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.078+0000 7f9c9f7fe700 1 -- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c8000c5b0 msgr2=0x7f9c8000ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:42:02.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.078+0000 7f9ca4a2d700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c8000f020 0x7f9c800114b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:02.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.078+0000 7f9c9f7fe700 1 -- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c8000c5b0 msgr2=0x7f9c8000ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:42:02.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.078+0000 7f9c9f7fe700 1 --2- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c8000c5b0 0x7f9c8000ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:42:02.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.078+0000 7f9c9f7fe700 1 --2- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c8000c5b0 0x7f9c8000ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:42:02.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.078+0000 7f9ca4a2d700 1 -- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c8000f020 msgr2=0x7f9c800114b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:42:02.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.078+0000 7f9ca4a2d700 1 -- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c8000f020 msgr2=0x7f9c800114b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:42:02.086 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.078+0000 7f9ca4a2d700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c8000f020 0x7f9c800114b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:42:02.086 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.078+0000 7f9ca4a2d700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c8000f020 0x7f9c800114b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:42:02.154 INFO:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:42:02.157 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:42:02.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.279+0000 7f9c9f7fe700 1 --2- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c8000c5b0 0x7f9c8000ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:02.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.279+0000 7f9c9f7fe700 1 -- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c8000c5b0 msgr2=0x7f9c8000ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:42:02.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.279+0000 7f9c9f7fe700 1 -- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c8000c5b0 msgr2=0x7f9c8000ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:42:02.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.279+0000 7f9c9f7fe700 1 --2- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c8000c5b0 0x7f9c8000ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:42:02.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.279+0000 7f9c9f7fe700 1 --2- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c8000c5b0 0x7f9c8000ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:42:02.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.279+0000 7f9ca4a2d700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c8000f020 0x7f9c800114b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:02.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.279+0000 7f9ca4a2d700 1 -- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c8000f020 msgr2=0x7f9c800114b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:42:02.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.279+0000 7f9ca4a2d700 1 -- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c8000f020 msgr2=0x7f9c800114b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:42:02.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.279+0000 7f9ca4a2d700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c8000f020 0x7f9c800114b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:42:02.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.279+0000 7f9ca4a2d700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c8000f020 0x7f9c800114b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:42:02.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.680+0000 7f9c9f7fe700 1 --2- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c8000c5b0 0x7f9c8000ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:02.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.680+0000 7f9c9f7fe700 1 -- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c8000c5b0 msgr2=0x7f9c8000ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:42:02.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.680+0000 7f9c9f7fe700 1 -- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c8000c5b0 msgr2=0x7f9c8000ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:42:02.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.680+0000 7f9c9f7fe700 1 --2- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c8000c5b0 0x7f9c8000ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:42:02.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.680+0000 7f9c9f7fe700 1 --2- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c8000c5b0 0x7f9c8000ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:42:02.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.681+0000 7f9ca4a2d700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c8000f020 0x7f9c800114b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:02.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.681+0000 7f9ca4a2d700 1 -- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c8000f020 msgr2=0x7f9c800114b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:42:02.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.681+0000 7f9ca4a2d700 1 -- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c8000f020 msgr2=0x7f9c800114b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:42:02.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.681+0000 7f9ca4a2d700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c8000f020 0x7f9c800114b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:42:02.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.681+0000 7f9ca4a2d700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c8000f020 0x7f9c800114b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:42:02.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.746+0000 7f9c9effd700 1 -- 172.21.15.23:0/1050788353 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1287824548 0 0) 0x7f9ca00a9150 con 0x7f9ca00a8c70 2020-11-20T11:42:02.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.746+0000 7f9c9effd700 1 -- 172.21.15.23:0/1050788353 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c88003280 con 0x7f9ca00a8c70 2020-11-20T11:42:02.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.747+0000 7f9c9effd700 1 -- 172.21.15.23:0/1050788353 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1170696476 0 0) 0x7f9c88003280 con 0x7f9ca00a8c70 2020-11-20T11:42:02.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.747+0000 7f9c9effd700 1 -- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c8000f020 msgr2=0x7f9c800114b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:42:02.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.747+0000 7f9c9effd700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c8000f020 0x7f9c800114b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:02.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.747+0000 7f9c9effd700 1 -- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c8000c5b0 msgr2=0x7f9c8000ea40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:42:02.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.747+0000 7f9c9effd700 1 --2- 172.21.15.23:0/1050788353 >> v2:172.21.15.23:3300/0 conn(0x7f9c8000c5b0 0x7f9c8000ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:02.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.747+0000 7f9c9effd700 1 -- 172.21.15.23:0/1050788353 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca00a8480 con 0x7f9ca00a8c70 2020-11-20T11:42:02.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.747+0000 7f9c9effd700 1 -- 172.21.15.23:0/1050788353 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f9c98018320 con 0x7f9ca00a8c70 2020-11-20T11:42:02.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.748+0000 7f9c9effd700 1 -- 172.21.15.23:0/1050788353 >> v1:172.21.15.23:6789/0 conn(0x7f9ca00a8c70 legacy=0x7f9c8000ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:02.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.748+0000 7f9c9effd700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c8000f020 0x7f9c88003770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:02.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.748+0000 7f9c9effd700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c88003c60 0x7f9c88004520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:02.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.748+0000 7f9c9effd700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c880040e0 0x7f9c88004ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:02.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.748+0000 7f9c9effd700 1 -- 172.21.15.23:0/1050788353 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca00a9d60 con 0x7f9c88003c60 2020-11-20T11:42:02.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.748+0000 7f9c9effd700 1 -- 172.21.15.23:0/1050788353 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c800089a0 con 0x7f9c880040e0 2020-11-20T11:42:02.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.748+0000 7f9c9effd700 1 -- 172.21.15.23:0/1050788353 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c88005210 con 0x7f9c8000f020 2020-11-20T11:42:02.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.748+0000 7f9ca4a2d700 1 -- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c880040e0 msgr2=0x7f9c88004ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:42:02.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.748+0000 7f9ca4a2d700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c880040e0 0x7f9c88004ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:42:02.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.748+0000 7f9c9ffff700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c8000f020 0x7f9c88003770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:02.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.748+0000 7f9c9f7fe700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c88003c60 0x7f9c88004520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:02.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.749+0000 7f9c9f7fe700 1 -- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c8000f020 msgr2=0x7f9c88003770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:42:02.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.749+0000 7f9c9f7fe700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c8000f020 0x7f9c88003770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:02.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.749+0000 7f9c9f7fe700 1 -- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c880040e0 msgr2=0x7f9c88004ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:42:02.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.749+0000 7f9c9f7fe700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c880040e0 0x7f9c88004ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:02.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.749+0000 7f9c9f7fe700 1 -- 172.21.15.23:0/1050788353 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c88003280 con 0x7f9c88003c60 2020-11-20T11:42:02.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.749+0000 7f9c9f7fe700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c88003c60 0x7f9c88004520 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f9c800061f0 tx=0x7f9c94009040).ready entity=mon.0 client_cookie=87d43e54df4198cb server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:02.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.751+0000 7f9c9effd700 1 -- 172.21.15.23:0/1050788353 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c94018640 con 0x7f9c88003c60 2020-11-20T11:42:02.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.751+0000 7f9c9effd700 1 -- 172.21.15.23:0/1050788353 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c940187a0 con 0x7f9c88003c60 2020-11-20T11:42:02.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.752+0000 7f9c9effd700 1 -- 172.21.15.23:0/1050788353 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c94018b20 con 0x7f9c88003c60 2020-11-20T11:42:02.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.752+0000 7f9ca6490700 1 -- 172.21.15.23:0/1050788353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c88003c60 msgr2=0x7f9c88004520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:02.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.752+0000 7f9ca6490700 1 --2- 172.21.15.23:0/1050788353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c88003c60 0x7f9c88004520 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f9c800061f0 tx=0x7f9c94009040).stop 2020-11-20T11:42:02.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.752+0000 7f9ca4a2d700 1 -- 172.21.15.23:0/1050788353 reap_dead start 2020-11-20T11:42:02.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.752+0000 7f9ca6490700 1 -- 172.21.15.23:0/1050788353 shutdown_connections 2020-11-20T11:42:02.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.753+0000 7f9ca6490700 1 -- 172.21.15.23:0/1050788353 >> 172.21.15.23:0/1050788353 conn(0x7f9ca00678e0 msgr2=0x7f9ca0067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:02.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.753+0000 7f9ca6490700 1 -- 172.21.15.23:0/1050788353 shutdown_connections 2020-11-20T11:42:02.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.753+0000 7f9ca6490700 1 -- 172.21.15.23:0/1050788353 wait complete. 2020-11-20T11:42:02.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.754+0000 7f9ca6490700 1 Processor -- start 2020-11-20T11:42:02.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.755+0000 7f9ca6490700 1 -- start start 2020-11-20T11:42:02.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.755+0000 7f9ca6490700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ca00a6590 0x7f9ca01421d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:02.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.755+0000 7f9ca6490700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ca012a0c0 0x7f9ca012a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:02.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.755+0000 7f9ca6490700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ca012aa30 0x7f9ca00a6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:02.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.755+0000 7f9ca6490700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca00a8110 con 0x7f9ca012aa30 2020-11-20T11:42:02.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.755+0000 7f9ca6490700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca0140d90 con 0x7f9ca012a0c0 2020-11-20T11:42:02.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.755+0000 7f9ca6490700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca00b0c20 con 0x7f9ca00a6590 2020-11-20T11:42:02.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.756+0000 7f9c9f7fe700 1 -- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ca012a0c0 msgr2=0x7f9ca012a540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:42:02.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.756+0000 7f9c9f7fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ca012a0c0 0x7f9ca012a540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:42:02.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.756+0000 7f9c9ffff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ca00a6590 0x7f9ca01421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:02.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.756+0000 7f9c9ffff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ca00a6590 0x7f9ca01421d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:55012/0 (socket says 172.21.15.23:55012) 2020-11-20T11:42:02.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.756+0000 7f9c9ffff700 1 -- 172.21.15.23:0/439126312 learned_addr learned my addr 172.21.15.23:0/439126312 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:42:02.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.756+0000 7f9c9ffff700 1 -- 172.21.15.23:0/439126312 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ca012a0c0 msgr2=0x7f9ca012a540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:42:02.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.756+0000 7f9c9ffff700 1 --2- 172.21.15.23:0/439126312 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ca012a0c0 0x7f9ca012a540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:02.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.756+0000 7f9c9ffff700 1 -- 172.21.15.23:0/439126312 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ca012aa30 msgr2=0x7f9ca00a6970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:42:02.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.756+0000 7f9c9ffff700 1 --2- 172.21.15.23:0/439126312 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ca012aa30 0x7f9ca00a6970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:02.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.756+0000 7f9c9ffff700 1 -- 172.21.15.23:0/439126312 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca00a7e70 con 0x7f9ca00a6590 2020-11-20T11:42:02.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.756+0000 7f9c9ffff700 1 --2- 172.21.15.23:0/439126312 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ca00a6590 0x7f9ca01421d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f9c98014c00 tx=0x7f9c98018fd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:02.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.757+0000 7f9c7ffff700 1 -- 172.21.15.23:0/439126312 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c9802c070 con 0x7f9ca00a6590 2020-11-20T11:42:02.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.757+0000 7f9c7ffff700 1 -- 172.21.15.23:0/439126312 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c9801b030 con 0x7f9ca00a6590 2020-11-20T11:42:02.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.757+0000 7f9ca6490700 1 -- 172.21.15.23:0/439126312 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ca0140490 con 0x7f9ca00a6590 2020-11-20T11:42:02.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.757+0000 7f9c7ffff700 1 -- 172.21.15.23:0/439126312 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c9801b3d0 con 0x7f9ca00a6590 2020-11-20T11:42:02.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.757+0000 7f9ca6490700 1 -- 172.21.15.23:0/439126312 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ca012ae90 con 0x7f9ca00a6590 2020-11-20T11:42:02.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.757+0000 7f9ca6490700 1 -- 172.21.15.23:0/439126312 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ca005c7d0 con 0x7f9ca00a6590 2020-11-20T11:42:02.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.758+0000 7f9c7ffff700 1 -- 172.21.15.23:0/439126312 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9c9801c230 con 0x7f9ca00a6590 2020-11-20T11:42:02.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.758+0000 7f9c7ffff700 1 --2- 172.21.15.23:0/439126312 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9c78056d10 0x7f9c780591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:02.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.758+0000 7f9c7ffff700 1 -- 172.21.15.23:0/439126312 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f9c98082390 con 0x7f9ca00a6590 2020-11-20T11:42:02.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.759+0000 7f9c9f7fe700 1 --2- 172.21.15.23:0/439126312 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9c78056d10 0x7f9c780591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:02.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.764+0000 7f9c9f7fe700 1 --2- 172.21.15.23:0/439126312 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9c78056d10 0x7f9c780591d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f9c94007fd0 tx=0x7f9c94022000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:02.769 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:02.766+0000 7f9c7ffff700 1 -- 172.21.15.23:0/439126312 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9c98057550 con 0x7f9ca00a6590 2020-11-20T11:42:03.069 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.066+0000 7f9ca6490700 1 -- 172.21.15.23:0/439126312 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9ca0044710 con 0x7f9ca00a6590 2020-11-20T11:42:03.072 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.068+0000 7f9c7ffff700 1 -- 172.21.15.23:0/439126312 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1471 (secure 0 0 0) 0x7f9c9801c4c0 con 0x7f9ca00a6590 2020-11-20T11:42:03.072 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":90,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:42:03.073 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.070+0000 7f9ca6490700 1 -- 172.21.15.23:0/439126312 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9c78056d10 msgr2=0x7f9c780591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:03.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.070+0000 7f9ca6490700 1 --2- 172.21.15.23:0/439126312 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9c78056d10 0x7f9c780591d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f9c94007fd0 tx=0x7f9c94022000).stop 2020-11-20T11:42:03.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.070+0000 7f9ca6490700 1 -- 172.21.15.23:0/439126312 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ca00a6590 msgr2=0x7f9ca01421d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:03.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.070+0000 7f9ca6490700 1 --2- 172.21.15.23:0/439126312 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ca00a6590 0x7f9ca01421d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f9c98014c00 tx=0x7f9c98018fd0).stop 2020-11-20T11:42:03.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.070+0000 7f9ca6490700 1 -- 172.21.15.23:0/439126312 shutdown_connections 2020-11-20T11:42:03.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.070+0000 7f9ca6490700 1 --2- 172.21.15.23:0/439126312 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ca012aa30 0x7f9ca00a6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:03.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.070+0000 7f9ca6490700 1 --2- 172.21.15.23:0/439126312 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9c78056d10 0x7f9c780591d0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:03.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.070+0000 7f9ca6490700 1 --2- 172.21.15.23:0/439126312 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ca00a6590 0x7f9ca01421d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:03.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.070+0000 7f9ca6490700 1 --2- 172.21.15.23:0/439126312 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9ca012a0c0 0x7f9ca012a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:03.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.070+0000 7f9ca6490700 1 -- 172.21.15.23:0/439126312 >> 172.21.15.23:0/439126312 conn(0x7f9ca00678e0 msgr2=0x7f9ca0132310 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:03.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.071+0000 7f9ca6490700 1 -- 172.21.15.23:0/439126312 shutdown_connections 2020-11-20T11:42:03.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.071+0000 7f9ca6490700 1 -- 172.21.15.23:0/439126312 wait complete. 2020-11-20T11:42:03.095 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2020-11-20T11:42:03.095 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:42:03.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.323+0000 7ff16fa7b700 1 Processor -- start 2020-11-20T11:42:03.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.323+0000 7ff16fa7b700 1 -- start start 2020-11-20T11:42:03.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.324+0000 7ff16fa7b700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff16807f7f0 0x7ff1680a5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:03.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.324+0000 7ff16fa7b700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1680a4330 con 0x7ff16807fbd0 2020-11-20T11:42:03.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.324+0000 7ff16fa7b700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1680ad600 con 0x7ff168063fe0 2020-11-20T11:42:03.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.324+0000 7ff16fa7b700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff168062670 con 0x7ff16807f7f0 2020-11-20T11:42:03.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.324+0000 7ff16d817700 1 -- >> v2:172.21.15.155:3300/0 conn(0x7ff16807f7f0 msgr2=0x7ff1680a5a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:42:03.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.324+0000 7ff16e018700 1 -- >> v1:172.21.15.155:6789/0 conn(0x7ff168063fe0 legacy=0x7ff168061f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:42:03.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.324+0000 7ff16d817700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff16807f7f0 0x7ff1680a5a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:42:03.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.324+0000 7ff16d016700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7ff16807fbd0 0x7ff168061850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:59028/0 (socket says 172.21.15.23:59028) 2020-11-20T11:42:03.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.324+0000 7ff16d016700 1 -- 172.21.15.23:0/4058293649 learned_addr learned my addr 172.21.15.23:0/4058293649 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:42:03.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.324+0000 7ff16c815700 1 -- 172.21.15.23:0/4058293649 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 394102708 0 0) 0x7ff1680a4330 con 0x7ff16807fbd0 2020-11-20T11:42:03.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.324+0000 7ff16c815700 1 -- 172.21.15.23:0/4058293649 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff150003280 con 0x7ff16807fbd0 2020-11-20T11:42:03.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.325+0000 7ff16c815700 1 -- 172.21.15.23:0/4058293649 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2175730127 0 0) 0x7ff150003280 con 0x7ff16807fbd0 2020-11-20T11:42:03.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.325+0000 7ff16c815700 1 -- 172.21.15.23:0/4058293649 >> v2:172.21.15.155:3300/0 conn(0x7ff16807f7f0 msgr2=0x7ff1680a5a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:42:03.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.477+0000 7ff16c815700 1 --2- 172.21.15.23:0/4058293649 >> v2:172.21.15.155:3300/0 conn(0x7ff16807f7f0 0x7ff1680a5a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:03.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.477+0000 7ff16c815700 1 -- 172.21.15.23:0/4058293649 >> v1:172.21.15.155:6789/0 conn(0x7ff168063fe0 legacy=0x7ff168061f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:42:03.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.477+0000 7ff16c815700 1 -- 172.21.15.23:0/4058293649 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1680a4560 con 0x7ff16807fbd0 2020-11-20T11:42:03.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.477+0000 7ff16c815700 1 -- 172.21.15.23:0/4058293649 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7ff15c001520 con 0x7ff16807fbd0 2020-11-20T11:42:03.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.477+0000 7ff16c815700 1 -- 172.21.15.23:0/4058293649 >> v1:172.21.15.23:6789/0 conn(0x7ff16807fbd0 legacy=0x7ff168061850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:03.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.477+0000 7ff16c815700 1 --2- 172.21.15.23:0/4058293649 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff150003770 0x7ff150003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:03.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.477+0000 7ff16c815700 1 --2- 172.21.15.23:0/4058293649 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff150004520 0x7ff1500049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:03.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.477+0000 7ff16c815700 1 --2- 172.21.15.23:0/4058293649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff150004e90 0x7ff15000b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:03.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.477+0000 7ff16c815700 1 -- 172.21.15.23:0/4058293649 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff168062670 con 0x7ff150004e90 2020-11-20T11:42:03.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.477+0000 7ff16c815700 1 -- 172.21.15.23:0/4058293649 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff150004230 con 0x7ff150003770 2020-11-20T11:42:03.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.477+0000 7ff16c815700 1 -- 172.21.15.23:0/4058293649 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff15000bc60 con 0x7ff150004520 2020-11-20T11:42:03.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.477+0000 7ff16e018700 1 --2- 172.21.15.23:0/4058293649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff150004e90 0x7ff15000b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:03.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.477+0000 7ff16d817700 1 -- 172.21.15.23:0/4058293649 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff150003770 msgr2=0x7ff150003b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:42:03.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.477+0000 7ff16d817700 1 --2- 172.21.15.23:0/4058293649 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff150003770 0x7ff150003b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:42:03.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.477+0000 7ff16d016700 1 --2- 172.21.15.23:0/4058293649 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff150004520 0x7ff1500049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:03.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.478+0000 7ff16e018700 1 -- 172.21.15.23:0/4058293649 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff150004520 msgr2=0x7ff1500049a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:42:03.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.478+0000 7ff16e018700 1 --2- 172.21.15.23:0/4058293649 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff150004520 0x7ff1500049a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:03.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.478+0000 7ff16e018700 1 -- 172.21.15.23:0/4058293649 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff150003770 msgr2=0x7ff150003b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:42:03.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.478+0000 7ff16e018700 1 --2- 172.21.15.23:0/4058293649 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff150003770 0x7ff150003b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:03.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.478+0000 7ff16e018700 1 -- 172.21.15.23:0/4058293649 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff150003280 con 0x7ff150004e90 2020-11-20T11:42:03.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.478+0000 7ff16e018700 1 -- 172.21.15.23:0/4058293649 reap_dead start 2020-11-20T11:42:03.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.478+0000 7ff16e018700 1 --2- 172.21.15.23:0/4058293649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff150004e90 0x7ff15000b770 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7ff15800b940 tx=0x7ff158006e00).ready entity=mon.0 client_cookie=c306770f2ece6fdf server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:03.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.478+0000 7ff16c815700 1 -- 172.21.15.23:0/4058293649 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1580050d0 con 0x7ff150004e90 2020-11-20T11:42:03.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.478+0000 7ff16c815700 1 -- 172.21.15.23:0/4058293649 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff158005250 con 0x7ff150004e90 2020-11-20T11:42:03.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.478+0000 7ff16c815700 1 -- 172.21.15.23:0/4058293649 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff158003850 con 0x7ff150004e90 2020-11-20T11:42:03.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.478+0000 7ff16fa7b700 1 -- 172.21.15.23:0/4058293649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff150004e90 msgr2=0x7ff15000b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:03.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.478+0000 7ff16fa7b700 1 --2- 172.21.15.23:0/4058293649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff150004e90 0x7ff15000b770 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7ff15800b940 tx=0x7ff158006e00).stop 2020-11-20T11:42:03.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.479+0000 7ff16fa7b700 1 -- 172.21.15.23:0/4058293649 shutdown_connections 2020-11-20T11:42:03.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.479+0000 7ff16fa7b700 1 --2- 172.21.15.23:0/4058293649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff150004e90 0x7ff15000b770 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:03.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.479+0000 7ff16fa7b700 1 -- 172.21.15.23:0/4058293649 >> 172.21.15.23:0/4058293649 conn(0x7ff168060c80 msgr2=0x7ff1680acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:03.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.479+0000 7ff16fa7b700 1 -- 172.21.15.23:0/4058293649 shutdown_connections 2020-11-20T11:42:03.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.479+0000 7ff16fa7b700 1 -- 172.21.15.23:0/4058293649 wait complete. 2020-11-20T11:42:03.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.481+0000 7ff16fa7b700 1 Processor -- start 2020-11-20T11:42:03.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.481+0000 7ff16fa7b700 1 -- start start 2020-11-20T11:42:03.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.481+0000 7ff16fa7b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff150004e90 0x7ff168139b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:03.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.481+0000 7ff16fa7b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff168138250 0x7ff1681386d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:03.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.482+0000 7ff16fa7b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1680aad00 0x7ff1680ab180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:03.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.482+0000 7ff16fa7b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff16813be70 con 0x7ff1680aad00 2020-11-20T11:42:03.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.482+0000 7ff16fa7b700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff16813c110 con 0x7ff168138250 2020-11-20T11:42:03.497 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.482+0000 7ff16fa7b700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff16813c3b0 con 0x7ff150004e90 2020-11-20T11:42:03.497 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.482+0000 7ff16d016700 1 -- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff168138250 msgr2=0x7ff1681386d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:42:03.497 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.482+0000 7ff16d016700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff168138250 0x7ff1681386d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:42:03.497 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.482+0000 7ff16d817700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff150004e90 0x7ff168139b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:03.497 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.482+0000 7ff16d817700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff150004e90 0x7ff168139b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:55028/0 (socket says 172.21.15.23:55028) 2020-11-20T11:42:03.498 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.482+0000 7ff16d817700 1 -- 172.21.15.23:0/2464104817 learned_addr learned my addr 172.21.15.23:0/2464104817 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:42:03.498 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.482+0000 7ff16e018700 1 --2- 172.21.15.23:0/2464104817 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1680aad00 0x7ff1680ab180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:03.498 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.482+0000 7ff16d817700 1 -- 172.21.15.23:0/2464104817 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff168138250 msgr2=0x7ff1681386d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:42:03.498 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.482+0000 7ff16d817700 1 --2- 172.21.15.23:0/2464104817 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff168138250 0x7ff1681386d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:03.498 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.482+0000 7ff16d817700 1 -- 172.21.15.23:0/2464104817 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1680aad00 msgr2=0x7ff1680ab180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:03.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.482+0000 7ff16d817700 1 --2- 172.21.15.23:0/2464104817 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1680aad00 0x7ff1680ab180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:03.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.482+0000 7ff16d817700 1 -- 172.21.15.23:0/2464104817 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff16813c650 con 0x7ff150004e90 2020-11-20T11:42:03.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.482+0000 7ff16d817700 1 --2- 172.21.15.23:0/2464104817 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff150004e90 0x7ff168139b60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7ff16000bb90 tx=0x7ff160009310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:03.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.483+0000 7ff165ffb700 1 -- 172.21.15.23:0/2464104817 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff160003d30 con 0x7ff150004e90 2020-11-20T11:42:03.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.483+0000 7ff165ffb700 1 -- 172.21.15.23:0/2464104817 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff160011030 con 0x7ff150004e90 2020-11-20T11:42:03.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.483+0000 7ff165ffb700 1 -- 172.21.15.23:0/2464104817 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1600063c0 con 0x7ff150004e90 2020-11-20T11:42:03.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.483+0000 7ff16fa7b700 1 -- 172.21.15.23:0/2464104817 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff16813caf0 con 0x7ff150004e90 2020-11-20T11:42:03.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.483+0000 7ff16fa7b700 1 -- 172.21.15.23:0/2464104817 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff16813a050 con 0x7ff150004e90 2020-11-20T11:42:03.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.484+0000 7ff16fa7b700 1 --2- 172.21.15.23:0/2464104817 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1680ab670 0x7ff168147340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:03.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.484+0000 7ff16fa7b700 1 -- 172.21.15.23:0/2464104817 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff168062b00 con 0x7ff1680ab670 2020-11-20T11:42:03.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.485+0000 7ff165ffb700 1 -- 172.21.15.23:0/2464104817 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff1600067b0 con 0x7ff150004e90 2020-11-20T11:42:03.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.485+0000 7ff165ffb700 1 --2- 172.21.15.23:0/2464104817 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff140056d10 0x7ff1400591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:03.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.485+0000 7ff165ffb700 1 -- 172.21.15.23:0/2464104817 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7ff16006fb00 con 0x7ff150004e90 2020-11-20T11:42:03.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.485+0000 7ff16d016700 1 --2- 172.21.15.23:0/2464104817 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1680ab670 0x7ff168147340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:03.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.485+0000 7ff16e018700 1 --2- 172.21.15.23:0/2464104817 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff140056d10 0x7ff1400591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:03.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.486+0000 7ff16d016700 1 -- 172.21.15.23:0/2464104817 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff168061650 con 0x7ff1680ab670 2020-11-20T11:42:03.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.486+0000 7ff16d016700 1 --2- 172.21.15.23:0/2464104817 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1680ab670 0x7ff168147340 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7ff1681493a0 tx=0x7ff1680a46f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:03.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.486+0000 7ff165ffb700 1 -- 172.21.15.23:0/2464104817 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff15c00a5f0 con 0x7ff1680ab670 2020-11-20T11:42:03.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.486+0000 7ff16e018700 1 --2- 172.21.15.23:0/2464104817 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff140056d10 0x7ff1400591d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7ff150005500 tx=0x7ff158015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:03.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.486+0000 7ff165ffb700 1 -- 172.21.15.23:0/2464104817 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7ff15c010040 con 0x7ff1680ab670 2020-11-20T11:42:03.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.487+0000 7ff165ffb700 1 -- 172.21.15.23:0/2464104817 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1680ab670 msgr2=0x7ff168147340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:03.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.487+0000 7ff165ffb700 1 --2- 172.21.15.23:0/2464104817 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1680ab670 0x7ff168147340 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7ff1681493a0 tx=0x7ff1680a46f0).stop 2020-11-20T11:42:03.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.487+0000 7ff165ffb700 1 -- 172.21.15.23:0/2464104817 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1680ab670 msgr2=0x7ff168147340 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:42:03.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.487+0000 7ff165ffb700 1 --2- 172.21.15.23:0/2464104817 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1680ab670 0x7ff168147340 secure :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0x7ff1681493a0 tx=0x7ff1680a46f0).stop 2020-11-20T11:42:03.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.499+0000 7ff16fa7b700 1 --2- 172.21.15.23:0/2464104817 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff168147830 0x7ff168147c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:03.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.499+0000 7ff16fa7b700 1 -- 172.21.15.23:0/2464104817 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff16805c520 con 0x7ff168147830 2020-11-20T11:42:03.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.500+0000 7ff16d016700 1 --2- 172.21.15.23:0/2464104817 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff168147830 0x7ff168147c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:03.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.500+0000 7ff16d016700 1 -- 172.21.15.23:0/2464104817 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff168044710 con 0x7ff168147830 2020-11-20T11:42:03.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.501+0000 7ff16d016700 1 --2- 172.21.15.23:0/2464104817 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff168147830 0x7ff168147c40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7ff1681493a0 tx=0x7ff15c013000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:03.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.501+0000 7ff165ffb700 1 -- 172.21.15.23:0/2464104817 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff15c016e00 con 0x7ff168147830 2020-11-20T11:42:03.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.502+0000 7ff165ffb700 1 -- 172.21.15.23:0/2464104817 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4774 (secure 0 0 0) 0x7ff15c0143a0 con 0x7ff168147830 2020-11-20T11:42:03.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.502+0000 7ff165ffb700 1 -- 172.21.15.23:0/2464104817 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff168147830 msgr2=0x7ff168147c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:03.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.502+0000 7ff165ffb700 1 --2- 172.21.15.23:0/2464104817 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff168147830 0x7ff168147c40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7ff1681493a0 tx=0x7ff15c013000).stop 2020-11-20T11:42:03.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.502+0000 7ff165ffb700 1 -- 172.21.15.23:0/2464104817 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff168147830 msgr2=0x7ff168147c40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:42:03.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.502+0000 7ff165ffb700 1 --2- 172.21.15.23:0/2464104817 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff168147830 0x7ff168147c40 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:03.507 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:42:03.507 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:42:03.508 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:42:03.508 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:42:03.508 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 90, 2020-11-20T11:42:03.508 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:42:03.508 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:42:03.509 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:42:03.509 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:03.509 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 0, 2020-11-20T11:42:03.509 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:42:03.509 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:42:03.510 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:42:03.510 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:42:03.510 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:42:03.510 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:42:03.510 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:42:03.511 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:42:03.511 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:42:03.511 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:42:03.511 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:42:03.511 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:03.512 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:42:03.512 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:42:03.512 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:42:03.512 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:42:03.512 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:42:03.512 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:42:03.513 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:42:03.513 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:42:03.513 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:42:03.513 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:42:03.513 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:03.514 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:03.514 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:42:03.514 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:42:03.514 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:42:03.514 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:42:03.515 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:42:03.515 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:42:03.515 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:42:03.515 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:42:03.515 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:42:03.516 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:42:03.516 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:42:03.516 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:42:03.516 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:42:03.516 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:42:03.517 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:42:03.517 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:42:03.517 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:42:03.517 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:42:03.517 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:42:03.518 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:42:03.518 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:42:03.518 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:42:03.518 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:42:03.518 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:03.519 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:42:03.519 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:03.519 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:42:03.519 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:03.519 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:42:03.520 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:42:03.520 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:42:03.520 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:42:03.520 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:03.520 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:42:03.520 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:42:03.521 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:03.521 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:03.521 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:03.521 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:42:03.522 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:42:03.522 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:03.522 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:03.522 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:03.522 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:03.523 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:42:03.523 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:42:03.523 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:42:03.523 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:42:03.524 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:42:03.524 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:03.524 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:03.524 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:42:03.524 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:42:03.525 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:42:03.525 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:42:03.525 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:03.525 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:42:03.525 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:42:03.526 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:03.526 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:03.526 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:03.526 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:42:03.526 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:42:03.527 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:03.527 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:03.527 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:03.527 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:03.528 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:42:03.528 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:42:03.528 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:42:03.528 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:42:03.528 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:42:03.529 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:03.529 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:03.529 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:42:03.529 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:42:03.529 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:42:03.530 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:42:03.530 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:03.530 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:42:03.530 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:42:03.530 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:03.531 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:03.531 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:03.531 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:42:03.531 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:42:03.531 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:03.531 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:03.532 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:03.532 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:03.532 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:42:03.532 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:42:03.532 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:42:03.533 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:42:03.533 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:42:03.533 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:03.533 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:03.533 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:03.534 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:42:03.534 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:42:03.534 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:03.534 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:03.534 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:03.535 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:42:03.535 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:03.535 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:03.535 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:42:03.536 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:03.536 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:03.536 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:03.536 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:42:03.536 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:03.536 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:03.537 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:42:03.537 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:03.537 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:03.537 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:03.537 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 8 2020-11-20T11:42:03.538 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:03.538 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:03.538 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:42:03.538 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:03.538 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:03.539 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:03.539 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 4 2020-11-20T11:42:03.539 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:03.539 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:03.539 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:42:03.540 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:03.540 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:03.540 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:03.540 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:42:03.540 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:03.541 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:03.541 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:03.541 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:42:03.541 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:42:03.543 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.503+0000 7ff16fa7b700 1 -- 172.21.15.23:0/2464104817 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff140056d10 msgr2=0x7ff1400591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:03.543 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.504+0000 7ff16fa7b700 1 --2- 172.21.15.23:0/2464104817 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff140056d10 0x7ff1400591d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7ff150005500 tx=0x7ff158015000).stop 2020-11-20T11:42:03.543 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.504+0000 7ff16fa7b700 1 -- 172.21.15.23:0/2464104817 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff150004e90 msgr2=0x7ff168139b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:03.543 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.504+0000 7ff16fa7b700 1 --2- 172.21.15.23:0/2464104817 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff150004e90 0x7ff168139b60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7ff16000bb90 tx=0x7ff160009310).stop 2020-11-20T11:42:03.544 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.506+0000 7ff16e018700 1 -- 172.21.15.23:0/2464104817 reap_dead start 2020-11-20T11:42:03.544 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.506+0000 7ff16e018700 1 -- 172.21.15.23:0/2464104817 reap_dead start 2020-11-20T11:42:03.544 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.506+0000 7ff16fa7b700 1 -- 172.21.15.23:0/2464104817 shutdown_connections 2020-11-20T11:42:03.544 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.506+0000 7ff16fa7b700 1 -- 172.21.15.23:0/2464104817 >> 172.21.15.23:0/2464104817 conn(0x7ff168060c80 msgr2=0x7ff1680acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:03.544 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.506+0000 7ff16fa7b700 1 -- 172.21.15.23:0/2464104817 shutdown_connections 2020-11-20T11:42:03.545 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.506+0000 7ff16fa7b700 1 -- 172.21.15.23:0/2464104817 wait complete. 2020-11-20T11:42:03.545 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:42:03.744 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.741+0000 7fcfa87fe700 1 Processor -- start 2020-11-20T11:42:03.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.741+0000 7fcfa87fe700 1 -- start start 2020-11-20T11:42:03.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.742+0000 7fcfa87fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fcfa00aae90 0x7fcfa01381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:03.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.742+0000 7fcfa659a700 1 -- >> v1:172.21.15.155:6789/0 conn(0x7fcfa00aaab0 legacy=0x7fcfa013ed40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:42:03.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.742+0000 7fcfa87fe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfa013aeb0 0x7fcfa00a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:03.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.742+0000 7fcfa5d99700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fcfa00aae90 0x7fcfa01381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:03.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.742+0000 7fcfa5d99700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fcfa00aae90 0x7fcfa01381a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44802/0 (socket says 172.21.15.23:44802) 2020-11-20T11:42:03.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.742+0000 7fcfa5d99700 1 -- 172.21.15.23:0/2674180662 learned_addr learned my addr 172.21.15.23:0/2674180662 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:42:03.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.742+0000 7fcfa87fe700 1 -- 172.21.15.23:0/2674180662 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa013e600 con 0x7fcfa00aaab0 2020-11-20T11:42:03.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.742+0000 7fcfa87fe700 1 -- 172.21.15.23:0/2674180662 --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfa013e8a0 con 0x7fcfa00aae90 2020-11-20T11:42:03.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.742+0000 7fcfa87fe700 1 -- 172.21.15.23:0/2674180662 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa00ab6d0 con 0x7fcfa013aeb0 2020-11-20T11:42:03.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.742+0000 7fcfa6d9b700 1 --2- 172.21.15.23:0/2674180662 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfa013aeb0 0x7fcfa00a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:03.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.743+0000 7fcfa5d99700 1 -- 172.21.15.23:0/2674180662 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfa013aeb0 msgr2=0x7fcfa00a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:42:03.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.743+0000 7fcfa5d99700 1 --2- 172.21.15.23:0/2674180662 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfa013aeb0 0x7fcfa00a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:03.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.743+0000 7fcfa5d99700 1 -- 172.21.15.23:0/2674180662 >> v1:172.21.15.155:6789/0 conn(0x7fcfa00aaab0 legacy=0x7fcfa013ed40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:42:03.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.743+0000 7fcfa5d99700 1 -- 172.21.15.23:0/2674180662 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa0138690 con 0x7fcfa00aae90 2020-11-20T11:42:03.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.743+0000 7fcfa5d99700 1 --2- 172.21.15.23:0/2674180662 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfa00aae90 0x7fcfa01381a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fcf90009a10 tx=0x7fcf90006ef0).ready entity=mon.0 client_cookie=fd1fa3165b9d7dc8 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:03.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.745+0000 7fcfa5598700 1 -- 172.21.15.23:0/2674180662 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9000b820 con 0x7fcfa00aae90 2020-11-20T11:42:03.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.745+0000 7fcfa5598700 1 -- 172.21.15.23:0/2674180662 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf90000e90 con 0x7fcfa00aae90 2020-11-20T11:42:03.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.745+0000 7fcfa5598700 1 -- 172.21.15.23:0/2674180662 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9000bb90 con 0x7fcfa00aae90 2020-11-20T11:42:03.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.746+0000 7fcfa87fe700 1 -- 172.21.15.23:0/2674180662 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfa00aae90 msgr2=0x7fcfa01381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:03.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.746+0000 7fcfa87fe700 1 --2- 172.21.15.23:0/2674180662 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfa00aae90 0x7fcfa01381a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7fcf90009a10 tx=0x7fcf90006ef0).stop 2020-11-20T11:42:03.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.746+0000 7fcfa87fe700 1 -- 172.21.15.23:0/2674180662 shutdown_connections 2020-11-20T11:42:03.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.746+0000 7fcfa87fe700 1 --2- 172.21.15.23:0/2674180662 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfa013aeb0 0x7fcfa00a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:03.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.746+0000 7fcfa87fe700 1 --2- 172.21.15.23:0/2674180662 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfa00aae90 0x7fcfa01381a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:03.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.746+0000 7fcfa87fe700 1 -- 172.21.15.23:0/2674180662 >> 172.21.15.23:0/2674180662 conn(0x7fcfa00678e0 msgr2=0x7fcfa0067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:03.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.746+0000 7fcfa87fe700 1 -- 172.21.15.23:0/2674180662 shutdown_connections 2020-11-20T11:42:03.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.746+0000 7fcfa87fe700 1 -- 172.21.15.23:0/2674180662 wait complete. 2020-11-20T11:42:03.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.747+0000 7fcfa87fe700 1 Processor -- start 2020-11-20T11:42:03.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.748+0000 7fcfa87fe700 1 -- start start 2020-11-20T11:42:03.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.748+0000 7fcfa87fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfa00aaab0 0x7fcfa00a7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:03.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.749+0000 7fcfa659a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfa00aaab0 0x7fcfa00a7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:03.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.749+0000 7fcfa659a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfa00aaab0 0x7fcfa00a7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44806/0 (socket says 172.21.15.23:44806) 2020-11-20T11:42:03.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.749+0000 7fcfa87fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfa00aae90 0x7fcfa00690c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:03.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.749+0000 7fcfa659a700 1 -- 172.21.15.23:0/3068754133 learned_addr learned my addr 172.21.15.23:0/3068754133 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:42:03.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.749+0000 7fcfa87fe700 1 --2- 172.21.15.23:0/3068754133 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfa009ee20 0x7fcfa00a64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:03.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.749+0000 7fcfa87fe700 1 -- 172.21.15.23:0/3068754133 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa00a5650 con 0x7fcfa00aaab0 2020-11-20T11:42:03.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.749+0000 7fcfa87fe700 1 -- 172.21.15.23:0/3068754133 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa014d860 con 0x7fcfa00aae90 2020-11-20T11:42:03.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.749+0000 7fcfa87fe700 1 -- 172.21.15.23:0/3068754133 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa00a79a0 con 0x7fcfa009ee20 2020-11-20T11:42:03.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.749+0000 7fcfa5d99700 1 -- 172.21.15.23:0/3068754133 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfa00aae90 msgr2=0x7fcfa00690c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:42:03.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.749+0000 7fcfa5d99700 1 --2- 172.21.15.23:0/3068754133 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfa00aae90 0x7fcfa00690c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:42:03.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.749+0000 7fcfa659a700 1 -- 172.21.15.23:0/3068754133 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfa009ee20 msgr2=0x7fcfa00a64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:03.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.749+0000 7fcfa659a700 1 --2- 172.21.15.23:0/3068754133 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfa009ee20 0x7fcfa00a64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:03.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.749+0000 7fcfa659a700 1 -- 172.21.15.23:0/3068754133 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfa00aae90 msgr2=0x7fcfa00690c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:42:03.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.749+0000 7fcfa659a700 1 --2- 172.21.15.23:0/3068754133 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcfa00aae90 0x7fcfa00690c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:03.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.749+0000 7fcfa659a700 1 -- 172.21.15.23:0/3068754133 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa00a7c40 con 0x7fcfa00aaab0 2020-11-20T11:42:03.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.749+0000 7fcfa659a700 1 --2- 172.21.15.23:0/3068754133 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfa00aaab0 0x7fcfa00a7370 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7fcf9c009790 tx=0x7fcf9c006ea0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:03.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.749+0000 7fcf8e7fc700 1 -- 172.21.15.23:0/3068754133 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c006170 con 0x7fcfa00aaab0 2020-11-20T11:42:03.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.750+0000 7fcf8e7fc700 1 -- 172.21.15.23:0/3068754133 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf9c0062d0 con 0x7fcfa00aaab0 2020-11-20T11:42:03.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.750+0000 7fcfa87fe700 1 -- 172.21.15.23:0/3068754133 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa00a6050 con 0x7fcfa00aaab0 2020-11-20T11:42:03.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.750+0000 7fcf8e7fc700 1 -- 172.21.15.23:0/3068754133 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c003920 con 0x7fcfa00aaab0 2020-11-20T11:42:03.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.750+0000 7fcfa87fe700 1 -- 172.21.15.23:0/3068754133 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa00a6c20 con 0x7fcfa00aaab0 2020-11-20T11:42:03.769 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.751+0000 7fcfa87fe700 1 --2- 172.21.15.23:0/3068754133 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfa013aeb0 0x7fcfa013d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:03.769 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.751+0000 7fcfa87fe700 1 -- 172.21.15.23:0/3068754133 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa005c7d0 con 0x7fcfa013aeb0 2020-11-20T11:42:03.769 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.751+0000 7fcf8e7fc700 1 -- 172.21.15.23:0/3068754133 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fcf9c017030 con 0x7fcfa00aaab0 2020-11-20T11:42:03.769 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.751+0000 7fcf8e7fc700 1 --2- 172.21.15.23:0/3068754133 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcf78056d10 0x7fcf780591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:03.770 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.751+0000 7fcf8e7fc700 1 -- 172.21.15.23:0/3068754133 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fcf9c06f290 con 0x7fcfa00aaab0 2020-11-20T11:42:03.770 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.751+0000 7fcfa5d99700 1 --2- 172.21.15.23:0/3068754133 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfa013aeb0 0x7fcfa013d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:03.770 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.751+0000 7fcfa6d9b700 1 --2- 172.21.15.23:0/3068754133 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcf78056d10 0x7fcf780591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:03.770 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.752+0000 7fcfa5d99700 1 -- 172.21.15.23:0/3068754133 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcfa00a61e0 con 0x7fcfa013aeb0 2020-11-20T11:42:03.770 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.752+0000 7fcfa6d9b700 1 --2- 172.21.15.23:0/3068754133 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcf78056d10 0x7fcf780591d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fcfa009f260 tx=0x7fcf98009220).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:03.771 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.752+0000 7fcfa5d99700 1 --2- 172.21.15.23:0/3068754133 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfa013aeb0 0x7fcfa013d350 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fcf90009a10 tx=0x7fcf90016000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:03.771 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.752+0000 7fcf8e7fc700 1 -- 172.21.15.23:0/3068754133 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf90011b20 con 0x7fcfa013aeb0 2020-11-20T11:42:03.771 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.752+0000 7fcf8e7fc700 1 -- 172.21.15.23:0/3068754133 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fcf90011c80 con 0x7fcfa013aeb0 2020-11-20T11:42:03.771 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.752+0000 7fcf8e7fc700 1 -- 172.21.15.23:0/3068754133 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfa013aeb0 msgr2=0x7fcfa013d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:03.771 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.752+0000 7fcf8e7fc700 1 --2- 172.21.15.23:0/3068754133 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfa013aeb0 0x7fcfa013d350 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fcf90009a10 tx=0x7fcf90016000).stop 2020-11-20T11:42:03.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.752+0000 7fcf8e7fc700 1 -- 172.21.15.23:0/3068754133 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfa013aeb0 msgr2=0x7fcfa013d350 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:42:03.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.752+0000 7fcf8e7fc700 1 --2- 172.21.15.23:0/3068754133 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfa013aeb0 0x7fcfa013d350 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:03.774 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.763+0000 7fcfa87fe700 1 --2- 172.21.15.23:0/3068754133 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfa00601b0 0x7fcfa014ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:03.774 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.763+0000 7fcfa87fe700 1 -- 172.21.15.23:0/3068754133 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa0044710 con 0x7fcfa00601b0 2020-11-20T11:42:03.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.764+0000 7fcfa5d99700 1 --2- 172.21.15.23:0/3068754133 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfa00601b0 0x7fcfa014ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:03.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.764+0000 7fcfa5d99700 1 -- 172.21.15.23:0/3068754133 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fcfa005c280 con 0x7fcfa00601b0 2020-11-20T11:42:03.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.764+0000 7fcfa5d99700 1 --2- 172.21.15.23:0/3068754133 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfa00601b0 0x7fcfa014ddc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fcfa005c960 tx=0x7fcf9001e000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:03.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.764+0000 7fcf8e7fc700 1 -- 172.21.15.23:0/3068754133 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf90026920 con 0x7fcfa00601b0 2020-11-20T11:42:03.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.765+0000 7fcf8e7fc700 1 -- 172.21.15.23:0/3068754133 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4432 (secure 0 0 0) 0x7fcf90011810 con 0x7fcfa00601b0 2020-11-20T11:42:03.776 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.765+0000 7fcf8e7fc700 1 -- 172.21.15.23:0/3068754133 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfa00601b0 msgr2=0x7fcfa014ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:03.776 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.765+0000 7fcf8e7fc700 1 --2- 172.21.15.23:0/3068754133 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfa00601b0 0x7fcfa014ddc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fcfa005c960 tx=0x7fcf9001e000).stop 2020-11-20T11:42:03.776 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.765+0000 7fcf8e7fc700 1 -- 172.21.15.23:0/3068754133 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfa00601b0 msgr2=0x7fcfa014ddc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:42:03.776 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.765+0000 7fcf8e7fc700 1 --2- 172.21.15.23:0/3068754133 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcfa00601b0 0x7fcfa014ddc0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:03.776 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.765+0000 7fcfa87fe700 1 -- 172.21.15.23:0/3068754133 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcf78056d10 msgr2=0x7fcf780591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:03.777 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.765+0000 7fcfa87fe700 1 --2- 172.21.15.23:0/3068754133 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcf78056d10 0x7fcf780591d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fcfa009f260 tx=0x7fcf98009220).stop 2020-11-20T11:42:03.777 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.765+0000 7fcfa87fe700 1 -- 172.21.15.23:0/3068754133 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfa00aaab0 msgr2=0x7fcfa00a7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:03.777 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.765+0000 7fcfa87fe700 1 --2- 172.21.15.23:0/3068754133 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfa00aaab0 0x7fcfa00a7370 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7fcf9c009790 tx=0x7fcf9c006ea0).stop 2020-11-20T11:42:03.777 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.765+0000 7fcfa6d9b700 1 -- 172.21.15.23:0/3068754133 reap_dead start 2020-11-20T11:42:03.777 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.765+0000 7fcfa6d9b700 1 -- 172.21.15.23:0/3068754133 reap_dead start 2020-11-20T11:42:03.778 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.765+0000 7fcfa87fe700 1 -- 172.21.15.23:0/3068754133 shutdown_connections 2020-11-20T11:42:03.778 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.765+0000 7fcfa87fe700 1 -- 172.21.15.23:0/3068754133 >> 172.21.15.23:0/3068754133 conn(0x7fcfa00678e0 msgr2=0x7fcfa00acf80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:03.778 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.765+0000 7fcfa87fe700 1 -- 172.21.15.23:0/3068754133 shutdown_connections 2020-11-20T11:42:03.778 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:03.765+0000 7fcfa87fe700 1 -- 172.21.15.23:0/3068754133 wait complete. 2020-11-20T11:42:03.779 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:42:03.779 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:42:03.779 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:42:03.779 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:42:03.779 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 90, 2020-11-20T11:42:03.780 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:42:03.780 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:42:03.780 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:42:03.780 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:03.780 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 1, 2020-11-20T11:42:03.781 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:42:03.781 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:42:03.781 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:42:03.781 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:42:03.781 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:42:03.782 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:42:03.782 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:42:03.782 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:42:03.782 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:42:03.782 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:42:03.783 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:42:03.783 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:03.783 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:42:03.783 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:42:03.783 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:42:03.784 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:42:03.784 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:42:03.784 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:42:03.784 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:42:03.785 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:42:03.785 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:42:03.785 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:42:03.785 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:03.785 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:03.786 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:42:03.786 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:42:03.786 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:42:03.786 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:42:03.786 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:42:03.787 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:42:03.787 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:42:03.787 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:42:03.787 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:42:03.787 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:42:03.788 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:42:03.788 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:42:03.788 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:42:03.788 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:42:03.788 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:42:03.789 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:42:03.789 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:42:03.789 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:42:03.789 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:42:03.790 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:42:03.790 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:42:03.790 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:42:03.790 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:42:03.790 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:03.791 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:42:03.791 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:03.791 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:42:03.791 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:03.791 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:42:03.792 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:42:03.792 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:42:03.792 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:42:03.792 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:03.792 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:42:03.793 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:42:03.793 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:03.793 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:03.793 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:03.793 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:42:03.794 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:42:03.794 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:03.794 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:03.794 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:03.794 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:03.795 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:42:03.795 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:42:03.795 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:42:03.795 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:42:03.795 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:42:03.795 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:03.796 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:03.796 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:42:03.796 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:42:03.796 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:42:03.796 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:42:03.797 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:03.797 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:42:03.797 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:42:03.797 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:03.797 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:03.798 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:03.798 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:42:03.798 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:42:03.798 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:03.798 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:03.799 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:03.799 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:03.799 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:42:03.799 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:42:03.799 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:42:03.800 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:42:03.800 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:42:03.800 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:03.800 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:03.800 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:42:03.801 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:42:03.801 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:42:03.801 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:42:03.801 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:03.801 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:42:03.802 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:42:03.802 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:03.802 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:03.802 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:03.802 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:42:03.803 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:42:03.803 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:03.803 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:03.803 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:03.803 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:03.804 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:42:03.804 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:42:03.804 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:42:03.804 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:42:03.804 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:42:03.804 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:03.805 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:03.805 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:03.805 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:42:03.805 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:42:03.805 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:03.806 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:03.806 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:03.806 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:42:03.806 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:03.806 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:03.807 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:42:03.807 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:03.807 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:03.807 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:03.807 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:42:03.808 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:03.808 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:03.808 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:42:03.808 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:03.808 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:03.809 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:03.809 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:42:03.809 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:03.809 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:03.809 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:03.810 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:42:03.810 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:42:03.811 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2020-11-20T11:42:23.811 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2020-11-20T11:42:23.812 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2020-11-20T11:42:23.812 INFO:tasks.ceph.mon.b:Restarting daemon 2020-11-20T11:42:23.813 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2020-11-20T11:42:23.816 INFO:tasks.ceph.mon.b:Started 2020-11-20T11:42:23.816 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:42:23.816 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:42:24.610 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.606+0000 7fb393fff700 1 Processor -- start 2020-11-20T11:42:24.613 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.610+0000 7fb393fff700 1 -- start start 2020-11-20T11:42:24.614 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.611+0000 7fb393fff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3940a2080 0x7fb3940aef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:24.615 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.611+0000 7fb393fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb394067a10 0x7fb394067e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:24.615 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.611+0000 7fb393fff700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3940aaab0 con 0x7fb3940a2460 2020-11-20T11:42:24.615 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.611+0000 7fb393fff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb39413aa50 con 0x7fb394067a10 2020-11-20T11:42:24.615 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.611+0000 7fb392ffd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3940a2080 0x7fb3940aef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:24.616 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.611+0000 7fb3937fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb394067a10 0x7fb394067e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:24.616 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.611+0000 7fb3937fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb394067a10 0x7fb394067e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44824/0 (socket says 172.21.15.23:44824) 2020-11-20T11:42:24.616 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.611+0000 7fb3937fe700 1 -- 172.21.15.23:0/2903552697 learned_addr learned my addr 172.21.15.23:0/2903552697 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:42:24.616 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.611+0000 7fb392ffd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3940a2080 0x7fb3940aef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:55062/0 (socket says 172.21.15.23:55062) 2020-11-20T11:42:24.616 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.611+0000 7fb393fff700 1 -- 172.21.15.23:0/2903552697 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3940aad50 con 0x7fb3940a2080 2020-11-20T11:42:24.617 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.611+0000 7fb3937fe700 1 -- 172.21.15.23:0/2903552697 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3940a2080 msgr2=0x7fb3940aef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:42:24.617 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.611+0000 7fb3937fe700 1 --2- 172.21.15.23:0/2903552697 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3940a2080 0x7fb3940aef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:24.617 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.611+0000 7fb3937fe700 1 -- 172.21.15.23:0/2903552697 >> v1:172.21.15.155:6789/0 conn(0x7fb3940a2460 legacy=0x7fb394137df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:42:24.617 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.611+0000 7fb3937fe700 1 -- 172.21.15.23:0/2903552697 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb39413b920 con 0x7fb394067a10 2020-11-20T11:42:24.618 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.612+0000 7fb3937fe700 1 --2- 172.21.15.23:0/2903552697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb394067a10 0x7fb394067e60 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fb38c009d90 tx=0x7fb38c0092d0).ready entity=mon.0 client_cookie=236ba1d8d0ef39b6 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:24.620 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.612+0000 7fb391ffb700 1 -- 172.21.15.23:0/2903552697 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38c00bbb0 con 0x7fb394067a10 2020-11-20T11:42:24.620 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.612+0000 7fb391ffb700 1 -- 172.21.15.23:0/2903552697 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb38c00bd10 con 0x7fb394067a10 2020-11-20T11:42:24.620 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.612+0000 7fb391ffb700 1 -- 172.21.15.23:0/2903552697 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38c011610 con 0x7fb394067a10 2020-11-20T11:42:24.621 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.613+0000 7fb393fff700 1 -- 172.21.15.23:0/2903552697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb394067a10 msgr2=0x7fb394067e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:24.621 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.613+0000 7fb393fff700 1 --2- 172.21.15.23:0/2903552697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb394067a10 0x7fb394067e60 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fb38c009d90 tx=0x7fb38c0092d0).stop 2020-11-20T11:42:24.621 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.613+0000 7fb393fff700 1 -- 172.21.15.23:0/2903552697 shutdown_connections 2020-11-20T11:42:24.621 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.613+0000 7fb393fff700 1 --2- 172.21.15.23:0/2903552697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb394067a10 0x7fb394067e60 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:24.621 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.613+0000 7fb393fff700 1 --2- 172.21.15.23:0/2903552697 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3940a2080 0x7fb3940aef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:24.622 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.613+0000 7fb393fff700 1 -- 172.21.15.23:0/2903552697 >> 172.21.15.23:0/2903552697 conn(0x7fb394063980 msgr2=0x7fb394063d90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:24.622 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.613+0000 7fb393fff700 1 -- 172.21.15.23:0/2903552697 shutdown_connections 2020-11-20T11:42:24.622 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.613+0000 7fb393fff700 1 -- 172.21.15.23:0/2903552697 wait complete. 2020-11-20T11:42:24.622 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.615+0000 7fb393fff700 1 Processor -- start 2020-11-20T11:42:24.623 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.615+0000 7fb393fff700 1 -- start start 2020-11-20T11:42:24.623 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.615+0000 7fb393fff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb394067a10 0x7fb394141870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:24.623 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.615+0000 7fb393fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3940a2080 0x7fb394149b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:24.623 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.615+0000 7fb393fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3940a2460 0x7fb3941372a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:24.623 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.615+0000 7fb393fff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3940a5100 con 0x7fb3940a2080 2020-11-20T11:42:24.623 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.615+0000 7fb393fff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3940a53a0 con 0x7fb3940a2460 2020-11-20T11:42:24.624 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.615+0000 7fb393fff700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3940a5680 con 0x7fb394067a10 2020-11-20T11:42:24.624 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.616+0000 7fb392ffd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb394067a10 0x7fb394141870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:24.624 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.616+0000 7fb392ffd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb394067a10 0x7fb394141870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:55066/0 (socket says 172.21.15.23:55066) 2020-11-20T11:42:24.624 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.616+0000 7fb392ffd700 1 -- 172.21.15.23:0/2860389347 learned_addr learned my addr 172.21.15.23:0/2860389347 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:42:24.625 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.616+0000 7fb3927fc700 1 --2- 172.21.15.23:0/2860389347 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3940a2080 0x7fb394149b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:24.625 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.616+0000 7fb392ffd700 1 -- 172.21.15.23:0/2860389347 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3940a2460 msgr2=0x7fb3941372a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:24.625 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.616+0000 7fb392ffd700 1 --2- 172.21.15.23:0/2860389347 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3940a2460 0x7fb3941372a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:24.625 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.616+0000 7fb392ffd700 1 -- 172.21.15.23:0/2860389347 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3940a2080 msgr2=0x7fb394149b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:24.625 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.616+0000 7fb392ffd700 1 --2- 172.21.15.23:0/2860389347 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3940a2080 0x7fb394149b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:24.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.616+0000 7fb392ffd700 1 -- 172.21.15.23:0/2860389347 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb39413d3c0 con 0x7fb394067a10 2020-11-20T11:42:24.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.616+0000 7fb3927fc700 1 --2- 172.21.15.23:0/2860389347 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3940a2080 0x7fb394149b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:42:24.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.616+0000 7fb392ffd700 1 --2- 172.21.15.23:0/2860389347 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb394067a10 0x7fb394141870 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fb38400eba0 tx=0x7fb384006dd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:24.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.616+0000 7fb37affd700 1 -- 172.21.15.23:0/2860389347 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38400c8c0 con 0x7fb394067a10 2020-11-20T11:42:24.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.617+0000 7fb37affd700 1 -- 172.21.15.23:0/2860389347 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb38400ca20 con 0x7fb394067a10 2020-11-20T11:42:24.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.617+0000 7fb37affd700 1 -- 172.21.15.23:0/2860389347 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38400cda0 con 0x7fb394067a10 2020-11-20T11:42:24.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.617+0000 7fb393fff700 1 -- 172.21.15.23:0/2860389347 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb39414a000 con 0x7fb394067a10 2020-11-20T11:42:24.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.617+0000 7fb393fff700 1 -- 172.21.15.23:0/2860389347 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3940a2f60 con 0x7fb394067a10 2020-11-20T11:42:24.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.618+0000 7fb393fff700 1 -- 172.21.15.23:0/2860389347 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3940a5840 con 0x7fb394067a10 2020-11-20T11:42:24.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.623+0000 7fb37affd700 1 -- 172.21.15.23:0/2860389347 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb384012600 con 0x7fb394067a10 2020-11-20T11:42:24.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.623+0000 7fb37affd700 1 --2- 172.21.15.23:0/2860389347 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb370056d60 0x7fb370059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:24.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.623+0000 7fb37affd700 1 -- 172.21.15.23:0/2860389347 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fb384070fb0 con 0x7fb394067a10 2020-11-20T11:42:24.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.623+0000 7fb3927fc700 1 --2- 172.21.15.23:0/2860389347 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb370056d60 0x7fb370059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:24.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.624+0000 7fb3927fc700 1 --2- 172.21.15.23:0/2860389347 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb370056d60 0x7fb370059220 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fb39413d830 tx=0x7fb388008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:24.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.628+0000 7fb37affd700 1 -- 172.21.15.23:0/2860389347 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb3840462a0 con 0x7fb394067a10 2020-11-20T11:42:24.957 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.954+0000 7fb393fff700 1 -- 172.21.15.23:0/2860389347 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb394057dc0 con 0x7fb394067a10 2020-11-20T11:42:24.958 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.955+0000 7fb37affd700 1 -- 172.21.15.23:0/2860389347 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1472 (secure 0 0 0) 0x7fb3840498c0 con 0x7fb394067a10 2020-11-20T11:42:24.960 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":90,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:42:24.968 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.964+0000 7fb378ff9700 1 -- 172.21.15.23:0/2860389347 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb370056d60 msgr2=0x7fb370059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:24.969 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.964+0000 7fb378ff9700 1 --2- 172.21.15.23:0/2860389347 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb370056d60 0x7fb370059220 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fb39413d830 tx=0x7fb388008040).stop 2020-11-20T11:42:24.969 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.964+0000 7fb378ff9700 1 -- 172.21.15.23:0/2860389347 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb394067a10 msgr2=0x7fb394141870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:24.970 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.964+0000 7fb378ff9700 1 --2- 172.21.15.23:0/2860389347 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb394067a10 0x7fb394141870 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fb38400eba0 tx=0x7fb384006dd0).stop 2020-11-20T11:42:24.970 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.964+0000 7fb378ff9700 1 -- 172.21.15.23:0/2860389347 shutdown_connections 2020-11-20T11:42:24.970 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.964+0000 7fb378ff9700 1 --2- 172.21.15.23:0/2860389347 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3940a2460 0x7fb3941372a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:24.971 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.964+0000 7fb378ff9700 1 --2- 172.21.15.23:0/2860389347 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb370056d60 0x7fb370059220 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:24.971 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.964+0000 7fb378ff9700 1 --2- 172.21.15.23:0/2860389347 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb394067a10 0x7fb394141870 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:24.972 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.964+0000 7fb378ff9700 1 --2- 172.21.15.23:0/2860389347 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3940a2080 0x7fb394149b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:24.972 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.964+0000 7fb378ff9700 1 -- 172.21.15.23:0/2860389347 >> 172.21.15.23:0/2860389347 conn(0x7fb394063980 msgr2=0x7fb394068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:24.972 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.964+0000 7fb378ff9700 1 -- 172.21.15.23:0/2860389347 shutdown_connections 2020-11-20T11:42:24.973 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:24.964+0000 7fb378ff9700 1 -- 172.21.15.23:0/2860389347 wait complete. 2020-11-20T11:42:27.988 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:42:28.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.436+0000 7f2f0e324700 1 Processor -- start 2020-11-20T11:42:28.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.436+0000 7f2f0e324700 1 -- start start 2020-11-20T11:42:28.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.437+0000 7f2f0e324700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2f0813aeb0 0x7f2f080a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:28.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.437+0000 7f2f0e324700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f081388b0 con 0x7f2f080aaab0 2020-11-20T11:42:28.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.437+0000 7f2f0e324700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f0813e600 con 0x7f2f080aae90 2020-11-20T11:42:28.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.437+0000 7f2f0e324700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0813e8a0 con 0x7f2f0813aeb0 2020-11-20T11:42:28.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.437+0000 7f2f07fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f2f080aaab0 0x7f2f0813ed40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:59086/0 (socket says 172.21.15.23:59086) 2020-11-20T11:42:28.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.437+0000 7f2f07fff700 1 -- 172.21.15.23:0/1552900183 learned_addr learned my addr 172.21.15.23:0/1552900183 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:42:28.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.438+0000 7f2f06ffd700 1 -- 172.21.15.23:0/1552900183 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 960650443 0 0) 0x7f2f081388b0 con 0x7f2f080aaab0 2020-11-20T11:42:28.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.438+0000 7f2f06ffd700 1 -- 172.21.15.23:0/1552900183 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ef4003280 con 0x7f2f080aaab0 2020-11-20T11:42:28.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.438+0000 7f2f06ffd700 1 -- 172.21.15.23:0/1552900183 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 365698752 0 0) 0x7f2ef4003280 con 0x7f2f080aaab0 2020-11-20T11:42:28.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.438+0000 7f2f06ffd700 1 -- 172.21.15.23:0/1552900183 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2f0813aeb0 msgr2=0x7f2f080a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:42:28.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.438+0000 7f2f06ffd700 1 --2- 172.21.15.23:0/1552900183 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2f0813aeb0 0x7f2f080a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:28.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.438+0000 7f2f06ffd700 1 -- 172.21.15.23:0/1552900183 >> v1:172.21.15.155:6789/0 conn(0x7f2f080aae90 legacy=0x7f2f081381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:42:28.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.438+0000 7f2f06ffd700 1 -- 172.21.15.23:0/1552900183 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f080ab6d0 con 0x7f2f080aaab0 2020-11-20T11:42:28.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.438+0000 7f2f06ffd700 1 -- 172.21.15.23:0/1552900183 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f2ef8001520 con 0x7f2f080aaab0 2020-11-20T11:42:28.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.438+0000 7f2f06ffd700 1 -- 172.21.15.23:0/1552900183 >> v1:172.21.15.23:6789/0 conn(0x7f2f080aaab0 legacy=0x7f2f0813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:28.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.438+0000 7f2f06ffd700 1 --2- 172.21.15.23:0/1552900183 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2ef4003820 0x7f2ef4003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:28.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.438+0000 7f2f06ffd700 1 --2- 172.21.15.23:0/1552900183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2ef4004520 0x7f2ef40049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:28.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.438+0000 7f2f06ffd700 1 --2- 172.21.15.23:0/1552900183 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ef4004e90 0x7f2ef400b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:28.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.438+0000 7f2f06ffd700 1 -- 172.21.15.23:0/1552900183 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0813e8a0 con 0x7f2ef4004e90 2020-11-20T11:42:28.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.438+0000 7f2f06ffd700 1 -- 172.21.15.23:0/1552900183 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef4005310 con 0x7f2ef4004520 2020-11-20T11:42:28.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.438+0000 7f2f06ffd700 1 -- 172.21.15.23:0/1552900183 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef400bc60 con 0x7f2ef4003820 2020-11-20T11:42:28.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.439+0000 7f2f07fff700 1 --2- 172.21.15.23:0/1552900183 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2ef4003820 0x7f2ef4003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:28.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.439+0000 7f2f07fff700 1 -- 172.21.15.23:0/1552900183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2ef4004520 msgr2=0x7f2ef40049a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:42:28.448 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.439+0000 7f2f07fff700 1 --2- 172.21.15.23:0/1552900183 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2ef4004520 0x7f2ef40049a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:28.448 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.439+0000 7f2f07fff700 1 -- 172.21.15.23:0/1552900183 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ef4004e90 msgr2=0x7f2ef400b770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:42:28.448 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.439+0000 7f2f07fff700 1 --2- 172.21.15.23:0/1552900183 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ef4004e90 0x7f2ef400b770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:28.448 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.439+0000 7f2f07fff700 1 -- 172.21.15.23:0/1552900183 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef4003280 con 0x7f2ef4003820 2020-11-20T11:42:28.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.439+0000 7f2f07fff700 1 --2- 172.21.15.23:0/1552900183 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2ef4003820 0x7f2ef4003c20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f2ef800c440 tx=0x7f2ef8007790).ready entity=mon.2 client_cookie=af0fc9d72edc1de8 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:28.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.439+0000 7f2f06ffd700 1 -- 172.21.15.23:0/1552900183 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef8007e10 con 0x7f2ef4003820 2020-11-20T11:42:28.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.440+0000 7f2f06ffd700 1 -- 172.21.15.23:0/1552900183 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef80072e0 con 0x7f2ef4003820 2020-11-20T11:42:28.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.440+0000 7f2f06ffd700 1 -- 172.21.15.23:0/1552900183 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef8005840 con 0x7f2ef4003820 2020-11-20T11:42:28.451 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.442+0000 7f2f0e324700 1 -- 172.21.15.23:0/1552900183 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2ef4003820 msgr2=0x7f2ef4003c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:28.451 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.442+0000 7f2f0e324700 1 --2- 172.21.15.23:0/1552900183 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2ef4003820 0x7f2ef4003c20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f2ef800c440 tx=0x7f2ef8007790).stop 2020-11-20T11:42:28.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.442+0000 7f2f0c8c1700 1 -- 172.21.15.23:0/1552900183 reap_dead start 2020-11-20T11:42:28.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.442+0000 7f2f0e324700 1 -- 172.21.15.23:0/1552900183 shutdown_connections 2020-11-20T11:42:28.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.442+0000 7f2f0e324700 1 -- 172.21.15.23:0/1552900183 >> 172.21.15.23:0/1552900183 conn(0x7f2f080678e0 msgr2=0x7f2f08067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:28.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.443+0000 7f2f0e324700 1 -- 172.21.15.23:0/1552900183 shutdown_connections 2020-11-20T11:42:28.453 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.443+0000 7f2f0e324700 1 -- 172.21.15.23:0/1552900183 wait complete. 2020-11-20T11:42:28.453 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.444+0000 7f2f0e324700 1 Processor -- start 2020-11-20T11:42:28.453 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.444+0000 7f2f0e324700 1 -- start start 2020-11-20T11:42:28.453 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.445+0000 7f2f0e324700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2ef4003820 0x7f2f080a7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:28.454 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.445+0000 7f2f0e324700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f0813d770 0x7f2f080a6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:28.454 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.445+0000 7f2f0e324700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f0814d8c0 0x7f2f0814dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:28.454 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.445+0000 7f2f0e324700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0813c080 con 0x7f2f0813d770 2020-11-20T11:42:28.454 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.445+0000 7f2f0e324700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0813d070 con 0x7f2f0814d8c0 2020-11-20T11:42:28.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.445+0000 7f2f0e324700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f080a7a60 con 0x7f2ef4003820 2020-11-20T11:42:28.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.445+0000 7f2f077fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f0813d770 0x7f2f080a6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:28.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.445+0000 7f2f077fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f0813d770 0x7f2f080a6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44844/0 (socket says 172.21.15.23:44844) 2020-11-20T11:42:28.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.445+0000 7f2f077fe700 1 -- 172.21.15.23:0/3294168088 learned_addr learned my addr 172.21.15.23:0/3294168088 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:42:28.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.445+0000 7f2f07fff700 1 --2- 172.21.15.23:0/3294168088 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2ef4003820 0x7f2f080a7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:28.456 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.446+0000 7f2f0c8c1700 1 --2- 172.21.15.23:0/3294168088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f0814d8c0 0x7f2f0814dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:28.456 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.446+0000 7f2f0c8c1700 1 -- 172.21.15.23:0/3294168088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f0814d8c0 msgr2=0x7f2f0814dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2020-11-20T11:42:28.456 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.446+0000 7f2f0c8c1700 1 -- 172.21.15.23:0/3294168088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f0814d8c0 msgr2=0x7f2f0814dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:42:28.456 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.446+0000 7f2f0c8c1700 1 --2- 172.21.15.23:0/3294168088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f0814d8c0 0x7f2f0814dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:42:28.456 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.446+0000 7f2f0c8c1700 1 --2- 172.21.15.23:0/3294168088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f0814d8c0 0x7f2f0814dd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:42:28.457 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.446+0000 7f2f077fe700 1 -- 172.21.15.23:0/3294168088 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2ef4003820 msgr2=0x7f2f080a7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:28.457 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.446+0000 7f2f077fe700 1 --2- 172.21.15.23:0/3294168088 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2ef4003820 0x7f2f080a7570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:28.457 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.446+0000 7f2f077fe700 1 -- 172.21.15.23:0/3294168088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f0814d8c0 msgr2=0x7f2f0814dd20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:42:28.457 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.446+0000 7f2f077fe700 1 --2- 172.21.15.23:0/3294168088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f0814d8c0 0x7f2f0814dd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:28.457 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.446+0000 7f2f077fe700 1 -- 172.21.15.23:0/3294168088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0813d2d0 con 0x7f2f0813d770 2020-11-20T11:42:28.458 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.446+0000 7f2f077fe700 1 --2- 172.21.15.23:0/3294168088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f0813d770 0x7f2f080a6d40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f2efc009690 tx=0x7f2efc006db0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:28.458 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.447+0000 7f2eeffff700 1 -- 172.21.15.23:0/3294168088 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2efc00d4a0 con 0x7f2f0813d770 2020-11-20T11:42:28.458 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.447+0000 7f2eeffff700 1 -- 172.21.15.23:0/3294168088 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2efc00d620 con 0x7f2f0813d770 2020-11-20T11:42:28.458 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.447+0000 7f2f0e324700 1 -- 172.21.15.23:0/3294168088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f0813df80 con 0x7f2f0813d770 2020-11-20T11:42:28.458 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.447+0000 7f2f0e324700 1 -- 172.21.15.23:0/3294168088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f080a7230 con 0x7f2f0813d770 2020-11-20T11:42:28.459 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.447+0000 7f2eeffff700 1 -- 172.21.15.23:0/3294168088 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2efc00d9e0 con 0x7f2f0813d770 2020-11-20T11:42:28.459 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.449+0000 7f2eeffff700 1 -- 172.21.15.23:0/3294168088 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2efc01a060 con 0x7f2f0813d770 2020-11-20T11:42:28.459 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.449+0000 7f2eeffff700 1 --2- 172.21.15.23:0/3294168088 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2ee0056d60 0x7f2ee0059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:28.460 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.449+0000 7f2eeffff700 1 -- 172.21.15.23:0/3294168088 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f2efc06f9a0 con 0x7f2f0813d770 2020-11-20T11:42:28.460 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.449+0000 7f2f07fff700 1 --2- 172.21.15.23:0/3294168088 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2ee0056d60 0x7f2ee0059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:28.460 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.450+0000 7f2f07fff700 1 --2- 172.21.15.23:0/3294168088 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2ee0056d60 0x7f2ee0059220 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f2f08136120 tx=0x7f2ef8015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:28.460 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.454+0000 7f2f0e324700 1 -- 172.21.15.23:0/3294168088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ee4004fd0 con 0x7f2f0813d770 2020-11-20T11:42:28.464 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.461+0000 7f2eeffff700 1 -- 172.21.15.23:0/3294168088 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2efc0486c0 con 0x7f2f0813d770 2020-11-20T11:42:28.791 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.787+0000 7f2f0e324700 1 -- 172.21.15.23:0/3294168088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2ee4001670 con 0x7f2f0813d770 2020-11-20T11:42:28.792 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.789+0000 7f2eeffff700 1 -- 172.21.15.23:0/3294168088 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1477 (secure 0 0 0) 0x7f2efc04c020 con 0x7f2f0813d770 2020-11-20T11:42:28.792 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:42:28.796 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.793+0000 7f2f0e324700 1 -- 172.21.15.23:0/3294168088 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2ee0056d60 msgr2=0x7f2ee0059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:28.797 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.793+0000 7f2f0e324700 1 --2- 172.21.15.23:0/3294168088 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2ee0056d60 0x7f2ee0059220 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f2f08136120 tx=0x7f2ef8015000).stop 2020-11-20T11:42:28.797 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.793+0000 7f2f0e324700 1 -- 172.21.15.23:0/3294168088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f0813d770 msgr2=0x7f2f080a6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:28.797 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.793+0000 7f2f0e324700 1 --2- 172.21.15.23:0/3294168088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f0813d770 0x7f2f080a6d40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f2efc009690 tx=0x7f2efc006db0).stop 2020-11-20T11:42:28.797 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.793+0000 7f2f0e324700 1 -- 172.21.15.23:0/3294168088 shutdown_connections 2020-11-20T11:42:28.798 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.793+0000 7f2f0e324700 1 --2- 172.21.15.23:0/3294168088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2f0814d8c0 0x7f2f0814dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:28.798 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.793+0000 7f2f0e324700 1 --2- 172.21.15.23:0/3294168088 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2ee0056d60 0x7f2ee0059220 secure :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0x7f2f08136120 tx=0x7f2ef8015000).stop 2020-11-20T11:42:28.798 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.793+0000 7f2f0e324700 1 --2- 172.21.15.23:0/3294168088 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2ef4003820 0x7f2f080a7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:28.798 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.793+0000 7f2f0e324700 1 --2- 172.21.15.23:0/3294168088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f0813d770 0x7f2f080a6d40 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:28.799 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.793+0000 7f2f0e324700 1 -- 172.21.15.23:0/3294168088 >> 172.21.15.23:0/3294168088 conn(0x7f2f080678e0 msgr2=0x7f2f08140490 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:28.799 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.794+0000 7f2f0e324700 1 -- 172.21.15.23:0/3294168088 shutdown_connections 2020-11-20T11:42:28.800 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:28.794+0000 7f2f0e324700 1 -- 172.21.15.23:0/3294168088 wait complete. 2020-11-20T11:42:28.813 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:42:28.814 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:42:29.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.006+0000 7f83342e6700 1 Processor -- start 2020-11-20T11:42:29.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.006+0000 7f83342e6700 1 -- start start 2020-11-20T11:42:29.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.007+0000 7f83342e6700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f832c060ca0 0x7f832c13eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:29.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.007+0000 7f83342e6700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f832c1386f0 con 0x7f832c0608c0 2020-11-20T11:42:29.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.007+0000 7f83342e6700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f832c13e1b0 con 0x7f832c0a1ac0 2020-11-20T11:42:29.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.007+0000 7f8332082700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f832c0608c0 0x7f832c137fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:59100/0 (socket says 172.21.15.23:59100) 2020-11-20T11:42:29.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.007+0000 7f8332082700 1 -- 172.21.15.23:0/376826642 learned_addr learned my addr 172.21.15.23:0/376826642 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:42:29.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.007+0000 7f83342e6700 1 -- 172.21.15.23:0/376826642 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f832c13e450 con 0x7f832c060ca0 2020-11-20T11:42:29.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.007+0000 7f8331881700 1 --2- 172.21.15.23:0/376826642 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f832c060ca0 0x7f832c13eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:29.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.007+0000 7f8331080700 1 -- 172.21.15.23:0/376826642 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3932677248 0 0) 0x7f832c1386f0 con 0x7f832c0608c0 2020-11-20T11:42:29.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.007+0000 7f8331080700 1 -- 172.21.15.23:0/376826642 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8318003280 con 0x7f832c0608c0 2020-11-20T11:42:29.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.007+0000 7f8331881700 1 -- 172.21.15.23:0/376826642 >> v1:172.21.15.155:6789/0 conn(0x7f832c0a1ac0 legacy=0x7f832c0aabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:29.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.007+0000 7f8331080700 1 -- 172.21.15.23:0/376826642 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4054511502 0 0) 0x7f832c13e1b0 con 0x7f832c0a1ac0 2020-11-20T11:42:29.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.007+0000 7f8331881700 1 -- 172.21.15.23:0/376826642 >> v1:172.21.15.23:6789/0 conn(0x7f832c0608c0 legacy=0x7f832c137fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:29.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.007+0000 7f8331881700 1 -- 172.21.15.23:0/376826642 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f832c13e6f0 con 0x7f832c060ca0 2020-11-20T11:42:29.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.008+0000 7f8331881700 1 --2- 172.21.15.23:0/376826642 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f832c060ca0 0x7f832c13eb90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f832000ba00 tx=0x7f8320006e30).ready entity=mon.2 client_cookie=9b71e62937b62f7d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:29.014 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.008+0000 7f8331080700 1 -- 172.21.15.23:0/376826642 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83200097e0 con 0x7f832c060ca0 2020-11-20T11:42:29.014 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.008+0000 7f8331080700 1 -- 172.21.15.23:0/376826642 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8320000e90 con 0x7f832c060ca0 2020-11-20T11:42:29.014 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.008+0000 7f8331080700 1 -- 172.21.15.23:0/376826642 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8320009b50 con 0x7f832c060ca0 2020-11-20T11:42:29.015 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.009+0000 7f83342e6700 1 -- 172.21.15.23:0/376826642 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f832c060ca0 msgr2=0x7f832c13eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:29.015 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.009+0000 7f83342e6700 1 --2- 172.21.15.23:0/376826642 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f832c060ca0 0x7f832c13eb90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f832000ba00 tx=0x7f8320006e30).stop 2020-11-20T11:42:29.020 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.017+0000 7f83342e6700 1 -- 172.21.15.23:0/376826642 shutdown_connections 2020-11-20T11:42:29.020 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.017+0000 7f83342e6700 1 --2- 172.21.15.23:0/376826642 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f832c060ca0 0x7f832c13eb90 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:29.020 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.017+0000 7f83342e6700 1 -- 172.21.15.23:0/376826642 >> 172.21.15.23:0/376826642 conn(0x7f832c066750 msgr2=0x7f832c0601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:29.021 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.017+0000 7f83342e6700 1 -- 172.21.15.23:0/376826642 shutdown_connections 2020-11-20T11:42:29.021 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.017+0000 7f83342e6700 1 -- 172.21.15.23:0/376826642 wait complete. 2020-11-20T11:42:29.021 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.018+0000 7f83342e6700 1 Processor -- start 2020-11-20T11:42:29.022 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.019+0000 7f83342e6700 1 -- start start 2020-11-20T11:42:29.023 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.019+0000 7f83342e6700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f832c0608c0 0x7f832c13d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:29.023 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.019+0000 7f83342e6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f832c060ca0 0x7f832c14a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:29.023 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.019+0000 7f83342e6700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f832c0a1ac0 0x7f832c14bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:29.023 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.020+0000 7f8331881700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f832c060ca0 0x7f832c14a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:29.023 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.020+0000 7f8331881700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f832c060ca0 0x7f832c14a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44856/0 (socket says 172.21.15.23:44856) 2020-11-20T11:42:29.024 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.020+0000 7f83342e6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832c14b5c0 con 0x7f832c060ca0 2020-11-20T11:42:29.024 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.020+0000 7f83342e6700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832c14b860 con 0x7f832c0a1ac0 2020-11-20T11:42:29.024 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.020+0000 7f83342e6700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f832c14bb00 con 0x7f832c0608c0 2020-11-20T11:42:29.024 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.020+0000 7f8331881700 1 -- 172.21.15.23:0/3187890337 learned_addr learned my addr 172.21.15.23:0/3187890337 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:42:29.025 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.020+0000 7f8332883700 1 --2- 172.21.15.23:0/3187890337 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f832c0a1ac0 0x7f832c14bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:29.025 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.020+0000 7f8331881700 1 -- 172.21.15.23:0/3187890337 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f832c0608c0 msgr2=0x7f832c13d240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:42:29.025 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.020+0000 7f8331881700 1 --2- 172.21.15.23:0/3187890337 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f832c0608c0 0x7f832c13d240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:29.026 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.020+0000 7f8331881700 1 -- 172.21.15.23:0/3187890337 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f832c0a1ac0 msgr2=0x7f832c14bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:29.026 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.020+0000 7f8331881700 1 --2- 172.21.15.23:0/3187890337 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f832c0a1ac0 0x7f832c14bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:29.026 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.020+0000 7f8331881700 1 -- 172.21.15.23:0/3187890337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f832c14b240 con 0x7f832c060ca0 2020-11-20T11:42:29.026 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.021+0000 7f8331881700 1 --2- 172.21.15.23:0/3187890337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f832c060ca0 0x7f832c14a590 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f832c1388a0 tx=0x7f8320015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:29.028 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.022+0000 7f832a7fc700 1 -- 172.21.15.23:0/3187890337 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8320011b70 con 0x7f832c060ca0 2020-11-20T11:42:29.028 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.022+0000 7f83342e6700 1 -- 172.21.15.23:0/3187890337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f832c14aa80 con 0x7f832c060ca0 2020-11-20T11:42:29.029 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.022+0000 7f83342e6700 1 -- 172.21.15.23:0/3187890337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f832c14c6d0 con 0x7f832c060ca0 2020-11-20T11:42:29.029 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.022+0000 7f832a7fc700 1 -- 172.21.15.23:0/3187890337 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8320011cd0 con 0x7f832c060ca0 2020-11-20T11:42:29.029 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.022+0000 7f832a7fc700 1 -- 172.21.15.23:0/3187890337 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832001d660 con 0x7f832c060ca0 2020-11-20T11:42:29.029 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.023+0000 7f83342e6700 1 --2- 172.21.15.23:0/3187890337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f832c13ace0 0x7f832c13b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:29.029 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.023+0000 7f83342e6700 1 -- 172.21.15.23:0/3187890337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832c14c9b0 con 0x7f832c13ace0 2020-11-20T11:42:29.030 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.024+0000 7f8332082700 1 --2- 172.21.15.23:0/3187890337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f832c13ace0 0x7f832c13b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:29.030 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.024+0000 7f832a7fc700 1 -- 172.21.15.23:0/3187890337 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f832000f070 con 0x7f832c060ca0 2020-11-20T11:42:29.030 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.024+0000 7f8332082700 1 -- 172.21.15.23:0/3187890337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f832c14acf0 con 0x7f832c13ace0 2020-11-20T11:42:29.030 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.025+0000 7f8332082700 1 --2- 172.21.15.23:0/3187890337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f832c13ace0 0x7f832c13b0c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f832400c960 tx=0x7f8324007da0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:29.031 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.026+0000 7f832a7fc700 1 --2- 172.21.15.23:0/3187890337 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8304056d60 0x7f8304059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:29.031 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.026+0000 7f832a7fc700 1 -- 172.21.15.23:0/3187890337 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f8320071e00 con 0x7f832c060ca0 2020-11-20T11:42:29.032 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.026+0000 7f8332883700 1 --2- 172.21.15.23:0/3187890337 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8304056d60 0x7f8304059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:29.032 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.026+0000 7f832a7fc700 1 -- 172.21.15.23:0/3187890337 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832400cea0 con 0x7f832c13ace0 2020-11-20T11:42:29.032 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.026+0000 7f832a7fc700 1 -- 172.21.15.23:0/3187890337 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f8324005bc0 con 0x7f832c13ace0 2020-11-20T11:42:29.032 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.026+0000 7f832a7fc700 1 -- 172.21.15.23:0/3187890337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f832c13ace0 msgr2=0x7f832c13b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:29.032 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.026+0000 7f832a7fc700 1 --2- 172.21.15.23:0/3187890337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f832c13ace0 0x7f832c13b0c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f832400c960 tx=0x7f8324007da0).stop 2020-11-20T11:42:29.033 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.026+0000 7f832a7fc700 1 -- 172.21.15.23:0/3187890337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f832c13ace0 msgr2=0x7f832c13b0c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:42:29.033 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.026+0000 7f832a7fc700 1 --2- 172.21.15.23:0/3187890337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f832c13ace0 0x7f832c13b0c0 secure :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0x7f832400c960 tx=0x7f8324007da0).stop 2020-11-20T11:42:29.033 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.027+0000 7f8332883700 1 --2- 172.21.15.23:0/3187890337 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8304056d60 0x7f8304059220 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f831c000f60 tx=0x7f831c007bb0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:29.039 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.036+0000 7f83342e6700 1 --2- 172.21.15.23:0/3187890337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f832c14d340 0x7f832c14d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:29.039 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.036+0000 7f83342e6700 1 -- 172.21.15.23:0/3187890337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832c14b700 con 0x7f832c14d340 2020-11-20T11:42:29.043 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.041+0000 7f8332082700 1 --2- 172.21.15.23:0/3187890337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f832c14d340 0x7f832c14d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:29.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.043+0000 7f8332082700 1 -- 172.21.15.23:0/3187890337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f832c14b3d0 con 0x7f832c14d340 2020-11-20T11:42:29.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.046+0000 7f8332082700 1 --2- 172.21.15.23:0/3187890337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f832c14d340 0x7f832c14d750 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f832400c960 tx=0x7f8324014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:29.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.053+0000 7f832a7fc700 1 -- 172.21.15.23:0/3187890337 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832401c040 con 0x7f832c14d340 2020-11-20T11:42:29.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.053+0000 7f832a7fc700 1 -- 172.21.15.23:0/3187890337 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4785 (secure 0 0 0) 0x7f832400ac20 con 0x7f832c14d340 2020-11-20T11:42:29.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.053+0000 7f832a7fc700 1 -- 172.21.15.23:0/3187890337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f832c14d340 msgr2=0x7f832c14d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:29.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.053+0000 7f832a7fc700 1 --2- 172.21.15.23:0/3187890337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f832c14d340 0x7f832c14d750 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f832400c960 tx=0x7f8324014000).stop 2020-11-20T11:42:29.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.053+0000 7f832a7fc700 1 -- 172.21.15.23:0/3187890337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f832c14d340 msgr2=0x7f832c14d750 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:42:29.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.053+0000 7f832a7fc700 1 --2- 172.21.15.23:0/3187890337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f832c14d340 0x7f832c14d750 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:29.059 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:42:29.059 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:42:29.059 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:42:29.059 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:42:29.060 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 92, 2020-11-20T11:42:29.060 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:42:29.060 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:42:29.060 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:42:29.060 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:42:29.061 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:29.061 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 3, 2020-11-20T11:42:29.061 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:42:29.061 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:42:29.061 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:42:29.062 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:42:29.062 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:42:29.062 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:42:29.062 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:42:29.063 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:42:29.063 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:42:29.063 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:42:29.063 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:42:29.063 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:29.064 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:42:29.064 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:42:29.064 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:42:29.064 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:42:29.064 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:42:29.065 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:42:29.065 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:42:29.065 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:42:29.065 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:42:29.065 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:42:29.066 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:29.066 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.066 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:42:29.066 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:42:29.066 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:42:29.067 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:42:29.067 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:42:29.067 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:42:29.067 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:42:29.068 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:42:29.069 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:42:29.069 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:42:29.069 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:42:29.069 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:42:29.070 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:42:29.070 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:42:29.070 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:42:29.070 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:42:29.071 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:42:29.071 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:42:29.071 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:42:29.071 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:42:29.071 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:42:29.072 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:42:29.072 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:42:29.072 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:29.073 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:42:29.073 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.073 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:42:29.073 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.074 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:42:29.074 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:42:29.075 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:42:29.075 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:42:29.075 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.075 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:42:29.076 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:42:29.076 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:29.076 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.076 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.077 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:42:29.077 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:42:29.077 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:29.077 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:29.078 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:29.078 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.078 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:42:29.079 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:42:29.079 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:42:29.079 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:42:29.079 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:42:29.080 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.080 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.080 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:42:29.080 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:42:29.080 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:42:29.081 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:42:29.081 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.081 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:42:29.081 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:42:29.081 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:29.082 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.082 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.082 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:42:29.082 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:42:29.082 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:29.083 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:29.083 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:29.083 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.083 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:42:29.083 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:42:29.084 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:42:29.084 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:42:29.084 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:42:29.084 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.084 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.085 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:42:29.085 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:42:29.085 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:42:29.085 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:42:29.085 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.086 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:42:29.086 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:42:29.086 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:29.086 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.087 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.087 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:42:29.087 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:42:29.087 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:29.087 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:29.088 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:29.088 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.088 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:42:29.088 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:42:29.088 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:42:29.089 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:42:29.089 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:42:29.089 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:29.089 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:29.089 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.090 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:42:29.090 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:42:29.090 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.090 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:29.091 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:29.091 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:42:29.091 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:29.091 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:29.091 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:42:29.092 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.092 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:29.092 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:29.092 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:42:29.092 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:29.093 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:29.093 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:42:29.093 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.093 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:29.093 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:29.094 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 8 2020-11-20T11:42:29.094 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:29.094 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:29.094 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:42:29.094 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.095 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:29.095 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:29.095 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:42:29.095 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:29.095 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:29.096 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:42:29.096 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.096 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:29.096 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:29.096 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:42:29.097 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:29.097 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:29.097 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.097 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:42:29.098 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:42:29.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.055+0000 7f83342e6700 1 -- 172.21.15.23:0/3187890337 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8304056d60 msgr2=0x7f8304059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:29.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.055+0000 7f83342e6700 1 --2- 172.21.15.23:0/3187890337 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8304056d60 0x7f8304059220 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f831c000f60 tx=0x7f831c007bb0).stop 2020-11-20T11:42:29.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.055+0000 7f83342e6700 1 -- 172.21.15.23:0/3187890337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f832c060ca0 msgr2=0x7f832c14a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:29.099 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.055+0000 7f83342e6700 1 --2- 172.21.15.23:0/3187890337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f832c060ca0 0x7f832c14a590 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f832c1388a0 tx=0x7f8320015000).stop 2020-11-20T11:42:29.099 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.055+0000 7f8332883700 1 -- 172.21.15.23:0/3187890337 reap_dead start 2020-11-20T11:42:29.099 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.055+0000 7f8332883700 1 -- 172.21.15.23:0/3187890337 reap_dead start 2020-11-20T11:42:29.099 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.056+0000 7f83342e6700 1 -- 172.21.15.23:0/3187890337 shutdown_connections 2020-11-20T11:42:29.099 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.056+0000 7f83342e6700 1 -- 172.21.15.23:0/3187890337 >> 172.21.15.23:0/3187890337 conn(0x7f832c066750 msgr2=0x7f832c137a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:29.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.056+0000 7f83342e6700 1 -- 172.21.15.23:0/3187890337 shutdown_connections 2020-11-20T11:42:29.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.056+0000 7f83342e6700 1 -- 172.21.15.23:0/3187890337 wait complete. 2020-11-20T11:42:29.101 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:42:29.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.268+0000 7f828f875700 1 Processor -- start 2020-11-20T11:42:29.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.268+0000 7f828f875700 1 -- start start 2020-11-20T11:42:29.281 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.269+0000 7f828f875700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8288067eb0 0x7f8288066700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:29.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.269+0000 7f828f875700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8288068430 con 0x7f828805bc80 2020-11-20T11:42:29.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.269+0000 7f828f875700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82880686d0 con 0x7f828813ae70 2020-11-20T11:42:29.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.269+0000 7f828f875700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8288067470 con 0x7f8288067eb0 2020-11-20T11:42:29.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.270+0000 7f828d611700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f828805bc80 0x7f82881345e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:59116/0 (socket says 172.21.15.23:59116) 2020-11-20T11:42:29.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.270+0000 7f828d611700 1 -- 172.21.15.23:0/1113478832 learned_addr learned my addr 172.21.15.23:0/1113478832 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:42:29.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.270+0000 7f828ce10700 1 --2- 172.21.15.23:0/1113478832 >> v2:172.21.15.23:3300/0 conn(0x7f8288067eb0 0x7f8288066700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:29.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.270+0000 7f8277fff700 1 -- 172.21.15.23:0/1113478832 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3951108252 0 0) 0x7f82880686d0 con 0x7f828813ae70 2020-11-20T11:42:29.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.270+0000 7f8277fff700 1 -- 172.21.15.23:0/1113478832 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8270003280 con 0x7f828813ae70 2020-11-20T11:42:29.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.270+0000 7f828ce10700 1 -- 172.21.15.23:0/1113478832 >> v1:172.21.15.155:6789/0 conn(0x7f828813ae70 legacy=0x7f8288134cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:29.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.270+0000 7f828ce10700 1 -- 172.21.15.23:0/1113478832 >> v1:172.21.15.23:6789/0 conn(0x7f828805bc80 legacy=0x7f82881345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:29.287 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.270+0000 7f828ce10700 1 -- 172.21.15.23:0/1113478832 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8288066260 con 0x7f8288067eb0 2020-11-20T11:42:29.287 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.270+0000 7f828ce10700 1 --2- 172.21.15.23:0/1113478832 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8288067eb0 0x7f8288066700 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f827800dc10 tx=0x7f8278006ec0).ready entity=mon.0 client_cookie=3beb0f799adea807 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:29.287 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.272+0000 7f8277fff700 1 -- 172.21.15.23:0/1113478832 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827800bb50 con 0x7f8288067eb0 2020-11-20T11:42:29.288 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.272+0000 7f8277fff700 1 -- 172.21.15.23:0/1113478832 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8278000e90 con 0x7f8288067eb0 2020-11-20T11:42:29.288 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.272+0000 7f8277fff700 1 -- 172.21.15.23:0/1113478832 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8278010470 con 0x7f8288067eb0 2020-11-20T11:42:29.288 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.275+0000 7f828f875700 1 -- 172.21.15.23:0/1113478832 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8288067eb0 msgr2=0x7f8288066700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:29.289 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.275+0000 7f828f875700 1 --2- 172.21.15.23:0/1113478832 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8288067eb0 0x7f8288066700 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f827800dc10 tx=0x7f8278006ec0).stop 2020-11-20T11:42:29.289 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.275+0000 7f828f875700 1 -- 172.21.15.23:0/1113478832 shutdown_connections 2020-11-20T11:42:29.290 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.275+0000 7f828f875700 1 --2- 172.21.15.23:0/1113478832 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8288067eb0 0x7f8288066700 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:29.290 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.275+0000 7f828f875700 1 -- 172.21.15.23:0/1113478832 >> 172.21.15.23:0/1113478832 conn(0x7f82880ad7a0 msgr2=0x7f82880adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:29.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.277+0000 7f828f875700 1 -- 172.21.15.23:0/1113478832 shutdown_connections 2020-11-20T11:42:29.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.277+0000 7f828f875700 1 -- 172.21.15.23:0/1113478832 wait complete. 2020-11-20T11:42:29.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.279+0000 7f828f875700 1 Processor -- start 2020-11-20T11:42:29.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.279+0000 7f828f875700 1 -- start start 2020-11-20T11:42:29.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.281+0000 7f828f875700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f828805bc80 0x7f82881346c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:29.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.281+0000 7f828f875700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8288067eb0 0x7f828809ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:29.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.281+0000 7f828f875700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f828813ae70 0x7f828809f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:29.298 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.281+0000 7f828f875700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f828813b2e0 con 0x7f828813ae70 2020-11-20T11:42:29.298 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.281+0000 7f828f875700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f828809f620 con 0x7f828805bc80 2020-11-20T11:42:29.299 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.281+0000 7f828f875700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82880a7b40 con 0x7f8288067eb0 2020-11-20T11:42:29.299 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.281+0000 7f828de12700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f828813ae70 0x7f828809f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:29.299 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.281+0000 7f828de12700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f828813ae70 0x7f828809f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44874/0 (socket says 172.21.15.23:44874) 2020-11-20T11:42:29.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.281+0000 7f828de12700 1 -- 172.21.15.23:0/2741540864 learned_addr learned my addr 172.21.15.23:0/2741540864 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:42:29.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.282+0000 7f828de12700 1 -- 172.21.15.23:0/2741540864 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8288067eb0 msgr2=0x7f828809ec40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:42:29.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.282+0000 7f828de12700 1 --2- 172.21.15.23:0/2741540864 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8288067eb0 0x7f828809ec40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:29.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.282+0000 7f828de12700 1 -- 172.21.15.23:0/2741540864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f828805bc80 msgr2=0x7f82881346c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2020-11-20T11:42:29.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.282+0000 7f828de12700 1 --2- 172.21.15.23:0/2741540864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f828805bc80 0x7f82881346c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:29.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.282+0000 7f828de12700 1 -- 172.21.15.23:0/2741540864 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82880a00a0 con 0x7f828813ae70 2020-11-20T11:42:29.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.282+0000 7f828de12700 1 --2- 172.21.15.23:0/2741540864 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f828813ae70 0x7f828809f130 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f828000cac0 tx=0x7f8280007e60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:29.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.282+0000 7f82757fa700 1 -- 172.21.15.23:0/2741540864 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828000a7f0 con 0x7f828813ae70 2020-11-20T11:42:29.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.282+0000 7f82757fa700 1 -- 172.21.15.23:0/2741540864 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f828000a950 con 0x7f828813ae70 2020-11-20T11:42:29.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.282+0000 7f828f875700 1 -- 172.21.15.23:0/2741540864 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82880a0340 con 0x7f828813ae70 2020-11-20T11:42:29.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.282+0000 7f828f875700 1 -- 172.21.15.23:0/2741540864 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8288134f00 con 0x7f828813ae70 2020-11-20T11:42:29.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.283+0000 7f82757fa700 1 -- 172.21.15.23:0/2741540864 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828000acf0 con 0x7f828813ae70 2020-11-20T11:42:29.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.283+0000 7f828f875700 1 --2- 172.21.15.23:0/2741540864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82880a2140 0x7f8288131e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:29.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.283+0000 7f828f875700 1 -- 172.21.15.23:0/2741540864 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f828809ff50 con 0x7f82880a2140 2020-11-20T11:42:29.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.285+0000 7f82757fa700 1 -- 172.21.15.23:0/2741540864 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8280010070 con 0x7f828813ae70 2020-11-20T11:42:29.308 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.285+0000 7f828d611700 1 --2- 172.21.15.23:0/2741540864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82880a2140 0x7f8288131e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:29.308 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.285+0000 7f82757fa700 1 --2- 172.21.15.23:0/2741540864 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8260056d90 0x7f8260059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:29.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.285+0000 7f82757fa700 1 -- 172.21.15.23:0/2741540864 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f828006ffc0 con 0x7f828813ae70 2020-11-20T11:42:29.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.286+0000 7f828ce10700 1 --2- 172.21.15.23:0/2741540864 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8260056d90 0x7f8260059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:29.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.286+0000 7f828d611700 1 -- 172.21.15.23:0/2741540864 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f82880a9000 con 0x7f82880a2140 2020-11-20T11:42:29.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.286+0000 7f828d611700 1 --2- 172.21.15.23:0/2741540864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82880a2140 0x7f8288131e30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f8284007fd0 tx=0x7f8284007b50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:29.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.286+0000 7f82757fa700 1 -- 172.21.15.23:0/2741540864 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82840072e0 con 0x7f82880a2140 2020-11-20T11:42:29.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.286+0000 7f828ce10700 1 --2- 172.21.15.23:0/2741540864 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8260056d90 0x7f8260059250 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f82880a7820 tx=0x7f82880685c0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:29.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.286+0000 7f82757fa700 1 -- 172.21.15.23:0/2741540864 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f8284007440 con 0x7f82880a2140 2020-11-20T11:42:29.313 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.286+0000 7f82757fa700 1 -- 172.21.15.23:0/2741540864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82880a2140 msgr2=0x7f8288131e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:29.314 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.286+0000 7f82757fa700 1 --2- 172.21.15.23:0/2741540864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82880a2140 0x7f8288131e30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f8284007fd0 tx=0x7f8284007b50).stop 2020-11-20T11:42:29.315 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.286+0000 7f82757fa700 1 -- 172.21.15.23:0/2741540864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82880a2140 msgr2=0x7f8288131e30 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:42:29.315 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.286+0000 7f82757fa700 1 --2- 172.21.15.23:0/2741540864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f82880a2140 0x7f8288131e30 secure :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0x7f8284007fd0 tx=0x7f8284007b50).stop 2020-11-20T11:42:29.321 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.302+0000 7f828f875700 1 --2- 172.21.15.23:0/2741540864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8288132320 0x7f828809d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:29.322 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.302+0000 7f828f875700 1 -- 172.21.15.23:0/2741540864 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82880a04d0 con 0x7f8288132320 2020-11-20T11:42:29.323 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.302+0000 7f828d611700 1 --2- 172.21.15.23:0/2741540864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8288132320 0x7f828809d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:29.324 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.302+0000 7f828d611700 1 -- 172.21.15.23:0/2741540864 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8288062fe0 con 0x7f8288132320 2020-11-20T11:42:29.325 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.304+0000 7f828d611700 1 --2- 172.21.15.23:0/2741540864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8288132320 0x7f828809d4a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f8284012000 tx=0x7f8284015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:29.326 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.305+0000 7f82757fa700 1 -- 172.21.15.23:0/2741540864 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8284014b80 con 0x7f8288132320 2020-11-20T11:42:29.326 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.305+0000 7f82757fa700 1 -- 172.21.15.23:0/2741540864 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4273 (secure 0 0 0) 0x7f8284014ce0 con 0x7f8288132320 2020-11-20T11:42:29.327 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.305+0000 7f82757fa700 1 -- 172.21.15.23:0/2741540864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8288132320 msgr2=0x7f828809d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:29.328 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.305+0000 7f82757fa700 1 --2- 172.21.15.23:0/2741540864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8288132320 0x7f828809d4a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f8284012000 tx=0x7f8284015000).stop 2020-11-20T11:42:29.328 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.305+0000 7f82757fa700 1 -- 172.21.15.23:0/2741540864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8288132320 msgr2=0x7f828809d4a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:42:29.330 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.305+0000 7f82757fa700 1 --2- 172.21.15.23:0/2741540864 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8288132320 0x7f828809d4a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:29.331 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.306+0000 7f828f875700 1 -- 172.21.15.23:0/2741540864 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8260056d90 msgr2=0x7f8260059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:29.331 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.306+0000 7f828f875700 1 --2- 172.21.15.23:0/2741540864 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8260056d90 0x7f8260059250 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f82880a7820 tx=0x7f82880685c0).stop 2020-11-20T11:42:29.333 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.306+0000 7f828f875700 1 -- 172.21.15.23:0/2741540864 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f828813ae70 msgr2=0x7f828809f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:29.333 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.306+0000 7f828f875700 1 --2- 172.21.15.23:0/2741540864 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f828813ae70 0x7f828809f130 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f828000cac0 tx=0x7f8280007e60).stop 2020-11-20T11:42:29.333 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.306+0000 7f828de12700 1 -- 172.21.15.23:0/2741540864 reap_dead start 2020-11-20T11:42:29.333 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.306+0000 7f828de12700 1 -- 172.21.15.23:0/2741540864 reap_dead start 2020-11-20T11:42:29.334 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.306+0000 7f828f875700 1 -- 172.21.15.23:0/2741540864 shutdown_connections 2020-11-20T11:42:29.334 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.306+0000 7f828f875700 1 -- 172.21.15.23:0/2741540864 >> 172.21.15.23:0/2741540864 conn(0x7f82880ad7a0 msgr2=0x7f82880adb80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:29.334 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.306+0000 7f828f875700 1 -- 172.21.15.23:0/2741540864 shutdown_connections 2020-11-20T11:42:29.334 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.306+0000 7f828f875700 1 -- 172.21.15.23:0/2741540864 wait complete. 2020-11-20T11:42:29.335 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:42:29.335 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:42:29.335 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:42:29.335 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:42:29.335 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 92, 2020-11-20T11:42:29.336 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:42:29.336 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:42:29.336 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:42:29.336 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:42:29.336 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:29.337 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 3, 2020-11-20T11:42:29.337 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:42:29.337 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:42:29.337 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:42:29.337 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:42:29.338 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:42:29.338 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:42:29.338 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:42:29.338 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:42:29.339 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:42:29.339 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:42:29.339 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:42:29.339 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:29.339 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:42:29.340 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:42:29.340 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:42:29.340 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:42:29.340 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:42:29.340 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:42:29.341 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:42:29.341 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:42:29.341 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:42:29.341 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:42:29.341 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:29.342 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.342 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:42:29.342 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:42:29.342 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:42:29.342 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:42:29.343 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:42:29.343 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:42:29.343 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:42:29.343 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:42:29.343 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:42:29.344 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:42:29.344 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:42:29.344 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:42:29.344 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:42:29.344 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:42:29.345 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:42:29.345 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:42:29.345 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:42:29.345 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:42:29.345 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:42:29.346 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:42:29.346 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:42:29.346 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:42:29.346 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:42:29.347 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:29.347 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:42:29.347 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.347 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:42:29.347 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.348 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:42:29.348 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:42:29.348 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:42:29.348 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:42:29.348 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.349 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:42:29.349 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:42:29.349 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:29.349 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.350 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.351 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:42:29.351 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:42:29.351 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:29.351 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:29.352 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:29.352 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.352 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:42:29.352 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:42:29.352 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:42:29.353 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:42:29.353 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:42:29.353 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.353 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.353 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:42:29.354 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:42:29.354 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:42:29.354 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:42:29.354 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.354 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:42:29.355 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:42:29.355 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:29.355 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.355 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.355 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:42:29.356 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:42:29.356 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:29.356 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:29.356 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:29.356 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.357 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:42:29.357 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:42:29.357 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:42:29.357 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:42:29.357 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:42:29.358 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.358 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.358 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:42:29.358 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:42:29.358 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:42:29.359 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:42:29.360 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.360 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:42:29.361 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:42:29.361 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:29.361 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.361 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.362 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:42:29.362 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:42:29.362 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:29.363 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:29.363 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:29.363 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.364 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:42:29.364 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:42:29.364 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:42:29.365 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:42:29.365 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:42:29.365 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:29.366 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:29.366 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.367 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:42:29.367 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:42:29.367 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.368 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:29.368 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:29.368 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:42:29.369 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:29.369 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:29.369 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:42:29.369 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.369 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:29.370 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:29.370 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:42:29.370 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:29.370 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:29.370 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.371 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:42:29.371 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:42:29.372 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:42:29.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.555+0000 7f08d75a1700 1 Processor -- start 2020-11-20T11:42:29.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.556+0000 7f08d75a1700 1 -- start start 2020-11-20T11:42:29.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.556+0000 7f08d75a1700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f08d00a6e10 0x7f08d0061850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:29.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.556+0000 7f08d75a1700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f08d0137870 0x7f08d013ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:29.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.556+0000 7f08d75a1700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d0062ad0 con 0x7f08d0137410 2020-11-20T11:42:29.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.556+0000 7f08d75a1700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08d00a61f0 con 0x7f08d0137870 2020-11-20T11:42:29.573 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.556+0000 7f08d75a1700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08d00a63e0 con 0x7f08d00a6e10 2020-11-20T11:42:29.573 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.557+0000 7f08d5b3e700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f08d0137870 0x7f08d013ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:29.573 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.557+0000 7f08d5b3e700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f08d0137870 0x7f08d013ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44886/0 (socket says 172.21.15.23:44886) 2020-11-20T11:42:29.573 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.557+0000 7f08d4b3c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f08d0137410 0x7f08d013a760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:59134/0 (socket says 172.21.15.23:59134) 2020-11-20T11:42:29.574 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.557+0000 7f08d5b3e700 1 -- 172.21.15.23:0/3367856538 learned_addr learned my addr 172.21.15.23:0/3367856538 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:42:29.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.557+0000 7f08d533d700 1 --2- 172.21.15.23:0/3367856538 >> v2:172.21.15.155:3300/0 conn(0x7f08d00a6e10 0x7f08d0061850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:29.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.557+0000 7f08d5b3e700 1 -- 172.21.15.23:0/3367856538 >> v2:172.21.15.155:3300/0 conn(0x7f08d00a6e10 msgr2=0x7f08d0061850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:42:29.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.557+0000 7f08d5b3e700 1 --2- 172.21.15.23:0/3367856538 >> v2:172.21.15.155:3300/0 conn(0x7f08d00a6e10 0x7f08d0061850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:29.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.557+0000 7f08d5b3e700 1 -- 172.21.15.23:0/3367856538 >> v1:172.21.15.23:6789/0 conn(0x7f08d0137410 legacy=0x7f08d013a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:29.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.557+0000 7f08d5b3e700 1 -- 172.21.15.23:0/3367856538 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d00a6680 con 0x7f08d0137870 2020-11-20T11:42:29.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.557+0000 7f08d5b3e700 1 --2- 172.21.15.23:0/3367856538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08d0137870 0x7f08d013ae70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f08d00aded0 tx=0x7f08c80092d0).ready entity=mon.0 client_cookie=9598b51462d03a6b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:29.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.557+0000 7f08bffff700 1 -- 172.21.15.23:0/3367856538 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c800bbf0 con 0x7f08d0137870 2020-11-20T11:42:29.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.557+0000 7f08bffff700 1 -- 172.21.15.23:0/3367856538 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08c800bd50 con 0x7f08d0137870 2020-11-20T11:42:29.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.557+0000 7f08bffff700 1 -- 172.21.15.23:0/3367856538 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c8011610 con 0x7f08d0137870 2020-11-20T11:42:29.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.558+0000 7f08d75a1700 1 -- 172.21.15.23:0/3367856538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08d0137870 msgr2=0x7f08d013ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:29.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.558+0000 7f08d75a1700 1 --2- 172.21.15.23:0/3367856538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08d0137870 0x7f08d013ae70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f08d00aded0 tx=0x7f08c80092d0).stop 2020-11-20T11:42:29.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.558+0000 7f08d75a1700 1 -- 172.21.15.23:0/3367856538 shutdown_connections 2020-11-20T11:42:29.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.558+0000 7f08d75a1700 1 --2- 172.21.15.23:0/3367856538 >> v2:172.21.15.155:3300/0 conn(0x7f08d00a6e10 0x7f08d0061850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:29.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.558+0000 7f08d75a1700 1 --2- 172.21.15.23:0/3367856538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08d0137870 0x7f08d013ae70 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:29.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.558+0000 7f08d75a1700 1 -- 172.21.15.23:0/3367856538 >> 172.21.15.23:0/3367856538 conn(0x7f08d0060c80 msgr2=0x7f08d00acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:29.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.558+0000 7f08d75a1700 1 -- 172.21.15.23:0/3367856538 shutdown_connections 2020-11-20T11:42:29.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.558+0000 7f08d75a1700 1 -- 172.21.15.23:0/3367856538 wait complete. 2020-11-20T11:42:29.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.559+0000 7f08d75a1700 1 Processor -- start 2020-11-20T11:42:29.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.559+0000 7f08d75a1700 1 -- start start 2020-11-20T11:42:29.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.560+0000 7f08d75a1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08d00a6e10 0x7f08d00ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:29.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.560+0000 7f08d75a1700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08d0137410 0x7f08d01318e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:29.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.560+0000 7f08d75a1700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08d0137870 0x7f08d0131dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:29.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.560+0000 7f08d75a1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d01322c0 con 0x7f08d00a6e10 2020-11-20T11:42:29.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.560+0000 7f08d75a1700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d0133660 con 0x7f08d0137410 2020-11-20T11:42:29.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.560+0000 7f08d75a1700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08d0133900 con 0x7f08d0137870 2020-11-20T11:42:29.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.560+0000 7f08d533d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08d00a6e10 0x7f08d00ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:29.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.560+0000 7f08d533d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08d00a6e10 0x7f08d00ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44888/0 (socket says 172.21.15.23:44888) 2020-11-20T11:42:29.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.560+0000 7f08d533d700 1 -- 172.21.15.23:0/3177669945 learned_addr learned my addr 172.21.15.23:0/3177669945 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:42:29.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.560+0000 7f08d4b3c700 1 --2- 172.21.15.23:0/3177669945 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08d0137410 0x7f08d01318e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:29.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.560+0000 7f08d5b3e700 1 --2- 172.21.15.23:0/3177669945 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08d0137870 0x7f08d0131dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:29.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.560+0000 7f08d533d700 1 -- 172.21.15.23:0/3177669945 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08d0137870 msgr2=0x7f08d0131dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:29.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.560+0000 7f08d533d700 1 --2- 172.21.15.23:0/3177669945 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08d0137870 0x7f08d0131dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:29.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.560+0000 7f08d533d700 1 -- 172.21.15.23:0/3177669945 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08d0137410 msgr2=0x7f08d01318e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:29.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.560+0000 7f08d533d700 1 --2- 172.21.15.23:0/3177669945 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08d0137410 0x7f08d01318e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:29.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.560+0000 7f08d533d700 1 -- 172.21.15.23:0/3177669945 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d0133ba0 con 0x7f08d00a6e10 2020-11-20T11:42:29.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.560+0000 7f08d4b3c700 1 --2- 172.21.15.23:0/3177669945 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08d0137410 0x7f08d01318e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:42:29.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.560+0000 7f08d533d700 1 --2- 172.21.15.23:0/3177669945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08d00a6e10 0x7f08d00ace20 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f08cc00eba0 tx=0x7f08cc006e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:29.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.560+0000 7f08bd7fa700 1 -- 172.21.15.23:0/3177669945 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08cc00c850 con 0x7f08d00a6e10 2020-11-20T11:42:29.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.560+0000 7f08bd7fa700 1 -- 172.21.15.23:0/3177669945 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08cc00c9b0 con 0x7f08d00a6e10 2020-11-20T11:42:29.584 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.560+0000 7f08d75a1700 1 -- 172.21.15.23:0/3177669945 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08d00ad8f0 con 0x7f08d00a6e10 2020-11-20T11:42:29.584 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.560+0000 7f08bd7fa700 1 -- 172.21.15.23:0/3177669945 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08cc00cd30 con 0x7f08d00a6e10 2020-11-20T11:42:29.584 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.561+0000 7f08d75a1700 1 -- 172.21.15.23:0/3177669945 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08d01310b0 con 0x7f08d00a6e10 2020-11-20T11:42:29.584 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.561+0000 7f08d75a1700 1 --2- 172.21.15.23:0/3177669945 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08d014b3f0 0x7f08d014b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:29.584 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.561+0000 7f08d75a1700 1 -- 172.21.15.23:0/3177669945 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08d005c520 con 0x7f08d014b3f0 2020-11-20T11:42:29.585 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.562+0000 7f08d4b3c700 1 --2- 172.21.15.23:0/3177669945 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08d014b3f0 0x7f08d014b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:29.585 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.562+0000 7f08bd7fa700 1 -- 172.21.15.23:0/3177669945 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f08cc012600 con 0x7f08d00a6e10 2020-11-20T11:42:29.585 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.562+0000 7f08d4b3c700 1 -- 172.21.15.23:0/3177669945 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f08d0044710 con 0x7f08d014b3f0 2020-11-20T11:42:29.585 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.562+0000 7f08d4b3c700 1 --2- 172.21.15.23:0/3177669945 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08d014b3f0 0x7f08d014b7d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f08d0133fb0 tx=0x7f08c0007bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:29.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.562+0000 7f08bd7fa700 1 --2- 172.21.15.23:0/3177669945 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f08a8056d60 0x7f08a8059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:29.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.562+0000 7f08bd7fa700 1 -- 172.21.15.23:0/3177669945 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f08cc070370 con 0x7f08d00a6e10 2020-11-20T11:42:29.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.562+0000 7f08d5b3e700 1 --2- 172.21.15.23:0/3177669945 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f08a8056d60 0x7f08a8059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:29.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.562+0000 7f08bd7fa700 1 -- 172.21.15.23:0/3177669945 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c0007440 con 0x7f08d014b3f0 2020-11-20T11:42:29.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.562+0000 7f08bd7fa700 1 -- 172.21.15.23:0/3177669945 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f08c0004e70 con 0x7f08d014b3f0 2020-11-20T11:42:29.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.562+0000 7f08bd7fa700 1 -- 172.21.15.23:0/3177669945 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08d014b3f0 msgr2=0x7f08d014b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:29.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.562+0000 7f08bd7fa700 1 --2- 172.21.15.23:0/3177669945 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08d014b3f0 0x7f08d014b7d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f08d0133fb0 tx=0x7f08c0007bc0).stop 2020-11-20T11:42:29.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.562+0000 7f08bd7fa700 1 -- 172.21.15.23:0/3177669945 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08d014b3f0 msgr2=0x7f08d014b7d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:42:29.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.562+0000 7f08bd7fa700 1 --2- 172.21.15.23:0/3177669945 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08d014b3f0 0x7f08d014b7d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:29.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.563+0000 7f08d5b3e700 1 --2- 172.21.15.23:0/3177669945 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f08a8056d60 0x7f08a8059220 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f08c8009310 tx=0x7f08c8016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:29.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.581+0000 7f08d75a1700 1 --2- 172.21.15.23:0/3177669945 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08d014bcc0 0x7f08d012dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:29.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.581+0000 7f08d75a1700 1 -- 172.21.15.23:0/3177669945 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08d00687a0 con 0x7f08d014bcc0 2020-11-20T11:42:29.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.582+0000 7f08d4b3c700 1 --2- 172.21.15.23:0/3177669945 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08d014bcc0 0x7f08d012dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:29.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.582+0000 7f08d4b3c700 1 -- 172.21.15.23:0/3177669945 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f08d005c280 con 0x7f08d014bcc0 2020-11-20T11:42:29.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.583+0000 7f08d4b3c700 1 --2- 172.21.15.23:0/3177669945 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08d014bcc0 0x7f08d012dcd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f08d0133fb0 tx=0x7f08c0017000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:29.595 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.592+0000 7f08bd7fa700 1 -- 172.21.15.23:0/3177669945 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c0015e90 con 0x7f08d014bcc0 2020-11-20T11:42:29.595 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.592+0000 7f08bd7fa700 1 -- 172.21.15.23:0/3177669945 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4443 (secure 0 0 0) 0x7f08c00233e0 con 0x7f08d014bcc0 2020-11-20T11:42:29.596 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.592+0000 7f08bd7fa700 1 -- 172.21.15.23:0/3177669945 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08d014bcc0 msgr2=0x7f08d012dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:29.596 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.592+0000 7f08bd7fa700 1 --2- 172.21.15.23:0/3177669945 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08d014bcc0 0x7f08d012dcd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f08d0133fb0 tx=0x7f08c0017000).stop 2020-11-20T11:42:29.596 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.592+0000 7f08bd7fa700 1 -- 172.21.15.23:0/3177669945 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08d014bcc0 msgr2=0x7f08d012dcd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:42:29.597 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.592+0000 7f08bd7fa700 1 --2- 172.21.15.23:0/3177669945 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08d014bcc0 0x7f08d012dcd0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:29.597 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:42:29.597 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:42:29.597 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:42:29.598 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:42:29.598 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 92, 2020-11-20T11:42:29.598 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:42:29.598 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:42:29.598 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:42:29.599 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:42:29.599 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:29.600 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 4, 2020-11-20T11:42:29.600 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:42:29.600 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:42:29.600 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:42:29.601 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:42:29.601 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:42:29.601 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:42:29.601 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:42:29.601 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:42:29.602 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:42:29.602 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:42:29.603 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:42:29.603 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:29.603 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:42:29.603 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:42:29.603 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:42:29.604 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:42:29.604 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:42:29.605 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:42:29.605 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:42:29.605 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:42:29.606 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:42:29.606 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:42:29.606 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:29.607 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.607 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:42:29.608 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:42:29.608 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:42:29.608 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:42:29.609 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:42:29.609 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:42:29.609 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:42:29.610 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:42:29.610 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:42:29.610 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:42:29.610 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:42:29.611 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:42:29.611 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:42:29.611 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:42:29.611 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:42:29.612 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:42:29.612 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:42:29.612 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:42:29.613 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:42:29.613 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:42:29.614 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:42:29.614 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:42:29.614 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:42:29.615 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:29.615 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:42:29.615 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.616 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:42:29.616 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.617 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:42:29.617 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:42:29.617 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:42:29.618 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:42:29.618 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.619 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:42:29.619 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:42:29.619 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:29.620 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.620 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.620 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:42:29.621 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:42:29.621 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:29.621 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:29.622 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:29.622 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.622 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:42:29.622 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:42:29.622 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:42:29.623 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:42:29.623 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:42:29.623 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.623 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.623 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:42:29.624 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:42:29.624 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:42:29.624 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:42:29.624 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.625 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:42:29.625 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:42:29.625 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:29.625 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.625 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.626 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:42:29.626 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:42:29.626 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:29.627 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:29.627 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:29.627 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.628 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:42:29.628 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:42:29.628 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:42:29.629 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:42:29.629 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:42:29.629 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.630 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.630 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:42:29.630 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:42:29.630 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:42:29.630 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:42:29.631 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.631 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:42:29.631 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:42:29.631 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:29.631 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.632 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.632 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:42:29.632 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:42:29.632 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:29.632 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:29.633 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:29.633 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.633 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:42:29.633 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:42:29.633 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:42:29.634 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:42:29.634 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:42:29.634 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:29.634 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:29.634 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.635 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:42:29.635 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:42:29.635 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.635 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:29.635 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:29.636 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:42:29.636 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:29.636 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:29.636 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:42:29.636 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.636 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:29.637 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:29.637 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:42:29.637 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:29.637 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:29.637 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:42:29.638 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:29.638 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:29.638 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:29.638 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:42:29.638 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:29.639 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:29.639 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:29.639 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:42:29.639 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:42:29.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.596+0000 7f08aeffd700 1 -- 172.21.15.23:0/3177669945 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f08a8056d60 msgr2=0x7f08a8059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:29.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.596+0000 7f08aeffd700 1 --2- 172.21.15.23:0/3177669945 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f08a8056d60 0x7f08a8059220 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f08c8009310 tx=0x7f08c8016000).stop 2020-11-20T11:42:29.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.596+0000 7f08d5b3e700 1 -- 172.21.15.23:0/3177669945 reap_dead start 2020-11-20T11:42:29.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.596+0000 7f08aeffd700 1 -- 172.21.15.23:0/3177669945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08d00a6e10 msgr2=0x7f08d00ace20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:29.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.596+0000 7f08aeffd700 1 --2- 172.21.15.23:0/3177669945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08d00a6e10 0x7f08d00ace20 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f08cc00eba0 tx=0x7f08cc006e30).stop 2020-11-20T11:42:29.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.596+0000 7f08aeffd700 1 -- 172.21.15.23:0/3177669945 shutdown_connections 2020-11-20T11:42:29.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.596+0000 7f08aeffd700 1 --2- 172.21.15.23:0/3177669945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08d00a6e10 0x7f08d00ace20 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:29.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.596+0000 7f08aeffd700 1 -- 172.21.15.23:0/3177669945 >> 172.21.15.23:0/3177669945 conn(0x7f08d0060c80 msgr2=0x7f08d01346b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:29.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.596+0000 7f08aeffd700 1 -- 172.21.15.23:0/3177669945 shutdown_connections 2020-11-20T11:42:29.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.596+0000 7f08aeffd700 1 -- 172.21.15.23:0/3177669945 wait complete. 2020-11-20T11:42:29.643 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2020-11-20T11:42:29.643 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2020-11-20T11:42:29.840 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.837+0000 7fda2ede1700 1 Processor -- start 2020-11-20T11:42:29.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.837+0000 7fda2ede1700 1 -- start start 2020-11-20T11:42:29.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.838+0000 7fda2ede1700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fda280aac70 0x7fda28142ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:29.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.838+0000 7fda2ede1700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda280a9690 con 0x7fda280a8c70 2020-11-20T11:42:29.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.838+0000 7fda2ede1700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda280a8480 con 0x7fda280ab480 2020-11-20T11:42:29.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.838+0000 7fda2ede1700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda280a9d60 con 0x7fda280aac70 2020-11-20T11:42:29.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.839+0000 7fda2d37e700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fda280ab480 0x7fda28149c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:37126/0 (socket says 172.21.15.23:37126) 2020-11-20T11:42:29.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.839+0000 7fda2d37e700 1 -- 172.21.15.23:0/638906362 learned_addr learned my addr 172.21.15.23:0/638906362 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:42:29.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.839+0000 7fda277fe700 1 -- 172.21.15.23:0/638906362 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129080152 0 0) 0x7fda280a8480 con 0x7fda280ab480 2020-11-20T11:42:29.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.839+0000 7fda277fe700 1 -- 172.21.15.23:0/638906362 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda08003280 con 0x7fda280ab480 2020-11-20T11:42:29.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.840+0000 7fda2cb7d700 1 --2- 172.21.15.23:0/638906362 >> v2:172.21.15.155:3300/0 conn(0x7fda280aac70 0x7fda28142ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:29.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.842+0000 7fda277fe700 1 -- 172.21.15.23:0/638906362 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4090624716 0 0) 0x7fda08003280 con 0x7fda280ab480 2020-11-20T11:42:29.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.842+0000 7fda2cb7d700 1 -- 172.21.15.23:0/638906362 >> v1:172.21.15.155:6789/0 conn(0x7fda280ab480 legacy=0x7fda28149c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:29.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.842+0000 7fda2cb7d700 1 -- 172.21.15.23:0/638906362 >> v1:172.21.15.23:6789/0 conn(0x7fda280a8c70 legacy=0x7fda2813c210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:42:29.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.842+0000 7fda2cb7d700 1 -- 172.21.15.23:0/638906362 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda280a9150 con 0x7fda280aac70 2020-11-20T11:42:29.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.842+0000 7fda2cb7d700 1 --2- 172.21.15.23:0/638906362 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fda280aac70 0x7fda28142ba0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fda20009c60 tx=0x7fda20009220).ready entity=mon.1 client_cookie=1ac3fdda078e4c1f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:29.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.843+0000 7fda277fe700 1 -- 172.21.15.23:0/638906362 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda10001520 con 0x7fda280aac70 2020-11-20T11:42:29.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.843+0000 7fda277fe700 1 -- 172.21.15.23:0/638906362 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda2000b750 con 0x7fda280aac70 2020-11-20T11:42:29.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.844+0000 7fda2ede1700 1 -- 172.21.15.23:0/638906362 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fda280aac70 msgr2=0x7fda28142ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:29.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.844+0000 7fda2ede1700 1 --2- 172.21.15.23:0/638906362 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fda280aac70 0x7fda28142ba0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fda20009c60 tx=0x7fda20009220).stop 2020-11-20T11:42:29.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.844+0000 7fda2ede1700 1 -- 172.21.15.23:0/638906362 shutdown_connections 2020-11-20T11:42:29.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.844+0000 7fda2ede1700 1 --2- 172.21.15.23:0/638906362 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fda280aac70 0x7fda28142ba0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:29.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.844+0000 7fda2ede1700 1 -- 172.21.15.23:0/638906362 >> 172.21.15.23:0/638906362 conn(0x7fda280678e0 msgr2=0x7fda28067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:29.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.844+0000 7fda2ede1700 1 -- 172.21.15.23:0/638906362 shutdown_connections 2020-11-20T11:42:29.850 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.844+0000 7fda2ede1700 1 -- 172.21.15.23:0/638906362 wait complete. 2020-11-20T11:42:29.852 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.846+0000 7fda2ede1700 1 Processor -- start 2020-11-20T11:42:29.852 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.846+0000 7fda2ede1700 1 -- start start 2020-11-20T11:42:29.852 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.847+0000 7fda2ede1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda280a8c70 0x7fda28144d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:29.853 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.847+0000 7fda2ede1700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fda280aac70 0x7fda280a7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:29.853 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.847+0000 7fda2ede1700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda280ab480 0x7fda28151c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:29.853 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.847+0000 7fda2ede1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda28143760 con 0x7fda280a8c70 2020-11-20T11:42:29.853 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.847+0000 7fda2ede1700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda280a78b0 con 0x7fda280aac70 2020-11-20T11:42:29.853 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.847+0000 7fda2ede1700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda280a7b50 con 0x7fda280ab480 2020-11-20T11:42:29.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.847+0000 7fda2d37e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda280ab480 0x7fda28151c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:29.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.847+0000 7fda2d37e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda280ab480 0x7fda28151c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:55146/0 (socket says 172.21.15.23:55146) 2020-11-20T11:42:29.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.847+0000 7fda2d37e700 1 -- 172.21.15.23:0/3770683249 learned_addr learned my addr 172.21.15.23:0/3770683249 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:42:29.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.847+0000 7fda2cb7d700 1 --2- 172.21.15.23:0/3770683249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda280a8c70 0x7fda28144d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:29.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.847+0000 7fda27fff700 1 --2- 172.21.15.23:0/3770683249 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fda280aac70 0x7fda280a7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:29.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.848+0000 7fda2cb7d700 1 -- 172.21.15.23:0/3770683249 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda280ab480 msgr2=0x7fda28151c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:29.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.848+0000 7fda2cb7d700 1 --2- 172.21.15.23:0/3770683249 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda280ab480 0x7fda28151c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:29.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.848+0000 7fda2cb7d700 1 -- 172.21.15.23:0/3770683249 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fda280aac70 msgr2=0x7fda280a7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:29.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.848+0000 7fda2cb7d700 1 --2- 172.21.15.23:0/3770683249 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fda280aac70 0x7fda280a7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:29.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.848+0000 7fda2cb7d700 1 -- 172.21.15.23:0/3770683249 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda280a84e0 con 0x7fda280a8c70 2020-11-20T11:42:29.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.848+0000 7fda2cb7d700 1 --2- 172.21.15.23:0/3770683249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda280a8c70 0x7fda28144d40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fda20013000 tx=0x7fda20016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:29.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.848+0000 7fda24ff9700 1 -- 172.21.15.23:0/3770683249 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda20011e40 con 0x7fda280a8c70 2020-11-20T11:42:29.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.848+0000 7fda24ff9700 1 -- 172.21.15.23:0/3770683249 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda2001c680 con 0x7fda280a8c70 2020-11-20T11:42:29.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.848+0000 7fda24ff9700 1 -- 172.21.15.23:0/3770683249 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2001ca20 con 0x7fda280a8c70 2020-11-20T11:42:29.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.848+0000 7fda2ede1700 1 -- 172.21.15.23:0/3770683249 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda28142ac0 con 0x7fda280a8c70 2020-11-20T11:42:29.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.848+0000 7fda2ede1700 1 -- 172.21.15.23:0/3770683249 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda280a6da0 con 0x7fda280a8c70 2020-11-20T11:42:29.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.850+0000 7fda2ede1700 1 -- 172.21.15.23:0/3770683249 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda2805c7d0 con 0x7fda280a8c70 2020-11-20T11:42:29.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.854+0000 7fda24ff9700 1 -- 172.21.15.23:0/3770683249 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fda20017070 con 0x7fda280a8c70 2020-11-20T11:42:29.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.855+0000 7fda24ff9700 1 --2- 172.21.15.23:0/3770683249 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fda04056d60 0x7fda04059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:29.859 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.855+0000 7fda24ff9700 1 -- 172.21.15.23:0/3770683249 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fda200729f0 con 0x7fda280a8c70 2020-11-20T11:42:29.860 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.857+0000 7fda27fff700 1 --2- 172.21.15.23:0/3770683249 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fda04056d60 0x7fda04059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:29.861 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.858+0000 7fda27fff700 1 --2- 172.21.15.23:0/3770683249 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fda04056d60 0x7fda04059220 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fda28140fb0 tx=0x7fda18009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:29.862 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:29.859+0000 7fda24ff9700 1 -- 172.21.15.23:0/3770683249 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fda2004b790 con 0x7fda280a8c70 2020-11-20T11:42:30.190 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:30.187+0000 7fda2ede1700 1 -- 172.21.15.23:0/3770683249 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fda28044710 con 0x7fda280a8c70 2020-11-20T11:42:30.191 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:30.188+0000 7fda24ff9700 1 -- 172.21.15.23:0/3770683249 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fda20015070 con 0x7fda280a8c70 2020-11-20T11:42:30.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:30.189+0000 7fda2ede1700 1 -- 172.21.15.23:0/3770683249 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fda04056d60 msgr2=0x7fda04059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:30.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:30.189+0000 7fda2ede1700 1 --2- 172.21.15.23:0/3770683249 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fda04056d60 0x7fda04059220 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fda28140fb0 tx=0x7fda18009040).stop 2020-11-20T11:42:30.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:30.189+0000 7fda2ede1700 1 -- 172.21.15.23:0/3770683249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda280a8c70 msgr2=0x7fda28144d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:30.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:30.189+0000 7fda2ede1700 1 --2- 172.21.15.23:0/3770683249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda280a8c70 0x7fda28144d40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fda20013000 tx=0x7fda20016000).stop 2020-11-20T11:42:30.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:30.189+0000 7fda2ede1700 1 -- 172.21.15.23:0/3770683249 shutdown_connections 2020-11-20T11:42:30.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:30.189+0000 7fda2ede1700 1 --2- 172.21.15.23:0/3770683249 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fda04056d60 0x7fda04059220 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:30.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:30.189+0000 7fda2ede1700 1 --2- 172.21.15.23:0/3770683249 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda280ab480 0x7fda28151c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:30.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:30.189+0000 7fda2ede1700 1 --2- 172.21.15.23:0/3770683249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda280a8c70 0x7fda28144d40 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:30.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:30.189+0000 7fda2ede1700 1 --2- 172.21.15.23:0/3770683249 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fda280aac70 0x7fda280a7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:30.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:30.189+0000 7fda2ede1700 1 -- 172.21.15.23:0/3770683249 >> 172.21.15.23:0/3770683249 conn(0x7fda280678e0 msgr2=0x7fda28068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:30.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:30.190+0000 7fda2ede1700 1 -- 172.21.15.23:0/3770683249 shutdown_connections 2020-11-20T11:42:30.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:30.190+0000 7fda2ede1700 1 -- 172.21.15.23:0/3770683249 wait complete. 2020-11-20T11:42:30.216 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2020-11-20T11:42:33.775 INFO:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:42:33.783 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:42:40.218 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:42:40.218 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:42:40.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.644+0000 7f4bb31a6700 1 Processor -- start 2020-11-20T11:42:40.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.644+0000 7f4bb31a6700 1 -- start start 2020-11-20T11:42:40.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.645+0000 7f4bb31a6700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4bac060a80 0x7f4bac09d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:40.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.645+0000 7f4bb31a6700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4bac09ded0 0x7f4bac0678b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:40.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.645+0000 7f4bb31a6700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bac0a9b40 con 0x7f4bac067da0 2020-11-20T11:42:40.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.645+0000 7f4bb31a6700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bac068230 con 0x7f4bac060a80 2020-11-20T11:42:40.652 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.645+0000 7f4bb31a6700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bac0aacb0 con 0x7f4bac09ded0 2020-11-20T11:42:40.653 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.646+0000 7f4bb1743700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f4bac067da0 0x7f4bac135650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:37142/0 (socket says 172.21.15.23:37142) 2020-11-20T11:42:40.653 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.646+0000 7f4bb1743700 1 -- 172.21.15.23:0/2549213980 learned_addr learned my addr 172.21.15.23:0/2549213980 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:42:40.656 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.646+0000 7f4ba3fff700 1 --2- 172.21.15.23:0/2549213980 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4bac09ded0 0x7f4bac0678b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:40.656 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.647+0000 7f4ba3fff700 1 -- 172.21.15.23:0/2549213980 >> v2:172.21.15.155:3300/0 conn(0x7f4bac060a80 msgr2=0x7f4bac09d9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2020-11-20T11:42:40.656 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.647+0000 7f4ba3fff700 1 --2- 172.21.15.23:0/2549213980 >> v2:172.21.15.155:3300/0 conn(0x7f4bac060a80 0x7f4bac09d9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:40.656 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.647+0000 7f4ba3fff700 1 -- 172.21.15.23:0/2549213980 >> v1:172.21.15.155:6789/0 conn(0x7f4bac067da0 legacy=0x7f4bac135650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:42:40.657 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.647+0000 7f4ba3fff700 1 -- 172.21.15.23:0/2549213980 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bac0a9da0 con 0x7f4bac09ded0 2020-11-20T11:42:40.657 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.647+0000 7f4ba3fff700 1 --2- 172.21.15.23:0/2549213980 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4bac09ded0 0x7f4bac0678b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f4bac068370 tx=0x7f4ba8006e30).ready entity=mon.2 client_cookie=1773d6d10e69651f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:40.657 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.647+0000 7f4ba37fe700 1 -- 172.21.15.23:0/2549213980 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba8010770 con 0x7f4bac09ded0 2020-11-20T11:42:40.657 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.647+0000 7f4ba37fe700 1 -- 172.21.15.23:0/2549213980 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba8000e90 con 0x7f4bac09ded0 2020-11-20T11:42:40.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.647+0000 7f4ba37fe700 1 -- 172.21.15.23:0/2549213980 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba80108d0 con 0x7f4bac09ded0 2020-11-20T11:42:40.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.650+0000 7f4bb31a6700 1 -- 172.21.15.23:0/2549213980 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4bac09ded0 msgr2=0x7f4bac0678b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:40.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.650+0000 7f4bb31a6700 1 --2- 172.21.15.23:0/2549213980 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4bac09ded0 0x7f4bac0678b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f4bac068370 tx=0x7f4ba8006e30).stop 2020-11-20T11:42:40.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.657+0000 7f4bb31a6700 1 -- 172.21.15.23:0/2549213980 shutdown_connections 2020-11-20T11:42:40.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.657+0000 7f4bb31a6700 1 --2- 172.21.15.23:0/2549213980 >> v2:172.21.15.155:3300/0 conn(0x7f4bac060a80 0x7f4bac09d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:40.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.657+0000 7f4bb31a6700 1 --2- 172.21.15.23:0/2549213980 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4bac09ded0 0x7f4bac0678b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:40.661 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.657+0000 7f4bb31a6700 1 -- 172.21.15.23:0/2549213980 >> 172.21.15.23:0/2549213980 conn(0x7f4bac066730 msgr2=0x7f4bac0601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:40.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.658+0000 7f4bb31a6700 1 -- 172.21.15.23:0/2549213980 shutdown_connections 2020-11-20T11:42:40.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.660+0000 7f4bb31a6700 1 -- 172.21.15.23:0/2549213980 wait complete. 2020-11-20T11:42:40.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.662+0000 7f4bb31a6700 1 Processor -- start 2020-11-20T11:42:40.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.662+0000 7f4bb31a6700 1 -- start start 2020-11-20T11:42:40.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.663+0000 7f4bb31a6700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4bac060a80 0x7f4bac138360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:40.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.663+0000 7f4bb31a6700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4bac067da0 0x7f4bac0ad820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:40.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.663+0000 7f4bb31a6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bac09ded0 0x7f4bac124330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:40.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.663+0000 7f4bb31a6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bac0add10 con 0x7f4bac09ded0 2020-11-20T11:42:40.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.663+0000 7f4bb31a6700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bac129de0 con 0x7f4bac067da0 2020-11-20T11:42:40.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.663+0000 7f4bb31a6700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bac12a080 con 0x7f4bac060a80 2020-11-20T11:42:40.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.663+0000 7f4bb1743700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bac09ded0 0x7f4bac124330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:40.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.663+0000 7f4bb1743700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bac09ded0 0x7f4bac124330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44922/0 (socket says 172.21.15.23:44922) 2020-11-20T11:42:40.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.663+0000 7f4bb1743700 1 -- 172.21.15.23:0/1515204846 learned_addr learned my addr 172.21.15.23:0/1515204846 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:42:40.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.663+0000 7f4bb0f42700 1 --2- 172.21.15.23:0/1515204846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4bac060a80 0x7f4bac138360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:40.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.664+0000 7f4bb0f42700 1 -- 172.21.15.23:0/1515204846 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4bac067da0 msgr2=0x7f4bac0ad820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:42:40.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.664+0000 7f4bb0f42700 1 --2- 172.21.15.23:0/1515204846 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4bac067da0 0x7f4bac0ad820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:40.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.664+0000 7f4bb0f42700 1 -- 172.21.15.23:0/1515204846 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bac09ded0 msgr2=0x7f4bac124330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:40.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.664+0000 7f4bb0f42700 1 --2- 172.21.15.23:0/1515204846 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bac09ded0 0x7f4bac124330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:40.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.664+0000 7f4bb0f42700 1 -- 172.21.15.23:0/1515204846 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bac136830 con 0x7f4bac060a80 2020-11-20T11:42:40.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.664+0000 7f4bb1743700 1 --2- 172.21.15.23:0/1515204846 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bac09ded0 0x7f4bac124330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:42:40.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.664+0000 7f4bb0f42700 1 --2- 172.21.15.23:0/1515204846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4bac060a80 0x7f4bac138360 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f4b9c0096e0 tx=0x7f4b9c006d70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:40.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.664+0000 7f4ba0ff9700 1 -- 172.21.15.23:0/1515204846 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c00d4a0 con 0x7f4bac060a80 2020-11-20T11:42:40.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.664+0000 7f4ba0ff9700 1 -- 172.21.15.23:0/1515204846 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b9c00d600 con 0x7f4bac060a80 2020-11-20T11:42:40.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.664+0000 7f4bb31a6700 1 -- 172.21.15.23:0/1515204846 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bac12ab90 con 0x7f4bac060a80 2020-11-20T11:42:40.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.664+0000 7f4bb31a6700 1 -- 172.21.15.23:0/1515204846 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bac13a890 con 0x7f4bac060a80 2020-11-20T11:42:40.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.665+0000 7f4ba0ff9700 1 -- 172.21.15.23:0/1515204846 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c00d9a0 con 0x7f4bac060a80 2020-11-20T11:42:40.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.666+0000 7f4ba0ff9700 1 -- 172.21.15.23:0/1515204846 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4b9c018020 con 0x7f4bac060a80 2020-11-20T11:42:40.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.666+0000 7f4ba0ff9700 1 --2- 172.21.15.23:0/1515204846 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4b84056d60 0x7f4b84059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:40.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.667+0000 7f4ba0ff9700 1 -- 172.21.15.23:0/1515204846 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f4b9c06f6d0 con 0x7f4bac060a80 2020-11-20T11:42:40.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.667+0000 7f4bb31a6700 1 -- 172.21.15.23:0/1515204846 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bac044710 con 0x7f4bac060a80 2020-11-20T11:42:40.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.667+0000 7f4ba3fff700 1 --2- 172.21.15.23:0/1515204846 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4b84056d60 0x7f4b84059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:40.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.667+0000 7f4ba3fff700 1 --2- 172.21.15.23:0/1515204846 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4b84056d60 0x7f4b84059220 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f4bac138b70 tx=0x7f4ba8017000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:40.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:40.677+0000 7f4ba0ff9700 1 -- 172.21.15.23:0/1515204846 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f4b9c0449c0 con 0x7f4bac060a80 2020-11-20T11:42:41.037 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.034+0000 7f4bb31a6700 1 -- 172.21.15.23:0/1515204846 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4bac05c280 con 0x7f4bac060a80 2020-11-20T11:42:41.037 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.034+0000 7f4ba0ff9700 1 -- 172.21.15.23:0/1515204846 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7f4b9c047fe0 con 0x7f4bac060a80 2020-11-20T11:42:41.038 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:42:41.040 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.036+0000 7f4bb31a6700 1 -- 172.21.15.23:0/1515204846 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4b84056d60 msgr2=0x7f4b84059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:41.040 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.036+0000 7f4bb31a6700 1 --2- 172.21.15.23:0/1515204846 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4b84056d60 0x7f4b84059220 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f4bac138b70 tx=0x7f4ba8017000).stop 2020-11-20T11:42:41.040 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.036+0000 7f4bb31a6700 1 -- 172.21.15.23:0/1515204846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4bac060a80 msgr2=0x7f4bac138360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:41.041 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.036+0000 7f4bb31a6700 1 --2- 172.21.15.23:0/1515204846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4bac060a80 0x7f4bac138360 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f4b9c0096e0 tx=0x7f4b9c006d70).stop 2020-11-20T11:42:41.041 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.036+0000 7f4bb31a6700 1 -- 172.21.15.23:0/1515204846 shutdown_connections 2020-11-20T11:42:41.041 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.036+0000 7f4bb31a6700 1 --2- 172.21.15.23:0/1515204846 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bac09ded0 0x7f4bac124330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:41.041 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.036+0000 7f4bb31a6700 1 --2- 172.21.15.23:0/1515204846 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4b84056d60 0x7f4b84059220 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:41.041 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.036+0000 7f4bb31a6700 1 --2- 172.21.15.23:0/1515204846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4bac060a80 0x7f4bac138360 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:41.042 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.036+0000 7f4bb31a6700 1 --2- 172.21.15.23:0/1515204846 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4bac067da0 0x7f4bac0ad820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:41.042 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.036+0000 7f4bb31a6700 1 -- 172.21.15.23:0/1515204846 >> 172.21.15.23:0/1515204846 conn(0x7f4bac066730 msgr2=0x7f4bac068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:41.042 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.036+0000 7f4bb31a6700 1 -- 172.21.15.23:0/1515204846 shutdown_connections 2020-11-20T11:42:41.042 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.036+0000 7f4bb31a6700 1 -- 172.21.15.23:0/1515204846 wait complete. 2020-11-20T11:42:41.056 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:42:41.057 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2020-11-20T11:42:41.057 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:42:41.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.227+0000 7f4cb359e700 1 Processor -- start 2020-11-20T11:42:41.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.227+0000 7f4cb359e700 1 -- start start 2020-11-20T11:42:41.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.228+0000 7f4cb359e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4cac007f50 0x7f4cac037550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:41.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.228+0000 7f4cb359e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4cac03b300 0x7f4cac0d2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:41.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.228+0000 7f4cb359e700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cac037ec0 con 0x7f4cac03a770 2020-11-20T11:42:41.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.228+0000 7f4cb359e700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cac038160 con 0x7f4cac007f50 2020-11-20T11:42:41.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.228+0000 7f4cb359e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac038400 con 0x7f4cac03b300 2020-11-20T11:42:41.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.228+0000 7f4cb259c700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4cac007f50 0x7f4cac037550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:41.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.228+0000 7f4cb259c700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4cac007f50 0x7f4cac037550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:37824/0 (socket says 172.21.15.23:37824) 2020-11-20T11:42:41.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.228+0000 7f4cb259c700 1 -- 172.21.15.23:0/3761473195 learned_addr learned my addr 172.21.15.23:0/3761473195 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:42:41.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.229+0000 7f4cb2d9d700 1 --2- 172.21.15.23:0/3761473195 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4cac03b300 0x7f4cac0d2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:41.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.229+0000 7f4cb259c700 1 -- 172.21.15.23:0/3761473195 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4cac03b300 msgr2=0x7f4cac0d2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:42:41.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.229+0000 7f4cb259c700 1 --2- 172.21.15.23:0/3761473195 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4cac03b300 0x7f4cac0d2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:41.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.229+0000 7f4cb259c700 1 -- 172.21.15.23:0/3761473195 >> v1:172.21.15.155:6789/0 conn(0x7f4cac03a770 legacy=0x7f4cac03abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:41.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.229+0000 7f4cb259c700 1 -- 172.21.15.23:0/3761473195 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cac008330 con 0x7f4cac007f50 2020-11-20T11:42:41.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.229+0000 7f4cb259c700 1 --2- 172.21.15.23:0/3761473195 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4cac007f50 0x7f4cac037550 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f4ca4009a10 tx=0x7f4ca4006ef0).ready entity=mon.1 client_cookie=2a151817756dabc2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:41.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.233+0000 7f4cb159a700 1 -- 172.21.15.23:0/3761473195 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca400b7a0 con 0x7f4cac007f50 2020-11-20T11:42:41.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.234+0000 7f4cb159a700 1 -- 172.21.15.23:0/3761473195 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ca4000e90 con 0x7f4cac007f50 2020-11-20T11:42:41.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.234+0000 7f4cb159a700 1 -- 172.21.15.23:0/3761473195 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca400bb10 con 0x7f4cac007f50 2020-11-20T11:42:41.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.234+0000 7f4cb359e700 1 -- 172.21.15.23:0/3761473195 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4cac007f50 msgr2=0x7f4cac037550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:41.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.234+0000 7f4cb359e700 1 --2- 172.21.15.23:0/3761473195 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4cac007f50 0x7f4cac037550 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f4ca4009a10 tx=0x7f4ca4006ef0).stop 2020-11-20T11:42:41.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.234+0000 7f4cb359e700 1 -- 172.21.15.23:0/3761473195 shutdown_connections 2020-11-20T11:42:41.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.234+0000 7f4cb359e700 1 --2- 172.21.15.23:0/3761473195 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4cac007f50 0x7f4cac037550 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:41.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.234+0000 7f4cb359e700 1 --2- 172.21.15.23:0/3761473195 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4cac03b300 0x7f4cac0d2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:41.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.234+0000 7f4cb359e700 1 -- 172.21.15.23:0/3761473195 >> 172.21.15.23:0/3761473195 conn(0x7f4cac00cd80 msgr2=0x7f4cac007680 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:41.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.235+0000 7f4cb359e700 1 -- 172.21.15.23:0/3761473195 shutdown_connections 2020-11-20T11:42:41.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.235+0000 7f4cb359e700 1 -- 172.21.15.23:0/3761473195 wait complete. 2020-11-20T11:42:41.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.236+0000 7f4cb359e700 1 Processor -- start 2020-11-20T11:42:41.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.236+0000 7f4cb359e700 1 -- start start 2020-11-20T11:42:41.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.237+0000 7f4cb359e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4cac007f50 0x7f4cac0cc350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:41.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.237+0000 7f4cb359e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cac03a770 0x7f4cac0da4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:41.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.237+0000 7f4cb359e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4cac03b300 0x7f4cac0da990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:41.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.237+0000 7f4cb359e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac0c9da0 con 0x7f4cac03a770 2020-11-20T11:42:41.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.237+0000 7f4cb359e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac0c9fd0 con 0x7f4cac03b300 2020-11-20T11:42:41.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.237+0000 7f4cb359e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac0ca270 con 0x7f4cac007f50 2020-11-20T11:42:41.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.237+0000 7f4cb1d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cac03a770 0x7f4cac0da4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:41.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.237+0000 7f4cb2d9d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4cac03b300 0x7f4cac0da990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:41.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.237+0000 7f4cb2d9d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4cac03b300 0x7f4cac0da990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:37830/0 (socket says 172.21.15.23:37830) 2020-11-20T11:42:41.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.237+0000 7f4cb1d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cac03a770 0x7f4cac0da4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44932/0 (socket says 172.21.15.23:44932) 2020-11-20T11:42:41.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.237+0000 7f4cb2d9d700 1 -- 172.21.15.23:0/3035542790 learned_addr learned my addr 172.21.15.23:0/3035542790 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:42:41.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.237+0000 7f4cb259c700 1 --2- 172.21.15.23:0/3035542790 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4cac007f50 0x7f4cac0cc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:41.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.238+0000 7f4cb259c700 1 -- 172.21.15.23:0/3035542790 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4cac03b300 msgr2=0x7f4cac0da990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:41.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.238+0000 7f4cb259c700 1 --2- 172.21.15.23:0/3035542790 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4cac03b300 0x7f4cac0da990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:41.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.238+0000 7f4cb259c700 1 -- 172.21.15.23:0/3035542790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cac03a770 msgr2=0x7f4cac0da4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:41.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.238+0000 7f4cb259c700 1 --2- 172.21.15.23:0/3035542790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cac03a770 0x7f4cac0da4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:41.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.238+0000 7f4cb259c700 1 -- 172.21.15.23:0/3035542790 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cac0ca710 con 0x7f4cac007f50 2020-11-20T11:42:41.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.238+0000 7f4cb2d9d700 1 --2- 172.21.15.23:0/3035542790 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4cac03b300 0x7f4cac0da990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:42:41.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.238+0000 7f4cb1d9b700 1 --2- 172.21.15.23:0/3035542790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cac03a770 0x7f4cac0da4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:42:41.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.238+0000 7f4cb259c700 1 --2- 172.21.15.23:0/3035542790 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4cac007f50 0x7f4cac0cc350 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f4cac036bb0 tx=0x7f4ca4016000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:41.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.239+0000 7f4ca27fc700 1 -- 172.21.15.23:0/3035542790 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca40068d0 con 0x7f4cac007f50 2020-11-20T11:42:41.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.239+0000 7f4ca27fc700 1 -- 172.21.15.23:0/3035542790 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ca4011a70 con 0x7f4cac007f50 2020-11-20T11:42:41.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.239+0000 7f4cb359e700 1 -- 172.21.15.23:0/3035542790 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cac0ca900 con 0x7f4cac007f50 2020-11-20T11:42:41.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.239+0000 7f4ca27fc700 1 -- 172.21.15.23:0/3035542790 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca4011e10 con 0x7f4cac007f50 2020-11-20T11:42:41.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.239+0000 7f4cb359e700 1 -- 172.21.15.23:0/3035542790 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cac0cad70 con 0x7f4cac007f50 2020-11-20T11:42:41.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.240+0000 7f4cb359e700 1 --2- 172.21.15.23:0/3035542790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cac03d4d0 0x7f4cac03d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:41.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.240+0000 7f4cb359e700 1 -- 172.21.15.23:0/3035542790 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac0092d0 con 0x7f4cac03d4d0 2020-11-20T11:42:41.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.240+0000 7f4cb1d9b700 1 --2- 172.21.15.23:0/3035542790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cac03d4d0 0x7f4cac03d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:41.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.240+0000 7f4ca27fc700 1 -- 172.21.15.23:0/3035542790 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4ca4017070 con 0x7f4cac007f50 2020-11-20T11:42:41.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.240+0000 7f4cb1d9b700 1 -- 172.21.15.23:0/3035542790 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4cac0ca430 con 0x7f4cac03d4d0 2020-11-20T11:42:41.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.240+0000 7f4cb1d9b700 1 --2- 172.21.15.23:0/3035542790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cac03d4d0 0x7f4cac03d8b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f4cb405f8e0 tx=0x7f4cb405e740).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:41.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.240+0000 7f4ca27fc700 1 --2- 172.21.15.23:0/3035542790 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4c90056de0 0x7f4c900592a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:41.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.240+0000 7f4ca27fc700 1 -- 172.21.15.23:0/3035542790 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f4ca4082540 con 0x7f4cac007f50 2020-11-20T11:42:41.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.241+0000 7f4ca27fc700 1 -- 172.21.15.23:0/3035542790 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb4084070 con 0x7f4cac03d4d0 2020-11-20T11:42:41.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.241+0000 7f4ca27fc700 1 -- 172.21.15.23:0/3035542790 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f4cb40685f0 con 0x7f4cac03d4d0 2020-11-20T11:42:41.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.241+0000 7f4ca27fc700 1 -- 172.21.15.23:0/3035542790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cac03d4d0 msgr2=0x7f4cac03d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:41.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.241+0000 7f4ca27fc700 1 --2- 172.21.15.23:0/3035542790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cac03d4d0 0x7f4cac03d8b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f4cb405f8e0 tx=0x7f4cb405e740).stop 2020-11-20T11:42:41.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.241+0000 7f4cb2d9d700 1 --2- 172.21.15.23:0/3035542790 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4c90056de0 0x7f4c900592a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:41.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.241+0000 7f4ca27fc700 1 -- 172.21.15.23:0/3035542790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cac03d4d0 msgr2=0x7f4cac03d8b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:42:41.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.241+0000 7f4ca27fc700 1 --2- 172.21.15.23:0/3035542790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cac03d4d0 0x7f4cac03d8b0 secure :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0x7f4cb405f8e0 tx=0x7f4cb405e740).stop 2020-11-20T11:42:41.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.241+0000 7f4cb2d9d700 1 --2- 172.21.15.23:0/3035542790 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4c90056de0 0x7f4c900592a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f4cac0cc040 tx=0x7f4ca8008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:41.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.251+0000 7f4cb359e700 1 --2- 172.21.15.23:0/3035542790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cac03dda0 0x7f4cac0dd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:41.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.251+0000 7f4cb359e700 1 -- 172.21.15.23:0/3035542790 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac0ca5d0 con 0x7f4cac03dda0 2020-11-20T11:42:41.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.251+0000 7f4cb1d9b700 1 --2- 172.21.15.23:0/3035542790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cac03dda0 0x7f4cac0dd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:41.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.252+0000 7f4cb1d9b700 1 -- 172.21.15.23:0/3035542790 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4cac005440 con 0x7f4cac03dda0 2020-11-20T11:42:41.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.252+0000 7f4cb1d9b700 1 --2- 172.21.15.23:0/3035542790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cac03dda0 0x7f4cac0dd4f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f4cb406ec20 tx=0x7f4cb4065000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:41.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.252+0000 7f4ca27fc700 1 -- 172.21.15.23:0/3035542790 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb4087780 con 0x7f4cac03dda0 2020-11-20T11:42:41.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.253+0000 7f4ca27fc700 1 -- 172.21.15.23:0/3035542790 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4786 (secure 0 0 0) 0x7f4cb40878e0 con 0x7f4cac03dda0 2020-11-20T11:42:41.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.253+0000 7f4ca27fc700 1 -- 172.21.15.23:0/3035542790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cac03dda0 msgr2=0x7f4cac0dd4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:41.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.253+0000 7f4ca27fc700 1 --2- 172.21.15.23:0/3035542790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cac03dda0 0x7f4cac0dd4f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f4cb406ec20 tx=0x7f4cb4065000).stop 2020-11-20T11:42:41.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.253+0000 7f4ca27fc700 1 -- 172.21.15.23:0/3035542790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cac03dda0 msgr2=0x7f4cac0dd4f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:42:41.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.253+0000 7f4ca27fc700 1 --2- 172.21.15.23:0/3035542790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4cac03dda0 0x7f4cac0dd4f0 secure :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0x7f4cb406ec20 tx=0x7f4cb4065000).stop 2020-11-20T11:42:41.258 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:42:41.258 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:42:41.258 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:42:41.258 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:42:41.259 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 92, 2020-11-20T11:42:41.259 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:42:41.259 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:42:41.259 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:42:41.259 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:42:41.260 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:41.260 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 15, 2020-11-20T11:42:41.260 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:42:41.260 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:42:41.260 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:42:41.261 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:42:41.261 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:42:41.261 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:42:41.261 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:42:41.262 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:42:41.262 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:42:41.262 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:42:41.262 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:42:41.262 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:41.262 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:42:41.263 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:42:41.263 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:42:41.263 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:42:41.263 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:42:41.263 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:42:41.264 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:42:41.264 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:42:41.264 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:42:41.264 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:42:41.264 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:41.265 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.265 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:42:41.265 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:42:41.265 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:42:41.265 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:42:41.266 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:42:41.266 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:42:41.266 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:42:41.266 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:42:41.266 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:42:41.267 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:42:41.267 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:42:41.267 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:42:41.267 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:42:41.267 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:42:41.268 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:42:41.268 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:42:41.268 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:42:41.268 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:42:41.268 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:42:41.269 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:42:41.269 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:42:41.269 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:42:41.269 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:42:41.269 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:41.270 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:42:41.270 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.270 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:42:41.270 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.270 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:42:41.271 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:42:41.271 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:42:41.271 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:42:41.271 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.272 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:42:41.272 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:42:41.272 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:41.272 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.273 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.273 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:42:41.273 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:42:41.273 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:41.274 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:41.274 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:41.274 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.274 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:42:41.274 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:42:41.275 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:42:41.275 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:42:41.275 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:42:41.275 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.275 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.276 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:42:41.276 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:42:41.276 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:42:41.276 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:42:41.276 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.277 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:42:41.277 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:42:41.277 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:41.277 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.278 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.278 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:42:41.278 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:42:41.278 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:41.279 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:41.279 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:41.279 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.280 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:42:41.280 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:42:41.280 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:42:41.280 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:42:41.280 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:42:41.281 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.281 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.281 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:42:41.281 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:42:41.281 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:42:41.282 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:42:41.282 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.282 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:42:41.283 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:42:41.283 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:41.283 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.283 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.283 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:42:41.284 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:42:41.284 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:41.284 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:41.284 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:41.284 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.285 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:42:41.285 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:42:41.285 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:42:41.285 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:42:41.286 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:42:41.286 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:41.286 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:41.286 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.286 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:42:41.287 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:42:41.287 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.287 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:41.287 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:41.287 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:42:41.288 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:41.288 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:41.288 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:42:41.288 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.289 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:41.289 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:41.289 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:42:41.289 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:41.290 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:41.290 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:42:41.290 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.290 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:41.291 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:41.291 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 8 2020-11-20T11:42:41.291 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:41.291 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:41.291 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:42:41.292 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.292 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:41.292 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:41.292 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 4 2020-11-20T11:42:41.292 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:41.293 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:41.293 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:42:41.293 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.293 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:41.293 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:41.294 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:42:41.294 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:41.294 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:41.294 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.294 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:42:41.295 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:42:41.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.267+0000 7f4cb359e700 1 -- 172.21.15.23:0/3035542790 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4c90056de0 msgr2=0x7f4c900592a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:41.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.267+0000 7f4cb359e700 1 --2- 172.21.15.23:0/3035542790 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4c90056de0 0x7f4c900592a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f4cac0cc040 tx=0x7f4ca8008040).stop 2020-11-20T11:42:41.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.267+0000 7f4cb359e700 1 -- 172.21.15.23:0/3035542790 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4cac007f50 msgr2=0x7f4cac0cc350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:41.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.267+0000 7f4cb359e700 1 --2- 172.21.15.23:0/3035542790 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4cac007f50 0x7f4cac0cc350 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f4cac036bb0 tx=0x7f4ca4016000).stop 2020-11-20T11:42:41.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.267+0000 7f4cb2d9d700 1 -- 172.21.15.23:0/3035542790 reap_dead start 2020-11-20T11:42:41.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.267+0000 7f4cb2d9d700 1 -- 172.21.15.23:0/3035542790 reap_dead start 2020-11-20T11:42:41.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.267+0000 7f4cb359e700 1 -- 172.21.15.23:0/3035542790 shutdown_connections 2020-11-20T11:42:41.298 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.267+0000 7f4cb359e700 1 -- 172.21.15.23:0/3035542790 >> 172.21.15.23:0/3035542790 conn(0x7f4cac00cd80 msgr2=0x7f4cac0c79b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:41.298 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.267+0000 7f4cb359e700 1 -- 172.21.15.23:0/3035542790 shutdown_connections 2020-11-20T11:42:41.298 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.267+0000 7f4cb359e700 1 -- 172.21.15.23:0/3035542790 wait complete. 2020-11-20T11:42:41.299 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:42:41.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.485+0000 7f8a3a81b700 1 Processor -- start 2020-11-20T11:42:41.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.485+0000 7f8a3a81b700 1 -- start start 2020-11-20T11:42:41.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.486+0000 7f8a3a81b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8a34061b90 0x7f8a340a92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:41.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.486+0000 7f8a3a81b700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f8a340a97d0 0x7f8a3413b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:41.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.486+0000 7f8a3a81b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8a3413b700 0x7f8a3413bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:41.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.486+0000 7f8a3a81b700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a3413c0a0 con 0x7f8a3413b700 2020-11-20T11:42:41.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.486+0000 7f8a3a81b700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a340a9c50 con 0x7f8a340a97d0 2020-11-20T11:42:41.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.486+0000 7f8a3a81b700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a34067190 con 0x7f8a34061b90 2020-11-20T11:42:41.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.486+0000 7f8a38db8700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8a3413b700 0x7f8a3413bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:41.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.486+0000 7f8a337fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f8a340a97d0 0x7f8a3413b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:41.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.486+0000 7f8a38db8700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8a3413b700 0x7f8a3413bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44948/0 (socket says 172.21.15.23:44948) 2020-11-20T11:42:41.515 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.486+0000 7f8a38db8700 1 -- 172.21.15.23:0/74532991 learned_addr learned my addr 172.21.15.23:0/74532991 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:42:41.515 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.486+0000 7f8a33fff700 1 --2- 172.21.15.23:0/74532991 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8a34061b90 0x7f8a340a92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:41.515 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.486+0000 7f8a38db8700 1 -- 172.21.15.23:0/74532991 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8a34061b90 msgr2=0x7f8a340a92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:42:41.515 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.486+0000 7f8a38db8700 1 --2- 172.21.15.23:0/74532991 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8a34061b90 0x7f8a340a92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:41.516 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.486+0000 7f8a38db8700 1 -- 172.21.15.23:0/74532991 >> v2:172.21.15.155:3300/0 conn(0x7f8a340a97d0 msgr2=0x7f8a3413b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:42:41.516 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.486+0000 7f8a38db8700 1 --2- 172.21.15.23:0/74532991 >> v2:172.21.15.155:3300/0 conn(0x7f8a340a97d0 0x7f8a3413b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:41.516 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.486+0000 7f8a38db8700 1 -- 172.21.15.23:0/74532991 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a34067430 con 0x7f8a3413b700 2020-11-20T11:42:41.516 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.486+0000 7f8a38db8700 1 --2- 172.21.15.23:0/74532991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a3413b700 0x7f8a3413bbb0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f8a34069dd0 tx=0x7f8a28006ef0).ready entity=mon.0 client_cookie=5885839768eff1b1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:41.516 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.487+0000 7f8a32ffd700 1 -- 172.21.15.23:0/74532991 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2800b680 con 0x7f8a3413b700 2020-11-20T11:42:41.517 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.487+0000 7f8a32ffd700 1 -- 172.21.15.23:0/74532991 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a28000e90 con 0x7f8a3413b700 2020-11-20T11:42:41.517 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.487+0000 7f8a32ffd700 1 -- 172.21.15.23:0/74532991 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2800b9f0 con 0x7f8a3413b700 2020-11-20T11:42:41.517 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.487+0000 7f8a3a81b700 1 -- 172.21.15.23:0/74532991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a3413b700 msgr2=0x7f8a3413bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:41.517 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.487+0000 7f8a3a81b700 1 --2- 172.21.15.23:0/74532991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a3413b700 0x7f8a3413bbb0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f8a34069dd0 tx=0x7f8a28006ef0).stop 2020-11-20T11:42:41.518 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.487+0000 7f8a3a81b700 1 -- 172.21.15.23:0/74532991 shutdown_connections 2020-11-20T11:42:41.518 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.487+0000 7f8a3a81b700 1 --2- 172.21.15.23:0/74532991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a3413b700 0x7f8a3413bbb0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:41.518 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.487+0000 7f8a3a81b700 1 --2- 172.21.15.23:0/74532991 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8a34061b90 0x7f8a340a92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:41.518 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.487+0000 7f8a3a81b700 1 --2- 172.21.15.23:0/74532991 >> v2:172.21.15.155:3300/0 conn(0x7f8a340a97d0 0x7f8a3413b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:41.518 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.487+0000 7f8a3a81b700 1 -- 172.21.15.23:0/74532991 >> 172.21.15.23:0/74532991 conn(0x7f8a34063640 msgr2=0x7f8a340612c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:41.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.487+0000 7f8a3a81b700 1 -- 172.21.15.23:0/74532991 shutdown_connections 2020-11-20T11:42:41.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.487+0000 7f8a3a81b700 1 -- 172.21.15.23:0/74532991 wait complete. 2020-11-20T11:42:41.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.488+0000 7f8a3a81b700 1 Processor -- start 2020-11-20T11:42:41.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.489+0000 7f8a3a81b700 1 -- start start 2020-11-20T11:42:41.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.489+0000 7f8a3a81b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a34061b90 0x7f8a340a3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:41.520 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.489+0000 7f8a3a81b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8a340a97d0 0x7f8a3414aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:41.520 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.489+0000 7f8a3a81b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a3413b700 0x7f8a34134320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:41.520 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.489+0000 7f8a3a81b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a340a54f0 con 0x7f8a3413b700 2020-11-20T11:42:41.520 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.489+0000 7f8a3a81b700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a340a2e60 con 0x7f8a34061b90 2020-11-20T11:42:41.520 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.489+0000 7f8a3a81b700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3414a320 con 0x7f8a340a97d0 2020-11-20T11:42:41.521 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.489+0000 7f8a38db8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a3413b700 0x7f8a34134320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:41.521 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.489+0000 7f8a38db8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a3413b700 0x7f8a34134320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44954/0 (socket says 172.21.15.23:44954) 2020-11-20T11:42:41.521 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.489+0000 7f8a38db8700 1 -- 172.21.15.23:0/3833606186 learned_addr learned my addr 172.21.15.23:0/3833606186 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:42:41.521 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.489+0000 7f8a337fe700 1 --2- 172.21.15.23:0/3833606186 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8a340a97d0 0x7f8a3414aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:41.521 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.489+0000 7f8a33fff700 1 --2- 172.21.15.23:0/3833606186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a34061b90 0x7f8a340a3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:41.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.490+0000 7f8a38db8700 1 -- 172.21.15.23:0/3833606186 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8a340a97d0 msgr2=0x7f8a3414aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:41.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.490+0000 7f8a38db8700 1 --2- 172.21.15.23:0/3833606186 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8a340a97d0 0x7f8a3414aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:41.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.490+0000 7f8a38db8700 1 -- 172.21.15.23:0/3833606186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a34061b90 msgr2=0x7f8a340a3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:41.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.490+0000 7f8a38db8700 1 --2- 172.21.15.23:0/3833606186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a34061b90 0x7f8a340a3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:41.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.490+0000 7f8a38db8700 1 -- 172.21.15.23:0/3833606186 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a3414a5c0 con 0x7f8a3413b700 2020-11-20T11:42:41.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.490+0000 7f8a38db8700 1 --2- 172.21.15.23:0/3833606186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a3413b700 0x7f8a34134320 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f8a28009fd0 tx=0x7f8a28016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:41.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.490+0000 7f8a13fff700 1 -- 172.21.15.23:0/3833606186 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a28011bc0 con 0x7f8a3413b700 2020-11-20T11:42:41.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.490+0000 7f8a13fff700 1 -- 172.21.15.23:0/3833606186 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a28011d20 con 0x7f8a3413b700 2020-11-20T11:42:41.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.490+0000 7f8a13fff700 1 -- 172.21.15.23:0/3833606186 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2801d660 con 0x7f8a3413b700 2020-11-20T11:42:41.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.490+0000 7f8a3a81b700 1 -- 172.21.15.23:0/3833606186 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a3414af50 con 0x7f8a3413b700 2020-11-20T11:42:41.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.490+0000 7f8a3a81b700 1 -- 172.21.15.23:0/3833606186 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a34134810 con 0x7f8a3413b700 2020-11-20T11:42:41.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.491+0000 7f8a3a81b700 1 --2- 172.21.15.23:0/3833606186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a34134a60 0x7f8a340a2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:41.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.491+0000 7f8a3a81b700 1 -- 172.21.15.23:0/3833606186 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a34061fd0 con 0x7f8a34134a60 2020-11-20T11:42:41.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.491+0000 7f8a33fff700 1 --2- 172.21.15.23:0/3833606186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a34134a60 0x7f8a340a2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:41.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.492+0000 7f8a13fff700 1 -- 172.21.15.23:0/3833606186 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8a28017040 con 0x7f8a3413b700 2020-11-20T11:42:41.525 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.492+0000 7f8a33fff700 1 -- 172.21.15.23:0/3833606186 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8a3414a750 con 0x7f8a34134a60 2020-11-20T11:42:41.525 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.492+0000 7f8a33fff700 1 --2- 172.21.15.23:0/3833606186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a34134a60 0x7f8a340a2270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f8a2c011fa0 tx=0x7f8a2c009220).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:41.526 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.492+0000 7f8a13fff700 1 --2- 172.21.15.23:0/3833606186 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8a0c056d60 0x7f8a0c059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:41.526 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.492+0000 7f8a13fff700 1 -- 172.21.15.23:0/3833606186 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f8a28071db0 con 0x7f8a3413b700 2020-11-20T11:42:41.526 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.492+0000 7f8a13fff700 1 -- 172.21.15.23:0/3833606186 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c014440 con 0x7f8a34134a60 2020-11-20T11:42:41.526 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.492+0000 7f8a337fe700 1 --2- 172.21.15.23:0/3833606186 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8a0c056d60 0x7f8a0c059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:41.526 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.492+0000 7f8a13fff700 1 -- 172.21.15.23:0/3833606186 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f8a2c0145a0 con 0x7f8a34134a60 2020-11-20T11:42:41.527 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.492+0000 7f8a13fff700 1 -- 172.21.15.23:0/3833606186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a34134a60 msgr2=0x7f8a340a2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:41.527 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.492+0000 7f8a13fff700 1 --2- 172.21.15.23:0/3833606186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a34134a60 0x7f8a340a2270 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f8a2c011fa0 tx=0x7f8a2c009220).stop 2020-11-20T11:42:41.527 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.492+0000 7f8a13fff700 1 -- 172.21.15.23:0/3833606186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a34134a60 msgr2=0x7f8a340a2270 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:42:41.527 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.492+0000 7f8a13fff700 1 --2- 172.21.15.23:0/3833606186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a34134a60 0x7f8a340a2270 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0x7f8a2c011fa0 tx=0x7f8a2c009220).stop 2020-11-20T11:42:41.527 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.493+0000 7f8a337fe700 1 --2- 172.21.15.23:0/3833606186 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8a0c056d60 0x7f8a0c059220 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f8a34061310 tx=0x7f8a24006e30).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:41.528 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.501+0000 7f8a3a81b700 1 --2- 172.21.15.23:0/3833606186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a3405c070 0x7f8a340a2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:41.528 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.501+0000 7f8a3a81b700 1 -- 172.21.15.23:0/3833606186 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a340a2c50 con 0x7f8a3405c070 2020-11-20T11:42:41.528 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.501+0000 7f8a33fff700 1 --2- 172.21.15.23:0/3833606186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a3405c070 0x7f8a340a2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:41.528 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.502+0000 7f8a33fff700 1 -- 172.21.15.23:0/3833606186 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8a340ab870 con 0x7f8a3405c070 2020-11-20T11:42:41.528 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.502+0000 7f8a33fff700 1 --2- 172.21.15.23:0/3833606186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a3405c070 0x7f8a340a2760 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f8a2c019000 tx=0x7f8a2c01c000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:41.529 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.502+0000 7f8a13fff700 1 -- 172.21.15.23:0/3833606186 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c01dd50 con 0x7f8a3405c070 2020-11-20T11:42:41.529 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.502+0000 7f8a13fff700 1 -- 172.21.15.23:0/3833606186 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4274 (secure 0 0 0) 0x7f8a2c00fd00 con 0x7f8a3405c070 2020-11-20T11:42:41.529 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.502+0000 7f8a13fff700 1 -- 172.21.15.23:0/3833606186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a3405c070 msgr2=0x7f8a340a2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:41.529 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.502+0000 7f8a13fff700 1 --2- 172.21.15.23:0/3833606186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a3405c070 0x7f8a340a2760 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f8a2c019000 tx=0x7f8a2c01c000).stop 2020-11-20T11:42:41.530 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.502+0000 7f8a13fff700 1 -- 172.21.15.23:0/3833606186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a3405c070 msgr2=0x7f8a340a2760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:42:41.530 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.502+0000 7f8a13fff700 1 --2- 172.21.15.23:0/3833606186 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8a3405c070 0x7f8a340a2760 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:41.530 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.503+0000 7f8a3a81b700 1 -- 172.21.15.23:0/3833606186 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8a0c056d60 msgr2=0x7f8a0c059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:41.530 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.503+0000 7f8a3a81b700 1 --2- 172.21.15.23:0/3833606186 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8a0c056d60 0x7f8a0c059220 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f8a34061310 tx=0x7f8a24006e30).stop 2020-11-20T11:42:41.530 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.503+0000 7f8a3a81b700 1 -- 172.21.15.23:0/3833606186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a3413b700 msgr2=0x7f8a34134320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:41.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.503+0000 7f8a3a81b700 1 --2- 172.21.15.23:0/3833606186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a3413b700 0x7f8a34134320 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f8a28009fd0 tx=0x7f8a28016000).stop 2020-11-20T11:42:41.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.503+0000 7f8a38db8700 1 -- 172.21.15.23:0/3833606186 reap_dead start 2020-11-20T11:42:41.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.503+0000 7f8a38db8700 1 -- 172.21.15.23:0/3833606186 reap_dead start 2020-11-20T11:42:41.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.503+0000 7f8a3a81b700 1 -- 172.21.15.23:0/3833606186 shutdown_connections 2020-11-20T11:42:41.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.503+0000 7f8a3a81b700 1 -- 172.21.15.23:0/3833606186 >> 172.21.15.23:0/3833606186 conn(0x7f8a34063640 msgr2=0x7f8a34134ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:41.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.503+0000 7f8a3a81b700 1 -- 172.21.15.23:0/3833606186 shutdown_connections 2020-11-20T11:42:41.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.503+0000 7f8a3a81b700 1 -- 172.21.15.23:0/3833606186 wait complete. 2020-11-20T11:42:41.532 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:42:41.532 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:42:41.532 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:42:41.533 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:42:41.533 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 92, 2020-11-20T11:42:41.533 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:42:41.533 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:42:41.534 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:42:41.534 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:42:41.534 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:41.534 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 16, 2020-11-20T11:42:41.534 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:42:41.535 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:42:41.535 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:42:41.535 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:42:41.535 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:42:41.535 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:42:41.536 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:42:41.536 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:42:41.536 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:42:41.536 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:42:41.536 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:42:41.537 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:41.537 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:42:41.537 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:42:41.537 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:42:41.537 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:42:41.538 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:42:41.538 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:42:41.538 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:42:41.538 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:42:41.538 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:42:41.539 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:42:41.539 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:41.539 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.539 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:42:41.539 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:42:41.540 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:42:41.540 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:42:41.540 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:42:41.540 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:42:41.540 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:42:41.541 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:42:41.541 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:42:41.541 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:42:41.541 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:42:41.541 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:42:41.542 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:42:41.542 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:42:41.542 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:42:41.542 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:42:41.542 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:42:41.543 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:42:41.543 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:42:41.543 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:42:41.543 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:42:41.543 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:42:41.544 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:42:41.544 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:41.544 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:42:41.544 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.544 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:42:41.544 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.545 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:42:41.545 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:42:41.545 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:42:41.545 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:42:41.545 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.546 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:42:41.546 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:42:41.546 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:41.546 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.546 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.547 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:42:41.547 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:42:41.547 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:41.547 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:41.547 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:41.548 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.548 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:42:41.548 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:42:41.548 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:42:41.548 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:42:41.549 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:42:41.549 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.549 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.549 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:42:41.549 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:42:41.550 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:42:41.550 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:42:41.550 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.550 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:42:41.550 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:42:41.551 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:41.551 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.551 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.551 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:42:41.551 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:42:41.552 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:41.552 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:41.552 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:41.552 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.552 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:42:41.553 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:42:41.553 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:42:41.553 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:42:41.553 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:42:41.553 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.554 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.554 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:42:41.554 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:42:41.554 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:42:41.554 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:42:41.555 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.555 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:42:41.555 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:42:41.555 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:41.555 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.556 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.556 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:42:41.556 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:42:41.556 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:41.556 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:41.557 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:41.557 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.557 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:42:41.557 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:42:41.557 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:42:41.558 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:42:41.558 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:42:41.558 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:41.558 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:41.558 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.559 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:42:41.559 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:42:41.559 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.559 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:41.559 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:41.560 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:42:41.560 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:41.560 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:41.560 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:42:41.560 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.561 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:41.561 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:41.561 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:42:41.561 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:41.561 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:41.562 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.562 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:42:41.562 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:42:41.563 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:42:41.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.759+0000 7faa4fffe700 1 Processor -- start 2020-11-20T11:42:41.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.759+0000 7faa4fffe700 1 -- start start 2020-11-20T11:42:41.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.760+0000 7faa4fffe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7faa480619d0 0x7faa480a9740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:41.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.760+0000 7faa4fffe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7faa48061db0 0x7faa4813c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:41.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.760+0000 7faa4fffe700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa480a5690 con 0x7faa4813c790 2020-11-20T11:42:41.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.760+0000 7faa4fffe700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa480a5eb0 con 0x7faa48061db0 2020-11-20T11:42:41.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.760+0000 7faa4fffe700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa480a6150 con 0x7faa480619d0 2020-11-20T11:42:41.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.763+0000 7faa4d599700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7faa48061db0 0x7faa4813c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:41.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.763+0000 7faa4d599700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7faa48061db0 0x7faa4813c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44962/0 (socket says 172.21.15.23:44962) 2020-11-20T11:42:41.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.763+0000 7faa4d599700 1 -- 172.21.15.23:0/2144141304 learned_addr learned my addr 172.21.15.23:0/2144141304 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:42:41.769 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.763+0000 7faa4d599700 1 -- 172.21.15.23:0/2144141304 >> v2:172.21.15.155:3300/0 conn(0x7faa480619d0 msgr2=0x7faa480a9740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:42:41.769 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.763+0000 7faa4d599700 1 --2- 172.21.15.23:0/2144141304 >> v2:172.21.15.155:3300/0 conn(0x7faa480619d0 0x7faa480a9740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:41.769 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.763+0000 7faa4d599700 1 -- 172.21.15.23:0/2144141304 >> v1:172.21.15.155:6789/0 conn(0x7faa4813c790 legacy=0x7faa480aa9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:41.769 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.763+0000 7faa4d599700 1 -- 172.21.15.23:0/2144141304 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa480650b0 con 0x7faa48061db0 2020-11-20T11:42:41.769 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.763+0000 7faa4d599700 1 --2- 172.21.15.23:0/2144141304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa48061db0 0x7faa4813c2a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7faa480af030 tx=0x7faa38006ef0).ready entity=mon.0 client_cookie=c37230426db7120b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:41.770 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.763+0000 7faa4cd98700 1 -- 172.21.15.23:0/2144141304 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3800b860 con 0x7faa48061db0 2020-11-20T11:42:41.770 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.763+0000 7faa4cd98700 1 -- 172.21.15.23:0/2144141304 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa38000e90 con 0x7faa48061db0 2020-11-20T11:42:41.770 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.763+0000 7faa4cd98700 1 -- 172.21.15.23:0/2144141304 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3800bbd0 con 0x7faa48061db0 2020-11-20T11:42:41.770 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.764+0000 7faa4fffe700 1 -- 172.21.15.23:0/2144141304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa48061db0 msgr2=0x7faa4813c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:41.771 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.764+0000 7faa4fffe700 1 --2- 172.21.15.23:0/2144141304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa48061db0 0x7faa4813c2a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7faa480af030 tx=0x7faa38006ef0).stop 2020-11-20T11:42:41.771 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.764+0000 7faa4fffe700 1 -- 172.21.15.23:0/2144141304 shutdown_connections 2020-11-20T11:42:41.771 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.764+0000 7faa4fffe700 1 --2- 172.21.15.23:0/2144141304 >> v2:172.21.15.155:3300/0 conn(0x7faa480619d0 0x7faa480a9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:41.771 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.764+0000 7faa4fffe700 1 --2- 172.21.15.23:0/2144141304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa48061db0 0x7faa4813c2a0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:41.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.764+0000 7faa4fffe700 1 -- 172.21.15.23:0/2144141304 >> 172.21.15.23:0/2144141304 conn(0x7faa48063640 msgr2=0x7faa480612c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:41.788 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.785+0000 7faa4fffe700 1 -- 172.21.15.23:0/2144141304 shutdown_connections 2020-11-20T11:42:41.789 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.785+0000 7faa4fffe700 1 -- 172.21.15.23:0/2144141304 wait complete. 2020-11-20T11:42:41.790 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.787+0000 7faa4fffe700 1 Processor -- start 2020-11-20T11:42:41.790 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.787+0000 7faa4fffe700 1 -- start start 2020-11-20T11:42:41.792 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.788+0000 7faa4fffe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa48061db0 0x7faa480a28e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:41.792 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.788+0000 7faa4fffe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faa4813c790 0x7faa4809de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:41.792 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.788+0000 7faa4fffe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faa4809e300 0x7faa48135240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:41.793 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.788+0000 7faa4dd9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa48061db0 0x7faa480a28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:41.793 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.788+0000 7faa4dd9a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa48061db0 0x7faa480a28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44966/0 (socket says 172.21.15.23:44966) 2020-11-20T11:42:41.793 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.788+0000 7faa4dd9a700 1 -- 172.21.15.23:0/562907382 learned_addr learned my addr 172.21.15.23:0/562907382 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:42:41.793 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.788+0000 7faa4fffe700 1 -- 172.21.15.23:0/562907382 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa480a3e10 con 0x7faa48061db0 2020-11-20T11:42:41.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.788+0000 7faa4fffe700 1 -- 172.21.15.23:0/562907382 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa480a40b0 con 0x7faa4809e300 2020-11-20T11:42:41.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.788+0000 7faa4fffe700 1 -- 172.21.15.23:0/562907382 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa480a4350 con 0x7faa4813c790 2020-11-20T11:42:41.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.788+0000 7faa4e59b700 1 --2- 172.21.15.23:0/562907382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faa4809e300 0x7faa48135240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:41.795 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.788+0000 7faa4dd9a700 1 -- 172.21.15.23:0/562907382 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faa4813c790 msgr2=0x7faa4809de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:41.795 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.788+0000 7faa4dd9a700 1 --2- 172.21.15.23:0/562907382 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faa4813c790 0x7faa4809de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:41.795 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.789+0000 7faa4dd9a700 1 -- 172.21.15.23:0/562907382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faa4809e300 msgr2=0x7faa48135240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:41.795 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.789+0000 7faa4dd9a700 1 --2- 172.21.15.23:0/562907382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7faa4809e300 0x7faa48135240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:41.795 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.789+0000 7faa4dd9a700 1 -- 172.21.15.23:0/562907382 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa4809e760 con 0x7faa48061db0 2020-11-20T11:42:41.796 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.789+0000 7faa4dd9a700 1 --2- 172.21.15.23:0/562907382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa48061db0 0x7faa480a28e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7faa44009690 tx=0x7faa44006dc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:41.796 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.789+0000 7faa35ffb700 1 -- 172.21.15.23:0/562907382 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa4400d4a0 con 0x7faa48061db0 2020-11-20T11:42:41.796 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.789+0000 7faa35ffb700 1 -- 172.21.15.23:0/562907382 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa4400d600 con 0x7faa48061db0 2020-11-20T11:42:41.796 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.789+0000 7faa35ffb700 1 -- 172.21.15.23:0/562907382 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa4400d9c0 con 0x7faa48061db0 2020-11-20T11:42:41.796 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.789+0000 7faa4fffe700 1 -- 172.21.15.23:0/562907382 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa48067eb0 con 0x7faa48061db0 2020-11-20T11:42:41.798 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.789+0000 7faa4fffe700 1 -- 172.21.15.23:0/562907382 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa480674f0 con 0x7faa48061db0 2020-11-20T11:42:41.798 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.790+0000 7faa4fffe700 1 --2- 172.21.15.23:0/562907382 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faa4805c070 0x7faa48135730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:41.798 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.791+0000 7faa4fffe700 1 -- 172.21.15.23:0/562907382 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa480a9360 con 0x7faa4805c070 2020-11-20T11:42:41.799 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.791+0000 7faa4d599700 1 --2- 172.21.15.23:0/562907382 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faa4805c070 0x7faa48135730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:41.799 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.791+0000 7faa35ffb700 1 -- 172.21.15.23:0/562907382 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7faa4400f070 con 0x7faa48061db0 2020-11-20T11:42:41.799 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.791+0000 7faa35ffb700 1 --2- 172.21.15.23:0/562907382 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7faa20056d60 0x7faa20059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:41.799 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.791+0000 7faa35ffb700 1 -- 172.21.15.23:0/562907382 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7faa4406f690 con 0x7faa48061db0 2020-11-20T11:42:41.799 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.792+0000 7faa4e59b700 1 --2- 172.21.15.23:0/562907382 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7faa20056d60 0x7faa20059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:41.800 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.792+0000 7faa4d599700 1 -- 172.21.15.23:0/562907382 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faa480a3c90 con 0x7faa4805c070 2020-11-20T11:42:41.800 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.792+0000 7faa4d599700 1 --2- 172.21.15.23:0/562907382 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faa4805c070 0x7faa48135730 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7faa480a59e0 tx=0x7faa38016000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:41.800 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.792+0000 7faa35ffb700 1 -- 172.21.15.23:0/562907382 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa38011b60 con 0x7faa4805c070 2020-11-20T11:42:41.800 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.792+0000 7faa4e59b700 1 --2- 172.21.15.23:0/562907382 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7faa20056d60 0x7faa20059220 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7faa40004290 tx=0x7faa40007f50).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:41.800 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.792+0000 7faa35ffb700 1 -- 172.21.15.23:0/562907382 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7faa38011cc0 con 0x7faa4805c070 2020-11-20T11:42:41.801 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.792+0000 7faa35ffb700 1 -- 172.21.15.23:0/562907382 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faa4805c070 msgr2=0x7faa48135730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:41.802 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.792+0000 7faa35ffb700 1 --2- 172.21.15.23:0/562907382 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faa4805c070 0x7faa48135730 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7faa480a59e0 tx=0x7faa38016000).stop 2020-11-20T11:42:41.802 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.792+0000 7faa35ffb700 1 -- 172.21.15.23:0/562907382 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faa4805c070 msgr2=0x7faa48135730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:42:41.802 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.792+0000 7faa35ffb700 1 --2- 172.21.15.23:0/562907382 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faa4805c070 0x7faa48135730 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:41.804 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.801+0000 7faa4fffe700 1 --2- 172.21.15.23:0/562907382 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faa480622d0 0x7faa48066010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:41.804 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.801+0000 7faa4fffe700 1 -- 172.21.15.23:0/562907382 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa480a3f50 con 0x7faa480622d0 2020-11-20T11:42:41.805 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.801+0000 7faa4d599700 1 --2- 172.21.15.23:0/562907382 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faa480622d0 0x7faa48066010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:41.805 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.802+0000 7faa4d599700 1 -- 172.21.15.23:0/562907382 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7faa48044710 con 0x7faa480622d0 2020-11-20T11:42:41.805 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.802+0000 7faa4d599700 1 --2- 172.21.15.23:0/562907382 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faa480622d0 0x7faa48066010 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7faa38006c80 tx=0x7faa3801d000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:41.806 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.802+0000 7faa35ffb700 1 -- 172.21.15.23:0/562907382 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3801ca00 con 0x7faa480622d0 2020-11-20T11:42:41.806 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.803+0000 7faa35ffb700 1 -- 172.21.15.23:0/562907382 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4444 (secure 0 0 0) 0x7faa38011850 con 0x7faa480622d0 2020-11-20T11:42:41.807 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.803+0000 7faa35ffb700 1 -- 172.21.15.23:0/562907382 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faa480622d0 msgr2=0x7faa48066010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:41.807 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.803+0000 7faa35ffb700 1 --2- 172.21.15.23:0/562907382 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faa480622d0 0x7faa48066010 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7faa38006c80 tx=0x7faa3801d000).stop 2020-11-20T11:42:41.807 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.803+0000 7faa35ffb700 1 -- 172.21.15.23:0/562907382 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faa480622d0 msgr2=0x7faa48066010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:42:41.807 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.803+0000 7faa35ffb700 1 --2- 172.21.15.23:0/562907382 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7faa480622d0 0x7faa48066010 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:41.808 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.803+0000 7faa4fffe700 1 -- 172.21.15.23:0/562907382 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7faa20056d60 msgr2=0x7faa20059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:41.808 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.803+0000 7faa4fffe700 1 --2- 172.21.15.23:0/562907382 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7faa20056d60 0x7faa20059220 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7faa40004290 tx=0x7faa40007f50).stop 2020-11-20T11:42:41.808 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.803+0000 7faa4fffe700 1 -- 172.21.15.23:0/562907382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa48061db0 msgr2=0x7faa480a28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:41.808 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.803+0000 7faa4fffe700 1 --2- 172.21.15.23:0/562907382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faa48061db0 0x7faa480a28e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7faa44009690 tx=0x7faa44006dc0).stop 2020-11-20T11:42:41.808 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.803+0000 7faa4e59b700 1 -- 172.21.15.23:0/562907382 reap_dead start 2020-11-20T11:42:41.809 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.803+0000 7faa4e59b700 1 -- 172.21.15.23:0/562907382 reap_dead start 2020-11-20T11:42:41.809 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.803+0000 7faa4fffe700 1 -- 172.21.15.23:0/562907382 shutdown_connections 2020-11-20T11:42:41.809 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.803+0000 7faa4fffe700 1 -- 172.21.15.23:0/562907382 >> 172.21.15.23:0/562907382 conn(0x7faa48063640 msgr2=0x7faa480ad8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:41.809 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:42:41.810 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:42:41.810 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:42:41.810 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:42:41.810 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 92, 2020-11-20T11:42:41.811 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:42:41.811 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:42:41.811 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:42:41.811 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:42:41.811 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:41.812 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 16, 2020-11-20T11:42:41.812 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:42:41.812 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:42:41.812 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:42:41.812 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:42:41.813 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:42:41.813 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:42:41.813 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:42:41.813 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:42:41.813 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:42:41.814 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:42:41.814 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:42:41.814 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:41.814 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:42:41.814 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:42:41.815 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:42:41.815 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:42:41.815 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:42:41.815 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:42:41.815 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:42:41.816 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:42:41.816 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:42:41.816 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:42:41.816 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:41.816 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.817 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:42:41.817 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:42:41.817 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:42:41.817 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:42:41.817 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:42:41.818 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:42:41.818 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:42:41.818 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:42:41.818 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:42:41.818 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:42:41.819 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:42:41.819 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:42:41.819 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:42:41.819 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:42:41.819 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:42:41.820 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:42:41.820 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:42:41.820 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:42:41.820 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:42:41.820 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:42:41.820 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:42:41.821 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:42:41.821 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:42:41.821 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:41.821 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:42:41.821 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.822 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:42:41.822 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.822 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:42:41.822 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:42:41.822 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:42:41.823 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:42:41.823 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.823 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:42:41.823 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:42:41.823 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:41.824 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.824 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.824 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:42:41.824 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:42:41.824 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:41.825 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:41.825 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:41.825 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.825 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:42:41.825 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:42:41.826 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:42:41.826 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:42:41.826 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:42:41.826 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.826 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.827 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:42:41.827 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:42:41.827 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:42:41.827 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:42:41.827 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.828 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:42:41.828 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:42:41.828 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:41.828 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.828 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.829 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:42:41.829 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:42:41.829 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:41.829 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:41.829 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:41.830 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.830 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:42:41.830 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:42:41.830 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:42:41.830 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:42:41.830 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:42:41.831 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.831 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.831 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:42:41.831 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:42:41.831 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:42:41.832 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:42:41.832 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.832 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:42:41.832 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:42:41.832 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:41.833 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.833 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.833 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:42:41.833 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:42:41.833 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:41.834 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:41.834 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:41.834 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.834 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:42:41.834 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:42:41.835 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:42:41.835 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:42:41.835 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:42:41.835 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:41.835 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:41.836 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.836 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:42:41.836 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:42:41.836 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.836 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:41.837 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:41.837 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:42:41.837 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:41.837 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:41.837 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:42:41.838 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.838 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:41.838 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:41.838 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:42:41.838 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:41.839 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:41.839 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:42:41.839 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:41.839 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:41.839 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:41.839 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:42:41.840 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:41.840 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:41.840 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:41.840 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:42:41.840 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:42:41.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.805+0000 7faa4fffe700 1 -- 172.21.15.23:0/562907382 shutdown_connections 2020-11-20T11:42:41.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:41.809+0000 7faa4fffe700 1 -- 172.21.15.23:0/562907382 wait complete. 2020-11-20T11:42:41.842 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2020-11-20T11:42:41.843 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2020-11-20T11:42:41.843 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2020-11-20T11:42:41.843 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2020-11-20T11:42:41.843 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2020-11-20T11:42:41.844 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T11:42:41.856 INFO:tasks.ceph.mon.c:Stopped 2020-11-20T11:42:41.857 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2020-11-20T11:42:41.857 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:42:42.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.046+0000 7f3574f65700 1 Processor -- start 2020-11-20T11:42:42.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.046+0000 7f3574f65700 1 -- start start 2020-11-20T11:42:42.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.047+0000 7f3574f65700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f35700615a0 0x7f3570137280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:42.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.047+0000 7f3574f65700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f35700a7260 0x7f35700a95c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:42.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.047+0000 7f3574f65700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3570137770 con 0x7f35700611c0 2020-11-20T11:42:42.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.047+0000 7f3574f65700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3570064a80 con 0x7f35700615a0 2020-11-20T11:42:42.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.047+0000 7f3574f65700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3570137a00 con 0x7f35700a7260 2020-11-20T11:42:42.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.047+0000 7f356dd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f35700615a0 0x7f3570137280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:42.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.047+0000 7f356dd9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f35700615a0 0x7f3570137280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44980/0 (socket says 172.21.15.23:44980) 2020-11-20T11:42:42.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.047+0000 7f356dd9b700 1 -- 172.21.15.23:0/3507443383 learned_addr learned my addr 172.21.15.23:0/3507443383 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:42:42.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.048+0000 7f356ed9d700 1 --2- 172.21.15.23:0/3507443383 >> v2:172.21.15.155:3300/0 conn(0x7f35700a7260 0x7f35700a95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:42.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.048+0000 7f356dd9b700 1 -- 172.21.15.23:0/3507443383 >> v2:172.21.15.155:3300/0 conn(0x7f35700a7260 msgr2=0x7f35700a95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:42:42.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.048+0000 7f356dd9b700 1 --2- 172.21.15.23:0/3507443383 >> v2:172.21.15.155:3300/0 conn(0x7f35700a7260 0x7f35700a95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:42.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.048+0000 7f356dd9b700 1 -- 172.21.15.23:0/3507443383 >> v1:172.21.15.155:6789/0 conn(0x7f35700611c0 legacy=0x7f35700a6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:42.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.048+0000 7f356dd9b700 1 -- 172.21.15.23:0/3507443383 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3570137ca0 con 0x7f35700615a0 2020-11-20T11:42:42.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.048+0000 7f356dd9b700 1 --2- 172.21.15.23:0/3507443383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35700615a0 0x7f3570137280 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f35700a6a70 tx=0x7f3560006ef0).ready entity=mon.0 client_cookie=f869c50cee6c6db4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:42.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.048+0000 7f356d59a700 1 -- 172.21.15.23:0/3507443383 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3560016d80 con 0x7f35700615a0 2020-11-20T11:42:42.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.048+0000 7f356d59a700 1 -- 172.21.15.23:0/3507443383 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3560000e90 con 0x7f35700615a0 2020-11-20T11:42:42.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.049+0000 7f356d59a700 1 -- 172.21.15.23:0/3507443383 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35600098f0 con 0x7f35700615a0 2020-11-20T11:42:42.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.049+0000 7f3574f65700 1 -- 172.21.15.23:0/3507443383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35700615a0 msgr2=0x7f3570137280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:42.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.049+0000 7f3574f65700 1 --2- 172.21.15.23:0/3507443383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35700615a0 0x7f3570137280 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f35700a6a70 tx=0x7f3560006ef0).stop 2020-11-20T11:42:42.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.050+0000 7f3574f65700 1 -- 172.21.15.23:0/3507443383 shutdown_connections 2020-11-20T11:42:42.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.050+0000 7f3574f65700 1 --2- 172.21.15.23:0/3507443383 >> v2:172.21.15.155:3300/0 conn(0x7f35700a7260 0x7f35700a95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:42.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.050+0000 7f3574f65700 1 --2- 172.21.15.23:0/3507443383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35700615a0 0x7f3570137280 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:42.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.050+0000 7f3574f65700 1 -- 172.21.15.23:0/3507443383 >> 172.21.15.23:0/3507443383 conn(0x7f35700ae230 msgr2=0x7f35700aa210 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:42.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.050+0000 7f3574f65700 1 -- 172.21.15.23:0/3507443383 shutdown_connections 2020-11-20T11:42:42.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.050+0000 7f3574f65700 1 -- 172.21.15.23:0/3507443383 wait complete. 2020-11-20T11:42:42.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.051+0000 7f3574f65700 1 Processor -- start 2020-11-20T11:42:42.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.051+0000 7f3574f65700 1 -- start start 2020-11-20T11:42:42.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.052+0000 7f3574f65700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35700611c0 0x7f35700678a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:42.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.052+0000 7f3574f65700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35700615a0 0x7f3570131f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:42.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.052+0000 7f3574f65700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35700a7260 0x7f3570138680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:42.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.052+0000 7f3574f65700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3570068450 con 0x7f35700a7260 2020-11-20T11:42:42.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.052+0000 7f3574f65700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35700686f0 con 0x7f35700611c0 2020-11-20T11:42:42.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.052+0000 7f3574f65700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3570065860 con 0x7f35700615a0 2020-11-20T11:42:42.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.052+0000 7f356ed9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35700a7260 0x7f3570138680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:42.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.052+0000 7f356dd9b700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35700615a0 msgr2=0x7f3570131f40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:42:42.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.052+0000 7f356dd9b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35700615a0 0x7f3570131f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:42:42.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.052+0000 7f356ed9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35700a7260 0x7f3570138680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44986/0 (socket says 172.21.15.23:44986) 2020-11-20T11:42:42.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.052+0000 7f356ed9d700 1 -- 172.21.15.23:0/4158600171 learned_addr learned my addr 172.21.15.23:0/4158600171 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:42:42.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.052+0000 7f356e59c700 1 --2- 172.21.15.23:0/4158600171 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35700611c0 0x7f35700678a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:42.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.053+0000 7f356ed9d700 1 -- 172.21.15.23:0/4158600171 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35700615a0 msgr2=0x7f3570131f40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:42:42.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.053+0000 7f356ed9d700 1 --2- 172.21.15.23:0/4158600171 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35700615a0 0x7f3570131f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:42.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.053+0000 7f356ed9d700 1 -- 172.21.15.23:0/4158600171 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35700611c0 msgr2=0x7f35700678a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:42.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.053+0000 7f356ed9d700 1 --2- 172.21.15.23:0/4158600171 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35700611c0 0x7f35700678a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:42.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.053+0000 7f356ed9d700 1 -- 172.21.15.23:0/4158600171 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3570067390 con 0x7f35700a7260 2020-11-20T11:42:42.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.053+0000 7f356ed9d700 1 --2- 172.21.15.23:0/4158600171 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35700a7260 0x7f3570138680 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f356800ffd0 tx=0x7f35680080e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:42.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.053+0000 7f355e7fc700 1 -- 172.21.15.23:0/4158600171 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356800f470 con 0x7f35700a7260 2020-11-20T11:42:42.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.053+0000 7f355e7fc700 1 -- 172.21.15.23:0/4158600171 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3568002d90 con 0x7f35700a7260 2020-11-20T11:42:42.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.053+0000 7f3574f65700 1 -- 172.21.15.23:0/4158600171 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3570134f10 con 0x7f35700a7260 2020-11-20T11:42:42.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.053+0000 7f355e7fc700 1 -- 172.21.15.23:0/4158600171 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3568018420 con 0x7f35700a7260 2020-11-20T11:42:42.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.053+0000 7f3574f65700 1 -- 172.21.15.23:0/4158600171 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35701399f0 con 0x7f35700a7260 2020-11-20T11:42:42.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.054+0000 7f3574f65700 1 -- 172.21.15.23:0/4158600171 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3570061e10 con 0x7f35700a7260 2020-11-20T11:42:42.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.055+0000 7f355e7fc700 1 -- 172.21.15.23:0/4158600171 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f356800e040 con 0x7f35700a7260 2020-11-20T11:42:42.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.056+0000 7f355e7fc700 1 --2- 172.21.15.23:0/4158600171 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3544056d10 0x7f35440591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:42.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.056+0000 7f355e7fc700 1 -- 172.21.15.23:0/4158600171 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f3568070b90 con 0x7f35700a7260 2020-11-20T11:42:42.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.056+0000 7f356e59c700 1 --2- 172.21.15.23:0/4158600171 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3544056d10 0x7f35440591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:42.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.056+0000 7f356e59c700 1 --2- 172.21.15.23:0/4158600171 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3544056d10 0x7f35440591d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f3564007fd0 tx=0x7f3564007bd0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:42.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.059+0000 7f355e7fc700 1 -- 172.21.15.23:0/4158600171 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3568045f00 con 0x7f35700a7260 2020-11-20T11:42:42.411 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.408+0000 7f3574f65700 1 -- 172.21.15.23:0/4158600171 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f357005c320 con 0x7f35700a7260 2020-11-20T11:42:42.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.408+0000 7f355e7fc700 1 -- 172.21.15.23:0/4158600171 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7f357005c320 con 0x7f35700a7260 2020-11-20T11:42:42.412 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:42:42.414 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.411+0000 7f3574f65700 1 -- 172.21.15.23:0/4158600171 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3544056d10 msgr2=0x7f35440591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:42.414 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.411+0000 7f3574f65700 1 --2- 172.21.15.23:0/4158600171 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3544056d10 0x7f35440591d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f3564007fd0 tx=0x7f3564007bd0).stop 2020-11-20T11:42:42.415 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.411+0000 7f3574f65700 1 -- 172.21.15.23:0/4158600171 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35700a7260 msgr2=0x7f3570138680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:42.415 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.411+0000 7f3574f65700 1 --2- 172.21.15.23:0/4158600171 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35700a7260 0x7f3570138680 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f356800ffd0 tx=0x7f35680080e0).stop 2020-11-20T11:42:42.415 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.411+0000 7f3574f65700 1 -- 172.21.15.23:0/4158600171 shutdown_connections 2020-11-20T11:42:42.415 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.411+0000 7f3574f65700 1 --2- 172.21.15.23:0/4158600171 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35700a7260 0x7f3570138680 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:42.416 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.411+0000 7f3574f65700 1 --2- 172.21.15.23:0/4158600171 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35700611c0 0x7f35700678a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:42.416 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.411+0000 7f3574f65700 1 --2- 172.21.15.23:0/4158600171 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3544056d10 0x7f35440591d0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:42.416 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.411+0000 7f3574f65700 1 --2- 172.21.15.23:0/4158600171 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35700615a0 0x7f3570131f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:42.416 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.411+0000 7f3574f65700 1 -- 172.21.15.23:0/4158600171 >> 172.21.15.23:0/4158600171 conn(0x7f35700ae230 msgr2=0x7f35700aa210 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:42.416 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.411+0000 7f3574f65700 1 -- 172.21.15.23:0/4158600171 shutdown_connections 2020-11-20T11:42:42.417 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:42.411+0000 7f3574f65700 1 -- 172.21.15.23:0/4158600171 wait complete. 2020-11-20T11:42:45.429 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:42:45.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.639+0000 7f59c7a5a700 1 Processor -- start 2020-11-20T11:42:45.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.640+0000 7f59c7a5a700 1 -- start start 2020-11-20T11:42:45.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.641+0000 7f59c7a5a700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59c00a9510 0x7f59c00a3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:45.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.641+0000 7f59c7a5a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f59c0137d10 0x7f59c0138160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:45.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.641+0000 7f59c7a5a700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c00a3860 con 0x7f59c00ad7c0 2020-11-20T11:42:45.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.641+0000 7f59c7a5a700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59c00a3320 con 0x7f59c0137d10 2020-11-20T11:42:45.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.641+0000 7f59c7a5a700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59c00a35c0 con 0x7f59c00a9510 2020-11-20T11:42:45.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.641+0000 7f59c57f6700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f59c00ad7c0 0x7f59c00a8e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:37226/0 (socket says 172.21.15.23:37226) 2020-11-20T11:42:45.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.641+0000 7f59c57f6700 1 -- 172.21.15.23:0/905948134 learned_addr learned my addr 172.21.15.23:0/905948134 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:42:45.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.641+0000 7f59c4ff5700 1 -- 172.21.15.23:0/905948134 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59c00a9510 msgr2=0x7f59c00a3a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:42:45.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.641+0000 7f59c4ff5700 1 --2- 172.21.15.23:0/905948134 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59c00a9510 0x7f59c00a3a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:42:45.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.641+0000 7f59c5ff7700 1 --2- 172.21.15.23:0/905948134 >> v2:172.21.15.23:3300/0 conn(0x7f59c0137d10 0x7f59c0138160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:45.653 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.642+0000 7f59c5ff7700 1 -- 172.21.15.23:0/905948134 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59c00a9510 msgr2=0x7f59c00a3a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:42:45.653 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.642+0000 7f59c5ff7700 1 --2- 172.21.15.23:0/905948134 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59c00a9510 0x7f59c00a3a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:45.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.642+0000 7f59c5ff7700 1 -- 172.21.15.23:0/905948134 >> v1:172.21.15.155:6789/0 conn(0x7f59c00ad7c0 legacy=0x7f59c00a8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:45.656 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.642+0000 7f59c5ff7700 1 -- 172.21.15.23:0/905948134 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c00adba0 con 0x7f59c0137d10 2020-11-20T11:42:45.657 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.642+0000 7f59c5ff7700 1 --2- 172.21.15.23:0/905948134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59c0137d10 0x7f59c0138160 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f59b8006fd0 tx=0x7f59b8006e30).ready entity=mon.0 client_cookie=b9359a647782d9b8 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:45.657 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.643+0000 7f59affff700 1 -- 172.21.15.23:0/905948134 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b80097e0 con 0x7f59c0137d10 2020-11-20T11:42:45.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.643+0000 7f59affff700 1 -- 172.21.15.23:0/905948134 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59b8000e90 con 0x7f59c0137d10 2020-11-20T11:42:45.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.643+0000 7f59affff700 1 -- 172.21.15.23:0/905948134 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b8009b50 con 0x7f59c0137d10 2020-11-20T11:42:45.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.644+0000 7f59c7a5a700 1 -- 172.21.15.23:0/905948134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59c0137d10 msgr2=0x7f59c0138160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:45.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.644+0000 7f59c7a5a700 1 --2- 172.21.15.23:0/905948134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59c0137d10 0x7f59c0138160 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f59b8006fd0 tx=0x7f59b8006e30).stop 2020-11-20T11:42:45.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.667+0000 7f59c7a5a700 1 -- 172.21.15.23:0/905948134 shutdown_connections 2020-11-20T11:42:45.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.667+0000 7f59c7a5a700 1 --2- 172.21.15.23:0/905948134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59c0137d10 0x7f59c0138160 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:45.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.667+0000 7f59c7a5a700 1 --2- 172.21.15.23:0/905948134 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59c00a9510 0x7f59c00a3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:45.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.667+0000 7f59c7a5a700 1 -- 172.21.15.23:0/905948134 >> 172.21.15.23:0/905948134 conn(0x7f59c0060c80 msgr2=0x7f59c00acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:45.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.669+0000 7f59c7a5a700 1 -- 172.21.15.23:0/905948134 shutdown_connections 2020-11-20T11:42:45.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.669+0000 7f59c7a5a700 1 -- 172.21.15.23:0/905948134 wait complete. 2020-11-20T11:42:45.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.675+0000 7f59c7a5a700 1 Processor -- start 2020-11-20T11:42:45.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.675+0000 7f59c7a5a700 1 -- start start 2020-11-20T11:42:45.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.676+0000 7f59c7a5a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59c00a9510 0x7f59c00ac1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:45.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.676+0000 7f59c7a5a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59c00ad7c0 0x7f59c014a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:45.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.676+0000 7f59c7a5a700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59c0137d10 0x7f59c014aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:45.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.676+0000 7f59c7a5a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c014a1f0 con 0x7f59c00ad7c0 2020-11-20T11:42:45.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.676+0000 7f59c7a5a700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c014afc0 con 0x7f59c00a9510 2020-11-20T11:42:45.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.676+0000 7f59c7a5a700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59c013a610 con 0x7f59c0137d10 2020-11-20T11:42:45.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.676+0000 7f59c5ff7700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59c0137d10 msgr2=0x7f59c014aad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:42:45.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.676+0000 7f59c5ff7700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59c0137d10 0x7f59c014aad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:42:45.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.676+0000 7f59c4ff5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59c00ad7c0 0x7f59c014a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:45.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.676+0000 7f59c57f6700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59c00a9510 0x7f59c00ac1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:45.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.676+0000 7f59c4ff5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59c00ad7c0 0x7f59c014a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45010/0 (socket says 172.21.15.23:45010) 2020-11-20T11:42:45.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.676+0000 7f59c4ff5700 1 -- 172.21.15.23:0/2420226782 learned_addr learned my addr 172.21.15.23:0/2420226782 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:42:45.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.677+0000 7f59c4ff5700 1 -- 172.21.15.23:0/2420226782 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59c0137d10 msgr2=0x7f59c014aad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:42:45.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.677+0000 7f59c4ff5700 1 --2- 172.21.15.23:0/2420226782 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59c0137d10 0x7f59c014aad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:45.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.677+0000 7f59c4ff5700 1 -- 172.21.15.23:0/2420226782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59c00a9510 msgr2=0x7f59c00ac1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:45.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.677+0000 7f59c4ff5700 1 --2- 172.21.15.23:0/2420226782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59c00a9510 0x7f59c00ac1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:45.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.677+0000 7f59c4ff5700 1 -- 172.21.15.23:0/2420226782 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c013a8b0 con 0x7f59c00ad7c0 2020-11-20T11:42:45.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.677+0000 7f59c57f6700 1 --2- 172.21.15.23:0/2420226782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59c00a9510 0x7f59c00ac1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:42:45.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.677+0000 7f59c4ff5700 1 --2- 172.21.15.23:0/2420226782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59c00ad7c0 0x7f59c014a5e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f59b000bb60 tx=0x7f59b0009270).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:45.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.677+0000 7f59ad7fa700 1 -- 172.21.15.23:0/2420226782 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b0003ca0 con 0x7f59c00ad7c0 2020-11-20T11:42:45.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.677+0000 7f59ad7fa700 1 -- 172.21.15.23:0/2420226782 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59b0003e00 con 0x7f59c00ad7c0 2020-11-20T11:42:45.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.677+0000 7f59ad7fa700 1 -- 172.21.15.23:0/2420226782 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b000c720 con 0x7f59c00ad7c0 2020-11-20T11:42:45.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.677+0000 7f59c7a5a700 1 -- 172.21.15.23:0/2420226782 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59c013a290 con 0x7f59c00ad7c0 2020-11-20T11:42:45.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.677+0000 7f59c7a5a700 1 -- 172.21.15.23:0/2420226782 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59c00653c0 con 0x7f59c00ad7c0 2020-11-20T11:42:45.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.678+0000 7f59c7a5a700 1 -- 172.21.15.23:0/2420226782 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59c014a340 con 0x7f59c00ad7c0 2020-11-20T11:42:45.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.679+0000 7f59ad7fa700 1 -- 172.21.15.23:0/2420226782 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f59b000ca90 con 0x7f59c00ad7c0 2020-11-20T11:42:45.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.679+0000 7f59ad7fa700 1 --2- 172.21.15.23:0/2420226782 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5998056d10 0x7f59980591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:45.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.679+0000 7f59ad7fa700 1 -- 172.21.15.23:0/2420226782 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f59b0011020 con 0x7f59c00ad7c0 2020-11-20T11:42:45.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.679+0000 7f59c57f6700 1 --2- 172.21.15.23:0/2420226782 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5998056d10 0x7f59980591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:45.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.679+0000 7f59c57f6700 1 --2- 172.21.15.23:0/2420226782 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5998056d10 0x7f59980591d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f59bc001730 tx=0x7f59bc007bc0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:45.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:45.683+0000 7f59ad7fa700 1 -- 172.21.15.23:0/2420226782 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f59c014a340 con 0x7f59c00ad7c0 2020-11-20T11:42:46.045 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:46.042+0000 7f59c7a5a700 1 -- 172.21.15.23:0/2420226782 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f59c013a480 con 0x7f59c00ad7c0 2020-11-20T11:42:46.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:46.043+0000 7f59ad7fa700 1 -- 172.21.15.23:0/2420226782 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7f59b0044e30 con 0x7f59c00ad7c0 2020-11-20T11:42:46.047 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:42:46.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:46.045+0000 7f59c7a5a700 1 -- 172.21.15.23:0/2420226782 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5998056d10 msgr2=0x7f59980591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:46.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:46.045+0000 7f59c7a5a700 1 --2- 172.21.15.23:0/2420226782 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5998056d10 0x7f59980591d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f59bc001730 tx=0x7f59bc007bc0).stop 2020-11-20T11:42:46.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:46.045+0000 7f59c7a5a700 1 -- 172.21.15.23:0/2420226782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59c00ad7c0 msgr2=0x7f59c014a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:46.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:46.045+0000 7f59c7a5a700 1 --2- 172.21.15.23:0/2420226782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59c00ad7c0 0x7f59c014a5e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f59b000bb60 tx=0x7f59b0009270).stop 2020-11-20T11:42:46.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:46.046+0000 7f59c7a5a700 1 -- 172.21.15.23:0/2420226782 shutdown_connections 2020-11-20T11:42:46.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:46.046+0000 7f59c7a5a700 1 --2- 172.21.15.23:0/2420226782 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5998056d10 0x7f59980591d0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:46.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:46.046+0000 7f59c7a5a700 1 --2- 172.21.15.23:0/2420226782 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f59c0137d10 0x7f59c014aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:46.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:46.046+0000 7f59c7a5a700 1 --2- 172.21.15.23:0/2420226782 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f59c00a9510 0x7f59c00ac1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:46.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:46.046+0000 7f59c7a5a700 1 --2- 172.21.15.23:0/2420226782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59c00ad7c0 0x7f59c014a5e0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:46.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:46.046+0000 7f59c7a5a700 1 -- 172.21.15.23:0/2420226782 >> 172.21.15.23:0/2420226782 conn(0x7f59c0060c80 msgr2=0x7f59c0135890 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:46.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:46.050+0000 7f59c7a5a700 1 -- 172.21.15.23:0/2420226782 shutdown_connections 2020-11-20T11:42:46.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:46.050+0000 7f59c7a5a700 1 -- 172.21.15.23:0/2420226782 wait complete. 2020-11-20T11:42:49.079 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:42:49.278 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.275+0000 7fe0277c2700 1 Processor -- start 2020-11-20T11:42:49.279 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.275+0000 7fe0277c2700 1 -- start start 2020-11-20T11:42:49.281 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.276+0000 7fe0277c2700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe0200aae90 0x7fe0201381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:49.281 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.276+0000 7fe0277c2700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe02013aeb0 0x7fe0200a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:49.281 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.276+0000 7fe0277c2700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe02013e600 con 0x7fe0200aaab0 2020-11-20T11:42:49.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.276+0000 7fe0277c2700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe02013e8a0 con 0x7fe0200aae90 2020-11-20T11:42:49.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.276+0000 7fe0277c2700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0200ab6d0 con 0x7fe02013aeb0 2020-11-20T11:42:49.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.276+0000 7fe024d5d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe0200aae90 0x7fe0201381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:49.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.276+0000 7fe024d5d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe0200aae90 0x7fe0201381a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45020/0 (socket says 172.21.15.23:45020) 2020-11-20T11:42:49.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.276+0000 7fe024d5d700 1 -- 172.21.15.23:0/3036008812 learned_addr learned my addr 172.21.15.23:0/3036008812 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:42:49.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.276+0000 7fe02555e700 1 --1- 172.21.15.23:0/3036008812 >> v1:172.21.15.23:6789/0 conn(0x7fe0200aaab0 0x7fe02013ed40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:59268/0 (socket says 172.21.15.23:59268) 2020-11-20T11:42:49.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.276+0000 7fe025d5f700 1 --2- 172.21.15.23:0/3036008812 >> v2:172.21.15.155:3300/0 conn(0x7fe02013aeb0 0x7fe0200a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:49.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.277+0000 7fe024d5d700 1 -- 172.21.15.23:0/3036008812 >> v2:172.21.15.155:3300/0 conn(0x7fe02013aeb0 msgr2=0x7fe0200a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:42:49.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.277+0000 7fe024d5d700 1 --2- 172.21.15.23:0/3036008812 >> v2:172.21.15.155:3300/0 conn(0x7fe02013aeb0 0x7fe0200a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:49.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.277+0000 7fe024d5d700 1 -- 172.21.15.23:0/3036008812 >> v1:172.21.15.23:6789/0 conn(0x7fe0200aaab0 legacy=0x7fe02013ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:49.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.277+0000 7fe024d5d700 1 -- 172.21.15.23:0/3036008812 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe020138690 con 0x7fe0200aae90 2020-11-20T11:42:49.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.277+0000 7fe024d5d700 1 --2- 172.21.15.23:0/3036008812 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0200aae90 0x7fe0201381a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fe020062f40 tx=0x7fe010006ef0).ready entity=mon.0 client_cookie=ecc09d218019552b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:49.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.277+0000 7fe00ffff700 1 -- 172.21.15.23:0/3036008812 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01000b860 con 0x7fe0200aae90 2020-11-20T11:42:49.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.277+0000 7fe00ffff700 1 -- 172.21.15.23:0/3036008812 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe010000e90 con 0x7fe0200aae90 2020-11-20T11:42:49.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.277+0000 7fe00ffff700 1 -- 172.21.15.23:0/3036008812 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01000bbd0 con 0x7fe0200aae90 2020-11-20T11:42:49.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.278+0000 7fe0277c2700 1 -- 172.21.15.23:0/3036008812 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0200aae90 msgr2=0x7fe0201381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:49.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.278+0000 7fe0277c2700 1 --2- 172.21.15.23:0/3036008812 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0200aae90 0x7fe0201381a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fe020062f40 tx=0x7fe010006ef0).stop 2020-11-20T11:42:49.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.278+0000 7fe0277c2700 1 -- 172.21.15.23:0/3036008812 shutdown_connections 2020-11-20T11:42:49.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.278+0000 7fe0277c2700 1 --2- 172.21.15.23:0/3036008812 >> v2:172.21.15.155:3300/0 conn(0x7fe02013aeb0 0x7fe0200a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:49.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.278+0000 7fe0277c2700 1 --2- 172.21.15.23:0/3036008812 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0200aae90 0x7fe0201381a0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:49.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.278+0000 7fe0277c2700 1 -- 172.21.15.23:0/3036008812 >> 172.21.15.23:0/3036008812 conn(0x7fe0200678e0 msgr2=0x7fe020067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:49.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.278+0000 7fe0277c2700 1 -- 172.21.15.23:0/3036008812 shutdown_connections 2020-11-20T11:42:49.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.278+0000 7fe0277c2700 1 -- 172.21.15.23:0/3036008812 wait complete. 2020-11-20T11:42:49.289 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.279+0000 7fe0277c2700 1 Processor -- start 2020-11-20T11:42:49.289 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.279+0000 7fe0277c2700 1 -- start start 2020-11-20T11:42:49.289 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.280+0000 7fe0277c2700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0200aaab0 0x7fe0200b0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:49.290 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.280+0000 7fe0277c2700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0200aae90 0x7fe0200a60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:49.290 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.280+0000 7fe0277c2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0200a65c0 0x7fe02009ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:49.290 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.280+0000 7fe0277c2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0200a7370 con 0x7fe0200a65c0 2020-11-20T11:42:49.290 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.280+0000 7fe0277c2700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0200a7d10 con 0x7fe0200aae90 2020-11-20T11:42:49.290 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.280+0000 7fe0277c2700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0200a75d0 con 0x7fe0200aaab0 2020-11-20T11:42:49.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.280+0000 7fe02555e700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0200aaab0 msgr2=0x7fe0200b0b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:42:49.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.280+0000 7fe02555e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0200aaab0 0x7fe0200b0b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:42:49.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.280+0000 7fe025d5f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0200a65c0 0x7fe02009ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:49.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.280+0000 7fe025d5f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0200a65c0 0x7fe02009ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45028/0 (socket says 172.21.15.23:45028) 2020-11-20T11:42:49.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.280+0000 7fe025d5f700 1 -- 172.21.15.23:0/572436507 learned_addr learned my addr 172.21.15.23:0/572436507 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:42:49.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.280+0000 7fe024d5d700 1 --2- 172.21.15.23:0/572436507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0200aae90 0x7fe0200a60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:49.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.280+0000 7fe025d5f700 1 -- 172.21.15.23:0/572436507 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0200aaab0 msgr2=0x7fe0200b0b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:42:49.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.280+0000 7fe025d5f700 1 --2- 172.21.15.23:0/572436507 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0200aaab0 0x7fe0200b0b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:49.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.280+0000 7fe025d5f700 1 -- 172.21.15.23:0/572436507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0200aae90 msgr2=0x7fe0200a60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:49.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.280+0000 7fe025d5f700 1 --2- 172.21.15.23:0/572436507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0200aae90 0x7fe0200a60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:49.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.280+0000 7fe025d5f700 1 -- 172.21.15.23:0/572436507 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0200a7870 con 0x7fe0200a65c0 2020-11-20T11:42:49.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.280+0000 7fe025d5f700 1 --2- 172.21.15.23:0/572436507 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0200a65c0 0x7fe02009ea20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7fe01800ed20 tx=0x7fe018009220).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:49.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.280+0000 7fe00d7fa700 1 -- 172.21.15.23:0/572436507 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01800ca60 con 0x7fe0200a65c0 2020-11-20T11:42:49.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.280+0000 7fe00d7fa700 1 -- 172.21.15.23:0/572436507 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe01800cbc0 con 0x7fe0200a65c0 2020-11-20T11:42:49.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.280+0000 7fe0277c2700 1 -- 172.21.15.23:0/572436507 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe02009ef10 con 0x7fe0200a65c0 2020-11-20T11:42:49.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.280+0000 7fe00d7fa700 1 -- 172.21.15.23:0/572436507 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0180123f0 con 0x7fe0200a65c0 2020-11-20T11:42:49.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.280+0000 7fe0277c2700 1 -- 172.21.15.23:0/572436507 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0201335b0 con 0x7fe0200a65c0 2020-11-20T11:42:49.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.281+0000 7fe00d7fa700 1 -- 172.21.15.23:0/572436507 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe018022070 con 0x7fe0200a65c0 2020-11-20T11:42:49.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.281+0000 7fe0277c2700 1 -- 172.21.15.23:0/572436507 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe020068660 con 0x7fe0200a65c0 2020-11-20T11:42:49.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.282+0000 7fe00d7fa700 1 --2- 172.21.15.23:0/572436507 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fdff8056d10 0x7fdff80591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:49.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.282+0000 7fe00d7fa700 1 -- 172.21.15.23:0/572436507 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fe0180709f0 con 0x7fe0200a65c0 2020-11-20T11:42:49.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.282+0000 7fe02555e700 1 --2- 172.21.15.23:0/572436507 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fdff8056d10 0x7fdff80591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:49.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.282+0000 7fe02555e700 1 --2- 172.21.15.23:0/572436507 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fdff8056d10 0x7fdff80591d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fe01c001120 tx=0x7fe01c007820).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:49.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.286+0000 7fe00d7fa700 1 -- 172.21.15.23:0/572436507 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe018072e60 con 0x7fe0200a65c0 2020-11-20T11:42:49.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.634+0000 7fe0277c2700 1 -- 172.21.15.23:0/572436507 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe02005c7b0 con 0x7fe0200a65c0 2020-11-20T11:42:49.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.634+0000 7fe00d7fa700 1 -- 172.21.15.23:0/572436507 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7fe018016070 con 0x7fe0200a65c0 2020-11-20T11:42:49.639 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:42:49.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.637+0000 7fe0277c2700 1 -- 172.21.15.23:0/572436507 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fdff8056d10 msgr2=0x7fdff80591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:49.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.637+0000 7fe0277c2700 1 --2- 172.21.15.23:0/572436507 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fdff8056d10 0x7fdff80591d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fe01c001120 tx=0x7fe01c007820).stop 2020-11-20T11:42:49.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.637+0000 7fe0277c2700 1 -- 172.21.15.23:0/572436507 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0200a65c0 msgr2=0x7fe02009ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:49.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.637+0000 7fe0277c2700 1 --2- 172.21.15.23:0/572436507 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0200a65c0 0x7fe02009ea20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7fe01800ed20 tx=0x7fe018009220).stop 2020-11-20T11:42:49.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.637+0000 7fe0277c2700 1 -- 172.21.15.23:0/572436507 shutdown_connections 2020-11-20T11:42:49.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.637+0000 7fe0277c2700 1 --2- 172.21.15.23:0/572436507 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe0200a65c0 0x7fe02009ea20 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:49.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.637+0000 7fe0277c2700 1 --2- 172.21.15.23:0/572436507 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fdff8056d10 0x7fdff80591d0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:49.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.637+0000 7fe0277c2700 1 --2- 172.21.15.23:0/572436507 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe0200aaab0 0x7fe0200b0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:49.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.637+0000 7fe0277c2700 1 --2- 172.21.15.23:0/572436507 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe0200aae90 0x7fe0200a60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:49.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.637+0000 7fe0277c2700 1 -- 172.21.15.23:0/572436507 >> 172.21.15.23:0/572436507 conn(0x7fe0200678e0 msgr2=0x7fe020068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:49.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.637+0000 7fe0277c2700 1 -- 172.21.15.23:0/572436507 shutdown_connections 2020-11-20T11:42:49.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:49.637+0000 7fe0277c2700 1 -- 172.21.15.23:0/572436507 wait complete. 2020-11-20T11:42:52.658 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:42:52.852 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:52.848+0000 7fcb1b057700 1 Processor -- start 2020-11-20T11:42:52.915 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:52.848+0000 7fcb1b057700 1 -- start start 2020-11-20T11:42:52.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:52.849+0000 7fcb1b057700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fcb140aae90 0x7fcb141381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:52.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:52.849+0000 7fcb1b057700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fcb1413aeb0 0x7fcb140a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:52.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:52.849+0000 7fcb1b057700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb1413e600 con 0x7fcb140aaab0 2020-11-20T11:42:52.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:52.849+0000 7fcb1b057700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb1413e8a0 con 0x7fcb140aae90 2020-11-20T11:42:52.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:52.849+0000 7fcb1b057700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb140ab6d0 con 0x7fcb1413aeb0 2020-11-20T11:42:52.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:52.849+0000 7fcb13fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fcb140aae90 0x7fcb141381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:52.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:52.849+0000 7fcb13fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fcb140aae90 0x7fcb141381a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45034/0 (socket says 172.21.15.23:45034) 2020-11-20T11:42:52.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:52.849+0000 7fcb13fff700 1 -- 172.21.15.23:0/2014415475 learned_addr learned my addr 172.21.15.23:0/2014415475 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:42:52.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:52.849+0000 7fcb13fff700 1 -- 172.21.15.23:0/2014415475 >> v2:172.21.15.23:3300/0 conn(0x7fcb140aae90 msgr2=0x7fcb141381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:42:52.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:52.849+0000 7fcb13fff700 1 -- 172.21.15.23:0/2014415475 >> v2:172.21.15.23:3300/0 conn(0x7fcb140aae90 msgr2=0x7fcb141381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:42:52.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:52.849+0000 7fcb13fff700 1 --2- 172.21.15.23:0/2014415475 >> v2:172.21.15.23:3300/0 conn(0x7fcb140aae90 0x7fcb141381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:42:52.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:52.849+0000 7fcb195f4700 1 --2- 172.21.15.23:0/2014415475 >> v2:172.21.15.155:3300/0 conn(0x7fcb1413aeb0 0x7fcb140a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:52.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:52.849+0000 7fcb13fff700 1 --2- 172.21.15.23:0/2014415475 >> v2:172.21.15.23:3300/0 conn(0x7fcb140aae90 0x7fcb141381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:42:52.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:52.849+0000 7fcb195f4700 1 -- 172.21.15.23:0/2014415475 >> v2:172.21.15.155:3300/0 conn(0x7fcb1413aeb0 msgr2=0x7fcb140a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:42:52.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:52.849+0000 7fcb195f4700 1 -- 172.21.15.23:0/2014415475 >> v2:172.21.15.155:3300/0 conn(0x7fcb1413aeb0 msgr2=0x7fcb140a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:42:52.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:52.849+0000 7fcb195f4700 1 --2- 172.21.15.23:0/2014415475 >> v2:172.21.15.155:3300/0 conn(0x7fcb1413aeb0 0x7fcb140a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:42:52.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:52.849+0000 7fcb195f4700 1 --2- 172.21.15.23:0/2014415475 >> v2:172.21.15.155:3300/0 conn(0x7fcb1413aeb0 0x7fcb140a8c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:42:53.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:53.050+0000 7fcb195f4700 1 --2- 172.21.15.23:0/2014415475 >> v2:172.21.15.155:3300/0 conn(0x7fcb1413aeb0 0x7fcb140a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:53.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:53.050+0000 7fcb195f4700 1 -- 172.21.15.23:0/2014415475 >> v2:172.21.15.155:3300/0 conn(0x7fcb1413aeb0 msgr2=0x7fcb140a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:42:53.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:53.050+0000 7fcb195f4700 1 -- 172.21.15.23:0/2014415475 >> v2:172.21.15.155:3300/0 conn(0x7fcb1413aeb0 msgr2=0x7fcb140a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:42:53.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:53.050+0000 7fcb195f4700 1 --2- 172.21.15.23:0/2014415475 >> v2:172.21.15.155:3300/0 conn(0x7fcb1413aeb0 0x7fcb140a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:42:53.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:53.050+0000 7fcb195f4700 1 --2- 172.21.15.23:0/2014415475 >> v2:172.21.15.155:3300/0 conn(0x7fcb1413aeb0 0x7fcb140a8c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:42:53.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:53.052+0000 7fcb13fff700 1 --2- 172.21.15.23:0/2014415475 >> v2:172.21.15.23:3300/0 conn(0x7fcb140aae90 0x7fcb141381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:53.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:53.052+0000 7fcb13fff700 1 -- 172.21.15.23:0/2014415475 >> v2:172.21.15.23:3300/0 conn(0x7fcb140aae90 msgr2=0x7fcb141381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:42:53.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:53.052+0000 7fcb13fff700 1 -- 172.21.15.23:0/2014415475 >> v2:172.21.15.23:3300/0 conn(0x7fcb140aae90 msgr2=0x7fcb141381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:42:53.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:53.052+0000 7fcb13fff700 1 --2- 172.21.15.23:0/2014415475 >> v2:172.21.15.23:3300/0 conn(0x7fcb140aae90 0x7fcb141381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:42:53.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:53.052+0000 7fcb13fff700 1 --2- 172.21.15.23:0/2014415475 >> v2:172.21.15.23:3300/0 conn(0x7fcb140aae90 0x7fcb141381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:42:53.454 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:53.451+0000 7fcb195f4700 1 --2- 172.21.15.23:0/2014415475 >> v2:172.21.15.155:3300/0 conn(0x7fcb1413aeb0 0x7fcb140a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:53.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:53.451+0000 7fcb195f4700 1 -- 172.21.15.23:0/2014415475 >> v2:172.21.15.155:3300/0 conn(0x7fcb1413aeb0 msgr2=0x7fcb140a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:42:53.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:53.451+0000 7fcb195f4700 1 -- 172.21.15.23:0/2014415475 >> v2:172.21.15.155:3300/0 conn(0x7fcb1413aeb0 msgr2=0x7fcb140a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:42:53.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:53.451+0000 7fcb195f4700 1 --2- 172.21.15.23:0/2014415475 >> v2:172.21.15.155:3300/0 conn(0x7fcb1413aeb0 0x7fcb140a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:42:53.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:53.451+0000 7fcb195f4700 1 --2- 172.21.15.23:0/2014415475 >> v2:172.21.15.155:3300/0 conn(0x7fcb1413aeb0 0x7fcb140a8c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:42:53.456 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:53.453+0000 7fcb13fff700 1 --2- 172.21.15.23:0/2014415475 >> v2:172.21.15.23:3300/0 conn(0x7fcb140aae90 0x7fcb141381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:53.456 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:53.453+0000 7fcb13fff700 1 -- 172.21.15.23:0/2014415475 >> v2:172.21.15.23:3300/0 conn(0x7fcb140aae90 msgr2=0x7fcb141381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:42:53.457 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:53.453+0000 7fcb13fff700 1 -- 172.21.15.23:0/2014415475 >> v2:172.21.15.23:3300/0 conn(0x7fcb140aae90 msgr2=0x7fcb141381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:42:53.457 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:53.453+0000 7fcb13fff700 1 --2- 172.21.15.23:0/2014415475 >> v2:172.21.15.23:3300/0 conn(0x7fcb140aae90 0x7fcb141381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:42:53.457 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:53.453+0000 7fcb13fff700 1 --2- 172.21.15.23:0/2014415475 >> v2:172.21.15.23:3300/0 conn(0x7fcb140aae90 0x7fcb141381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:42:54.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:54.252+0000 7fcb195f4700 1 --2- 172.21.15.23:0/2014415475 >> v2:172.21.15.155:3300/0 conn(0x7fcb1413aeb0 0x7fcb140a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:54.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:54.252+0000 7fcb195f4700 1 -- 172.21.15.23:0/2014415475 >> v2:172.21.15.155:3300/0 conn(0x7fcb1413aeb0 msgr2=0x7fcb140a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:42:54.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:54.252+0000 7fcb195f4700 1 -- 172.21.15.23:0/2014415475 >> v2:172.21.15.155:3300/0 conn(0x7fcb1413aeb0 msgr2=0x7fcb140a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:42:54.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:54.252+0000 7fcb195f4700 1 --2- 172.21.15.23:0/2014415475 >> v2:172.21.15.155:3300/0 conn(0x7fcb1413aeb0 0x7fcb140a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:42:54.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:54.252+0000 7fcb195f4700 1 --2- 172.21.15.23:0/2014415475 >> v2:172.21.15.155:3300/0 conn(0x7fcb1413aeb0 0x7fcb140a8c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:42:54.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:54.254+0000 7fcb13fff700 1 --2- 172.21.15.23:0/2014415475 >> v2:172.21.15.23:3300/0 conn(0x7fcb140aae90 0x7fcb141381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:54.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:54.254+0000 7fcb13fff700 1 -- 172.21.15.23:0/2014415475 >> v2:172.21.15.23:3300/0 conn(0x7fcb140aae90 msgr2=0x7fcb141381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:42:54.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:54.254+0000 7fcb13fff700 1 -- 172.21.15.23:0/2014415475 >> v2:172.21.15.23:3300/0 conn(0x7fcb140aae90 msgr2=0x7fcb141381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:42:54.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:54.254+0000 7fcb13fff700 1 --2- 172.21.15.23:0/2014415475 >> v2:172.21.15.23:3300/0 conn(0x7fcb140aae90 0x7fcb141381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:42:54.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:54.254+0000 7fcb13fff700 1 --2- 172.21.15.23:0/2014415475 >> v2:172.21.15.23:3300/0 conn(0x7fcb140aae90 0x7fcb141381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:42:55.852 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:55.849+0000 7fcb127fc700 1 -- 172.21.15.23:0/2014415475 >> v2:172.21.15.155:3300/0 conn(0x7fcb1413aeb0 msgr2=0x7fcb140a8c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:42:55.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:55.849+0000 7fcb127fc700 1 --2- 172.21.15.23:0/2014415475 >> v2:172.21.15.155:3300/0 conn(0x7fcb1413aeb0 0x7fcb140a8c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:55.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:55.849+0000 7fcb127fc700 1 -- 172.21.15.23:0/2014415475 >> v2:172.21.15.23:3300/0 conn(0x7fcb140aae90 msgr2=0x7fcb141381a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:42:55.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:55.849+0000 7fcb127fc700 1 --2- 172.21.15.23:0/2014415475 >> v2:172.21.15.23:3300/0 conn(0x7fcb140aae90 0x7fcb141381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:55.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:55.849+0000 7fcb127fc700 1 -- 172.21.15.23:0/2014415475 >> v1:172.21.15.23:6789/0 conn(0x7fcb140aaab0 legacy=0x7fcb1413ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:55.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:55.849+0000 7fcb127fc700 1 --2- 172.21.15.23:0/2014415475 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcaf40033a0 0x7fcaf40057f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:55.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:55.849+0000 7fcb127fc700 1 -- 172.21.15.23:0/2014415475 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb14138690 con 0x7fcaf4005e30 2020-11-20T11:42:55.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:55.849+0000 7fcb127fc700 1 -- 172.21.15.23:0/2014415475 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1413e8a0 con 0x7fcaf40033a0 2020-11-20T11:42:55.870 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:55.849+0000 7fcb18df3700 1 -- 172.21.15.23:0/2014415475 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcaf40033a0 msgr2=0x7fcaf40057f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:42:55.870 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:55.849+0000 7fcb18df3700 1 --2- 172.21.15.23:0/2014415475 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcaf40033a0 0x7fcaf40057f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:42:56.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.050+0000 7fcb18df3700 1 -- 172.21.15.23:0/2014415475 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcaf40033a0 msgr2=0x7fcaf40057f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:42:56.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.050+0000 7fcb18df3700 1 --2- 172.21.15.23:0/2014415475 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcaf40033a0 0x7fcaf40057f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:42:56.332 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.328+0000 7fcb137fe700 1 -- 172.21.15.23:0/2014415475 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498896210 0 0) 0x7fcb14138690 con 0x7fcaf4005e30 2020-11-20T11:42:56.332 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.328+0000 7fcb137fe700 1 -- 172.21.15.23:0/2014415475 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb00003280 con 0x7fcaf4005e30 2020-11-20T11:42:56.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.343+0000 7fcb137fe700 1 -- 172.21.15.23:0/2014415475 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3065317202 0 0) 0x7fcb00003280 con 0x7fcaf4005e30 2020-11-20T11:42:56.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.343+0000 7fcb137fe700 1 -- 172.21.15.23:0/2014415475 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcaf40033a0 msgr2=0x7fcaf40057f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:42:56.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.343+0000 7fcb137fe700 1 --2- 172.21.15.23:0/2014415475 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcaf40033a0 0x7fcaf40057f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:56.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.343+0000 7fcb137fe700 1 -- 172.21.15.23:0/2014415475 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf400ac30 con 0x7fcaf4005e30 2020-11-20T11:42:56.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.343+0000 7fcb137fe700 1 -- 172.21.15.23:0/2014415475 <== mon.1 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7fcb0800d1c0 con 0x7fcaf4005e30 2020-11-20T11:42:56.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.343+0000 7fcb137fe700 1 -- 172.21.15.23:0/2014415475 >> v1:172.21.15.155:6789/0 conn(0x7fcaf4005e30 legacy=0x7fcaf40082f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:56.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.343+0000 7fcb137fe700 1 --2- 172.21.15.23:0/2014415475 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb00003770 0x7fcb00003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:56.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.343+0000 7fcb137fe700 1 --2- 172.21.15.23:0/2014415475 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb00004080 0x7fcb00004560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:56.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.343+0000 7fcb137fe700 1 --2- 172.21.15.23:0/2014415475 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcb00004c00 0x7fcb0000b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:56.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.343+0000 7fcb137fe700 1 -- 172.21.15.23:0/2014415475 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1413e8a0 con 0x7fcb00003770 2020-11-20T11:42:56.354 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.343+0000 7fcb137fe700 1 -- 172.21.15.23:0/2014415475 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb00005220 con 0x7fcb00004080 2020-11-20T11:42:56.354 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.343+0000 7fcb137fe700 1 -- 172.21.15.23:0/2014415475 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb000053e0 con 0x7fcb00004c00 2020-11-20T11:42:56.355 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.343+0000 7fcb195f4700 1 -- 172.21.15.23:0/2014415475 reap_dead start 2020-11-20T11:42:56.355 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.343+0000 7fcb195f4700 1 -- 172.21.15.23:0/2014415475 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcb00004c00 msgr2=0x7fcb0000b770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:42:56.356 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.343+0000 7fcb195f4700 1 --2- 172.21.15.23:0/2014415475 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcb00004c00 0x7fcb0000b770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:42:56.356 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.344+0000 7fcb18df3700 1 --2- 172.21.15.23:0/2014415475 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb00003770 0x7fcb00003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:56.357 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.344+0000 7fcb13fff700 1 --2- 172.21.15.23:0/2014415475 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb00004080 0x7fcb00004560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:56.361 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.344+0000 7fcb18df3700 1 -- 172.21.15.23:0/2014415475 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcb00004c00 msgr2=0x7fcb0000b770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:42:56.361 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.344+0000 7fcb18df3700 1 --2- 172.21.15.23:0/2014415475 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcb00004c00 0x7fcb0000b770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:56.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.344+0000 7fcb18df3700 1 -- 172.21.15.23:0/2014415475 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb00004080 msgr2=0x7fcb00004560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:42:56.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.344+0000 7fcb18df3700 1 --2- 172.21.15.23:0/2014415475 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb00004080 0x7fcb00004560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:56.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.344+0000 7fcb18df3700 1 -- 172.21.15.23:0/2014415475 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf400ac30 con 0x7fcb00003770 2020-11-20T11:42:56.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.344+0000 7fcb18df3700 1 --2- 172.21.15.23:0/2014415475 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb00003770 0x7fcb00003b70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fcb140b23d0 tx=0x7fcb04007b50).ready entity=mon.0 client_cookie=7977fc3302bec495 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:56.363 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.345+0000 7fcb137fe700 1 -- 172.21.15.23:0/2014415475 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb04001ea0 con 0x7fcb00003770 2020-11-20T11:42:56.363 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.345+0000 7fcb137fe700 1 -- 172.21.15.23:0/2014415475 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb040072e0 con 0x7fcb00003770 2020-11-20T11:42:56.363 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.345+0000 7fcb137fe700 1 -- 172.21.15.23:0/2014415475 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0400c870 con 0x7fcb00003770 2020-11-20T11:42:56.363 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.346+0000 7fcb1b057700 1 -- 172.21.15.23:0/2014415475 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb00003770 msgr2=0x7fcb00003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:56.363 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.346+0000 7fcb1b057700 1 --2- 172.21.15.23:0/2014415475 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb00003770 0x7fcb00003b70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fcb140b23d0 tx=0x7fcb04007b50).stop 2020-11-20T11:42:56.364 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.346+0000 7fcb1b057700 1 -- 172.21.15.23:0/2014415475 shutdown_connections 2020-11-20T11:42:56.364 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.346+0000 7fcb1b057700 1 --2- 172.21.15.23:0/2014415475 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb00004080 0x7fcb00004560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:56.364 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.346+0000 7fcb1b057700 1 --2- 172.21.15.23:0/2014415475 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb00003770 0x7fcb00003b70 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:56.364 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.346+0000 7fcb1b057700 1 --2- 172.21.15.23:0/2014415475 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcb00004c00 0x7fcb0000b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:56.365 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.346+0000 7fcb1b057700 1 -- 172.21.15.23:0/2014415475 >> 172.21.15.23:0/2014415475 conn(0x7fcb140678e0 msgr2=0x7fcb14067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:56.365 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.346+0000 7fcb1b057700 1 -- 172.21.15.23:0/2014415475 shutdown_connections 2020-11-20T11:42:56.365 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.346+0000 7fcb1b057700 1 -- 172.21.15.23:0/2014415475 wait complete. 2020-11-20T11:42:56.365 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.347+0000 7fcb1b057700 1 Processor -- start 2020-11-20T11:42:56.365 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.347+0000 7fcb1b057700 1 -- start start 2020-11-20T11:42:56.366 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.348+0000 7fcb1b057700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcb00003770 0x7fcb1413ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:56.366 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.348+0000 7fcb1b057700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb00004080 0x7fcb140a5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:56.366 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.348+0000 7fcb1b057700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb00004c00 0x7fcb140a6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:56.366 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.348+0000 7fcb1b057700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb140aba00 con 0x7fcb00004c00 2020-11-20T11:42:56.366 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.348+0000 7fcb1b057700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1413c080 con 0x7fcb00004080 2020-11-20T11:42:56.367 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.348+0000 7fcb1b057700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb140a5810 con 0x7fcb00003770 2020-11-20T11:42:56.367 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.348+0000 7fcb18df3700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcb00003770 msgr2=0x7fcb1413ffa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:42:56.368 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.348+0000 7fcb18df3700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcb00003770 0x7fcb1413ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:42:56.368 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.348+0000 7fcb195f4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb00004c00 0x7fcb140a6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:56.369 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.348+0000 7fcb13fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb00004080 0x7fcb140a5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:56.369 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.348+0000 7fcb195f4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb00004c00 0x7fcb140a6130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45066/0 (socket says 172.21.15.23:45066) 2020-11-20T11:42:56.369 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.348+0000 7fcb195f4700 1 -- 172.21.15.23:0/1525523762 learned_addr learned my addr 172.21.15.23:0/1525523762 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:42:56.369 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.348+0000 7fcb13fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb00004080 0x7fcb140a5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:37960/0 (socket says 172.21.15.23:37960) 2020-11-20T11:42:56.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.349+0000 7fcb13fff700 1 -- 172.21.15.23:0/1525523762 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcb00003770 msgr2=0x7fcb1413ffa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:42:56.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.349+0000 7fcb13fff700 1 --2- 172.21.15.23:0/1525523762 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcb00003770 0x7fcb1413ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:56.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.349+0000 7fcb13fff700 1 -- 172.21.15.23:0/1525523762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb00004c00 msgr2=0x7fcb140a6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:56.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.349+0000 7fcb13fff700 1 --2- 172.21.15.23:0/1525523762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb00004c00 0x7fcb140a6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:56.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.349+0000 7fcb13fff700 1 -- 172.21.15.23:0/1525523762 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb140a7dd0 con 0x7fcb00004080 2020-11-20T11:42:56.371 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.349+0000 7fcb13fff700 1 --2- 172.21.15.23:0/1525523762 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb00004080 0x7fcb140a5c40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fcb08007fd0 tx=0x7fcb08008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:56.371 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.349+0000 7fcb10ff9700 1 -- 172.21.15.23:0/1525523762 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb08011d90 con 0x7fcb00004080 2020-11-20T11:42:56.371 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.349+0000 7fcb10ff9700 1 -- 172.21.15.23:0/1525523762 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb08015420 con 0x7fcb00004080 2020-11-20T11:42:56.371 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.349+0000 7fcb1b057700 1 -- 172.21.15.23:0/1525523762 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb140a8070 con 0x7fcb00004080 2020-11-20T11:42:56.372 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.350+0000 7fcb1b057700 1 -- 172.21.15.23:0/1525523762 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb140a7970 con 0x7fcb00004080 2020-11-20T11:42:56.372 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.350+0000 7fcb10ff9700 1 -- 172.21.15.23:0/1525523762 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb080157e0 con 0x7fcb00004080 2020-11-20T11:42:56.372 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.352+0000 7fcb10ff9700 1 -- 172.21.15.23:0/1525523762 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fcb08025070 con 0x7fcb00004080 2020-11-20T11:42:56.372 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.352+0000 7fcb10ff9700 1 --2- 172.21.15.23:0/1525523762 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcaec056d10 0x7fcaec0591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:56.372 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.352+0000 7fcb1b057700 1 -- 172.21.15.23:0/1525523762 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb1405c7d0 con 0x7fcb00004080 2020-11-20T11:42:56.373 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.352+0000 7fcb18df3700 1 --2- 172.21.15.23:0/1525523762 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcaec056d10 0x7fcaec0591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:56.374 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.353+0000 7fcb18df3700 1 --2- 172.21.15.23:0/1525523762 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcaec056d10 0x7fcaec0591d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fcb04000c00 tx=0x7fcb04015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:56.374 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.354+0000 7fcb10ff9700 1 -- 172.21.15.23:0/1525523762 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fcb080499f0 con 0x7fcb00004080 2020-11-20T11:42:56.375 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.360+0000 7fcb10ff9700 1 -- 172.21.15.23:0/1525523762 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fcb08045d90 con 0x7fcb00004080 2020-11-20T11:42:56.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.791+0000 7fcb1b057700 1 -- 172.21.15.23:0/1525523762 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fcb14044710 con 0x7fcb00004080 2020-11-20T11:42:56.795 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.791+0000 7fcb10ff9700 1 -- 172.21.15.23:0/1525523762 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1471 (secure 0 0 0) 0x7fcb080493b0 con 0x7fcb00004080 2020-11-20T11:42:56.795 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":96,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:42:56.797 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.793+0000 7fcb1b057700 1 -- 172.21.15.23:0/1525523762 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcaec056d10 msgr2=0x7fcaec0591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:56.797 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.793+0000 7fcb1b057700 1 --2- 172.21.15.23:0/1525523762 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcaec056d10 0x7fcaec0591d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fcb04000c00 tx=0x7fcb04015000).stop 2020-11-20T11:42:56.797 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.793+0000 7fcb1b057700 1 -- 172.21.15.23:0/1525523762 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb00004080 msgr2=0x7fcb140a5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:56.797 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.793+0000 7fcb1b057700 1 --2- 172.21.15.23:0/1525523762 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb00004080 0x7fcb140a5c40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fcb08007fd0 tx=0x7fcb08008040).stop 2020-11-20T11:42:56.798 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.793+0000 7fcb1b057700 1 -- 172.21.15.23:0/1525523762 shutdown_connections 2020-11-20T11:42:56.798 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.793+0000 7fcb1b057700 1 --2- 172.21.15.23:0/1525523762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb00004c00 0x7fcb140a6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:56.798 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.793+0000 7fcb1b057700 1 --2- 172.21.15.23:0/1525523762 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fcaec056d10 0x7fcaec0591d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:56.798 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.793+0000 7fcb1b057700 1 --2- 172.21.15.23:0/1525523762 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcb00003770 0x7fcb1413ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:56.798 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.793+0000 7fcb1b057700 1 --2- 172.21.15.23:0/1525523762 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcb00004080 0x7fcb140a5c40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:56.799 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.793+0000 7fcb1b057700 1 -- 172.21.15.23:0/1525523762 >> 172.21.15.23:0/1525523762 conn(0x7fcb140678e0 msgr2=0x7fcb141367f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:56.799 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.794+0000 7fcb1b057700 1 -- 172.21.15.23:0/1525523762 shutdown_connections 2020-11-20T11:42:56.799 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:56.796+0000 7fcb1b057700 1 -- 172.21.15.23:0/1525523762 wait complete. 2020-11-20T11:42:56.825 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2020-11-20T11:42:56.825 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:42:57.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.341+0000 7f9952aee700 1 Processor -- start 2020-11-20T11:42:57.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.341+0000 7f9952aee700 1 -- start start 2020-11-20T11:42:57.345 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.342+0000 7f9952aee700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f994c060a80 0x7f994c09d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:57.346 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.342+0000 7f9952aee700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f994c067fc0 0x7f994c0abbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:57.346 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.342+0000 7f9952aee700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f994c0a9b40 con 0x7f994c09ded0 2020-11-20T11:42:57.346 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.342+0000 7f9952aee700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994c0aacb0 con 0x7f994c060a80 2020-11-20T11:42:57.346 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.342+0000 7f9952aee700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994c0a9da0 con 0x7f994c067fc0 2020-11-20T11:42:57.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.342+0000 7f995088a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f994c060a80 0x7f994c09d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:57.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.342+0000 7f995088a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f994c060a80 0x7f994c09d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45070/0 (socket says 172.21.15.23:45070) 2020-11-20T11:42:57.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.342+0000 7f995088a700 1 -- 172.21.15.23:0/2032007544 learned_addr learned my addr 172.21.15.23:0/2032007544 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:42:57.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.343+0000 7f995088a700 1 -- 172.21.15.23:0/2032007544 >> v2:172.21.15.155:3300/0 conn(0x7f994c067fc0 msgr2=0x7f994c0abbd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:42:57.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.343+0000 7f995088a700 1 --2- 172.21.15.23:0/2032007544 >> v2:172.21.15.155:3300/0 conn(0x7f994c067fc0 0x7f994c0abbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:57.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.343+0000 7f995088a700 1 -- 172.21.15.23:0/2032007544 >> v1:172.21.15.23:6789/0 conn(0x7f994c09ded0 legacy=0x7f994c0678b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:57.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.343+0000 7f995088a700 1 -- 172.21.15.23:0/2032007544 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994c0aa040 con 0x7f994c060a80 2020-11-20T11:42:57.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.346+0000 7f995088a700 1 --2- 172.21.15.23:0/2032007544 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f994c060a80 0x7f994c09d9e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f994c09d8c0 tx=0x7f9940006ef0).ready entity=mon.0 client_cookie=27ca0987e348056d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:57.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.346+0000 7f994b7fe700 1 -- 172.21.15.23:0/2032007544 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994000b760 con 0x7f994c060a80 2020-11-20T11:42:57.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.346+0000 7f994b7fe700 1 -- 172.21.15.23:0/2032007544 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9940000e90 con 0x7f994c060a80 2020-11-20T11:42:57.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.346+0000 7f994b7fe700 1 -- 172.21.15.23:0/2032007544 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994000bad0 con 0x7f994c060a80 2020-11-20T11:42:57.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.347+0000 7f9952aee700 1 -- 172.21.15.23:0/2032007544 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f994c060a80 msgr2=0x7f994c09d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:57.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.347+0000 7f9952aee700 1 --2- 172.21.15.23:0/2032007544 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f994c060a80 0x7f994c09d9e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f994c09d8c0 tx=0x7f9940006ef0).stop 2020-11-20T11:42:57.356 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.353+0000 7f9952aee700 1 -- 172.21.15.23:0/2032007544 shutdown_connections 2020-11-20T11:42:57.357 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.353+0000 7f9952aee700 1 --2- 172.21.15.23:0/2032007544 >> v2:172.21.15.155:3300/0 conn(0x7f994c067fc0 0x7f994c0abbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:57.357 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.353+0000 7f9952aee700 1 --2- 172.21.15.23:0/2032007544 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f994c060a80 0x7f994c09d9e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:57.357 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.353+0000 7f9952aee700 1 -- 172.21.15.23:0/2032007544 >> 172.21.15.23:0/2032007544 conn(0x7f994c066730 msgr2=0x7f994c0601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:57.360 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.357+0000 7f9952aee700 1 -- 172.21.15.23:0/2032007544 shutdown_connections 2020-11-20T11:42:57.360 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.357+0000 7f9952aee700 1 -- 172.21.15.23:0/2032007544 wait complete. 2020-11-20T11:42:57.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.359+0000 7f9952aee700 1 Processor -- start 2020-11-20T11:42:57.366 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.363+0000 7f9952aee700 1 -- start start 2020-11-20T11:42:57.368 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.364+0000 7f9952aee700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c060a80 0x7f994c14e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:57.368 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.364+0000 7f9952aee700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f994c067fc0 0x7f994c0a7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:57.369 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.364+0000 7f9952aee700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f994c09ded0 0x7f994c12f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:57.369 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.364+0000 7f9952aee700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c0a6830 con 0x7f994c09ded0 2020-11-20T11:42:57.369 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.364+0000 7f9952aee700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c136910 con 0x7f994c067fc0 2020-11-20T11:42:57.369 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.364+0000 7f9952aee700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994c061c40 con 0x7f994c060a80 2020-11-20T11:42:57.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.365+0000 7f995088a700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c060a80 msgr2=0x7f994c14e680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:42:57.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.365+0000 7f995088a700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c060a80 0x7f994c14e680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:42:57.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.365+0000 7f994bfff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f994c067fc0 0x7f994c0a7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:57.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.365+0000 7f994bfff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f994c067fc0 0x7f994c0a7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:37972/0 (socket says 172.21.15.23:37972) 2020-11-20T11:42:57.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.365+0000 7f994bfff700 1 -- 172.21.15.23:0/973295918 learned_addr learned my addr 172.21.15.23:0/973295918 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:42:57.371 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.365+0000 7f995108b700 1 --2- 172.21.15.23:0/973295918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f994c09ded0 0x7f994c12f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:57.371 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.365+0000 7f994bfff700 1 -- 172.21.15.23:0/973295918 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c060a80 msgr2=0x7f994c14e680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:42:57.371 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.365+0000 7f994bfff700 1 --2- 172.21.15.23:0/973295918 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c060a80 0x7f994c14e680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:57.371 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.365+0000 7f994bfff700 1 -- 172.21.15.23:0/973295918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f994c09ded0 msgr2=0x7f994c12f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:57.372 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.365+0000 7f994bfff700 1 --2- 172.21.15.23:0/973295918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f994c09ded0 0x7f994c12f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:57.372 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.365+0000 7f994bfff700 1 -- 172.21.15.23:0/973295918 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994c061ee0 con 0x7f994c067fc0 2020-11-20T11:42:57.372 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.365+0000 7f994bfff700 1 --2- 172.21.15.23:0/973295918 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f994c067fc0 0x7f994c0a7b40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f993800c7e0 tx=0x7f9938007c40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:57.373 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.367+0000 7f9948ff9700 1 -- 172.21.15.23:0/973295918 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9938010510 con 0x7f994c067fc0 2020-11-20T11:42:57.373 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.367+0000 7f9948ff9700 1 -- 172.21.15.23:0/973295918 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9938010670 con 0x7f994c067fc0 2020-11-20T11:42:57.374 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.367+0000 7f9952aee700 1 -- 172.21.15.23:0/973295918 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f994c14f600 con 0x7f994c067fc0 2020-11-20T11:42:57.374 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.367+0000 7f9948ff9700 1 -- 172.21.15.23:0/973295918 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9938010a10 con 0x7f994c067fc0 2020-11-20T11:42:57.374 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.367+0000 7f9952aee700 1 -- 172.21.15.23:0/973295918 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f994c136230 con 0x7f994c067fc0 2020-11-20T11:42:57.374 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.368+0000 7f9948ff9700 1 -- 172.21.15.23:0/973295918 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9938010070 con 0x7f994c067fc0 2020-11-20T11:42:57.375 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.369+0000 7f9948ff9700 1 --2- 172.21.15.23:0/973295918 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9924056d60 0x7f9924059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:57.375 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.369+0000 7f9948ff9700 1 -- 172.21.15.23:0/973295918 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f993806f850 con 0x7f994c067fc0 2020-11-20T11:42:57.375 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.369+0000 7f995088a700 1 --2- 172.21.15.23:0/973295918 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9924056d60 0x7f9924059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:57.375 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.369+0000 7f995088a700 1 --2- 172.21.15.23:0/973295918 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9924056d60 0x7f9924059220 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f9940006f30 tx=0x7f9940016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:57.376 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.373+0000 7f9952aee700 1 --2- 172.21.15.23:0/973295918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f994c063020 0x7f994c0622a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:57.376 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.373+0000 7f9952aee700 1 -- 172.21.15.23:0/973295918 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c14ec80 con 0x7f994c063020 2020-11-20T11:42:57.376 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.373+0000 7f995108b700 1 --2- 172.21.15.23:0/973295918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f994c063020 0x7f994c0622a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:57.379 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.375+0000 7f995108b700 1 -- 172.21.15.23:0/973295918 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f994c062070 con 0x7f994c063020 2020-11-20T11:42:57.379 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.375+0000 7f995108b700 1 --2- 172.21.15.23:0/973295918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f994c063020 0x7f994c0622a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f994c0ab3a0 tx=0x7f99440090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:57.379 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.376+0000 7f9948ff9700 1 -- 172.21.15.23:0/973295918 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994400f940 con 0x7f994c063020 2020-11-20T11:42:57.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.376+0000 7f9948ff9700 1 -- 172.21.15.23:0/973295918 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f994400faa0 con 0x7f994c063020 2020-11-20T11:42:57.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.376+0000 7f9948ff9700 1 -- 172.21.15.23:0/973295918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f994c063020 msgr2=0x7f994c0622a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:57.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.376+0000 7f9948ff9700 1 --2- 172.21.15.23:0/973295918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f994c063020 0x7f994c0622a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f994c0ab3a0 tx=0x7f99440090f0).stop 2020-11-20T11:42:57.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.376+0000 7f9948ff9700 1 -- 172.21.15.23:0/973295918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f994c063020 msgr2=0x7f994c0622a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:42:57.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.376+0000 7f9948ff9700 1 --2- 172.21.15.23:0/973295918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f994c063020 0x7f994c0622a0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:57.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.387+0000 7f9952aee700 1 --2- 172.21.15.23:0/973295918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f994c151bf0 0x7f994c152000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:57.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.388+0000 7f995108b700 1 --2- 172.21.15.23:0/973295918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f994c151bf0 0x7f994c152000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:57.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.388+0000 7f9952aee700 1 -- 172.21.15.23:0/973295918 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c0a66e0 con 0x7f994c151bf0 2020-11-20T11:42:57.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.388+0000 7f995108b700 1 -- 172.21.15.23:0/973295918 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f994c0a6310 con 0x7f994c151bf0 2020-11-20T11:42:57.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.389+0000 7f995108b700 1 --2- 172.21.15.23:0/973295918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f994c151bf0 0x7f994c152000 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f9944016000 tx=0x7f9944019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:57.394 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.391+0000 7f9948ff9700 1 -- 172.21.15.23:0/973295918 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9944024480 con 0x7f994c151bf0 2020-11-20T11:42:57.395 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.392+0000 7f9948ff9700 1 -- 172.21.15.23:0/973295918 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4774 (secure 0 0 0) 0x7f994400fc10 con 0x7f994c151bf0 2020-11-20T11:42:57.395 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.392+0000 7f9948ff9700 1 -- 172.21.15.23:0/973295918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f994c151bf0 msgr2=0x7f994c152000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:57.395 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.392+0000 7f9948ff9700 1 --2- 172.21.15.23:0/973295918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f994c151bf0 0x7f994c152000 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f9944016000 tx=0x7f9944019000).stop 2020-11-20T11:42:57.396 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.392+0000 7f9948ff9700 1 -- 172.21.15.23:0/973295918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f994c151bf0 msgr2=0x7f994c152000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:42:57.396 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.392+0000 7f9948ff9700 1 --2- 172.21.15.23:0/973295918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f994c151bf0 0x7f994c152000 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:57.400 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:42:57.400 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:42:57.400 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:42:57.401 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:42:57.401 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 96, 2020-11-20T11:42:57.401 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:42:57.402 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:42:57.402 INFO:teuthology.orchestra.run.smithi023.stdout: 1 2020-11-20T11:42:57.402 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:57.402 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 1, 2020-11-20T11:42:57.402 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:42:57.403 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:42:57.403 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:42:57.403 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:42:57.403 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:42:57.404 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:42:57.404 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:42:57.404 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:42:57.404 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:42:57.404 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:42:57.405 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:42:57.405 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:57.405 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:42:57.405 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:42:57.406 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:42:57.406 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:42:57.406 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:42:57.406 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:42:57.406 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:42:57.407 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:42:57.407 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:42:57.407 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:42:57.407 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:57.407 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:57.408 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:42:57.408 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:42:57.408 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:42:57.408 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:42:57.409 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:42:57.409 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:42:57.410 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:42:57.410 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:42:57.410 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:42:57.410 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:42:57.410 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:42:57.411 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:42:57.411 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:42:57.411 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:42:57.411 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:42:57.412 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:42:57.412 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:42:57.412 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:42:57.412 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:42:57.413 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:42:57.413 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:42:57.413 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:42:57.414 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:42:57.414 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:57.414 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:42:57.414 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:57.415 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:42:57.415 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:57.415 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:42:57.415 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:42:57.416 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:42:57.416 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:42:57.416 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:57.416 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:42:57.417 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:42:57.417 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:57.417 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:57.417 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:57.420 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:42:57.420 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:42:57.420 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:57.420 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:57.421 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:57.421 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:57.421 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:42:57.421 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:42:57.422 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:42:57.422 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:42:57.422 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:42:57.423 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:57.423 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:57.423 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:42:57.424 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:42:57.424 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:42:57.425 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:42:57.425 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:57.425 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:42:57.425 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:42:57.426 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:57.426 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:57.426 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:57.426 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:42:57.427 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:42:57.427 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:57.427 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:57.427 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:57.428 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:57.428 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:42:57.428 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:42:57.429 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:42:57.429 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:42:57.429 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:42:57.430 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:57.430 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:57.430 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:42:57.431 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:42:57.431 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:42:57.432 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:42:57.432 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:57.432 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:42:57.433 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:42:57.433 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:57.433 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:57.434 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:57.434 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:42:57.434 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:42:57.434 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:57.435 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:57.435 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:57.435 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:57.435 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:42:57.436 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:42:57.436 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:42:57.436 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:42:57.436 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:42:57.437 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:57.437 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:57.438 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:57.438 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:42:57.438 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:42:57.438 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:57.439 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:57.439 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:57.439 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:42:57.440 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:57.440 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:57.440 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:42:57.441 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:57.441 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:57.441 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:57.442 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:42:57.442 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:57.442 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:57.443 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:42:57.443 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:57.443 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:57.444 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:57.444 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 8 2020-11-20T11:42:57.444 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:57.444 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:57.445 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:42:57.445 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:57.445 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:57.445 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:57.446 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 4 2020-11-20T11:42:57.446 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:57.446 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:57.446 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:42:57.447 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:57.447 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:57.447 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:57.448 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:42:57.448 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:57.448 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:57.448 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:57.449 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:42:57.449 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:42:57.450 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.397+0000 7f9952aee700 1 -- 172.21.15.23:0/973295918 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9924056d60 msgr2=0x7f9924059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:57.451 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.397+0000 7f9952aee700 1 --2- 172.21.15.23:0/973295918 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9924056d60 0x7f9924059220 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f9940006f30 tx=0x7f9940016000).stop 2020-11-20T11:42:57.451 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.398+0000 7f995108b700 1 -- 172.21.15.23:0/973295918 reap_dead start 2020-11-20T11:42:57.451 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.398+0000 7f9952aee700 1 -- 172.21.15.23:0/973295918 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f994c067fc0 msgr2=0x7f994c0a7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:57.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.398+0000 7f9952aee700 1 --2- 172.21.15.23:0/973295918 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f994c067fc0 0x7f994c0a7b40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f993800c7e0 tx=0x7f9938007c40).stop 2020-11-20T11:42:57.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.398+0000 7f9952aee700 1 -- 172.21.15.23:0/973295918 shutdown_connections 2020-11-20T11:42:57.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.398+0000 7f9952aee700 1 --2- 172.21.15.23:0/973295918 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f994c067fc0 0x7f994c0a7b40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:57.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.398+0000 7f9952aee700 1 -- 172.21.15.23:0/973295918 >> 172.21.15.23:0/973295918 conn(0x7f994c066730 msgr2=0x7f994c068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:57.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.398+0000 7f9952aee700 1 -- 172.21.15.23:0/973295918 shutdown_connections 2020-11-20T11:42:57.453 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.398+0000 7f9952aee700 1 -- 172.21.15.23:0/973295918 wait complete. 2020-11-20T11:42:57.453 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:42:57.653 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.650+0000 7fed1ff36700 1 Processor -- start 2020-11-20T11:42:57.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.651+0000 7fed1ff36700 1 -- start start 2020-11-20T11:42:57.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.652+0000 7fed1ff36700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fed180a8c70 0x7fed1813c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:57.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.652+0000 7fed1ff36700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed180a9690 con 0x7fed180aac70 2020-11-20T11:42:57.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.652+0000 7fed1ff36700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed180a8480 con 0x7fed181432b0 2020-11-20T11:42:57.656 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.652+0000 7fed1ff36700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed180a9d60 con 0x7fed180a8c70 2020-11-20T11:42:57.656 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.652+0000 7fed1dcd2700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fed180aac70 0x7fed18142ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:59336/0 (socket says 172.21.15.23:59336) 2020-11-20T11:42:57.656 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.652+0000 7fed1dcd2700 1 -- 172.21.15.23:0/2484630773 learned_addr learned my addr 172.21.15.23:0/2484630773 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:42:57.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.652+0000 7fed1ccd0700 1 -- 172.21.15.23:0/2484630773 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37388049 0 0) 0x7fed180a9690 con 0x7fed180aac70 2020-11-20T11:42:57.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.652+0000 7fed1ccd0700 1 -- 172.21.15.23:0/2484630773 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed00003280 con 0x7fed180aac70 2020-11-20T11:42:57.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.652+0000 7fed1ccd0700 1 -- 172.21.15.23:0/2484630773 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1852994625 0 0) 0x7fed180a8480 con 0x7fed181432b0 2020-11-20T11:42:57.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.652+0000 7fed1ccd0700 1 -- 172.21.15.23:0/2484630773 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed180a9690 con 0x7fed181432b0 2020-11-20T11:42:57.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.653+0000 7fed1ccd0700 1 -- 172.21.15.23:0/2484630773 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4119887844 0 0) 0x7fed00003280 con 0x7fed180aac70 2020-11-20T11:42:57.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.653+0000 7fed1ccd0700 1 -- 172.21.15.23:0/2484630773 >> v2:172.21.15.155:3300/0 conn(0x7fed180a8c70 msgr2=0x7fed1813c210 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2020-11-20T11:42:57.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.653+0000 7fed1ccd0700 1 --2- 172.21.15.23:0/2484630773 >> v2:172.21.15.155:3300/0 conn(0x7fed180a8c70 0x7fed1813c210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:57.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.653+0000 7fed1ccd0700 1 -- 172.21.15.23:0/2484630773 >> v1:172.21.15.155:6789/0 conn(0x7fed181432b0 legacy=0x7fed1813c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:57.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.653+0000 7fed1ccd0700 1 -- 172.21.15.23:0/2484630773 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed180a9150 con 0x7fed180aac70 2020-11-20T11:42:57.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.653+0000 7fed1ccd0700 1 -- 172.21.15.23:0/2484630773 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7fed14001520 con 0x7fed180aac70 2020-11-20T11:42:57.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.653+0000 7fed1ccd0700 1 -- 172.21.15.23:0/2484630773 >> v1:172.21.15.23:6789/0 conn(0x7fed180aac70 legacy=0x7fed18142ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:57.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.653+0000 7fed1ccd0700 1 --2- 172.21.15.23:0/2484630773 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed000049b0 0x7fed00004db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:57.661 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.653+0000 7fed1ccd0700 1 --2- 172.21.15.23:0/2484630773 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fed000034c0 0x7fed00003940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:57.661 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.653+0000 7fed1ccd0700 1 --2- 172.21.15.23:0/2484630773 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed00003e30 0x7fed0000b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:57.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.653+0000 7fed1ccd0700 1 -- 172.21.15.23:0/2484630773 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed180a9d60 con 0x7fed000049b0 2020-11-20T11:42:57.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.653+0000 7fed1ccd0700 1 -- 172.21.15.23:0/2484630773 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0000bc60 con 0x7fed00003e30 2020-11-20T11:42:57.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.653+0000 7fed1ccd0700 1 -- 172.21.15.23:0/2484630773 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0000bda0 con 0x7fed000034c0 2020-11-20T11:42:57.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.653+0000 7fed1dcd2700 1 --2- 172.21.15.23:0/2484630773 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed000049b0 0x7fed00004db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:57.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.654+0000 7fed1dcd2700 1 -- 172.21.15.23:0/2484630773 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fed000034c0 msgr2=0x7fed00003940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:42:57.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.654+0000 7fed1dcd2700 1 --2- 172.21.15.23:0/2484630773 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fed000034c0 0x7fed00003940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:57.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.654+0000 7fed1dcd2700 1 -- 172.21.15.23:0/2484630773 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed00003e30 msgr2=0x7fed0000b770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:42:57.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.654+0000 7fed1dcd2700 1 --2- 172.21.15.23:0/2484630773 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed00003e30 0x7fed0000b770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:57.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.654+0000 7fed1dcd2700 1 -- 172.21.15.23:0/2484630773 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed00003280 con 0x7fed000049b0 2020-11-20T11:42:57.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.654+0000 7fed1dcd2700 1 --2- 172.21.15.23:0/2484630773 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed000049b0 0x7fed00004db0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fed1400c230 tx=0x7fed18142890).ready entity=mon.0 client_cookie=f4764432af199ca3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:57.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.654+0000 7fed1ccd0700 1 -- 172.21.15.23:0/2484630773 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed140070d0 con 0x7fed000049b0 2020-11-20T11:42:57.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.654+0000 7fed1ccd0700 1 -- 172.21.15.23:0/2484630773 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed14007230 con 0x7fed000049b0 2020-11-20T11:42:57.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.654+0000 7fed1ccd0700 1 -- 172.21.15.23:0/2484630773 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed14005630 con 0x7fed000049b0 2020-11-20T11:42:57.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.655+0000 7fed1e4d3700 1 -- 172.21.15.23:0/2484630773 reap_dead start 2020-11-20T11:42:57.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.655+0000 7fed1ff36700 1 -- 172.21.15.23:0/2484630773 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed000049b0 msgr2=0x7fed00004db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:57.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.655+0000 7fed1ff36700 1 --2- 172.21.15.23:0/2484630773 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed000049b0 0x7fed00004db0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fed1400c230 tx=0x7fed18142890).stop 2020-11-20T11:42:57.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.655+0000 7fed1ff36700 1 -- 172.21.15.23:0/2484630773 shutdown_connections 2020-11-20T11:42:57.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.655+0000 7fed1ff36700 1 --2- 172.21.15.23:0/2484630773 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed000049b0 0x7fed00004db0 secure :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0x7fed1400c230 tx=0x7fed18142890).stop 2020-11-20T11:42:57.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.655+0000 7fed1ff36700 1 -- 172.21.15.23:0/2484630773 >> 172.21.15.23:0/2484630773 conn(0x7fed180678e0 msgr2=0x7fed18067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:57.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.686+0000 7fed1ff36700 1 -- 172.21.15.23:0/2484630773 shutdown_connections 2020-11-20T11:42:57.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.687+0000 7fed1ff36700 1 -- 172.21.15.23:0/2484630773 wait complete. 2020-11-20T11:42:57.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.688+0000 7fed1ff36700 1 Processor -- start 2020-11-20T11:42:57.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.688+0000 7fed1ff36700 1 -- start start 2020-11-20T11:42:57.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.689+0000 7fed1ff36700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed18141f10 0x7fed181575a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:57.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.689+0000 7fed1ff36700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed180b1fe0 0x7fed18157a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:57.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.689+0000 7fed1ff36700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fed180a5170 0x7fed18154350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:57.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.689+0000 7fed1ff36700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed18157f80 con 0x7fed180b1fe0 2020-11-20T11:42:57.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.689+0000 7fed1ff36700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed180a7c80 con 0x7fed18141f10 2020-11-20T11:42:57.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.689+0000 7fed1ff36700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed180a7f20 con 0x7fed180a5170 2020-11-20T11:42:57.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.689+0000 7fed1e4d3700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fed180a5170 msgr2=0x7fed18154350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:42:57.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.689+0000 7fed1e4d3700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fed180a5170 0x7fed18154350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:42:57.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.689+0000 7fed1d4d1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed180b1fe0 0x7fed18157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:57.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.689+0000 7fed1d4d1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed180b1fe0 0x7fed18157a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45094/0 (socket says 172.21.15.23:45094) 2020-11-20T11:42:57.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.689+0000 7fed1d4d1700 1 -- 172.21.15.23:0/455924193 learned_addr learned my addr 172.21.15.23:0/455924193 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:42:57.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.690+0000 7fed1dcd2700 1 --2- 172.21.15.23:0/455924193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed18141f10 0x7fed181575a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:57.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.690+0000 7fed1d4d1700 1 -- 172.21.15.23:0/455924193 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fed180a5170 msgr2=0x7fed18154350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:42:57.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.690+0000 7fed1d4d1700 1 --2- 172.21.15.23:0/455924193 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fed180a5170 0x7fed18154350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:42:57.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.690+0000 7fed1d4d1700 1 -- 172.21.15.23:0/455924193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed18141f10 msgr2=0x7fed181575a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:57.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.690+0000 7fed1d4d1700 1 --2- 172.21.15.23:0/455924193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed18141f10 0x7fed181575a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:57.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.690+0000 7fed1d4d1700 1 -- 172.21.15.23:0/455924193 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed180a81c0 con 0x7fed180b1fe0 2020-11-20T11:42:57.699 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.690+0000 7fed1d4d1700 1 --2- 172.21.15.23:0/455924193 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed180b1fe0 0x7fed18157a90 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fed08009660 tx=0x7fed08006cf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:57.699 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.690+0000 7fed05ffb700 1 -- 172.21.15.23:0/455924193 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed08012410 con 0x7fed180b1fe0 2020-11-20T11:42:57.699 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.691+0000 7fed1ff36700 1 -- 172.21.15.23:0/455924193 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed181422f0 con 0x7fed180b1fe0 2020-11-20T11:42:57.699 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.691+0000 7fed1ff36700 1 -- 172.21.15.23:0/455924193 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed18154840 con 0x7fed180b1fe0 2020-11-20T11:42:57.700 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.691+0000 7fed05ffb700 1 -- 172.21.15.23:0/455924193 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed08012570 con 0x7fed180b1fe0 2020-11-20T11:42:57.700 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.691+0000 7fed05ffb700 1 -- 172.21.15.23:0/455924193 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed08012910 con 0x7fed180b1fe0 2020-11-20T11:42:57.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.692+0000 7fed05ffb700 1 -- 172.21.15.23:0/455924193 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fed08018020 con 0x7fed180b1fe0 2020-11-20T11:42:57.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.693+0000 7fed05ffb700 1 --2- 172.21.15.23:0/455924193 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fecf0056d10 0x7fecf00591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:57.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.693+0000 7fed05ffb700 1 -- 172.21.15.23:0/455924193 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fed0806f980 con 0x7fed180b1fe0 2020-11-20T11:42:57.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.693+0000 7fed1dcd2700 1 --2- 172.21.15.23:0/455924193 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fecf0056d10 0x7fecf00591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:57.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.694+0000 7fed1dcd2700 1 --2- 172.21.15.23:0/455924193 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fecf0056d10 0x7fecf00591d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fed180ae590 tx=0x7fed14013000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:57.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.698+0000 7fed1ff36700 1 --2- 172.21.15.23:0/455924193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed180601b0 0x7fed180b0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:57.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.698+0000 7fed1ff36700 1 -- 172.21.15.23:0/455924193 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed180a8350 con 0x7fed180601b0 2020-11-20T11:42:57.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.698+0000 7fed1e4d3700 1 --2- 172.21.15.23:0/455924193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed180601b0 0x7fed180b0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:57.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.699+0000 7fed1e4d3700 1 -- 172.21.15.23:0/455924193 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fed1813e260 con 0x7fed180601b0 2020-11-20T11:42:57.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.699+0000 7fed1e4d3700 1 --2- 172.21.15.23:0/455924193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed180601b0 0x7fed180b0210 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fed180acf20 tx=0x7fed10007dd0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:57.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.699+0000 7fed05ffb700 1 -- 172.21.15.23:0/455924193 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1000ce70 con 0x7fed180601b0 2020-11-20T11:42:57.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.699+0000 7fed05ffb700 1 -- 172.21.15.23:0/455924193 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fed10002240 con 0x7fed180601b0 2020-11-20T11:42:57.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.699+0000 7fed05ffb700 1 -- 172.21.15.23:0/455924193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed180601b0 msgr2=0x7fed180b0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:57.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.699+0000 7fed05ffb700 1 --2- 172.21.15.23:0/455924193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed180601b0 0x7fed180b0210 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fed180acf20 tx=0x7fed10007dd0).stop 2020-11-20T11:42:57.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.699+0000 7fed05ffb700 1 -- 172.21.15.23:0/455924193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed180601b0 msgr2=0x7fed180b0210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:42:57.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.699+0000 7fed05ffb700 1 --2- 172.21.15.23:0/455924193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed180601b0 0x7fed180b0210 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:42:57.710 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.707+0000 7fed1ff36700 1 --2- 172.21.15.23:0/455924193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed18154b40 0x7fed180b0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:42:57.710 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.707+0000 7fed1ff36700 1 -- 172.21.15.23:0/455924193 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1813e700 con 0x7fed18154b40 2020-11-20T11:42:57.710 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.707+0000 7fed1e4d3700 1 --2- 172.21.15.23:0/455924193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed18154b40 0x7fed180b0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:42:57.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.708+0000 7fed1e4d3700 1 -- 172.21.15.23:0/455924193 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fed18142ec0 con 0x7fed18154b40 2020-11-20T11:42:57.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.708+0000 7fed1e4d3700 1 --2- 172.21.15.23:0/455924193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed18154b40 0x7fed180b0700 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fed10007fd0 tx=0x7fed10014000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:42:57.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.708+0000 7fed05ffb700 1 -- 172.21.15.23:0/455924193 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed100076b0 con 0x7fed18154b40 2020-11-20T11:42:57.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.708+0000 7fed05ffb700 1 -- 172.21.15.23:0/455924193 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4432 (secure 0 0 0) 0x7fed10002030 con 0x7fed18154b40 2020-11-20T11:42:57.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.708+0000 7fed05ffb700 1 -- 172.21.15.23:0/455924193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed18154b40 msgr2=0x7fed180b0700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:57.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.708+0000 7fed05ffb700 1 --2- 172.21.15.23:0/455924193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed18154b40 0x7fed180b0700 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fed10007fd0 tx=0x7fed10014000).stop 2020-11-20T11:42:57.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.708+0000 7fed05ffb700 1 -- 172.21.15.23:0/455924193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed18154b40 msgr2=0x7fed180b0700 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:42:57.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.708+0000 7fed05ffb700 1 --2- 172.21.15.23:0/455924193 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fed18154b40 0x7fed180b0700 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0x7fed10007fd0 tx=0x7fed10014000).stop 2020-11-20T11:42:57.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.708+0000 7fed1ff36700 1 -- 172.21.15.23:0/455924193 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fecf0056d10 msgr2=0x7fecf00591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:57.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.708+0000 7fed1ff36700 1 --2- 172.21.15.23:0/455924193 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fecf0056d10 0x7fecf00591d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fed180ae590 tx=0x7fed14013000).stop 2020-11-20T11:42:57.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.708+0000 7fed1ff36700 1 -- 172.21.15.23:0/455924193 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed180b1fe0 msgr2=0x7fed18157a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:42:57.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.708+0000 7fed1ff36700 1 --2- 172.21.15.23:0/455924193 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed180b1fe0 0x7fed18157a90 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fed08009660 tx=0x7fed08006cf0).stop 2020-11-20T11:42:57.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.708+0000 7fed1e4d3700 1 -- 172.21.15.23:0/455924193 reap_dead start 2020-11-20T11:42:57.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.708+0000 7fed1e4d3700 1 -- 172.21.15.23:0/455924193 reap_dead start 2020-11-20T11:42:57.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.708+0000 7fed1ff36700 1 -- 172.21.15.23:0/455924193 shutdown_connections 2020-11-20T11:42:57.716 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.708+0000 7fed1ff36700 1 -- 172.21.15.23:0/455924193 >> 172.21.15.23:0/455924193 conn(0x7fed180678e0 msgr2=0x7fed18139230 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:42:57.716 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.708+0000 7fed1ff36700 1 -- 172.21.15.23:0/455924193 shutdown_connections 2020-11-20T11:42:57.716 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:42:57.709+0000 7fed1ff36700 1 -- 172.21.15.23:0/455924193 wait complete. 2020-11-20T11:42:57.716 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:42:57.716 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:42:57.717 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:42:57.717 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:42:57.717 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 96, 2020-11-20T11:42:57.717 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:42:57.718 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:42:57.718 INFO:teuthology.orchestra.run.smithi023.stdout: 1 2020-11-20T11:42:57.718 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:57.718 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 1, 2020-11-20T11:42:57.718 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:42:57.719 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:42:57.719 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:42:57.719 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:42:57.719 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:42:57.719 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:42:57.720 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:42:57.720 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:42:57.720 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:42:57.720 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:42:57.720 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:42:57.721 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:57.721 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:42:57.722 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:42:57.722 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:42:57.722 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:42:57.722 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:42:57.722 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:42:57.723 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:42:57.723 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:42:57.723 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:42:57.723 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:42:57.724 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:57.724 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:57.724 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:42:57.724 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:42:57.724 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:42:57.725 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:42:57.725 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:42:57.725 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:42:57.725 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:42:57.725 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:42:57.726 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:42:57.726 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:42:57.726 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:42:57.726 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:42:57.726 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:42:57.727 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:42:57.727 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:42:57.727 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:42:57.727 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:42:57.727 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:42:57.728 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:42:57.728 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:42:57.728 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:42:57.728 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:42:57.728 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:42:57.729 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:57.729 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:42:57.729 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:57.729 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:42:57.729 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:57.730 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:42:57.730 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:42:57.730 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:42:57.730 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:42:57.730 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:57.731 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:42:57.731 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:42:57.731 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:57.731 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:57.731 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:57.732 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:42:57.732 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:42:57.732 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:57.732 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:57.732 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:57.733 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:57.733 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:42:57.733 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:42:57.733 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:42:57.733 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:42:57.734 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:42:57.734 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:57.734 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:57.734 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:42:57.734 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:42:57.735 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:42:57.735 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:42:57.735 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:57.735 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:42:57.735 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:42:57.736 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:57.736 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:57.736 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:57.736 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:42:57.736 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:42:57.737 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:57.737 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:57.737 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:57.737 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:57.738 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:42:57.738 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:42:57.738 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:42:57.738 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:42:57.738 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:42:57.739 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:57.739 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:57.739 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:42:57.739 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:42:57.739 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:42:57.740 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:42:57.740 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:57.740 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:42:57.740 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:42:57.740 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:57.741 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:57.741 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:57.741 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:42:57.741 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:42:57.741 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:42:57.742 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:57.742 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:57.742 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:57.742 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:42:57.742 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:42:57.743 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:42:57.743 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:42:57.743 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:42:57.743 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:57.743 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:57.744 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:57.744 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:42:57.744 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:42:57.744 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:57.744 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:57.745 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:57.745 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:42:57.745 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:57.745 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:57.745 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:42:57.746 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:57.746 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:57.746 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:57.746 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:42:57.746 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:57.747 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:42:57.747 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:42:57.747 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:42:57.747 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:42:57.747 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:42:57.748 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:42:57.748 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:42:57.748 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:42:57.748 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:42:57.748 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:42:57.749 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:42:57.750 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2020-11-20T11:43:04.793 INFO:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:43:04.796 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:43:17.751 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2020-11-20T11:43:17.752 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2020-11-20T11:43:17.752 INFO:tasks.ceph.mon.c:Restarting daemon 2020-11-20T11:43:17.752 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2020-11-20T11:43:17.756 INFO:tasks.ceph.mon.c:Started 2020-11-20T11:43:17.757 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:43:17.757 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:43:18.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.697+0000 7f630359e700 1 Processor -- start 2020-11-20T11:43:18.707 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.697+0000 7f630359e700 1 -- start start 2020-11-20T11:43:18.707 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.698+0000 7f630359e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63040ad1e0 0x7f63040a60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:18.707 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.698+0000 7f630359e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6304136960 0x7f63041412d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:18.707 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.698+0000 7f630359e700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6304136e00 con 0x7f63040ad5c0 2020-11-20T11:43:18.708 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.698+0000 7f630359e700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63041370a0 con 0x7f6304136960 2020-11-20T11:43:18.708 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.698+0000 7f630359e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6304137340 con 0x7f63040ad1e0 2020-11-20T11:43:18.708 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.699+0000 7f6301d9b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f63040ad5c0 0x7f6304133bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:59360/0 (socket says 172.21.15.23:59360) 2020-11-20T11:43:18.708 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.699+0000 7f6301d9b700 1 -- 172.21.15.23:0/2515181814 learned_addr learned my addr 172.21.15.23:0/2515181814 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:43:18.709 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.699+0000 7f630159a700 1 -- 172.21.15.23:0/2515181814 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 485170349 0 0) 0x7f6304136e00 con 0x7f63040ad5c0 2020-11-20T11:43:18.709 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.699+0000 7f630159a700 1 -- 172.21.15.23:0/2515181814 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62e8003280 con 0x7f63040ad5c0 2020-11-20T11:43:18.709 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.699+0000 7f630159a700 1 -- 172.21.15.23:0/2515181814 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3422842381 0 0) 0x7f62e8003280 con 0x7f63040ad5c0 2020-11-20T11:43:18.709 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.699+0000 7f630159a700 1 -- 172.21.15.23:0/2515181814 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63040ad1e0 msgr2=0x7f63040a60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:43:18.709 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.699+0000 7f630159a700 1 --2- 172.21.15.23:0/2515181814 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63040ad1e0 0x7f63040a60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:18.710 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.699+0000 7f630159a700 1 -- 172.21.15.23:0/2515181814 >> v2:172.21.15.155:3300/0 conn(0x7f6304136960 msgr2=0x7f63041412d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2020-11-20T11:43:18.710 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.699+0000 7f630159a700 1 --2- 172.21.15.23:0/2515181814 >> v2:172.21.15.155:3300/0 conn(0x7f6304136960 0x7f63041412d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:18.710 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.699+0000 7f630159a700 1 -- 172.21.15.23:0/2515181814 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63041377e0 con 0x7f63040ad5c0 2020-11-20T11:43:18.710 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.699+0000 7f630159a700 1 -- 172.21.15.23:0/2515181814 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f62f4001520 con 0x7f63040ad5c0 2020-11-20T11:43:18.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.699+0000 7f630159a700 1 -- 172.21.15.23:0/2515181814 >> v1:172.21.15.23:6789/0 conn(0x7f63040ad5c0 legacy=0x7f6304133bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:18.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.699+0000 7f630159a700 1 --2- 172.21.15.23:0/2515181814 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63040ad1e0 0x7f62e8003770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:18.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.699+0000 7f630159a700 1 --2- 172.21.15.23:0/2515181814 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62e8003c80 0x7f62e8004520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:18.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.699+0000 7f630159a700 1 --2- 172.21.15.23:0/2515181814 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62e8004a10 0x7f62e8004e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:18.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.699+0000 7f630159a700 1 -- 172.21.15.23:0/2515181814 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63041370a0 con 0x7f62e8004a10 2020-11-20T11:43:18.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.699+0000 7f630159a700 1 -- 172.21.15.23:0/2515181814 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6304137340 con 0x7f62e8003c80 2020-11-20T11:43:18.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.699+0000 7f630159a700 1 -- 172.21.15.23:0/2515181814 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62e8004240 con 0x7f63040ad1e0 2020-11-20T11:43:18.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.700+0000 7f6302d9d700 1 --2- 172.21.15.23:0/2515181814 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62e8004a10 0x7f62e8004e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:18.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.700+0000 7f630259c700 1 --2- 172.21.15.23:0/2515181814 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63040ad1e0 0x7f62e8003770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:18.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.700+0000 7f6301d9b700 1 --2- 172.21.15.23:0/2515181814 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62e8003c80 0x7f62e8004520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:18.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.700+0000 7f6302d9d700 1 -- 172.21.15.23:0/2515181814 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63040ad1e0 msgr2=0x7f62e8003770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:43:18.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.700+0000 7f6302d9d700 1 --2- 172.21.15.23:0/2515181814 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f63040ad1e0 0x7f62e8003770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:18.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.700+0000 7f6302d9d700 1 -- 172.21.15.23:0/2515181814 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62e8003c80 msgr2=0x7f62e8004520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:43:18.716 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.700+0000 7f6302d9d700 1 --2- 172.21.15.23:0/2515181814 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62e8003c80 0x7f62e8004520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:18.716 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.700+0000 7f6302d9d700 1 -- 172.21.15.23:0/2515181814 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e8003280 con 0x7f62e8004a10 2020-11-20T11:43:18.716 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.700+0000 7f6302d9d700 1 --2- 172.21.15.23:0/2515181814 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62e8004a10 0x7f62e8004e90 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f62fc00fd90 tx=0x7f62fc0091a0).ready entity=mon.0 client_cookie=ac8056298932e2a1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:18.716 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.700+0000 7f630159a700 1 -- 172.21.15.23:0/2515181814 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc009d40 con 0x7f62e8004a10 2020-11-20T11:43:18.717 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.700+0000 7f630159a700 1 -- 172.21.15.23:0/2515181814 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62fc018440 con 0x7f62e8004a10 2020-11-20T11:43:18.717 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.700+0000 7f630159a700 1 -- 172.21.15.23:0/2515181814 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc018800 con 0x7f62e8004a10 2020-11-20T11:43:18.717 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.701+0000 7f630359e700 1 -- 172.21.15.23:0/2515181814 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62e8004a10 msgr2=0x7f62e8004e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:18.717 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.701+0000 7f630359e700 1 --2- 172.21.15.23:0/2515181814 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62e8004a10 0x7f62e8004e90 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f62fc00fd90 tx=0x7f62fc0091a0).stop 2020-11-20T11:43:18.718 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.701+0000 7f6302d9d700 1 -- 172.21.15.23:0/2515181814 reap_dead start 2020-11-20T11:43:18.718 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.701+0000 7f630359e700 1 -- 172.21.15.23:0/2515181814 shutdown_connections 2020-11-20T11:43:18.718 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.701+0000 7f630359e700 1 -- 172.21.15.23:0/2515181814 >> 172.21.15.23:0/2515181814 conn(0x7f6304061850 msgr2=0x7f6304061c60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:18.718 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.701+0000 7f630359e700 1 -- 172.21.15.23:0/2515181814 shutdown_connections 2020-11-20T11:43:18.718 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.701+0000 7f630359e700 1 -- 172.21.15.23:0/2515181814 wait complete. 2020-11-20T11:43:18.719 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.702+0000 7f630359e700 1 Processor -- start 2020-11-20T11:43:18.719 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.702+0000 7f630359e700 1 -- start start 2020-11-20T11:43:18.721 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.703+0000 7f630359e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62e8004a10 0x7f6304149050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:18.722 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.703+0000 7f630359e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6304066720 0x7f630409d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:18.722 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.703+0000 7f630359e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63040a31f0 0x7f63040a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:18.722 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.703+0000 7f630359e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63040a3b60 con 0x7f63040a31f0 2020-11-20T11:43:18.722 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.703+0000 7f630359e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63040652a0 con 0x7f62e8004a10 2020-11-20T11:43:18.723 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.703+0000 7f630359e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6304065540 con 0x7f6304066720 2020-11-20T11:43:18.723 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.703+0000 7f630259c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62e8004a10 0x7f6304149050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:18.723 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.703+0000 7f6302d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63040a31f0 0x7f63040a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:18.723 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.703+0000 7f630259c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62e8004a10 0x7f6304149050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:38016/0 (socket says 172.21.15.23:38016) 2020-11-20T11:43:18.723 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.703+0000 7f630259c700 1 -- 172.21.15.23:0/3527456403 learned_addr learned my addr 172.21.15.23:0/3527456403 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:43:18.724 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.703+0000 7f6301d9b700 1 --2- 172.21.15.23:0/3527456403 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6304066720 0x7f630409d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:18.724 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.704+0000 7f6301d9b700 1 -- 172.21.15.23:0/3527456403 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6304066720 msgr2=0x7f630409d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2020-11-20T11:43:18.724 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.704+0000 7f6301d9b700 1 -- 172.21.15.23:0/3527456403 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6304066720 msgr2=0x7f630409d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:43:18.724 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.704+0000 7f6301d9b700 1 --2- 172.21.15.23:0/3527456403 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6304066720 0x7f630409d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:43:18.724 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.704+0000 7f6301d9b700 1 --2- 172.21.15.23:0/3527456403 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6304066720 0x7f630409d740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:43:18.725 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.704+0000 7f6302d9d700 1 -- 172.21.15.23:0/3527456403 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6304066720 msgr2=0x7f630409d740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:43:18.725 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.704+0000 7f6302d9d700 1 --2- 172.21.15.23:0/3527456403 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6304066720 0x7f630409d740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:18.725 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.704+0000 7f6302d9d700 1 -- 172.21.15.23:0/3527456403 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62e8004a10 msgr2=0x7f6304149050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:18.725 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.704+0000 7f6302d9d700 1 --2- 172.21.15.23:0/3527456403 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62e8004a10 0x7f6304149050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:18.726 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.704+0000 7f6302d9d700 1 -- 172.21.15.23:0/3527456403 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63040657e0 con 0x7f63040a31f0 2020-11-20T11:43:18.726 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.704+0000 7f630259c700 1 --2- 172.21.15.23:0/3527456403 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62e8004a10 0x7f6304149050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:43:18.726 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.704+0000 7f6302d9d700 1 --2- 172.21.15.23:0/3527456403 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63040a31f0 0x7f63040a3670 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f62fc017000 tx=0x7f62fc018000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:18.726 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.704+0000 7f62f27fc700 1 -- 172.21.15.23:0/3527456403 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc018dc0 con 0x7f63040a31f0 2020-11-20T11:43:18.726 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.704+0000 7f62f27fc700 1 -- 172.21.15.23:0/3527456403 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62fc017800 con 0x7f63040a31f0 2020-11-20T11:43:18.727 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.704+0000 7f630359e700 1 -- 172.21.15.23:0/3527456403 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f630413a860 con 0x7f63040a31f0 2020-11-20T11:43:18.727 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.704+0000 7f62f27fc700 1 -- 172.21.15.23:0/3527456403 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc017bc0 con 0x7f63040a31f0 2020-11-20T11:43:18.727 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.704+0000 7f630359e700 1 -- 172.21.15.23:0/3527456403 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f630409dc30 con 0x7f63040a31f0 2020-11-20T11:43:18.727 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.705+0000 7f630359e700 1 -- 172.21.15.23:0/3527456403 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6304057dc0 con 0x7f63040a31f0 2020-11-20T11:43:18.727 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.706+0000 7f62f27fc700 1 -- 172.21.15.23:0/3527456403 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f62fc021020 con 0x7f63040a31f0 2020-11-20T11:43:18.728 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.706+0000 7f62f27fc700 1 --2- 172.21.15.23:0/3527456403 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f62d8056d60 0x7f62d8059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:18.728 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.706+0000 7f62f27fc700 1 -- 172.21.15.23:0/3527456403 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f62fc075730 con 0x7f63040a31f0 2020-11-20T11:43:18.728 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.706+0000 7f630259c700 1 --2- 172.21.15.23:0/3527456403 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f62d8056d60 0x7f62d8059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:18.728 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.707+0000 7f630259c700 1 --2- 172.21.15.23:0/3527456403 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f62d8056d60 0x7f62d8059220 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f62f8006e40 tx=0x7f62f8006f90).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:18.728 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:18.710+0000 7f62f27fc700 1 -- 172.21.15.23:0/3527456403 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f62fc04a990 con 0x7f63040a31f0 2020-11-20T11:43:19.184 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:19.180+0000 7f630359e700 1 -- 172.21.15.23:0/3527456403 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f63041342e0 con 0x7f63040a31f0 2020-11-20T11:43:19.184 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:19.180+0000 7f62f27fc700 1 -- 172.21.15.23:0/3527456403 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1472 (secure 0 0 0) 0x7f62fc029070 con 0x7f63040a31f0 2020-11-20T11:43:19.185 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":96,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:43:19.186 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:19.182+0000 7f630359e700 1 -- 172.21.15.23:0/3527456403 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f62d8056d60 msgr2=0x7f62d8059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:19.187 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:19.182+0000 7f630359e700 1 --2- 172.21.15.23:0/3527456403 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f62d8056d60 0x7f62d8059220 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f62f8006e40 tx=0x7f62f8006f90).stop 2020-11-20T11:43:19.187 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:19.182+0000 7f630359e700 1 -- 172.21.15.23:0/3527456403 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63040a31f0 msgr2=0x7f63040a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:19.187 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:19.182+0000 7f630359e700 1 --2- 172.21.15.23:0/3527456403 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63040a31f0 0x7f63040a3670 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f62fc017000 tx=0x7f62fc018000).stop 2020-11-20T11:43:19.187 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:19.182+0000 7f630359e700 1 -- 172.21.15.23:0/3527456403 shutdown_connections 2020-11-20T11:43:19.188 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:19.182+0000 7f630359e700 1 --2- 172.21.15.23:0/3527456403 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f63040a31f0 0x7f63040a3670 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:19.188 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:19.182+0000 7f630359e700 1 --2- 172.21.15.23:0/3527456403 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f62e8004a10 0x7f6304149050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:19.188 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:19.182+0000 7f630359e700 1 --2- 172.21.15.23:0/3527456403 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f62d8056d60 0x7f62d8059220 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:19.188 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:19.182+0000 7f630359e700 1 --2- 172.21.15.23:0/3527456403 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6304066720 0x7f630409d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:19.188 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:19.182+0000 7f630359e700 1 -- 172.21.15.23:0/3527456403 >> 172.21.15.23:0/3527456403 conn(0x7f6304061850 msgr2=0x7f630413d000 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:19.189 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:19.183+0000 7f630359e700 1 -- 172.21.15.23:0/3527456403 shutdown_connections 2020-11-20T11:43:19.189 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:19.183+0000 7f630359e700 1 -- 172.21.15.23:0/3527456403 wait complete. 2020-11-20T11:43:22.511 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:43:22.723 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:22.719+0000 7f9950eb7700 1 Processor -- start 2020-11-20T11:43:22.725 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:22.719+0000 7f9950eb7700 1 -- start start 2020-11-20T11:43:22.725 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:22.720+0000 7f9950eb7700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c063fe0 0x7f994c13f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:22.726 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:22.720+0000 7f9950eb7700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f994c0a69f0 0x7f994c13fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:22.726 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:22.720+0000 7f9950eb7700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f994c0a3cc0 con 0x7f994c0ad7c0 2020-11-20T11:43:22.726 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:22.720+0000 7f9950eb7700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994c0a4640 con 0x7f994c0a69f0 2020-11-20T11:43:22.726 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:22.720+0000 7f9950eb7700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994c0a5ba0 con 0x7f994c063fe0 2020-11-20T11:43:22.726 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:22.721+0000 7f9949d9b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c063fe0 0x7f994c13f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:22.727 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:22.721+0000 7f9949d9b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c063fe0 0x7f994c13f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:55370/0 (socket says 172.21.15.23:55370) 2020-11-20T11:43:22.727 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:22.721+0000 7f9949d9b700 1 -- 172.21.15.23:0/2673773584 learned_addr learned my addr 172.21.15.23:0/2673773584 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:43:22.727 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:22.721+0000 7f9949d9b700 1 -- 172.21.15.23:0/2673773584 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c063fe0 msgr2=0x7f994c13f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:43:22.728 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:22.721+0000 7f9949d9b700 1 -- 172.21.15.23:0/2673773584 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c063fe0 msgr2=0x7f994c13f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:43:22.728 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:22.721+0000 7f9949d9b700 1 --2- 172.21.15.23:0/2673773584 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c063fe0 0x7f994c13f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:43:22.728 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:22.721+0000 7f9949d9b700 1 --2- 172.21.15.23:0/2673773584 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c063fe0 0x7f994c13f570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:43:22.729 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:22.722+0000 7f994ad9d700 1 --2- 172.21.15.23:0/2673773584 >> v2:172.21.15.155:3300/0 conn(0x7f994c0a69f0 0x7f994c13fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:22.729 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:22.723+0000 7f994ad9d700 1 -- 172.21.15.23:0/2673773584 >> v2:172.21.15.155:3300/0 conn(0x7f994c0a69f0 msgr2=0x7f994c13fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:43:22.729 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:22.723+0000 7f994ad9d700 1 -- 172.21.15.23:0/2673773584 >> v2:172.21.15.155:3300/0 conn(0x7f994c0a69f0 msgr2=0x7f994c13fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:43:22.729 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:22.723+0000 7f994ad9d700 1 --2- 172.21.15.23:0/2673773584 >> v2:172.21.15.155:3300/0 conn(0x7f994c0a69f0 0x7f994c13fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:43:22.730 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:22.723+0000 7f994ad9d700 1 --2- 172.21.15.23:0/2673773584 >> v2:172.21.15.155:3300/0 conn(0x7f994c0a69f0 0x7f994c13fa60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:43:22.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:22.922+0000 7f9949d9b700 1 --2- 172.21.15.23:0/2673773584 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c063fe0 0x7f994c13f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:22.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:22.922+0000 7f9949d9b700 1 -- 172.21.15.23:0/2673773584 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c063fe0 msgr2=0x7f994c13f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:43:22.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:22.922+0000 7f9949d9b700 1 -- 172.21.15.23:0/2673773584 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c063fe0 msgr2=0x7f994c13f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:43:22.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:22.922+0000 7f9949d9b700 1 --2- 172.21.15.23:0/2673773584 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c063fe0 0x7f994c13f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:43:22.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:22.922+0000 7f9949d9b700 1 --2- 172.21.15.23:0/2673773584 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c063fe0 0x7f994c13f570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:43:22.928 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:22.924+0000 7f994ad9d700 1 --2- 172.21.15.23:0/2673773584 >> v2:172.21.15.155:3300/0 conn(0x7f994c0a69f0 0x7f994c13fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:22.928 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:22.924+0000 7f994ad9d700 1 -- 172.21.15.23:0/2673773584 >> v2:172.21.15.155:3300/0 conn(0x7f994c0a69f0 msgr2=0x7f994c13fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:43:22.929 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:22.924+0000 7f994ad9d700 1 -- 172.21.15.23:0/2673773584 >> v2:172.21.15.155:3300/0 conn(0x7f994c0a69f0 msgr2=0x7f994c13fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:43:22.929 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:22.924+0000 7f994ad9d700 1 --2- 172.21.15.23:0/2673773584 >> v2:172.21.15.155:3300/0 conn(0x7f994c0a69f0 0x7f994c13fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:43:22.929 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:22.924+0000 7f994ad9d700 1 --2- 172.21.15.23:0/2673773584 >> v2:172.21.15.155:3300/0 conn(0x7f994c0a69f0 0x7f994c13fa60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:43:23.327 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:23.323+0000 7f9949d9b700 1 --2- 172.21.15.23:0/2673773584 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c063fe0 0x7f994c13f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:23.334 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:23.323+0000 7f9949d9b700 1 -- 172.21.15.23:0/2673773584 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c063fe0 msgr2=0x7f994c13f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:43:23.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:23.323+0000 7f9949d9b700 1 -- 172.21.15.23:0/2673773584 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c063fe0 msgr2=0x7f994c13f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:43:23.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:23.323+0000 7f9949d9b700 1 --2- 172.21.15.23:0/2673773584 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c063fe0 0x7f994c13f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:43:23.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:23.323+0000 7f9949d9b700 1 --2- 172.21.15.23:0/2673773584 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c063fe0 0x7f994c13f570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:43:23.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:23.325+0000 7f994ad9d700 1 --2- 172.21.15.23:0/2673773584 >> v2:172.21.15.155:3300/0 conn(0x7f994c0a69f0 0x7f994c13fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:23.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:23.325+0000 7f994ad9d700 1 -- 172.21.15.23:0/2673773584 >> v2:172.21.15.155:3300/0 conn(0x7f994c0a69f0 msgr2=0x7f994c13fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:43:23.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:23.325+0000 7f994ad9d700 1 -- 172.21.15.23:0/2673773584 >> v2:172.21.15.155:3300/0 conn(0x7f994c0a69f0 msgr2=0x7f994c13fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:43:23.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:23.325+0000 7f994ad9d700 1 --2- 172.21.15.23:0/2673773584 >> v2:172.21.15.155:3300/0 conn(0x7f994c0a69f0 0x7f994c13fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:43:23.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:23.325+0000 7f994ad9d700 1 --2- 172.21.15.23:0/2673773584 >> v2:172.21.15.155:3300/0 conn(0x7f994c0a69f0 0x7f994c13fa60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:43:24.130 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.126+0000 7f9949d9b700 1 --2- 172.21.15.23:0/2673773584 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c063fe0 0x7f994c13f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:24.130 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.126+0000 7f9949d9b700 1 -- 172.21.15.23:0/2673773584 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c063fe0 msgr2=0x7f994c13f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:43:24.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.126+0000 7f9949d9b700 1 -- 172.21.15.23:0/2673773584 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c063fe0 msgr2=0x7f994c13f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:43:24.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.126+0000 7f9949d9b700 1 --2- 172.21.15.23:0/2673773584 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c063fe0 0x7f994c13f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:43:24.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.126+0000 7f9949d9b700 1 --2- 172.21.15.23:0/2673773584 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c063fe0 0x7f994c13f570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:43:24.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.127+0000 7f994ad9d700 1 --2- 172.21.15.23:0/2673773584 >> v2:172.21.15.155:3300/0 conn(0x7f994c0a69f0 0x7f994c13fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:24.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.127+0000 7f994ad9d700 1 -- 172.21.15.23:0/2673773584 >> v2:172.21.15.155:3300/0 conn(0x7f994c0a69f0 msgr2=0x7f994c13fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:43:24.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.127+0000 7f994ad9d700 1 -- 172.21.15.23:0/2673773584 >> v2:172.21.15.155:3300/0 conn(0x7f994c0a69f0 msgr2=0x7f994c13fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:43:24.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.127+0000 7f994ad9d700 1 --2- 172.21.15.23:0/2673773584 >> v2:172.21.15.155:3300/0 conn(0x7f994c0a69f0 0x7f994c13fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:43:24.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.127+0000 7f994ad9d700 1 --2- 172.21.15.23:0/2673773584 >> v2:172.21.15.155:3300/0 conn(0x7f994c0a69f0 0x7f994c13fa60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:43:24.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.384+0000 7f994959a700 1 -- 172.21.15.23:0/2673773584 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3758539299 0 0) 0x7f994c0a3cc0 con 0x7f994c0ad7c0 2020-11-20T11:43:24.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.384+0000 7f994959a700 1 -- 172.21.15.23:0/2673773584 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9930003280 con 0x7f994c0ad7c0 2020-11-20T11:43:24.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.384+0000 7f994959a700 1 -- 172.21.15.23:0/2673773584 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 514079031 0 0) 0x7f9930003280 con 0x7f994c0ad7c0 2020-11-20T11:43:24.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.384+0000 7f994959a700 1 -- 172.21.15.23:0/2673773584 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c063fe0 msgr2=0x7f994c13f570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:43:24.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.384+0000 7f994959a700 1 --2- 172.21.15.23:0/2673773584 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c063fe0 0x7f994c13f570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:24.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.384+0000 7f994959a700 1 -- 172.21.15.23:0/2673773584 >> v2:172.21.15.155:3300/0 conn(0x7f994c0a69f0 msgr2=0x7f994c13fa60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:43:24.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.384+0000 7f994959a700 1 --2- 172.21.15.23:0/2673773584 >> v2:172.21.15.155:3300/0 conn(0x7f994c0a69f0 0x7f994c13fa60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:24.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.384+0000 7f994959a700 1 -- 172.21.15.23:0/2673773584 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994c0a5e40 con 0x7f994c0ad7c0 2020-11-20T11:43:24.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.384+0000 7f994959a700 1 -- 172.21.15.23:0/2673773584 <== mon.1 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f9940001520 con 0x7f994c0ad7c0 2020-11-20T11:43:24.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.384+0000 7f994959a700 1 -- 172.21.15.23:0/2673773584 >> v1:172.21.15.155:6789/0 conn(0x7f994c0ad7c0 legacy=0x7f994c0a62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:24.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.384+0000 7f994959a700 1 --2- 172.21.15.23:0/2673773584 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c063fe0 0x7f99300037f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:24.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.384+0000 7f994959a700 1 --2- 172.21.15.23:0/2673773584 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9930003d50 0x7f9930004520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:24.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.384+0000 7f994959a700 1 --2- 172.21.15.23:0/2673773584 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9930004a10 0x7f9930004e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:24.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.384+0000 7f994959a700 1 -- 172.21.15.23:0/2673773584 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c0a4640 con 0x7f9930003d50 2020-11-20T11:43:24.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.384+0000 7f994959a700 1 -- 172.21.15.23:0/2673773584 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c0a5ba0 con 0x7f9930004a10 2020-11-20T11:43:24.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.384+0000 7f994959a700 1 -- 172.21.15.23:0/2673773584 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9930004310 con 0x7f994c063fe0 2020-11-20T11:43:24.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.384+0000 7f9949d9b700 1 --2- 172.21.15.23:0/2673773584 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9930003d50 0x7f9930004520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:24.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.385+0000 7f994ad9d700 1 --2- 172.21.15.23:0/2673773584 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9930004a10 0x7f9930004e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:24.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.384+0000 7f994a59c700 1 --2- 172.21.15.23:0/2673773584 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c063fe0 0x7f99300037f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:24.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.385+0000 7f9949d9b700 1 -- 172.21.15.23:0/2673773584 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c063fe0 msgr2=0x7f99300037f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:43:24.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.385+0000 7f9949d9b700 1 --2- 172.21.15.23:0/2673773584 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c063fe0 0x7f99300037f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:24.394 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.385+0000 7f9949d9b700 1 -- 172.21.15.23:0/2673773584 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9930004a10 msgr2=0x7f9930004e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:43:24.394 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.385+0000 7f9949d9b700 1 --2- 172.21.15.23:0/2673773584 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9930004a10 0x7f9930004e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:24.394 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.385+0000 7f9949d9b700 1 -- 172.21.15.23:0/2673773584 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9930003280 con 0x7f9930003d50 2020-11-20T11:43:24.394 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.385+0000 7f9949d9b700 1 --2- 172.21.15.23:0/2673773584 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9930003d50 0x7f9930004520 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f993c011b80 tx=0x7f993c008040).ready entity=mon.0 client_cookie=b6308681fede2fa6 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:24.395 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.388+0000 7f994959a700 1 -- 172.21.15.23:0/2673773584 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c013980 con 0x7f9930003d50 2020-11-20T11:43:24.396 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.388+0000 7f994959a700 1 -- 172.21.15.23:0/2673773584 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993c013ae0 con 0x7f9930003d50 2020-11-20T11:43:24.396 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.388+0000 7f994959a700 1 -- 172.21.15.23:0/2673773584 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c013e60 con 0x7f9930003d50 2020-11-20T11:43:24.396 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.390+0000 7f9950eb7700 1 -- 172.21.15.23:0/2673773584 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9930003d50 msgr2=0x7f9930004520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:24.396 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.390+0000 7f9950eb7700 1 --2- 172.21.15.23:0/2673773584 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9930003d50 0x7f9930004520 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f993c011b80 tx=0x7f993c008040).stop 2020-11-20T11:43:24.397 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.390+0000 7f994ad9d700 1 -- 172.21.15.23:0/2673773584 reap_dead start 2020-11-20T11:43:24.397 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.391+0000 7f9950eb7700 1 -- 172.21.15.23:0/2673773584 shutdown_connections 2020-11-20T11:43:24.397 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.391+0000 7f9950eb7700 1 -- 172.21.15.23:0/2673773584 >> 172.21.15.23:0/2673773584 conn(0x7f994c060c80 msgr2=0x7f994c0acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:24.397 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.391+0000 7f9950eb7700 1 -- 172.21.15.23:0/2673773584 shutdown_connections 2020-11-20T11:43:24.397 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.391+0000 7f9950eb7700 1 -- 172.21.15.23:0/2673773584 wait complete. 2020-11-20T11:43:24.399 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.392+0000 7f9950eb7700 1 Processor -- start 2020-11-20T11:43:24.399 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.392+0000 7f9950eb7700 1 -- start start 2020-11-20T11:43:24.399 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.393+0000 7f9950eb7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9930003d50 0x7f994c0ace00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:24.399 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.393+0000 7f9950eb7700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f994c067260 0x7f994c149f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:24.400 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.393+0000 7f9950eb7700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c14a420 0x7f994c1200e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:24.400 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.393+0000 7f9950eb7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c09e7e0 con 0x7f9930003d50 2020-11-20T11:43:24.400 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.393+0000 7f9950eb7700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c09e2a0 con 0x7f994c067260 2020-11-20T11:43:24.400 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.393+0000 7f9950eb7700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994c09e540 con 0x7f994c14a420 2020-11-20T11:43:24.400 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.393+0000 7f994a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9930003d50 0x7f994c0ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:24.401 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.393+0000 7f994a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9930003d50 0x7f994c0ace00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45154/0 (socket says 172.21.15.23:45154) 2020-11-20T11:43:24.401 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.393+0000 7f994a59c700 1 -- 172.21.15.23:0/221685156 learned_addr learned my addr 172.21.15.23:0/221685156 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:43:24.401 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.393+0000 7f994ad9d700 1 --2- 172.21.15.23:0/221685156 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c14a420 0x7f994c1200e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:24.401 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.394+0000 7f994ad9d700 1 -- 172.21.15.23:0/221685156 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c14a420 msgr2=0x7f994c1200e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2020-11-20T11:43:24.402 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.394+0000 7f994ad9d700 1 -- 172.21.15.23:0/221685156 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c14a420 msgr2=0x7f994c1200e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:43:24.402 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.394+0000 7f9949d9b700 1 --2- 172.21.15.23:0/221685156 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f994c067260 0x7f994c149f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:24.402 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.394+0000 7f994ad9d700 1 --2- 172.21.15.23:0/221685156 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c14a420 0x7f994c1200e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:43:24.402 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.394+0000 7f994ad9d700 1 --2- 172.21.15.23:0/221685156 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c14a420 0x7f994c1200e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:43:24.402 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.394+0000 7f9949d9b700 1 -- 172.21.15.23:0/221685156 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c14a420 msgr2=0x7f994c1200e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:43:24.403 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.394+0000 7f9949d9b700 1 --2- 172.21.15.23:0/221685156 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c14a420 0x7f994c1200e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:24.403 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.394+0000 7f9949d9b700 1 -- 172.21.15.23:0/221685156 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9930003d50 msgr2=0x7f994c0ace00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:24.403 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.394+0000 7f9949d9b700 1 --2- 172.21.15.23:0/221685156 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9930003d50 0x7f994c0ace00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:24.403 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.394+0000 7f9949d9b700 1 -- 172.21.15.23:0/221685156 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994c09df90 con 0x7f994c067260 2020-11-20T11:43:24.403 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.394+0000 7f9949d9b700 1 --2- 172.21.15.23:0/221685156 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f994c067260 0x7f994c149f30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f993c006d00 tx=0x7f993c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:24.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.394+0000 7f993a7fc700 1 -- 172.21.15.23:0/221685156 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c025070 con 0x7f994c067260 2020-11-20T11:43:24.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.395+0000 7f993a7fc700 1 -- 172.21.15.23:0/221685156 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993c006a40 con 0x7f994c067260 2020-11-20T11:43:24.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.395+0000 7f993a7fc700 1 -- 172.21.15.23:0/221685156 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c01f890 con 0x7f994c067260 2020-11-20T11:43:24.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.395+0000 7f9950eb7700 1 -- 172.21.15.23:0/221685156 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f994c09e9d0 con 0x7f994c067260 2020-11-20T11:43:24.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.395+0000 7f9950eb7700 1 -- 172.21.15.23:0/221685156 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f994c0678a0 con 0x7f994c067260 2020-11-20T11:43:24.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.401+0000 7f9950eb7700 1 -- 172.21.15.23:0/221685156 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f994c140c60 con 0x7f994c067260 2020-11-20T11:43:24.407 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.403+0000 7f993a7fc700 1 -- 172.21.15.23:0/221685156 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f993c01c030 con 0x7f994c067260 2020-11-20T11:43:24.407 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.403+0000 7f993a7fc700 1 --2- 172.21.15.23:0/221685156 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9920056d60 0x7f9920059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:24.407 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.403+0000 7f993a7fc700 1 -- 172.21.15.23:0/221685156 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f993c071a90 con 0x7f994c067260 2020-11-20T11:43:24.408 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.404+0000 7f994a59c700 1 --2- 172.21.15.23:0/221685156 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9920056d60 0x7f9920059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:24.409 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.405+0000 7f994a59c700 1 --2- 172.21.15.23:0/221685156 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9920056d60 0x7f9920059220 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f994c0ad510 tx=0x7f9940009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:24.415 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.411+0000 7f993a7fc700 1 -- 172.21.15.23:0/221685156 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f993c04a860 con 0x7f994c067260 2020-11-20T11:43:24.820 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.815+0000 7f9950eb7700 1 -- 172.21.15.23:0/221685156 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f994c140a60 con 0x7f994c067260 2020-11-20T11:43:24.820 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.816+0000 7f993a7fc700 1 -- 172.21.15.23:0/221685156 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1472 (secure 0 0 0) 0x7f993c02e020 con 0x7f994c067260 2020-11-20T11:43:24.824 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":100,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:43:24.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.822+0000 7f9950eb7700 1 -- 172.21.15.23:0/221685156 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9920056d60 msgr2=0x7f9920059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:24.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.822+0000 7f9950eb7700 1 --2- 172.21.15.23:0/221685156 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9920056d60 0x7f9920059220 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f994c0ad510 tx=0x7f9940009040).stop 2020-11-20T11:43:24.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.822+0000 7f9950eb7700 1 -- 172.21.15.23:0/221685156 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f994c067260 msgr2=0x7f994c149f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:24.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.822+0000 7f9950eb7700 1 --2- 172.21.15.23:0/221685156 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f994c067260 0x7f994c149f30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f993c006d00 tx=0x7f993c008040).stop 2020-11-20T11:43:24.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.823+0000 7f9950eb7700 1 -- 172.21.15.23:0/221685156 shutdown_connections 2020-11-20T11:43:24.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.823+0000 7f9950eb7700 1 --2- 172.21.15.23:0/221685156 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9920056d60 0x7f9920059220 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:24.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.823+0000 7f9950eb7700 1 --2- 172.21.15.23:0/221685156 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994c14a420 0x7f994c1200e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:24.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.823+0000 7f9950eb7700 1 --2- 172.21.15.23:0/221685156 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9930003d50 0x7f994c0ace00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:24.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.823+0000 7f9950eb7700 1 --2- 172.21.15.23:0/221685156 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f994c067260 0x7f994c149f30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:24.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.823+0000 7f9950eb7700 1 -- 172.21.15.23:0/221685156 >> 172.21.15.23:0/221685156 conn(0x7f994c060c80 msgr2=0x7f994c131920 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:24.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.823+0000 7f9950eb7700 1 -- 172.21.15.23:0/221685156 shutdown_connections 2020-11-20T11:43:24.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:24.823+0000 7f9950eb7700 1 -- 172.21.15.23:0/221685156 wait complete. 2020-11-20T11:43:27.852 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:43:28.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.303+0000 7f3a74b43700 1 Processor -- start 2020-11-20T11:43:28.308 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.303+0000 7f3a74b43700 1 -- start start 2020-11-20T11:43:28.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.304+0000 7f3a74b43700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3a700aaab0 0x7f3a7013ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:28.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.304+0000 7f3a74b43700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3a7013aeb0 0x7f3a700a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:28.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.304+0000 7f3a74b43700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a701388b0 con 0x7f3a700aae90 2020-11-20T11:43:28.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.304+0000 7f3a74b43700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a7013e600 con 0x7f3a7013aeb0 2020-11-20T11:43:28.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.304+0000 7f3a74b43700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7013e8a0 con 0x7f3a700aaab0 2020-11-20T11:43:28.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.304+0000 7f3a6ed9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3a7013aeb0 0x7f3a700a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:28.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.304+0000 7f3a6ed9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3a7013aeb0 0x7f3a700a8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45162/0 (socket says 172.21.15.23:45162) 2020-11-20T11:43:28.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.304+0000 7f3a6ed9d700 1 -- 172.21.15.23:0/3842738874 learned_addr learned my addr 172.21.15.23:0/3842738874 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:43:28.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.304+0000 7f3a6dd9b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f3a700aae90 0x7f3a701381a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:59412/0 (socket says 172.21.15.23:59412) 2020-11-20T11:43:28.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.304+0000 7f3a6e59c700 1 --2- 172.21.15.23:0/3842738874 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3a700aaab0 0x7f3a7013ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:28.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.305+0000 7f3a6ed9d700 1 -- 172.21.15.23:0/3842738874 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3a700aaab0 msgr2=0x7f3a7013ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:43:28.319 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.305+0000 7f3a6ed9d700 1 --2- 172.21.15.23:0/3842738874 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3a700aaab0 0x7f3a7013ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:28.319 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.305+0000 7f3a6ed9d700 1 -- 172.21.15.23:0/3842738874 >> v1:172.21.15.23:6789/0 conn(0x7f3a700aae90 legacy=0x7f3a701381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:28.319 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.305+0000 7f3a6ed9d700 1 -- 172.21.15.23:0/3842738874 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a700ab6d0 con 0x7f3a7013aeb0 2020-11-20T11:43:28.320 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.305+0000 7f3a6ed9d700 1 --2- 172.21.15.23:0/3842738874 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a7013aeb0 0x7f3a700a8c60 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f3a68009d90 tx=0x7f3a680092d0).ready entity=mon.0 client_cookie=46a4b814b5d1027c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:28.320 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.305+0000 7f3a6d59a700 1 -- 172.21.15.23:0/3842738874 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6800bbf0 con 0x7f3a7013aeb0 2020-11-20T11:43:28.320 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.305+0000 7f3a6d59a700 1 -- 172.21.15.23:0/3842738874 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a6800bd50 con 0x7f3a7013aeb0 2020-11-20T11:43:28.320 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.305+0000 7f3a6d59a700 1 -- 172.21.15.23:0/3842738874 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a68011610 con 0x7f3a7013aeb0 2020-11-20T11:43:28.321 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.305+0000 7f3a74b43700 1 -- 172.21.15.23:0/3842738874 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a7013aeb0 msgr2=0x7f3a700a8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:28.321 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.305+0000 7f3a74b43700 1 --2- 172.21.15.23:0/3842738874 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a7013aeb0 0x7f3a700a8c60 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f3a68009d90 tx=0x7f3a680092d0).stop 2020-11-20T11:43:28.321 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.305+0000 7f3a74b43700 1 -- 172.21.15.23:0/3842738874 shutdown_connections 2020-11-20T11:43:28.322 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.305+0000 7f3a74b43700 1 --2- 172.21.15.23:0/3842738874 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3a700aaab0 0x7f3a7013ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:28.322 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.305+0000 7f3a74b43700 1 --2- 172.21.15.23:0/3842738874 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a7013aeb0 0x7f3a700a8c60 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:28.322 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.305+0000 7f3a74b43700 1 -- 172.21.15.23:0/3842738874 >> 172.21.15.23:0/3842738874 conn(0x7f3a700678e0 msgr2=0x7f3a70067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:28.322 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.305+0000 7f3a74b43700 1 -- 172.21.15.23:0/3842738874 shutdown_connections 2020-11-20T11:43:28.323 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.305+0000 7f3a74b43700 1 -- 172.21.15.23:0/3842738874 wait complete. 2020-11-20T11:43:28.323 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.306+0000 7f3a74b43700 1 Processor -- start 2020-11-20T11:43:28.323 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.306+0000 7f3a74b43700 1 -- start start 2020-11-20T11:43:28.324 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.307+0000 7f3a74b43700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3a700aaab0 0x7f3a700b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:28.324 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.307+0000 7f3a74b43700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a700aae90 0x7f3a7014d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:28.324 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.307+0000 7f3a74b43700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a7013c470 0x7f3a7013c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:28.324 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.307+0000 7f3a74b43700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7014df90 con 0x7f3a7013c470 2020-11-20T11:43:28.324 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.307+0000 7f3a74b43700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7014e230 con 0x7f3a700aae90 2020-11-20T11:43:28.325 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.307+0000 7f3a74b43700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7014e4d0 con 0x7f3a700aaab0 2020-11-20T11:43:28.325 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.307+0000 7f3a6ed9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a7013c470 0x7f3a7013c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:28.325 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.307+0000 7f3a6ed9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a7013c470 0x7f3a7013c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45170/0 (socket says 172.21.15.23:45170) 2020-11-20T11:43:28.325 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.307+0000 7f3a6ed9d700 1 -- 172.21.15.23:0/1526671760 learned_addr learned my addr 172.21.15.23:0/1526671760 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:43:28.325 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.307+0000 7f3a6dd9b700 1 --2- 172.21.15.23:0/1526671760 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a700aae90 0x7f3a7014d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:28.326 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.307+0000 7f3a6ed9d700 1 -- 172.21.15.23:0/1526671760 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3a700aaab0 msgr2=0x7f3a700b09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:28.326 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.307+0000 7f3a6ed9d700 1 --2- 172.21.15.23:0/1526671760 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3a700aaab0 0x7f3a700b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:28.326 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.307+0000 7f3a6ed9d700 1 -- 172.21.15.23:0/1526671760 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a700aae90 msgr2=0x7f3a7014d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:28.326 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.307+0000 7f3a6ed9d700 1 --2- 172.21.15.23:0/1526671760 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a700aae90 0x7f3a7014d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:28.326 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.307+0000 7f3a6ed9d700 1 -- 172.21.15.23:0/1526671760 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a7013cda0 con 0x7f3a7013c470 2020-11-20T11:43:28.327 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.308+0000 7f3a6ed9d700 1 --2- 172.21.15.23:0/1526671760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a7013c470 0x7f3a7013c8b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f3a68007fd0 tx=0x7f3a68016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:28.327 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.308+0000 7f3a5e7fc700 1 -- 172.21.15.23:0/1526671760 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a68011650 con 0x7f3a7013c470 2020-11-20T11:43:28.327 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.308+0000 7f3a5e7fc700 1 -- 172.21.15.23:0/1526671760 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a6800bec0 con 0x7f3a7013c470 2020-11-20T11:43:28.327 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.308+0000 7f3a74b43700 1 -- 172.21.15.23:0/1526671760 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a7014dbe0 con 0x7f3a7013c470 2020-11-20T11:43:28.328 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.308+0000 7f3a5e7fc700 1 -- 172.21.15.23:0/1526671760 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a68010da0 con 0x7f3a7013c470 2020-11-20T11:43:28.328 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.308+0000 7f3a74b43700 1 -- 172.21.15.23:0/1526671760 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a700a4450 con 0x7f3a7013c470 2020-11-20T11:43:28.329 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.309+0000 7f3a74b43700 1 -- 172.21.15.23:0/1526671760 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a7005c7d0 con 0x7f3a7013c470 2020-11-20T11:43:28.329 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.320+0000 7f3a5e7fc700 1 -- 172.21.15.23:0/1526671760 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3a68017070 con 0x7f3a7013c470 2020-11-20T11:43:28.330 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.320+0000 7f3a5e7fc700 1 --2- 172.21.15.23:0/1526671760 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3a44056d60 0x7f3a44059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:28.330 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.320+0000 7f3a5e7fc700 1 -- 172.21.15.23:0/1526671760 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f3a68071d70 con 0x7f3a7013c470 2020-11-20T11:43:28.331 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.327+0000 7f3a6e59c700 1 --2- 172.21.15.23:0/1526671760 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3a44056d60 0x7f3a44059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:28.332 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.327+0000 7f3a5e7fc700 1 -- 172.21.15.23:0/1526671760 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3a68047060 con 0x7f3a7013c470 2020-11-20T11:43:28.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.495+0000 7f3a6e59c700 1 --2- 172.21.15.23:0/1526671760 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3a44056d60 0x7f3a44059220 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f3a60006fd0 tx=0x7f3a60006e30).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:28.721 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.717+0000 7f3a74b43700 1 -- 172.21.15.23:0/1526671760 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3a7013c2b0 con 0x7f3a7013c470 2020-11-20T11:43:28.722 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.718+0000 7f3a5e7fc700 1 -- 172.21.15.23:0/1526671760 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7f3a68015030 con 0x7f3a7013c470 2020-11-20T11:43:28.722 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":102,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:43:28.724 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.720+0000 7f3a4bfff700 1 -- 172.21.15.23:0/1526671760 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3a44056d60 msgr2=0x7f3a44059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:28.724 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.720+0000 7f3a4bfff700 1 --2- 172.21.15.23:0/1526671760 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3a44056d60 0x7f3a44059220 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f3a60006fd0 tx=0x7f3a60006e30).stop 2020-11-20T11:43:28.725 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.720+0000 7f3a4bfff700 1 -- 172.21.15.23:0/1526671760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a7013c470 msgr2=0x7f3a7013c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:28.725 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.720+0000 7f3a4bfff700 1 --2- 172.21.15.23:0/1526671760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a7013c470 0x7f3a7013c8b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f3a68007fd0 tx=0x7f3a68016000).stop 2020-11-20T11:43:28.725 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.720+0000 7f3a4bfff700 1 -- 172.21.15.23:0/1526671760 shutdown_connections 2020-11-20T11:43:28.725 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.720+0000 7f3a4bfff700 1 --2- 172.21.15.23:0/1526671760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a7013c470 0x7f3a7013c8b0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:28.726 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.720+0000 7f3a4bfff700 1 --2- 172.21.15.23:0/1526671760 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3a44056d60 0x7f3a44059220 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:28.729 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.720+0000 7f3a4bfff700 1 --2- 172.21.15.23:0/1526671760 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3a700aaab0 0x7f3a700b09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:28.730 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.720+0000 7f3a4bfff700 1 --2- 172.21.15.23:0/1526671760 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3a700aae90 0x7f3a7014d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:28.730 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.720+0000 7f3a4bfff700 1 -- 172.21.15.23:0/1526671760 >> 172.21.15.23:0/1526671760 conn(0x7f3a700678e0 msgr2=0x7f3a700acf80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:28.731 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.721+0000 7f3a4bfff700 1 -- 172.21.15.23:0/1526671760 shutdown_connections 2020-11-20T11:43:28.731 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.721+0000 7f3a4bfff700 1 -- 172.21.15.23:0/1526671760 wait complete. 2020-11-20T11:43:28.739 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:43:28.739 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:43:29.000 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.996+0000 7f61ba77b700 1 Processor -- start 2020-11-20T11:43:29.001 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.996+0000 7f61ba77b700 1 -- start start 2020-11-20T11:43:29.002 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.997+0000 7f61ba77b700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f61b4060a80 0x7f61b4061950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:29.003 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.997+0000 7f61ba77b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f61b40a9d40 0x7f61b40aa1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:29.003 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.997+0000 7f61ba77b700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b4067bc0 con 0x7f61b4137920 2020-11-20T11:43:29.003 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.997+0000 7f61ba77b700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61b4067e60 con 0x7f61b40a9d40 2020-11-20T11:43:29.003 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.997+0000 7f61ba77b700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61b4068100 con 0x7f61b4060a80 2020-11-20T11:43:29.004 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.997+0000 7f61b37fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f61b40a9d40 0x7f61b40aa1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:29.004 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.997+0000 7f61b37fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f61b40a9d40 0x7f61b40aa1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45174/0 (socket says 172.21.15.23:45174) 2020-11-20T11:43:29.004 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.997+0000 7f61b37fe700 1 -- 172.21.15.23:0/2453777271 learned_addr learned my addr 172.21.15.23:0/2453777271 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:43:29.004 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.997+0000 7f61b3fff700 1 --2- 172.21.15.23:0/2453777271 >> v2:172.21.15.155:3300/0 conn(0x7f61b4060a80 0x7f61b4061950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:29.004 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.997+0000 7f61b37fe700 1 -- 172.21.15.23:0/2453777271 >> v2:172.21.15.155:3300/0 conn(0x7f61b4060a80 msgr2=0x7f61b4061950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:43:29.005 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.997+0000 7f61b37fe700 1 --2- 172.21.15.23:0/2453777271 >> v2:172.21.15.155:3300/0 conn(0x7f61b4060a80 0x7f61b4061950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:29.005 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.997+0000 7f61b37fe700 1 -- 172.21.15.23:0/2453777271 >> v1:172.21.15.23:6789/0 conn(0x7f61b4137920 legacy=0x7f61b4137d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:29.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.997+0000 7f61b37fe700 1 -- 172.21.15.23:0/2453777271 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b4060e60 con 0x7f61b40a9d40 2020-11-20T11:43:29.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.998+0000 7f61b37fe700 1 --2- 172.21.15.23:0/2453777271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b40a9d40 0x7f61b40aa1c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f61b40b01e0 tx=0x7f61a8006ef0).ready entity=mon.0 client_cookie=ebd3e6211a072c93 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:29.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.998+0000 7f61b2ffd700 1 -- 172.21.15.23:0/2453777271 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a800b860 con 0x7f61b40a9d40 2020-11-20T11:43:29.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.998+0000 7f61b2ffd700 1 -- 172.21.15.23:0/2453777271 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a8000e90 con 0x7f61b40a9d40 2020-11-20T11:43:29.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.998+0000 7f61b2ffd700 1 -- 172.21.15.23:0/2453777271 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a800bbd0 con 0x7f61b40a9d40 2020-11-20T11:43:29.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.998+0000 7f61ba77b700 1 -- 172.21.15.23:0/2453777271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b40a9d40 msgr2=0x7f61b40aa1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:29.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.998+0000 7f61ba77b700 1 --2- 172.21.15.23:0/2453777271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b40a9d40 0x7f61b40aa1c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f61b40b01e0 tx=0x7f61a8006ef0).stop 2020-11-20T11:43:29.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.998+0000 7f61ba77b700 1 -- 172.21.15.23:0/2453777271 shutdown_connections 2020-11-20T11:43:29.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.998+0000 7f61ba77b700 1 --2- 172.21.15.23:0/2453777271 >> v2:172.21.15.155:3300/0 conn(0x7f61b4060a80 0x7f61b4061950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:29.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.998+0000 7f61ba77b700 1 --2- 172.21.15.23:0/2453777271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b40a9d40 0x7f61b40aa1c0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:29.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.998+0000 7f61ba77b700 1 -- 172.21.15.23:0/2453777271 >> 172.21.15.23:0/2453777271 conn(0x7f61b4066730 msgr2=0x7f61b40601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:29.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.998+0000 7f61ba77b700 1 -- 172.21.15.23:0/2453777271 shutdown_connections 2020-11-20T11:43:29.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.998+0000 7f61ba77b700 1 -- 172.21.15.23:0/2453777271 wait complete. 2020-11-20T11:43:29.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:28.999+0000 7f61ba77b700 1 Processor -- start 2020-11-20T11:43:29.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.000+0000 7f61ba77b700 1 -- start start 2020-11-20T11:43:29.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.000+0000 7f61ba77b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f61b4060a80 0x7f61b414a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:29.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.000+0000 7f61ba77b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f61b40a9d40 0x7f61b413e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:29.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.000+0000 7f61ba77b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b4137920 0x7f61b413f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:29.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.000+0000 7f61ba77b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b413eb50 con 0x7f61b4137920 2020-11-20T11:43:29.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.000+0000 7f61ba77b700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b413edf0 con 0x7f61b4060a80 2020-11-20T11:43:29.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.000+0000 7f61ba77b700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b4139010 con 0x7f61b40a9d40 2020-11-20T11:43:29.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.000+0000 7f61b8d18700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b4137920 0x7f61b413f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:29.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.000+0000 7f61b8d18700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b4137920 0x7f61b413f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45184/0 (socket says 172.21.15.23:45184) 2020-11-20T11:43:29.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.000+0000 7f61b8d18700 1 -- 172.21.15.23:0/2393852704 learned_addr learned my addr 172.21.15.23:0/2393852704 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:43:29.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.000+0000 7f61b37fe700 1 --2- 172.21.15.23:0/2393852704 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f61b40a9d40 0x7f61b413e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:29.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.000+0000 7f61b3fff700 1 --2- 172.21.15.23:0/2393852704 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f61b4060a80 0x7f61b414a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:29.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.001+0000 7f61b8d18700 1 -- 172.21.15.23:0/2393852704 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f61b40a9d40 msgr2=0x7f61b413e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:29.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.001+0000 7f61b8d18700 1 --2- 172.21.15.23:0/2393852704 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f61b40a9d40 0x7f61b413e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:29.014 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.001+0000 7f61b8d18700 1 -- 172.21.15.23:0/2393852704 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f61b4060a80 msgr2=0x7f61b414a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:29.014 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.001+0000 7f61b8d18700 1 --2- 172.21.15.23:0/2393852704 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f61b4060a80 0x7f61b414a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:29.014 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.001+0000 7f61b8d18700 1 -- 172.21.15.23:0/2393852704 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b413e810 con 0x7f61b4137920 2020-11-20T11:43:29.014 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.001+0000 7f61b8d18700 1 --2- 172.21.15.23:0/2393852704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b4137920 0x7f61b413f530 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f61ac00ca90 tx=0x7f61ac0013a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:29.015 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.001+0000 7f6193fff700 1 -- 172.21.15.23:0/2393852704 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac00a5e0 con 0x7f61b4137920 2020-11-20T11:43:29.015 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.001+0000 7f6193fff700 1 -- 172.21.15.23:0/2393852704 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61ac00a740 con 0x7f61b4137920 2020-11-20T11:43:29.015 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.001+0000 7f6193fff700 1 -- 172.21.15.23:0/2393852704 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac00aac0 con 0x7f61b4137920 2020-11-20T11:43:29.015 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.001+0000 7f61ba77b700 1 -- 172.21.15.23:0/2393852704 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b413fa20 con 0x7f61b4137920 2020-11-20T11:43:29.015 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.001+0000 7f61ba77b700 1 -- 172.21.15.23:0/2393852704 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b4138bd0 con 0x7f61b4137920 2020-11-20T11:43:29.018 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.002+0000 7f61ba77b700 1 --2- 172.21.15.23:0/2393852704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b413f0e0 0x7f61b413a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:29.018 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.002+0000 7f61ba77b700 1 -- 172.21.15.23:0/2393852704 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b413fc10 con 0x7f61b413f0e0 2020-11-20T11:43:29.018 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.002+0000 7f61b3fff700 1 --2- 172.21.15.23:0/2393852704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b413f0e0 0x7f61b413a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:29.018 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.002+0000 7f6193fff700 1 -- 172.21.15.23:0/2393852704 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f61ac018020 con 0x7f61b4137920 2020-11-20T11:43:29.018 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.003+0000 7f61b3fff700 1 -- 172.21.15.23:0/2393852704 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f61b40a6310 con 0x7f61b413f0e0 2020-11-20T11:43:29.019 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.003+0000 7f6193fff700 1 --2- 172.21.15.23:0/2393852704 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f618c056d60 0x7f618c059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:29.019 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.003+0000 7f6193fff700 1 -- 172.21.15.23:0/2393852704 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f61ac06fe40 con 0x7f61b4137920 2020-11-20T11:43:29.019 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.003+0000 7f61b3fff700 1 --2- 172.21.15.23:0/2393852704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b413f0e0 0x7f61b413a2c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f61a4011cb0 tx=0x7f61a4006e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:29.019 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.003+0000 7f61b37fe700 1 --2- 172.21.15.23:0/2393852704 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f618c056d60 0x7f618c059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:29.019 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.004+0000 7f61b37fe700 1 --2- 172.21.15.23:0/2393852704 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f618c056d60 0x7f618c059220 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f61b4147ff0 tx=0x7f61a8016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:29.020 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.004+0000 7f6193fff700 1 -- 172.21.15.23:0/2393852704 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a4011ec0 con 0x7f61b413f0e0 2020-11-20T11:43:29.020 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.004+0000 7f6193fff700 1 -- 172.21.15.23:0/2393852704 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f61a400fd60 con 0x7f61b413f0e0 2020-11-20T11:43:29.020 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.004+0000 7f6193fff700 1 -- 172.21.15.23:0/2393852704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b413f0e0 msgr2=0x7f61b413a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:29.020 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.004+0000 7f6193fff700 1 --2- 172.21.15.23:0/2393852704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b413f0e0 0x7f61b413a2c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f61a4011cb0 tx=0x7f61a4006e30).stop 2020-11-20T11:43:29.020 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.004+0000 7f6193fff700 1 -- 172.21.15.23:0/2393852704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b413f0e0 msgr2=0x7f61b413a2c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:43:29.021 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.004+0000 7f6193fff700 1 --2- 172.21.15.23:0/2393852704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b413f0e0 0x7f61b413a2c0 secure :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0x7f61a4011cb0 tx=0x7f61a4006e30).stop 2020-11-20T11:43:29.021 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.012+0000 7f61ba77b700 1 --2- 172.21.15.23:0/2393852704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b413a7b0 0x7f61b414d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:29.021 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.012+0000 7f61ba77b700 1 -- 172.21.15.23:0/2393852704 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b413ec90 con 0x7f61b413a7b0 2020-11-20T11:43:29.021 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.012+0000 7f61b3fff700 1 --2- 172.21.15.23:0/2393852704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b413a7b0 0x7f61b414d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:29.021 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.012+0000 7f61b3fff700 1 -- 172.21.15.23:0/2393852704 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f61b4141280 con 0x7f61b413a7b0 2020-11-20T11:43:29.022 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.013+0000 7f61b3fff700 1 --2- 172.21.15.23:0/2393852704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b413a7b0 0x7f61b414d9a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f61a4018000 tx=0x7f61a401b000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:29.022 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.014+0000 7f6193fff700 1 -- 172.21.15.23:0/2393852704 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a4027060 con 0x7f61b413a7b0 2020-11-20T11:43:29.023 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.015+0000 7f6193fff700 1 -- 172.21.15.23:0/2393852704 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4786 (secure 0 0 0) 0x7f61a400bab0 con 0x7f61b413a7b0 2020-11-20T11:43:29.023 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.015+0000 7f6193fff700 1 -- 172.21.15.23:0/2393852704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b413a7b0 msgr2=0x7f61b414d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:29.023 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.015+0000 7f6193fff700 1 --2- 172.21.15.23:0/2393852704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b413a7b0 0x7f61b414d9a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f61a4018000 tx=0x7f61a401b000).stop 2020-11-20T11:43:29.023 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.015+0000 7f6193fff700 1 -- 172.21.15.23:0/2393852704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b413a7b0 msgr2=0x7f61b414d9a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:43:29.023 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.015+0000 7f6193fff700 1 --2- 172.21.15.23:0/2393852704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b413a7b0 0x7f61b414d9a0 secure :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0x7f61a4018000 tx=0x7f61a401b000).stop 2020-11-20T11:43:29.027 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:43:29.027 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:43:29.027 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:43:29.027 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:43:29.028 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 102, 2020-11-20T11:43:29.028 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:43:29.028 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:43:29.028 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:43:29.028 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:43:29.029 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:29.029 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 3, 2020-11-20T11:43:29.029 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:43:29.029 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:43:29.029 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:43:29.030 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:43:29.030 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:43:29.030 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:43:29.030 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:43:29.031 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:43:29.031 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:43:29.031 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:43:29.031 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:43:29.031 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:29.032 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:43:29.032 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:43:29.032 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:43:29.032 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:43:29.032 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:43:29.032 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:43:29.033 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:43:29.033 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:43:29.033 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:43:29.033 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:43:29.034 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:29.034 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.034 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:43:29.034 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:43:29.034 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:43:29.034 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:43:29.035 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:43:29.035 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:43:29.035 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:43:29.035 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:43:29.035 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:43:29.036 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:43:29.036 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:43:29.036 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:43:29.036 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:43:29.036 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:43:29.037 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:43:29.037 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:43:29.037 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:43:29.037 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:43:29.038 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:43:29.038 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:43:29.038 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:43:29.038 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:43:29.038 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:43:29.039 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:29.039 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:43:29.039 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.039 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:43:29.039 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.040 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:43:29.040 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:43:29.040 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:43:29.040 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:43:29.040 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.040 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:43:29.041 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:43:29.041 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:29.041 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.041 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.042 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:43:29.042 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:43:29.042 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:29.042 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:29.043 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:29.043 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.043 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:43:29.043 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:43:29.044 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:43:29.044 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:43:29.044 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:43:29.044 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.044 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.045 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:43:29.045 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:43:29.045 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:43:29.045 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:43:29.045 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.046 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:43:29.046 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:43:29.046 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:29.046 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.046 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.047 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:43:29.047 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:43:29.047 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:29.047 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:29.047 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:29.048 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.048 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:43:29.048 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:43:29.048 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:43:29.048 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:43:29.049 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:43:29.049 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.049 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.049 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:43:29.049 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:43:29.050 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:43:29.050 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:43:29.050 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.050 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:43:29.050 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:43:29.051 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:29.051 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.051 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.051 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:43:29.051 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:43:29.052 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:29.052 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:29.052 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:29.052 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.052 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:43:29.053 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:43:29.053 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:43:29.053 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:43:29.053 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:43:29.053 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:29.054 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:29.054 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.054 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:43:29.054 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:43:29.054 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.055 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:43:29.055 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:43:29.055 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:43:29.055 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:29.055 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:29.056 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:43:29.056 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.056 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:43:29.057 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:43:29.057 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:43:29.057 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:29.057 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:29.057 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:43:29.058 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.058 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:43:29.058 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:43:29.058 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 8 2020-11-20T11:43:29.058 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:29.059 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:29.059 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:43:29.059 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.059 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:43:29.059 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:43:29.060 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:43:29.060 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:29.060 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:29.060 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:43:29.061 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.061 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:43:29.061 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:43:29.061 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:43:29.062 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:29.062 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:29.062 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.062 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:43:29.062 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:43:29.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.024+0000 7f61ba77b700 1 -- 172.21.15.23:0/2393852704 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f618c056d60 msgr2=0x7f618c059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:29.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.024+0000 7f61ba77b700 1 --2- 172.21.15.23:0/2393852704 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f618c056d60 0x7f618c059220 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f61b4147ff0 tx=0x7f61a8016000).stop 2020-11-20T11:43:29.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.025+0000 7f61ba77b700 1 -- 172.21.15.23:0/2393852704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b4137920 msgr2=0x7f61b413f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:29.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.025+0000 7f61ba77b700 1 --2- 172.21.15.23:0/2393852704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61b4137920 0x7f61b413f530 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f61ac00ca90 tx=0x7f61ac0013a0).stop 2020-11-20T11:43:29.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.025+0000 7f61b8d18700 1 -- 172.21.15.23:0/2393852704 reap_dead start 2020-11-20T11:43:29.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.025+0000 7f61b8d18700 1 -- 172.21.15.23:0/2393852704 reap_dead start 2020-11-20T11:43:29.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.025+0000 7f61ba77b700 1 -- 172.21.15.23:0/2393852704 shutdown_connections 2020-11-20T11:43:29.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.025+0000 7f61ba77b700 1 -- 172.21.15.23:0/2393852704 >> 172.21.15.23:0/2393852704 conn(0x7f61b4066730 msgr2=0x7f61b40601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:29.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.025+0000 7f61ba77b700 1 -- 172.21.15.23:0/2393852704 shutdown_connections 2020-11-20T11:43:29.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.025+0000 7f61ba77b700 1 -- 172.21.15.23:0/2393852704 wait complete. 2020-11-20T11:43:29.067 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:43:29.276 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.270+0000 7fb772bc3700 1 Processor -- start 2020-11-20T11:43:29.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.270+0000 7fb772bc3700 1 -- start start 2020-11-20T11:43:29.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.271+0000 7fb772bc3700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb76c0608c0 0x7fb76c0aa0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:29.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.271+0000 7fb772bc3700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb76c138ff0 0x7fb76c139480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:29.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.271+0000 7fb772bc3700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb76c0aa590 con 0x7fb76c060ca0 2020-11-20T11:43:29.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.271+0000 7fb772bc3700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb76c0ab300 con 0x7fb76c0608c0 2020-11-20T11:43:29.278 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.271+0000 7fb772bc3700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c0aabe0 con 0x7fb76c138ff0 2020-11-20T11:43:29.278 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.274+0000 7fb771160700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb76c138ff0 0x7fb76c139480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:29.279 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.274+0000 7fb771160700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb76c138ff0 0x7fb76c139480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:55434/0 (socket says 172.21.15.23:55434) 2020-11-20T11:43:29.279 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.274+0000 7fb771160700 1 -- 172.21.15.23:0/450097270 learned_addr learned my addr 172.21.15.23:0/450097270 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:43:29.280 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.275+0000 7fb771160700 1 -- 172.21.15.23:0/450097270 >> v2:172.21.15.155:3300/0 conn(0x7fb76c0608c0 msgr2=0x7fb76c0aa0a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:43:29.280 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.275+0000 7fb771160700 1 --2- 172.21.15.23:0/450097270 >> v2:172.21.15.155:3300/0 conn(0x7fb76c0608c0 0x7fb76c0aa0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:29.280 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.275+0000 7fb771160700 1 -- 172.21.15.23:0/450097270 >> v1:172.21.15.155:6789/0 conn(0x7fb76c060ca0 legacy=0x7fb76c0a6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:29.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.280+0000 7fb771160700 1 -- 172.21.15.23:0/450097270 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76c0aae80 con 0x7fb76c138ff0 2020-11-20T11:43:29.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.280+0000 7fb771160700 1 --2- 172.21.15.23:0/450097270 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb76c138ff0 0x7fb76c139480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fb75c009a10 tx=0x7fb75c006ef0).ready entity=mon.2 client_cookie=6be8b189c41e652d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:29.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.280+0000 7fb76b7fe700 1 -- 172.21.15.23:0/450097270 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75c00b860 con 0x7fb76c138ff0 2020-11-20T11:43:29.285 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.281+0000 7fb76b7fe700 1 -- 172.21.15.23:0/450097270 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb75c000e90 con 0x7fb76c138ff0 2020-11-20T11:43:29.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.281+0000 7fb76b7fe700 1 -- 172.21.15.23:0/450097270 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75c00bbd0 con 0x7fb76c138ff0 2020-11-20T11:43:29.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.282+0000 7fb772bc3700 1 -- 172.21.15.23:0/450097270 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb76c138ff0 msgr2=0x7fb76c139480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:29.286 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.282+0000 7fb772bc3700 1 --2- 172.21.15.23:0/450097270 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb76c138ff0 0x7fb76c139480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fb75c009a10 tx=0x7fb75c006ef0).stop 2020-11-20T11:43:29.287 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.282+0000 7fb772bc3700 1 -- 172.21.15.23:0/450097270 shutdown_connections 2020-11-20T11:43:29.287 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.282+0000 7fb772bc3700 1 --2- 172.21.15.23:0/450097270 >> v2:172.21.15.155:3300/0 conn(0x7fb76c0608c0 0x7fb76c0aa0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:29.287 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.282+0000 7fb772bc3700 1 --2- 172.21.15.23:0/450097270 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb76c138ff0 0x7fb76c139480 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:29.287 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.282+0000 7fb772bc3700 1 -- 172.21.15.23:0/450097270 >> 172.21.15.23:0/450097270 conn(0x7fb76c066750 msgr2=0x7fb76c0601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:29.288 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.283+0000 7fb772bc3700 1 -- 172.21.15.23:0/450097270 shutdown_connections 2020-11-20T11:43:29.289 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.283+0000 7fb772bc3700 1 -- 172.21.15.23:0/450097270 wait complete. 2020-11-20T11:43:29.289 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.285+0000 7fb772bc3700 1 Processor -- start 2020-11-20T11:43:29.289 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.285+0000 7fb772bc3700 1 -- start start 2020-11-20T11:43:29.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.286+0000 7fb772bc3700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb76c060ca0 0x7fb76c0a1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:29.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.286+0000 7fb772bc3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb76c138ff0 0x7fb76c13d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:29.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.286+0000 7fb772bc3700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb76c0630a0 0x7fb76c13dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:29.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.286+0000 7fb76bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb76c138ff0 0x7fb76c13d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:29.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.286+0000 7fb76bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb76c138ff0 0x7fb76c13d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45196/0 (socket says 172.21.15.23:45196) 2020-11-20T11:43:29.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.286+0000 7fb77095f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb76c060ca0 0x7fb76c0a1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:29.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.286+0000 7fb77095f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb76c060ca0 0x7fb76c0a1aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:38092/0 (socket says 172.21.15.23:38092) 2020-11-20T11:43:29.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.286+0000 7fb76bfff700 1 -- 172.21.15.23:0/1969257468 learned_addr learned my addr 172.21.15.23:0/1969257468 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:43:29.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.286+0000 7fb772bc3700 1 -- 172.21.15.23:0/1969257468 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c13e1e0 con 0x7fb76c138ff0 2020-11-20T11:43:29.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.286+0000 7fb772bc3700 1 -- 172.21.15.23:0/1969257468 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c13e8c0 con 0x7fb76c060ca0 2020-11-20T11:43:29.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.286+0000 7fb772bc3700 1 -- 172.21.15.23:0/1969257468 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c13eb60 con 0x7fb76c0630a0 2020-11-20T11:43:29.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.287+0000 7fb76bfff700 1 -- 172.21.15.23:0/1969257468 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb76c0630a0 msgr2=0x7fb76c13dcf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2020-11-20T11:43:29.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.287+0000 7fb76bfff700 1 --2- 172.21.15.23:0/1969257468 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb76c0630a0 0x7fb76c13dcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:29.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.287+0000 7fb76bfff700 1 -- 172.21.15.23:0/1969257468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb76c060ca0 msgr2=0x7fb76c0a1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:29.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.287+0000 7fb76bfff700 1 --2- 172.21.15.23:0/1969257468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb76c060ca0 0x7fb76c0a1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:29.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.287+0000 7fb76bfff700 1 -- 172.21.15.23:0/1969257468 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76c13e5b0 con 0x7fb76c138ff0 2020-11-20T11:43:29.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.287+0000 7fb76bfff700 1 --2- 172.21.15.23:0/1969257468 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb76c138ff0 0x7fb76c13d800 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fb76400ca70 tx=0x7fb764001530).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:29.299 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.289+0000 7fb768ff9700 1 -- 172.21.15.23:0/1969257468 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7640068b0 con 0x7fb76c138ff0 2020-11-20T11:43:29.299 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.289+0000 7fb768ff9700 1 -- 172.21.15.23:0/1969257468 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb764006a10 con 0x7fb76c138ff0 2020-11-20T11:43:29.299 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.289+0000 7fb768ff9700 1 -- 172.21.15.23:0/1969257468 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb764006db0 con 0x7fb76c138ff0 2020-11-20T11:43:29.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.289+0000 7fb772bc3700 1 -- 172.21.15.23:0/1969257468 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb76c13ee00 con 0x7fb76c138ff0 2020-11-20T11:43:29.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.289+0000 7fb772bc3700 1 -- 172.21.15.23:0/1969257468 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb76c13a840 con 0x7fb76c138ff0 2020-11-20T11:43:29.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.292+0000 7fb772bc3700 1 --2- 172.21.15.23:0/1969257468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb76c1334c0 0x7fb76c1338a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:29.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.292+0000 7fb772bc3700 1 -- 172.21.15.23:0/1969257468 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c13d560 con 0x7fb76c1334c0 2020-11-20T11:43:29.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.293+0000 7fb77095f700 1 --2- 172.21.15.23:0/1969257468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb76c1334c0 0x7fb76c1338a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:29.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.293+0000 7fb77095f700 1 -- 172.21.15.23:0/1969257468 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb76c044710 con 0x7fb76c1334c0 2020-11-20T11:43:29.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.293+0000 7fb77095f700 1 --2- 172.21.15.23:0/1969257468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb76c1334c0 0x7fb76c1338a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fb76c064a70 tx=0x7fb760008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:29.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.294+0000 7fb768ff9700 1 -- 172.21.15.23:0/1969257468 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76000ce00 con 0x7fb76c1334c0 2020-11-20T11:43:29.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.294+0000 7fb768ff9700 1 -- 172.21.15.23:0/1969257468 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fb76000b8a0 con 0x7fb76c1334c0 2020-11-20T11:43:29.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.294+0000 7fb768ff9700 1 -- 172.21.15.23:0/1969257468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb76c1334c0 msgr2=0x7fb76c1338a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:29.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.294+0000 7fb768ff9700 1 --2- 172.21.15.23:0/1969257468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb76c1334c0 0x7fb76c1338a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fb76c064a70 tx=0x7fb760008040).stop 2020-11-20T11:43:29.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.294+0000 7fb768ff9700 1 -- 172.21.15.23:0/1969257468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb76c1334c0 msgr2=0x7fb76c1338a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:43:29.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.294+0000 7fb768ff9700 1 --2- 172.21.15.23:0/1969257468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb76c1334c0 0x7fb76c1338a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:29.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.295+0000 7fb768ff9700 1 -- 172.21.15.23:0/1969257468 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb764018020 con 0x7fb76c138ff0 2020-11-20T11:43:29.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.295+0000 7fb768ff9700 1 --2- 172.21.15.23:0/1969257468 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb740057500 0x7fb7400599c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:29.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.296+0000 7fb768ff9700 1 -- 172.21.15.23:0/1969257468 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fb76406f600 con 0x7fb76c138ff0 2020-11-20T11:43:29.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.296+0000 7fb77095f700 1 --2- 172.21.15.23:0/1969257468 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb740057500 0x7fb7400599c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:29.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.296+0000 7fb77095f700 1 --2- 172.21.15.23:0/1969257468 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb740057500 0x7fb7400599c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fb76000f010 tx=0x7fb760016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:29.318 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.314+0000 7fb772bc3700 1 --2- 172.21.15.23:0/1969257468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb76c0612c0 0x7fb76c0616a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:29.318 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.314+0000 7fb772bc3700 1 -- 172.21.15.23:0/1969257468 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c13e740 con 0x7fb76c0612c0 2020-11-20T11:43:29.320 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.315+0000 7fb771160700 1 --2- 172.21.15.23:0/1969257468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb76c0612c0 0x7fb76c0616a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:29.320 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.316+0000 7fb771160700 1 -- 172.21.15.23:0/1969257468 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb76c05c280 con 0x7fb76c0612c0 2020-11-20T11:43:29.320 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.316+0000 7fb771160700 1 --2- 172.21.15.23:0/1969257468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb76c0612c0 0x7fb76c0616a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fb75c013000 tx=0x7fb75c016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:29.321 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.317+0000 7fb768ff9700 1 -- 172.21.15.23:0/1969257468 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75c011db0 con 0x7fb76c0612c0 2020-11-20T11:43:29.321 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.317+0000 7fb768ff9700 1 -- 172.21.15.23:0/1969257468 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4444 (secure 0 0 0) 0x7fb75c01d450 con 0x7fb76c0612c0 2020-11-20T11:43:29.322 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.317+0000 7fb768ff9700 1 -- 172.21.15.23:0/1969257468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb76c0612c0 msgr2=0x7fb76c0616a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:29.322 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.317+0000 7fb768ff9700 1 --2- 172.21.15.23:0/1969257468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb76c0612c0 0x7fb76c0616a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fb75c013000 tx=0x7fb75c016000).stop 2020-11-20T11:43:29.322 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.317+0000 7fb768ff9700 1 -- 172.21.15.23:0/1969257468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb76c0612c0 msgr2=0x7fb76c0616a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:43:29.323 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:43:29.323 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:43:29.323 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:43:29.323 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:43:29.324 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 102, 2020-11-20T11:43:29.324 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:43:29.324 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:43:29.324 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:43:29.324 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:43:29.325 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:29.325 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 3, 2020-11-20T11:43:29.325 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:43:29.325 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:43:29.325 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:43:29.326 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:43:29.326 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:43:29.326 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:43:29.326 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:43:29.327 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:43:29.327 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:43:29.327 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:43:29.327 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:43:29.328 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:29.328 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:43:29.328 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:43:29.328 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:43:29.329 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:43:29.329 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:43:29.329 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:43:29.329 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:43:29.329 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:43:29.330 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:43:29.330 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:43:29.330 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:29.330 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.330 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:43:29.331 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:43:29.331 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:43:29.331 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:43:29.331 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:43:29.331 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:43:29.332 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:43:29.332 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:43:29.332 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:43:29.332 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:43:29.332 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:43:29.333 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:43:29.333 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:43:29.333 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:43:29.333 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:43:29.333 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:43:29.334 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:43:29.334 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:43:29.334 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:43:29.334 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:43:29.334 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:43:29.335 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:43:29.335 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:43:29.335 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:29.335 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:43:29.335 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.336 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:43:29.336 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.336 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:43:29.336 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:43:29.336 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:43:29.337 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:43:29.337 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.337 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:43:29.338 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:43:29.338 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:29.338 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.338 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.338 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:43:29.339 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:43:29.339 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:29.339 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:29.339 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:29.339 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.339 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:43:29.340 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:43:29.340 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:43:29.340 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:43:29.340 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:43:29.340 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.341 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.341 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:43:29.341 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:43:29.341 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:43:29.342 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:43:29.342 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.342 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:43:29.342 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:43:29.342 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:29.343 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.343 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.343 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:43:29.343 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:43:29.343 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:29.344 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:29.344 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:29.344 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.344 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:43:29.344 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:43:29.345 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:43:29.345 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:43:29.345 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:43:29.345 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.345 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.346 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:43:29.346 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:43:29.346 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:43:29.346 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:43:29.346 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.347 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:43:29.347 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:43:29.347 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:29.347 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.348 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.348 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:43:29.348 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:43:29.348 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:29.348 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:29.349 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:29.349 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.349 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:43:29.349 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:43:29.349 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:43:29.350 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:43:29.350 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:43:29.350 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:29.350 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:29.350 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.351 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:43:29.351 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:43:29.351 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.351 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:43:29.351 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:43:29.352 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:43:29.352 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:29.352 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:29.352 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:43:29.352 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.353 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:43:29.353 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:43:29.353 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:43:29.353 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:29.353 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:29.354 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:43:29.354 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.354 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:43:29.354 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:43:29.354 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:43:29.355 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:29.355 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:29.355 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.355 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:43:29.355 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:43:29.356 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.318+0000 7fb768ff9700 1 --2- 172.21.15.23:0/1969257468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb76c0612c0 0x7fb76c0616a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:29.357 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.326+0000 7fb772bc3700 1 -- 172.21.15.23:0/1969257468 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb740057500 msgr2=0x7fb7400599c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:29.357 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.326+0000 7fb772bc3700 1 --2- 172.21.15.23:0/1969257468 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb740057500 0x7fb7400599c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fb76000f010 tx=0x7fb760016000).stop 2020-11-20T11:43:29.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.326+0000 7fb771160700 1 -- 172.21.15.23:0/1969257468 reap_dead start 2020-11-20T11:43:29.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.326+0000 7fb772bc3700 1 -- 172.21.15.23:0/1969257468 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb76c138ff0 msgr2=0x7fb76c13d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:29.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.326+0000 7fb772bc3700 1 --2- 172.21.15.23:0/1969257468 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb76c138ff0 0x7fb76c13d800 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fb76400ca70 tx=0x7fb764001530).stop 2020-11-20T11:43:29.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.326+0000 7fb772bc3700 1 -- 172.21.15.23:0/1969257468 shutdown_connections 2020-11-20T11:43:29.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.326+0000 7fb772bc3700 1 --2- 172.21.15.23:0/1969257468 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb76c138ff0 0x7fb76c13d800 secure :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0x7fb76400ca70 tx=0x7fb764001530).stop 2020-11-20T11:43:29.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.326+0000 7fb772bc3700 1 -- 172.21.15.23:0/1969257468 >> 172.21.15.23:0/1969257468 conn(0x7fb76c066750 msgr2=0x7fb76c0a7960 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:29.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.326+0000 7fb772bc3700 1 -- 172.21.15.23:0/1969257468 shutdown_connections 2020-11-20T11:43:29.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.326+0000 7fb772bc3700 1 -- 172.21.15.23:0/1969257468 wait complete. 2020-11-20T11:43:29.360 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:43:29.574 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.569+0000 7fadf3ff2700 1 Processor -- start 2020-11-20T11:43:29.574 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.569+0000 7fadf3ff2700 1 -- start start 2020-11-20T11:43:29.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.570+0000 7fadf3ff2700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fadec05bc80 0x7fadec0a9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:29.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.570+0000 7fadf3ff2700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fadec0a9cf0 0x7fadec0601b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:29.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.570+0000 7fadf3ff2700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadec067430 con 0x7fadec065780 2020-11-20T11:43:29.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.570+0000 7fadf3ff2700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadec0a8ca0 con 0x7fadec05bc80 2020-11-20T11:43:29.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.570+0000 7fadf3ff2700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadec05c060 con 0x7fadec0a9cf0 2020-11-20T11:43:29.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.570+0000 7fadf1d8e700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fadec05bc80 0x7fadec0a9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:29.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.570+0000 7fadf158d700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fadec065780 0x7fadec1379e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:59458/0 (socket says 172.21.15.23:59458) 2020-11-20T11:43:29.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.570+0000 7fadf1d8e700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fadec05bc80 0x7fadec0a9140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45208/0 (socket says 172.21.15.23:45208) 2020-11-20T11:43:29.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.570+0000 7fadf158d700 1 -- 172.21.15.23:0/1151192763 learned_addr learned my addr 172.21.15.23:0/1151192763 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:43:29.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.570+0000 7fadf258f700 1 --2- 172.21.15.23:0/1151192763 >> v2:172.21.15.155:3300/0 conn(0x7fadec0a9cf0 0x7fadec0601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:29.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.571+0000 7fadf1d8e700 1 -- 172.21.15.23:0/1151192763 >> v2:172.21.15.155:3300/0 conn(0x7fadec0a9cf0 msgr2=0x7fadec0601b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:43:29.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.571+0000 7fadf1d8e700 1 --2- 172.21.15.23:0/1151192763 >> v2:172.21.15.155:3300/0 conn(0x7fadec0a9cf0 0x7fadec0601b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:29.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.571+0000 7fadf1d8e700 1 -- 172.21.15.23:0/1151192763 >> v1:172.21.15.23:6789/0 conn(0x7fadec065780 legacy=0x7fadec1379e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:29.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.571+0000 7fadf1d8e700 1 -- 172.21.15.23:0/1151192763 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadec0a8810 con 0x7fadec05bc80 2020-11-20T11:43:29.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.571+0000 7fadf1d8e700 1 --2- 172.21.15.23:0/1151192763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadec05bc80 0x7fadec0a9140 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fadec066c80 tx=0x7fade4009220).ready entity=mon.0 client_cookie=67d8d039f73d5d2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:29.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.571+0000 7fadf0d8c700 1 -- 172.21.15.23:0/1151192763 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade400ba00 con 0x7fadec05bc80 2020-11-20T11:43:29.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.571+0000 7fadf0d8c700 1 -- 172.21.15.23:0/1151192763 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fade400bb60 con 0x7fadec05bc80 2020-11-20T11:43:29.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.571+0000 7fadf0d8c700 1 -- 172.21.15.23:0/1151192763 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade4011400 con 0x7fadec05bc80 2020-11-20T11:43:29.584 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.572+0000 7fadf3ff2700 1 -- 172.21.15.23:0/1151192763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadec05bc80 msgr2=0x7fadec0a9140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:29.584 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.572+0000 7fadf3ff2700 1 --2- 172.21.15.23:0/1151192763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadec05bc80 0x7fadec0a9140 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fadec066c80 tx=0x7fade4009220).stop 2020-11-20T11:43:29.585 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.572+0000 7fadf3ff2700 1 -- 172.21.15.23:0/1151192763 shutdown_connections 2020-11-20T11:43:29.585 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.572+0000 7fadf3ff2700 1 --2- 172.21.15.23:0/1151192763 >> v2:172.21.15.155:3300/0 conn(0x7fadec0a9cf0 0x7fadec0601b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:29.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.572+0000 7fadf3ff2700 1 --2- 172.21.15.23:0/1151192763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadec05bc80 0x7fadec0a9140 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:29.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.572+0000 7fadf3ff2700 1 -- 172.21.15.23:0/1151192763 >> 172.21.15.23:0/1151192763 conn(0x7fadec0ad7a0 msgr2=0x7fadec0adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:29.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.577+0000 7fadf3ff2700 1 -- 172.21.15.23:0/1151192763 shutdown_connections 2020-11-20T11:43:29.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.577+0000 7fadf3ff2700 1 -- 172.21.15.23:0/1151192763 wait complete. 2020-11-20T11:43:29.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.578+0000 7fadf3ff2700 1 Processor -- start 2020-11-20T11:43:29.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.578+0000 7fadf3ff2700 1 -- start start 2020-11-20T11:43:29.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.579+0000 7fadf3ff2700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fadec05bc80 0x7fadec0a77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:29.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.579+0000 7fadf3ff2700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fadec065780 0x7fadec13e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:29.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.579+0000 7fadf3ff2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadec0a9cf0 0x7fadec1383b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:29.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.580+0000 7fadf1d8e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fadec05bc80 0x7fadec0a77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:29.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.580+0000 7fadf1d8e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fadec05bc80 0x7fadec0a77d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:55454/0 (socket says 172.21.15.23:55454) 2020-11-20T11:43:29.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.580+0000 7fadf158d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fadec065780 0x7fadec13e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:29.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.580+0000 7fadf1d8e700 1 -- 172.21.15.23:0/180377636 learned_addr learned my addr 172.21.15.23:0/180377636 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:43:29.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.580+0000 7fadf3ff2700 1 -- 172.21.15.23:0/180377636 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadec0a0410 con 0x7fadec0a9cf0 2020-11-20T11:43:29.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.580+0000 7fadf3ff2700 1 -- 172.21.15.23:0/180377636 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadec0a1920 con 0x7fadec065780 2020-11-20T11:43:29.593 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.580+0000 7fadf3ff2700 1 -- 172.21.15.23:0/180377636 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadec0a1bc0 con 0x7fadec05bc80 2020-11-20T11:43:29.593 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.580+0000 7fadf258f700 1 --2- 172.21.15.23:0/180377636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadec0a9cf0 0x7fadec1383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:29.596 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.581+0000 7fadf258f700 1 -- 172.21.15.23:0/180377636 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fadec05bc80 msgr2=0x7fadec0a77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:29.597 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.581+0000 7fadf258f700 1 --2- 172.21.15.23:0/180377636 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fadec05bc80 0x7fadec0a77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:29.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.581+0000 7fadf258f700 1 -- 172.21.15.23:0/180377636 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fadec065780 msgr2=0x7fadec13e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:29.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.581+0000 7fadf258f700 1 --2- 172.21.15.23:0/180377636 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fadec065780 0x7fadec13e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:29.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.581+0000 7fadf258f700 1 -- 172.21.15.23:0/180377636 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadec0a2060 con 0x7fadec0a9cf0 2020-11-20T11:43:29.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.581+0000 7fadf258f700 1 --2- 172.21.15.23:0/180377636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadec0a9cf0 0x7fadec1383b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7faddc00e970 tx=0x7faddc006e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:29.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.582+0000 7fade9ffb700 1 -- 172.21.15.23:0/180377636 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faddc00c850 con 0x7fadec0a9cf0 2020-11-20T11:43:29.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.582+0000 7fade9ffb700 1 -- 172.21.15.23:0/180377636 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faddc00c9b0 con 0x7fadec0a9cf0 2020-11-20T11:43:29.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.582+0000 7fadf3ff2700 1 -- 172.21.15.23:0/180377636 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadec1388a0 con 0x7fadec0a9cf0 2020-11-20T11:43:29.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.582+0000 7fadf3ff2700 1 -- 172.21.15.23:0/180377636 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadec0a0090 con 0x7fadec0a9cf0 2020-11-20T11:43:29.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.583+0000 7fade9ffb700 1 -- 172.21.15.23:0/180377636 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faddc00cd30 con 0x7fadec0a9cf0 2020-11-20T11:43:29.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.584+0000 7fadf1d8e700 1 --2- 172.21.15.23:0/180377636 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fadec05bc80 0x7fadec0a77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:43:29.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.584+0000 7fadf158d700 1 --2- 172.21.15.23:0/180377636 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fadec065780 0x7fadec13e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:43:29.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.588+0000 7fade9ffb700 1 -- 172.21.15.23:0/180377636 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7faddc012600 con 0x7fadec0a9cf0 2020-11-20T11:43:29.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.588+0000 7fade9ffb700 1 --2- 172.21.15.23:0/180377636 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fadc4056db0 0x7fadc4059270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:29.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.588+0000 7fade9ffb700 1 -- 172.21.15.23:0/180377636 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7faddc06fb50 con 0x7fadec0a9cf0 2020-11-20T11:43:29.601 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.588+0000 7fadf1d8e700 1 --2- 172.21.15.23:0/180377636 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fadc4056db0 0x7fadc4059270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:29.601 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.589+0000 7fadcb7fe700 1 --2- 172.21.15.23:0/180377636 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fadec09d4a0 0x7fadec09d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:29.601 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.589+0000 7fadcb7fe700 1 -- 172.21.15.23:0/180377636 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadec09efc0 con 0x7fadec09d4a0 2020-11-20T11:43:29.601 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.589+0000 7fadf158d700 1 --2- 172.21.15.23:0/180377636 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fadec09d4a0 0x7fadec09d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:29.601 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.589+0000 7fadf158d700 1 -- 172.21.15.23:0/180377636 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fadec0a1d80 con 0x7fadec09d4a0 2020-11-20T11:43:29.602 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.590+0000 7fadf158d700 1 --2- 172.21.15.23:0/180377636 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fadec09d4a0 0x7fadec09d880 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fadec0685c0 tx=0x7fade0009040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:29.602 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.591+0000 7fade9ffb700 1 -- 172.21.15.23:0/180377636 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade0007320 con 0x7fadec09d4a0 2020-11-20T11:43:29.602 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.591+0000 7fadf1d8e700 1 --2- 172.21.15.23:0/180377636 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fadc4056db0 0x7fadc4059270 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fade40090b0 tx=0x7fade4016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:29.602 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.591+0000 7fade9ffb700 1 -- 172.21.15.23:0/180377636 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fade0007480 con 0x7fadec09d4a0 2020-11-20T11:43:29.603 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.591+0000 7fade9ffb700 1 -- 172.21.15.23:0/180377636 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fadec09d4a0 msgr2=0x7fadec09d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:29.603 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.591+0000 7fade9ffb700 1 --2- 172.21.15.23:0/180377636 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fadec09d4a0 0x7fadec09d880 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fadec0685c0 tx=0x7fade0009040).stop 2020-11-20T11:43:29.603 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.591+0000 7fade9ffb700 1 -- 172.21.15.23:0/180377636 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fadec09d4a0 msgr2=0x7fadec09d880 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:43:29.603 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.591+0000 7fade9ffb700 1 --2- 172.21.15.23:0/180377636 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fadec09d4a0 0x7fadec09d880 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:29.606 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.600+0000 7fadcb7fe700 1 --2- 172.21.15.23:0/180377636 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fadec09dd70 0x7fadec132970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:29.607 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.600+0000 7fadcb7fe700 1 -- 172.21.15.23:0/180377636 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadec13e940 con 0x7fadec09dd70 2020-11-20T11:43:29.608 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.601+0000 7fadf158d700 1 --2- 172.21.15.23:0/180377636 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fadec09dd70 0x7fadec132970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:29.608 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.601+0000 7fadf158d700 1 -- 172.21.15.23:0/180377636 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fadec062fe0 con 0x7fadec09dd70 2020-11-20T11:43:29.609 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.601+0000 7fadf158d700 1 --2- 172.21.15.23:0/180377636 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fadec09dd70 0x7fadec132970 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fade0014000 tx=0x7fade0017000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:29.611 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.603+0000 7fade9ffb700 1 -- 172.21.15.23:0/180377636 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade0013c50 con 0x7fadec09dd70 2020-11-20T11:43:29.612 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.603+0000 7fade9ffb700 1 -- 172.21.15.23:0/180377636 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4274 (secure 0 0 0) 0x7fade0007110 con 0x7fadec09dd70 2020-11-20T11:43:29.612 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.603+0000 7fade9ffb700 1 -- 172.21.15.23:0/180377636 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fadec09dd70 msgr2=0x7fadec132970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:29.614 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.603+0000 7fade9ffb700 1 --2- 172.21.15.23:0/180377636 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fadec09dd70 0x7fadec132970 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fade0014000 tx=0x7fade0017000).stop 2020-11-20T11:43:29.614 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.603+0000 7fade9ffb700 1 -- 172.21.15.23:0/180377636 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fadec09dd70 msgr2=0x7fadec132970 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:43:29.615 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.603+0000 7fade9ffb700 1 --2- 172.21.15.23:0/180377636 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fadec09dd70 0x7fadec132970 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:29.615 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.604+0000 7fadcb7fe700 1 -- 172.21.15.23:0/180377636 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fadc4056db0 msgr2=0x7fadc4059270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:29.615 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.604+0000 7fadcb7fe700 1 --2- 172.21.15.23:0/180377636 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fadc4056db0 0x7fadc4059270 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fade40090b0 tx=0x7fade4016000).stop 2020-11-20T11:43:29.616 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.604+0000 7fadcb7fe700 1 -- 172.21.15.23:0/180377636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadec0a9cf0 msgr2=0x7fadec1383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:29.616 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.604+0000 7fadcb7fe700 1 --2- 172.21.15.23:0/180377636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadec0a9cf0 0x7fadec1383b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7faddc00e970 tx=0x7faddc006e30).stop 2020-11-20T11:43:29.617 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.604+0000 7fadf258f700 1 -- 172.21.15.23:0/180377636 reap_dead start 2020-11-20T11:43:29.619 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.604+0000 7fadf258f700 1 -- 172.21.15.23:0/180377636 reap_dead start 2020-11-20T11:43:29.619 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.604+0000 7fadcb7fe700 1 -- 172.21.15.23:0/180377636 shutdown_connections 2020-11-20T11:43:29.620 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.604+0000 7fadcb7fe700 1 -- 172.21.15.23:0/180377636 >> 172.21.15.23:0/180377636 conn(0x7fadec0ad7a0 msgr2=0x7fadec13b540 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:29.620 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.604+0000 7fadcb7fe700 1 -- 172.21.15.23:0/180377636 shutdown_connections 2020-11-20T11:43:29.621 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:29.604+0000 7fadcb7fe700 1 -- 172.21.15.23:0/180377636 wait complete. 2020-11-20T11:43:29.621 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:43:29.622 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:43:29.623 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:43:29.623 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:43:29.624 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 102, 2020-11-20T11:43:29.625 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:43:29.625 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:43:29.627 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:43:29.627 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:43:29.627 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:29.627 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 4, 2020-11-20T11:43:29.627 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:43:29.628 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:43:29.628 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:43:29.628 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:43:29.628 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:43:29.628 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:43:29.629 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:43:29.629 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:43:29.629 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:43:29.629 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:43:29.630 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:43:29.630 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:29.630 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:43:29.630 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:43:29.631 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:43:29.631 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:43:29.632 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:43:29.632 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:43:29.632 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:43:29.632 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:43:29.632 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:43:29.633 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:43:29.634 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:29.635 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.635 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:43:29.635 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:43:29.636 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:43:29.636 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:43:29.636 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:43:29.636 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:43:29.636 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:43:29.637 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:43:29.637 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:43:29.637 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:43:29.637 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:43:29.638 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:43:29.638 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:43:29.638 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:43:29.638 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:43:29.638 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:43:29.639 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:43:29.639 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:43:29.639 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:43:29.639 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:43:29.639 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:43:29.640 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:43:29.640 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:43:29.640 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:29.640 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:43:29.641 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.641 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:43:29.641 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.642 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:43:29.642 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:43:29.642 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:43:29.642 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:43:29.643 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.643 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:43:29.643 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:43:29.643 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:29.643 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.644 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.644 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:43:29.644 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:43:29.644 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:29.644 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:29.645 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:29.645 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.645 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:43:29.645 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:43:29.645 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:43:29.646 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:43:29.646 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:43:29.646 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.646 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.647 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:43:29.647 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:43:29.647 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:43:29.647 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:43:29.647 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.648 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:43:29.648 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:43:29.648 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:29.648 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.648 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.649 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:43:29.649 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:43:29.649 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:29.649 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:29.649 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:29.650 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.650 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:43:29.650 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:43:29.650 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:43:29.650 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:43:29.651 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:43:29.651 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.651 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.651 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:43:29.651 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:43:29.652 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:43:29.652 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:43:29.653 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.653 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:43:29.653 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:43:29.653 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:29.653 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.654 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.654 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:43:29.654 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:43:29.654 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:29.654 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:29.655 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:29.655 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.655 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:43:29.655 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:43:29.655 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:43:29.656 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:43:29.656 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:43:29.656 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:29.656 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:29.657 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.657 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:43:29.657 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:43:29.657 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.657 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:43:29.658 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:43:29.658 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:43:29.658 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:29.658 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:29.658 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:43:29.659 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:29.659 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:43:29.659 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:43:29.659 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:43:29.659 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:29.660 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:29.660 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:29.660 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:43:29.660 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:43:29.661 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2020-11-20T11:43:29.662 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2020-11-20T11:43:30.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.130+0000 7f38377be700 1 Processor -- start 2020-11-20T11:43:30.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.131+0000 7f38377be700 1 -- start start 2020-11-20T11:43:30.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.131+0000 7f38377be700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f38300aae90 0x7f38301381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:30.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.131+0000 7f38377be700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f383013aeb0 0x7f38300a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:30.137 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.131+0000 7f38377be700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f383013e600 con 0x7f38300aaab0 2020-11-20T11:43:30.137 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.131+0000 7f38377be700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f383013e8a0 con 0x7f38300aae90 2020-11-20T11:43:30.137 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.131+0000 7f38377be700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38300ab6d0 con 0x7f383013aeb0 2020-11-20T11:43:30.137 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.131+0000 7f3834d59700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f38300aae90 0x7f38301381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:30.137 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.132+0000 7f3834d59700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f38300aae90 0x7f38301381a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45226/0 (socket says 172.21.15.23:45226) 2020-11-20T11:43:30.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.132+0000 7f3834d59700 1 -- 172.21.15.23:0/2127668841 learned_addr learned my addr 172.21.15.23:0/2127668841 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:43:30.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.132+0000 7f3834d59700 1 -- 172.21.15.23:0/2127668841 >> v2:172.21.15.155:3300/0 conn(0x7f383013aeb0 msgr2=0x7f38300a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:43:30.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.132+0000 7f3834d59700 1 --2- 172.21.15.23:0/2127668841 >> v2:172.21.15.155:3300/0 conn(0x7f383013aeb0 0x7f38300a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:30.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.132+0000 7f3834d59700 1 -- 172.21.15.23:0/2127668841 >> v1:172.21.15.23:6789/0 conn(0x7f38300aaab0 legacy=0x7f383013ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:30.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.132+0000 7f3834d59700 1 -- 172.21.15.23:0/2127668841 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3830138690 con 0x7f38300aae90 2020-11-20T11:43:30.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.133+0000 7f3834d59700 1 --2- 172.21.15.23:0/2127668841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38300aae90 0x7f38301381a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f3830062f40 tx=0x7f382c006ef0).ready entity=mon.0 client_cookie=1f078a58d3f9ea35 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:30.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.136+0000 7f381ffff700 1 -- 172.21.15.23:0/2127668841 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c00b860 con 0x7f38300aae90 2020-11-20T11:43:30.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.136+0000 7f381ffff700 1 -- 172.21.15.23:0/2127668841 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382c000e90 con 0x7f38300aae90 2020-11-20T11:43:30.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.136+0000 7f381ffff700 1 -- 172.21.15.23:0/2127668841 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c00bbd0 con 0x7f38300aae90 2020-11-20T11:43:30.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.137+0000 7f38377be700 1 -- 172.21.15.23:0/2127668841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38300aae90 msgr2=0x7f38301381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:30.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.137+0000 7f38377be700 1 --2- 172.21.15.23:0/2127668841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38300aae90 0x7f38301381a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f3830062f40 tx=0x7f382c006ef0).stop 2020-11-20T11:43:30.149 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.145+0000 7f38377be700 1 -- 172.21.15.23:0/2127668841 shutdown_connections 2020-11-20T11:43:30.149 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.145+0000 7f38377be700 1 --2- 172.21.15.23:0/2127668841 >> v2:172.21.15.155:3300/0 conn(0x7f383013aeb0 0x7f38300a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:30.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.145+0000 7f38377be700 1 --2- 172.21.15.23:0/2127668841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38300aae90 0x7f38301381a0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:30.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.145+0000 7f38377be700 1 -- 172.21.15.23:0/2127668841 >> 172.21.15.23:0/2127668841 conn(0x7f38300678e0 msgr2=0x7f3830067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:30.152 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.148+0000 7f38377be700 1 -- 172.21.15.23:0/2127668841 shutdown_connections 2020-11-20T11:43:30.155 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.151+0000 7f38377be700 1 -- 172.21.15.23:0/2127668841 wait complete. 2020-11-20T11:43:30.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.152+0000 7f38377be700 1 Processor -- start 2020-11-20T11:43:30.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.152+0000 7f38377be700 1 -- start start 2020-11-20T11:43:30.157 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.153+0000 7f38377be700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38300aaab0 0x7f383013d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:30.157 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.153+0000 7f38377be700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38300aae90 0x7f383014aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:30.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.153+0000 7f38377be700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38300a7490 0x7f383014af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:30.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.153+0000 7f38377be700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383014b400 con 0x7f38300aaab0 2020-11-20T11:43:30.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.153+0000 7f38377be700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38300a79a0 con 0x7f38300aae90 2020-11-20T11:43:30.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.153+0000 7f38377be700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38300a7c40 con 0x7f38300a7490 2020-11-20T11:43:30.159 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.153+0000 7f383555a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38300aaab0 0x7f383013d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:30.159 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.153+0000 7f3835d5b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38300a7490 0x7f383014af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:30.159 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.153+0000 7f383555a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38300aaab0 0x7f383013d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45230/0 (socket says 172.21.15.23:45230) 2020-11-20T11:43:30.159 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.153+0000 7f3835d5b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38300a7490 0x7f383014af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:55474/0 (socket says 172.21.15.23:55474) 2020-11-20T11:43:30.159 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.153+0000 7f383555a700 1 -- 172.21.15.23:0/1324841243 learned_addr learned my addr 172.21.15.23:0/1324841243 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:43:30.160 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.153+0000 7f3834d59700 1 --2- 172.21.15.23:0/1324841243 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38300aae90 0x7f383014aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:30.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.154+0000 7f3835d5b700 1 -- 172.21.15.23:0/1324841243 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38300aae90 msgr2=0x7f383014aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:30.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.154+0000 7f3835d5b700 1 --2- 172.21.15.23:0/1324841243 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38300aae90 0x7f383014aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:30.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.154+0000 7f3835d5b700 1 -- 172.21.15.23:0/1324841243 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38300aaab0 msgr2=0x7f383013d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:30.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.154+0000 7f3835d5b700 1 --2- 172.21.15.23:0/1324841243 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38300aaab0 0x7f383013d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:30.162 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.154+0000 7f3835d5b700 1 -- 172.21.15.23:0/1324841243 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38300b10d0 con 0x7f38300a7490 2020-11-20T11:43:30.162 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.154+0000 7f3834d59700 1 --2- 172.21.15.23:0/1324841243 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38300aae90 0x7f383014aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:43:30.162 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.154+0000 7f3835d5b700 1 --2- 172.21.15.23:0/1324841243 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38300a7490 0x7f383014af10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f382800bae0 tx=0x7f3828009120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:30.162 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.154+0000 7f381d7fa700 1 -- 172.21.15.23:0/1324841243 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382800c640 con 0x7f38300a7490 2020-11-20T11:43:30.162 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.154+0000 7f381d7fa700 1 -- 172.21.15.23:0/1324841243 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382800c7a0 con 0x7f38300a7490 2020-11-20T11:43:30.163 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.154+0000 7f38377be700 1 -- 172.21.15.23:0/1324841243 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f383014a510 con 0x7f38300a7490 2020-11-20T11:43:30.163 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.154+0000 7f381d7fa700 1 -- 172.21.15.23:0/1324841243 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382800c950 con 0x7f38300a7490 2020-11-20T11:43:30.163 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.154+0000 7f38377be700 1 -- 172.21.15.23:0/1324841243 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f383013fe50 con 0x7f38300a7490 2020-11-20T11:43:30.164 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.156+0000 7f38377be700 1 -- 172.21.15.23:0/1324841243 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f383005c7d0 con 0x7f38300a7490 2020-11-20T11:43:30.164 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.156+0000 7f381d7fa700 1 -- 172.21.15.23:0/1324841243 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f382801c070 con 0x7f38300a7490 2020-11-20T11:43:30.164 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.157+0000 7f381d7fa700 1 --2- 172.21.15.23:0/1324841243 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3808056d60 0x7f3808059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:30.164 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.157+0000 7f381d7fa700 1 -- 172.21.15.23:0/1324841243 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f382806f680 con 0x7f38300a7490 2020-11-20T11:43:30.165 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.160+0000 7f383555a700 1 --2- 172.21.15.23:0/1324841243 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3808056d60 0x7f3808059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:30.165 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.161+0000 7f383555a700 1 --2- 172.21.15.23:0/1324841243 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3808056d60 0x7f3808059220 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f3820003880 tx=0x7f3820000bc0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:30.172 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.164+0000 7f381d7fa700 1 -- 172.21.15.23:0/1324841243 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3828044910 con 0x7f38300a7490 2020-11-20T11:43:30.525 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.517+0000 7f38377be700 1 -- 172.21.15.23:0/1324841243 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f3830044710 con 0x7f38300a7490 2020-11-20T11:43:30.526 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.522+0000 7f381d7fa700 1 -- 172.21.15.23:0/1324841243 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f382801a070 con 0x7f38300a7490 2020-11-20T11:43:30.533 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.528+0000 7f38377be700 1 -- 172.21.15.23:0/1324841243 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3808056d60 msgr2=0x7f3808059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:30.533 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.528+0000 7f38377be700 1 --2- 172.21.15.23:0/1324841243 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3808056d60 0x7f3808059220 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f3820003880 tx=0x7f3820000bc0).stop 2020-11-20T11:43:30.533 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.528+0000 7f38377be700 1 -- 172.21.15.23:0/1324841243 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38300a7490 msgr2=0x7f383014af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:30.534 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.528+0000 7f38377be700 1 --2- 172.21.15.23:0/1324841243 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38300a7490 0x7f383014af10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f382800bae0 tx=0x7f3828009120).stop 2020-11-20T11:43:30.534 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.529+0000 7f38377be700 1 -- 172.21.15.23:0/1324841243 shutdown_connections 2020-11-20T11:43:30.534 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.529+0000 7f38377be700 1 --2- 172.21.15.23:0/1324841243 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3808056d60 0x7f3808059220 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:30.534 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.529+0000 7f38377be700 1 --2- 172.21.15.23:0/1324841243 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38300a7490 0x7f383014af10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:30.535 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.529+0000 7f38377be700 1 --2- 172.21.15.23:0/1324841243 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38300aaab0 0x7f383013d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:30.535 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.529+0000 7f38377be700 1 --2- 172.21.15.23:0/1324841243 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38300aae90 0x7f383014aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:30.535 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.529+0000 7f38377be700 1 -- 172.21.15.23:0/1324841243 >> 172.21.15.23:0/1324841243 conn(0x7f38300678e0 msgr2=0x7f38300aba00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:30.536 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.529+0000 7f38377be700 1 -- 172.21.15.23:0/1324841243 shutdown_connections 2020-11-20T11:43:30.536 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:30.529+0000 7f38377be700 1 -- 172.21.15.23:0/1324841243 wait complete. 2020-11-20T11:43:30.571 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2020-11-20T11:43:35.746 INFO:teuthology.orchestra.run.smithi023:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:43:35.749 INFO:teuthology.orchestra.run.smithi155:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-11-20T11:43:36.137 INFO:tasks.ceph.osd.0.smithi023.stderr:2020-11-20T11:43:36.132+0000 7f4450b23700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 41989) UID: 0 2020-11-20T11:43:36.138 INFO:tasks.ceph.osd.2.smithi023.stderr:2020-11-20T11:43:36.132+0000 7f069d7e5700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 41989) UID: 0 2020-11-20T11:43:36.142 INFO:tasks.ceph.osd.3.smithi023.stderr:2020-11-20T11:43:36.135+0000 7faee90db700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 41989) UID: 0 2020-11-20T11:43:36.144 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:43:36.135+0000 7f595f468700 -1 received signal: Hangup from killall ceph-osd -1 (PID: 41989) UID: 0 2020-11-20T11:43:40.572 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:43:40.573 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:43:40.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.707+0000 7f905130e700 1 Processor -- start 2020-11-20T11:43:40.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.708+0000 7f905130e700 1 -- start start 2020-11-20T11:43:40.723 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.709+0000 7f905130e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f904c0acca0 0x7f904c0a8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:40.723 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.709+0000 7f905130e700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f904c0ad080 0x7f904c0a8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:40.723 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.709+0000 7f905130e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f904c0a9020 0x7f904c0a9790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:40.724 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.709+0000 7f905130e700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f904c0a9560 con 0x7f904c0ad080 2020-11-20T11:43:40.724 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.709+0000 7f905130e700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f904c0a77e0 con 0x7f904c0acca0 2020-11-20T11:43:40.724 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.709+0000 7f905130e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f904c0a5a30 con 0x7f904c0a9020 2020-11-20T11:43:40.725 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.709+0000 7f904b7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f904c0ad080 0x7f904c0a8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:40.725 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.709+0000 7f904b7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f904c0ad080 0x7f904c0a8b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45238/0 (socket says 172.21.15.23:45238) 2020-11-20T11:43:40.725 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.709+0000 7f904b7fe700 1 -- 172.21.15.23:0/4263562387 learned_addr learned my addr 172.21.15.23:0/4263562387 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:43:40.725 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.709+0000 7f904bfff700 1 --2- 172.21.15.23:0/4263562387 >> v2:172.21.15.155:3300/0 conn(0x7f904c0acca0 0x7f904c0a8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:40.726 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.709+0000 7f9050b0d700 1 --2- 172.21.15.23:0/4263562387 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f904c0a9020 0x7f904c0a9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:40.726 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.709+0000 7f904b7fe700 1 -- 172.21.15.23:0/4263562387 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f904c0a9020 msgr2=0x7f904c0a9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:43:40.726 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.709+0000 7f904b7fe700 1 --2- 172.21.15.23:0/4263562387 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f904c0a9020 0x7f904c0a9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:40.727 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.709+0000 7f904b7fe700 1 -- 172.21.15.23:0/4263562387 >> v2:172.21.15.155:3300/0 conn(0x7f904c0acca0 msgr2=0x7f904c0a8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:43:40.727 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.709+0000 7f904b7fe700 1 --2- 172.21.15.23:0/4263562387 >> v2:172.21.15.155:3300/0 conn(0x7f904c0acca0 0x7f904c0a8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:40.727 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.709+0000 7f904b7fe700 1 -- 172.21.15.23:0/4263562387 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f904c0a79d0 con 0x7f904c0ad080 2020-11-20T11:43:40.728 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.709+0000 7f904bfff700 1 --2- 172.21.15.23:0/4263562387 >> v2:172.21.15.155:3300/0 conn(0x7f904c0acca0 0x7f904c0a8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:43:40.728 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.709+0000 7f904b7fe700 1 --2- 172.21.15.23:0/4263562387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f904c0ad080 0x7f904c0a8b30 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f904c062870 tx=0x7f903c006ef0).ready entity=mon.0 client_cookie=c857f5e9ec4731a2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:40.728 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.710+0000 7f904affd700 1 -- 172.21.15.23:0/4263562387 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f903c00b680 con 0x7f904c0ad080 2020-11-20T11:43:40.728 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.710+0000 7f904affd700 1 -- 172.21.15.23:0/4263562387 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f903c000e90 con 0x7f904c0ad080 2020-11-20T11:43:40.729 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.710+0000 7f904affd700 1 -- 172.21.15.23:0/4263562387 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f903c00b9f0 con 0x7f904c0ad080 2020-11-20T11:43:40.729 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.710+0000 7f905130e700 1 -- 172.21.15.23:0/4263562387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f904c0ad080 msgr2=0x7f904c0a8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:40.730 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.710+0000 7f905130e700 1 --2- 172.21.15.23:0/4263562387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f904c0ad080 0x7f904c0a8b30 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f904c062870 tx=0x7f903c006ef0).stop 2020-11-20T11:43:40.730 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.710+0000 7f905130e700 1 -- 172.21.15.23:0/4263562387 shutdown_connections 2020-11-20T11:43:40.730 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.710+0000 7f905130e700 1 --2- 172.21.15.23:0/4263562387 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f904c0a9020 0x7f904c0a9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:40.731 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.710+0000 7f905130e700 1 --2- 172.21.15.23:0/4263562387 >> v2:172.21.15.155:3300/0 conn(0x7f904c0acca0 0x7f904c0a8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:40.731 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.710+0000 7f905130e700 1 --2- 172.21.15.23:0/4263562387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f904c0ad080 0x7f904c0a8b30 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:40.731 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.710+0000 7f905130e700 1 -- 172.21.15.23:0/4263562387 >> 172.21.15.23:0/4263562387 conn(0x7f904c061850 msgr2=0x7f904c061c60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:40.731 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.710+0000 7f905130e700 1 -- 172.21.15.23:0/4263562387 shutdown_connections 2020-11-20T11:43:40.732 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.710+0000 7f905130e700 1 -- 172.21.15.23:0/4263562387 wait complete. 2020-11-20T11:43:40.732 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.711+0000 7f905130e700 1 Processor -- start 2020-11-20T11:43:40.732 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.712+0000 7f905130e700 1 -- start start 2020-11-20T11:43:40.733 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.712+0000 7f905130e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f904c0a9020 0x7f904c13bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:40.733 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.713+0000 7f905130e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f904c0acca0 0x7f904c13e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:40.733 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.713+0000 7f905130e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f904c0ad080 0x7f904c149be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:40.733 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.713+0000 7f905130e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904c13edd0 con 0x7f904c0a9020 2020-11-20T11:43:40.734 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.713+0000 7f905130e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904c13ca60 con 0x7f904c0ad080 2020-11-20T11:43:40.734 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.713+0000 7f905130e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f904c13cd00 con 0x7f904c0acca0 2020-11-20T11:43:40.734 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.713+0000 7f904bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f904c0a9020 0x7f904c13bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:40.735 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.713+0000 7f904bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f904c0a9020 0x7f904c13bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45244/0 (socket says 172.21.15.23:45244) 2020-11-20T11:43:40.735 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.713+0000 7f904bfff700 1 -- 172.21.15.23:0/3851898725 learned_addr learned my addr 172.21.15.23:0/3851898725 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:43:40.735 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.713+0000 7f9050b0d700 1 --2- 172.21.15.23:0/3851898725 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f904c0ad080 0x7f904c149be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:40.736 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.713+0000 7f904b7fe700 1 --2- 172.21.15.23:0/3851898725 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f904c0acca0 0x7f904c13e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:40.736 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.713+0000 7f904bfff700 1 -- 172.21.15.23:0/3851898725 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f904c0acca0 msgr2=0x7f904c13e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:40.736 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.713+0000 7f904bfff700 1 --2- 172.21.15.23:0/3851898725 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f904c0acca0 0x7f904c13e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:40.736 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.713+0000 7f904bfff700 1 -- 172.21.15.23:0/3851898725 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f904c0ad080 msgr2=0x7f904c149be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:40.737 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.713+0000 7f904bfff700 1 --2- 172.21.15.23:0/3851898725 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f904c0ad080 0x7f904c149be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:40.737 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.713+0000 7f904bfff700 1 -- 172.21.15.23:0/3851898725 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f904c13cfa0 con 0x7f904c0a9020 2020-11-20T11:43:40.737 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.713+0000 7f904bfff700 1 --2- 172.21.15.23:0/3851898725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f904c0a9020 0x7f904c13bf20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f904000ef50 tx=0x7f9040006e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:40.738 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.713+0000 7f902bfff700 1 -- 172.21.15.23:0/3851898725 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904000cab0 con 0x7f904c0a9020 2020-11-20T11:43:40.738 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.713+0000 7f902bfff700 1 -- 172.21.15.23:0/3851898725 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f904000cc10 con 0x7f904c0a9020 2020-11-20T11:43:40.738 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.713+0000 7f905130e700 1 -- 172.21.15.23:0/3851898725 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f904c13d440 con 0x7f904c0a9020 2020-11-20T11:43:40.738 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.713+0000 7f902bfff700 1 -- 172.21.15.23:0/3851898725 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90400123f0 con 0x7f904c0a9020 2020-11-20T11:43:40.739 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.713+0000 7f905130e700 1 -- 172.21.15.23:0/3851898725 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f904c09d4a0 con 0x7f904c0a9020 2020-11-20T11:43:40.739 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.714+0000 7f905130e700 1 -- 172.21.15.23:0/3851898725 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f904c13d130 con 0x7f904c0a9020 2020-11-20T11:43:40.739 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.715+0000 7f902bfff700 1 -- 172.21.15.23:0/3851898725 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9040012760 con 0x7f904c0a9020 2020-11-20T11:43:40.740 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.715+0000 7f902bfff700 1 --2- 172.21.15.23:0/3851898725 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9024056d60 0x7f9024059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:40.740 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.716+0000 7f902bfff700 1 -- 172.21.15.23:0/3851898725 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f9040070680 con 0x7f904c0a9020 2020-11-20T11:43:40.740 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.716+0000 7f904b7fe700 1 --2- 172.21.15.23:0/3851898725 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9024056d60 0x7f9024059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:40.740 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.716+0000 7f904b7fe700 1 --2- 172.21.15.23:0/3851898725 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9024056d60 0x7f9024059220 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f903c00b440 tx=0x7f903c016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:40.741 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:40.719+0000 7f902bfff700 1 -- 172.21.15.23:0/3851898725 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9040045860 con 0x7f904c0a9020 2020-11-20T11:43:41.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.135+0000 7f905130e700 1 -- 172.21.15.23:0/3851898725 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f904c057dc0 con 0x7f904c0a9020 2020-11-20T11:43:41.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.135+0000 7f902bfff700 1 -- 172.21.15.23:0/3851898725 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1479 (secure 0 0 0) 0x7f9040048e80 con 0x7f904c0a9020 2020-11-20T11:43:41.139 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":102,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:43:41.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.140+0000 7f905130e700 1 -- 172.21.15.23:0/3851898725 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9024056d60 msgr2=0x7f9024059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:41.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.140+0000 7f905130e700 1 --2- 172.21.15.23:0/3851898725 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9024056d60 0x7f9024059220 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f903c00b440 tx=0x7f903c016000).stop 2020-11-20T11:43:41.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.140+0000 7f905130e700 1 -- 172.21.15.23:0/3851898725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f904c0a9020 msgr2=0x7f904c13bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:41.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.140+0000 7f905130e700 1 --2- 172.21.15.23:0/3851898725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f904c0a9020 0x7f904c13bf20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f904000ef50 tx=0x7f9040006e30).stop 2020-11-20T11:43:41.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.141+0000 7f905130e700 1 -- 172.21.15.23:0/3851898725 shutdown_connections 2020-11-20T11:43:41.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.141+0000 7f905130e700 1 --2- 172.21.15.23:0/3851898725 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f904c0ad080 0x7f904c149be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:41.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.141+0000 7f905130e700 1 --2- 172.21.15.23:0/3851898725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f904c0a9020 0x7f904c13bf20 secure :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0x7f904000ef50 tx=0x7f9040006e30).stop 2020-11-20T11:43:41.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.141+0000 7f905130e700 1 --2- 172.21.15.23:0/3851898725 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9024056d60 0x7f9024059220 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:41.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.141+0000 7f905130e700 1 --2- 172.21.15.23:0/3851898725 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f904c0acca0 0x7f904c13e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:41.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.141+0000 7f905130e700 1 -- 172.21.15.23:0/3851898725 >> 172.21.15.23:0/3851898725 conn(0x7f904c061850 msgr2=0x7f904c0a4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:41.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.141+0000 7f905130e700 1 -- 172.21.15.23:0/3851898725 shutdown_connections 2020-11-20T11:43:41.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.141+0000 7f905130e700 1 -- 172.21.15.23:0/3851898725 wait complete. 2020-11-20T11:43:41.171 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:43:41.172 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2020-11-20T11:43:41.172 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:43:41.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.526+0000 7ff3b315f700 1 Processor -- start 2020-11-20T11:43:41.533 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.526+0000 7ff3b315f700 1 -- start start 2020-11-20T11:43:41.533 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.527+0000 7ff3b315f700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3ac07f7f0 0x7ff3ac0a5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:41.533 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.527+0000 7ff3b315f700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff3ac07fbd0 0x7ff3ac0a5f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:41.533 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.527+0000 7ff3b315f700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3ac0a4330 con 0x7ff3ac063fe0 2020-11-20T11:43:41.535 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.527+0000 7ff3b315f700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3ac0a4560 con 0x7ff3ac07fbd0 2020-11-20T11:43:41.536 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.527+0000 7ff3b315f700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac0ad830 con 0x7ff3ac07f7f0 2020-11-20T11:43:41.536 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.528+0000 7ff3b0efb700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3ac07f7f0 0x7ff3ac0a5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:41.536 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.528+0000 7ff3b0efb700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3ac07f7f0 0x7ff3ac0a5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:55492/0 (socket says 172.21.15.23:55492) 2020-11-20T11:43:41.537 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.528+0000 7ff3b0efb700 1 -- 172.21.15.23:0/3360169068 learned_addr learned my addr 172.21.15.23:0/3360169068 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:43:41.537 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.528+0000 7ff3abfff700 1 --2- 172.21.15.23:0/3360169068 >> v2:172.21.15.23:3300/0 conn(0x7ff3ac07fbd0 0x7ff3ac0a5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:41.537 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.529+0000 7ff3b0efb700 1 -- 172.21.15.23:0/3360169068 >> v2:172.21.15.23:3300/0 conn(0x7ff3ac07fbd0 msgr2=0x7ff3ac0a5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:43:41.537 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.529+0000 7ff3b0efb700 1 --2- 172.21.15.23:0/3360169068 >> v2:172.21.15.23:3300/0 conn(0x7ff3ac07fbd0 0x7ff3ac0a5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:41.537 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.529+0000 7ff3b0efb700 1 -- 172.21.15.23:0/3360169068 >> v1:172.21.15.23:6789/0 conn(0x7ff3ac063fe0 legacy=0x7ff3ac061850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:41.538 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.529+0000 7ff3b0efb700 1 -- 172.21.15.23:0/3360169068 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ac0ad600 con 0x7ff3ac07f7f0 2020-11-20T11:43:41.538 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.529+0000 7ff3b0efb700 1 --2- 172.21.15.23:0/3360169068 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3ac07f7f0 0x7ff3ac0a5a90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7ff39c0099f0 tx=0x7ff39c006ef0).ready entity=mon.2 client_cookie=4def6aa80aca231a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:41.538 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.528+0000 7ff3ab7fe700 1 -- 172.21.15.23:0/3360169068 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1960969741 0 0) 0x7ff3ac0a4330 con 0x7ff3ac063fe0 2020-11-20T11:43:41.538 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.529+0000 7ff3ab7fe700 1 -- 172.21.15.23:0/3360169068 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39c009e10 con 0x7ff3ac07f7f0 2020-11-20T11:43:41.538 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.529+0000 7ff3ab7fe700 1 -- 172.21.15.23:0/3360169068 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff39c000e90 con 0x7ff3ac07f7f0 2020-11-20T11:43:41.539 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.529+0000 7ff3ab7fe700 1 -- 172.21.15.23:0/3360169068 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39c01ba90 con 0x7ff3ac07f7f0 2020-11-20T11:43:41.539 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.530+0000 7ff3b315f700 1 -- 172.21.15.23:0/3360169068 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3ac07f7f0 msgr2=0x7ff3ac0a5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:41.539 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.530+0000 7ff3b315f700 1 --2- 172.21.15.23:0/3360169068 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3ac07f7f0 0x7ff3ac0a5a90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7ff39c0099f0 tx=0x7ff39c006ef0).stop 2020-11-20T11:43:41.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.552+0000 7ff3b315f700 1 -- 172.21.15.23:0/3360169068 shutdown_connections 2020-11-20T11:43:41.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.552+0000 7ff3b315f700 1 --2- 172.21.15.23:0/3360169068 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3ac07f7f0 0x7ff3ac0a5a90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:41.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.552+0000 7ff3b315f700 1 --2- 172.21.15.23:0/3360169068 >> v2:172.21.15.23:3300/0 conn(0x7ff3ac07fbd0 0x7ff3ac0a5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:41.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.552+0000 7ff3b315f700 1 -- 172.21.15.23:0/3360169068 >> 172.21.15.23:0/3360169068 conn(0x7ff3ac060c80 msgr2=0x7ff3ac0acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:41.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.552+0000 7ff3b315f700 1 -- 172.21.15.23:0/3360169068 shutdown_connections 2020-11-20T11:43:41.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.552+0000 7ff3b315f700 1 -- 172.21.15.23:0/3360169068 wait complete. 2020-11-20T11:43:41.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.554+0000 7ff3b315f700 1 Processor -- start 2020-11-20T11:43:41.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.554+0000 7ff3b315f700 1 -- start start 2020-11-20T11:43:41.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.555+0000 7ff3b315f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3ac063fe0 0x7ff3ac066080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:41.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.555+0000 7ff3b315f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3ac07f7f0 0x7ff3ac1482d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:41.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.555+0000 7ff3b315f700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3ac1487c0 0x7ff3ac09e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:41.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.555+0000 7ff3b315f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac0a3550 con 0x7ff3ac07f7f0 2020-11-20T11:43:41.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.555+0000 7ff3b315f700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac09f8e0 con 0x7ff3ac063fe0 2020-11-20T11:43:41.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.555+0000 7ff3b315f700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac09fb80 con 0x7ff3ac1487c0 2020-11-20T11:43:41.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.555+0000 7ff3b0efb700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3ac063fe0 0x7ff3ac066080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:41.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.555+0000 7ff3b0efb700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3ac063fe0 0x7ff3ac066080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:38154/0 (socket says 172.21.15.23:38154) 2020-11-20T11:43:41.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.556+0000 7ff3b0efb700 1 -- 172.21.15.23:0/4271578088 learned_addr learned my addr 172.21.15.23:0/4271578088 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:43:41.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.556+0000 7ff3abfff700 1 --2- 172.21.15.23:0/4271578088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3ac07f7f0 0x7ff3ac1482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:41.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.556+0000 7ff3b16fc700 1 --2- 172.21.15.23:0/4271578088 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3ac1487c0 0x7ff3ac09e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:41.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.556+0000 7ff3b0efb700 1 -- 172.21.15.23:0/4271578088 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3ac1487c0 msgr2=0x7ff3ac09e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:41.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.556+0000 7ff3b0efb700 1 --2- 172.21.15.23:0/4271578088 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff3ac1487c0 0x7ff3ac09e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:41.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.556+0000 7ff3b0efb700 1 -- 172.21.15.23:0/4271578088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3ac07f7f0 msgr2=0x7ff3ac1482d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:41.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.556+0000 7ff3b0efb700 1 --2- 172.21.15.23:0/4271578088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3ac07f7f0 0x7ff3ac1482d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:41.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.556+0000 7ff3b0efb700 1 -- 172.21.15.23:0/4271578088 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ac09fe20 con 0x7ff3ac063fe0 2020-11-20T11:43:41.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.556+0000 7ff3b0efb700 1 --2- 172.21.15.23:0/4271578088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3ac063fe0 0x7ff3ac066080 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7ff3ac0ad970 tx=0x7ff39c01d000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:41.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.557+0000 7ff3a8ff9700 1 -- 172.21.15.23:0/4271578088 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39c01bad0 con 0x7ff3ac063fe0 2020-11-20T11:43:41.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.557+0000 7ff3b315f700 1 -- 172.21.15.23:0/4271578088 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3ac09f5b0 con 0x7ff3ac063fe0 2020-11-20T11:43:41.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.557+0000 7ff3b315f700 1 -- 172.21.15.23:0/4271578088 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3ac0a1ac0 con 0x7ff3ac063fe0 2020-11-20T11:43:41.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.557+0000 7ff3a8ff9700 1 -- 172.21.15.23:0/4271578088 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff39c020070 con 0x7ff3ac063fe0 2020-11-20T11:43:41.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.557+0000 7ff3a8ff9700 1 -- 172.21.15.23:0/4271578088 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39c00ae40 con 0x7ff3ac063fe0 2020-11-20T11:43:41.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.559+0000 7ff3b315f700 1 --2- 172.21.15.23:0/4271578088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3ac07fbd0 0x7ff3ac09ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:41.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.559+0000 7ff3b315f700 1 -- 172.21.15.23:0/4271578088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac061650 con 0x7ff3ac07fbd0 2020-11-20T11:43:41.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.559+0000 7ff3abfff700 1 --2- 172.21.15.23:0/4271578088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3ac07fbd0 0x7ff3ac09ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:41.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.560+0000 7ff3abfff700 1 -- 172.21.15.23:0/4271578088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff3ac09ffb0 con 0x7ff3ac07fbd0 2020-11-20T11:43:41.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.560+0000 7ff3abfff700 1 --2- 172.21.15.23:0/4271578088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3ac07fbd0 0x7ff3ac09ea50 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7ff3ac062d10 tx=0x7ff3a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:41.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.568+0000 7ff3a8ff9700 1 -- 172.21.15.23:0/4271578088 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff39c02e030 con 0x7ff3ac063fe0 2020-11-20T11:43:41.574 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.568+0000 7ff3a8ff9700 1 --2- 172.21.15.23:0/4271578088 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff388056db0 0x7ff388059270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:41.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.568+0000 7ff3a8ff9700 1 -- 172.21.15.23:0/4271578088 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a400e040 con 0x7ff3ac07fbd0 2020-11-20T11:43:41.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.568+0000 7ff3a8ff9700 1 -- 172.21.15.23:0/4271578088 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7ff39c078ea0 con 0x7ff3ac063fe0 2020-11-20T11:43:41.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.569+0000 7ff3a8ff9700 1 -- 172.21.15.23:0/4271578088 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7ff3a400f040 con 0x7ff3ac07fbd0 2020-11-20T11:43:41.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.569+0000 7ff3a8ff9700 1 -- 172.21.15.23:0/4271578088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3ac07fbd0 msgr2=0x7ff3ac09ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:41.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.569+0000 7ff3a8ff9700 1 --2- 172.21.15.23:0/4271578088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3ac07fbd0 0x7ff3ac09ea50 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7ff3ac062d10 tx=0x7ff3a400b040).stop 2020-11-20T11:43:41.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.569+0000 7ff3a8ff9700 1 -- 172.21.15.23:0/4271578088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3ac07fbd0 msgr2=0x7ff3ac09ea50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:43:41.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.569+0000 7ff3a8ff9700 1 --2- 172.21.15.23:0/4271578088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3ac07fbd0 0x7ff3ac09ea50 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:41.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.569+0000 7ff3b16fc700 1 --2- 172.21.15.23:0/4271578088 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff388056db0 0x7ff388059270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:41.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.569+0000 7ff3b16fc700 1 --2- 172.21.15.23:0/4271578088 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff388056db0 0x7ff388059270 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7ff3a00011e0 tx=0x7ff3a00079f0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:41.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.586+0000 7ff3b315f700 1 --2- 172.21.15.23:0/4271578088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3ac09d4a0 0x7ff3ac09d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:41.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.586+0000 7ff3b315f700 1 -- 172.21.15.23:0/4271578088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac09fa20 con 0x7ff3ac09d4a0 2020-11-20T11:43:41.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.586+0000 7ff3abfff700 1 --2- 172.21.15.23:0/4271578088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3ac09d4a0 0x7ff3ac09d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:41.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.587+0000 7ff3abfff700 1 -- 172.21.15.23:0/4271578088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff3ac0a0150 con 0x7ff3ac09d4a0 2020-11-20T11:43:41.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.587+0000 7ff3abfff700 1 --2- 172.21.15.23:0/4271578088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3ac09d4a0 0x7ff3ac09d8b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7ff3ac061790 tx=0x7ff3a401b000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:41.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.587+0000 7ff3a8ff9700 1 -- 172.21.15.23:0/4271578088 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a40156f0 con 0x7ff3ac09d4a0 2020-11-20T11:43:41.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.587+0000 7ff3a8ff9700 1 -- 172.21.15.23:0/4271578088 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4787 (secure 0 0 0) 0x7ff3a4015850 con 0x7ff3ac09d4a0 2020-11-20T11:43:41.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.587+0000 7ff3a8ff9700 1 -- 172.21.15.23:0/4271578088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3ac09d4a0 msgr2=0x7ff3ac09d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:41.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.587+0000 7ff3a8ff9700 1 --2- 172.21.15.23:0/4271578088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3ac09d4a0 0x7ff3ac09d8b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7ff3ac061790 tx=0x7ff3a401b000).stop 2020-11-20T11:43:41.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.587+0000 7ff3a8ff9700 1 -- 172.21.15.23:0/4271578088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3ac09d4a0 msgr2=0x7ff3ac09d8b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:43:41.593 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.587+0000 7ff3a8ff9700 1 --2- 172.21.15.23:0/4271578088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3ac09d4a0 0x7ff3ac09d8b0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:41.594 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:43:41.594 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:43:41.594 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:43:41.595 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:43:41.595 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 102, 2020-11-20T11:43:41.595 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:43:41.595 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:43:41.595 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:43:41.596 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:43:41.596 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:41.596 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 16, 2020-11-20T11:43:41.596 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:43:41.596 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:43:41.597 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:43:41.597 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:43:41.597 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:43:41.597 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:43:41.598 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:43:41.598 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:43:41.598 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:43:41.598 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:43:41.598 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:43:41.599 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:41.599 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:43:41.599 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:43:41.599 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:43:41.599 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:43:41.600 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:43:41.600 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:43:41.600 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:43:41.600 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:43:41.600 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:43:41.601 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:43:41.601 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:41.601 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:41.601 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:43:41.601 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:43:41.602 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:43:41.602 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:43:41.602 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:43:41.602 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:43:41.602 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:43:41.603 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:43:41.603 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:43:41.603 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:43:41.603 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:43:41.603 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:43:41.604 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:43:41.604 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:43:41.604 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:43:41.604 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:43:41.604 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:43:41.605 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:43:41.605 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:43:41.605 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:43:41.605 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:43:41.605 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:43:41.606 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:43:41.606 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:41.606 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:43:41.606 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:41.606 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:43:41.607 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:41.607 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:43:41.607 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:43:41.607 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:43:41.607 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:43:41.608 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:41.608 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:43:41.608 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:43:41.608 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:41.608 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:41.609 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:41.609 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:43:41.609 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:43:41.609 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:41.609 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:41.610 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:41.610 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:41.610 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:43:41.610 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:43:41.610 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:43:41.610 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:43:41.611 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:43:41.611 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:41.611 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:41.611 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:43:41.612 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:43:41.612 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:43:41.612 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:43:41.612 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:41.612 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:43:41.613 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:43:41.613 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:41.613 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:41.613 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:41.613 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:43:41.614 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:43:41.614 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:41.614 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:41.614 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:41.614 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:41.615 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:43:41.615 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:43:41.615 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:43:41.615 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:43:41.615 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:43:41.616 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:41.616 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:41.616 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:43:41.616 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:43:41.616 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:43:41.617 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:43:41.617 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:41.617 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:43:41.617 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:43:41.617 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:41.618 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:41.618 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:41.618 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:43:41.618 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:43:41.618 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:41.619 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:41.619 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:41.619 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:41.619 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:43:41.619 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:43:41.619 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:43:41.620 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:43:41.620 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:43:41.620 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:41.620 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:41.620 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:41.621 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:43:41.621 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:43:41.621 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:41.621 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:43:41.621 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:43:41.622 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:43:41.622 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:41.622 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:41.622 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:43:41.622 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:41.623 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:43:41.623 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:43:41.623 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:43:41.623 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:41.623 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:41.624 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:43:41.624 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:41.624 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:43:41.624 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:43:41.625 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 8 2020-11-20T11:43:41.626 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:41.627 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:41.627 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:43:41.628 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:41.628 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:43:41.628 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:43:41.629 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 4 2020-11-20T11:43:41.629 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:41.629 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:41.629 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:43:41.629 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:41.630 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:43:41.630 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:43:41.630 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:43:41.630 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:41.630 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:41.631 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:41.631 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:43:41.631 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:43:41.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.590+0000 7ff3927fc700 1 -- 172.21.15.23:0/4271578088 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff388056db0 msgr2=0x7ff388059270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:41.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.590+0000 7ff3927fc700 1 --2- 172.21.15.23:0/4271578088 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff388056db0 0x7ff388059270 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7ff3a00011e0 tx=0x7ff3a00079f0).stop 2020-11-20T11:43:41.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.590+0000 7ff3927fc700 1 -- 172.21.15.23:0/4271578088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3ac063fe0 msgr2=0x7ff3ac066080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:41.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.590+0000 7ff3927fc700 1 --2- 172.21.15.23:0/4271578088 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff3ac063fe0 0x7ff3ac066080 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7ff3ac0ad970 tx=0x7ff39c01d000).stop 2020-11-20T11:43:41.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.590+0000 7ff3b16fc700 1 -- 172.21.15.23:0/4271578088 reap_dead start 2020-11-20T11:43:41.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.590+0000 7ff3b16fc700 1 -- 172.21.15.23:0/4271578088 reap_dead start 2020-11-20T11:43:41.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.590+0000 7ff3927fc700 1 -- 172.21.15.23:0/4271578088 shutdown_connections 2020-11-20T11:43:41.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.590+0000 7ff3927fc700 1 -- 172.21.15.23:0/4271578088 >> 172.21.15.23:0/4271578088 conn(0x7ff3ac060c80 msgr2=0x7ff3ac14a280 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:41.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.590+0000 7ff3927fc700 1 -- 172.21.15.23:0/4271578088 shutdown_connections 2020-11-20T11:43:41.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.590+0000 7ff3927fc700 1 -- 172.21.15.23:0/4271578088 wait complete. 2020-11-20T11:43:41.643 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:43:41.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.917+0000 7f37809d2700 1 Processor -- start 2020-11-20T11:43:41.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.917+0000 7f37809d2700 1 -- start start 2020-11-20T11:43:41.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.918+0000 7f37809d2700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f377c0a8f90 0x7f377c0a6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:41.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.918+0000 7f37809d2700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f377c0a5000 con 0x7f377c061380 2020-11-20T11:43:41.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.918+0000 7f37809d2700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f377c0a5d30 con 0x7f377c0a68f0 2020-11-20T11:43:41.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.918+0000 7f37809d2700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f377c0a5fd0 con 0x7f377c0a8f90 2020-11-20T11:43:41.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.918+0000 7f377b59e700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f377c0a68f0 0x7f377c0a6da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:37496/0 (socket says 172.21.15.23:37496) 2020-11-20T11:43:41.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.918+0000 7f377b59e700 1 -- 172.21.15.23:0/3630313253 learned_addr learned my addr 172.21.15.23:0/3630313253 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:43:41.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.920+0000 7f377a59c700 1 --2- 172.21.15.23:0/3630313253 >> v2:172.21.15.155:3300/0 conn(0x7f377c0a8f90 0x7f377c0a6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:41.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.921+0000 7f377a59c700 1 -- 172.21.15.23:0/3630313253 >> v1:172.21.15.155:6789/0 conn(0x7f377c0a68f0 legacy=0x7f377c0a6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:41.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.921+0000 7f377a59c700 1 -- 172.21.15.23:0/3630313253 >> v1:172.21.15.23:6789/0 conn(0x7f377c061380 legacy=0x7f377c0a8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:43:41.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.923+0000 7f3779d9b700 1 -- 172.21.15.23:0/3630313253 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2471318329 0 0) 0x7f377c0a5000 con 0x7f377c061380 2020-11-20T11:43:41.929 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.926+0000 7f377a59c700 1 -- 172.21.15.23:0/3630313253 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f377c0a8140 con 0x7f377c0a8f90 2020-11-20T11:43:41.930 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.927+0000 7f377a59c700 1 --2- 172.21.15.23:0/3630313253 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f377c0a8f90 0x7f377c0a6400 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f3770006fd0 tx=0x7f3770006e30).ready entity=mon.1 client_cookie=b0cee08a78b68a16 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:41.930 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.927+0000 7f3779d9b700 1 -- 172.21.15.23:0/3630313253 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3770009760 con 0x7f377c0a8f90 2020-11-20T11:43:41.930 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.927+0000 7f3779d9b700 1 -- 172.21.15.23:0/3630313253 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3770000e90 con 0x7f377c0a8f90 2020-11-20T11:43:41.931 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.927+0000 7f3779d9b700 1 -- 172.21.15.23:0/3630313253 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3770009ad0 con 0x7f377c0a8f90 2020-11-20T11:43:41.931 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.928+0000 7f37809d2700 1 -- 172.21.15.23:0/3630313253 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f377c0a8f90 msgr2=0x7f377c0a6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:41.931 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.928+0000 7f37809d2700 1 --2- 172.21.15.23:0/3630313253 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f377c0a8f90 0x7f377c0a6400 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f3770006fd0 tx=0x7f3770006e30).stop 2020-11-20T11:43:41.932 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.930+0000 7f37809d2700 1 -- 172.21.15.23:0/3630313253 shutdown_connections 2020-11-20T11:43:41.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.930+0000 7f37809d2700 1 --2- 172.21.15.23:0/3630313253 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f377c0a8f90 0x7f377c0a6400 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:41.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.930+0000 7f37809d2700 1 -- 172.21.15.23:0/3630313253 >> 172.21.15.23:0/3630313253 conn(0x7f377c0ae0a0 msgr2=0x7f377c0aa0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:41.934 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.931+0000 7f37809d2700 1 -- 172.21.15.23:0/3630313253 shutdown_connections 2020-11-20T11:43:41.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.932+0000 7f37809d2700 1 -- 172.21.15.23:0/3630313253 wait complete. 2020-11-20T11:43:41.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.933+0000 7f37809d2700 1 Processor -- start 2020-11-20T11:43:41.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.933+0000 7f37809d2700 1 -- start start 2020-11-20T11:43:41.942 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.934+0000 7f37809d2700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f377c061380 0x7f377c13f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:41.942 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.934+0000 7f37809d2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f377c0a68f0 0x7f377c065920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:41.942 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.934+0000 7f37809d2700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f377c0a8f90 0x7f377c09dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:41.942 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.934+0000 7f37809d2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377c09fd80 con 0x7f377c0a68f0 2020-11-20T11:43:41.942 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.934+0000 7f37809d2700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377c13fb90 con 0x7f377c0a8f90 2020-11-20T11:43:41.943 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.934+0000 7f37809d2700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f377c09e1b0 con 0x7f377c061380 2020-11-20T11:43:41.943 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.934+0000 7f377a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f377c0a68f0 0x7f377c065920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:41.943 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.934+0000 7f377a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f377c0a68f0 0x7f377c065920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45280/0 (socket says 172.21.15.23:45280) 2020-11-20T11:43:41.943 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.934+0000 7f377a59c700 1 -- 172.21.15.23:0/2984032652 learned_addr learned my addr 172.21.15.23:0/2984032652 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:43:41.943 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.935+0000 7f377ad9d700 1 --2- 172.21.15.23:0/2984032652 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f377c061380 0x7f377c13f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:41.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.935+0000 7f377a59c700 1 -- 172.21.15.23:0/2984032652 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f377c061380 msgr2=0x7f377c13f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:41.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.935+0000 7f377a59c700 1 --2- 172.21.15.23:0/2984032652 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f377c061380 0x7f377c13f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:41.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.935+0000 7f377a59c700 1 -- 172.21.15.23:0/2984032652 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f377c0a8f90 msgr2=0x7f377c09dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:41.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.935+0000 7f377a59c700 1 --2- 172.21.15.23:0/2984032652 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f377c0a8f90 0x7f377c09dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:41.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.935+0000 7f377a59c700 1 -- 172.21.15.23:0/2984032652 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f377c09d4a0 con 0x7f377c0a68f0 2020-11-20T11:43:41.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.935+0000 7f377a59c700 1 --2- 172.21.15.23:0/2984032652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f377c0a68f0 0x7f377c065920 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f377000b190 tx=0x7f3770015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:41.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.935+0000 7f3762ffd700 1 -- 172.21.15.23:0/2984032652 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3770011a50 con 0x7f377c0a68f0 2020-11-20T11:43:41.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.935+0000 7f3762ffd700 1 -- 172.21.15.23:0/2984032652 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3770011bb0 con 0x7f377c0a68f0 2020-11-20T11:43:41.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.935+0000 7f37809d2700 1 -- 172.21.15.23:0/2984032652 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f377c13f8f0 con 0x7f377c0a68f0 2020-11-20T11:43:41.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.935+0000 7f3762ffd700 1 -- 172.21.15.23:0/2984032652 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3770040470 con 0x7f377c0a68f0 2020-11-20T11:43:41.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.935+0000 7f37809d2700 1 -- 172.21.15.23:0/2984032652 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f377c09eb70 con 0x7f377c0a68f0 2020-11-20T11:43:41.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.936+0000 7f37809d2700 1 --2- 172.21.15.23:0/2984032652 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f377c14d6f0 0x7f377c14df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:41.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.936+0000 7f37809d2700 1 -- 172.21.15.23:0/2984032652 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377c069c70 con 0x7f377c14d6f0 2020-11-20T11:43:41.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.937+0000 7f3762ffd700 1 -- 172.21.15.23:0/2984032652 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3770047030 con 0x7f377c0a68f0 2020-11-20T11:43:41.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.937+0000 7f3762ffd700 1 --2- 172.21.15.23:0/2984032652 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3758056d60 0x7f3758059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:41.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.937+0000 7f3762ffd700 1 -- 172.21.15.23:0/2984032652 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f3770093700 con 0x7f377c0a68f0 2020-11-20T11:43:41.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.937+0000 7f377b59e700 1 --2- 172.21.15.23:0/2984032652 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3758056d60 0x7f3758059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:41.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.938+0000 7f377ad9d700 1 --2- 172.21.15.23:0/2984032652 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f377c14d6f0 0x7f377c14df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:41.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.938+0000 7f377b59e700 1 --2- 172.21.15.23:0/2984032652 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3758056d60 0x7f3758059220 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f3774005960 tx=0x7f3774007b10).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:41.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.943+0000 7f377ad9d700 1 -- 172.21.15.23:0/2984032652 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f377c09d9e0 con 0x7f377c14d6f0 2020-11-20T11:43:41.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.943+0000 7f377ad9d700 1 --2- 172.21.15.23:0/2984032652 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f377c14d6f0 0x7f377c14df50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f376c00f770 tx=0x7f376c009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:41.952 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.946+0000 7f3762ffd700 1 -- 172.21.15.23:0/2984032652 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376c011590 con 0x7f377c14d6f0 2020-11-20T11:43:41.953 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.946+0000 7f3762ffd700 1 -- 172.21.15.23:0/2984032652 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f376c0116f0 con 0x7f377c14d6f0 2020-11-20T11:43:41.953 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.946+0000 7f3762ffd700 1 -- 172.21.15.23:0/2984032652 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f377c14d6f0 msgr2=0x7f377c14df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:41.953 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.946+0000 7f3762ffd700 1 --2- 172.21.15.23:0/2984032652 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f377c14d6f0 0x7f377c14df50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f376c00f770 tx=0x7f376c009040).stop 2020-11-20T11:43:41.953 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.946+0000 7f3762ffd700 1 -- 172.21.15.23:0/2984032652 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f377c14d6f0 msgr2=0x7f377c14df50 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:43:41.954 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.946+0000 7f3762ffd700 1 --2- 172.21.15.23:0/2984032652 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f377c14d6f0 0x7f377c14df50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:41.964 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.958+0000 7f37809d2700 1 --2- 172.21.15.23:0/2984032652 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f377c061980 0x7f377c0a2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:41.964 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.958+0000 7f37809d2700 1 -- 172.21.15.23:0/2984032652 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377c057dc0 con 0x7f377c061980 2020-11-20T11:43:41.964 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.958+0000 7f377ad9d700 1 --2- 172.21.15.23:0/2984032652 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f377c061980 0x7f377c0a2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:41.964 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.959+0000 7f377ad9d700 1 -- 172.21.15.23:0/2984032652 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f377c044710 con 0x7f377c061980 2020-11-20T11:43:41.965 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.959+0000 7f377ad9d700 1 --2- 172.21.15.23:0/2984032652 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f377c061980 0x7f377c0a2900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f376c015000 tx=0x7f376c018000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:41.965 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.960+0000 7f3762ffd700 1 -- 172.21.15.23:0/2984032652 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376c014c50 con 0x7f377c061980 2020-11-20T11:43:41.965 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.960+0000 7f3762ffd700 1 -- 172.21.15.23:0/2984032652 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4445 (secure 0 0 0) 0x7f376c014db0 con 0x7f377c061980 2020-11-20T11:43:41.965 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.960+0000 7f3762ffd700 1 -- 172.21.15.23:0/2984032652 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f377c061980 msgr2=0x7f377c0a2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:41.965 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.960+0000 7f3762ffd700 1 --2- 172.21.15.23:0/2984032652 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f377c061980 0x7f377c0a2900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f376c015000 tx=0x7f376c018000).stop 2020-11-20T11:43:41.966 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.960+0000 7f3762ffd700 1 -- 172.21.15.23:0/2984032652 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f377c061980 msgr2=0x7f377c0a2900 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:43:41.966 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.960+0000 7f3762ffd700 1 --2- 172.21.15.23:0/2984032652 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f377c061980 0x7f377c0a2900 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:41.966 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.960+0000 7f37809d2700 1 -- 172.21.15.23:0/2984032652 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3758056d60 msgr2=0x7f3758059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:41.966 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.960+0000 7f37809d2700 1 --2- 172.21.15.23:0/2984032652 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3758056d60 0x7f3758059220 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f3774005960 tx=0x7f3774007b10).stop 2020-11-20T11:43:41.966 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.960+0000 7f37809d2700 1 -- 172.21.15.23:0/2984032652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f377c0a68f0 msgr2=0x7f377c065920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:41.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.960+0000 7f37809d2700 1 --2- 172.21.15.23:0/2984032652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f377c0a68f0 0x7f377c065920 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f377000b190 tx=0x7f3770015000).stop 2020-11-20T11:43:41.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.960+0000 7f377b59e700 1 -- 172.21.15.23:0/2984032652 reap_dead start 2020-11-20T11:43:41.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.960+0000 7f37809d2700 1 -- 172.21.15.23:0/2984032652 shutdown_connections 2020-11-20T11:43:41.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.960+0000 7f37809d2700 1 --2- 172.21.15.23:0/2984032652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f377c0a68f0 0x7f377c065920 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:41.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.960+0000 7f37809d2700 1 -- 172.21.15.23:0/2984032652 >> 172.21.15.23:0/2984032652 conn(0x7f377c0ae0a0 msgr2=0x7f377c0a61d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:41.968 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.960+0000 7f37809d2700 1 -- 172.21.15.23:0/2984032652 shutdown_connections 2020-11-20T11:43:41.968 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:43:41.968 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:43:41.968 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:43:41.969 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:43:41.969 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 102, 2020-11-20T11:43:41.969 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:43:41.969 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:43:41.969 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:43:41.970 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:43:41.970 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:41.970 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 16, 2020-11-20T11:43:41.970 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:43:41.970 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:43:41.971 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:43:41.971 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:43:41.971 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:43:41.971 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:43:41.972 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:43:41.972 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:43:41.972 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:43:41.972 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:43:41.972 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:43:41.973 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:41.973 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:43:41.973 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:43:41.973 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:43:41.974 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:43:41.974 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:43:41.974 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:43:41.975 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:43:41.975 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:43:41.975 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:43:41.975 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:43:41.975 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:41.976 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:41.976 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:43:41.976 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:43:41.976 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:43:41.976 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:43:41.977 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:43:41.977 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:43:41.977 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:43:41.977 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:43:41.978 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:43:41.978 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:43:41.978 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:43:41.978 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:43:41.978 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:43:41.979 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:43:41.979 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:43:41.979 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:43:41.979 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:43:41.979 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:43:41.980 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:43:41.980 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:43:41.980 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:43:41.980 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:43:41.980 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:43:41.981 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:41.981 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:43:41.981 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:41.981 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:43:41.981 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:41.982 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:43:41.982 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:43:41.982 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:43:41.982 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:43:41.982 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:41.983 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:43:41.983 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:43:41.983 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:41.983 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:41.984 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:41.984 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:43:41.984 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:43:41.984 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:41.984 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:41.985 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:41.985 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:41.985 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:43:41.985 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:43:41.986 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:43:41.986 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:43:41.986 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:43:41.986 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:41.987 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:41.987 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:43:41.987 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:43:41.987 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:43:41.987 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:43:41.988 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:41.988 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:43:41.988 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:43:41.988 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:41.989 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:41.989 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:41.990 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:43:41.990 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:43:41.990 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:41.991 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:41.991 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:41.991 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:41.991 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:43:41.992 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:43:41.992 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:43:41.992 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:43:41.992 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:43:41.992 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:41.993 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:41.993 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:43:41.993 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:43:41.993 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:43:41.993 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:43:41.994 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:41.994 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:43:41.994 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:43:41.994 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:41.994 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:41.995 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:41.995 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:43:41.995 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:43:41.995 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:41.995 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:41.996 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:41.996 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:41.996 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:43:41.996 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:43:41.996 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:43:41.997 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:43:41.997 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:43:41.997 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:41.997 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:41.998 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:41.998 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:43:41.998 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:43:41.998 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:41.998 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:43:41.999 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:43:41.999 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:43:41.999 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:41.999 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:41.999 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:43:42.000 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:42.000 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:43:42.000 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:43:42.000 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:43:42.000 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:42.001 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:42.001 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:43:42.001 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:42.001 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:43:42.001 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:43:42.002 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:43:42.002 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:42.002 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:42.002 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:42.002 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:43:42.003 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:43:42.003 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:41.966+0000 7f37809d2700 1 -- 172.21.15.23:0/2984032652 wait complete. 2020-11-20T11:43:42.021 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:43:42.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.334+0000 7f274feed700 1 Processor -- start 2020-11-20T11:43:42.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.334+0000 7f274feed700 1 -- start start 2020-11-20T11:43:42.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.334+0000 7f274feed700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f27480619d0 0x7f27480a9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:42.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.335+0000 7f274feed700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f2748066170 0x7f2748137e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:42.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.335+0000 7f274dc89700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f27480619d0 0x7f27480a9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:42.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.335+0000 7f274dc89700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f27480619d0 0x7f27480a9260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:38184/0 (socket says 172.21.15.23:38184) 2020-11-20T11:43:42.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.335+0000 7f274e48a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f2748066170 0x7f2748137e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:42.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.335+0000 7f274e48a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f2748066170 0x7f2748137e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45292/0 (socket says 172.21.15.23:45292) 2020-11-20T11:43:42.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.335+0000 7f274dc89700 1 -- 172.21.15.23:0/2356694412 learned_addr learned my addr 172.21.15.23:0/2356694412 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:43:42.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.335+0000 7f274feed700 1 -- 172.21.15.23:0/2356694412 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27480610b0 con 0x7f2748061db0 2020-11-20T11:43:42.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.335+0000 7f274feed700 1 -- 172.21.15.23:0/2356694412 --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27480a9750 con 0x7f2748066170 2020-11-20T11:43:42.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.335+0000 7f274feed700 1 -- 172.21.15.23:0/2356694412 --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2748066680 con 0x7f27480619d0 2020-11-20T11:43:42.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.335+0000 7f274d488700 1 --1- 172.21.15.23:0/2356694412 >> v1:172.21.15.155:6789/0 conn(0x7f2748061db0 0x7f27480609a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:37514/0 (socket says 172.21.15.23:37514) 2020-11-20T11:43:42.346 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.336+0000 7f274cc87700 1 -- 172.21.15.23:0/2356694412 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3041529999 0 0) 0x7f27480610b0 con 0x7f2748061db0 2020-11-20T11:43:42.346 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.337+0000 7f274cc87700 1 -- 172.21.15.23:0/2356694412 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2730003280 con 0x7f2748061db0 2020-11-20T11:43:42.346 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.337+0000 7f274e48a700 1 -- 172.21.15.23:0/2356694412 >> v2:172.21.15.155:3300/0 conn(0x7f27480619d0 msgr2=0x7f27480a9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:43:42.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.337+0000 7f274e48a700 1 --2- 172.21.15.23:0/2356694412 >> v2:172.21.15.155:3300/0 conn(0x7f27480619d0 0x7f27480a9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:42.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.337+0000 7f274e48a700 1 -- 172.21.15.23:0/2356694412 >> v1:172.21.15.155:6789/0 conn(0x7f2748061db0 legacy=0x7f27480609a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:42.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.337+0000 7f274e48a700 1 -- 172.21.15.23:0/2356694412 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2748066920 con 0x7f2748066170 2020-11-20T11:43:42.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.337+0000 7f274e48a700 1 --2- 172.21.15.23:0/2356694412 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2748066170 0x7f2748137e90 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f27480af030 tx=0x7f2740009220).ready entity=mon.0 client_cookie=6049e4b9a568b522 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:42.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.337+0000 7f274cc87700 1 -- 172.21.15.23:0/2356694412 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274000b950 con 0x7f2748066170 2020-11-20T11:43:42.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.337+0000 7f274cc87700 1 -- 172.21.15.23:0/2356694412 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f274000bab0 con 0x7f2748066170 2020-11-20T11:43:42.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.337+0000 7f274cc87700 1 -- 172.21.15.23:0/2356694412 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274000be30 con 0x7f2748066170 2020-11-20T11:43:42.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.338+0000 7f274feed700 1 -- 172.21.15.23:0/2356694412 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2748066170 msgr2=0x7f2748137e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:42.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.338+0000 7f274feed700 1 --2- 172.21.15.23:0/2356694412 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2748066170 0x7f2748137e90 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f27480af030 tx=0x7f2740009220).stop 2020-11-20T11:43:42.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.338+0000 7f274feed700 1 -- 172.21.15.23:0/2356694412 shutdown_connections 2020-11-20T11:43:42.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.338+0000 7f274feed700 1 --2- 172.21.15.23:0/2356694412 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2748066170 0x7f2748137e90 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:42.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.338+0000 7f274feed700 1 --2- 172.21.15.23:0/2356694412 >> v2:172.21.15.155:3300/0 conn(0x7f27480619d0 0x7f27480a9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:42.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.338+0000 7f274feed700 1 -- 172.21.15.23:0/2356694412 >> 172.21.15.23:0/2356694412 conn(0x7f2748063640 msgr2=0x7f27480612c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:42.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.338+0000 7f274feed700 1 -- 172.21.15.23:0/2356694412 shutdown_connections 2020-11-20T11:43:42.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.338+0000 7f274feed700 1 -- 172.21.15.23:0/2356694412 wait complete. 2020-11-20T11:43:42.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.339+0000 7f274feed700 1 Processor -- start 2020-11-20T11:43:42.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.339+0000 7f274feed700 1 -- start start 2020-11-20T11:43:42.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.340+0000 7f274feed700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27480619d0 0x7f27480abec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:42.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.340+0000 7f274feed700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2748061db0 0x7f274814ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:42.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.340+0000 7f274feed700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2748066170 0x7f27480ae490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:42.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.340+0000 7f274feed700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274814a360 con 0x7f27480619d0 2020-11-20T11:43:42.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.340+0000 7f274feed700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274814a600 con 0x7f2748066170 2020-11-20T11:43:42.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.340+0000 7f274feed700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f274814a8a0 con 0x7f2748061db0 2020-11-20T11:43:42.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.340+0000 7f274dc89700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27480619d0 0x7f27480abec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:42.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.340+0000 7f274e48a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2748066170 0x7f27480ae490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:42.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.340+0000 7f274d488700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2748061db0 0x7f274814ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:42.354 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.340+0000 7f274dc89700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27480619d0 0x7f27480abec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45294/0 (socket says 172.21.15.23:45294) 2020-11-20T11:43:42.354 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.340+0000 7f274dc89700 1 -- 172.21.15.23:0/488935672 learned_addr learned my addr 172.21.15.23:0/488935672 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:43:42.354 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.340+0000 7f274e48a700 1 --2- 172.21.15.23:0/488935672 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2748066170 0x7f27480ae490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:38194/0 (socket says 172.21.15.23:38194) 2020-11-20T11:43:42.354 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.340+0000 7f274dc89700 1 -- 172.21.15.23:0/488935672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2748061db0 msgr2=0x7f274814ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:42.354 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.340+0000 7f274dc89700 1 --2- 172.21.15.23:0/488935672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2748061db0 0x7f274814ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:42.355 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.340+0000 7f274dc89700 1 -- 172.21.15.23:0/488935672 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2748066170 msgr2=0x7f27480ae490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:42.355 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.340+0000 7f274dc89700 1 --2- 172.21.15.23:0/488935672 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2748066170 0x7f27480ae490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:42.355 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.340+0000 7f274dc89700 1 -- 172.21.15.23:0/488935672 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27480aeca0 con 0x7f27480619d0 2020-11-20T11:43:42.355 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.340+0000 7f274d488700 1 --2- 172.21.15.23:0/488935672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2748061db0 0x7f274814ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:43:42.355 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.340+0000 7f274dc89700 1 --2- 172.21.15.23:0/488935672 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27480619d0 0x7f27480abec0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f274400eba0 tx=0x7f2744006ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:42.356 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.340+0000 7f2735ffb700 1 -- 172.21.15.23:0/488935672 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274400c850 con 0x7f27480619d0 2020-11-20T11:43:42.356 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.340+0000 7f2735ffb700 1 -- 172.21.15.23:0/488935672 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f274400c9b0 con 0x7f27480619d0 2020-11-20T11:43:42.356 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.340+0000 7f274feed700 1 -- 172.21.15.23:0/488935672 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2748141d40 con 0x7f27480619d0 2020-11-20T11:43:42.356 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.341+0000 7f2735ffb700 1 -- 172.21.15.23:0/488935672 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274400cd30 con 0x7f27480619d0 2020-11-20T11:43:42.356 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.341+0000 7f274feed700 1 -- 172.21.15.23:0/488935672 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27481422d0 con 0x7f27480619d0 2020-11-20T11:43:42.357 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.341+0000 7f274feed700 1 --2- 172.21.15.23:0/488935672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f274805c070 0x7f2748133230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:42.357 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.342+0000 7f274feed700 1 -- 172.21.15.23:0/488935672 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f274814a1e0 con 0x7f274805c070 2020-11-20T11:43:42.357 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.342+0000 7f274d488700 1 --2- 172.21.15.23:0/488935672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f274805c070 0x7f2748133230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:42.357 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.342+0000 7f2735ffb700 1 -- 172.21.15.23:0/488935672 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2744022070 con 0x7f27480619d0 2020-11-20T11:43:42.357 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.342+0000 7f274d488700 1 -- 172.21.15.23:0/488935672 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f27480aee90 con 0x7f274805c070 2020-11-20T11:43:42.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.342+0000 7f274d488700 1 --2- 172.21.15.23:0/488935672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f274805c070 0x7f2748133230 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f2748147d80 tx=0x7f2738007a50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:42.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.342+0000 7f2735ffb700 1 --2- 172.21.15.23:0/488935672 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2720056d60 0x7f2720059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:42.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.342+0000 7f2735ffb700 1 -- 172.21.15.23:0/488935672 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f2744070390 con 0x7f27480619d0 2020-11-20T11:43:42.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.342+0000 7f2735ffb700 1 -- 172.21.15.23:0/488935672 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f273800fa30 con 0x7f274805c070 2020-11-20T11:43:42.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.342+0000 7f274e48a700 1 --2- 172.21.15.23:0/488935672 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2720056d60 0x7f2720059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:42.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.343+0000 7f2735ffb700 1 -- 172.21.15.23:0/488935672 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f273800fb90 con 0x7f274805c070 2020-11-20T11:43:42.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.343+0000 7f2735ffb700 1 -- 172.21.15.23:0/488935672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f274805c070 msgr2=0x7f2748133230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:42.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.343+0000 7f2735ffb700 1 --2- 172.21.15.23:0/488935672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f274805c070 0x7f2748133230 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f2748147d80 tx=0x7f2738007a50).stop 2020-11-20T11:43:42.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.343+0000 7f2735ffb700 1 -- 172.21.15.23:0/488935672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f274805c070 msgr2=0x7f2748133230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:43:42.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.343+0000 7f2735ffb700 1 --2- 172.21.15.23:0/488935672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f274805c070 0x7f2748133230 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:42.360 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.343+0000 7f274e48a700 1 --2- 172.21.15.23:0/488935672 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2720056d60 0x7f2720059220 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f27400090b0 tx=0x7f2740016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:42.361 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.358+0000 7f274feed700 1 --2- 172.21.15.23:0/488935672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27480622d0 0x7f2748133720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:42.361 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.358+0000 7f274feed700 1 -- 172.21.15.23:0/488935672 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f274813e300 con 0x7f27480622d0 2020-11-20T11:43:42.361 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.358+0000 7f274d488700 1 --2- 172.21.15.23:0/488935672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27480622d0 0x7f2748133720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:42.363 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.358+0000 7f274d488700 1 -- 172.21.15.23:0/488935672 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2748138860 con 0x7f27480622d0 2020-11-20T11:43:42.363 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.358+0000 7f274d488700 1 --2- 172.21.15.23:0/488935672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27480622d0 0x7f2748133720 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f2738016000 tx=0x7f2738017000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:42.364 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.359+0000 7f2735ffb700 1 -- 172.21.15.23:0/488935672 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27380150d0 con 0x7f27480622d0 2020-11-20T11:43:42.364 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.359+0000 7f2735ffb700 1 -- 172.21.15.23:0/488935672 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4275 (secure 0 0 0) 0x7f2738015230 con 0x7f27480622d0 2020-11-20T11:43:42.364 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.359+0000 7f2735ffb700 1 -- 172.21.15.23:0/488935672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27480622d0 msgr2=0x7f2748133720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:42.364 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.359+0000 7f2735ffb700 1 --2- 172.21.15.23:0/488935672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27480622d0 0x7f2748133720 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f2738016000 tx=0x7f2738017000).stop 2020-11-20T11:43:42.364 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.359+0000 7f2735ffb700 1 -- 172.21.15.23:0/488935672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27480622d0 msgr2=0x7f2748133720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:43:42.365 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.359+0000 7f2735ffb700 1 --2- 172.21.15.23:0/488935672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27480622d0 0x7f2748133720 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:42.365 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.359+0000 7f274feed700 1 -- 172.21.15.23:0/488935672 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2720056d60 msgr2=0x7f2720059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:42.365 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.359+0000 7f274feed700 1 --2- 172.21.15.23:0/488935672 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2720056d60 0x7f2720059220 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f27400090b0 tx=0x7f2740016000).stop 2020-11-20T11:43:42.365 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.359+0000 7f274feed700 1 -- 172.21.15.23:0/488935672 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27480619d0 msgr2=0x7f27480abec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:42.366 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.359+0000 7f274feed700 1 --2- 172.21.15.23:0/488935672 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27480619d0 0x7f27480abec0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f274400eba0 tx=0x7f2744006ef0).stop 2020-11-20T11:43:42.366 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.359+0000 7f274e48a700 1 -- 172.21.15.23:0/488935672 reap_dead start 2020-11-20T11:43:42.366 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.359+0000 7f274e48a700 1 -- 172.21.15.23:0/488935672 reap_dead start 2020-11-20T11:43:42.366 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.359+0000 7f274feed700 1 -- 172.21.15.23:0/488935672 shutdown_connections 2020-11-20T11:43:42.366 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.360+0000 7f274feed700 1 -- 172.21.15.23:0/488935672 >> 172.21.15.23:0/488935672 conn(0x7f2748063640 msgr2=0x7f27480608c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:42.367 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.360+0000 7f274feed700 1 -- 172.21.15.23:0/488935672 shutdown_connections 2020-11-20T11:43:42.367 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.360+0000 7f274feed700 1 -- 172.21.15.23:0/488935672 wait complete. 2020-11-20T11:43:42.367 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:43:42.367 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:43:42.367 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:43:42.368 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:43:42.368 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 102, 2020-11-20T11:43:42.368 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:43:42.368 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:43:42.368 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:43:42.369 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:43:42.369 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:42.369 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 16, 2020-11-20T11:43:42.369 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:43:42.370 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:43:42.370 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:43:42.370 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:43:42.370 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:43:42.370 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:43:42.370 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:43:42.371 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:43:42.371 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:43:42.371 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:43:42.371 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:43:42.371 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:42.372 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:43:42.372 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:43:42.372 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:43:42.372 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:43:42.372 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:43:42.373 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:43:42.373 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:43:42.373 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:43:42.373 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:43:42.373 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:43:42.374 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:42.374 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:42.374 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:43:42.374 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:43:42.374 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:43:42.375 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:43:42.375 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:43:42.375 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:43:42.375 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:43:42.375 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:43:42.376 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:43:42.376 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:43:42.376 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:43:42.376 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:43:42.376 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:43:42.377 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:43:42.377 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:43:42.377 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:43:42.377 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:43:42.377 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:43:42.378 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:43:42.378 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:43:42.378 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:43:42.378 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:43:42.378 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:43:42.379 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:42.379 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:43:42.379 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:42.379 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:43:42.379 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:42.379 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:43:42.380 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:43:42.380 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:43:42.380 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:43:42.380 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:42.381 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:43:42.381 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:43:42.381 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:42.381 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:42.381 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:42.382 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:43:42.382 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:43:42.382 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:42.382 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:42.382 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:42.383 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:42.383 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:43:42.383 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:43:42.383 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:43:42.383 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:43:42.384 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:43:42.384 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:42.384 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:42.384 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:43:42.384 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:43:42.384 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:43:42.385 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:43:42.385 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:42.385 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:43:42.385 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:43:42.386 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:42.386 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:42.386 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:42.386 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:43:42.386 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:43:42.387 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:42.387 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:42.387 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:42.387 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:42.387 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:43:42.387 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:43:42.388 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:43:42.388 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:43:42.388 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:43:42.388 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:42.388 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:42.389 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:43:42.389 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:43:42.389 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:43:42.389 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:43:42.389 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:42.390 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:43:42.390 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:43:42.390 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:42.390 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:42.390 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:42.391 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:43:42.391 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:43:42.391 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:42.391 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:42.391 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:42.392 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:42.392 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:43:42.392 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:43:42.392 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:43:42.392 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:43:42.392 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:43:42.393 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:42.393 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:42.393 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:42.393 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:43:42.393 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:43:42.394 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:42.394 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:43:42.394 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:43:42.394 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:43:42.395 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:42.395 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:42.395 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:43:42.395 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:42.395 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:43:42.395 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:43:42.396 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:43:42.396 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:42.396 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:42.396 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:42.397 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:43:42.397 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:43:42.398 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2020-11-20T11:43:42.398 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2020-11-20T11:43:42.398 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2020-11-20T11:43:42.399 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2020-11-20T11:43:42.399 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2020-11-20T11:43:42.399 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T11:43:42.413 INFO:tasks.ceph.mon.a:Stopped 2020-11-20T11:43:42.413 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2020-11-20T11:43:42.414 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:43:42.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.586+0000 7f41f54c0700 1 Processor -- start 2020-11-20T11:43:42.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.586+0000 7f41f54c0700 1 -- start start 2020-11-20T11:43:42.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.587+0000 7f41f54c0700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41f0068260 0x7f41f0065780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:42.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.587+0000 7f41f54c0700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f41f0065c70 0x7f41f0137870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:42.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.587+0000 7f41f54c0700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f00a9000 con 0x7f41f0137d60 2020-11-20T11:43:42.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.587+0000 7f41f54c0700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41f0134c40 con 0x7f41f0065c70 2020-11-20T11:43:42.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.587+0000 7f41f54c0700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41f00663a0 con 0x7f41f0068260 2020-11-20T11:43:42.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.588+0000 7f41ee7fc700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f41f0065c70 msgr2=0x7f41f0137870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:43:42.593 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.588+0000 7f41ef7fe700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f41f0137d60 0x7f41f013a6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:37574/0 (socket says 172.21.15.23:37574) 2020-11-20T11:43:42.593 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.588+0000 7f41ef7fe700 1 -- 172.21.15.23:0/995134043 learned_addr learned my addr 172.21.15.23:0/995134043 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:43:42.593 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.588+0000 7f41ee7fc700 1 --2- 172.21.15.23:0/995134043 >> v2:172.21.15.23:3300/0 conn(0x7f41f0065c70 0x7f41f0137870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:43:42.594 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.588+0000 7f41eeffd700 1 --2- 172.21.15.23:0/995134043 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41f0068260 0x7f41f0065780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:42.594 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.589+0000 7f41eeffd700 1 -- 172.21.15.23:0/995134043 >> v2:172.21.15.23:3300/0 conn(0x7f41f0065c70 msgr2=0x7f41f0137870 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:43:42.594 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.589+0000 7f41eeffd700 1 --2- 172.21.15.23:0/995134043 >> v2:172.21.15.23:3300/0 conn(0x7f41f0065c70 0x7f41f0137870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:42.595 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.589+0000 7f41eeffd700 1 -- 172.21.15.23:0/995134043 >> v1:172.21.15.155:6789/0 conn(0x7f41f0137d60 legacy=0x7f41f013a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:42.595 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.589+0000 7f41eeffd700 1 -- 172.21.15.23:0/995134043 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f0137170 con 0x7f41f0068260 2020-11-20T11:43:42.595 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.589+0000 7f41eeffd700 1 --2- 172.21.15.23:0/995134043 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41f0068260 0x7f41f0065780 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f41e4006fd0 tx=0x7f41e4006e30).ready entity=mon.2 client_cookie=89f1d95b2224be58 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:42.597 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.592+0000 7f41edffb700 1 -- 172.21.15.23:0/995134043 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e40096e0 con 0x7f41f0068260 2020-11-20T11:43:42.597 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.592+0000 7f41edffb700 1 -- 172.21.15.23:0/995134043 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e4000e90 con 0x7f41f0068260 2020-11-20T11:43:42.597 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.592+0000 7f41edffb700 1 -- 172.21.15.23:0/995134043 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e4009a50 con 0x7f41f0068260 2020-11-20T11:43:42.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.592+0000 7f41f54c0700 1 -- 172.21.15.23:0/995134043 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41f0068260 msgr2=0x7f41f0065780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:42.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.592+0000 7f41f54c0700 1 --2- 172.21.15.23:0/995134043 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41f0068260 0x7f41f0065780 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f41e4006fd0 tx=0x7f41e4006e30).stop 2020-11-20T11:43:42.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.592+0000 7f41f54c0700 1 -- 172.21.15.23:0/995134043 shutdown_connections 2020-11-20T11:43:42.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.592+0000 7f41f54c0700 1 --2- 172.21.15.23:0/995134043 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41f0068260 0x7f41f0065780 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:42.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.593+0000 7f41f54c0700 1 --2- 172.21.15.23:0/995134043 >> v2:172.21.15.23:3300/0 conn(0x7f41f0065c70 0x7f41f0137870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:42.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.593+0000 7f41f54c0700 1 -- 172.21.15.23:0/995134043 >> 172.21.15.23:0/995134043 conn(0x7f41f00ad7a0 msgr2=0x7f41f00adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:42.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.593+0000 7f41f54c0700 1 -- 172.21.15.23:0/995134043 shutdown_connections 2020-11-20T11:43:42.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.593+0000 7f41f54c0700 1 -- 172.21.15.23:0/995134043 wait complete. 2020-11-20T11:43:42.602 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.594+0000 7f41f54c0700 1 Processor -- start 2020-11-20T11:43:42.603 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.594+0000 7f41f54c0700 1 -- start start 2020-11-20T11:43:42.603 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.595+0000 7f41f54c0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41f0068260 0x7f41f00ac730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:42.603 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.595+0000 7f41f54c0700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41f0137d60 0x7f41f0127ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:42.603 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.595+0000 7f41f54c0700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41f01226f0 0x7f41f0122b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:42.604 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.595+0000 7f41f54c0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f00a7640 con 0x7f41f0068260 2020-11-20T11:43:42.604 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.595+0000 7f41f54c0700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f00a78e0 con 0x7f41f0137d60 2020-11-20T11:43:42.604 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.595+0000 7f41f54c0700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41f00acc20 con 0x7f41f01226f0 2020-11-20T11:43:42.604 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.595+0000 7f41eeffd700 1 -- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41f0068260 msgr2=0x7f41f00ac730 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:43:42.605 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.595+0000 7f41eeffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41f0068260 0x7f41f00ac730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:43:42.605 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.595+0000 7f41ef7fe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41f01226f0 0x7f41f0122b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:42.605 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.595+0000 7f41ee7fc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41f0137d60 0x7f41f0127ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:42.605 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.595+0000 7f41ef7fe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41f01226f0 0x7f41f0122b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:55598/0 (socket says 172.21.15.23:55598) 2020-11-20T11:43:42.606 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.595+0000 7f41ef7fe700 1 -- 172.21.15.23:0/832822408 learned_addr learned my addr 172.21.15.23:0/832822408 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:43:42.606 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.595+0000 7f41ef7fe700 1 -- 172.21.15.23:0/832822408 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41f0137d60 msgr2=0x7f41f0127ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:42.606 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.595+0000 7f41ef7fe700 1 --2- 172.21.15.23:0/832822408 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41f0137d60 0x7f41f0127ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:42.607 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.595+0000 7f41ef7fe700 1 -- 172.21.15.23:0/832822408 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41f0068260 msgr2=0x7f41f00ac730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:43:42.607 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.595+0000 7f41ef7fe700 1 --2- 172.21.15.23:0/832822408 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41f0068260 0x7f41f00ac730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:42.607 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.595+0000 7f41ef7fe700 1 -- 172.21.15.23:0/832822408 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f00a7250 con 0x7f41f01226f0 2020-11-20T11:43:42.608 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.595+0000 7f41ef7fe700 1 --2- 172.21.15.23:0/832822408 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41f01226f0 0x7f41f0122b70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f41e800cae0 tx=0x7f41e8007f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:42.608 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.595+0000 7f41d6ffd700 1 -- 172.21.15.23:0/832822408 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e8005bc0 con 0x7f41f01226f0 2020-11-20T11:43:42.608 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.595+0000 7f41d6ffd700 1 -- 172.21.15.23:0/832822408 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e8005d20 con 0x7f41f01226f0 2020-11-20T11:43:42.608 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.596+0000 7f41f54c0700 1 -- 172.21.15.23:0/832822408 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41f01283d0 con 0x7f41f01226f0 2020-11-20T11:43:42.608 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.596+0000 7f41d6ffd700 1 -- 172.21.15.23:0/832822408 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e8007320 con 0x7f41f01226f0 2020-11-20T11:43:42.609 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.596+0000 7f41f54c0700 1 -- 172.21.15.23:0/832822408 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41f00a57e0 con 0x7f41f01226f0 2020-11-20T11:43:42.609 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.596+0000 7f41f54c0700 1 -- 172.21.15.23:0/832822408 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41f0067c50 con 0x7f41f01226f0 2020-11-20T11:43:42.610 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.597+0000 7f41d6ffd700 1 -- 172.21.15.23:0/832822408 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f41e8010030 con 0x7f41f01226f0 2020-11-20T11:43:42.610 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.597+0000 7f41d6ffd700 1 --2- 172.21.15.23:0/832822408 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f41cc056d40 0x7f41cc059200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:42.610 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.597+0000 7f41d6ffd700 1 -- 172.21.15.23:0/832822408 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f41e806ffa0 con 0x7f41f01226f0 2020-11-20T11:43:42.611 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.598+0000 7f41eeffd700 1 --2- 172.21.15.23:0/832822408 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f41cc056d40 0x7f41cc059200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:42.611 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.598+0000 7f41eeffd700 1 --2- 172.21.15.23:0/832822408 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f41cc056d40 0x7f41cc059200 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f41e4006e70 tx=0x7f41e4015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:42.611 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.602+0000 7f41d6ffd700 1 -- 172.21.15.23:0/832822408 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f41f0067c50 con 0x7f41f01226f0 2020-11-20T11:43:42.895 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.892+0000 7f41f54c0700 1 -- 172.21.15.23:0/832822408 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f41f00a74a0 con 0x7f41f01226f0 2020-11-20T11:43:42.896 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.892+0000 7f41d6ffd700 1 -- 172.21.15.23:0/832822408 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1479 (secure 0 0 0) 0x7f41f00a74a0 con 0x7f41f01226f0 2020-11-20T11:43:42.896 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":102,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:43:42.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.894+0000 7f41f54c0700 1 -- 172.21.15.23:0/832822408 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f41cc056d40 msgr2=0x7f41cc059200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:42.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.894+0000 7f41f54c0700 1 --2- 172.21.15.23:0/832822408 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f41cc056d40 0x7f41cc059200 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f41e4006e70 tx=0x7f41e4015000).stop 2020-11-20T11:43:42.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.894+0000 7f41f54c0700 1 -- 172.21.15.23:0/832822408 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41f01226f0 msgr2=0x7f41f0122b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:42.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.894+0000 7f41f54c0700 1 --2- 172.21.15.23:0/832822408 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41f01226f0 0x7f41f0122b70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f41e800cae0 tx=0x7f41e8007f90).stop 2020-11-20T11:43:42.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.894+0000 7f41f54c0700 1 -- 172.21.15.23:0/832822408 shutdown_connections 2020-11-20T11:43:42.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.894+0000 7f41f54c0700 1 --2- 172.21.15.23:0/832822408 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f41cc056d40 0x7f41cc059200 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:42.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.894+0000 7f41f54c0700 1 --2- 172.21.15.23:0/832822408 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f41f01226f0 0x7f41f0122b70 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:42.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.894+0000 7f41f54c0700 1 --2- 172.21.15.23:0/832822408 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41f0068260 0x7f41f00ac730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:42.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.894+0000 7f41f54c0700 1 --2- 172.21.15.23:0/832822408 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f41f0137d60 0x7f41f0127ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:42.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.894+0000 7f41f54c0700 1 -- 172.21.15.23:0/832822408 >> 172.21.15.23:0/832822408 conn(0x7f41f00ad7a0 msgr2=0x7f41f005bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:42.900 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.894+0000 7f41f54c0700 1 -- 172.21.15.23:0/832822408 shutdown_connections 2020-11-20T11:43:42.900 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:42.894+0000 7f41f54c0700 1 -- 172.21.15.23:0/832822408 wait complete. 2020-11-20T11:43:45.912 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:43:46.179 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.175+0000 7fd6f372b700 1 Processor -- start 2020-11-20T11:43:46.179 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.176+0000 7fd6f372b700 1 -- start start 2020-11-20T11:43:46.181 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.176+0000 7fd6f372b700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd6ec061b90 0x7fd6ec0a92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:46.181 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.176+0000 7fd6f372b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd6ec0a97d0 0x7fd6ec13b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:46.182 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.176+0000 7fd6f372b700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6ec0a9c80 con 0x7fd6ec13b700 2020-11-20T11:43:46.182 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.176+0000 7fd6f372b700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6ec067190 con 0x7fd6ec0a97d0 2020-11-20T11:43:46.182 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.176+0000 7fd6f372b700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6ec067430 con 0x7fd6ec061b90 2020-11-20T11:43:46.183 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.176+0000 7fd6f0cc6700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7fd6ec0a97d0 msgr2=0x7fd6ec13b210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:43:46.183 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.177+0000 7fd6f0cc6700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd6ec0a97d0 0x7fd6ec13b210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:43:46.184 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.177+0000 7fd6f14c7700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd6ec061b90 0x7fd6ec0a92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:46.184 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.177+0000 7fd6f14c7700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd6ec061b90 0x7fd6ec0a92e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:38258/0 (socket says 172.21.15.23:38258) 2020-11-20T11:43:46.184 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.177+0000 7fd6f14c7700 1 -- 172.21.15.23:0/2675044768 learned_addr learned my addr 172.21.15.23:0/2675044768 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:43:46.185 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.177+0000 7fd6f1cc8700 1 -- 172.21.15.23:0/2675044768 >> v1:172.21.15.23:6789/0 conn(0x7fd6ec13b700 legacy=0x7fd6ec137e90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2020-11-20T11:43:46.187 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.184+0000 7fd6f14c7700 1 -- 172.21.15.23:0/2675044768 >> v2:172.21.15.23:3300/0 conn(0x7fd6ec0a97d0 msgr2=0x7fd6ec13b210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:43:46.187 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.184+0000 7fd6f14c7700 1 --2- 172.21.15.23:0/2675044768 >> v2:172.21.15.23:3300/0 conn(0x7fd6ec0a97d0 0x7fd6ec13b210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:46.188 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.184+0000 7fd6f14c7700 1 -- 172.21.15.23:0/2675044768 >> v1:172.21.15.23:6789/0 conn(0x7fd6ec13b700 legacy=0x7fd6ec137e90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:43:46.188 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.184+0000 7fd6f14c7700 1 -- 172.21.15.23:0/2675044768 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6ec067860 con 0x7fd6ec061b90 2020-11-20T11:43:46.188 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.184+0000 7fd6f14c7700 1 --2- 172.21.15.23:0/2675044768 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6ec061b90 0x7fd6ec0a92e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fd6ec069cb0 tx=0x7fd6e400b570).ready entity=mon.1 client_cookie=8f5be46d76766ebf server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:46.189 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.184+0000 7fd6dbfff700 1 -- 172.21.15.23:0/2675044768 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e400de10 con 0x7fd6ec061b90 2020-11-20T11:43:46.189 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.184+0000 7fd6dbfff700 1 -- 172.21.15.23:0/2675044768 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6e4010680 con 0x7fd6ec061b90 2020-11-20T11:43:46.190 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.184+0000 7fd6dbfff700 1 -- 172.21.15.23:0/2675044768 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e4010a00 con 0x7fd6ec061b90 2020-11-20T11:43:46.190 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.185+0000 7fd6f372b700 1 -- 172.21.15.23:0/2675044768 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6ec061b90 msgr2=0x7fd6ec0a92e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:46.191 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.185+0000 7fd6f372b700 1 --2- 172.21.15.23:0/2675044768 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6ec061b90 0x7fd6ec0a92e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fd6ec069cb0 tx=0x7fd6e400b570).stop 2020-11-20T11:43:46.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.189+0000 7fd6f372b700 1 -- 172.21.15.23:0/2675044768 shutdown_connections 2020-11-20T11:43:46.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.189+0000 7fd6f372b700 1 --2- 172.21.15.23:0/2675044768 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6ec061b90 0x7fd6ec0a92e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:46.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.189+0000 7fd6f372b700 1 --2- 172.21.15.23:0/2675044768 >> v2:172.21.15.23:3300/0 conn(0x7fd6ec0a97d0 0x7fd6ec13b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:46.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.189+0000 7fd6f372b700 1 -- 172.21.15.23:0/2675044768 >> 172.21.15.23:0/2675044768 conn(0x7fd6ec063640 msgr2=0x7fd6ec0612c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:46.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.189+0000 7fd6f372b700 1 -- 172.21.15.23:0/2675044768 shutdown_connections 2020-11-20T11:43:46.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.190+0000 7fd6f372b700 1 -- 172.21.15.23:0/2675044768 wait complete. 2020-11-20T11:43:46.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.191+0000 7fd6f372b700 1 Processor -- start 2020-11-20T11:43:46.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.193+0000 7fd6f372b700 1 -- start start 2020-11-20T11:43:46.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.195+0000 7fd6f372b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6ec061b90 0x7fd6ec134a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:46.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.195+0000 7fd6f372b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6ec0a97d0 0x7fd6ec0a6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:46.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.195+0000 7fd6f372b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6ec13b700 0x7fd6ec0a2990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:46.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.195+0000 7fd6f372b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ec0a5370 con 0x7fd6ec0a97d0 2020-11-20T11:43:46.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.195+0000 7fd6f372b700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ec0a5610 con 0x7fd6ec061b90 2020-11-20T11:43:46.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.195+0000 7fd6f372b700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ec0a58b0 con 0x7fd6ec13b700 2020-11-20T11:43:46.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.195+0000 7fd6f0cc6700 1 -- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6ec0a97d0 msgr2=0x7fd6ec0a6130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:43:46.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.195+0000 7fd6f0cc6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6ec0a97d0 0x7fd6ec0a6130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:43:46.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.195+0000 7fd6f14c7700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6ec061b90 0x7fd6ec134a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:46.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.195+0000 7fd6f14c7700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6ec061b90 0x7fd6ec134a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:38266/0 (socket says 172.21.15.23:38266) 2020-11-20T11:43:46.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.195+0000 7fd6f14c7700 1 -- 172.21.15.23:0/498867753 learned_addr learned my addr 172.21.15.23:0/498867753 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:43:46.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.195+0000 7fd6f1cc8700 1 --2- 172.21.15.23:0/498867753 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6ec13b700 0x7fd6ec0a2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:46.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.195+0000 7fd6f14c7700 1 -- 172.21.15.23:0/498867753 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6ec13b700 msgr2=0x7fd6ec0a2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:46.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.195+0000 7fd6f14c7700 1 --2- 172.21.15.23:0/498867753 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6ec13b700 0x7fd6ec0a2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:46.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.195+0000 7fd6f14c7700 1 -- 172.21.15.23:0/498867753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6ec0a97d0 msgr2=0x7fd6ec0a6130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:43:46.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.195+0000 7fd6f14c7700 1 --2- 172.21.15.23:0/498867753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6ec0a97d0 0x7fd6ec0a6130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:46.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.195+0000 7fd6f14c7700 1 -- 172.21.15.23:0/498867753 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6ec0ad320 con 0x7fd6ec061b90 2020-11-20T11:43:46.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.196+0000 7fd6f14c7700 1 --2- 172.21.15.23:0/498867753 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6ec061b90 0x7fd6ec134a00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fd6e400b2c0 tx=0x7fd6ec0a83a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:46.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.196+0000 7fd6d97fa700 1 -- 172.21.15.23:0/498867753 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e4019b50 con 0x7fd6ec061b90 2020-11-20T11:43:46.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.196+0000 7fd6d97fa700 1 -- 172.21.15.23:0/498867753 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6e4019cb0 con 0x7fd6ec061b90 2020-11-20T11:43:46.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.196+0000 7fd6f372b700 1 -- 172.21.15.23:0/498867753 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6ec0a5d50 con 0x7fd6ec061b90 2020-11-20T11:43:46.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.196+0000 7fd6d97fa700 1 -- 172.21.15.23:0/498867753 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e4022650 con 0x7fd6ec061b90 2020-11-20T11:43:46.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.196+0000 7fd6f372b700 1 -- 172.21.15.23:0/498867753 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6ec0a2e80 con 0x7fd6ec061b90 2020-11-20T11:43:46.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.197+0000 7fd6f372b700 1 -- 172.21.15.23:0/498867753 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6ec0a1850 con 0x7fd6ec061b90 2020-11-20T11:43:46.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.197+0000 7fd6d97fa700 1 -- 172.21.15.23:0/498867753 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd6e4028020 con 0x7fd6ec061b90 2020-11-20T11:43:46.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.198+0000 7fd6d97fa700 1 --2- 172.21.15.23:0/498867753 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd6c4056d10 0x7fd6c40591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:46.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.198+0000 7fd6d97fa700 1 -- 172.21.15.23:0/498867753 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fd6e40720a0 con 0x7fd6ec061b90 2020-11-20T11:43:46.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.198+0000 7fd6f0cc6700 1 --2- 172.21.15.23:0/498867753 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd6c4056d10 0x7fd6c40591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:46.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.198+0000 7fd6f0cc6700 1 --2- 172.21.15.23:0/498867753 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd6c4056d10 0x7fd6c40591d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fd6e8006fd0 tx=0x7fd6e8006b20).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:46.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.203+0000 7fd6d97fa700 1 -- 172.21.15.23:0/498867753 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd6e4047390 con 0x7fd6ec061b90 2020-11-20T11:43:46.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.671+0000 7fd6f372b700 1 -- 172.21.15.23:0/498867753 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd6ec044710 con 0x7fd6ec061b90 2020-11-20T11:43:46.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.672+0000 7fd6d97fa700 1 -- 172.21.15.23:0/498867753 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1479 (secure 0 0 0) 0x7fd6e404a9b0 con 0x7fd6ec061b90 2020-11-20T11:43:46.677 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":102,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:43:46.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.679+0000 7fd6f372b700 1 -- 172.21.15.23:0/498867753 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd6c4056d10 msgr2=0x7fd6c40591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:46.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.679+0000 7fd6f372b700 1 --2- 172.21.15.23:0/498867753 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd6c4056d10 0x7fd6c40591d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fd6e8006fd0 tx=0x7fd6e8006b20).stop 2020-11-20T11:43:46.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.679+0000 7fd6f372b700 1 -- 172.21.15.23:0/498867753 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6ec061b90 msgr2=0x7fd6ec134a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:46.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.679+0000 7fd6f372b700 1 --2- 172.21.15.23:0/498867753 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6ec061b90 0x7fd6ec134a00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fd6e400b2c0 tx=0x7fd6ec0a83a0).stop 2020-11-20T11:43:46.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.679+0000 7fd6f372b700 1 -- 172.21.15.23:0/498867753 shutdown_connections 2020-11-20T11:43:46.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.679+0000 7fd6f372b700 1 --2- 172.21.15.23:0/498867753 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd6c4056d10 0x7fd6c40591d0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:46.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.679+0000 7fd6f372b700 1 --2- 172.21.15.23:0/498867753 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6ec13b700 0x7fd6ec0a2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:46.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.679+0000 7fd6f372b700 1 --2- 172.21.15.23:0/498867753 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6ec061b90 0x7fd6ec134a00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:46.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.679+0000 7fd6f372b700 1 --2- 172.21.15.23:0/498867753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6ec0a97d0 0x7fd6ec0a6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:46.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.679+0000 7fd6f372b700 1 -- 172.21.15.23:0/498867753 >> 172.21.15.23:0/498867753 conn(0x7fd6ec063640 msgr2=0x7fd6ec0aae70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:46.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.680+0000 7fd6f372b700 1 -- 172.21.15.23:0/498867753 shutdown_connections 2020-11-20T11:43:46.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:46.680+0000 7fd6f372b700 1 -- 172.21.15.23:0/498867753 wait complete. 2020-11-20T11:43:49.745 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:43:50.280 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.275+0000 7f28d863f700 1 Processor -- start 2020-11-20T11:43:50.280 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.276+0000 7f28d863f700 1 -- start start 2020-11-20T11:43:50.280 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.277+0000 7f28d863f700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28d0061a10 0x7f28d00a6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:50.281 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.277+0000 7f28d863f700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d00a4970 con 0x7f28d0136ea0 2020-11-20T11:43:50.281 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.277+0000 7f28d863f700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d00a60e0 con 0x7f28d013f800 2020-11-20T11:43:50.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.277+0000 7f28d863f700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28d00a5ef0 con 0x7f28d0061a10 2020-11-20T11:43:50.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.277+0000 7f28d5bda700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f28d0136ea0 legacy=0x7f28d00a8ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2020-11-20T11:43:50.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.277+0000 7f28d63db700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28d0061a10 0x7f28d00a6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:50.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.277+0000 7f28d6bdc700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f28d013f800 0x7f28d0141bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:37600/0 (socket says 172.21.15.23:37600) 2020-11-20T11:43:50.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.277+0000 7f28d6bdc700 1 -- 172.21.15.23:0/1131083509 learned_addr learned my addr 172.21.15.23:0/1131083509 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:43:50.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.278+0000 7f28d63db700 1 -- 172.21.15.23:0/1131083509 >> v1:172.21.15.155:6789/0 conn(0x7f28d013f800 legacy=0x7f28d0141bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:50.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.278+0000 7f28d63db700 1 -- 172.21.15.23:0/1131083509 >> v1:172.21.15.23:6789/0 conn(0x7f28d0136ea0 legacy=0x7f28d00a8ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:43:50.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.278+0000 7f28d63db700 1 -- 172.21.15.23:0/1131083509 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d0061df0 con 0x7f28d0061a10 2020-11-20T11:43:50.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.278+0000 7f28d63db700 1 --2- 172.21.15.23:0/1131083509 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28d0061a10 0x7f28d00a6580 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f28d0061fe0 tx=0x7f28c8009220).ready entity=mon.2 client_cookie=3afd48712949c51e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:50.287 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.278+0000 7f28d53d9700 1 -- 172.21.15.23:0/1131083509 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c800bbb0 con 0x7f28d0061a10 2020-11-20T11:43:50.288 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.278+0000 7f28d53d9700 1 -- 172.21.15.23:0/1131083509 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28c800bd10 con 0x7f28d0061a10 2020-11-20T11:43:50.288 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.278+0000 7f28d53d9700 1 -- 172.21.15.23:0/1131083509 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c8011610 con 0x7f28d0061a10 2020-11-20T11:43:50.288 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.279+0000 7f28d863f700 1 -- 172.21.15.23:0/1131083509 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28d0061a10 msgr2=0x7f28d00a6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:50.288 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.279+0000 7f28d863f700 1 --2- 172.21.15.23:0/1131083509 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28d0061a10 0x7f28d00a6580 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f28d0061fe0 tx=0x7f28c8009220).stop 2020-11-20T11:43:50.288 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.280+0000 7f28d863f700 1 -- 172.21.15.23:0/1131083509 shutdown_connections 2020-11-20T11:43:50.289 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.280+0000 7f28d863f700 1 --2- 172.21.15.23:0/1131083509 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28d0061a10 0x7f28d00a6580 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:50.289 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.280+0000 7f28d863f700 1 -- 172.21.15.23:0/1131083509 >> 172.21.15.23:0/1131083509 conn(0x7f28d0060c80 msgr2=0x7f28d00acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:50.290 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.281+0000 7f28d863f700 1 -- 172.21.15.23:0/1131083509 shutdown_connections 2020-11-20T11:43:50.290 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.284+0000 7f28d863f700 1 -- 172.21.15.23:0/1131083509 wait complete. 2020-11-20T11:43:50.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.285+0000 7f28d863f700 1 Processor -- start 2020-11-20T11:43:50.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.285+0000 7f28d863f700 1 -- start start 2020-11-20T11:43:50.298 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.286+0000 7f28d863f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28d0061a10 0x7f28d009dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:50.298 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.286+0000 7f28d863f700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28d0136ea0 0x7f28d0065f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:50.298 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.286+0000 7f28d863f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28d013f800 0x7f28d00a2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:50.299 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.286+0000 7f28d863f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d0064f80 con 0x7f28d013f800 2020-11-20T11:43:50.299 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.286+0000 7f28d863f700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d00672d0 con 0x7f28d0061a10 2020-11-20T11:43:50.299 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.286+0000 7f28d863f700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28d00668f0 con 0x7f28d0136ea0 2020-11-20T11:43:50.299 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.286+0000 7f28d6bdc700 1 -- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28d013f800 msgr2=0x7f28d00a2970 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:43:50.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.286+0000 7f28d6bdc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28d013f800 0x7f28d00a2970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:43:50.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.286+0000 7f28d5bda700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28d0136ea0 0x7f28d0065f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:50.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.286+0000 7f28d5bda700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28d0136ea0 0x7f28d0065f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:55624/0 (socket says 172.21.15.23:55624) 2020-11-20T11:43:50.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.286+0000 7f28d5bda700 1 -- 172.21.15.23:0/3632212071 learned_addr learned my addr 172.21.15.23:0/3632212071 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:43:50.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.287+0000 7f28d5bda700 1 -- 172.21.15.23:0/3632212071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28d0061a10 msgr2=0x7f28d009dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:50.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.287+0000 7f28d5bda700 1 --2- 172.21.15.23:0/3632212071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28d0061a10 0x7f28d009dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:50.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.287+0000 7f28d5bda700 1 -- 172.21.15.23:0/3632212071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28d013f800 msgr2=0x7f28d00a2970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:43:50.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.287+0000 7f28d5bda700 1 --2- 172.21.15.23:0/3632212071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28d013f800 0x7f28d00a2970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:50.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.287+0000 7f28d5bda700 1 -- 172.21.15.23:0/3632212071 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d0066b90 con 0x7f28d0136ea0 2020-11-20T11:43:50.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.287+0000 7f28d5bda700 1 --2- 172.21.15.23:0/3632212071 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28d0136ea0 0x7f28d0065f30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f28c0009790 tx=0x7f28c0006e70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:50.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.287+0000 7f28be7fc700 1 -- 172.21.15.23:0/3632212071 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c0006170 con 0x7f28d0136ea0 2020-11-20T11:43:50.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.287+0000 7f28be7fc700 1 -- 172.21.15.23:0/3632212071 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28c00062f0 con 0x7f28d0136ea0 2020-11-20T11:43:50.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.287+0000 7f28d863f700 1 -- 172.21.15.23:0/3632212071 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28d0066e30 con 0x7f28d0136ea0 2020-11-20T11:43:50.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.288+0000 7f28be7fc700 1 -- 172.21.15.23:0/3632212071 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c00038f0 con 0x7f28d0136ea0 2020-11-20T11:43:50.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.288+0000 7f28d863f700 1 -- 172.21.15.23:0/3632212071 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28d012b570 con 0x7f28d0136ea0 2020-11-20T11:43:50.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.289+0000 7f28be7fc700 1 -- 172.21.15.23:0/3632212071 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f28c0017030 con 0x7f28d0136ea0 2020-11-20T11:43:50.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.289+0000 7f28be7fc700 1 --2- 172.21.15.23:0/3632212071 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f28a8056d10 0x7f28a80591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:50.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.289+0000 7f28be7fc700 1 -- 172.21.15.23:0/3632212071 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f28c006f250 con 0x7f28d0136ea0 2020-11-20T11:43:50.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.289+0000 7f28d863f700 1 -- 172.21.15.23:0/3632212071 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28d0137600 con 0x7f28d0136ea0 2020-11-20T11:43:50.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.289+0000 7f28d63db700 1 --2- 172.21.15.23:0/3632212071 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f28a8056d10 0x7f28a80591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:50.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.290+0000 7f28d63db700 1 --2- 172.21.15.23:0/3632212071 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f28a8056d10 0x7f28a80591d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f28c8009260 tx=0x7f28c8016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:50.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.300+0000 7f28be7fc700 1 -- 172.21.15.23:0/3632212071 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f28c0048020 con 0x7f28d0136ea0 2020-11-20T11:43:50.796 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.793+0000 7f28d863f700 1 -- 172.21.15.23:0/3632212071 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f28d0061650 con 0x7f28d0136ea0 2020-11-20T11:43:50.796 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.793+0000 7f28be7fc700 1 -- 172.21.15.23:0/3632212071 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1479 (secure 0 0 0) 0x7f28c0047c10 con 0x7f28d0136ea0 2020-11-20T11:43:50.797 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":102,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:43:50.798 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.795+0000 7f28d863f700 1 -- 172.21.15.23:0/3632212071 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f28a8056d10 msgr2=0x7f28a80591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:50.799 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.795+0000 7f28d863f700 1 --2- 172.21.15.23:0/3632212071 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f28a8056d10 0x7f28a80591d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f28c8009260 tx=0x7f28c8016000).stop 2020-11-20T11:43:50.799 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.795+0000 7f28d863f700 1 -- 172.21.15.23:0/3632212071 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28d0136ea0 msgr2=0x7f28d0065f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:50.799 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.795+0000 7f28d863f700 1 --2- 172.21.15.23:0/3632212071 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28d0136ea0 0x7f28d0065f30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f28c0009790 tx=0x7f28c0006e70).stop 2020-11-20T11:43:50.799 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.795+0000 7f28d863f700 1 -- 172.21.15.23:0/3632212071 shutdown_connections 2020-11-20T11:43:50.800 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.795+0000 7f28d863f700 1 --2- 172.21.15.23:0/3632212071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28d013f800 0x7f28d00a2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:50.800 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.795+0000 7f28d863f700 1 --2- 172.21.15.23:0/3632212071 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28d0061a10 0x7f28d009dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:50.800 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.795+0000 7f28d863f700 1 --2- 172.21.15.23:0/3632212071 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f28a8056d10 0x7f28a80591d0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:50.800 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.795+0000 7f28d863f700 1 --2- 172.21.15.23:0/3632212071 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28d0136ea0 0x7f28d0065f30 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:50.800 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.795+0000 7f28d863f700 1 -- 172.21.15.23:0/3632212071 >> 172.21.15.23:0/3632212071 conn(0x7f28d0060c80 msgr2=0x7f28d012ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:50.801 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.795+0000 7f28d863f700 1 -- 172.21.15.23:0/3632212071 shutdown_connections 2020-11-20T11:43:50.801 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:50.795+0000 7f28d863f700 1 -- 172.21.15.23:0/3632212071 wait complete. 2020-11-20T11:43:53.811 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:43:55.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:55.483+0000 7f5bf10dc700 1 Processor -- start 2020-11-20T11:43:55.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:55.483+0000 7f5bf10dc700 1 -- start start 2020-11-20T11:43:55.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:55.484+0000 7f5bf10dc700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec061a10 0x7f5bec0a6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:55.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:55.484+0000 7f5bf10dc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5bec0ad600 0x7f5bec136ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:55.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:55.484+0000 7f5bf10dc700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bec0a4970 con 0x7f5bec0a8ad0 2020-11-20T11:43:55.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:55.484+0000 7f5bf10dc700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bec0a5ef0 con 0x7f5bec0ad600 2020-11-20T11:43:55.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:55.484+0000 7f5bf10dc700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bec0a5490 con 0x7f5bec061a10 2020-11-20T11:43:55.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:55.484+0000 7f5bea59c700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f5bec0ad600 msgr2=0x7f5bec136ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:43:55.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:55.484+0000 7f5bea59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5bec0ad600 0x7f5bec136ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:43:55.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:55.484+0000 7f5beb59e700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f5bec0a8ad0 0x7f5bec13a470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:37616/0 (socket says 172.21.15.23:37616) 2020-11-20T11:43:55.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:55.484+0000 7f5beb59e700 1 -- 172.21.15.23:0/2521257698 learned_addr learned my addr 172.21.15.23:0/2521257698 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:43:55.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:55.484+0000 7f5bead9d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec061a10 0x7f5bec0a6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:55.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:55.486+0000 7f5bead9d700 1 -- 172.21.15.23:0/2521257698 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec061a10 msgr2=0x7f5bec0a6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:43:55.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:55.486+0000 7f5bead9d700 1 -- 172.21.15.23:0/2521257698 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec061a10 msgr2=0x7f5bec0a6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:43:55.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:55.486+0000 7f5bead9d700 1 --2- 172.21.15.23:0/2521257698 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec061a10 0x7f5bec0a6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:43:55.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:55.486+0000 7f5bead9d700 1 --2- 172.21.15.23:0/2521257698 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec061a10 0x7f5bec0a6580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:43:55.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:55.685+0000 7f5bea59c700 1 -- 172.21.15.23:0/2521257698 >> v2:172.21.15.23:3300/0 conn(0x7f5bec0ad600 msgr2=0x7f5bec136ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:43:55.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:55.685+0000 7f5bea59c700 1 --2- 172.21.15.23:0/2521257698 >> v2:172.21.15.23:3300/0 conn(0x7f5bec0ad600 0x7f5bec136ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:43:55.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:55.687+0000 7f5bead9d700 1 --2- 172.21.15.23:0/2521257698 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec061a10 0x7f5bec0a6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:55.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:55.687+0000 7f5bead9d700 1 -- 172.21.15.23:0/2521257698 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec061a10 msgr2=0x7f5bec0a6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:43:55.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:55.687+0000 7f5bead9d700 1 -- 172.21.15.23:0/2521257698 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec061a10 msgr2=0x7f5bec0a6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:43:55.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:55.687+0000 7f5bead9d700 1 --2- 172.21.15.23:0/2521257698 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec061a10 0x7f5bec0a6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:43:55.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:55.687+0000 7f5bead9d700 1 --2- 172.21.15.23:0/2521257698 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec061a10 0x7f5bec0a6580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:43:56.091 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:56.088+0000 7f5bea59c700 1 -- 172.21.15.23:0/2521257698 >> v2:172.21.15.23:3300/0 conn(0x7f5bec0ad600 msgr2=0x7f5bec136ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:43:56.091 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:56.088+0000 7f5bea59c700 1 --2- 172.21.15.23:0/2521257698 >> v2:172.21.15.23:3300/0 conn(0x7f5bec0ad600 0x7f5bec136ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:43:56.092 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:56.089+0000 7f5bead9d700 1 --2- 172.21.15.23:0/2521257698 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec061a10 0x7f5bec0a6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:56.092 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:56.089+0000 7f5bead9d700 1 -- 172.21.15.23:0/2521257698 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec061a10 msgr2=0x7f5bec0a6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:43:56.092 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:56.089+0000 7f5bead9d700 1 -- 172.21.15.23:0/2521257698 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec061a10 msgr2=0x7f5bec0a6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:43:56.093 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:56.089+0000 7f5bead9d700 1 --2- 172.21.15.23:0/2521257698 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec061a10 0x7f5bec0a6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:43:56.093 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:56.089+0000 7f5bead9d700 1 --2- 172.21.15.23:0/2521257698 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec061a10 0x7f5bec0a6580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:43:56.892 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:56.889+0000 7f5bea59c700 1 -- 172.21.15.23:0/2521257698 >> v2:172.21.15.23:3300/0 conn(0x7f5bec0ad600 msgr2=0x7f5bec136ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:43:56.892 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:56.889+0000 7f5bea59c700 1 --2- 172.21.15.23:0/2521257698 >> v2:172.21.15.23:3300/0 conn(0x7f5bec0ad600 0x7f5bec136ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:43:56.893 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:56.890+0000 7f5bead9d700 1 --2- 172.21.15.23:0/2521257698 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec061a10 0x7f5bec0a6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:56.893 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:56.890+0000 7f5bead9d700 1 -- 172.21.15.23:0/2521257698 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec061a10 msgr2=0x7f5bec0a6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:43:56.893 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:56.890+0000 7f5bead9d700 1 -- 172.21.15.23:0/2521257698 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec061a10 msgr2=0x7f5bec0a6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:43:56.894 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:56.890+0000 7f5bead9d700 1 --2- 172.21.15.23:0/2521257698 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec061a10 0x7f5bec0a6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:43:56.894 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:56.890+0000 7f5bead9d700 1 --2- 172.21.15.23:0/2521257698 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec061a10 0x7f5bec0a6580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:43:57.402 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.398+0000 7f5be9d9b700 1 -- 172.21.15.23:0/2521257698 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2579864325 0 0) 0x7f5bec0a4970 con 0x7f5bec0a8ad0 2020-11-20T11:43:57.402 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.399+0000 7f5be9d9b700 1 -- 172.21.15.23:0/2521257698 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bd8003280 con 0x7f5bec0a8ad0 2020-11-20T11:43:57.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.401+0000 7f5be9d9b700 1 -- 172.21.15.23:0/2521257698 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 603421459 0 0) 0x7f5bd8003280 con 0x7f5bec0a8ad0 2020-11-20T11:43:57.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.401+0000 7f5be9d9b700 1 -- 172.21.15.23:0/2521257698 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec061a10 msgr2=0x7f5bec0a6580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:43:57.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.401+0000 7f5be9d9b700 1 --2- 172.21.15.23:0/2521257698 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec061a10 0x7f5bec0a6580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:57.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.401+0000 7f5be9d9b700 1 -- 172.21.15.23:0/2521257698 >> v2:172.21.15.23:3300/0 conn(0x7f5bec0ad600 msgr2=0x7f5bec136ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:43:57.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.401+0000 7f5be9d9b700 1 --2- 172.21.15.23:0/2521257698 >> v2:172.21.15.23:3300/0 conn(0x7f5bec0ad600 0x7f5bec136ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:57.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.401+0000 7f5be9d9b700 1 -- 172.21.15.23:0/2521257698 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bec0a60e0 con 0x7f5bec0a8ad0 2020-11-20T11:43:57.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.401+0000 7f5be9d9b700 1 -- 172.21.15.23:0/2521257698 <== mon.1 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f5be4001880 con 0x7f5bec0a8ad0 2020-11-20T11:43:57.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.401+0000 7f5be9d9b700 1 -- 172.21.15.23:0/2521257698 >> v1:172.21.15.155:6789/0 conn(0x7f5bec0a8ad0 legacy=0x7f5bec13a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:57.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.401+0000 7f5be9d9b700 1 --2- 172.21.15.23:0/2521257698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bd8003770 0x7f5bd8003b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:57.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.401+0000 7f5be9d9b700 1 --2- 172.21.15.23:0/2521257698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bd80040a0 0x7f5bd8004580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:57.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.401+0000 7f5be9d9b700 1 --2- 172.21.15.23:0/2521257698 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec061a10 0x7f5bd8004be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:57.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.401+0000 7f5be9d9b700 1 -- 172.21.15.23:0/2521257698 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bec0a5490 con 0x7f5bd80040a0 2020-11-20T11:43:57.407 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.401+0000 7f5be9d9b700 1 -- 172.21.15.23:0/2521257698 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd8005120 con 0x7f5bd8003770 2020-11-20T11:43:57.407 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.401+0000 7f5be9d9b700 1 -- 172.21.15.23:0/2521257698 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd80052e0 con 0x7f5bec061a10 2020-11-20T11:43:57.407 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.401+0000 7f5bea59c700 1 -- 172.21.15.23:0/2521257698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bd80040a0 msgr2=0x7f5bd8004580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:43:57.407 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.401+0000 7f5bea59c700 1 --2- 172.21.15.23:0/2521257698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bd80040a0 0x7f5bd8004580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:43:57.407 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.401+0000 7f5beb59e700 1 --2- 172.21.15.23:0/2521257698 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec061a10 0x7f5bd8004be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:57.408 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.401+0000 7f5bead9d700 1 --2- 172.21.15.23:0/2521257698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bd8003770 0x7f5bd8003b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:57.411 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.401+0000 7f5bead9d700 1 -- 172.21.15.23:0/2521257698 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec061a10 msgr2=0x7f5bd8004be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:43:57.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.401+0000 7f5bead9d700 1 --2- 172.21.15.23:0/2521257698 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec061a10 0x7f5bd8004be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:57.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.401+0000 7f5bead9d700 1 -- 172.21.15.23:0/2521257698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bd80040a0 msgr2=0x7f5bd8004580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:43:57.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.401+0000 7f5bead9d700 1 --2- 172.21.15.23:0/2521257698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bd80040a0 0x7f5bd8004580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:57.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.401+0000 7f5bead9d700 1 -- 172.21.15.23:0/2521257698 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd8003280 con 0x7f5bd8003770 2020-11-20T11:43:57.413 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.401+0000 7f5beb59e700 1 --2- 172.21.15.23:0/2521257698 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec061a10 0x7f5bd8004be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:43:57.413 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.402+0000 7f5bead9d700 1 --2- 172.21.15.23:0/2521257698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bd8003770 0x7f5bd8003b90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f5be0011a20 tx=0x7f5be0008040).ready entity=mon.1 client_cookie=6f99e854846220a5 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:57.413 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.403+0000 7f5be9d9b700 1 -- 172.21.15.23:0/2521257698 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be00139e0 con 0x7f5bd8003770 2020-11-20T11:43:57.414 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.403+0000 7f5be9d9b700 1 -- 172.21.15.23:0/2521257698 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5be0013b40 con 0x7f5bd8003770 2020-11-20T11:43:57.414 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.403+0000 7f5be9d9b700 1 -- 172.21.15.23:0/2521257698 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be0013ec0 con 0x7f5bd8003770 2020-11-20T11:43:57.414 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.404+0000 7f5bf10dc700 1 -- 172.21.15.23:0/2521257698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bd8003770 msgr2=0x7f5bd8003b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:57.414 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.404+0000 7f5bf10dc700 1 --2- 172.21.15.23:0/2521257698 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bd8003770 0x7f5bd8003b90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f5be0011a20 tx=0x7f5be0008040).stop 2020-11-20T11:43:57.414 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.404+0000 7f5beb59e700 1 -- 172.21.15.23:0/2521257698 reap_dead start 2020-11-20T11:43:57.415 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.404+0000 7f5bf10dc700 1 -- 172.21.15.23:0/2521257698 shutdown_connections 2020-11-20T11:43:57.415 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.404+0000 7f5bf10dc700 1 -- 172.21.15.23:0/2521257698 >> 172.21.15.23:0/2521257698 conn(0x7f5bec060c80 msgr2=0x7f5bec0acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:57.415 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.404+0000 7f5bf10dc700 1 -- 172.21.15.23:0/2521257698 shutdown_connections 2020-11-20T11:43:57.415 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.404+0000 7f5bf10dc700 1 -- 172.21.15.23:0/2521257698 wait complete. 2020-11-20T11:43:57.415 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.405+0000 7f5bf10dc700 1 Processor -- start 2020-11-20T11:43:57.416 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.405+0000 7f5bf10dc700 1 -- start start 2020-11-20T11:43:57.416 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.406+0000 7f5bf10dc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bd8003770 0x7f5bec139020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:57.416 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.406+0000 7f5bf10dc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bec0a2400 0x7f5bec0a2880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:57.416 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.406+0000 7f5bf10dc700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec1334e0 0x7f5bec133940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:57.416 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.406+0000 7f5bf10dc700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bec133080 con 0x7f5bd8003770 2020-11-20T11:43:57.417 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.406+0000 7f5bf10dc700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bec0a15b0 con 0x7f5bec0a2400 2020-11-20T11:43:57.417 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.406+0000 7f5bf10dc700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bec0a2d70 con 0x7f5bec1334e0 2020-11-20T11:43:57.417 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.406+0000 7f5bead9d700 1 -- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bd8003770 msgr2=0x7f5bec139020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:43:57.417 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.406+0000 7f5bead9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bd8003770 0x7f5bec139020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:43:57.417 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.406+0000 7f5bea59c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bec0a2400 0x7f5bec0a2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:57.418 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.406+0000 7f5beb59e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec1334e0 0x7f5bec133940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:57.418 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.406+0000 7f5beb59e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec1334e0 0x7f5bec133940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:55658/0 (socket says 172.21.15.23:55658) 2020-11-20T11:43:57.418 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.406+0000 7f5beb59e700 1 -- 172.21.15.23:0/1932559080 learned_addr learned my addr 172.21.15.23:0/1932559080 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:43:57.418 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.406+0000 7f5bea59c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bec0a2400 0x7f5bec0a2880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:38312/0 (socket says 172.21.15.23:38312) 2020-11-20T11:43:57.418 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.406+0000 7f5beb59e700 1 -- 172.21.15.23:0/1932559080 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bec0a2400 msgr2=0x7f5bec0a2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:57.419 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.406+0000 7f5beb59e700 1 --2- 172.21.15.23:0/1932559080 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bec0a2400 0x7f5bec0a2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:57.419 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.406+0000 7f5beb59e700 1 -- 172.21.15.23:0/1932559080 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bd8003770 msgr2=0x7f5bec139020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:43:57.419 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.406+0000 7f5beb59e700 1 --2- 172.21.15.23:0/1932559080 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bd8003770 0x7f5bec139020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:57.419 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.406+0000 7f5beb59e700 1 -- 172.21.15.23:0/1932559080 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bec0a1800 con 0x7f5bec1334e0 2020-11-20T11:43:57.419 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.406+0000 7f5beb59e700 1 --2- 172.21.15.23:0/1932559080 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec1334e0 0x7f5bec133940 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f5bec0ada20 tx=0x7f5bec0a6270).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:57.420 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.406+0000 7f5bd2ffd700 1 -- 172.21.15.23:0/1932559080 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be400fcc0 con 0x7f5bec1334e0 2020-11-20T11:43:57.420 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.407+0000 7f5bd2ffd700 1 -- 172.21.15.23:0/1932559080 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5be400fe20 con 0x7f5bec1334e0 2020-11-20T11:43:57.420 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.407+0000 7f5bf10dc700 1 -- 172.21.15.23:0/1932559080 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bec0a1aa0 con 0x7f5bec1334e0 2020-11-20T11:43:57.420 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.407+0000 7f5bd2ffd700 1 -- 172.21.15.23:0/1932559080 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be40125a0 con 0x7f5bec1334e0 2020-11-20T11:43:57.421 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.407+0000 7f5bf10dc700 1 -- 172.21.15.23:0/1932559080 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bec0a1270 con 0x7f5bec1334e0 2020-11-20T11:43:57.421 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.407+0000 7f5bf10dc700 1 -- 172.21.15.23:0/1932559080 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bec061650 con 0x7f5bec1334e0 2020-11-20T11:43:57.421 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.408+0000 7f5bd2ffd700 1 -- 172.21.15.23:0/1932559080 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5be4001590 con 0x7f5bec1334e0 2020-11-20T11:43:57.421 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.408+0000 7f5bd2ffd700 1 --2- 172.21.15.23:0/1932559080 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5bc8056d10 0x7f5bc80591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:57.421 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.408+0000 7f5bd2ffd700 1 -- 172.21.15.23:0/1932559080 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f5be406fb90 con 0x7f5bec1334e0 2020-11-20T11:43:57.422 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.408+0000 7f5bead9d700 1 --2- 172.21.15.23:0/1932559080 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5bc8056d10 0x7f5bc80591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:57.422 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.408+0000 7f5bead9d700 1 --2- 172.21.15.23:0/1932559080 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5bc8056d10 0x7f5bc80591d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f5bec138b90 tx=0x7f5be0008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:57.422 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.412+0000 7f5bd2ffd700 1 -- 172.21.15.23:0/1932559080 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5be4044e80 con 0x7f5bec1334e0 2020-11-20T11:43:57.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.683+0000 7f5bf10dc700 1 -- 172.21.15.23:0/1932559080 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5bec05c520 con 0x7f5bec1334e0 2020-11-20T11:43:57.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.684+0000 7f5bd2ffd700 1 -- 172.21.15.23:0/1932559080 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1472 (secure 0 0 0) 0x7f5be4019030 con 0x7f5bec1334e0 2020-11-20T11:43:57.687 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":104,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:43:57.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.685+0000 7f5bf10dc700 1 -- 172.21.15.23:0/1932559080 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5bc8056d10 msgr2=0x7f5bc80591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:57.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.685+0000 7f5bf10dc700 1 --2- 172.21.15.23:0/1932559080 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5bc8056d10 0x7f5bc80591d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f5bec138b90 tx=0x7f5be0008040).stop 2020-11-20T11:43:57.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.685+0000 7f5bf10dc700 1 -- 172.21.15.23:0/1932559080 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec1334e0 msgr2=0x7f5bec133940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:57.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.685+0000 7f5bf10dc700 1 --2- 172.21.15.23:0/1932559080 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec1334e0 0x7f5bec133940 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f5bec0ada20 tx=0x7f5bec0a6270).stop 2020-11-20T11:43:57.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.685+0000 7f5bf10dc700 1 -- 172.21.15.23:0/1932559080 shutdown_connections 2020-11-20T11:43:57.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.685+0000 7f5bf10dc700 1 --2- 172.21.15.23:0/1932559080 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5bc8056d10 0x7f5bc80591d0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:57.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.685+0000 7f5bf10dc700 1 --2- 172.21.15.23:0/1932559080 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5bec1334e0 0x7f5bec133940 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:57.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.685+0000 7f5bf10dc700 1 --2- 172.21.15.23:0/1932559080 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5bd8003770 0x7f5bec139020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:57.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.685+0000 7f5bf10dc700 1 --2- 172.21.15.23:0/1932559080 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5bec0a2400 0x7f5bec0a2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:57.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.685+0000 7f5bf10dc700 1 -- 172.21.15.23:0/1932559080 >> 172.21.15.23:0/1932559080 conn(0x7f5bec060c80 msgr2=0x7f5bec135350 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:57.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.686+0000 7f5bf10dc700 1 -- 172.21.15.23:0/1932559080 shutdown_connections 2020-11-20T11:43:57.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:57.686+0000 7f5bf10dc700 1 -- 172.21.15.23:0/1932559080 wait complete. 2020-11-20T11:43:57.700 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2020-11-20T11:43:57.700 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:43:58.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.589+0000 7f7b328da700 1 Processor -- start 2020-11-20T11:43:58.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.589+0000 7f7b328da700 1 -- start start 2020-11-20T11:43:58.594 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.589+0000 7f7b328da700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b2c0aae90 0x7f7b2c1381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:58.595 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.589+0000 7f7b328da700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f7b2c13aeb0 0x7f7b2c0a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:58.595 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.589+0000 7f7b328da700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2c13e600 con 0x7f7b2c0aaab0 2020-11-20T11:43:58.595 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.589+0000 7f7b328da700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b2c13e8a0 con 0x7f7b2c13aeb0 2020-11-20T11:43:58.595 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.589+0000 7f7b328da700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2c0ab6d0 con 0x7f7b2c0aae90 2020-11-20T11:43:58.595 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.589+0000 7f7b30e77700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f7b2c13aeb0 msgr2=0x7f7b2c0a8c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:43:58.596 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.589+0000 7f7b30e77700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f7b2c13aeb0 0x7f7b2c0a8c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:43:58.596 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.590+0000 7f7b2b7fe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b2c0aae90 0x7f7b2c1381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:58.597 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.590+0000 7f7b2bfff700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f7b2c0aaab0 0x7f7b2c13ed40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:37646/0 (socket says 172.21.15.23:37646) 2020-11-20T11:43:58.597 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.590+0000 7f7b2bfff700 1 -- 172.21.15.23:0/1756099753 learned_addr learned my addr 172.21.15.23:0/1756099753 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:43:58.597 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.590+0000 7f7b2affd700 1 -- 172.21.15.23:0/1756099753 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4291604140 0 0) 0x7f7b2c13e600 con 0x7f7b2c0aaab0 2020-11-20T11:43:58.597 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.590+0000 7f7b2affd700 1 -- 172.21.15.23:0/1756099753 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b18003280 con 0x7f7b2c0aaab0 2020-11-20T11:43:58.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.590+0000 7f7b2b7fe700 1 -- 172.21.15.23:0/1756099753 >> v2:172.21.15.23:3300/0 conn(0x7f7b2c13aeb0 msgr2=0x7f7b2c0a8c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:43:58.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.590+0000 7f7b2b7fe700 1 --2- 172.21.15.23:0/1756099753 >> v2:172.21.15.23:3300/0 conn(0x7f7b2c13aeb0 0x7f7b2c0a8c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:58.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.590+0000 7f7b2b7fe700 1 -- 172.21.15.23:0/1756099753 >> v1:172.21.15.155:6789/0 conn(0x7f7b2c0aaab0 legacy=0x7f7b2c13ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:58.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.590+0000 7f7b2b7fe700 1 -- 172.21.15.23:0/1756099753 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2c138690 con 0x7f7b2c0aae90 2020-11-20T11:43:58.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.590+0000 7f7b2b7fe700 1 --2- 172.21.15.23:0/1756099753 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b2c0aae90 0x7f7b2c1381a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f7b20006fd0 tx=0x7f7b20006e30).ready entity=mon.2 client_cookie=ebafffce30ca1f70 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:58.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.590+0000 7f7b2affd700 1 -- 172.21.15.23:0/1756099753 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b200097e0 con 0x7f7b2c0aae90 2020-11-20T11:43:58.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.590+0000 7f7b2affd700 1 -- 172.21.15.23:0/1756099753 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b20000e90 con 0x7f7b2c0aae90 2020-11-20T11:43:58.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.590+0000 7f7b2affd700 1 -- 172.21.15.23:0/1756099753 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b20009b50 con 0x7f7b2c0aae90 2020-11-20T11:43:58.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.591+0000 7f7b328da700 1 -- 172.21.15.23:0/1756099753 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b2c0aae90 msgr2=0x7f7b2c1381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:58.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.591+0000 7f7b328da700 1 --2- 172.21.15.23:0/1756099753 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b2c0aae90 0x7f7b2c1381a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f7b20006fd0 tx=0x7f7b20006e30).stop 2020-11-20T11:43:58.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.591+0000 7f7b328da700 1 -- 172.21.15.23:0/1756099753 shutdown_connections 2020-11-20T11:43:58.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.591+0000 7f7b328da700 1 --2- 172.21.15.23:0/1756099753 >> v2:172.21.15.23:3300/0 conn(0x7f7b2c13aeb0 0x7f7b2c0a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:58.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.591+0000 7f7b328da700 1 --2- 172.21.15.23:0/1756099753 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b2c0aae90 0x7f7b2c1381a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:58.601 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.591+0000 7f7b328da700 1 -- 172.21.15.23:0/1756099753 >> 172.21.15.23:0/1756099753 conn(0x7f7b2c0678e0 msgr2=0x7f7b2c067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:58.601 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.591+0000 7f7b328da700 1 -- 172.21.15.23:0/1756099753 shutdown_connections 2020-11-20T11:43:58.601 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.591+0000 7f7b328da700 1 -- 172.21.15.23:0/1756099753 wait complete. 2020-11-20T11:43:58.603 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.592+0000 7f7b328da700 1 Processor -- start 2020-11-20T11:43:58.604 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.592+0000 7f7b328da700 1 -- start start 2020-11-20T11:43:58.604 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.593+0000 7f7b328da700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b2c0aaab0 0x7f7b2c13d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:58.604 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.593+0000 7f7b328da700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b2c0aae90 0x7f7b2c0a77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:58.604 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.593+0000 7f7b328da700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b2c0a5170 0x7f7b2c14d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:58.604 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.593+0000 7f7b328da700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2c14a6c0 con 0x7f7b2c0aae90 2020-11-20T11:43:58.605 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.593+0000 7f7b328da700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2c14a960 con 0x7f7b2c0a5170 2020-11-20T11:43:58.605 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.593+0000 7f7b328da700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2c14ac00 con 0x7f7b2c0aaab0 2020-11-20T11:43:58.605 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.593+0000 7f7b2b7fe700 1 -- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b2c0aae90 msgr2=0x7f7b2c0a77b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:43:58.605 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.593+0000 7f7b2b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b2c0aae90 0x7f7b2c0a77b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:43:58.605 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.593+0000 7f7b2bfff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b2c0aaab0 0x7f7b2c13d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:58.606 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.593+0000 7f7b2bfff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b2c0aaab0 0x7f7b2c13d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:55668/0 (socket says 172.21.15.23:55668) 2020-11-20T11:43:58.606 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.593+0000 7f7b2bfff700 1 -- 172.21.15.23:0/2445020376 learned_addr learned my addr 172.21.15.23:0/2445020376 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:43:58.606 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.593+0000 7f7b30e77700 1 --2- 172.21.15.23:0/2445020376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b2c0a5170 0x7f7b2c14d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:58.606 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.593+0000 7f7b2bfff700 1 -- 172.21.15.23:0/2445020376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b2c0a5170 msgr2=0x7f7b2c14d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:58.606 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.593+0000 7f7b2bfff700 1 --2- 172.21.15.23:0/2445020376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b2c0a5170 0x7f7b2c14d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:58.607 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.593+0000 7f7b2bfff700 1 -- 172.21.15.23:0/2445020376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b2c0aae90 msgr2=0x7f7b2c0a77b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:43:58.607 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.593+0000 7f7b2bfff700 1 --2- 172.21.15.23:0/2445020376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b2c0aae90 0x7f7b2c0a77b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:58.607 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.593+0000 7f7b2bfff700 1 -- 172.21.15.23:0/2445020376 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2c0a7ca0 con 0x7f7b2c0aaab0 2020-11-20T11:43:58.607 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.593+0000 7f7b2bfff700 1 --2- 172.21.15.23:0/2445020376 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b2c0aaab0 0x7f7b2c13d510 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f7b1c00c770 tx=0x7f7b1c007b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:58.607 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.593+0000 7f7b0bfff700 1 -- 172.21.15.23:0/2445020376 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c007320 con 0x7f7b2c0aaab0 2020-11-20T11:43:58.608 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.594+0000 7f7b0bfff700 1 -- 172.21.15.23:0/2445020376 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1c007480 con 0x7f7b2c0aaab0 2020-11-20T11:43:58.608 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.594+0000 7f7b328da700 1 -- 172.21.15.23:0/2445020376 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b2c14b0a0 con 0x7f7b2c0aaab0 2020-11-20T11:43:58.608 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.594+0000 7f7b0bfff700 1 -- 172.21.15.23:0/2445020376 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c005840 con 0x7f7b2c0aaab0 2020-11-20T11:43:58.608 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.594+0000 7f7b328da700 1 -- 172.21.15.23:0/2445020376 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b2c0a55b0 con 0x7f7b2c0aaab0 2020-11-20T11:43:58.609 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.594+0000 7f7b328da700 1 --2- 172.21.15.23:0/2445020376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b2c13aeb0 0x7f7b2c14dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:58.609 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.594+0000 7f7b328da700 1 -- 172.21.15.23:0/2445020376 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2c0a5830 con 0x7f7b2c13aeb0 2020-11-20T11:43:58.612 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.595+0000 7f7b2b7fe700 1 --2- 172.21.15.23:0/2445020376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b2c13aeb0 0x7f7b2c14dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:58.612 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.595+0000 7f7b0bfff700 1 -- 172.21.15.23:0/2445020376 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7b1c010030 con 0x7f7b2c0aaab0 2020-11-20T11:43:58.612 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.595+0000 7f7b2b7fe700 1 -- 172.21.15.23:0/2445020376 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7b2c14a510 con 0x7f7b2c13aeb0 2020-11-20T11:43:58.613 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.595+0000 7f7b2b7fe700 1 --2- 172.21.15.23:0/2445020376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b2c13aeb0 0x7f7b2c14dd90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f7b20006fd0 tx=0x7f7b20015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:58.613 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.595+0000 7f7b0bfff700 1 --2- 172.21.15.23:0/2445020376 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7b04056d40 0x7f7b04059200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:58.613 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.595+0000 7f7b0bfff700 1 -- 172.21.15.23:0/2445020376 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f7b1c06fd20 con 0x7f7b2c0aaab0 2020-11-20T11:43:58.613 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.595+0000 7f7b0bfff700 1 -- 172.21.15.23:0/2445020376 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b20011ba0 con 0x7f7b2c13aeb0 2020-11-20T11:43:58.613 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.595+0000 7f7b30e77700 1 --2- 172.21.15.23:0/2445020376 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7b04056d40 0x7f7b04059200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:58.614 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.595+0000 7f7b0bfff700 1 -- 172.21.15.23:0/2445020376 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f7b20011d00 con 0x7f7b2c13aeb0 2020-11-20T11:43:58.614 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.595+0000 7f7b0bfff700 1 -- 172.21.15.23:0/2445020376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b2c13aeb0 msgr2=0x7f7b2c14dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:58.614 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.595+0000 7f7b0bfff700 1 --2- 172.21.15.23:0/2445020376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b2c13aeb0 0x7f7b2c14dd90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f7b20006fd0 tx=0x7f7b20015000).stop 2020-11-20T11:43:58.614 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.595+0000 7f7b0bfff700 1 -- 172.21.15.23:0/2445020376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b2c13aeb0 msgr2=0x7f7b2c14dd90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:43:58.614 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.595+0000 7f7b0bfff700 1 --2- 172.21.15.23:0/2445020376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b2c13aeb0 0x7f7b2c14dd90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:58.615 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.601+0000 7f7b30e77700 1 --2- 172.21.15.23:0/2445020376 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7b04056d40 0x7f7b04059200 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f7b24006fd0 tx=0x7f7b240090f0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:58.615 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.607+0000 7f7b328da700 1 --2- 172.21.15.23:0/2445020376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b2c0601b0 0x7f7b2c14e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:58.615 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.607+0000 7f7b328da700 1 -- 172.21.15.23:0/2445020376 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2c14e770 con 0x7f7b2c0601b0 2020-11-20T11:43:58.615 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.607+0000 7f7b2b7fe700 1 --2- 172.21.15.23:0/2445020376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b2c0601b0 0x7f7b2c14e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:58.615 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.607+0000 7f7b2b7fe700 1 -- 172.21.15.23:0/2445020376 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7b2c05c7d0 con 0x7f7b2c0601b0 2020-11-20T11:43:58.616 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.608+0000 7f7b2b7fe700 1 --2- 172.21.15.23:0/2445020376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b2c0601b0 0x7f7b2c14e280 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f7b2c0a5970 tx=0x7f7b2001e000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:58.616 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.608+0000 7f7b0bfff700 1 -- 172.21.15.23:0/2445020376 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2001d7a0 con 0x7f7b2c0601b0 2020-11-20T11:43:58.616 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.608+0000 7f7b0bfff700 1 -- 172.21.15.23:0/2445020376 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4775 (secure 0 0 0) 0x7f7b20011890 con 0x7f7b2c0601b0 2020-11-20T11:43:58.616 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.608+0000 7f7b0bfff700 1 -- 172.21.15.23:0/2445020376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b2c0601b0 msgr2=0x7f7b2c14e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:58.616 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.608+0000 7f7b0bfff700 1 --2- 172.21.15.23:0/2445020376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b2c0601b0 0x7f7b2c14e280 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f7b2c0a5970 tx=0x7f7b2001e000).stop 2020-11-20T11:43:58.617 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.608+0000 7f7b0bfff700 1 -- 172.21.15.23:0/2445020376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b2c0601b0 msgr2=0x7f7b2c14e280 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:43:58.617 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.608+0000 7f7b0bfff700 1 --2- 172.21.15.23:0/2445020376 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b2c0601b0 0x7f7b2c14e280 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0x7f7b2c0a5970 tx=0x7f7b2001e000).stop 2020-11-20T11:43:58.617 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.608+0000 7f7b328da700 1 -- 172.21.15.23:0/2445020376 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7b04056d40 msgr2=0x7f7b04059200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:58.617 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.608+0000 7f7b328da700 1 --2- 172.21.15.23:0/2445020376 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7b04056d40 0x7f7b04059200 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f7b24006fd0 tx=0x7f7b240090f0).stop 2020-11-20T11:43:58.617 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.608+0000 7f7b328da700 1 -- 172.21.15.23:0/2445020376 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b2c0aaab0 msgr2=0x7f7b2c13d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:58.618 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.608+0000 7f7b328da700 1 --2- 172.21.15.23:0/2445020376 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b2c0aaab0 0x7f7b2c13d510 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f7b1c00c770 tx=0x7f7b1c007b10).stop 2020-11-20T11:43:58.618 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.608+0000 7f7b30e77700 1 -- 172.21.15.23:0/2445020376 reap_dead start 2020-11-20T11:43:58.618 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.608+0000 7f7b30e77700 1 -- 172.21.15.23:0/2445020376 reap_dead start 2020-11-20T11:43:58.618 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.608+0000 7f7b328da700 1 -- 172.21.15.23:0/2445020376 shutdown_connections 2020-11-20T11:43:58.618 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.608+0000 7f7b328da700 1 -- 172.21.15.23:0/2445020376 >> 172.21.15.23:0/2445020376 conn(0x7f7b2c0678e0 msgr2=0x7f7b2c068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:58.619 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.609+0000 7f7b328da700 1 -- 172.21.15.23:0/2445020376 shutdown_connections 2020-11-20T11:43:58.619 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:58.609+0000 7f7b328da700 1 -- 172.21.15.23:0/2445020376 wait complete. 2020-11-20T11:43:58.619 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:43:58.619 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:43:58.619 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:43:58.620 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:43:58.620 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 104, 2020-11-20T11:43:58.621 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:43:58.621 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:43:58.621 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:43:58.621 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:58.621 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 1, 2020-11-20T11:43:58.622 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:43:58.622 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:43:58.622 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:43:58.622 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:43:58.622 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:43:58.623 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:43:58.623 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:43:58.623 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:43:58.623 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:43:58.623 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:43:58.624 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:43:58.624 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:58.624 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:43:58.624 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:43:58.624 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:43:58.625 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:43:58.625 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:43:58.625 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:43:58.625 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:43:58.625 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:43:58.626 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:43:58.626 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:43:58.626 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:58.626 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:58.626 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:43:58.627 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:43:58.627 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:43:58.627 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:43:58.627 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:43:58.627 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:43:58.628 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:43:58.628 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:43:58.628 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:43:58.628 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:43:58.628 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:43:58.629 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:43:58.629 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:43:58.629 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:43:58.629 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:43:58.629 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:43:58.630 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:43:58.630 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:43:58.630 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:43:58.630 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:43:58.630 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:43:58.631 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:43:58.631 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:43:58.631 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:58.631 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:43:58.631 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:58.632 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:43:58.632 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:58.632 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:43:58.632 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:43:58.632 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:43:58.633 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:43:58.633 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:58.633 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:43:58.633 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:43:58.633 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:58.634 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:58.634 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:58.634 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:43:58.634 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:43:58.634 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:58.635 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:58.635 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:58.635 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:58.635 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:43:58.635 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:43:58.636 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:43:58.636 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:43:58.636 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:43:58.636 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:58.636 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:58.637 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:43:58.637 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:43:58.637 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:43:58.637 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:43:58.637 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:58.638 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:43:58.638 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:43:58.638 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:58.638 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:58.638 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:58.638 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:43:58.639 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:43:58.639 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:58.639 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:58.639 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:58.640 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:58.640 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:43:58.640 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:43:58.640 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:43:58.640 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:43:58.640 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:43:58.641 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:58.641 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:58.641 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:43:58.641 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:43:58.641 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:43:58.642 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:43:58.642 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:58.642 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:43:58.642 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:43:58.643 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:58.643 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:58.643 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:58.643 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:43:58.643 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:43:58.643 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:58.644 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:58.644 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:58.644 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:58.644 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:43:58.644 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:43:58.645 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:43:58.645 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:43:58.645 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:43:58.645 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:58.646 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:58.646 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:58.646 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:43:58.646 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:43:58.646 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:58.646 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:43:58.647 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:43:58.647 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:43:58.647 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:58.647 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:58.647 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:43:58.648 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:58.648 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:43:58.648 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:43:58.648 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:43:58.648 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:58.649 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:58.649 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:43:58.649 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:58.649 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:43:58.649 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:43:58.650 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:43:58.650 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:58.650 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:58.650 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:43:58.650 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:58.651 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:43:58.651 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:43:58.651 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:43:58.651 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:58.651 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:58.652 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:43:58.652 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:58.652 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:43:58.652 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:43:58.652 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:43:58.653 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:58.653 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:58.653 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:58.653 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:43:58.653 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:43:58.655 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:43:59.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.103+0000 7f1b3c463700 1 Processor -- start 2020-11-20T11:43:59.107 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.103+0000 7f1b3c463700 1 -- start start 2020-11-20T11:43:59.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.104+0000 7f1b3c463700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1b34063fe0 0x7f1b3413c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:59.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.104+0000 7f1b3c463700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b340678e0 0x7f1b34065b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:59.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.104+0000 7f1b3c463700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b34062b10 con 0x7f1b340ad7c0 2020-11-20T11:43:59.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.104+0000 7f1b3c463700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b3413d2d0 con 0x7f1b34063fe0 2020-11-20T11:43:59.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.104+0000 7f1b3c463700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b34065760 con 0x7f1b340678e0 2020-11-20T11:43:59.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.104+0000 7f1b3a1ff700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f1b340ad7c0 legacy=0x7f1b340671d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2020-11-20T11:43:59.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.105+0000 7f1b399fe700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f1b34063fe0 msgr2=0x7f1b3413c3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:43:59.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.105+0000 7f1b399fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1b34063fe0 0x7f1b3413c3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:43:59.110 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.105+0000 7f1b3aa00700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b340678e0 0x7f1b34065b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:59.110 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.105+0000 7f1b3aa00700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b340678e0 0x7f1b34065b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:55684/0 (socket says 172.21.15.23:55684) 2020-11-20T11:43:59.110 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.105+0000 7f1b3aa00700 1 -- 172.21.15.23:0/592321994 learned_addr learned my addr 172.21.15.23:0/592321994 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:43:59.111 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.105+0000 7f1b3aa00700 1 -- 172.21.15.23:0/592321994 >> v2:172.21.15.23:3300/0 conn(0x7f1b34063fe0 msgr2=0x7f1b3413c3a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:43:59.111 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.105+0000 7f1b3aa00700 1 --2- 172.21.15.23:0/592321994 >> v2:172.21.15.23:3300/0 conn(0x7f1b34063fe0 0x7f1b3413c3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:59.111 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.105+0000 7f1b3aa00700 1 -- 172.21.15.23:0/592321994 >> v1:172.21.15.23:6789/0 conn(0x7f1b340ad7c0 legacy=0x7f1b340671d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:43:59.112 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.105+0000 7f1b3aa00700 1 -- 172.21.15.23:0/592321994 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b340a7db0 con 0x7f1b340678e0 2020-11-20T11:43:59.112 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.106+0000 7f1b3aa00700 1 --2- 172.21.15.23:0/592321994 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b340678e0 0x7f1b34065b50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f1b2c009d90 tx=0x7f1b2c0092d0).ready entity=mon.2 client_cookie=9aaf847b0789a6c4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:59.112 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.106+0000 7f1b391fd700 1 -- 172.21.15.23:0/592321994 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c00bbb0 con 0x7f1b340678e0 2020-11-20T11:43:59.112 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.106+0000 7f1b391fd700 1 -- 172.21.15.23:0/592321994 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b2c00bd10 con 0x7f1b340678e0 2020-11-20T11:43:59.112 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.106+0000 7f1b391fd700 1 -- 172.21.15.23:0/592321994 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c011610 con 0x7f1b340678e0 2020-11-20T11:43:59.113 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.107+0000 7f1b3c463700 1 -- 172.21.15.23:0/592321994 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b340678e0 msgr2=0x7f1b34065b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:59.113 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.107+0000 7f1b3c463700 1 --2- 172.21.15.23:0/592321994 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b340678e0 0x7f1b34065b50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f1b2c009d90 tx=0x7f1b2c0092d0).stop 2020-11-20T11:43:59.113 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.107+0000 7f1b3c463700 1 -- 172.21.15.23:0/592321994 shutdown_connections 2020-11-20T11:43:59.113 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.107+0000 7f1b3c463700 1 --2- 172.21.15.23:0/592321994 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b340678e0 0x7f1b34065b50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:59.113 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.107+0000 7f1b3c463700 1 --2- 172.21.15.23:0/592321994 >> v2:172.21.15.23:3300/0 conn(0x7f1b34063fe0 0x7f1b3413c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:59.114 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.107+0000 7f1b3c463700 1 -- 172.21.15.23:0/592321994 >> 172.21.15.23:0/592321994 conn(0x7f1b34060c80 msgr2=0x7f1b340acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:59.116 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.114+0000 7f1b3c463700 1 -- 172.21.15.23:0/592321994 shutdown_connections 2020-11-20T11:43:59.118 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.114+0000 7f1b3c463700 1 -- 172.21.15.23:0/592321994 wait complete. 2020-11-20T11:43:59.118 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.115+0000 7f1b3c463700 1 Processor -- start 2020-11-20T11:43:59.118 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.115+0000 7f1b3c463700 1 -- start start 2020-11-20T11:43:59.119 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.116+0000 7f1b3c463700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b34063fe0 0x7f1b3409e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:59.119 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.116+0000 7f1b3c463700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b340678e0 0x7f1b34156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:59.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.116+0000 7f1b3c463700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b340ad7c0 0x7f1b340a0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:59.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.116+0000 7f1b3c463700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b34067f60 con 0x7f1b340ad7c0 2020-11-20T11:43:59.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.116+0000 7f1b3c463700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3409d5b0 con 0x7f1b34063fe0 2020-11-20T11:43:59.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.116+0000 7f1b3c463700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3409d810 con 0x7f1b340678e0 2020-11-20T11:43:59.121 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.116+0000 7f1b3aa00700 1 -- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b340ad7c0 msgr2=0x7f1b340a0410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:43:59.121 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.116+0000 7f1b3aa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b340ad7c0 0x7f1b340a0410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:43:59.121 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.116+0000 7f1b399fe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b340678e0 0x7f1b34156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:59.121 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.117+0000 7f1b399fe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b340678e0 0x7f1b34156ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:55690/0 (socket says 172.21.15.23:55690) 2020-11-20T11:43:59.121 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.117+0000 7f1b399fe700 1 -- 172.21.15.23:0/881196115 learned_addr learned my addr 172.21.15.23:0/881196115 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:43:59.122 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.117+0000 7f1b3a1ff700 1 --2- 172.21.15.23:0/881196115 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b34063fe0 0x7f1b3409e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:59.122 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.117+0000 7f1b399fe700 1 -- 172.21.15.23:0/881196115 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b34063fe0 msgr2=0x7f1b3409e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:59.122 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.117+0000 7f1b399fe700 1 --2- 172.21.15.23:0/881196115 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b34063fe0 0x7f1b3409e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:59.122 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.117+0000 7f1b399fe700 1 -- 172.21.15.23:0/881196115 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b340ad7c0 msgr2=0x7f1b340a0410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:43:59.123 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.117+0000 7f1b399fe700 1 --2- 172.21.15.23:0/881196115 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b340ad7c0 0x7f1b340a0410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:43:59.123 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.117+0000 7f1b399fe700 1 -- 172.21.15.23:0/881196115 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3409dab0 con 0x7f1b340678e0 2020-11-20T11:43:59.125 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.121+0000 7f1b399fe700 1 --2- 172.21.15.23:0/881196115 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b340678e0 0x7f1b34156ea0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f1b24009710 tx=0x7f1b24006e20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:59.125 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.122+0000 7f1b227fc700 1 -- 172.21.15.23:0/881196115 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b240050d0 con 0x7f1b340678e0 2020-11-20T11:43:59.126 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.122+0000 7f1b3c463700 1 -- 172.21.15.23:0/881196115 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b3409df50 con 0x7f1b340678e0 2020-11-20T11:43:59.126 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.122+0000 7f1b3c463700 1 -- 172.21.15.23:0/881196115 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b341575d0 con 0x7f1b340678e0 2020-11-20T11:43:59.126 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.123+0000 7f1b227fc700 1 -- 172.21.15.23:0/881196115 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b24005230 con 0x7f1b340678e0 2020-11-20T11:43:59.128 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.125+0000 7f1b227fc700 1 -- 172.21.15.23:0/881196115 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b24003820 con 0x7f1b340678e0 2020-11-20T11:43:59.128 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.125+0000 7f1b3c463700 1 --2- 172.21.15.23:0/881196115 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b340a09c0 0x7f1b34153ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:59.129 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.125+0000 7f1b3c463700 1 -- 172.21.15.23:0/881196115 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b340a6c80 con 0x7f1b340a09c0 2020-11-20T11:43:59.129 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.125+0000 7f1b227fc700 1 -- 172.21.15.23:0/881196115 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1b2401a060 con 0x7f1b340678e0 2020-11-20T11:43:59.129 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.126+0000 7f1b3a1ff700 1 --2- 172.21.15.23:0/881196115 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b340a09c0 0x7f1b34153ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:59.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.126+0000 7f1b227fc700 1 --2- 172.21.15.23:0/881196115 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1b0c056d60 0x7f1b0c059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:59.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.126+0000 7f1b227fc700 1 -- 172.21.15.23:0/881196115 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f1b2406fa40 con 0x7f1b340678e0 2020-11-20T11:43:59.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.126+0000 7f1b3aa00700 1 --2- 172.21.15.23:0/881196115 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1b0c056d60 0x7f1b0c059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:59.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.126+0000 7f1b3aa00700 1 --2- 172.21.15.23:0/881196115 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1b0c056d60 0x7f1b0c059220 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f1b2c009310 tx=0x7f1b2c016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:59.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.129+0000 7f1b3a1ff700 1 -- 172.21.15.23:0/881196115 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1b34061650 con 0x7f1b340a09c0 2020-11-20T11:43:59.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.129+0000 7f1b3a1ff700 1 --2- 172.21.15.23:0/881196115 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b340a09c0 0x7f1b34153ea0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f1b300036e0 tx=0x7f1b30006e10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:59.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.133+0000 7f1b227fc700 1 -- 172.21.15.23:0/881196115 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3000e620 con 0x7f1b340a09c0 2020-11-20T11:43:59.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.133+0000 7f1b227fc700 1 -- 172.21.15.23:0/881196115 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f1b3000e780 con 0x7f1b340a09c0 2020-11-20T11:43:59.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.133+0000 7f1b227fc700 1 -- 172.21.15.23:0/881196115 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b340a09c0 msgr2=0x7f1b34153ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:59.137 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.133+0000 7f1b227fc700 1 --2- 172.21.15.23:0/881196115 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b340a09c0 0x7f1b34153ea0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f1b300036e0 tx=0x7f1b30006e10).stop 2020-11-20T11:43:59.137 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.133+0000 7f1b227fc700 1 -- 172.21.15.23:0/881196115 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b340a09c0 msgr2=0x7f1b34153ea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:43:59.137 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.133+0000 7f1b227fc700 1 --2- 172.21.15.23:0/881196115 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b340a09c0 0x7f1b34153ea0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:59.149 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.146+0000 7f1b3c463700 1 --2- 172.21.15.23:0/881196115 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b34154390 0x7f1b3415a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:43:59.149 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.146+0000 7f1b3c463700 1 -- 172.21.15.23:0/881196115 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3405c520 con 0x7f1b34154390 2020-11-20T11:43:59.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.146+0000 7f1b3a1ff700 1 --2- 172.21.15.23:0/881196115 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b34154390 0x7f1b3415a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:43:59.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.147+0000 7f1b3a1ff700 1 -- 172.21.15.23:0/881196115 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1b34044710 con 0x7f1b34154390 2020-11-20T11:43:59.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.147+0000 7f1b3a1ff700 1 --2- 172.21.15.23:0/881196115 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b34154390 0x7f1b3415a470 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f1b300036e0 tx=0x7f1b30013000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:43:59.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.147+0000 7f1b227fc700 1 -- 172.21.15.23:0/881196115 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3001a350 con 0x7f1b34154390 2020-11-20T11:43:59.152 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.147+0000 7f1b227fc700 1 -- 172.21.15.23:0/881196115 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4773 (secure 0 0 0) 0x7f1b3000e8f0 con 0x7f1b34154390 2020-11-20T11:43:59.152 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.147+0000 7f1b227fc700 1 -- 172.21.15.23:0/881196115 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b34154390 msgr2=0x7f1b3415a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:59.152 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.147+0000 7f1b227fc700 1 --2- 172.21.15.23:0/881196115 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b34154390 0x7f1b3415a470 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f1b300036e0 tx=0x7f1b30013000).stop 2020-11-20T11:43:59.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.147+0000 7f1b227fc700 1 -- 172.21.15.23:0/881196115 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b34154390 msgr2=0x7f1b3415a470 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:43:59.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.147+0000 7f1b227fc700 1 --2- 172.21.15.23:0/881196115 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b34154390 0x7f1b3415a470 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:43:59.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.148+0000 7f1b3c463700 1 -- 172.21.15.23:0/881196115 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1b0c056d60 msgr2=0x7f1b0c059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:59.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.148+0000 7f1b3c463700 1 --2- 172.21.15.23:0/881196115 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1b0c056d60 0x7f1b0c059220 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f1b2c009310 tx=0x7f1b2c016000).stop 2020-11-20T11:43:59.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.148+0000 7f1b3c463700 1 -- 172.21.15.23:0/881196115 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b340678e0 msgr2=0x7f1b34156ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:43:59.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.148+0000 7f1b3c463700 1 --2- 172.21.15.23:0/881196115 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b340678e0 0x7f1b34156ea0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f1b24009710 tx=0x7f1b24006e20).stop 2020-11-20T11:43:59.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.148+0000 7f1b3aa00700 1 -- 172.21.15.23:0/881196115 reap_dead start 2020-11-20T11:43:59.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.148+0000 7f1b3aa00700 1 -- 172.21.15.23:0/881196115 reap_dead start 2020-11-20T11:43:59.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.148+0000 7f1b3c463700 1 -- 172.21.15.23:0/881196115 shutdown_connections 2020-11-20T11:43:59.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.148+0000 7f1b3c463700 1 -- 172.21.15.23:0/881196115 >> 172.21.15.23:0/881196115 conn(0x7f1b34060c80 msgr2=0x7f1b34066e60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:43:59.155 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.148+0000 7f1b3c463700 1 -- 172.21.15.23:0/881196115 shutdown_connections 2020-11-20T11:43:59.155 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:43:59.148+0000 7f1b3c463700 1 -- 172.21.15.23:0/881196115 wait complete. 2020-11-20T11:43:59.155 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:43:59.155 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:43:59.155 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:43:59.156 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:43:59.156 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 104, 2020-11-20T11:43:59.156 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:43:59.156 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:43:59.156 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:43:59.157 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:59.157 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 1, 2020-11-20T11:43:59.157 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:43:59.157 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:43:59.157 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:43:59.158 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:43:59.158 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:43:59.158 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:43:59.158 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:43:59.158 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:43:59.159 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:43:59.159 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:43:59.159 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:43:59.159 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:59.159 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:43:59.160 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:43:59.160 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:43:59.160 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:43:59.160 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:43:59.160 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:43:59.160 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:43:59.161 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:43:59.161 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:43:59.161 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:43:59.161 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:59.161 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:59.162 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:43:59.162 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:43:59.162 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:43:59.162 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:43:59.162 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:43:59.163 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:43:59.163 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:43:59.163 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:43:59.163 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:43:59.163 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:43:59.164 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:43:59.164 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:43:59.164 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:43:59.164 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:43:59.164 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:43:59.164 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:43:59.165 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:43:59.165 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:43:59.165 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:43:59.165 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:43:59.165 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:43:59.166 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:43:59.166 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:43:59.166 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:59.166 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:43:59.166 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:59.167 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:43:59.167 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:59.167 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:43:59.167 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:43:59.167 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:43:59.167 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:43:59.168 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:59.168 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:43:59.168 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:43:59.168 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:59.168 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:59.169 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:59.169 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:43:59.169 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:43:59.169 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:59.169 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:59.170 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:59.170 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:59.170 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:43:59.170 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:43:59.170 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:43:59.171 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:43:59.171 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:43:59.171 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:59.171 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:59.171 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:43:59.172 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:43:59.172 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:43:59.172 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:43:59.172 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:59.172 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:43:59.173 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:43:59.173 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:59.173 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:59.173 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:59.173 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:43:59.174 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:43:59.174 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:59.174 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:59.174 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:59.174 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:59.175 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:43:59.175 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:43:59.175 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:43:59.175 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:43:59.175 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:43:59.176 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:59.176 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:59.176 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:43:59.176 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:43:59.176 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:43:59.177 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:43:59.177 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:59.177 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:43:59.177 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:43:59.177 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:59.178 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:59.178 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:59.178 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:43:59.178 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:43:59.178 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:43:59.179 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:59.179 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:59.179 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:59.179 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:43:59.179 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:43:59.180 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:43:59.180 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:43:59.180 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:43:59.180 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:59.180 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:59.181 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:59.181 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:43:59.181 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:43:59.181 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:59.181 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:43:59.182 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:43:59.182 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:43:59.182 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:59.182 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:59.182 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:43:59.183 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:59.183 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:43:59.183 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:43:59.183 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:43:59.183 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:59.184 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:59.184 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:43:59.184 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:59.184 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:43:59.184 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:43:59.185 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:43:59.185 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:59.185 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:59.185 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:43:59.185 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:59.186 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:43:59.186 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:43:59.186 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:43:59.186 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:59.186 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:43:59.186 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:43:59.187 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:43:59.187 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:43:59.187 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:43:59.187 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:43:59.188 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:43:59.188 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:43:59.188 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:43:59.188 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:43:59.188 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:43:59.189 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2020-11-20T11:44:19.191 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2020-11-20T11:44:19.191 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2020-11-20T11:44:19.192 INFO:tasks.ceph.mon.a:Restarting daemon 2020-11-20T11:44:19.192 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2020-11-20T11:44:19.195 INFO:tasks.ceph.mon.a:Started 2020-11-20T11:44:19.195 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:44:19.195 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:44:22.527 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.522+0000 7f5484f1d700 1 Processor -- start 2020-11-20T11:44:22.529 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.524+0000 7f5484f1d700 1 -- start start 2020-11-20T11:44:22.530 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.525+0000 7f5484f1d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54800ad1e0 0x7f54800a60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:22.530 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.525+0000 7f5484f1d700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5480136960 0x7f54801412d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:22.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.525+0000 7f5484f1d700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5480136e00 con 0x7f54800ad5c0 2020-11-20T11:44:22.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.525+0000 7f5484f1d700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54801370a0 con 0x7f5480136960 2020-11-20T11:44:22.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.525+0000 7f5484f1d700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5480137340 con 0x7f54800ad1e0 2020-11-20T11:44:22.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.525+0000 7f547effd700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f54800ad5c0 0x7f5480133bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:37698/0 (socket says 172.21.15.23:37698) 2020-11-20T11:44:22.540 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.525+0000 7f547effd700 1 -- 172.21.15.23:0/3365928947 learned_addr learned my addr 172.21.15.23:0/3365928947 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:44:22.541 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.525+0000 7f547ffff700 1 --2- 172.21.15.23:0/3365928947 >> v2:172.21.15.155:3300/0 conn(0x7f5480136960 0x7f54801412d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:22.541 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.526+0000 7f547f7fe700 1 --2- 172.21.15.23:0/3365928947 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54800ad1e0 0x7f54800a60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:22.541 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.526+0000 7f547e7fc700 1 -- 172.21.15.23:0/3365928947 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1018080772 0 0) 0x7f5480136e00 con 0x7f54800ad5c0 2020-11-20T11:44:22.542 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.526+0000 7f547e7fc700 1 -- 172.21.15.23:0/3365928947 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f546c003280 con 0x7f54800ad5c0 2020-11-20T11:44:22.542 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.526+0000 7f547ffff700 1 -- 172.21.15.23:0/3365928947 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54800ad1e0 msgr2=0x7f54800a60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:44:22.542 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.526+0000 7f547ffff700 1 --2- 172.21.15.23:0/3365928947 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54800ad1e0 0x7f54800a60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:22.543 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.526+0000 7f547ffff700 1 -- 172.21.15.23:0/3365928947 >> v1:172.21.15.155:6789/0 conn(0x7f54800ad5c0 legacy=0x7f5480133bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:22.543 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.526+0000 7f547ffff700 1 -- 172.21.15.23:0/3365928947 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54801377e0 con 0x7f5480136960 2020-11-20T11:44:22.543 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.526+0000 7f547ffff700 1 --2- 172.21.15.23:0/3365928947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5480136960 0x7f54801412d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f5470007fd0 tx=0x7f5470007ed0).ready entity=mon.1 client_cookie=b306d34f3e1d884f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:22.544 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.528+0000 7f547e7fc700 1 -- 172.21.15.23:0/3365928947 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547000be30 con 0x7f5480136960 2020-11-20T11:44:22.544 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.528+0000 7f547e7fc700 1 -- 172.21.15.23:0/3365928947 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5470000e90 con 0x7f5480136960 2020-11-20T11:44:22.544 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.528+0000 7f547e7fc700 1 -- 172.21.15.23:0/3365928947 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5470006a30 con 0x7f5480136960 2020-11-20T11:44:22.545 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.528+0000 7f5484f1d700 1 -- 172.21.15.23:0/3365928947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5480136960 msgr2=0x7f54801412d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:22.545 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.528+0000 7f5484f1d700 1 --2- 172.21.15.23:0/3365928947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5480136960 0x7f54801412d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f5470007fd0 tx=0x7f5470007ed0).stop 2020-11-20T11:44:22.545 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.530+0000 7f5484f1d700 1 -- 172.21.15.23:0/3365928947 shutdown_connections 2020-11-20T11:44:22.546 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.530+0000 7f5484f1d700 1 --2- 172.21.15.23:0/3365928947 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5480136960 0x7f54801412d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:22.546 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.530+0000 7f5484f1d700 1 --2- 172.21.15.23:0/3365928947 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54800ad1e0 0x7f54800a60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:22.547 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.530+0000 7f5484f1d700 1 -- 172.21.15.23:0/3365928947 >> 172.21.15.23:0/3365928947 conn(0x7f5480061850 msgr2=0x7f5480061c60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:44:22.547 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.536+0000 7f5484f1d700 1 -- 172.21.15.23:0/3365928947 shutdown_connections 2020-11-20T11:44:22.551 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.537+0000 7f5484f1d700 1 -- 172.21.15.23:0/3365928947 wait complete. 2020-11-20T11:44:22.551 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.538+0000 7f5484f1d700 1 Processor -- start 2020-11-20T11:44:22.551 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.538+0000 7f5484f1d700 1 -- start start 2020-11-20T11:44:22.552 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.539+0000 7f5484f1d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54800ad1e0 0x7f54800a3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:22.552 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.539+0000 7f5484f1d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f54800ad5c0 0x7f548009d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:22.552 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.539+0000 7f5484f1d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5480136960 0x7f548013c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:22.553 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.540+0000 7f5484f1d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5480138e00 con 0x7f5480136960 2020-11-20T11:44:22.553 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.540+0000 7f5484f1d700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548013b040 con 0x7f54800ad5c0 2020-11-20T11:44:22.553 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.540+0000 7f5484f1d700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54800a3bf0 con 0x7f54800ad1e0 2020-11-20T11:44:22.554 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.540+0000 7f547ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5480136960 0x7f548013c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:22.554 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.540+0000 7f547effd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f54800ad5c0 0x7f548009d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:22.554 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.540+0000 7f547ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5480136960 0x7f548013c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45480/0 (socket says 172.21.15.23:45480) 2020-11-20T11:44:22.554 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.540+0000 7f547ffff700 1 -- 172.21.15.23:0/2468412392 learned_addr learned my addr 172.21.15.23:0/2468412392 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:44:22.555 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.540+0000 7f547ffff700 1 -- 172.21.15.23:0/2468412392 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5480136960 msgr2=0x7f548013c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2020-11-20T11:44:22.555 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.540+0000 7f547ffff700 1 -- 172.21.15.23:0/2468412392 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5480136960 msgr2=0x7f548013c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:44:22.555 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.540+0000 7f547ffff700 1 --2- 172.21.15.23:0/2468412392 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5480136960 0x7f548013c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:44:22.556 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.540+0000 7f547ffff700 1 --2- 172.21.15.23:0/2468412392 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5480136960 0x7f548013c4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:44:22.556 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.540+0000 7f547f7fe700 1 --2- 172.21.15.23:0/2468412392 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54800ad1e0 0x7f54800a3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:22.556 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.541+0000 7f547effd700 1 -- 172.21.15.23:0/2468412392 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54800ad1e0 msgr2=0x7f54800a3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:22.557 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.541+0000 7f547effd700 1 --2- 172.21.15.23:0/2468412392 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54800ad1e0 0x7f54800a3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:22.557 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.541+0000 7f547effd700 1 -- 172.21.15.23:0/2468412392 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5480136960 msgr2=0x7f548013c4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:44:22.557 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.541+0000 7f547effd700 1 --2- 172.21.15.23:0/2468412392 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5480136960 0x7f548013c4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:22.558 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.541+0000 7f547effd700 1 -- 172.21.15.23:0/2468412392 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f548013b270 con 0x7f54800ad5c0 2020-11-20T11:44:22.558 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.541+0000 7f547effd700 1 --2- 172.21.15.23:0/2468412392 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f54800ad5c0 0x7f548009d970 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f548005c970 tx=0x7f54780061b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:22.559 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.541+0000 7f54677fe700 1 -- 172.21.15.23:0/2468412392 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5478007d40 con 0x7f54800ad5c0 2020-11-20T11:44:22.559 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.541+0000 7f54677fe700 1 -- 172.21.15.23:0/2468412392 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f547800e440 con 0x7f54800ad5c0 2020-11-20T11:44:22.559 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.541+0000 7f5484f1d700 1 -- 172.21.15.23:0/2468412392 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f548013b510 con 0x7f54800ad5c0 2020-11-20T11:44:22.560 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.541+0000 7f5484f1d700 1 -- 172.21.15.23:0/2468412392 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f548013c9e0 con 0x7f54800ad5c0 2020-11-20T11:44:22.560 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.541+0000 7f54677fe700 1 -- 172.21.15.23:0/2468412392 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5478007ea0 con 0x7f54800ad5c0 2020-11-20T11:44:22.560 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.543+0000 7f54677fe700 1 -- 172.21.15.23:0/2468412392 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f547800ea90 con 0x7f54800ad5c0 2020-11-20T11:44:22.560 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.543+0000 7f54677fe700 1 --2- 172.21.15.23:0/2468412392 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5458056de0 0x7f54580592a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:22.561 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.543+0000 7f54677fe700 1 -- 172.21.15.23:0/2468412392 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f5478070b80 con 0x7f54800ad5c0 2020-11-20T11:44:22.561 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.544+0000 7f5484f1d700 1 -- 172.21.15.23:0/2468412392 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5480057dc0 con 0x7f54800ad5c0 2020-11-20T11:44:22.561 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.544+0000 7f547f7fe700 1 --2- 172.21.15.23:0/2468412392 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5458056de0 0x7f54580592a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:22.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.544+0000 7f547f7fe700 1 --2- 172.21.15.23:0/2468412392 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5458056de0 0x7f54580592a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f547400dc20 tx=0x7f547400f040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:22.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:22.551+0000 7f54677fe700 1 -- 172.21.15.23:0/2468412392 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5478045ef0 con 0x7f54800ad5c0 2020-11-20T11:44:23.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:23.131+0000 7f5484f1d700 1 -- 172.21.15.23:0/2468412392 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f54801342e0 con 0x7f54800ad5c0 2020-11-20T11:44:23.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:23.132+0000 7f54677fe700 1 -- 172.21.15.23:0/2468412392 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1473 (secure 0 0 0) 0x7f5478049510 con 0x7f54800ad5c0 2020-11-20T11:44:23.136 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":104,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":25,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:44:23.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:23.133+0000 7f5484f1d700 1 -- 172.21.15.23:0/2468412392 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5458056de0 msgr2=0x7f54580592a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:23.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:23.133+0000 7f5484f1d700 1 --2- 172.21.15.23:0/2468412392 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5458056de0 0x7f54580592a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f547400dc20 tx=0x7f547400f040).stop 2020-11-20T11:44:23.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:23.133+0000 7f5484f1d700 1 -- 172.21.15.23:0/2468412392 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f54800ad5c0 msgr2=0x7f548009d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:23.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:23.133+0000 7f5484f1d700 1 --2- 172.21.15.23:0/2468412392 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f54800ad5c0 0x7f548009d970 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f548005c970 tx=0x7f54780061b0).stop 2020-11-20T11:44:23.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:23.133+0000 7f5484f1d700 1 -- 172.21.15.23:0/2468412392 shutdown_connections 2020-11-20T11:44:23.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:23.133+0000 7f5484f1d700 1 --2- 172.21.15.23:0/2468412392 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5480136960 0x7f548013c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:23.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:23.133+0000 7f5484f1d700 1 --2- 172.21.15.23:0/2468412392 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5458056de0 0x7f54580592a0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:23.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:23.133+0000 7f5484f1d700 1 --2- 172.21.15.23:0/2468412392 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f54800ad1e0 0x7f54800a3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:23.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:23.133+0000 7f5484f1d700 1 --2- 172.21.15.23:0/2468412392 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f54800ad5c0 0x7f548009d970 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:23.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:23.134+0000 7f5484f1d700 1 -- 172.21.15.23:0/2468412392 >> 172.21.15.23:0/2468412392 conn(0x7f5480061850 msgr2=0x7f5480135c90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:44:23.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:23.134+0000 7f5484f1d700 1 -- 172.21.15.23:0/2468412392 shutdown_connections 2020-11-20T11:44:23.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:23.134+0000 7f5484f1d700 1 -- 172.21.15.23:0/2468412392 wait complete. 2020-11-20T11:44:26.165 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:44:26.528 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:26.524+0000 7f50ba351700 1 Processor -- start 2020-11-20T11:44:26.528 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:26.525+0000 7f50ba351700 1 -- start start 2020-11-20T11:44:26.529 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:26.525+0000 7f50ba351700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f50b40aae90 0x7f50b41381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:26.529 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:26.525+0000 7f50ba351700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50b413aeb0 0x7f50b40a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:26.530 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:26.525+0000 7f50ba351700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b413e600 con 0x7f50b40aaab0 2020-11-20T11:44:26.530 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:26.525+0000 7f50ba351700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50b413e8a0 con 0x7f50b40aae90 2020-11-20T11:44:26.530 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:26.525+0000 7f50ba351700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b40ab6d0 con 0x7f50b413aeb0 2020-11-20T11:44:26.530 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:26.525+0000 7f50b3fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f50b40aaab0 0x7f50b413ed40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:59742/0 (socket says 172.21.15.23:59742) 2020-11-20T11:44:26.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:26.525+0000 7f50b3fff700 1 -- 172.21.15.23:0/2396581992 learned_addr learned my addr 172.21.15.23:0/2396581992 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:44:26.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:26.525+0000 7f50b37fe700 1 --2- 172.21.15.23:0/2396581992 >> v2:172.21.15.155:3300/0 conn(0x7f50b40aae90 0x7f50b41381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:26.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:26.525+0000 7f50b88ee700 1 --2- 172.21.15.23:0/2396581992 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50b413aeb0 0x7f50b40a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:26.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:26.526+0000 7f50b37fe700 1 -- 172.21.15.23:0/2396581992 >> v2:172.21.15.155:3300/0 conn(0x7f50b40aae90 msgr2=0x7f50b41381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:44:26.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:26.526+0000 7f50b37fe700 1 -- 172.21.15.23:0/2396581992 >> v2:172.21.15.155:3300/0 conn(0x7f50b40aae90 msgr2=0x7f50b41381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:44:26.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:26.526+0000 7f50b37fe700 1 --2- 172.21.15.23:0/2396581992 >> v2:172.21.15.155:3300/0 conn(0x7f50b40aae90 0x7f50b41381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:44:26.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:26.526+0000 7f50b88ee700 1 -- 172.21.15.23:0/2396581992 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50b413aeb0 msgr2=0x7f50b40a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:44:26.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:26.526+0000 7f50b88ee700 1 -- 172.21.15.23:0/2396581992 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50b413aeb0 msgr2=0x7f50b40a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:44:26.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:26.526+0000 7f50b88ee700 1 --2- 172.21.15.23:0/2396581992 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50b413aeb0 0x7f50b40a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:44:26.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:26.526+0000 7f50b37fe700 1 --2- 172.21.15.23:0/2396581992 >> v2:172.21.15.155:3300/0 conn(0x7f50b40aae90 0x7f50b41381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:44:26.533 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:26.526+0000 7f50b88ee700 1 --2- 172.21.15.23:0/2396581992 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50b413aeb0 0x7f50b40a8c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:44:26.730 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:26.727+0000 7f50b88ee700 1 --2- 172.21.15.23:0/2396581992 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50b413aeb0 0x7f50b40a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:26.731 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:26.727+0000 7f50b37fe700 1 --2- 172.21.15.23:0/2396581992 >> v2:172.21.15.155:3300/0 conn(0x7f50b40aae90 0x7f50b41381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:26.731 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:26.727+0000 7f50b88ee700 1 -- 172.21.15.23:0/2396581992 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50b413aeb0 msgr2=0x7f50b40a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:44:26.731 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:26.727+0000 7f50b88ee700 1 -- 172.21.15.23:0/2396581992 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50b413aeb0 msgr2=0x7f50b40a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:44:26.731 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:26.727+0000 7f50b37fe700 1 -- 172.21.15.23:0/2396581992 >> v2:172.21.15.155:3300/0 conn(0x7f50b40aae90 msgr2=0x7f50b41381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:44:26.732 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:26.727+0000 7f50b37fe700 1 -- 172.21.15.23:0/2396581992 >> v2:172.21.15.155:3300/0 conn(0x7f50b40aae90 msgr2=0x7f50b41381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:44:26.732 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:26.727+0000 7f50b88ee700 1 --2- 172.21.15.23:0/2396581992 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50b413aeb0 0x7f50b40a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:44:26.732 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:26.727+0000 7f50b37fe700 1 --2- 172.21.15.23:0/2396581992 >> v2:172.21.15.155:3300/0 conn(0x7f50b40aae90 0x7f50b41381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:44:26.732 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:26.727+0000 7f50b88ee700 1 --2- 172.21.15.23:0/2396581992 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50b413aeb0 0x7f50b40a8c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:44:26.732 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:26.727+0000 7f50b37fe700 1 --2- 172.21.15.23:0/2396581992 >> v2:172.21.15.155:3300/0 conn(0x7f50b40aae90 0x7f50b41381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:44:27.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:27.128+0000 7f50b37fe700 1 --2- 172.21.15.23:0/2396581992 >> v2:172.21.15.155:3300/0 conn(0x7f50b40aae90 0x7f50b41381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:27.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:27.128+0000 7f50b88ee700 1 --2- 172.21.15.23:0/2396581992 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50b413aeb0 0x7f50b40a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:27.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:27.128+0000 7f50b37fe700 1 -- 172.21.15.23:0/2396581992 >> v2:172.21.15.155:3300/0 conn(0x7f50b40aae90 msgr2=0x7f50b41381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:44:27.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:27.128+0000 7f50b88ee700 1 -- 172.21.15.23:0/2396581992 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50b413aeb0 msgr2=0x7f50b40a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:44:27.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:27.128+0000 7f50b37fe700 1 -- 172.21.15.23:0/2396581992 >> v2:172.21.15.155:3300/0 conn(0x7f50b40aae90 msgr2=0x7f50b41381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:44:27.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:27.128+0000 7f50b88ee700 1 -- 172.21.15.23:0/2396581992 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50b413aeb0 msgr2=0x7f50b40a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:44:27.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:27.128+0000 7f50b37fe700 1 --2- 172.21.15.23:0/2396581992 >> v2:172.21.15.155:3300/0 conn(0x7f50b40aae90 0x7f50b41381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:44:27.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:27.128+0000 7f50b88ee700 1 --2- 172.21.15.23:0/2396581992 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50b413aeb0 0x7f50b40a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:44:27.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:27.128+0000 7f50b37fe700 1 --2- 172.21.15.23:0/2396581992 >> v2:172.21.15.155:3300/0 conn(0x7f50b40aae90 0x7f50b41381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:44:27.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:27.128+0000 7f50b88ee700 1 --2- 172.21.15.23:0/2396581992 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50b413aeb0 0x7f50b40a8c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:44:27.932 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:27.929+0000 7f50b37fe700 1 --2- 172.21.15.23:0/2396581992 >> v2:172.21.15.155:3300/0 conn(0x7f50b40aae90 0x7f50b41381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:27.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:27.929+0000 7f50b88ee700 1 --2- 172.21.15.23:0/2396581992 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50b413aeb0 0x7f50b40a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:27.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:27.929+0000 7f50b37fe700 1 -- 172.21.15.23:0/2396581992 >> v2:172.21.15.155:3300/0 conn(0x7f50b40aae90 msgr2=0x7f50b41381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:44:27.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:27.929+0000 7f50b37fe700 1 -- 172.21.15.23:0/2396581992 >> v2:172.21.15.155:3300/0 conn(0x7f50b40aae90 msgr2=0x7f50b41381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:44:27.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:27.929+0000 7f50b37fe700 1 --2- 172.21.15.23:0/2396581992 >> v2:172.21.15.155:3300/0 conn(0x7f50b40aae90 0x7f50b41381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:44:27.934 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:27.929+0000 7f50b37fe700 1 --2- 172.21.15.23:0/2396581992 >> v2:172.21.15.155:3300/0 conn(0x7f50b40aae90 0x7f50b41381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:44:27.934 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:27.930+0000 7f50b88ee700 1 -- 172.21.15.23:0/2396581992 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50b413aeb0 msgr2=0x7f50b40a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:44:27.934 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:27.930+0000 7f50b88ee700 1 -- 172.21.15.23:0/2396581992 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50b413aeb0 msgr2=0x7f50b40a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:44:27.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:27.930+0000 7f50b88ee700 1 --2- 172.21.15.23:0/2396581992 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50b413aeb0 0x7f50b40a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:44:27.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:27.930+0000 7f50b88ee700 1 --2- 172.21.15.23:0/2396581992 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50b413aeb0 0x7f50b40a8c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:44:28.190 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.187+0000 7f50b2ffd700 1 -- 172.21.15.23:0/2396581992 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 388998005 0 0) 0x7f50b413e600 con 0x7f50b40aaab0 2020-11-20T11:44:28.190 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.187+0000 7f50b2ffd700 1 -- 172.21.15.23:0/2396581992 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5094003280 con 0x7f50b40aaab0 2020-11-20T11:44:28.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.187+0000 7f50b2ffd700 1 -- 172.21.15.23:0/2396581992 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2656515113 0 0) 0x7f5094003280 con 0x7f50b40aaab0 2020-11-20T11:44:28.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.187+0000 7f50b2ffd700 1 -- 172.21.15.23:0/2396581992 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50b413aeb0 msgr2=0x7f50b40a8c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:44:28.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.187+0000 7f50b2ffd700 1 --2- 172.21.15.23:0/2396581992 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50b413aeb0 0x7f50b40a8c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:28.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.187+0000 7f50b2ffd700 1 -- 172.21.15.23:0/2396581992 >> v2:172.21.15.155:3300/0 conn(0x7f50b40aae90 msgr2=0x7f50b41381a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:44:28.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.187+0000 7f50b2ffd700 1 --2- 172.21.15.23:0/2396581992 >> v2:172.21.15.155:3300/0 conn(0x7f50b40aae90 0x7f50b41381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:28.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.187+0000 7f50b2ffd700 1 -- 172.21.15.23:0/2396581992 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b4138690 con 0x7f50b40aaab0 2020-11-20T11:44:28.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.188+0000 7f50b2ffd700 1 -- 172.21.15.23:0/2396581992 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f509c001520 con 0x7f50b40aaab0 2020-11-20T11:44:28.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.188+0000 7f50b2ffd700 1 -- 172.21.15.23:0/2396581992 >> v1:172.21.15.23:6789/0 conn(0x7f50b40aaab0 legacy=0x7f50b413ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:28.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.188+0000 7f50b2ffd700 1 --2- 172.21.15.23:0/2396581992 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5094003770 0x7f5094003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:28.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.188+0000 7f50b2ffd700 1 --2- 172.21.15.23:0/2396581992 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5094004060 0x7f5094004540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:28.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.188+0000 7f50b2ffd700 1 --2- 172.21.15.23:0/2396581992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5094004b40 0x7f5094004fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:28.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.188+0000 7f50b2ffd700 1 -- 172.21.15.23:0/2396581992 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b413e8a0 con 0x7f5094004b40 2020-11-20T11:44:28.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.188+0000 7f50b2ffd700 1 -- 172.21.15.23:0/2396581992 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b40ab6d0 con 0x7f5094004060 2020-11-20T11:44:28.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.188+0000 7f50b2ffd700 1 -- 172.21.15.23:0/2396581992 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f509400b8c0 con 0x7f5094003770 2020-11-20T11:44:28.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.188+0000 7f50b88ee700 1 --2- 172.21.15.23:0/2396581992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5094004b40 0x7f5094004fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:28.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.189+0000 7f50b3fff700 1 --2- 172.21.15.23:0/2396581992 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5094003770 0x7f5094003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:28.197 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.189+0000 7f50b88ee700 1 -- 172.21.15.23:0/2396581992 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5094003770 msgr2=0x7f5094003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:44:28.197 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.189+0000 7f50b88ee700 1 --2- 172.21.15.23:0/2396581992 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5094003770 0x7f5094003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:28.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.189+0000 7f50b88ee700 1 -- 172.21.15.23:0/2396581992 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5094004060 msgr2=0x7f5094004540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:44:28.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.189+0000 7f50b88ee700 1 --2- 172.21.15.23:0/2396581992 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5094004060 0x7f5094004540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:44:28.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.189+0000 7f50b88ee700 1 -- 172.21.15.23:0/2396581992 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5094003280 con 0x7f5094004b40 2020-11-20T11:44:28.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.189+0000 7f50b88ee700 1 --2- 172.21.15.23:0/2396581992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5094004b40 0x7f5094004fc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f50ac009100 tx=0x7f50ac008040).ready entity=mon.0 client_cookie=a268d0e795358d33 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:28.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.192+0000 7f50b2ffd700 1 -- 172.21.15.23:0/2396581992 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50ac010ee0 con 0x7f5094004b40 2020-11-20T11:44:28.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.193+0000 7f50b2ffd700 1 -- 172.21.15.23:0/2396581992 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50ac00c070 con 0x7f5094004b40 2020-11-20T11:44:28.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.194+0000 7f50ba351700 1 -- 172.21.15.23:0/2396581992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5094004b40 msgr2=0x7f5094004fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:28.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.194+0000 7f50ba351700 1 --2- 172.21.15.23:0/2396581992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5094004b40 0x7f5094004fc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f50ac009100 tx=0x7f50ac008040).stop 2020-11-20T11:44:28.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.194+0000 7f50b88ee700 1 -- 172.21.15.23:0/2396581992 reap_dead start 2020-11-20T11:44:28.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.194+0000 7f50ba351700 1 -- 172.21.15.23:0/2396581992 shutdown_connections 2020-11-20T11:44:28.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.194+0000 7f50ba351700 1 -- 172.21.15.23:0/2396581992 >> 172.21.15.23:0/2396581992 conn(0x7f50b40678e0 msgr2=0x7f50b4067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:44:28.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.194+0000 7f50ba351700 1 -- 172.21.15.23:0/2396581992 shutdown_connections 2020-11-20T11:44:28.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.194+0000 7f50ba351700 1 -- 172.21.15.23:0/2396581992 wait complete. 2020-11-20T11:44:28.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.196+0000 7f50ba351700 1 Processor -- start 2020-11-20T11:44:28.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.196+0000 7f50ba351700 1 -- start start 2020-11-20T11:44:28.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.197+0000 7f50ba351700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5094004b40 0x7f50b409ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:28.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.197+0000 7f50ba351700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50b4065710 0x7f50b41568d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:28.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.197+0000 7f50ba351700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50b40a45d0 0x7f50b40a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:28.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.197+0000 7f50ba351700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b4064770 con 0x7f5094004b40 2020-11-20T11:44:28.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.197+0000 7f50ba351700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b4064a10 con 0x7f50b4065710 2020-11-20T11:44:28.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.197+0000 7f50ba351700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b4064cb0 con 0x7f50b40a45d0 2020-11-20T11:44:28.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.198+0000 7f50b37fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50b4065710 0x7f50b41568d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:28.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.198+0000 7f50b37fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50b4065710 0x7f50b41568d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:38414/0 (socket says 172.21.15.23:38414) 2020-11-20T11:44:28.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.198+0000 7f50b37fe700 1 -- 172.21.15.23:0/535053769 learned_addr learned my addr 172.21.15.23:0/535053769 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:44:28.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.198+0000 7f50b3fff700 1 --2- 172.21.15.23:0/535053769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5094004b40 0x7f50b409ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:28.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.198+0000 7f50b88ee700 1 --2- 172.21.15.23:0/535053769 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50b40a45d0 0x7f50b40a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:28.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.198+0000 7f50b37fe700 1 -- 172.21.15.23:0/535053769 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50b40a45d0 msgr2=0x7f50b40a4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:28.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.198+0000 7f50b37fe700 1 --2- 172.21.15.23:0/535053769 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50b40a45d0 0x7f50b40a4a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:28.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.198+0000 7f50b37fe700 1 -- 172.21.15.23:0/535053769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5094004b40 msgr2=0x7f50b409ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:28.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.198+0000 7f50b37fe700 1 --2- 172.21.15.23:0/535053769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5094004b40 0x7f50b409ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:28.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.198+0000 7f50b37fe700 1 -- 172.21.15.23:0/535053769 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b409e2a0 con 0x7f50b4065710 2020-11-20T11:44:28.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.198+0000 7f50b37fe700 1 --2- 172.21.15.23:0/535053769 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50b4065710 0x7f50b41568d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f50a4011930 tx=0x7f50a4008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:28.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.204+0000 7f50abfff700 1 -- 172.21.15.23:0/535053769 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a4011e70 con 0x7f50b4065710 2020-11-20T11:44:28.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.204+0000 7f50ba351700 1 -- 172.21.15.23:0/535053769 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b4157770 con 0x7f50b4065710 2020-11-20T11:44:28.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.204+0000 7f50ba351700 1 -- 172.21.15.23:0/535053769 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b4156dc0 con 0x7f50b4065710 2020-11-20T11:44:28.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.205+0000 7f50abfff700 1 -- 172.21.15.23:0/535053769 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50a400f9d0 con 0x7f50b4065710 2020-11-20T11:44:28.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.205+0000 7f50abfff700 1 -- 172.21.15.23:0/535053769 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a400fd50 con 0x7f50b4065710 2020-11-20T11:44:28.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.209+0000 7f50ba351700 1 -- 172.21.15.23:0/535053769 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b405c7d0 con 0x7f50b4065710 2020-11-20T11:44:28.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.212+0000 7f50abfff700 1 -- 172.21.15.23:0/535053769 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f50a4026070 con 0x7f50b4065710 2020-11-20T11:44:28.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.213+0000 7f50abfff700 1 --2- 172.21.15.23:0/535053769 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5090056db0 0x7f5090059270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:28.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.214+0000 7f50abfff700 1 -- 172.21.15.23:0/535053769 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f50a40708b0 con 0x7f50b4065710 2020-11-20T11:44:28.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.214+0000 7f50b3fff700 1 --2- 172.21.15.23:0/535053769 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5090056db0 0x7f5090059270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:28.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.215+0000 7f50b3fff700 1 --2- 172.21.15.23:0/535053769 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5090056db0 0x7f5090059270 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f509c0045a0 tx=0x7f509c009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:28.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.219+0000 7f50abfff700 1 -- 172.21.15.23:0/535053769 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f50a4045b10 con 0x7f50b4065710 2020-11-20T11:44:28.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.749+0000 7f50ba351700 1 -- 172.21.15.23:0/535053769 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f50b4044710 con 0x7f50b4065710 2020-11-20T11:44:29.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.750+0000 7f50abfff700 1 -- 172.21.15.23:0/535053769 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1472 (secure 0 0 0) 0x7f50a4049130 con 0x7f50b4065710 2020-11-20T11:44:29.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.751+0000 7f50ba351700 1 -- 172.21.15.23:0/535053769 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5090056db0 msgr2=0x7f5090059270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:29.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.751+0000 7f50ba351700 1 --2- 172.21.15.23:0/535053769 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5090056db0 0x7f5090059270 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f509c0045a0 tx=0x7f509c009040).stop 2020-11-20T11:44:29.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.751+0000 7f50ba351700 1 -- 172.21.15.23:0/535053769 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50b4065710 msgr2=0x7f50b41568d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:29.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.751+0000 7f50ba351700 1 --2- 172.21.15.23:0/535053769 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50b4065710 0x7f50b41568d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f50a4011930 tx=0x7f50a4008040).stop 2020-11-20T11:44:29.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.751+0000 7f50ba351700 1 -- 172.21.15.23:0/535053769 shutdown_connections 2020-11-20T11:44:29.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.751+0000 7f50ba351700 1 --2- 172.21.15.23:0/535053769 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5090056db0 0x7f5090059270 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:29.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.751+0000 7f50ba351700 1 --2- 172.21.15.23:0/535053769 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50b40a45d0 0x7f50b40a4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:29.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.751+0000 7f50ba351700 1 --2- 172.21.15.23:0/535053769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5094004b40 0x7f50b409ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:29.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.751+0000 7f50ba351700 1 --2- 172.21.15.23:0/535053769 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f50b4065710 0x7f50b41568d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:29.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.751+0000 7f50ba351700 1 -- 172.21.15.23:0/535053769 >> 172.21.15.23:0/535053769 conn(0x7f50b40678e0 msgr2=0x7f50b40b1360 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:44:29.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.752+0000 7f50ba351700 1 -- 172.21.15.23:0/535053769 shutdown_connections 2020-11-20T11:44:29.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:28.752+0000 7f50ba351700 1 -- 172.21.15.23:0/535053769 wait complete. 2020-11-20T11:44:29.064 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":108,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:44:32.066 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:44:32.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:32.561+0000 7f1e3db19700 1 Processor -- start 2020-11-20T11:44:32.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:32.561+0000 7f1e3db19700 1 -- start start 2020-11-20T11:44:32.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:32.562+0000 7f1e3db19700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1e380ad7c0 0x7f1e380a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:32.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:32.562+0000 7f1e3db19700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e380a67d0 0x7f1e38061f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:32.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:32.562+0000 7f1e3db19700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e380a3cc0 con 0x7f1e38063fe0 2020-11-20T11:44:32.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:32.562+0000 7f1e3db19700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e380a6c30 con 0x7f1e380ad7c0 2020-11-20T11:44:32.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:32.562+0000 7f1e3db19700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e380a4640 con 0x7f1e380a67d0 2020-11-20T11:44:32.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:32.564+0000 7f1e377fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1e380ad7c0 0x7f1e380a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:32.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:32.564+0000 7f1e377fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1e380ad7c0 0x7f1e380a62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45526/0 (socket says 172.21.15.23:45526) 2020-11-20T11:44:32.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:32.564+0000 7f1e377fe700 1 -- 172.21.15.23:0/4055738301 learned_addr learned my addr 172.21.15.23:0/4055738301 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:44:32.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:32.564+0000 7f1e377fe700 1 -- 172.21.15.23:0/4055738301 >> v2:172.21.15.23:3300/0 conn(0x7f1e380ad7c0 msgr2=0x7f1e380a62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:44:32.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:32.564+0000 7f1e377fe700 1 -- 172.21.15.23:0/4055738301 >> v2:172.21.15.23:3300/0 conn(0x7f1e380ad7c0 msgr2=0x7f1e380a62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:44:32.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:32.564+0000 7f1e377fe700 1 --2- 172.21.15.23:0/4055738301 >> v2:172.21.15.23:3300/0 conn(0x7f1e380ad7c0 0x7f1e380a62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:44:32.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:32.564+0000 7f1e377fe700 1 --2- 172.21.15.23:0/4055738301 >> v2:172.21.15.23:3300/0 conn(0x7f1e380ad7c0 0x7f1e380a62e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:44:32.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:32.565+0000 7f1e37fff700 1 --2- 172.21.15.23:0/4055738301 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e380a67d0 0x7f1e38061f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:32.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:32.565+0000 7f1e37fff700 1 -- 172.21.15.23:0/4055738301 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e380a67d0 msgr2=0x7f1e38061f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:44:32.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:32.565+0000 7f1e37fff700 1 -- 172.21.15.23:0/4055738301 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e380a67d0 msgr2=0x7f1e38061f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:44:32.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:32.565+0000 7f1e37fff700 1 --2- 172.21.15.23:0/4055738301 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e380a67d0 0x7f1e38061f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:44:32.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:32.565+0000 7f1e37fff700 1 --2- 172.21.15.23:0/4055738301 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e380a67d0 0x7f1e38061f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:44:32.773 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:32.770+0000 7f1e377fe700 1 --2- 172.21.15.23:0/4055738301 >> v2:172.21.15.23:3300/0 conn(0x7f1e380ad7c0 0x7f1e380a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:32.773 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:32.770+0000 7f1e377fe700 1 -- 172.21.15.23:0/4055738301 >> v2:172.21.15.23:3300/0 conn(0x7f1e380ad7c0 msgr2=0x7f1e380a62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:44:32.774 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:32.770+0000 7f1e377fe700 1 -- 172.21.15.23:0/4055738301 >> v2:172.21.15.23:3300/0 conn(0x7f1e380ad7c0 msgr2=0x7f1e380a62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:44:32.774 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:32.770+0000 7f1e377fe700 1 --2- 172.21.15.23:0/4055738301 >> v2:172.21.15.23:3300/0 conn(0x7f1e380ad7c0 0x7f1e380a62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:44:32.774 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:32.770+0000 7f1e377fe700 1 --2- 172.21.15.23:0/4055738301 >> v2:172.21.15.23:3300/0 conn(0x7f1e380ad7c0 0x7f1e380a62e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:44:32.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:32.771+0000 7f1e37fff700 1 --2- 172.21.15.23:0/4055738301 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e380a67d0 0x7f1e38061f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:32.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:32.771+0000 7f1e37fff700 1 -- 172.21.15.23:0/4055738301 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e380a67d0 msgr2=0x7f1e38061f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:44:32.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:32.771+0000 7f1e37fff700 1 -- 172.21.15.23:0/4055738301 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e380a67d0 msgr2=0x7f1e38061f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:44:32.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:32.771+0000 7f1e37fff700 1 --2- 172.21.15.23:0/4055738301 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e380a67d0 0x7f1e38061f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:44:32.776 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:32.771+0000 7f1e37fff700 1 --2- 172.21.15.23:0/4055738301 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e380a67d0 0x7f1e38061f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:44:33.174 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:33.171+0000 7f1e377fe700 1 --2- 172.21.15.23:0/4055738301 >> v2:172.21.15.23:3300/0 conn(0x7f1e380ad7c0 0x7f1e380a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:33.174 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:33.171+0000 7f1e377fe700 1 -- 172.21.15.23:0/4055738301 >> v2:172.21.15.23:3300/0 conn(0x7f1e380ad7c0 msgr2=0x7f1e380a62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:44:33.175 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:33.171+0000 7f1e377fe700 1 -- 172.21.15.23:0/4055738301 >> v2:172.21.15.23:3300/0 conn(0x7f1e380ad7c0 msgr2=0x7f1e380a62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:44:33.175 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:33.171+0000 7f1e377fe700 1 --2- 172.21.15.23:0/4055738301 >> v2:172.21.15.23:3300/0 conn(0x7f1e380ad7c0 0x7f1e380a62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:44:33.175 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:33.171+0000 7f1e377fe700 1 --2- 172.21.15.23:0/4055738301 >> v2:172.21.15.23:3300/0 conn(0x7f1e380ad7c0 0x7f1e380a62e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:44:33.176 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:33.172+0000 7f1e37fff700 1 --2- 172.21.15.23:0/4055738301 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e380a67d0 0x7f1e38061f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:33.176 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:33.172+0000 7f1e37fff700 1 -- 172.21.15.23:0/4055738301 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e380a67d0 msgr2=0x7f1e38061f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:44:33.176 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:33.172+0000 7f1e37fff700 1 -- 172.21.15.23:0/4055738301 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e380a67d0 msgr2=0x7f1e38061f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:44:33.176 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:33.172+0000 7f1e37fff700 1 --2- 172.21.15.23:0/4055738301 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e380a67d0 0x7f1e38061f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:44:33.177 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:33.172+0000 7f1e37fff700 1 --2- 172.21.15.23:0/4055738301 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e380a67d0 0x7f1e38061f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:44:33.975 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:33.972+0000 7f1e377fe700 1 --2- 172.21.15.23:0/4055738301 >> v2:172.21.15.23:3300/0 conn(0x7f1e380ad7c0 0x7f1e380a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:33.976 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:33.972+0000 7f1e377fe700 1 -- 172.21.15.23:0/4055738301 >> v2:172.21.15.23:3300/0 conn(0x7f1e380ad7c0 msgr2=0x7f1e380a62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:44:33.976 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:33.972+0000 7f1e377fe700 1 -- 172.21.15.23:0/4055738301 >> v2:172.21.15.23:3300/0 conn(0x7f1e380ad7c0 msgr2=0x7f1e380a62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:44:33.976 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:33.972+0000 7f1e377fe700 1 --2- 172.21.15.23:0/4055738301 >> v2:172.21.15.23:3300/0 conn(0x7f1e380ad7c0 0x7f1e380a62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:44:33.976 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:33.972+0000 7f1e377fe700 1 --2- 172.21.15.23:0/4055738301 >> v2:172.21.15.23:3300/0 conn(0x7f1e380ad7c0 0x7f1e380a62e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:44:33.977 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:33.973+0000 7f1e37fff700 1 --2- 172.21.15.23:0/4055738301 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e380a67d0 0x7f1e38061f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:33.977 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:33.973+0000 7f1e37fff700 1 -- 172.21.15.23:0/4055738301 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e380a67d0 msgr2=0x7f1e38061f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:44:33.978 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:33.973+0000 7f1e37fff700 1 -- 172.21.15.23:0/4055738301 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e380a67d0 msgr2=0x7f1e38061f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:44:33.978 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:33.973+0000 7f1e37fff700 1 --2- 172.21.15.23:0/4055738301 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e380a67d0 0x7f1e38061f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:44:33.978 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:33.973+0000 7f1e37fff700 1 --2- 172.21.15.23:0/4055738301 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e380a67d0 0x7f1e38061f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:44:34.181 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.178+0000 7f1e367fc700 1 -- 172.21.15.23:0/4055738301 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3564956706 0 0) 0x7f1e380a3cc0 con 0x7f1e38063fe0 2020-11-20T11:44:34.181 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.178+0000 7f1e367fc700 1 -- 172.21.15.23:0/4055738301 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e24003280 con 0x7f1e38063fe0 2020-11-20T11:44:34.184 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.181+0000 7f1e367fc700 1 -- 172.21.15.23:0/4055738301 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 702194853 0 0) 0x7f1e24003280 con 0x7f1e38063fe0 2020-11-20T11:44:34.184 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.181+0000 7f1e367fc700 1 -- 172.21.15.23:0/4055738301 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e380a67d0 msgr2=0x7f1e38061f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:44:34.184 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.181+0000 7f1e367fc700 1 --2- 172.21.15.23:0/4055738301 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e380a67d0 0x7f1e38061f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:34.185 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.181+0000 7f1e367fc700 1 -- 172.21.15.23:0/4055738301 >> v2:172.21.15.23:3300/0 conn(0x7f1e380ad7c0 msgr2=0x7f1e380a62e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:44:34.185 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.181+0000 7f1e367fc700 1 --2- 172.21.15.23:0/4055738301 >> v2:172.21.15.23:3300/0 conn(0x7f1e380ad7c0 0x7f1e380a62e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:34.185 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.181+0000 7f1e367fc700 1 -- 172.21.15.23:0/4055738301 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e380a5ba0 con 0x7f1e38063fe0 2020-11-20T11:44:34.185 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.181+0000 7f1e367fc700 1 -- 172.21.15.23:0/4055738301 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f1e2c001520 con 0x7f1e38063fe0 2020-11-20T11:44:34.186 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.181+0000 7f1e367fc700 1 -- 172.21.15.23:0/4055738301 >> v1:172.21.15.23:6789/0 conn(0x7f1e38063fe0 legacy=0x7f1e38061850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:34.186 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.181+0000 7f1e367fc700 1 --2- 172.21.15.23:0/4055738301 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e240037f0 0x7f1e24003c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:34.186 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.181+0000 7f1e367fc700 1 --2- 172.21.15.23:0/4055738301 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e380a67d0 0x7f1e24004520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:34.187 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.181+0000 7f1e367fc700 1 --2- 172.21.15.23:0/4055738301 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e24004a10 0x7f1e24004e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:34.187 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.181+0000 7f1e367fc700 1 -- 172.21.15.23:0/4055738301 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e380a6c30 con 0x7f1e240037f0 2020-11-20T11:44:34.187 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.181+0000 7f1e367fc700 1 -- 172.21.15.23:0/4055738301 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e380a4640 con 0x7f1e24004a10 2020-11-20T11:44:34.187 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.181+0000 7f1e367fc700 1 -- 172.21.15.23:0/4055738301 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e24004290 con 0x7f1e380a67d0 2020-11-20T11:44:34.187 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.182+0000 7f1e37fff700 1 --2- 172.21.15.23:0/4055738301 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e24004a10 0x7f1e24004e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:34.188 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.182+0000 7f1e36ffd700 1 --2- 172.21.15.23:0/4055738301 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e380a67d0 0x7f1e24004520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:34.188 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.182+0000 7f1e36ffd700 1 -- 172.21.15.23:0/4055738301 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e24004a10 msgr2=0x7f1e24004e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:44:34.188 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.182+0000 7f1e36ffd700 1 --2- 172.21.15.23:0/4055738301 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e24004a10 0x7f1e24004e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:34.188 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.182+0000 7f1e36ffd700 1 -- 172.21.15.23:0/4055738301 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e240037f0 msgr2=0x7f1e24003c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:44:34.189 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.182+0000 7f1e36ffd700 1 --2- 172.21.15.23:0/4055738301 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e240037f0 0x7f1e24003c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:44:34.189 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.182+0000 7f1e36ffd700 1 -- 172.21.15.23:0/4055738301 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e380a5ba0 con 0x7f1e380a67d0 2020-11-20T11:44:34.190 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.183+0000 7f1e36ffd700 1 --2- 172.21.15.23:0/4055738301 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e380a67d0 0x7f1e24004520 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f1e2c00c490 tx=0x7f1e2c007750).ready entity=mon.2 client_cookie=4bf201364a94501b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:34.190 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.183+0000 7f1e367fc700 1 -- 172.21.15.23:0/4055738301 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2c012400 con 0x7f1e380a67d0 2020-11-20T11:44:34.190 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.183+0000 7f1e367fc700 1 -- 172.21.15.23:0/4055738301 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e2c012580 con 0x7f1e380a67d0 2020-11-20T11:44:34.190 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.183+0000 7f1e367fc700 1 -- 172.21.15.23:0/4055738301 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2c012940 con 0x7f1e380a67d0 2020-11-20T11:44:34.190 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.184+0000 7f1e3db19700 1 -- 172.21.15.23:0/4055738301 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e380a67d0 msgr2=0x7f1e24004520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:34.191 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.184+0000 7f1e3db19700 1 --2- 172.21.15.23:0/4055738301 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e380a67d0 0x7f1e24004520 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f1e2c00c490 tx=0x7f1e2c007750).stop 2020-11-20T11:44:34.191 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.186+0000 7f1e37fff700 1 -- 172.21.15.23:0/4055738301 reap_dead start 2020-11-20T11:44:34.191 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.186+0000 7f1e3db19700 1 -- 172.21.15.23:0/4055738301 shutdown_connections 2020-11-20T11:44:34.191 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.186+0000 7f1e3db19700 1 -- 172.21.15.23:0/4055738301 >> 172.21.15.23:0/4055738301 conn(0x7f1e38060c80 msgr2=0x7f1e380acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:44:34.191 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.187+0000 7f1e3db19700 1 -- 172.21.15.23:0/4055738301 shutdown_connections 2020-11-20T11:44:34.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.187+0000 7f1e3db19700 1 -- 172.21.15.23:0/4055738301 wait complete. 2020-11-20T11:44:34.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.188+0000 7f1e3db19700 1 Processor -- start 2020-11-20T11:44:34.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.189+0000 7f1e3db19700 1 -- start start 2020-11-20T11:44:34.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.190+0000 7f1e3db19700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e380a67d0 0x7f1e38067c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:34.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.190+0000 7f1e3db19700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e381301f0 0x7f1e3809eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:34.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.190+0000 7f1e3db19700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e3809f3a0 0x7f1e38064ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:34.194 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.190+0000 7f1e3db19700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3809d5f0 con 0x7f1e381301f0 2020-11-20T11:44:34.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.190+0000 7f1e3db19700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e38067440 con 0x7f1e380a67d0 2020-11-20T11:44:34.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.190+0000 7f1e3db19700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e380676e0 con 0x7f1e3809f3a0 2020-11-20T11:44:34.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.190+0000 7f1e377fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e380a67d0 0x7f1e38067c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:34.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.190+0000 7f1e377fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e380a67d0 0x7f1e38067c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:38446/0 (socket says 172.21.15.23:38446) 2020-11-20T11:44:34.195 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.190+0000 7f1e377fe700 1 -- 172.21.15.23:0/3432919659 learned_addr learned my addr 172.21.15.23:0/3432919659 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:44:34.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.191+0000 7f1e377fe700 1 -- 172.21.15.23:0/3432919659 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e3809f3a0 msgr2=0x7f1e38064ef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:44:34.197 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.191+0000 7f1e377fe700 1 --2- 172.21.15.23:0/3432919659 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e3809f3a0 0x7f1e38064ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:44:34.197 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.191+0000 7f1e377fe700 1 -- 172.21.15.23:0/3432919659 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e381301f0 msgr2=0x7f1e3809eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:34.197 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.191+0000 7f1e377fe700 1 --2- 172.21.15.23:0/3432919659 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e381301f0 0x7f1e3809eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:44:34.197 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.191+0000 7f1e377fe700 1 -- 172.21.15.23:0/3432919659 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e3814abc0 con 0x7f1e380a67d0 2020-11-20T11:44:34.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.191+0000 7f1e377fe700 1 --2- 172.21.15.23:0/3432919659 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e380a67d0 0x7f1e38067c00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f1e30013d90 tx=0x7f1e30008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:34.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.193+0000 7f1e1f7fe700 1 -- 172.21.15.23:0/3432919659 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e30011cf0 con 0x7f1e380a67d0 2020-11-20T11:44:34.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.193+0000 7f1e3db19700 1 -- 172.21.15.23:0/3432919659 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e380a3450 con 0x7f1e380a67d0 2020-11-20T11:44:34.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.193+0000 7f1e3db19700 1 -- 172.21.15.23:0/3432919659 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e38130670 con 0x7f1e380a67d0 2020-11-20T11:44:34.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.193+0000 7f1e1f7fe700 1 -- 172.21.15.23:0/3432919659 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e30011e50 con 0x7f1e380a67d0 2020-11-20T11:44:34.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.193+0000 7f1e1f7fe700 1 -- 172.21.15.23:0/3432919659 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e30016690 con 0x7f1e380a67d0 2020-11-20T11:44:34.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.194+0000 7f1e3db19700 1 -- 172.21.15.23:0/3432919659 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e14004fd0 con 0x7f1e380a67d0 2020-11-20T11:44:34.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.195+0000 7f1e1f7fe700 1 -- 172.21.15.23:0/3432919659 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1e30026070 con 0x7f1e380a67d0 2020-11-20T11:44:34.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.195+0000 7f1e1f7fe700 1 --2- 172.21.15.23:0/3432919659 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1e18056de0 0x7f1e180592a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:34.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.195+0000 7f1e1f7fe700 1 -- 172.21.15.23:0/3432919659 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f1e30070cc0 con 0x7f1e380a67d0 2020-11-20T11:44:34.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.195+0000 7f1e36ffd700 1 --2- 172.21.15.23:0/3432919659 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1e18056de0 0x7f1e180592a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:34.201 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.196+0000 7f1e36ffd700 1 --2- 172.21.15.23:0/3432919659 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1e18056de0 0x7f1e180592a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f1e3809e9b0 tx=0x7f1e2c017000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:34.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.204+0000 7f1e1f7fe700 1 -- 172.21.15.23:0/3432919659 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1e30046030 con 0x7f1e380a67d0 2020-11-20T11:44:34.656 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.652+0000 7f1e3db19700 1 -- 172.21.15.23:0/3432919659 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1e14001670 con 0x7f1e380a67d0 2020-11-20T11:44:34.657 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.652+0000 7f1e1f7fe700 1 -- 172.21.15.23:0/3432919659 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1472 (secure 0 0 0) 0x7f1e30049650 con 0x7f1e380a67d0 2020-11-20T11:44:34.657 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":112,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:44:34.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.655+0000 7f1e3db19700 1 -- 172.21.15.23:0/3432919659 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1e18056de0 msgr2=0x7f1e180592a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:34.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.655+0000 7f1e3db19700 1 --2- 172.21.15.23:0/3432919659 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1e18056de0 0x7f1e180592a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f1e3809e9b0 tx=0x7f1e2c017000).stop 2020-11-20T11:44:34.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.655+0000 7f1e3db19700 1 -- 172.21.15.23:0/3432919659 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e380a67d0 msgr2=0x7f1e38067c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:34.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.655+0000 7f1e3db19700 1 --2- 172.21.15.23:0/3432919659 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e380a67d0 0x7f1e38067c00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f1e30013d90 tx=0x7f1e30008040).stop 2020-11-20T11:44:34.660 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.655+0000 7f1e3db19700 1 -- 172.21.15.23:0/3432919659 shutdown_connections 2020-11-20T11:44:34.661 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.655+0000 7f1e3db19700 1 --2- 172.21.15.23:0/3432919659 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1e18056de0 0x7f1e180592a0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:34.661 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.655+0000 7f1e3db19700 1 --2- 172.21.15.23:0/3432919659 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1e3809f3a0 0x7f1e38064ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:44:34.661 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.655+0000 7f1e3db19700 1 --2- 172.21.15.23:0/3432919659 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1e380a67d0 0x7f1e38067c00 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:34.661 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.655+0000 7f1e3db19700 1 --2- 172.21.15.23:0/3432919659 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e381301f0 0x7f1e3809eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:44:34.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.655+0000 7f1e3db19700 1 -- 172.21.15.23:0/3432919659 >> 172.21.15.23:0/3432919659 conn(0x7f1e38060c80 msgr2=0x7f1e38061fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:44:34.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.655+0000 7f1e3db19700 1 -- 172.21.15.23:0/3432919659 shutdown_connections 2020-11-20T11:44:34.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:34.655+0000 7f1e3db19700 1 -- 172.21.15.23:0/3432919659 wait complete. 2020-11-20T11:44:37.676 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:44:37.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.869+0000 7f282d7a2700 1 Processor -- start 2020-11-20T11:44:37.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.870+0000 7f282d7a2700 1 -- start start 2020-11-20T11:44:37.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.870+0000 7f282d7a2700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2828065e50 0x7f28280aad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:37.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.870+0000 7f282d7a2700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f28280a90d0 0x7f28280ab220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:37.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.870+0000 7f282d7a2700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2828138360 con 0x7f28280676a0 2020-11-20T11:44:37.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.870+0000 7f282d7a2700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2828067d10 con 0x7f28280a90d0 2020-11-20T11:44:37.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.870+0000 7f282d7a2700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2828066700 con 0x7f2828065e50 2020-11-20T11:44:37.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.870+0000 7f282cfa1700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f28280a90d0 0x7f28280ab220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:37.882 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.870+0000 7f28277fe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2828065e50 0x7f28280aad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:37.882 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.870+0000 7f2827fff700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f28280676a0 0x7f28280a89c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:37782/0 (socket says 172.21.15.23:37782) 2020-11-20T11:44:37.882 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.871+0000 7f28277fe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2828065e50 0x7f28280aad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:55798/0 (socket says 172.21.15.23:55798) 2020-11-20T11:44:37.882 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.871+0000 7f282cfa1700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f28280a90d0 0x7f28280ab220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:38458/0 (socket says 172.21.15.23:38458) 2020-11-20T11:44:37.882 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.871+0000 7f2827fff700 1 -- 172.21.15.23:0/3260071965 learned_addr learned my addr 172.21.15.23:0/3260071965 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:44:37.883 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.871+0000 7f28277fe700 1 -- 172.21.15.23:0/3260071965 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2828065e50 msgr2=0x7f28280aad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:44:37.883 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.871+0000 7f28277fe700 1 -- 172.21.15.23:0/3260071965 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2828065e50 msgr2=0x7f28280aad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:44:37.883 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.871+0000 7f28277fe700 1 --2- 172.21.15.23:0/3260071965 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2828065e50 0x7f28280aad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:44:37.883 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.871+0000 7f28277fe700 1 --2- 172.21.15.23:0/3260071965 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2828065e50 0x7f28280aad30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:44:37.884 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.871+0000 7f282cfa1700 1 -- 172.21.15.23:0/3260071965 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2828065e50 msgr2=0x7f28280aad30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:44:37.884 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.871+0000 7f282cfa1700 1 --2- 172.21.15.23:0/3260071965 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2828065e50 0x7f28280aad30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:37.884 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.871+0000 7f282cfa1700 1 -- 172.21.15.23:0/3260071965 >> v1:172.21.15.155:6789/0 conn(0x7f28280676a0 legacy=0x7f28280a89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:37.884 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.871+0000 7f282cfa1700 1 -- 172.21.15.23:0/3260071965 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2828138600 con 0x7f28280a90d0 2020-11-20T11:44:37.885 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.871+0000 7f2826ffd700 1 -- 172.21.15.23:0/3260071965 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1912585286 0 0) 0x7f2828138360 con 0x7f28280676a0 2020-11-20T11:44:37.885 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.871+0000 7f282cfa1700 1 --2- 172.21.15.23:0/3260071965 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28280a90d0 0x7f28280ab220 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f2820009ce0 tx=0x7f2820009220).ready entity=mon.1 client_cookie=549cfa2d4966f70b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:37.886 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.871+0000 7f2826ffd700 1 -- 172.21.15.23:0/3260071965 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282000de00 con 0x7f28280a90d0 2020-11-20T11:44:37.886 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.871+0000 7f2826ffd700 1 -- 172.21.15.23:0/3260071965 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2820010470 con 0x7f28280a90d0 2020-11-20T11:44:37.886 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.871+0000 7f2826ffd700 1 -- 172.21.15.23:0/3260071965 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28200107f0 con 0x7f28280a90d0 2020-11-20T11:44:37.887 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.872+0000 7f282d7a2700 1 -- 172.21.15.23:0/3260071965 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28280a90d0 msgr2=0x7f28280ab220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:37.887 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.872+0000 7f282d7a2700 1 --2- 172.21.15.23:0/3260071965 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28280a90d0 0x7f28280ab220 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f2820009ce0 tx=0x7f2820009220).stop 2020-11-20T11:44:37.888 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.872+0000 7f282d7a2700 1 -- 172.21.15.23:0/3260071965 shutdown_connections 2020-11-20T11:44:37.888 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.872+0000 7f282d7a2700 1 --2- 172.21.15.23:0/3260071965 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28280a90d0 0x7f28280ab220 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:37.888 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.872+0000 7f282d7a2700 1 --2- 172.21.15.23:0/3260071965 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f2828065e50 0x7f28280aad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:37.889 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.872+0000 7f282d7a2700 1 -- 172.21.15.23:0/3260071965 >> 172.21.15.23:0/3260071965 conn(0x7f28280612c0 msgr2=0x7f28280616d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:44:37.889 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.872+0000 7f282d7a2700 1 -- 172.21.15.23:0/3260071965 shutdown_connections 2020-11-20T11:44:37.889 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.872+0000 7f282d7a2700 1 -- 172.21.15.23:0/3260071965 wait complete. 2020-11-20T11:44:37.890 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.873+0000 7f282d7a2700 1 Processor -- start 2020-11-20T11:44:37.890 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.873+0000 7f282d7a2700 1 -- start start 2020-11-20T11:44:37.891 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.873+0000 7f282d7a2700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2828065e50 0x7f282813b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:37.891 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.873+0000 7f282d7a2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28280676a0 0x7f28280ad0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:37.891 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.873+0000 7f282d7a2700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28280a90d0 0x7f282813a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:37.892 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.873+0000 7f282d7a2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2828149070 con 0x7f28280676a0 2020-11-20T11:44:37.892 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.873+0000 7f282d7a2700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f282813bdb0 con 0x7f2828065e50 2020-11-20T11:44:37.893 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.873+0000 7f282d7a2700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f282813d100 con 0x7f28280a90d0 2020-11-20T11:44:37.893 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.874+0000 7f28277fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28280676a0 0x7f28280ad0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:37.893 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.874+0000 7f28277fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28280676a0 0x7f28280ad0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45566/0 (socket says 172.21.15.23:45566) 2020-11-20T11:44:37.894 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.874+0000 7f28277fe700 1 -- 172.21.15.23:0/340892108 learned_addr learned my addr 172.21.15.23:0/340892108 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:44:37.894 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.874+0000 7f2827fff700 1 --2- 172.21.15.23:0/340892108 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2828065e50 0x7f282813b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:37.895 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.874+0000 7f282cfa1700 1 --2- 172.21.15.23:0/340892108 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28280a90d0 0x7f282813a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:37.895 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.874+0000 7f282cfa1700 1 -- 172.21.15.23:0/340892108 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28280a90d0 msgr2=0x7f282813a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2020-11-20T11:44:37.895 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.874+0000 7f282cfa1700 1 -- 172.21.15.23:0/340892108 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28280a90d0 msgr2=0x7f282813a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:44:37.896 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.874+0000 7f282cfa1700 1 --2- 172.21.15.23:0/340892108 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28280a90d0 0x7f282813a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:44:37.896 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.874+0000 7f282cfa1700 1 --2- 172.21.15.23:0/340892108 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28280a90d0 0x7f282813a3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:44:37.896 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.874+0000 7f2827fff700 1 -- 172.21.15.23:0/340892108 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28280a90d0 msgr2=0x7f282813a3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:44:37.897 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.874+0000 7f2827fff700 1 --2- 172.21.15.23:0/340892108 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28280a90d0 0x7f282813a3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:37.897 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.874+0000 7f2827fff700 1 -- 172.21.15.23:0/340892108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28280676a0 msgr2=0x7f28280ad0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:37.897 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.874+0000 7f2827fff700 1 --2- 172.21.15.23:0/340892108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28280676a0 0x7f28280ad0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:37.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.874+0000 7f2827fff700 1 -- 172.21.15.23:0/340892108 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f282813d3a0 con 0x7f2828065e50 2020-11-20T11:44:37.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.874+0000 7f28277fe700 1 --2- 172.21.15.23:0/340892108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28280676a0 0x7f28280ad0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:44:37.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.874+0000 7f2827fff700 1 --2- 172.21.15.23:0/340892108 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2828065e50 0x7f282813b8c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f281000c710 tx=0x7f2810007a40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:37.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.874+0000 7f281ffff700 1 -- 172.21.15.23:0/340892108 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28100072e0 con 0x7f2828065e50 2020-11-20T11:44:37.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.874+0000 7f281ffff700 1 -- 172.21.15.23:0/340892108 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2810007440 con 0x7f2828065e50 2020-11-20T11:44:37.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.874+0000 7f281ffff700 1 -- 172.21.15.23:0/340892108 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2810005860 con 0x7f2828065e50 2020-11-20T11:44:37.900 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.874+0000 7f282d7a2700 1 -- 172.21.15.23:0/340892108 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f282813d640 con 0x7f2828065e50 2020-11-20T11:44:37.900 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.874+0000 7f282d7a2700 1 -- 172.21.15.23:0/340892108 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f282813a8d0 con 0x7f2828065e50 2020-11-20T11:44:37.900 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.875+0000 7f282d7a2700 1 -- 172.21.15.23:0/340892108 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f282813a140 con 0x7f2828065e50 2020-11-20T11:44:37.901 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.876+0000 7f281ffff700 1 -- 172.21.15.23:0/340892108 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2810010070 con 0x7f2828065e50 2020-11-20T11:44:37.901 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.876+0000 7f281ffff700 1 --2- 172.21.15.23:0/340892108 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2804056d60 0x7f2804059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:37.902 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.876+0000 7f281ffff700 1 -- 172.21.15.23:0/340892108 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f281006f730 con 0x7f2828065e50 2020-11-20T11:44:37.902 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.876+0000 7f28277fe700 1 --2- 172.21.15.23:0/340892108 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2804056d60 0x7f2804059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:37.903 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.876+0000 7f28277fe700 1 --2- 172.21.15.23:0/340892108 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2804056d60 0x7f2804059220 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f2828137820 tx=0x7f2818008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:37.903 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:37.880+0000 7f281ffff700 1 -- 172.21.15.23:0/340892108 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f282813a140 con 0x7f2828065e50 2020-11-20T11:44:38.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:38.289+0000 7f282d7a2700 1 -- 172.21.15.23:0/340892108 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2828063430 con 0x7f2828065e50 2020-11-20T11:44:38.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:38.290+0000 7f281ffff700 1 -- 172.21.15.23:0/340892108 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1472 (secure 0 0 0) 0x7f2810044a20 con 0x7f2828065e50 2020-11-20T11:44:38.304 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":112,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:44:38.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:38.306+0000 7f282d7a2700 1 -- 172.21.15.23:0/340892108 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2804056d60 msgr2=0x7f2804059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:38.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:38.306+0000 7f282d7a2700 1 --2- 172.21.15.23:0/340892108 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2804056d60 0x7f2804059220 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f2828137820 tx=0x7f2818008040).stop 2020-11-20T11:44:38.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:38.306+0000 7f282d7a2700 1 -- 172.21.15.23:0/340892108 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2828065e50 msgr2=0x7f282813b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:38.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:38.306+0000 7f282d7a2700 1 --2- 172.21.15.23:0/340892108 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2828065e50 0x7f282813b8c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f281000c710 tx=0x7f2810007a40).stop 2020-11-20T11:44:38.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:38.306+0000 7f282d7a2700 1 -- 172.21.15.23:0/340892108 shutdown_connections 2020-11-20T11:44:38.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:38.306+0000 7f282d7a2700 1 --2- 172.21.15.23:0/340892108 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f2804056d60 0x7f2804059220 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:38.313 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:38.306+0000 7f282d7a2700 1 --2- 172.21.15.23:0/340892108 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f28280a90d0 0x7f282813a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:38.313 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:38.306+0000 7f282d7a2700 1 --2- 172.21.15.23:0/340892108 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f2828065e50 0x7f282813b8c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:38.314 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:38.306+0000 7f282d7a2700 1 --2- 172.21.15.23:0/340892108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28280676a0 0x7f28280ad0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:38.314 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:38.306+0000 7f282d7a2700 1 -- 172.21.15.23:0/340892108 >> 172.21.15.23:0/340892108 conn(0x7f28280612c0 msgr2=0x7f28280616a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:44:38.314 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:38.306+0000 7f282d7a2700 1 -- 172.21.15.23:0/340892108 shutdown_connections 2020-11-20T11:44:38.315 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:38.306+0000 7f282d7a2700 1 -- 172.21.15.23:0/340892108 wait complete. 2020-11-20T11:44:41.391 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:44:41.736 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.733+0000 7fce90f0d700 1 Processor -- start 2020-11-20T11:44:41.737 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.733+0000 7fce90f0d700 1 -- start start 2020-11-20T11:44:41.737 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.734+0000 7fce90f0d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fce8c061a10 0x7fce8c0a6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:41.738 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.734+0000 7fce90f0d700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce8c0a4970 con 0x7fce8c13f800 2020-11-20T11:44:41.738 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.734+0000 7fce90f0d700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce8c0a60e0 con 0x7fce8c136ea0 2020-11-20T11:44:41.738 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.734+0000 7fce90f0d700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce8c0a5ef0 con 0x7fce8c061a10 2020-11-20T11:44:41.738 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.734+0000 7fce8a59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fce8c061a10 0x7fce8c0a6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:41.738 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.734+0000 7fce8a59c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fce8c061a10 0x7fce8c0a6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45600/0 (socket says 172.21.15.23:45600) 2020-11-20T11:44:41.739 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.734+0000 7fce8a59c700 1 -- 172.21.15.23:0/3949442168 learned_addr learned my addr 172.21.15.23:0/3949442168 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:44:41.740 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.735+0000 7fce8a59c700 1 -- 172.21.15.23:0/3949442168 >> v1:172.21.15.155:6789/0 conn(0x7fce8c136ea0 legacy=0x7fce8c0a8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:41.741 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.735+0000 7fce8a59c700 1 -- 172.21.15.23:0/3949442168 >> v1:172.21.15.23:6789/0 conn(0x7fce8c13f800 legacy=0x7fce8c141bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:44:41.741 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.735+0000 7fce8a59c700 1 -- 172.21.15.23:0/3949442168 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce8c061df0 con 0x7fce8c061a10 2020-11-20T11:44:41.741 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.735+0000 7fce8959a700 1 -- 172.21.15.23:0/3949442168 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3508918306 0 0) 0x7fce8c0a60e0 con 0x7fce8c136ea0 2020-11-20T11:44:41.741 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.735+0000 7fce8a59c700 1 --2- 172.21.15.23:0/3949442168 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce8c061a10 0x7fce8c0a6580 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fce7c0089a0 tx=0x7fce7c00d3e0).ready entity=mon.0 client_cookie=d17e93b3542ef3c4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:41.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.735+0000 7fce8959a700 1 -- 172.21.15.23:0/3949442168 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7c010070 con 0x7fce8c061a10 2020-11-20T11:44:41.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.735+0000 7fce8959a700 1 -- 172.21.15.23:0/3949442168 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7c000e90 con 0x7fce8c061a10 2020-11-20T11:44:41.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.735+0000 7fce8959a700 1 -- 172.21.15.23:0/3949442168 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7c015470 con 0x7fce8c061a10 2020-11-20T11:44:41.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.736+0000 7fce90f0d700 1 -- 172.21.15.23:0/3949442168 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce8c061a10 msgr2=0x7fce8c0a6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:41.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.736+0000 7fce90f0d700 1 --2- 172.21.15.23:0/3949442168 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce8c061a10 0x7fce8c0a6580 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fce7c0089a0 tx=0x7fce7c00d3e0).stop 2020-11-20T11:44:41.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.736+0000 7fce90f0d700 1 -- 172.21.15.23:0/3949442168 shutdown_connections 2020-11-20T11:44:41.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.736+0000 7fce90f0d700 1 --2- 172.21.15.23:0/3949442168 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce8c061a10 0x7fce8c0a6580 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:41.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.736+0000 7fce90f0d700 1 -- 172.21.15.23:0/3949442168 >> 172.21.15.23:0/3949442168 conn(0x7fce8c060c80 msgr2=0x7fce8c0acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:44:41.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.736+0000 7fce90f0d700 1 -- 172.21.15.23:0/3949442168 shutdown_connections 2020-11-20T11:44:41.744 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.736+0000 7fce90f0d700 1 -- 172.21.15.23:0/3949442168 wait complete. 2020-11-20T11:44:41.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.738+0000 7fce90f0d700 1 Processor -- start 2020-11-20T11:44:41.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.738+0000 7fce90f0d700 1 -- start start 2020-11-20T11:44:41.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.739+0000 7fce90f0d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fce8c061a10 0x7fce8c142dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:41.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.739+0000 7fce90f0d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce8c136ea0 0x7fce8c14eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:41.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.739+0000 7fce90f0d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fce8c13f800 0x7fce8c133e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:41.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.739+0000 7fce90f0d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8c14f9e0 con 0x7fce8c136ea0 2020-11-20T11:44:41.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.739+0000 7fce90f0d700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8c14f4a0 con 0x7fce8c061a10 2020-11-20T11:44:41.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.739+0000 7fce90f0d700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce8c14f740 con 0x7fce8c13f800 2020-11-20T11:44:41.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.739+0000 7fce89d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce8c136ea0 0x7fce8c14eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:41.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.739+0000 7fce89d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce8c136ea0 0x7fce8c14eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45606/0 (socket says 172.21.15.23:45606) 2020-11-20T11:44:41.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.739+0000 7fce89d9b700 1 -- 172.21.15.23:0/3238857854 learned_addr learned my addr 172.21.15.23:0/3238857854 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:44:41.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.739+0000 7fce8ad9d700 1 --2- 172.21.15.23:0/3238857854 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fce8c13f800 0x7fce8c133e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:41.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.740+0000 7fce89d9b700 1 -- 172.21.15.23:0/3238857854 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fce8c13f800 msgr2=0x7fce8c133e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:41.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.740+0000 7fce89d9b700 1 --2- 172.21.15.23:0/3238857854 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fce8c13f800 0x7fce8c133e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:41.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.740+0000 7fce89d9b700 1 -- 172.21.15.23:0/3238857854 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fce8c061a10 msgr2=0x7fce8c142dc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:44:41.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.740+0000 7fce89d9b700 1 --2- 172.21.15.23:0/3238857854 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fce8c061a10 0x7fce8c142dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:44:41.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.740+0000 7fce89d9b700 1 -- 172.21.15.23:0/3238857854 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce8c1432b0 con 0x7fce8c136ea0 2020-11-20T11:44:41.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.740+0000 7fce89d9b700 1 --2- 172.21.15.23:0/3238857854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce8c136ea0 0x7fce8c14eab0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fce8000c6e0 tx=0x7fce80007ab0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:41.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.743+0000 7fce7a7fc700 1 -- 172.21.15.23:0/3238857854 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8000e4a0 con 0x7fce8c136ea0 2020-11-20T11:44:41.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.743+0000 7fce7a7fc700 1 -- 172.21.15.23:0/3238857854 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce8000e620 con 0x7fce8c136ea0 2020-11-20T11:44:41.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.743+0000 7fce7a7fc700 1 -- 172.21.15.23:0/3238857854 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8000e9e0 con 0x7fce8c136ea0 2020-11-20T11:44:41.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.743+0000 7fce90f0d700 1 -- 172.21.15.23:0/3238857854 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce8c14efa0 con 0x7fce8c136ea0 2020-11-20T11:44:41.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.743+0000 7fce90f0d700 1 -- 172.21.15.23:0/3238857854 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce8c1344f0 con 0x7fce8c136ea0 2020-11-20T11:44:41.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.747+0000 7fce90f0d700 1 -- 172.21.15.23:0/3238857854 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce8c143440 con 0x7fce8c136ea0 2020-11-20T11:44:41.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.747+0000 7fce7a7fc700 1 -- 172.21.15.23:0/3238857854 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fce80019030 con 0x7fce8c136ea0 2020-11-20T11:44:41.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.748+0000 7fce7a7fc700 1 --2- 172.21.15.23:0/3238857854 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fce60056db0 0x7fce60059270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:41.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.748+0000 7fce8a59c700 1 --2- 172.21.15.23:0/3238857854 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fce60056db0 0x7fce60059270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:41.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.748+0000 7fce7a7fc700 1 -- 172.21.15.23:0/3238857854 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fce8006f460 con 0x7fce8c136ea0 2020-11-20T11:44:41.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.749+0000 7fce8a59c700 1 --2- 172.21.15.23:0/3238857854 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fce60056db0 0x7fce60059270 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fce8c142930 tx=0x7fce7c00d3e0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:41.904 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:41.901+0000 7fce7a7fc700 1 -- 172.21.15.23:0/3238857854 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fce80048200 con 0x7fce8c136ea0 2020-11-20T11:44:42.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.478+0000 7fce90f0d700 1 -- 172.21.15.23:0/3238857854 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fce8c061650 con 0x7fce8c136ea0 2020-11-20T11:44:42.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.481+0000 7fce7a7fc700 1 -- 172.21.15.23:0/3238857854 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7fce80047df0 con 0x7fce8c136ea0 2020-11-20T11:44:42.499 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":116,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:44:42.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.502+0000 7fce90f0d700 1 -- 172.21.15.23:0/3238857854 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fce60056db0 msgr2=0x7fce60059270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:42.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.502+0000 7fce90f0d700 1 --2- 172.21.15.23:0/3238857854 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fce60056db0 0x7fce60059270 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fce8c142930 tx=0x7fce7c00d3e0).stop 2020-11-20T11:44:42.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.502+0000 7fce90f0d700 1 -- 172.21.15.23:0/3238857854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce8c136ea0 msgr2=0x7fce8c14eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:42.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.502+0000 7fce90f0d700 1 --2- 172.21.15.23:0/3238857854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce8c136ea0 0x7fce8c14eab0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fce8000c6e0 tx=0x7fce80007ab0).stop 2020-11-20T11:44:42.509 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.506+0000 7fce90f0d700 1 -- 172.21.15.23:0/3238857854 shutdown_connections 2020-11-20T11:44:42.509 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.506+0000 7fce90f0d700 1 --2- 172.21.15.23:0/3238857854 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fce60056db0 0x7fce60059270 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:42.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.506+0000 7fce90f0d700 1 --2- 172.21.15.23:0/3238857854 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fce8c13f800 0x7fce8c133e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:42.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.506+0000 7fce90f0d700 1 --2- 172.21.15.23:0/3238857854 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fce8c061a10 0x7fce8c142dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:44:42.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.506+0000 7fce90f0d700 1 --2- 172.21.15.23:0/3238857854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fce8c136ea0 0x7fce8c14eab0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:42.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.506+0000 7fce90f0d700 1 -- 172.21.15.23:0/3238857854 >> 172.21.15.23:0/3238857854 conn(0x7fce8c060c80 msgr2=0x7fce8c135af0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:44:42.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.506+0000 7fce90f0d700 1 -- 172.21.15.23:0/3238857854 shutdown_connections 2020-11-20T11:44:42.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.506+0000 7fce90f0d700 1 -- 172.21.15.23:0/3238857854 wait complete. 2020-11-20T11:44:42.526 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:44:42.526 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:44:42.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.820+0000 7f5e0e5c9700 1 Processor -- start 2020-11-20T11:44:42.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.820+0000 7f5e0e5c9700 1 -- start start 2020-11-20T11:44:42.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.821+0000 7f5e0e5c9700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e08137870 0x7f5e08137cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:42.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.821+0000 7f5e0e5c9700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e080a9000 con 0x7f5e08133b90 2020-11-20T11:44:42.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.821+0000 7f5e0e5c9700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e08137170 con 0x7f5e08068260 2020-11-20T11:44:42.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.821+0000 7f5e0e5c9700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e08134c40 con 0x7f5e08137870 2020-11-20T11:44:42.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.821+0000 7f5e0cb66700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f5e08133b90 0x7f5e08134020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:59864/0 (socket says 172.21.15.23:59864) 2020-11-20T11:44:42.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.821+0000 7f5e0cb66700 1 -- 172.21.15.23:0/1260021773 learned_addr learned my addr 172.21.15.23:0/1260021773 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:44:42.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.821+0000 7f5e07fff700 1 --1- 172.21.15.23:0/1260021773 >> v1:172.21.15.155:6789/0 conn(0x7f5e08068260 0x7f5e08065780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:37836/0 (socket says 172.21.15.23:37836) 2020-11-20T11:44:42.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.821+0000 7f5e077fe700 1 --2- 172.21.15.23:0/1260021773 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e08137870 0x7f5e08137cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:42.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.822+0000 7f5e06ffd700 1 -- 172.21.15.23:0/1260021773 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2657084327 0 0) 0x7f5e080a9000 con 0x7f5e08133b90 2020-11-20T11:44:42.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.822+0000 7f5e06ffd700 1 -- 172.21.15.23:0/1260021773 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5de8003280 con 0x7f5e08133b90 2020-11-20T11:44:42.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.822+0000 7f5e06ffd700 1 -- 172.21.15.23:0/1260021773 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4111390957 0 0) 0x7f5e08137170 con 0x7f5e08068260 2020-11-20T11:44:42.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.822+0000 7f5e06ffd700 1 -- 172.21.15.23:0/1260021773 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e080a9000 con 0x7f5e08068260 2020-11-20T11:44:42.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.822+0000 7f5e077fe700 1 -- 172.21.15.23:0/1260021773 >> v1:172.21.15.155:6789/0 conn(0x7f5e08068260 legacy=0x7f5e08065780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:42.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.822+0000 7f5e077fe700 1 -- 172.21.15.23:0/1260021773 >> v1:172.21.15.23:6789/0 conn(0x7f5e08133b90 legacy=0x7f5e08134020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:42.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.822+0000 7f5e077fe700 1 -- 172.21.15.23:0/1260021773 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e08065e90 con 0x7f5e08137870 2020-11-20T11:44:42.832 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.822+0000 7f5e077fe700 1 --2- 172.21.15.23:0/1260021773 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e08137870 0x7f5e08137cd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f5e00009d40 tx=0x7f5e00009220).ready entity=mon.2 client_cookie=2a8d98dac228fc8 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:42.832 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.822+0000 7f5e06ffd700 1 -- 172.21.15.23:0/1260021773 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e00009e40 con 0x7f5e08137870 2020-11-20T11:44:42.832 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.822+0000 7f5e06ffd700 1 -- 172.21.15.23:0/1260021773 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e0000bac0 con 0x7f5e08137870 2020-11-20T11:44:42.832 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.822+0000 7f5e06ffd700 1 -- 172.21.15.23:0/1260021773 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0000be40 con 0x7f5e08137870 2020-11-20T11:44:42.833 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.823+0000 7f5e0e5c9700 1 -- 172.21.15.23:0/1260021773 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e08137870 msgr2=0x7f5e08137cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:42.833 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.823+0000 7f5e0e5c9700 1 --2- 172.21.15.23:0/1260021773 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e08137870 0x7f5e08137cd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f5e00009d40 tx=0x7f5e00009220).stop 2020-11-20T11:44:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.823+0000 7f5e0e5c9700 1 -- 172.21.15.23:0/1260021773 shutdown_connections 2020-11-20T11:44:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.823+0000 7f5e0e5c9700 1 --2- 172.21.15.23:0/1260021773 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e08137870 0x7f5e08137cd0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.823+0000 7f5e0e5c9700 1 -- 172.21.15.23:0/1260021773 >> 172.21.15.23:0/1260021773 conn(0x7f5e080ad7a0 msgr2=0x7f5e080adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:44:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.829+0000 7f5e0e5c9700 1 -- 172.21.15.23:0/1260021773 shutdown_connections 2020-11-20T11:44:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.829+0000 7f5e0e5c9700 1 -- 172.21.15.23:0/1260021773 wait complete. 2020-11-20T11:44:42.839 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.830+0000 7f5e0e5c9700 1 Processor -- start 2020-11-20T11:44:42.839 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.831+0000 7f5e0e5c9700 1 -- start start 2020-11-20T11:44:42.839 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.831+0000 7f5e0e5c9700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e08068260 0x7f5e080ace50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:42.840 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.831+0000 7f5e0e5c9700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e08133b90 0x7f5e08127aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:42.840 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.831+0000 7f5e0e5c9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e08137870 0x7f5e08127f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:42.840 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.831+0000 7f5e0e5c9700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e080a6170 con 0x7f5e08137870 2020-11-20T11:44:42.840 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.831+0000 7f5e0e5c9700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e080a6410 con 0x7f5e08133b90 2020-11-20T11:44:42.840 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.831+0000 7f5e0e5c9700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e08151990 con 0x7f5e08068260 2020-11-20T11:44:42.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.832+0000 7f5e077fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e08133b90 0x7f5e08127aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:42.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.832+0000 7f5e0cb66700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e08137870 0x7f5e08127f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:42.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.832+0000 7f5e077fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e08133b90 0x7f5e08127aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:38514/0 (socket says 172.21.15.23:38514) 2020-11-20T11:44:42.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.832+0000 7f5e077fe700 1 -- 172.21.15.23:0/915562879 learned_addr learned my addr 172.21.15.23:0/915562879 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:44:42.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.832+0000 7f5e07fff700 1 --2- 172.21.15.23:0/915562879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e08068260 0x7f5e080ace50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:42.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.832+0000 7f5e0cb66700 1 -- 172.21.15.23:0/915562879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e08068260 msgr2=0x7f5e080ace50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:42.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.832+0000 7f5e0cb66700 1 --2- 172.21.15.23:0/915562879 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e08068260 0x7f5e080ace50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:42.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.832+0000 7f5e0cb66700 1 -- 172.21.15.23:0/915562879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e08133b90 msgr2=0x7f5e08127aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:42.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.832+0000 7f5e0cb66700 1 --2- 172.21.15.23:0/915562879 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e08133b90 0x7f5e08127aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:42.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.832+0000 7f5e0cb66700 1 -- 172.21.15.23:0/915562879 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e08142170 con 0x7f5e08137870 2020-11-20T11:44:42.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.832+0000 7f5e0cb66700 1 --2- 172.21.15.23:0/915562879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e08137870 0x7f5e08127f90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f5df800c730 tx=0x7f5df80079c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:42.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.832+0000 7f5dfffff700 1 -- 172.21.15.23:0/915562879 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df80072e0 con 0x7f5e08137870 2020-11-20T11:44:42.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.833+0000 7f5e0e5c9700 1 -- 172.21.15.23:0/915562879 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e08142300 con 0x7f5e08137870 2020-11-20T11:44:42.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.833+0000 7f5e0e5c9700 1 -- 172.21.15.23:0/915562879 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e080a5bf0 con 0x7f5e08137870 2020-11-20T11:44:42.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.833+0000 7f5dfffff700 1 -- 172.21.15.23:0/915562879 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5df8005630 con 0x7f5e08137870 2020-11-20T11:44:42.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.833+0000 7f5dfffff700 1 -- 172.21.15.23:0/915562879 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df80057a0 con 0x7f5e08137870 2020-11-20T11:44:42.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.834+0000 7f5dfffff700 1 -- 172.21.15.23:0/915562879 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5df8010070 con 0x7f5e08137870 2020-11-20T11:44:42.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.835+0000 7f5dfffff700 1 --2- 172.21.15.23:0/915562879 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5de0056d60 0x7f5de0059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:42.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.835+0000 7f5dfffff700 1 -- 172.21.15.23:0/915562879 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f5df806f640 con 0x7f5e08137870 2020-11-20T11:44:42.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.835+0000 7f5e07fff700 1 --2- 172.21.15.23:0/915562879 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5de0056d60 0x7f5de0059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:43.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.838+0000 7f5e0e5c9700 1 --2- 172.21.15.23:0/915562879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5de4002a50 0x7f5de4004f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:43.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.838+0000 7f5e077fe700 1 --2- 172.21.15.23:0/915562879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5de4002a50 0x7f5de4004f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:43.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.839+0000 7f5e07fff700 1 --2- 172.21.15.23:0/915562879 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5de0056d60 0x7f5de0059220 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f5df0006fd0 tx=0x7f5df0008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:43.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.839+0000 7f5e0e5c9700 1 -- 172.21.15.23:0/915562879 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5de4007bb0 con 0x7f5de4002a50 2020-11-20T11:44:43.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.839+0000 7f5e077fe700 1 -- 172.21.15.23:0/915562879 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5de4007d70 con 0x7f5de4002a50 2020-11-20T11:44:43.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.839+0000 7f5e077fe700 1 --2- 172.21.15.23:0/915562879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5de4002a50 0x7f5de4004f10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f5e080a7d40 tx=0x7f5e00018000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:43.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.844+0000 7f5dfffff700 1 -- 172.21.15.23:0/915562879 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e00017d50 con 0x7f5de4002a50 2020-11-20T11:44:43.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.845+0000 7f5dfffff700 1 -- 172.21.15.23:0/915562879 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f5e00021400 con 0x7f5de4002a50 2020-11-20T11:44:43.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.845+0000 7f5dfffff700 1 -- 172.21.15.23:0/915562879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5de4002a50 msgr2=0x7f5de4004f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:43.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.845+0000 7f5dfffff700 1 --2- 172.21.15.23:0/915562879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5de4002a50 0x7f5de4004f10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f5e080a7d40 tx=0x7f5e00018000).stop 2020-11-20T11:44:43.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.845+0000 7f5dfffff700 1 -- 172.21.15.23:0/915562879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5de4002a50 msgr2=0x7f5de4004f10 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:44:43.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.845+0000 7f5dfffff700 1 --2- 172.21.15.23:0/915562879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5de4002a50 0x7f5de4004f10 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0x7f5e080a7d40 tx=0x7f5e00018000).stop 2020-11-20T11:44:43.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.875+0000 7f5e0e5c9700 1 --2- 172.21.15.23:0/915562879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5de4006bc0 0x7f5de4007050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:43.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.876+0000 7f5e077fe700 1 --2- 172.21.15.23:0/915562879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5de4006bc0 0x7f5de4007050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:43.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.876+0000 7f5e0e5c9700 1 -- 172.21.15.23:0/915562879 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5de4007660 con 0x7f5de4006bc0 2020-11-20T11:44:43.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.876+0000 7f5e077fe700 1 -- 172.21.15.23:0/915562879 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5de4007820 con 0x7f5de4006bc0 2020-11-20T11:44:43.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.877+0000 7f5e077fe700 1 --2- 172.21.15.23:0/915562879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5de4006bc0 0x7f5de4007050 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f5de4007d40 tx=0x7f5e00020000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:43.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.877+0000 7f5dfffff700 1 -- 172.21.15.23:0/915562879 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0002ac40 con 0x7f5de4006bc0 2020-11-20T11:44:43.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.877+0000 7f5dfffff700 1 -- 172.21.15.23:0/915562879 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4446 (secure 0 0 0) 0x7f5e0002ada0 con 0x7f5de4006bc0 2020-11-20T11:44:43.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.877+0000 7f5dfffff700 1 -- 172.21.15.23:0/915562879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5de4006bc0 msgr2=0x7f5de4007050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:43.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.877+0000 7f5dfffff700 1 --2- 172.21.15.23:0/915562879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5de4006bc0 0x7f5de4007050 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f5de4007d40 tx=0x7f5e00020000).stop 2020-11-20T11:44:43.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.877+0000 7f5dfffff700 1 -- 172.21.15.23:0/915562879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5de4006bc0 msgr2=0x7f5de4007050 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:44:43.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.877+0000 7f5dfffff700 1 --2- 172.21.15.23:0/915562879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5de4006bc0 0x7f5de4007050 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0x7f5de4007d40 tx=0x7f5e00020000).stop 2020-11-20T11:44:43.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.883+0000 7f5e0e5c9700 1 -- 172.21.15.23:0/915562879 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5de0056d60 msgr2=0x7f5de0059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:43.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.883+0000 7f5e0e5c9700 1 --2- 172.21.15.23:0/915562879 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5de0056d60 0x7f5de0059220 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f5df0006fd0 tx=0x7f5df0008040).stop 2020-11-20T11:44:43.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.883+0000 7f5e0e5c9700 1 -- 172.21.15.23:0/915562879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e08137870 msgr2=0x7f5e08127f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:43.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.883+0000 7f5e0e5c9700 1 --2- 172.21.15.23:0/915562879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e08137870 0x7f5e08127f90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f5df800c730 tx=0x7f5df80079c0).stop 2020-11-20T11:44:43.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.883+0000 7f5e0cb66700 1 -- 172.21.15.23:0/915562879 reap_dead start 2020-11-20T11:44:43.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.883+0000 7f5e0e5c9700 1 -- 172.21.15.23:0/915562879 shutdown_connections 2020-11-20T11:44:43.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.883+0000 7f5e0e5c9700 1 --2- 172.21.15.23:0/915562879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e08137870 0x7f5e08127f90 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:43.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.883+0000 7f5e0e5c9700 1 -- 172.21.15.23:0/915562879 >> 172.21.15.23:0/915562879 conn(0x7f5e080ad7a0 msgr2=0x7f5e08131180 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:44:43.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.883+0000 7f5e0e5c9700 1 -- 172.21.15.23:0/915562879 shutdown_connections 2020-11-20T11:44:43.298 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:42.883+0000 7f5e0e5c9700 1 -- 172.21.15.23:0/915562879 wait complete. 2020-11-20T11:44:43.298 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:44:43.298 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:44:43.298 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:44:43.299 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:44:43.299 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 116, 2020-11-20T11:44:43.299 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:44:43.299 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:44:43.299 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:44:43.300 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:44:43.300 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:43.300 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 2, 2020-11-20T11:44:43.300 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:44:43.301 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:44:43.301 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:44:43.301 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:44:43.301 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:44:43.302 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:44:43.302 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:44:43.302 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:44:43.302 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:44:43.302 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:44:43.303 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:44:43.303 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:43.303 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:44:43.303 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:44:43.303 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:44:43.303 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:44:43.304 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:44:43.304 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:44:43.304 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:44:43.304 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:44:43.304 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:44:43.305 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:44:43.305 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:43.305 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.305 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:44:43.306 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:44:43.306 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:44:43.306 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:44:43.306 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:44:43.306 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:44:43.306 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:44:43.307 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:44:43.307 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:44:43.307 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:44:43.307 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:44:43.307 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:44:43.308 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:44:43.308 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:44:43.308 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:44:43.308 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:44:43.308 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:44:43.309 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:44:43.309 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:44:43.309 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:44:43.309 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:44:43.309 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:44:43.310 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:44:43.310 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:43.310 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:44:43.310 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.310 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:44:43.311 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.311 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:44:43.311 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:44:43.311 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:44:43.311 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:44:43.312 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.312 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:44:43.312 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:44:43.312 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:43.312 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.313 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.313 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:44:43.313 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:44:43.313 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:43.313 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:43.314 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:43.314 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.314 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:44:43.314 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:44:43.314 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:44:43.315 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:44:43.315 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:44:43.315 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.315 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.315 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:44:43.316 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:44:43.316 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:44:43.316 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:44:43.316 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.316 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:44:43.317 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:44:43.317 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:43.317 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.317 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.317 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:44:43.318 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:44:43.318 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:43.318 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:43.318 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:43.318 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.319 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:44:43.319 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:44:43.319 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:44:43.319 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:44:43.319 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:44:43.320 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.320 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.320 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:44:43.320 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:44:43.320 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:44:43.321 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:44:43.321 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.321 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:44:43.321 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:44:43.321 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:43.322 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.322 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.322 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:44:43.322 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:44:43.322 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:43.323 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:43.323 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:43.323 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.323 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:44:43.323 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:44:43.324 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:44:43.324 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:44:43.324 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:44:43.324 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:43.324 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:43.325 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.326 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:44:43.326 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:44:43.326 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.326 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:44:43.326 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:44:43.327 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:44:43.327 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:43.327 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:43.327 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:44:43.327 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.328 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:44:43.328 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:44:43.328 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:44:43.328 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:43.329 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:43.329 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:44:43.329 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.329 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:44:43.329 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:44:43.330 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:44:43.330 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:43.330 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:43.330 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.330 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:44:43.331 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:44:43.331 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:44:43.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.472+0000 7fc55bed6700 1 Processor -- start 2020-11-20T11:44:43.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.472+0000 7fc55bed6700 1 -- start start 2020-11-20T11:44:43.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.473+0000 7fc55bed6700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc5540ad9e0 0x7fc55413af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:43.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.473+0000 7fc55bed6700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc55413be80 con 0x7fc5540ad600 2020-11-20T11:44:43.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.473+0000 7fc55bed6700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5540a6f10 con 0x7fc5540a6a70 2020-11-20T11:44:43.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.473+0000 7fc55bed6700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5540a71b0 con 0x7fc5540ad9e0 2020-11-20T11:44:43.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.473+0000 7fc559471700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc5540ad9e0 0x7fc55413af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:43.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.473+0000 7fc559c72700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fc5540ad600 0x7fc5540a90a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:59880/0 (socket says 172.21.15.23:59880) 2020-11-20T11:44:43.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.473+0000 7fc559c72700 1 -- 172.21.15.23:0/2641215405 learned_addr learned my addr 172.21.15.23:0/2641215405 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:44:43.481 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.473+0000 7fc559471700 1 -- 172.21.15.23:0/2641215405 >> v1:172.21.15.155:6789/0 conn(0x7fc5540a6a70 legacy=0x7fc55413b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:43.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.473+0000 7fc559471700 1 -- 172.21.15.23:0/2641215405 >> v1:172.21.15.23:6789/0 conn(0x7fc5540ad600 legacy=0x7fc5540a90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:43.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.473+0000 7fc559471700 1 -- 172.21.15.23:0/2641215405 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5540a4da0 con 0x7fc5540ad9e0 2020-11-20T11:44:43.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.473+0000 7fc559471700 1 --2- 172.21.15.23:0/2641215405 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5540ad9e0 0x7fc55413af90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fc550006fd0 tx=0x7fc550006e30).ready entity=mon.0 client_cookie=5bac989bd1fd55ba server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:43.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.474+0000 7fc558c70700 1 -- 172.21.15.23:0/2641215405 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5500097e0 con 0x7fc5540ad9e0 2020-11-20T11:44:43.482 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.474+0000 7fc558c70700 1 -- 172.21.15.23:0/2641215405 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc550000e90 con 0x7fc5540ad9e0 2020-11-20T11:44:43.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.474+0000 7fc558c70700 1 -- 172.21.15.23:0/2641215405 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc550009b50 con 0x7fc5540ad9e0 2020-11-20T11:44:43.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.474+0000 7fc55bed6700 1 -- 172.21.15.23:0/2641215405 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5540ad9e0 msgr2=0x7fc55413af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:43.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.474+0000 7fc55bed6700 1 --2- 172.21.15.23:0/2641215405 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5540ad9e0 0x7fc55413af90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fc550006fd0 tx=0x7fc550006e30).stop 2020-11-20T11:44:43.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.474+0000 7fc55bed6700 1 -- 172.21.15.23:0/2641215405 shutdown_connections 2020-11-20T11:44:43.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.474+0000 7fc55bed6700 1 --2- 172.21.15.23:0/2641215405 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5540ad9e0 0x7fc55413af90 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:43.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.474+0000 7fc55bed6700 1 -- 172.21.15.23:0/2641215405 >> 172.21.15.23:0/2641215405 conn(0x7fc554060c80 msgr2=0x7fc5540acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:44:43.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.474+0000 7fc55bed6700 1 -- 172.21.15.23:0/2641215405 shutdown_connections 2020-11-20T11:44:43.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.474+0000 7fc55bed6700 1 -- 172.21.15.23:0/2641215405 wait complete. 2020-11-20T11:44:43.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.475+0000 7fc55bed6700 1 Processor -- start 2020-11-20T11:44:43.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.475+0000 7fc55bed6700 1 -- start start 2020-11-20T11:44:43.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.476+0000 7fc55bed6700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc5540a6a70 0x7fc554066c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:43.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.476+0000 7fc55bed6700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc5540ad600 0x7fc55409d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:43.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.476+0000 7fc55bed6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5540ad9e0 0x7fc554149490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:43.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.476+0000 7fc55bed6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc554067950 con 0x7fc5540ad9e0 2020-11-20T11:44:43.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.476+0000 7fc55bed6700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc554065720 con 0x7fc5540a6a70 2020-11-20T11:44:43.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.476+0000 7fc55bed6700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5540659c0 con 0x7fc5540ad600 2020-11-20T11:44:43.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.476+0000 7fc55a473700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5540ad9e0 0x7fc554149490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:43.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.476+0000 7fc55a473700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5540ad9e0 0x7fc554149490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45636/0 (socket says 172.21.15.23:45636) 2020-11-20T11:44:43.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.476+0000 7fc55a473700 1 -- 172.21.15.23:0/1749542490 learned_addr learned my addr 172.21.15.23:0/1749542490 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:44:43.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.477+0000 7fc55a473700 1 -- 172.21.15.23:0/1749542490 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc5540ad600 msgr2=0x7fc55409d9a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:44:43.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.477+0000 7fc55a473700 1 --2- 172.21.15.23:0/1749542490 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc5540ad600 0x7fc55409d9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:44:43.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.477+0000 7fc55a473700 1 -- 172.21.15.23:0/1749542490 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc5540a6a70 msgr2=0x7fc554066c40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2020-11-20T11:44:43.489 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.477+0000 7fc55a473700 1 --2- 172.21.15.23:0/1749542490 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc5540a6a70 0x7fc554066c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:44:43.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.477+0000 7fc55a473700 1 -- 172.21.15.23:0/1749542490 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc554065c60 con 0x7fc5540ad9e0 2020-11-20T11:44:43.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.477+0000 7fc55a473700 1 --2- 172.21.15.23:0/1749542490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5540ad9e0 0x7fc554149490 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fc54c001760 tx=0x7fc54c001b80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:43.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.477+0000 7fc545ffb700 1 -- 172.21.15.23:0/1749542490 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54c013e80 con 0x7fc5540ad9e0 2020-11-20T11:44:43.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.477+0000 7fc55bed6700 1 -- 172.21.15.23:0/1749542490 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc554138690 con 0x7fc5540ad9e0 2020-11-20T11:44:43.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.477+0000 7fc545ffb700 1 -- 172.21.15.23:0/1749542490 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc54c012030 con 0x7fc5540ad9e0 2020-11-20T11:44:43.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.477+0000 7fc545ffb700 1 -- 172.21.15.23:0/1749542490 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54c006ac0 con 0x7fc5540ad9e0 2020-11-20T11:44:43.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.477+0000 7fc55bed6700 1 -- 172.21.15.23:0/1749542490 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc55409de90 con 0x7fc5540ad9e0 2020-11-20T11:44:43.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.479+0000 7fc545ffb700 1 -- 172.21.15.23:0/1749542490 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc54c028070 con 0x7fc5540ad9e0 2020-11-20T11:44:43.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.479+0000 7fc545ffb700 1 --2- 172.21.15.23:0/1749542490 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc52c056d60 0x7fc52c059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:43.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.479+0000 7fc545ffb700 1 -- 172.21.15.23:0/1749542490 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fc54c081340 con 0x7fc5540ad9e0 2020-11-20T11:44:43.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.479+0000 7fc559c72700 1 --2- 172.21.15.23:0/1749542490 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc52c056d60 0x7fc52c059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:43.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.481+0000 7fc559c72700 1 --2- 172.21.15.23:0/1749542490 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc52c056d60 0x7fc52c059220 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fc5480055e0 tx=0x7fc5480077d0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:43.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.482+0000 7fc55bed6700 1 --2- 172.21.15.23:0/1749542490 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc554065df0 0x7fc554149980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:43.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.482+0000 7fc55bed6700 1 -- 172.21.15.23:0/1749542490 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55405c520 con 0x7fc554065df0 2020-11-20T11:44:43.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.482+0000 7fc559471700 1 --2- 172.21.15.23:0/1749542490 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc554065df0 0x7fc554149980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:43.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.483+0000 7fc559471700 1 -- 172.21.15.23:0/1749542490 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc5541352f0 con 0x7fc554065df0 2020-11-20T11:44:43.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.483+0000 7fc559471700 1 --2- 172.21.15.23:0/1749542490 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc554065df0 0x7fc554149980 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fc550006fd0 tx=0x7fc550017000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:43.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.483+0000 7fc545ffb700 1 -- 172.21.15.23:0/1749542490 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc550010c60 con 0x7fc554065df0 2020-11-20T11:44:43.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.483+0000 7fc545ffb700 1 -- 172.21.15.23:0/1749542490 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fc550010dc0 con 0x7fc554065df0 2020-11-20T11:44:43.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.483+0000 7fc545ffb700 1 -- 172.21.15.23:0/1749542490 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc554065df0 msgr2=0x7fc554149980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:43.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.483+0000 7fc545ffb700 1 --2- 172.21.15.23:0/1749542490 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc554065df0 0x7fc554149980 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fc550006fd0 tx=0x7fc550017000).stop 2020-11-20T11:44:43.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.483+0000 7fc545ffb700 1 -- 172.21.15.23:0/1749542490 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc554065df0 msgr2=0x7fc554149980 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:44:43.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.483+0000 7fc545ffb700 1 --2- 172.21.15.23:0/1749542490 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc554065df0 0x7fc554149980 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:43.497 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.490+0000 7fc55bed6700 1 --2- 172.21.15.23:0/1749542490 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc554149e70 0x7fc554145ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:43.497 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.490+0000 7fc55bed6700 1 -- 172.21.15.23:0/1749542490 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc554044710 con 0x7fc554149e70 2020-11-20T11:44:43.497 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.490+0000 7fc559471700 1 --2- 172.21.15.23:0/1749542490 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc554149e70 0x7fc554145ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:43.498 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.491+0000 7fc559471700 1 -- 172.21.15.23:0/1749542490 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc5540687a0 con 0x7fc554149e70 2020-11-20T11:44:43.498 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.491+0000 7fc559471700 1 --2- 172.21.15.23:0/1749542490 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc554149e70 0x7fc554145ec0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fc55000b190 tx=0x7fc55001d040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:43.498 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.491+0000 7fc545ffb700 1 -- 172.21.15.23:0/1749542490 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55000f830 con 0x7fc554149e70 2020-11-20T11:44:43.498 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.491+0000 7fc545ffb700 1 -- 172.21.15.23:0/1749542490 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4784 (secure 0 0 0) 0x7fc550010800 con 0x7fc554149e70 2020-11-20T11:44:43.498 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.491+0000 7fc545ffb700 1 -- 172.21.15.23:0/1749542490 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc554149e70 msgr2=0x7fc554145ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:43.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.491+0000 7fc545ffb700 1 --2- 172.21.15.23:0/1749542490 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc554149e70 0x7fc554145ec0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fc55000b190 tx=0x7fc55001d040).stop 2020-11-20T11:44:43.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.491+0000 7fc545ffb700 1 -- 172.21.15.23:0/1749542490 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc554149e70 msgr2=0x7fc554145ec0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:44:43.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.491+0000 7fc545ffb700 1 --2- 172.21.15.23:0/1749542490 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc554149e70 0x7fc554145ec0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:43.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.491+0000 7fc55bed6700 1 -- 172.21.15.23:0/1749542490 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc52c056d60 msgr2=0x7fc52c059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:43.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.491+0000 7fc55bed6700 1 --2- 172.21.15.23:0/1749542490 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc52c056d60 0x7fc52c059220 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fc5480055e0 tx=0x7fc5480077d0).stop 2020-11-20T11:44:43.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.491+0000 7fc55bed6700 1 -- 172.21.15.23:0/1749542490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5540ad9e0 msgr2=0x7fc554149490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:43.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.491+0000 7fc55bed6700 1 --2- 172.21.15.23:0/1749542490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5540ad9e0 0x7fc554149490 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fc54c001760 tx=0x7fc54c001b80).stop 2020-11-20T11:44:43.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.491+0000 7fc55a473700 1 -- 172.21.15.23:0/1749542490 reap_dead start 2020-11-20T11:44:43.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.491+0000 7fc55a473700 1 -- 172.21.15.23:0/1749542490 reap_dead start 2020-11-20T11:44:43.500 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.491+0000 7fc55bed6700 1 -- 172.21.15.23:0/1749542490 shutdown_connections 2020-11-20T11:44:43.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.492+0000 7fc55bed6700 1 -- 172.21.15.23:0/1749542490 >> 172.21.15.23:0/1749542490 conn(0x7fc554060c80 msgr2=0x7fc55413c830 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:44:43.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.492+0000 7fc55bed6700 1 -- 172.21.15.23:0/1749542490 shutdown_connections 2020-11-20T11:44:43.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.492+0000 7fc55bed6700 1 -- 172.21.15.23:0/1749542490 wait complete. 2020-11-20T11:44:43.501 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:44:43.502 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:44:43.502 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:44:43.502 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:44:43.502 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 116, 2020-11-20T11:44:43.502 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:44:43.503 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:44:43.503 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:44:43.503 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:44:43.503 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:43.503 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 3, 2020-11-20T11:44:43.504 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:44:43.504 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:44:43.504 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:44:43.504 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:44:43.504 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:44:43.505 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:44:43.505 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:44:43.505 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:44:43.505 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:44:43.505 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:44:43.506 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:44:43.506 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:43.506 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:44:43.506 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:44:43.507 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:44:43.507 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:44:43.507 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:44:43.507 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:44:43.507 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:44:43.508 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:44:43.508 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:44:43.508 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:44:43.508 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:43.508 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.509 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:44:43.509 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:44:43.509 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:44:43.509 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:44:43.509 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:44:43.510 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:44:43.510 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:44:43.510 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:44:43.510 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:44:43.511 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:44:43.511 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:44:43.511 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:44:43.512 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:44:43.512 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:44:43.512 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:44:43.512 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:44:43.513 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:44:43.513 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:44:43.513 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:44:43.514 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:44:43.514 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:44:43.515 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:44:43.515 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:44:43.515 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:43.516 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:44:43.516 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.516 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:44:43.517 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.517 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:44:43.518 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:44:43.518 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:44:43.518 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:44:43.519 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.519 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:44:43.520 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:44:43.520 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:43.521 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.521 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.521 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:44:43.522 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:44:43.522 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:43.523 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:43.523 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:43.523 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.524 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:44:43.524 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:44:43.525 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:44:43.525 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:44:43.525 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:44:43.526 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.526 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.527 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:44:43.527 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:44:43.527 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:44:43.528 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:44:43.528 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.528 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:44:43.529 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:44:43.529 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:43.530 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.530 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.530 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:44:43.531 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:44:43.531 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:43.532 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:43.532 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:43.532 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.533 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:44:43.533 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:44:43.533 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:44:43.534 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:44:43.534 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:44:43.534 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.535 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.535 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:44:43.535 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:44:43.536 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:44:43.536 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:44:43.536 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.537 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:44:43.537 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:44:43.537 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:43.538 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.538 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.538 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:44:43.538 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:44:43.538 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:43.539 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:43.539 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:43.539 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.539 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:44:43.539 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:44:43.540 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:44:43.540 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:44:43.540 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:44:43.540 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:43.541 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:43.541 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.541 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:44:43.541 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:44:43.541 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.542 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:44:43.542 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:44:43.542 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:44:43.542 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:43.542 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:43.543 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:44:43.543 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.543 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:44:43.543 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:44:43.543 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:44:43.544 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:43.544 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:43.544 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:44:43.544 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.544 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:44:43.545 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:44:43.545 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:44:43.545 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:43.545 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:43.546 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:44:43.546 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.546 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:44:43.546 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:44:43.546 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:44:43.547 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:43.547 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:43.547 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:44:43.547 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.547 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:44:43.548 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:44:43.548 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:44:43.548 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:43.548 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:43.548 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.549 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:44:43.549 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:44:43.550 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:44:43.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.685+0000 7f04cf8ee700 1 Processor -- start 2020-11-20T11:44:43.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.686+0000 7f04cf8ee700 1 -- start start 2020-11-20T11:44:43.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.686+0000 7f04cf8ee700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f04c80a6e10 0x7f04c8061850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:43.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.686+0000 7f04cf8ee700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f04c8137410 0x7f04c8137870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:43.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.686+0000 7f04cf8ee700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f04c813a760 0x7f04c813abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:43.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.686+0000 7f04cf8ee700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04c80a61f0 con 0x7f04c813a760 2020-11-20T11:44:43.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.686+0000 7f04cf8ee700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04c8062ad0 con 0x7f04c8137410 2020-11-20T11:44:43.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.686+0000 7f04cf8ee700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c80a63e0 con 0x7f04c80a6e10 2020-11-20T11:44:43.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.687+0000 7f04cde8b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f04c813a760 0x7f04c813abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:43.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.687+0000 7f04cde8b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f04c813a760 0x7f04c813abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45650/0 (socket says 172.21.15.23:45650) 2020-11-20T11:44:43.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.687+0000 7f04cde8b700 1 -- 172.21.15.23:0/3361993967 learned_addr learned my addr 172.21.15.23:0/3361993967 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:44:43.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.687+0000 7f04cce89700 1 --2- 172.21.15.23:0/3361993967 >> v2:172.21.15.155:3300/0 conn(0x7f04c8137410 0x7f04c8137870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:43.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.687+0000 7f04cd68a700 1 --2- 172.21.15.23:0/3361993967 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f04c80a6e10 0x7f04c8061850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:43.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.687+0000 7f04cde8b700 1 -- 172.21.15.23:0/3361993967 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f04c80a6e10 msgr2=0x7f04c8061850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:44:43.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.687+0000 7f04cde8b700 1 --2- 172.21.15.23:0/3361993967 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f04c80a6e10 0x7f04c8061850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:43.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.687+0000 7f04cde8b700 1 -- 172.21.15.23:0/3361993967 >> v2:172.21.15.155:3300/0 conn(0x7f04c8137410 msgr2=0x7f04c8137870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:44:43.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.687+0000 7f04cde8b700 1 --2- 172.21.15.23:0/3361993967 >> v2:172.21.15.155:3300/0 conn(0x7f04c8137410 0x7f04c8137870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:43.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.687+0000 7f04cde8b700 1 -- 172.21.15.23:0/3361993967 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c80a6680 con 0x7f04c813a760 2020-11-20T11:44:43.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.687+0000 7f04cde8b700 1 --2- 172.21.15.23:0/3361993967 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04c813a760 0x7f04c813abf0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f04c8061140 tx=0x7f04c00092d0).ready entity=mon.0 client_cookie=ef3ca1632e201726 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:43.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.687+0000 7f04b7fff700 1 -- 172.21.15.23:0/3361993967 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c000bb40 con 0x7f04c813a760 2020-11-20T11:44:43.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.687+0000 7f04b7fff700 1 -- 172.21.15.23:0/3361993967 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04c000bca0 con 0x7f04c813a760 2020-11-20T11:44:43.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.687+0000 7f04b7fff700 1 -- 172.21.15.23:0/3361993967 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c0011610 con 0x7f04c813a760 2020-11-20T11:44:43.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.688+0000 7f04cf8ee700 1 -- 172.21.15.23:0/3361993967 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04c813a760 msgr2=0x7f04c813abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:43.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.688+0000 7f04cf8ee700 1 --2- 172.21.15.23:0/3361993967 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04c813a760 0x7f04c813abf0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f04c8061140 tx=0x7f04c00092d0).stop 2020-11-20T11:44:43.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.688+0000 7f04cf8ee700 1 -- 172.21.15.23:0/3361993967 shutdown_connections 2020-11-20T11:44:43.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.688+0000 7f04cf8ee700 1 --2- 172.21.15.23:0/3361993967 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04c813a760 0x7f04c813abf0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:43.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.688+0000 7f04cf8ee700 1 --2- 172.21.15.23:0/3361993967 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f04c80a6e10 0x7f04c8061850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:43.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.688+0000 7f04cf8ee700 1 --2- 172.21.15.23:0/3361993967 >> v2:172.21.15.155:3300/0 conn(0x7f04c8137410 0x7f04c8137870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:43.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.688+0000 7f04cf8ee700 1 -- 172.21.15.23:0/3361993967 >> 172.21.15.23:0/3361993967 conn(0x7f04c8060c80 msgr2=0x7f04c80acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:44:43.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.688+0000 7f04cf8ee700 1 -- 172.21.15.23:0/3361993967 shutdown_connections 2020-11-20T11:44:43.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.688+0000 7f04cf8ee700 1 -- 172.21.15.23:0/3361993967 wait complete. 2020-11-20T11:44:43.700 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.689+0000 7f04cf8ee700 1 Processor -- start 2020-11-20T11:44:43.700 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.689+0000 7f04cf8ee700 1 -- start start 2020-11-20T11:44:43.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.689+0000 7f04cf8ee700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04c80a6e10 0x7f04c8149330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:43.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.690+0000 7f04cf8ee700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f04c8137410 0x7f04c80ac370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:43.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.690+0000 7f04cf8ee700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f04c813a760 0x7f04c80ac860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:43.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.690+0000 7f04cf8ee700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c8067ba0 con 0x7f04c80a6e10 2020-11-20T11:44:43.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.690+0000 7f04cf8ee700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c8145960 con 0x7f04c813a760 2020-11-20T11:44:43.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.690+0000 7f04cf8ee700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c81499d0 con 0x7f04c8137410 2020-11-20T11:44:43.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.690+0000 7f04cd68a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04c80a6e10 0x7f04c8149330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:43.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.690+0000 7f04cd68a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04c80a6e10 0x7f04c8149330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45652/0 (socket says 172.21.15.23:45652) 2020-11-20T11:44:43.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.690+0000 7f04cd68a700 1 -- 172.21.15.23:0/3356748343 learned_addr learned my addr 172.21.15.23:0/3356748343 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:44:43.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.690+0000 7f04cde8b700 1 --2- 172.21.15.23:0/3356748343 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f04c813a760 0x7f04c80ac860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:43.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.690+0000 7f04cce89700 1 --2- 172.21.15.23:0/3356748343 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f04c8137410 0x7f04c80ac370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:43.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.690+0000 7f04cd68a700 1 -- 172.21.15.23:0/3356748343 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f04c8137410 msgr2=0x7f04c80ac370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:43.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.690+0000 7f04cd68a700 1 --2- 172.21.15.23:0/3356748343 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f04c8137410 0x7f04c80ac370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:43.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.690+0000 7f04cd68a700 1 -- 172.21.15.23:0/3356748343 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f04c813a760 msgr2=0x7f04c80ac860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:43.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.690+0000 7f04cd68a700 1 --2- 172.21.15.23:0/3356748343 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f04c813a760 0x7f04c80ac860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:43.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.690+0000 7f04cd68a700 1 -- 172.21.15.23:0/3356748343 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c8149c70 con 0x7f04c80a6e10 2020-11-20T11:44:43.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.690+0000 7f04cd68a700 1 --2- 172.21.15.23:0/3356748343 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04c80a6e10 0x7f04c8149330 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f04b800eba0 tx=0x7f04b8006e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:43.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.690+0000 7f04b57fa700 1 -- 172.21.15.23:0/3356748343 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b800c8e0 con 0x7f04c80a6e10 2020-11-20T11:44:43.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.690+0000 7f04b57fa700 1 -- 172.21.15.23:0/3356748343 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b800ca40 con 0x7f04c80a6e10 2020-11-20T11:44:43.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.690+0000 7f04b57fa700 1 -- 172.21.15.23:0/3356748343 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b800cdc0 con 0x7f04c80a6e10 2020-11-20T11:44:43.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.690+0000 7f04cf8ee700 1 -- 172.21.15.23:0/3356748343 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04c8145d90 con 0x7f04c80a6e10 2020-11-20T11:44:43.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.690+0000 7f04cf8ee700 1 -- 172.21.15.23:0/3356748343 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04c8146270 con 0x7f04c80a6e10 2020-11-20T11:44:43.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.691+0000 7f04cf8ee700 1 --2- 172.21.15.23:0/3356748343 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f04c8146490 0x7f04c8132070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:43.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.691+0000 7f04cf8ee700 1 -- 172.21.15.23:0/3356748343 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c8149820 con 0x7f04c8146490 2020-11-20T11:44:43.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.691+0000 7f04cce89700 1 --2- 172.21.15.23:0/3356748343 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f04c8146490 0x7f04c8132070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:43.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.692+0000 7f04b57fa700 1 -- 172.21.15.23:0/3356748343 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f04b8012600 con 0x7f04c80a6e10 2020-11-20T11:44:43.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.692+0000 7f04cce89700 1 -- 172.21.15.23:0/3356748343 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04c805c520 con 0x7f04c8146490 2020-11-20T11:44:43.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.692+0000 7f04cce89700 1 --2- 172.21.15.23:0/3356748343 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f04c8146490 0x7f04c8132070 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f04c80add30 tx=0x7f04c4006ec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:43.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.692+0000 7f04b57fa700 1 --2- 172.21.15.23:0/3356748343 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f04a0056d90 0x7f04a0059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:43.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.692+0000 7f04b57fa700 1 -- 172.21.15.23:0/3356748343 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f04b80703e0 con 0x7f04c80a6e10 2020-11-20T11:44:43.707 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.692+0000 7f04b57fa700 1 -- 172.21.15.23:0/3356748343 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c4011ed0 con 0x7f04c8146490 2020-11-20T11:44:43.707 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.692+0000 7f04cde8b700 1 --2- 172.21.15.23:0/3356748343 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f04a0056d90 0x7f04a0059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:43.707 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.692+0000 7f04b57fa700 1 -- 172.21.15.23:0/3356748343 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f04c4000e90 con 0x7f04c8146490 2020-11-20T11:44:43.707 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.692+0000 7f04b57fa700 1 -- 172.21.15.23:0/3356748343 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f04c8146490 msgr2=0x7f04c8132070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:43.707 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.692+0000 7f04b57fa700 1 --2- 172.21.15.23:0/3356748343 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f04c8146490 0x7f04c8132070 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f04c80add30 tx=0x7f04c4006ec0).stop 2020-11-20T11:44:43.708 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.692+0000 7f04b57fa700 1 -- 172.21.15.23:0/3356748343 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f04c8146490 msgr2=0x7f04c8132070 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:44:43.708 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.692+0000 7f04b57fa700 1 --2- 172.21.15.23:0/3356748343 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f04c8146490 0x7f04c8132070 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:43.708 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.692+0000 7f04cde8b700 1 --2- 172.21.15.23:0/3356748343 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f04a0056d90 0x7f04a0059250 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f04c0009130 tx=0x7f04c0016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:43.710 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.699+0000 7f04cf8ee700 1 --2- 172.21.15.23:0/3356748343 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f04c8132560 0x7f04c80aaf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:43.710 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.699+0000 7f04cf8ee700 1 -- 172.21.15.23:0/3356748343 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c8044710 con 0x7f04c8132560 2020-11-20T11:44:43.710 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.700+0000 7f04cce89700 1 --2- 172.21.15.23:0/3356748343 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f04c8132560 0x7f04c80aaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:43.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.700+0000 7f04cce89700 1 -- 172.21.15.23:0/3356748343 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f04c80687a0 con 0x7f04c8132560 2020-11-20T11:44:43.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.700+0000 7f04cce89700 1 --2- 172.21.15.23:0/3356748343 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f04c8132560 0x7f04c80aaf70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f04c80add30 tx=0x7f04c401e000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:43.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.700+0000 7f04b57fa700 1 -- 172.21.15.23:0/3356748343 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c4006020 con 0x7f04c8132560 2020-11-20T11:44:43.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.700+0000 7f04b57fa700 1 -- 172.21.15.23:0/3356748343 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4444 (secure 0 0 0) 0x7f04c4011cc0 con 0x7f04c8132560 2020-11-20T11:44:43.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.700+0000 7f04b57fa700 1 -- 172.21.15.23:0/3356748343 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f04c8132560 msgr2=0x7f04c80aaf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:43.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.700+0000 7f04b57fa700 1 --2- 172.21.15.23:0/3356748343 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f04c8132560 0x7f04c80aaf70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f04c80add30 tx=0x7f04c401e000).stop 2020-11-20T11:44:43.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.700+0000 7f04b57fa700 1 -- 172.21.15.23:0/3356748343 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f04c8132560 msgr2=0x7f04c80aaf70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:44:43.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.700+0000 7f04b57fa700 1 --2- 172.21.15.23:0/3356748343 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f04c8132560 0x7f04c80aaf70 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:43.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.701+0000 7f04cf8ee700 1 -- 172.21.15.23:0/3356748343 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f04a0056d90 msgr2=0x7f04a0059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:43.712 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.701+0000 7f04cf8ee700 1 --2- 172.21.15.23:0/3356748343 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f04a0056d90 0x7f04a0059250 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f04c0009130 tx=0x7f04c0016000).stop 2020-11-20T11:44:43.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.701+0000 7f04cf8ee700 1 -- 172.21.15.23:0/3356748343 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04c80a6e10 msgr2=0x7f04c8149330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:43.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.701+0000 7f04cf8ee700 1 --2- 172.21.15.23:0/3356748343 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04c80a6e10 0x7f04c8149330 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f04b800eba0 tx=0x7f04b8006e30).stop 2020-11-20T11:44:43.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.701+0000 7f04cde8b700 1 -- 172.21.15.23:0/3356748343 reap_dead start 2020-11-20T11:44:43.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.701+0000 7f04cde8b700 1 -- 172.21.15.23:0/3356748343 reap_dead start 2020-11-20T11:44:43.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.701+0000 7f04cf8ee700 1 -- 172.21.15.23:0/3356748343 shutdown_connections 2020-11-20T11:44:43.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.701+0000 7f04cf8ee700 1 -- 172.21.15.23:0/3356748343 >> 172.21.15.23:0/3356748343 conn(0x7f04c8060c80 msgr2=0x7f04c8135180 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:44:43.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.701+0000 7f04cf8ee700 1 -- 172.21.15.23:0/3356748343 shutdown_connections 2020-11-20T11:44:43.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.701+0000 7f04cf8ee700 1 -- 172.21.15.23:0/3356748343 wait complete. 2020-11-20T11:44:43.714 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:44:43.715 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:44:43.715 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:44:43.715 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:44:43.715 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 116, 2020-11-20T11:44:43.716 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:44:43.716 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:44:43.716 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:44:43.716 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:44:43.716 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:43.717 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 3, 2020-11-20T11:44:43.717 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:44:43.717 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:44:43.718 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:44:43.718 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:44:43.718 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:44:43.718 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:44:43.719 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:44:43.719 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:44:43.719 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:44:43.719 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:44:43.720 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:44:43.720 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:43.720 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:44:43.720 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:44:43.720 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:44:43.721 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:44:43.721 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:44:43.721 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:44:43.721 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:44:43.722 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:44:43.722 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:44:43.722 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:44:43.722 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:43.723 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.723 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:44:43.723 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:44:43.723 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:44:43.724 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:44:43.724 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:44:43.724 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:44:43.724 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:44:43.725 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:44:43.725 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:44:43.725 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:44:43.725 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:44:43.726 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:44:43.726 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:44:43.726 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:44:43.726 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:44:43.726 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:44:43.727 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:44:43.727 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:44:43.727 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:44:43.727 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:44:43.728 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:44:43.728 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:44:43.728 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:44:43.728 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:43.729 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:44:43.729 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.729 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:44:43.729 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.730 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:44:43.730 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:44:43.730 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:44:43.730 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:44:43.730 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.731 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:44:43.731 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:44:43.731 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:43.732 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.732 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.732 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:44:43.732 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:44:43.732 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:43.733 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:43.733 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:43.733 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.733 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:44:43.734 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:44:43.734 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:44:43.734 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:44:43.734 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:44:43.735 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.735 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.735 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:44:43.735 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:44:43.736 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:44:43.736 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:44:43.736 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.736 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:44:43.737 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:44:43.737 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:43.737 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.737 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.737 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:44:43.738 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:44:43.738 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:43.738 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:43.738 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:43.739 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.739 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:44:43.739 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:44:43.739 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:44:43.739 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:44:43.740 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:44:43.740 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.740 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.740 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:44:43.741 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:44:43.741 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:44:43.741 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:44:43.741 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.742 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:44:43.742 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:44:43.742 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:43.742 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.743 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.743 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:44:43.743 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:44:43.743 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:43.743 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:43.744 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:43.744 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.744 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:44:43.744 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:44:43.745 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:44:43.745 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:44:43.745 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:44:43.745 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:43.745 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:43.746 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.746 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:44:43.746 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:44:43.746 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.747 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:44:43.747 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:44:43.747 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:44:43.747 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:43.748 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:43.748 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:44:43.748 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.748 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:44:43.749 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:44:43.749 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:44:43.749 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:43.749 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:43.750 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:44:43.750 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:43.750 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:44:43.750 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:44:43.751 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:44:43.751 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:43.751 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:43.751 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:43.751 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:44:43.752 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:44:43.753 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2020-11-20T11:44:43.753 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2020-11-20T11:44:43.998 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.991+0000 7fd47d9fe700 1 Processor -- start 2020-11-20T11:44:43.998 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.991+0000 7fd47d9fe700 1 -- start start 2020-11-20T11:44:43.999 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.992+0000 7fd47d9fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd4780a99f0 0x7fd478137e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:43.999 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.992+0000 7fd47d9fe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd478138380 0x7fd478138810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:43.999 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.992+0000 7fd47d9fe700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd478138d00 con 0x7fd478061b90 2020-11-20T11:44:44.000 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.992+0000 7fd47d9fe700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd478067190 con 0x7fd4780a99f0 2020-11-20T11:44:44.000 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.992+0000 7fd47d9fe700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd478067430 con 0x7fd478138380 2020-11-20T11:44:44.000 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.992+0000 7fd4767fc700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd4780a99f0 0x7fd478137e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:44.001 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.992+0000 7fd4777fe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd478138380 0x7fd478138810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:44.002 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.993+0000 7fd4767fc700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd4780a99f0 0x7fd478137e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:38562/0 (socket says 172.21.15.23:38562) 2020-11-20T11:44:44.002 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.993+0000 7fd4767fc700 1 -- 172.21.15.23:0/1091128545 learned_addr learned my addr 172.21.15.23:0/1091128545 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:44:44.002 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.993+0000 7fd4777fe700 1 -- 172.21.15.23:0/1091128545 >> v2:172.21.15.155:3300/0 conn(0x7fd4780a99f0 msgr2=0x7fd478137e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:44:44.003 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.993+0000 7fd4777fe700 1 --2- 172.21.15.23:0/1091128545 >> v2:172.21.15.155:3300/0 conn(0x7fd4780a99f0 0x7fd478137e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:44.003 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.993+0000 7fd4777fe700 1 -- 172.21.15.23:0/1091128545 >> v1:172.21.15.23:6789/0 conn(0x7fd478061b90 legacy=0x7fd4780a92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:44:44.004 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.993+0000 7fd4777fe700 1 -- 172.21.15.23:0/1091128545 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd478067860 con 0x7fd478138380 2020-11-20T11:44:44.004 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:43.993+0000 7fd4777fe700 1 --2- 172.21.15.23:0/1091128545 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd478138380 0x7fd478138810 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fd46c006fd0 tx=0x7fd46c006e30).ready entity=mon.2 client_cookie=f226617fa30037e0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:44.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.000+0000 7fd475ffb700 1 -- 172.21.15.23:0/1091128545 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46c0097a0 con 0x7fd478138380 2020-11-20T11:44:44.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.000+0000 7fd475ffb700 1 -- 172.21.15.23:0/1091128545 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd46c000e90 con 0x7fd478138380 2020-11-20T11:44:44.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.001+0000 7fd47d9fe700 1 -- 172.21.15.23:0/1091128545 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd478138380 msgr2=0x7fd478138810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:44.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.001+0000 7fd47d9fe700 1 --2- 172.21.15.23:0/1091128545 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd478138380 0x7fd478138810 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fd46c006fd0 tx=0x7fd46c006e30).stop 2020-11-20T11:44:44.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.005+0000 7fd47d9fe700 1 -- 172.21.15.23:0/1091128545 shutdown_connections 2020-11-20T11:44:44.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.005+0000 7fd47d9fe700 1 --2- 172.21.15.23:0/1091128545 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd478138380 0x7fd478138810 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:44.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.005+0000 7fd47d9fe700 1 --2- 172.21.15.23:0/1091128545 >> v2:172.21.15.155:3300/0 conn(0x7fd4780a99f0 0x7fd478137e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:44.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.005+0000 7fd47d9fe700 1 -- 172.21.15.23:0/1091128545 >> 172.21.15.23:0/1091128545 conn(0x7fd478063640 msgr2=0x7fd4780612c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:44:44.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.005+0000 7fd47d9fe700 1 -- 172.21.15.23:0/1091128545 shutdown_connections 2020-11-20T11:44:44.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.005+0000 7fd47d9fe700 1 -- 172.21.15.23:0/1091128545 wait complete. 2020-11-20T11:44:44.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.007+0000 7fd47d9fe700 1 Processor -- start 2020-11-20T11:44:44.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.007+0000 7fd47d9fe700 1 -- start start 2020-11-20T11:44:44.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.008+0000 7fd47d9fe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd478061b90 0x7fd4780a6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:44.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.008+0000 7fd47d9fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4780a99f0 0x7fd4780a3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:44.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.008+0000 7fd47d9fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd478138380 0x7fd4780a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:44.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.008+0000 7fd47d9fe700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4780a4f40 con 0x7fd4780a99f0 2020-11-20T11:44:44.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.008+0000 7fd47d9fe700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4780a45d0 con 0x7fd478138380 2020-11-20T11:44:44.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.008+0000 7fd47d9fe700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4780a4800 con 0x7fd478061b90 2020-11-20T11:44:44.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.008+0000 7fd4777fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd478138380 0x7fd4780a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:44.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.008+0000 7fd4777fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd478138380 0x7fd4780a3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:38568/0 (socket says 172.21.15.23:38568) 2020-11-20T11:44:44.014 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.008+0000 7fd4777fe700 1 -- 172.21.15.23:0/3246320015 learned_addr learned my addr 172.21.15.23:0/3246320015 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:44:44.014 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.008+0000 7fd4767fc700 1 --2- 172.21.15.23:0/3246320015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4780a99f0 0x7fd4780a3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:44.015 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.008+0000 7fd476ffd700 1 --2- 172.21.15.23:0/3246320015 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd478061b90 0x7fd4780a6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:44.015 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.009+0000 7fd4767fc700 1 -- 172.21.15.23:0/3246320015 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd478061b90 msgr2=0x7fd4780a6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:44.015 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.009+0000 7fd4767fc700 1 --2- 172.21.15.23:0/3246320015 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd478061b90 0x7fd4780a6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:44.016 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.009+0000 7fd4767fc700 1 -- 172.21.15.23:0/3246320015 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd478138380 msgr2=0x7fd4780a3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:44.016 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.009+0000 7fd4767fc700 1 --2- 172.21.15.23:0/3246320015 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd478138380 0x7fd4780a3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:44.016 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.009+0000 7fd4767fc700 1 -- 172.21.15.23:0/3246320015 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4780a4aa0 con 0x7fd4780a99f0 2020-11-20T11:44:44.016 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.009+0000 7fd4767fc700 1 --2- 172.21.15.23:0/3246320015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4780a99f0 0x7fd4780a3420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fd468006a40 tx=0x7fd468008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:44.016 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.009+0000 7fd45effd700 1 -- 172.21.15.23:0/3246320015 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd468014170 con 0x7fd4780a99f0 2020-11-20T11:44:44.017 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.009+0000 7fd45effd700 1 -- 172.21.15.23:0/3246320015 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4680142d0 con 0x7fd4780a99f0 2020-11-20T11:44:44.017 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.009+0000 7fd47d9fe700 1 -- 172.21.15.23:0/3246320015 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4780a7320 con 0x7fd4780a99f0 2020-11-20T11:44:44.017 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.009+0000 7fd47d9fe700 1 -- 172.21.15.23:0/3246320015 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd47809ee10 con 0x7fd4780a99f0 2020-11-20T11:44:44.017 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.009+0000 7fd45effd700 1 -- 172.21.15.23:0/3246320015 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd468014650 con 0x7fd4780a99f0 2020-11-20T11:44:44.017 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.011+0000 7fd47d9fe700 1 -- 172.21.15.23:0/3246320015 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4780a8d10 con 0x7fd4780a99f0 2020-11-20T11:44:44.018 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.011+0000 7fd45effd700 1 -- 172.21.15.23:0/3246320015 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd468023030 con 0x7fd4780a99f0 2020-11-20T11:44:44.018 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.012+0000 7fd45effd700 1 --2- 172.21.15.23:0/3246320015 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd454067d70 0x7fd45406a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:44.018 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.012+0000 7fd45effd700 1 -- 172.21.15.23:0/3246320015 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fd46806f910 con 0x7fd4780a99f0 2020-11-20T11:44:44.020 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.017+0000 7fd476ffd700 1 --2- 172.21.15.23:0/3246320015 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd454067d70 0x7fd45406a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:44.021 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.018+0000 7fd45effd700 1 -- 172.21.15.23:0/3246320015 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd468044af0 con 0x7fd4780a99f0 2020-11-20T11:44:44.021 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.018+0000 7fd476ffd700 1 --2- 172.21.15.23:0/3246320015 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd454067d70 0x7fd45406a230 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fd470006fd0 tx=0x7fd470008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:44.806 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.803+0000 7fd47d9fe700 1 -- 172.21.15.23:0/3246320015 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fd478044710 con 0x7fd4780a99f0 2020-11-20T11:44:44.808 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.804+0000 7fd45effd700 1 -- 172.21.15.23:0/3246320015 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7fd468048110 con 0x7fd4780a99f0 2020-11-20T11:44:44.814 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.808+0000 7fd47d9fe700 1 -- 172.21.15.23:0/3246320015 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd454067d70 msgr2=0x7fd45406a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:44.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.808+0000 7fd47d9fe700 1 --2- 172.21.15.23:0/3246320015 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd454067d70 0x7fd45406a230 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fd470006fd0 tx=0x7fd470008040).stop 2020-11-20T11:44:44.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.808+0000 7fd47d9fe700 1 -- 172.21.15.23:0/3246320015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4780a99f0 msgr2=0x7fd4780a3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:44.816 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.808+0000 7fd47d9fe700 1 --2- 172.21.15.23:0/3246320015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4780a99f0 0x7fd4780a3420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fd468006a40 tx=0x7fd468008040).stop 2020-11-20T11:44:44.816 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.808+0000 7fd47d9fe700 1 -- 172.21.15.23:0/3246320015 shutdown_connections 2020-11-20T11:44:44.817 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.808+0000 7fd47d9fe700 1 --2- 172.21.15.23:0/3246320015 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd478138380 0x7fd4780a3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:44.819 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.808+0000 7fd47d9fe700 1 --2- 172.21.15.23:0/3246320015 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd454067d70 0x7fd45406a230 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:44.819 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.808+0000 7fd47d9fe700 1 --2- 172.21.15.23:0/3246320015 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd478061b90 0x7fd4780a6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:44.820 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.808+0000 7fd47d9fe700 1 --2- 172.21.15.23:0/3246320015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4780a99f0 0x7fd4780a3420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:44.820 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.808+0000 7fd47d9fe700 1 -- 172.21.15.23:0/3246320015 >> 172.21.15.23:0/3246320015 conn(0x7fd478063640 msgr2=0x7fd4781399a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:44:44.821 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.808+0000 7fd47d9fe700 1 -- 172.21.15.23:0/3246320015 shutdown_connections 2020-11-20T11:44:44.821 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:44.808+0000 7fd47d9fe700 1 -- 172.21.15.23:0/3246320015 wait complete. 2020-11-20T11:44:44.865 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2020-11-20T11:44:54.866 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:44:54.867 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:44:55.005 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.000+0000 7f31843e4700 1 Processor -- start 2020-11-20T11:44:55.006 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.000+0000 7f31843e4700 1 -- start start 2020-11-20T11:44:55.006 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.001+0000 7f31843e4700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f317c0a9370 0x7f317c0a97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:55.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.001+0000 7f31843e4700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f317c0a51f0 con 0x7f317c133db0 2020-11-20T11:44:55.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.001+0000 7f31843e4700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f317c0a5490 con 0x7f317c061380 2020-11-20T11:44:55.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.001+0000 7f31843e4700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f317c0a5730 con 0x7f317c0a9370 2020-11-20T11:44:55.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.001+0000 7f3182981700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f317c133db0 0x7f317c134220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:59932/0 (socket says 172.21.15.23:59932) 2020-11-20T11:44:55.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.001+0000 7f3182981700 1 -- 172.21.15.23:0/4050291084 learned_addr learned my addr 172.21.15.23:0/4050291084 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:44:55.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.002+0000 7f318117e700 1 -- 172.21.15.23:0/4050291084 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1811319087 0 0) 0x7f317c0a51f0 con 0x7f317c133db0 2020-11-20T11:44:55.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.002+0000 7f318117e700 1 -- 172.21.15.23:0/4050291084 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3164003280 con 0x7f317c133db0 2020-11-20T11:44:55.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.002+0000 7f318197f700 1 --2- 172.21.15.23:0/4050291084 >> v2:172.21.15.23:3300/0 conn(0x7f317c0a9370 0x7f317c0a97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:55.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.003+0000 7f318197f700 1 -- 172.21.15.23:0/4050291084 >> v1:172.21.15.155:6789/0 conn(0x7f317c061380 legacy=0x7f317c0a6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:44:55.014 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.003+0000 7f318197f700 1 -- 172.21.15.23:0/4050291084 >> v1:172.21.15.23:6789/0 conn(0x7f317c133db0 legacy=0x7f317c134220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:55.014 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.003+0000 7f318197f700 1 -- 172.21.15.23:0/4050291084 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f317c0a9cc0 con 0x7f317c0a9370 2020-11-20T11:44:55.014 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.003+0000 7f318197f700 1 --2- 172.21.15.23:0/4050291084 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f317c0a9370 0x7f317c0a97d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f316c00dc70 tx=0x7f316c006e00).ready entity=mon.0 client_cookie=bf7b72900551e0e0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:55.015 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.006+0000 7f318117e700 1 -- 172.21.15.23:0/4050291084 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31740018a0 con 0x7f317c0a9370 2020-11-20T11:44:55.015 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.006+0000 7f318117e700 1 -- 172.21.15.23:0/4050291084 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f316c000e90 con 0x7f317c0a9370 2020-11-20T11:44:55.016 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.006+0000 7f318117e700 1 -- 172.21.15.23:0/4050291084 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f316c00a670 con 0x7f317c0a9370 2020-11-20T11:44:55.016 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.007+0000 7f31843e4700 1 -- 172.21.15.23:0/4050291084 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f317c0a9370 msgr2=0x7f317c0a97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:55.016 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.007+0000 7f31843e4700 1 --2- 172.21.15.23:0/4050291084 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f317c0a9370 0x7f317c0a97d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f316c00dc70 tx=0x7f316c006e00).stop 2020-11-20T11:44:55.017 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.007+0000 7f31843e4700 1 -- 172.21.15.23:0/4050291084 shutdown_connections 2020-11-20T11:44:55.017 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.007+0000 7f31843e4700 1 --2- 172.21.15.23:0/4050291084 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f317c0a9370 0x7f317c0a97d0 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0x7f316c00dc70 tx=0x7f316c006e00).stop 2020-11-20T11:44:55.018 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.007+0000 7f31843e4700 1 -- 172.21.15.23:0/4050291084 >> 172.21.15.23:0/4050291084 conn(0x7f317c0ae280 msgr2=0x7f317c0aa290 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:44:55.018 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.010+0000 7f31843e4700 1 -- 172.21.15.23:0/4050291084 shutdown_connections 2020-11-20T11:44:55.023 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.010+0000 7f31843e4700 1 -- 172.21.15.23:0/4050291084 wait complete. 2020-11-20T11:44:55.024 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.011+0000 7f31843e4700 1 Processor -- start 2020-11-20T11:44:55.024 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.011+0000 7f31843e4700 1 -- start start 2020-11-20T11:44:55.024 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.012+0000 7f31843e4700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f317c061380 0x7f317c0ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:55.025 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.012+0000 7f31843e4700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f317c133db0 0x7f317c132d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:55.025 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.012+0000 7f31843e4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f317c133270 0x7f317c126e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:55.026 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.012+0000 7f31843e4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f317c0acb40 con 0x7f317c133270 2020-11-20T11:44:55.026 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.012+0000 7f31843e4700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f317c12aa70 con 0x7f317c133db0 2020-11-20T11:44:55.027 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.012+0000 7f31843e4700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f317c127ee0 con 0x7f317c061380 2020-11-20T11:44:55.027 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.012+0000 7f318197f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f317c133db0 0x7f317c132d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:55.027 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.012+0000 7f3182981700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f317c133270 0x7f317c126e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:55.028 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.012+0000 7f318197f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f317c133db0 0x7f317c132d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:38582/0 (socket says 172.21.15.23:38582) 2020-11-20T11:44:55.028 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.012+0000 7f318197f700 1 -- 172.21.15.23:0/521139365 learned_addr learned my addr 172.21.15.23:0/521139365 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:44:55.028 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.012+0000 7f3182180700 1 --2- 172.21.15.23:0/521139365 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f317c061380 0x7f317c0ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:55.029 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.013+0000 7f3182981700 1 -- 172.21.15.23:0/521139365 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f317c061380 msgr2=0x7f317c0ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:55.029 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.013+0000 7f3182981700 1 --2- 172.21.15.23:0/521139365 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f317c061380 0x7f317c0ab7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:55.030 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.013+0000 7f3182981700 1 -- 172.21.15.23:0/521139365 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f317c133db0 msgr2=0x7f317c132d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:55.030 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.013+0000 7f3182981700 1 --2- 172.21.15.23:0/521139365 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f317c133db0 0x7f317c132d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:55.030 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.013+0000 7f3182981700 1 -- 172.21.15.23:0/521139365 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f317c128180 con 0x7f317c133270 2020-11-20T11:44:55.031 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.013+0000 7f3182981700 1 --2- 172.21.15.23:0/521139365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f317c133270 0x7f317c126e70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f317400cb10 tx=0x7f3174001130).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:55.031 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.013+0000 7f316a7fc700 1 -- 172.21.15.23:0/521139365 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3174004ad0 con 0x7f317c133270 2020-11-20T11:44:55.032 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.013+0000 7f31843e4700 1 -- 172.21.15.23:0/521139365 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f317c128420 con 0x7f317c133270 2020-11-20T11:44:55.032 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.013+0000 7f31843e4700 1 -- 172.21.15.23:0/521139365 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f317c1275a0 con 0x7f317c133270 2020-11-20T11:44:55.032 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.013+0000 7f316a7fc700 1 -- 172.21.15.23:0/521139365 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3174004e40 con 0x7f317c133270 2020-11-20T11:44:55.033 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.013+0000 7f316a7fc700 1 -- 172.21.15.23:0/521139365 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3174007170 con 0x7f317c133270 2020-11-20T11:44:55.033 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.014+0000 7f31843e4700 1 -- 172.21.15.23:0/521139365 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f317c134b20 con 0x7f317c133270 2020-11-20T11:44:55.034 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.015+0000 7f316a7fc700 1 -- 172.21.15.23:0/521139365 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f317401b060 con 0x7f317c133270 2020-11-20T11:44:55.034 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.015+0000 7f316a7fc700 1 --2- 172.21.15.23:0/521139365 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3154056d60 0x7f3154059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:55.034 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.015+0000 7f316a7fc700 1 -- 172.21.15.23:0/521139365 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f317406fb80 con 0x7f317c133270 2020-11-20T11:44:55.035 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.015+0000 7f3182180700 1 --2- 172.21.15.23:0/521139365 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3154056d60 0x7f3154059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:55.035 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.016+0000 7f3182180700 1 --2- 172.21.15.23:0/521139365 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3154056d60 0x7f3154059220 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f3178006fd0 tx=0x7f3178008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:55.036 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.019+0000 7f316a7fc700 1 -- 172.21.15.23:0/521139365 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3174044e70 con 0x7f317c133270 2020-11-20T11:44:55.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.281+0000 7f31843e4700 1 -- 172.21.15.23:0/521139365 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f317c1285b0 con 0x7f317c133270 2020-11-20T11:44:55.284 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.281+0000 7f316a7fc700 1 -- 172.21.15.23:0/521139365 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1479 (secure 0 0 0) 0x7f317c1285b0 con 0x7f317c133270 2020-11-20T11:44:55.285 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":116,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:44:55.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.288+0000 7f31843e4700 1 -- 172.21.15.23:0/521139365 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3154056d60 msgr2=0x7f3154059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:55.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.288+0000 7f31843e4700 1 --2- 172.21.15.23:0/521139365 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3154056d60 0x7f3154059220 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f3178006fd0 tx=0x7f3178008040).stop 2020-11-20T11:44:55.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.288+0000 7f31843e4700 1 -- 172.21.15.23:0/521139365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f317c133270 msgr2=0x7f317c126e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:55.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.288+0000 7f31843e4700 1 --2- 172.21.15.23:0/521139365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f317c133270 0x7f317c126e70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f317400cb10 tx=0x7f3174001130).stop 2020-11-20T11:44:55.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.289+0000 7f31843e4700 1 -- 172.21.15.23:0/521139365 shutdown_connections 2020-11-20T11:44:55.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.289+0000 7f31843e4700 1 --2- 172.21.15.23:0/521139365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f317c133270 0x7f317c126e70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:55.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.289+0000 7f31843e4700 1 --2- 172.21.15.23:0/521139365 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3154056d60 0x7f3154059220 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:55.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.289+0000 7f31843e4700 1 --2- 172.21.15.23:0/521139365 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f317c061380 0x7f317c0ab7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:55.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.289+0000 7f31843e4700 1 --2- 172.21.15.23:0/521139365 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f317c133db0 0x7f317c132d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:55.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.289+0000 7f31843e4700 1 -- 172.21.15.23:0/521139365 >> 172.21.15.23:0/521139365 conn(0x7f317c0ae280 msgr2=0x7f317c068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:44:55.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.289+0000 7f31843e4700 1 -- 172.21.15.23:0/521139365 shutdown_connections 2020-11-20T11:44:55.294 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.289+0000 7f31843e4700 1 -- 172.21.15.23:0/521139365 wait complete. 2020-11-20T11:44:55.306 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:44:55.306 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2020-11-20T11:44:55.307 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:44:55.602 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.599+0000 7f3993c37700 1 Processor -- start 2020-11-20T11:44:55.603 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.600+0000 7f3993c37700 1 -- start start 2020-11-20T11:44:55.604 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.600+0000 7f3993c37700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f398c0aa0a0 0x7f398c139700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:55.604 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.600+0000 7f3993c37700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f398c0aabe0 con 0x7f398c0608c0 2020-11-20T11:44:55.604 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.600+0000 7f3993c37700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f398c0aae80 con 0x7f398c060ca0 2020-11-20T11:44:55.605 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.600+0000 7f3993c37700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f398c0ab300 con 0x7f398c0aa0a0 2020-11-20T11:44:55.605 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.600+0000 7f39919d3700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f398c0608c0 0x7f398c0a6d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:59942/0 (socket says 172.21.15.23:59942) 2020-11-20T11:44:55.605 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.600+0000 7f39919d3700 1 -- 172.21.15.23:0/2298469456 learned_addr learned my addr 172.21.15.23:0/2298469456 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:44:55.605 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.601+0000 7f39921d4700 1 --2- 172.21.15.23:0/2298469456 >> v2:172.21.15.155:3300/0 conn(0x7f398c0aa0a0 0x7f398c139700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:55.606 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.601+0000 7f39909d1700 1 -- 172.21.15.23:0/2298469456 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2993475854 0 0) 0x7f398c0aabe0 con 0x7f398c0608c0 2020-11-20T11:44:55.606 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.601+0000 7f39909d1700 1 -- 172.21.15.23:0/2298469456 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3974003280 con 0x7f398c0608c0 2020-11-20T11:44:55.610 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.601+0000 7f39921d4700 1 -- 172.21.15.23:0/2298469456 >> v1:172.21.15.155:6789/0 conn(0x7f398c060ca0 legacy=0x7f398c138ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:55.610 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.601+0000 7f39921d4700 1 -- 172.21.15.23:0/2298469456 >> v1:172.21.15.23:6789/0 conn(0x7f398c0608c0 legacy=0x7f398c0a6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:55.611 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.601+0000 7f39921d4700 1 -- 172.21.15.23:0/2298469456 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f398c0a26c0 con 0x7f398c0aa0a0 2020-11-20T11:44:55.611 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.601+0000 7f39921d4700 1 --2- 172.21.15.23:0/2298469456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f398c0aa0a0 0x7f398c139700 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f3984009ce0 tx=0x7f3984009220).ready entity=mon.1 client_cookie=87c98230dde0ca58 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:55.611 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.601+0000 7f39909d1700 1 -- 172.21.15.23:0/2298469456 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398400ba40 con 0x7f398c0aa0a0 2020-11-20T11:44:55.611 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.601+0000 7f39909d1700 1 -- 172.21.15.23:0/2298469456 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398400bba0 con 0x7f398c0aa0a0 2020-11-20T11:44:55.612 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.601+0000 7f39909d1700 1 -- 172.21.15.23:0/2298469456 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3984011400 con 0x7f398c0aa0a0 2020-11-20T11:44:55.612 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.602+0000 7f3993c37700 1 -- 172.21.15.23:0/2298469456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f398c0aa0a0 msgr2=0x7f398c139700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:55.612 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.602+0000 7f3993c37700 1 --2- 172.21.15.23:0/2298469456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f398c0aa0a0 0x7f398c139700 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f3984009ce0 tx=0x7f3984009220).stop 2020-11-20T11:44:55.613 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.602+0000 7f3993c37700 1 -- 172.21.15.23:0/2298469456 shutdown_connections 2020-11-20T11:44:55.613 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.602+0000 7f3993c37700 1 --2- 172.21.15.23:0/2298469456 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f398c0aa0a0 0x7f398c139700 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:55.614 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.602+0000 7f3993c37700 1 -- 172.21.15.23:0/2298469456 >> 172.21.15.23:0/2298469456 conn(0x7f398c066750 msgr2=0x7f398c0601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:44:55.614 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.602+0000 7f3993c37700 1 -- 172.21.15.23:0/2298469456 shutdown_connections 2020-11-20T11:44:55.614 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.602+0000 7f3993c37700 1 -- 172.21.15.23:0/2298469456 wait complete. 2020-11-20T11:44:55.615 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.603+0000 7f3993c37700 1 Processor -- start 2020-11-20T11:44:55.615 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.603+0000 7f3993c37700 1 -- start start 2020-11-20T11:44:55.615 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.604+0000 7f3993c37700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f398c0608c0 0x7f398c0ae7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:55.616 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.604+0000 7f3993c37700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f398c060ca0 0x7f398c09e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:55.616 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.604+0000 7f3993c37700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f398c0aa0a0 0x7f398c09d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:55.617 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.604+0000 7f3993c37700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398c09f1a0 con 0x7f398c0aa0a0 2020-11-20T11:44:55.617 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.604+0000 7f3993c37700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398c09ea40 con 0x7f398c0608c0 2020-11-20T11:44:55.617 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.604+0000 7f3993c37700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f398c09ec60 con 0x7f398c060ca0 2020-11-20T11:44:55.618 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.604+0000 7f39921d4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f398c0aa0a0 0x7f398c09d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:55.618 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.604+0000 7f39921d4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f398c0aa0a0 0x7f398c09d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45702/0 (socket says 172.21.15.23:45702) 2020-11-20T11:44:55.618 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.604+0000 7f39921d4700 1 -- 172.21.15.23:0/3760393612 learned_addr learned my addr 172.21.15.23:0/3760393612 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:44:55.619 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.604+0000 7f39919d3700 1 --2- 172.21.15.23:0/3760393612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f398c0608c0 0x7f398c0ae7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:55.619 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.604+0000 7f39911d2700 1 --2- 172.21.15.23:0/3760393612 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f398c060ca0 0x7f398c09e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:55.620 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.604+0000 7f39921d4700 1 -- 172.21.15.23:0/3760393612 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f398c060ca0 msgr2=0x7f398c09e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:55.620 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.604+0000 7f39921d4700 1 --2- 172.21.15.23:0/3760393612 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f398c060ca0 0x7f398c09e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:55.620 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.604+0000 7f39921d4700 1 -- 172.21.15.23:0/3760393612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f398c0608c0 msgr2=0x7f398c0ae7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:55.621 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.604+0000 7f39921d4700 1 --2- 172.21.15.23:0/3760393612 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f398c0608c0 0x7f398c0ae7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:55.621 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.604+0000 7f39921d4700 1 -- 172.21.15.23:0/3760393612 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f398c09ef00 con 0x7f398c0aa0a0 2020-11-20T11:44:55.621 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.604+0000 7f39921d4700 1 --2- 172.21.15.23:0/3760393612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f398c0aa0a0 0x7f398c09d820 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f3984009260 tx=0x7f3984016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:55.622 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.604+0000 7f3979ffb700 1 -- 172.21.15.23:0/3760393612 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3984010a90 con 0x7f398c0aa0a0 2020-11-20T11:44:55.622 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.604+0000 7f3979ffb700 1 -- 172.21.15.23:0/3760393612 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3984010bf0 con 0x7f398c0aa0a0 2020-11-20T11:44:55.622 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.604+0000 7f3979ffb700 1 -- 172.21.15.23:0/3760393612 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398401d480 con 0x7f398c0aa0a0 2020-11-20T11:44:55.622 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.604+0000 7f3993c37700 1 -- 172.21.15.23:0/3760393612 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f398c0a09c0 con 0x7f398c0aa0a0 2020-11-20T11:44:55.622 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.605+0000 7f3993c37700 1 -- 172.21.15.23:0/3760393612 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f398c09dd10 con 0x7f398c0aa0a0 2020-11-20T11:44:55.623 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.605+0000 7f3993c37700 1 --2- 172.21.15.23:0/3760393612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f398c133680 0x7f398c133a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:55.623 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.605+0000 7f3993c37700 1 -- 172.21.15.23:0/3760393612 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398c09d580 con 0x7f398c133680 2020-11-20T11:44:55.623 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.606+0000 7f39919d3700 1 --2- 172.21.15.23:0/3760393612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f398c133680 0x7f398c133a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:55.623 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.606+0000 7f39919d3700 1 -- 172.21.15.23:0/3760393612 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f398c0a4fa0 con 0x7f398c133680 2020-11-20T11:44:55.623 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.606+0000 7f3979ffb700 1 -- 172.21.15.23:0/3760393612 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3984017070 con 0x7f398c0aa0a0 2020-11-20T11:44:55.624 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.606+0000 7f39919d3700 1 --2- 172.21.15.23:0/3760393612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f398c133680 0x7f398c133a60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f398800f820 tx=0x7f3988009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:55.624 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.606+0000 7f3979ffb700 1 --2- 172.21.15.23:0/3760393612 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3964056d60 0x7f3964059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:55.624 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.606+0000 7f3979ffb700 1 -- 172.21.15.23:0/3760393612 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f3984072980 con 0x7f398c0aa0a0 2020-11-20T11:44:55.624 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.606+0000 7f3979ffb700 1 -- 172.21.15.23:0/3760393612 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39880070d0 con 0x7f398c133680 2020-11-20T11:44:55.624 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.606+0000 7f39911d2700 1 --2- 172.21.15.23:0/3760393612 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3964056d60 0x7f3964059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:55.625 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.606+0000 7f3979ffb700 1 -- 172.21.15.23:0/3760393612 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f3988007230 con 0x7f398c133680 2020-11-20T11:44:55.625 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.606+0000 7f3979ffb700 1 -- 172.21.15.23:0/3760393612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f398c133680 msgr2=0x7f398c133a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:55.625 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.606+0000 7f3979ffb700 1 --2- 172.21.15.23:0/3760393612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f398c133680 0x7f398c133a60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f398800f820 tx=0x7f3988009040).stop 2020-11-20T11:44:55.625 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.606+0000 7f3979ffb700 1 -- 172.21.15.23:0/3760393612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f398c133680 msgr2=0x7f398c133a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:44:55.625 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.606+0000 7f3979ffb700 1 --2- 172.21.15.23:0/3760393612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f398c133680 0x7f398c133a60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:55.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.607+0000 7f39911d2700 1 --2- 172.21.15.23:0/3760393612 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3964056d60 0x7f3964059220 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f398c0a6010 tx=0x7f397c009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:55.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.616+0000 7f3993c37700 1 --2- 172.21.15.23:0/3760393612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f398c025aa0 0x7f398c025eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:55.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.616+0000 7f3993c37700 1 -- 172.21.15.23:0/3760393612 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398c0ad7a0 con 0x7f398c025aa0 2020-11-20T11:44:55.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.617+0000 7f39919d3700 1 --2- 172.21.15.23:0/3760393612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f398c025aa0 0x7f398c025eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:55.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.617+0000 7f39919d3700 1 -- 172.21.15.23:0/3760393612 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f398c044710 con 0x7f398c025aa0 2020-11-20T11:44:55.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.617+0000 7f39919d3700 1 --2- 172.21.15.23:0/3760393612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f398c025aa0 0x7f398c025eb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f398800f820 tx=0x7f3988018000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:55.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.620+0000 7f3979ffb700 1 -- 172.21.15.23:0/3760393612 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398800d970 con 0x7f398c025aa0 2020-11-20T11:44:55.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.620+0000 7f3979ffb700 1 -- 172.21.15.23:0/3760393612 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4447 (secure 0 0 0) 0x7f398800dad0 con 0x7f398c025aa0 2020-11-20T11:44:55.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.620+0000 7f3979ffb700 1 -- 172.21.15.23:0/3760393612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f398c025aa0 msgr2=0x7f398c025eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:55.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.620+0000 7f3979ffb700 1 --2- 172.21.15.23:0/3760393612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f398c025aa0 0x7f398c025eb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f398800f820 tx=0x7f3988018000).stop 2020-11-20T11:44:55.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.620+0000 7f3979ffb700 1 -- 172.21.15.23:0/3760393612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f398c025aa0 msgr2=0x7f398c025eb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:44:55.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.620+0000 7f3979ffb700 1 --2- 172.21.15.23:0/3760393612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f398c025aa0 0x7f398c025eb0 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0x7f398800f820 tx=0x7f3988018000).stop 2020-11-20T11:44:55.629 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:44:55.629 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:44:55.629 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:44:55.630 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:44:55.630 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 116, 2020-11-20T11:44:55.630 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:44:55.630 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:44:55.631 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:44:55.631 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:44:55.631 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:55.631 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 15, 2020-11-20T11:44:55.631 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:44:55.632 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:44:55.632 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:44:55.632 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:44:55.632 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:44:55.632 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:44:55.633 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:44:55.633 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:44:55.633 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:44:55.633 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:44:55.633 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:44:55.634 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:55.634 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:44:55.634 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:44:55.634 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:44:55.634 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:44:55.635 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:44:55.635 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:44:55.635 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:44:55.635 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:44:55.635 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:44:55.636 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:44:55.636 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:55.636 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:55.636 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:44:55.636 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:44:55.637 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:44:55.637 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:44:55.637 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:44:55.637 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:44:55.637 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:44:55.638 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:44:55.638 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:44:55.638 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:44:55.638 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:44:55.638 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:44:55.639 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:44:55.639 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:44:55.639 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:44:55.640 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:44:55.640 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:44:55.640 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:44:55.641 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:44:55.641 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:44:55.641 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:44:55.642 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:44:55.642 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:44:55.643 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:55.643 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:44:55.643 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:55.644 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:44:55.644 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:55.644 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:44:55.645 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:44:55.645 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:44:55.645 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:44:55.646 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:55.646 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:44:55.646 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:44:55.647 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:55.647 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:55.647 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:55.648 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:44:55.648 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:44:55.648 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:55.648 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:55.648 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:55.649 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:55.649 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:44:55.649 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:44:55.650 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:44:55.650 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:44:55.650 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:44:55.650 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:55.650 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:55.651 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:44:55.651 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:44:55.651 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:44:55.651 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:44:55.651 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:55.651 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:44:55.652 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:44:55.652 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:55.652 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:55.652 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:55.652 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:44:55.653 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:44:55.653 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:55.653 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:55.653 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:55.654 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:55.654 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:44:55.654 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:44:55.654 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:44:55.654 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:44:55.655 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:44:55.655 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:55.655 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:55.655 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:44:55.655 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:44:55.656 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:44:55.656 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:44:55.656 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:55.656 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:44:55.657 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:44:55.657 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:55.657 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:55.657 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:55.657 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:44:55.658 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:44:55.658 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:55.658 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:55.658 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:55.658 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:55.659 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:44:55.659 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:44:55.659 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:44:55.659 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:44:55.659 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:44:55.660 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:55.660 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:55.660 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:55.660 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:44:55.660 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:44:55.660 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:55.661 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:44:55.661 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:44:55.661 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:44:55.661 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:55.662 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:55.662 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:44:55.662 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:55.662 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:44:55.663 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:44:55.663 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:44:55.663 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:55.663 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:55.663 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:44:55.663 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:55.664 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:44:55.664 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:44:55.664 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:44:55.664 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:55.664 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:55.665 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:55.665 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:44:55.665 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:44:55.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.624+0000 7f3993c37700 1 -- 172.21.15.23:0/3760393612 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3964056d60 msgr2=0x7f3964059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:55.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.624+0000 7f3993c37700 1 --2- 172.21.15.23:0/3760393612 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3964056d60 0x7f3964059220 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f398c0a6010 tx=0x7f397c009040).stop 2020-11-20T11:44:55.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.624+0000 7f3993c37700 1 -- 172.21.15.23:0/3760393612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f398c0aa0a0 msgr2=0x7f398c09d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:55.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.624+0000 7f3993c37700 1 --2- 172.21.15.23:0/3760393612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f398c0aa0a0 0x7f398c09d820 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f3984009260 tx=0x7f3984016000).stop 2020-11-20T11:44:55.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.624+0000 7f39921d4700 1 -- 172.21.15.23:0/3760393612 reap_dead start 2020-11-20T11:44:55.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.624+0000 7f39921d4700 1 -- 172.21.15.23:0/3760393612 reap_dead start 2020-11-20T11:44:55.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.626+0000 7f3993c37700 1 -- 172.21.15.23:0/3760393612 shutdown_connections 2020-11-20T11:44:55.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.626+0000 7f3993c37700 1 -- 172.21.15.23:0/3760393612 >> 172.21.15.23:0/3760393612 conn(0x7f398c066750 msgr2=0x7f398c0a74a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:44:55.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.626+0000 7f3993c37700 1 -- 172.21.15.23:0/3760393612 shutdown_connections 2020-11-20T11:44:55.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.626+0000 7f3993c37700 1 -- 172.21.15.23:0/3760393612 wait complete. 2020-11-20T11:44:55.669 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:44:55.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.869+0000 7f3598abc700 1 Processor -- start 2020-11-20T11:44:55.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.869+0000 7f3598abc700 1 -- start start 2020-11-20T11:44:55.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.870+0000 7f3598abc700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3594136ea0 0x7f35940a8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:55.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.870+0000 7f3598abc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f35940a8fc0 0x7f3594141810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:55.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.870+0000 7f3598abc700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35940a4970 con 0x7f3594061a10 2020-11-20T11:44:55.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.870+0000 7f3598abc700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35940a5ef0 con 0x7f35940a8fc0 2020-11-20T11:44:55.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.870+0000 7f3598abc700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35940a5490 con 0x7f3594136ea0 2020-11-20T11:44:55.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.870+0000 7f3592d9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f35940a8fc0 0x7f3594141810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:55.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.870+0000 7f3592d9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f35940a8fc0 0x7f3594141810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45712/0 (socket says 172.21.15.23:45712) 2020-11-20T11:44:55.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.870+0000 7f3592d9d700 1 -- 172.21.15.23:0/1408271225 learned_addr learned my addr 172.21.15.23:0/1408271225 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:44:55.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.871+0000 7f3592d9d700 1 -- 172.21.15.23:0/1408271225 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3594136ea0 msgr2=0x7f35940a8ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:44:55.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.871+0000 7f3592d9d700 1 --2- 172.21.15.23:0/1408271225 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3594136ea0 0x7f35940a8ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:44:55.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.871+0000 7f3592d9d700 1 -- 172.21.15.23:0/1408271225 >> v1:172.21.15.155:6789/0 conn(0x7f3594061a10 legacy=0x7f35940a6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:44:55.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.871+0000 7f3592d9d700 1 -- 172.21.15.23:0/1408271225 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35940a60e0 con 0x7f35940a8fc0 2020-11-20T11:44:55.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.871+0000 7f3592d9d700 1 --2- 172.21.15.23:0/1408271225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35940a8fc0 0x7f3594141810 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f3584009fa0 tx=0x7f3584006ef0).ready entity=mon.0 client_cookie=5f9bd41c60974e68 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:55.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.873+0000 7f359159a700 1 -- 172.21.15.23:0/1408271225 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3584013a90 con 0x7f35940a8fc0 2020-11-20T11:44:55.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.873+0000 7f359159a700 1 -- 172.21.15.23:0/1408271225 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3584000e90 con 0x7f35940a8fc0 2020-11-20T11:44:55.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.873+0000 7f359159a700 1 -- 172.21.15.23:0/1408271225 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3584013e00 con 0x7f35940a8fc0 2020-11-20T11:44:55.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.874+0000 7f3598abc700 1 -- 172.21.15.23:0/1408271225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35940a8fc0 msgr2=0x7f3594141810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:55.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.874+0000 7f3598abc700 1 --2- 172.21.15.23:0/1408271225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35940a8fc0 0x7f3594141810 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f3584009fa0 tx=0x7f3584006ef0).stop 2020-11-20T11:44:55.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.875+0000 7f3598abc700 1 -- 172.21.15.23:0/1408271225 shutdown_connections 2020-11-20T11:44:55.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.875+0000 7f3598abc700 1 --2- 172.21.15.23:0/1408271225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35940a8fc0 0x7f3594141810 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:55.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.875+0000 7f3598abc700 1 --2- 172.21.15.23:0/1408271225 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3594136ea0 0x7f35940a8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:44:55.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.875+0000 7f3598abc700 1 -- 172.21.15.23:0/1408271225 >> 172.21.15.23:0/1408271225 conn(0x7f3594060c80 msgr2=0x7f35940acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:44:55.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.895+0000 7f3598abc700 1 -- 172.21.15.23:0/1408271225 shutdown_connections 2020-11-20T11:44:55.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.895+0000 7f3598abc700 1 -- 172.21.15.23:0/1408271225 wait complete. 2020-11-20T11:44:55.900 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.896+0000 7f3598abc700 1 Processor -- start 2020-11-20T11:44:55.903 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.897+0000 7f3598abc700 1 -- start start 2020-11-20T11:44:55.903 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.897+0000 7f3598abc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3594061a10 0x7f359413ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:55.903 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.897+0000 7f3598abc700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35940a8fc0 0x7f359409d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:55.903 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.897+0000 7f3598abc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3594136ea0 0x7f3594067440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:55.904 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.897+0000 7f3598abc700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359409d990 con 0x7f3594136ea0 2020-11-20T11:44:55.904 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.897+0000 7f3598abc700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359409dbb0 con 0x7f3594061a10 2020-11-20T11:44:55.904 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.897+0000 7f3598abc700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3594066d00 con 0x7f35940a8fc0 2020-11-20T11:44:55.904 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.898+0000 7f3591d9b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35940a8fc0 0x7f359409d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:55.904 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.898+0000 7f359259c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3594061a10 0x7f359413ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:55.905 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.898+0000 7f3591d9b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35940a8fc0 0x7f359409d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:55954/0 (socket says 172.21.15.23:55954) 2020-11-20T11:44:55.905 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.898+0000 7f3591d9b700 1 -- 172.21.15.23:0/2561751483 learned_addr learned my addr 172.21.15.23:0/2561751483 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:44:55.905 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.898+0000 7f3591d9b700 1 -- 172.21.15.23:0/2561751483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3594061a10 msgr2=0x7f359413ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:55.905 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.898+0000 7f3591d9b700 1 --2- 172.21.15.23:0/2561751483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3594061a10 0x7f359413ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:55.906 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.898+0000 7f3591d9b700 1 -- 172.21.15.23:0/2561751483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3594136ea0 msgr2=0x7f3594067440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:55.906 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.898+0000 7f3591d9b700 1 --2- 172.21.15.23:0/2561751483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3594136ea0 0x7f3594067440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:44:55.906 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.898+0000 7f3591d9b700 1 -- 172.21.15.23:0/2561751483 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3594067930 con 0x7f35940a8fc0 2020-11-20T11:44:55.906 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.899+0000 7f3591d9b700 1 --2- 172.21.15.23:0/2561751483 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35940a8fc0 0x7f359409d4a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f358c00bb10 tx=0x7f358c0091a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:55.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.900+0000 7f35827fc700 1 -- 172.21.15.23:0/2561751483 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c00c800 con 0x7f35940a8fc0 2020-11-20T11:44:55.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.900+0000 7f35827fc700 1 -- 172.21.15.23:0/2561751483 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f358c00c980 con 0x7f35940a8fc0 2020-11-20T11:44:55.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.900+0000 7f35827fc700 1 -- 172.21.15.23:0/2561751483 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c00cb30 con 0x7f35940a8fc0 2020-11-20T11:44:55.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.900+0000 7f3598abc700 1 -- 172.21.15.23:0/2561751483 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f359415a990 con 0x7f35940a8fc0 2020-11-20T11:44:55.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.900+0000 7f3598abc700 1 -- 172.21.15.23:0/2561751483 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f359409f780 con 0x7f35940a8fc0 2020-11-20T11:44:55.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.903+0000 7f35827fc700 1 -- 172.21.15.23:0/2561751483 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f358c01c070 con 0x7f35940a8fc0 2020-11-20T11:44:55.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.904+0000 7f35827fc700 1 --2- 172.21.15.23:0/2561751483 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3570056d60 0x7f3570059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:55.909 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.904+0000 7f35827fc700 1 -- 172.21.15.23:0/2561751483 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f358c06f920 con 0x7f35940a8fc0 2020-11-20T11:44:55.909 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.904+0000 7f359259c700 1 --2- 172.21.15.23:0/2561751483 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3570056d60 0x7f3570059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:55.909 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.905+0000 7f359259c700 1 --2- 172.21.15.23:0/2561751483 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3570056d60 0x7f3570059220 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f3588006fd0 tx=0x7f3588008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:55.917 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.914+0000 7f3598abc700 1 --2- 172.21.15.23:0/2561751483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3568002a50 0x7f3568004f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:55.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.914+0000 7f3598abc700 1 -- 172.21.15.23:0/2561751483 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3568007bb0 con 0x7f3568002a50 2020-11-20T11:44:55.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.915+0000 7f3592d9d700 1 --2- 172.21.15.23:0/2561751483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3568002a50 0x7f3568004f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:55.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.915+0000 7f3592d9d700 1 -- 172.21.15.23:0/2561751483 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3568007d70 con 0x7f3568002a50 2020-11-20T11:44:55.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.915+0000 7f3592d9d700 1 --2- 172.21.15.23:0/2561751483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3568002a50 0x7f3568004f10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f35940643d0 tx=0x7f358401a000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:55.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.916+0000 7f35827fc700 1 -- 172.21.15.23:0/2561751483 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3584020070 con 0x7f3568002a50 2020-11-20T11:44:55.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.916+0000 7f35827fc700 1 -- 172.21.15.23:0/2561751483 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f3584013690 con 0x7f3568002a50 2020-11-20T11:44:55.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.916+0000 7f35827fc700 1 -- 172.21.15.23:0/2561751483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3568002a50 msgr2=0x7f3568004f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:55.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.916+0000 7f35827fc700 1 --2- 172.21.15.23:0/2561751483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3568002a50 0x7f3568004f10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f35940643d0 tx=0x7f358401a000).stop 2020-11-20T11:44:55.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.916+0000 7f35827fc700 1 -- 172.21.15.23:0/2561751483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3568002a50 msgr2=0x7f3568004f10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:44:55.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.916+0000 7f35827fc700 1 --2- 172.21.15.23:0/2561751483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3568002a50 0x7f3568004f10 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:55.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.932+0000 7f3598abc700 1 --2- 172.21.15.23:0/2561751483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3568006c00 0x7f3568007090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:55.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.932+0000 7f3598abc700 1 -- 172.21.15.23:0/2561751483 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35680076a0 con 0x7f3568006c00 2020-11-20T11:44:55.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.933+0000 7f3592d9d700 1 --2- 172.21.15.23:0/2561751483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3568006c00 0x7f3568007090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:55.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.933+0000 7f3592d9d700 1 -- 172.21.15.23:0/2561751483 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3568007860 con 0x7f3568006c00 2020-11-20T11:44:55.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.933+0000 7f3592d9d700 1 --2- 172.21.15.23:0/2561751483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3568006c00 0x7f3568007090 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f3568007d40 tx=0x7f358401c000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:55.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.934+0000 7f35827fc700 1 -- 172.21.15.23:0/2561751483 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3584012820 con 0x7f3568006c00 2020-11-20T11:44:55.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.934+0000 7f35827fc700 1 -- 172.21.15.23:0/2561751483 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4785 (secure 0 0 0) 0x7f358400cc10 con 0x7f3568006c00 2020-11-20T11:44:55.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.934+0000 7f35827fc700 1 -- 172.21.15.23:0/2561751483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3568006c00 msgr2=0x7f3568007090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:55.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.934+0000 7f35827fc700 1 --2- 172.21.15.23:0/2561751483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3568006c00 0x7f3568007090 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f3568007d40 tx=0x7f358401c000).stop 2020-11-20T11:44:55.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.934+0000 7f35827fc700 1 -- 172.21.15.23:0/2561751483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3568006c00 msgr2=0x7f3568007090 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:44:55.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.934+0000 7f35827fc700 1 --2- 172.21.15.23:0/2561751483 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3568006c00 0x7f3568007090 secure :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0x7f3568007d40 tx=0x7f358401c000).stop 2020-11-20T11:44:55.939 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:44:55.939 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:44:55.939 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:44:55.940 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:44:55.940 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 116, 2020-11-20T11:44:55.940 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:44:55.940 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:44:55.941 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:44:55.941 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:44:55.941 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:55.941 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 15, 2020-11-20T11:44:55.941 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:44:55.942 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:44:55.942 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:44:55.942 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:44:55.942 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:44:55.942 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:44:55.943 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:44:55.943 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:44:55.943 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:44:55.943 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:44:55.943 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:44:55.944 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:55.944 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:44:55.944 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:44:55.944 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:44:55.944 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:44:55.945 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:44:55.945 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:44:55.945 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:44:55.945 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:44:55.945 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:44:55.946 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:44:55.946 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:55.946 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:55.946 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:44:55.946 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:44:55.947 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:44:55.947 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:44:55.947 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:44:55.947 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:44:55.947 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:44:55.948 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:44:55.948 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:44:55.948 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:44:55.948 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:44:55.948 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:44:55.949 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:44:55.949 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:44:55.949 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:44:55.949 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:44:55.950 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:44:55.950 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:44:55.950 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:44:55.950 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:44:55.950 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:44:55.951 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:44:55.951 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:44:55.951 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:55.951 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:44:55.951 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:55.952 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:44:55.952 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:55.952 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:44:55.952 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:44:55.952 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:44:55.953 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:44:55.953 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:55.953 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:44:55.953 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:44:55.953 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:55.954 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:55.954 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:55.954 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:44:55.954 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:44:55.954 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:55.955 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:55.955 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:55.955 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:55.955 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:44:55.955 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:44:55.956 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:44:55.956 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:44:55.956 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:44:55.956 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:55.956 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:55.957 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:44:55.957 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:44:55.957 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:44:55.957 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:44:55.957 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:55.958 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:44:55.958 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:44:55.958 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:55.958 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:55.959 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:55.959 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:44:55.959 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:44:55.959 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:55.959 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:55.960 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:55.960 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:55.960 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:44:55.960 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:44:55.960 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:44:55.961 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:44:55.961 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:44:55.961 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:55.961 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:55.961 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:44:55.962 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:44:55.962 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:44:55.962 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:44:55.962 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:55.962 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:44:55.963 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:44:55.963 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:55.963 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:55.963 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:55.963 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:44:55.964 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:44:55.964 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:55.964 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:55.964 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:55.964 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:55.965 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:44:55.965 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:44:55.965 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:44:55.965 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:44:55.965 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:44:55.966 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:55.966 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:55.966 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:55.966 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:44:55.966 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:44:55.967 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:55.967 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:44:55.967 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:44:55.967 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:44:55.967 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:55.968 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:55.968 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:44:55.968 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:55.968 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:44:55.968 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:44:55.969 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:44:55.969 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:55.969 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:55.969 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:44:55.970 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:55.970 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:44:55.970 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:44:55.970 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:44:55.970 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:55.971 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:55.971 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:44:55.971 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:55.971 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:44:55.971 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:44:55.972 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:44:55.972 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:55.972 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:55.972 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:44:55.972 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:55.972 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:44:55.973 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:44:55.973 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:44:55.973 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:55.973 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:55.973 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:55.974 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:44:55.974 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:44:55.975 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.945+0000 7f3598abc700 1 -- 172.21.15.23:0/2561751483 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3570056d60 msgr2=0x7f3570059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:55.976 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.945+0000 7f3598abc700 1 --2- 172.21.15.23:0/2561751483 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3570056d60 0x7f3570059220 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f3588006fd0 tx=0x7f3588008040).stop 2020-11-20T11:44:55.976 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.945+0000 7f3592d9d700 1 -- 172.21.15.23:0/2561751483 reap_dead start 2020-11-20T11:44:55.976 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.945+0000 7f3598abc700 1 -- 172.21.15.23:0/2561751483 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35940a8fc0 msgr2=0x7f359409d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:55.976 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.945+0000 7f3598abc700 1 --2- 172.21.15.23:0/2561751483 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35940a8fc0 0x7f359409d4a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f358c00bb10 tx=0x7f358c0091a0).stop 2020-11-20T11:44:55.977 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.947+0000 7f3598abc700 1 -- 172.21.15.23:0/2561751483 shutdown_connections 2020-11-20T11:44:55.977 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.947+0000 7f3598abc700 1 --2- 172.21.15.23:0/2561751483 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35940a8fc0 0x7f359409d4a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:55.977 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.947+0000 7f3598abc700 1 -- 172.21.15.23:0/2561751483 >> 172.21.15.23:0/2561751483 conn(0x7f3594060c80 msgr2=0x7f35940a77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:44:55.977 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.947+0000 7f3598abc700 1 -- 172.21.15.23:0/2561751483 shutdown_connections 2020-11-20T11:44:55.977 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:55.947+0000 7f3598abc700 1 -- 172.21.15.23:0/2561751483 wait complete. 2020-11-20T11:44:55.978 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:44:56.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.137+0000 7f6d39088700 1 Processor -- start 2020-11-20T11:44:56.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.137+0000 7f6d39088700 1 -- start start 2020-11-20T11:44:56.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.138+0000 7f6d39088700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6d34060ca0 0x7f6d3413eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:56.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.138+0000 7f6d39088700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d340a1ac0 0x7f6d340aabc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:56.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.138+0000 7f6d39088700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d341386f0 con 0x7f6d340608c0 2020-11-20T11:44:56.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.138+0000 7f6d39088700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d3413e1b0 con 0x7f6d34060ca0 2020-11-20T11:44:56.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.138+0000 7f6d39088700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3413e450 con 0x7f6d340a1ac0 2020-11-20T11:44:56.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.138+0000 7f6d3259c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6d34060ca0 0x7f6d3413eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:56.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.138+0000 7f6d3259c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6d34060ca0 0x7f6d3413eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45728/0 (socket says 172.21.15.23:45728) 2020-11-20T11:44:56.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.138+0000 7f6d3259c700 1 -- 172.21.15.23:0/2372772210 learned_addr learned my addr 172.21.15.23:0/2372772210 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:44:56.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.138+0000 7f6d3359e700 1 --2- 172.21.15.23:0/2372772210 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d340a1ac0 0x7f6d340aabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:56.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.138+0000 7f6d3259c700 1 -- 172.21.15.23:0/2372772210 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d340a1ac0 msgr2=0x7f6d340aabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:44:56.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.138+0000 7f6d3259c700 1 --2- 172.21.15.23:0/2372772210 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d340a1ac0 0x7f6d340aabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:56.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.138+0000 7f6d3259c700 1 -- 172.21.15.23:0/2372772210 >> v1:172.21.15.155:6789/0 conn(0x7f6d340608c0 legacy=0x7f6d34137fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:56.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.138+0000 7f6d3259c700 1 -- 172.21.15.23:0/2372772210 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d3413e6f0 con 0x7f6d34060ca0 2020-11-20T11:44:56.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.138+0000 7f6d3259c700 1 --2- 172.21.15.23:0/2372772210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d34060ca0 0x7f6d3413eb90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f6d24009a10 tx=0x7f6d24006ef0).ready entity=mon.0 client_cookie=1413ae7edffbc3f5 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:56.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.138+0000 7f6d31d9b700 1 -- 172.21.15.23:0/2372772210 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2400b860 con 0x7f6d34060ca0 2020-11-20T11:44:56.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.139+0000 7f6d31d9b700 1 -- 172.21.15.23:0/2372772210 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d24000e90 con 0x7f6d34060ca0 2020-11-20T11:44:56.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.139+0000 7f6d31d9b700 1 -- 172.21.15.23:0/2372772210 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2400bbd0 con 0x7f6d34060ca0 2020-11-20T11:44:56.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.139+0000 7f6d39088700 1 -- 172.21.15.23:0/2372772210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d34060ca0 msgr2=0x7f6d3413eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:56.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.139+0000 7f6d39088700 1 --2- 172.21.15.23:0/2372772210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d34060ca0 0x7f6d3413eb90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f6d24009a10 tx=0x7f6d24006ef0).stop 2020-11-20T11:44:56.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.139+0000 7f6d39088700 1 -- 172.21.15.23:0/2372772210 shutdown_connections 2020-11-20T11:44:56.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.139+0000 7f6d39088700 1 --2- 172.21.15.23:0/2372772210 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d340a1ac0 0x7f6d340aabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:56.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.139+0000 7f6d39088700 1 --2- 172.21.15.23:0/2372772210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d34060ca0 0x7f6d3413eb90 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:56.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.139+0000 7f6d39088700 1 -- 172.21.15.23:0/2372772210 >> 172.21.15.23:0/2372772210 conn(0x7f6d34066750 msgr2=0x7f6d340601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:44:56.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.139+0000 7f6d39088700 1 -- 172.21.15.23:0/2372772210 shutdown_connections 2020-11-20T11:44:56.148 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.139+0000 7f6d39088700 1 -- 172.21.15.23:0/2372772210 wait complete. 2020-11-20T11:44:56.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.140+0000 7f6d39088700 1 Processor -- start 2020-11-20T11:44:56.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.140+0000 7f6d39088700 1 -- start start 2020-11-20T11:44:56.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.141+0000 7f6d39088700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d340608c0 0x7f6d34140040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:56.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.141+0000 7f6d39088700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d34060ca0 0x7f6d3413a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.141+0000 7f6d39088700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d340a1ac0 0x7f6d34140c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.141+0000 7f6d39088700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3413ad40 con 0x7f6d34060ca0 2020-11-20T11:44:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.141+0000 7f6d39088700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3413f6d0 con 0x7f6d340608c0 2020-11-20T11:44:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.141+0000 7f6d39088700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3413f900 con 0x7f6d340a1ac0 2020-11-20T11:44:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.141+0000 7f6d3259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d34060ca0 0x7f6d3413a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:56.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.141+0000 7f6d32d9d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d340608c0 0x7f6d34140040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:56.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.141+0000 7f6d3259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d34060ca0 0x7f6d3413a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45734/0 (socket says 172.21.15.23:45734) 2020-11-20T11:44:56.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.141+0000 7f6d3259c700 1 -- 172.21.15.23:0/3696378570 learned_addr learned my addr 172.21.15.23:0/3696378570 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:44:56.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.141+0000 7f6d3359e700 1 --2- 172.21.15.23:0/3696378570 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d340a1ac0 0x7f6d34140c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:56.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.141+0000 7f6d3259c700 1 -- 172.21.15.23:0/3696378570 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d340a1ac0 msgr2=0x7f6d34140c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:56.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.141+0000 7f6d3259c700 1 --2- 172.21.15.23:0/3696378570 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d340a1ac0 0x7f6d34140c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:56.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.141+0000 7f6d3259c700 1 -- 172.21.15.23:0/3696378570 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d340608c0 msgr2=0x7f6d34140040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:56.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.141+0000 7f6d3259c700 1 --2- 172.21.15.23:0/3696378570 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6d340608c0 0x7f6d34140040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:56.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.141+0000 7f6d3259c700 1 -- 172.21.15.23:0/3696378570 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d3413fba0 con 0x7f6d34060ca0 2020-11-20T11:44:56.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.141+0000 7f6d3259c700 1 --2- 172.21.15.23:0/3696378570 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d34060ca0 0x7f6d3413a850 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f6d2400c010 tx=0x7f6d24016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:56.155 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.142+0000 7f6d1affd700 1 -- 172.21.15.23:0/3696378570 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d24011ba0 con 0x7f6d34060ca0 2020-11-20T11:44:56.155 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.142+0000 7f6d1affd700 1 -- 172.21.15.23:0/3696378570 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d24011d00 con 0x7f6d34060ca0 2020-11-20T11:44:56.155 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.142+0000 7f6d1affd700 1 -- 172.21.15.23:0/3696378570 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2401d6b0 con 0x7f6d34060ca0 2020-11-20T11:44:56.155 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.142+0000 7f6d39088700 1 -- 172.21.15.23:0/3696378570 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d34133540 con 0x7f6d34060ca0 2020-11-20T11:44:56.155 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.142+0000 7f6d39088700 1 -- 172.21.15.23:0/3696378570 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d3413da30 con 0x7f6d34060ca0 2020-11-20T11:44:56.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.142+0000 7f6d39088700 1 --2- 172.21.15.23:0/3696378570 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d34139e10 0x7f6d3413b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:56.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.142+0000 7f6d39088700 1 -- 172.21.15.23:0/3696378570 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3413dbf0 con 0x7f6d34139e10 2020-11-20T11:44:56.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.142+0000 7f6d32d9d700 1 --2- 172.21.15.23:0/3696378570 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d34139e10 0x7f6d3413b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:56.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.143+0000 7f6d1affd700 1 -- 172.21.15.23:0/3696378570 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6d24017070 con 0x7f6d34060ca0 2020-11-20T11:44:56.156 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.143+0000 7f6d32d9d700 1 -- 172.21.15.23:0/3696378570 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d3413ed20 con 0x7f6d34139e10 2020-11-20T11:44:56.157 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.143+0000 7f6d32d9d700 1 --2- 172.21.15.23:0/3696378570 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d34139e10 0x7f6d3413b430 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f6d2800f700 tx=0x7f6d28007bd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:56.157 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.143+0000 7f6d1affd700 1 --2- 172.21.15.23:0/3696378570 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6d10056d60 0x7f6d10059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:56.157 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.143+0000 7f6d1affd700 1 -- 172.21.15.23:0/3696378570 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f6d24071d50 con 0x7f6d34060ca0 2020-11-20T11:44:56.157 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.143+0000 7f6d1affd700 1 -- 172.21.15.23:0/3696378570 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d28011590 con 0x7f6d34139e10 2020-11-20T11:44:56.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.143+0000 7f6d3359e700 1 --2- 172.21.15.23:0/3696378570 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6d10056d60 0x7f6d10059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:56.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.143+0000 7f6d1affd700 1 -- 172.21.15.23:0/3696378570 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f6d280116f0 con 0x7f6d34139e10 2020-11-20T11:44:56.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.143+0000 7f6d1affd700 1 -- 172.21.15.23:0/3696378570 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d34139e10 msgr2=0x7f6d3413b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:56.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.143+0000 7f6d1affd700 1 --2- 172.21.15.23:0/3696378570 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d34139e10 0x7f6d3413b430 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f6d2800f700 tx=0x7f6d28007bd0).stop 2020-11-20T11:44:56.158 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.143+0000 7f6d1affd700 1 -- 172.21.15.23:0/3696378570 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d34139e10 msgr2=0x7f6d3413b430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:44:56.159 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.143+0000 7f6d1affd700 1 --2- 172.21.15.23:0/3696378570 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d34139e10 0x7f6d3413b430 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:56.159 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.144+0000 7f6d3359e700 1 --2- 172.21.15.23:0/3696378570 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6d10056d60 0x7f6d10059220 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f6d34139d30 tx=0x7f6d2c008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:56.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.152+0000 7f6d39088700 1 --2- 172.21.15.23:0/3696378570 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d3413a1f0 0x7f6d3414dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:44:56.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.152+0000 7f6d39088700 1 -- 172.21.15.23:0/3696378570 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3413fd30 con 0x7f6d3413a1f0 2020-11-20T11:44:56.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.152+0000 7f6d32d9d700 1 --2- 172.21.15.23:0/3696378570 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d3413a1f0 0x7f6d3414dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:44:56.161 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.153+0000 7f6d32d9d700 1 -- 172.21.15.23:0/3696378570 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6d34066d60 con 0x7f6d3413a1f0 2020-11-20T11:44:56.162 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.153+0000 7f6d32d9d700 1 --2- 172.21.15.23:0/3696378570 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d3413a1f0 0x7f6d3414dac0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f6d28015000 tx=0x7f6d28018000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:44:56.162 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.153+0000 7f6d1affd700 1 -- 172.21.15.23:0/3696378570 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d28007240 con 0x7f6d3413a1f0 2020-11-20T11:44:56.162 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.153+0000 7f6d1affd700 1 -- 172.21.15.23:0/3696378570 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4445 (secure 0 0 0) 0x7f6d28011380 con 0x7f6d3413a1f0 2020-11-20T11:44:56.162 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.153+0000 7f6d1affd700 1 -- 172.21.15.23:0/3696378570 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d3413a1f0 msgr2=0x7f6d3414dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:56.162 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.153+0000 7f6d1affd700 1 --2- 172.21.15.23:0/3696378570 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d3413a1f0 0x7f6d3414dac0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f6d28015000 tx=0x7f6d28018000).stop 2020-11-20T11:44:56.163 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.153+0000 7f6d1affd700 1 -- 172.21.15.23:0/3696378570 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d3413a1f0 msgr2=0x7f6d3414dac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:44:56.163 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.153+0000 7f6d1affd700 1 --2- 172.21.15.23:0/3696378570 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6d3413a1f0 0x7f6d3414dac0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:44:56.163 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.154+0000 7f6d39088700 1 -- 172.21.15.23:0/3696378570 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6d10056d60 msgr2=0x7f6d10059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:56.163 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.154+0000 7f6d39088700 1 --2- 172.21.15.23:0/3696378570 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6d10056d60 0x7f6d10059220 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f6d34139d30 tx=0x7f6d2c008040).stop 2020-11-20T11:44:56.163 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.154+0000 7f6d39088700 1 -- 172.21.15.23:0/3696378570 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d34060ca0 msgr2=0x7f6d3413a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:44:56.164 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.154+0000 7f6d39088700 1 --2- 172.21.15.23:0/3696378570 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d34060ca0 0x7f6d3413a850 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f6d2400c010 tx=0x7f6d24016000).stop 2020-11-20T11:44:56.164 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.154+0000 7f6d3359e700 1 -- 172.21.15.23:0/3696378570 reap_dead start 2020-11-20T11:44:56.164 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.154+0000 7f6d3359e700 1 -- 172.21.15.23:0/3696378570 reap_dead start 2020-11-20T11:44:56.164 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.154+0000 7f6d39088700 1 -- 172.21.15.23:0/3696378570 shutdown_connections 2020-11-20T11:44:56.164 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.154+0000 7f6d39088700 1 -- 172.21.15.23:0/3696378570 >> 172.21.15.23:0/3696378570 conn(0x7f6d34066750 msgr2=0x7f6d340ad890 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:44:56.165 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.154+0000 7f6d39088700 1 -- 172.21.15.23:0/3696378570 shutdown_connections 2020-11-20T11:44:56.165 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:44:56.154+0000 7f6d39088700 1 -- 172.21.15.23:0/3696378570 wait complete. 2020-11-20T11:44:56.165 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:44:56.165 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:44:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:44:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:44:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 116, 2020-11-20T11:44:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:44:56.166 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:44:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:44:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:44:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 15, 2020-11-20T11:44:56.167 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:44:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:44:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:44:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:44:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:44:56.168 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:44:56.169 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:44:56.169 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:44:56.169 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:44:56.169 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:44:56.169 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:44:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:44:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:44:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:44:56.170 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:44:56.171 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:44:56.171 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:44:56.171 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:44:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:44:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:44:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:44:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:56.172 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:56.173 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:44:56.173 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:44:56.173 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:44:56.173 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:44:56.173 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:44:56.174 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:44:56.174 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:44:56.174 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:44:56.174 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:44:56.174 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:44:56.175 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:44:56.175 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:44:56.175 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:44:56.175 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:44:56.175 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:44:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:44:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:44:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:44:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:44:56.176 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:44:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:44:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:44:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:44:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:56.177 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:44:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:44:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:44:56.178 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:44:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:44:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:44:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:44:56.179 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:44:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:44:56.180 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:44:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:56.181 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:44:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:44:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:44:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:44:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:44:56.182 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:44:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:44:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:44:56.183 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:44:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:44:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:44:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:56.184 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:44:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:44:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:56.185 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:44:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:44:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:44:56.186 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:44:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:44:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:44:56.187 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:44:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:44:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:44:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:44:56.188 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:44:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:44:56.189 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:44:56.190 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:44:56.190 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:56.190 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:56.190 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:56.190 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:44:56.191 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:44:56.191 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:44:56.191 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:44:56.191 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:44:56.191 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:44:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:44:56.192 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:44:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:44:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:44:56.193 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:44:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:44:56.194 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:44:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:44:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:44:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:44:56.195 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:44:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:44:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:44:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:44:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:44:56.196 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:44:56.197 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:44:56.197 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:44:56.197 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:44:56.198 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2020-11-20T11:44:56.198 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2020-11-20T11:44:56.199 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2020-11-20T11:44:56.199 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2020-11-20T11:44:56.199 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2020-11-20T11:44:56.200 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T11:45:00.663 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2020-11-20T11:45:02.301 INFO:tasks.ceph.mon.c:Stopped 2020-11-20T11:45:02.301 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2020-11-20T11:45:02.302 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:45:02.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.744+0000 7ff7648de700 1 Processor -- start 2020-11-20T11:45:02.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.750+0000 7ff7648de700 1 -- start start 2020-11-20T11:45:02.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.751+0000 7ff7648de700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff7600a8110 0x7ff7600acca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:02.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.751+0000 7ff7648de700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7600a84f0 con 0x7ff760118a00 2020-11-20T11:45:02.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.751+0000 7ff7648de700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7600a5a30 con 0x7ff7600a8b30 2020-11-20T11:45:02.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.751+0000 7ff7648de700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7600a9560 con 0x7ff7600a8110 2020-11-20T11:45:02.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.751+0000 7ff75ed9d700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff7600a8110 0x7ff7600acca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:02.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.751+0000 7ff75ed9d700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff7600a8110 0x7ff7600acca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:38664/0 (socket says 172.21.15.23:38664) 2020-11-20T11:45:02.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.751+0000 7ff75ed9d700 1 -- 172.21.15.23:0/2441122394 learned_addr learned my addr 172.21.15.23:0/2441122394 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:45:02.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.752+0000 7ff75ed9d700 1 -- 172.21.15.23:0/2441122394 >> v1:172.21.15.155:6789/0 conn(0x7ff7600a8b30 legacy=0x7ff7600a9790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:45:02.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.752+0000 7ff75ed9d700 1 -- 172.21.15.23:0/2441122394 >> v1:172.21.15.23:6789/0 conn(0x7ff760118a00 legacy=0x7ff7600a9ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:45:02.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.752+0000 7ff75ed9d700 1 -- 172.21.15.23:0/2441122394 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7600a79d0 con 0x7ff7600a8110 2020-11-20T11:45:02.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.752+0000 7ff75ed9d700 1 --2- 172.21.15.23:0/2441122394 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7600a8110 0x7ff7600acca0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7ff75401a9a0 tx=0x7ff75401ad60).ready entity=mon.1 client_cookie=3500f193cde54a73 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:02.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.753+0000 7ff75dd9b700 1 -- 172.21.15.23:0/2441122394 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75401c7f0 con 0x7ff7600a8110 2020-11-20T11:45:02.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.753+0000 7ff75dd9b700 1 -- 172.21.15.23:0/2441122394 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff754000e90 con 0x7ff7600a8110 2020-11-20T11:45:02.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.753+0000 7ff75dd9b700 1 -- 172.21.15.23:0/2441122394 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75401cb60 con 0x7ff7600a8110 2020-11-20T11:45:02.771 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.754+0000 7ff7648de700 1 -- 172.21.15.23:0/2441122394 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7600a8110 msgr2=0x7ff7600acca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:02.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.754+0000 7ff7648de700 1 --2- 172.21.15.23:0/2441122394 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7600a8110 0x7ff7600acca0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7ff75401a9a0 tx=0x7ff75401ad60).stop 2020-11-20T11:45:02.773 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.759+0000 7ff7648de700 1 -- 172.21.15.23:0/2441122394 shutdown_connections 2020-11-20T11:45:02.773 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.759+0000 7ff7648de700 1 --2- 172.21.15.23:0/2441122394 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7600a8110 0x7ff7600acca0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:02.774 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.759+0000 7ff7648de700 1 -- 172.21.15.23:0/2441122394 >> 172.21.15.23:0/2441122394 conn(0x7ff760061850 msgr2=0x7ff760061c60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:02.776 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.771+0000 7ff7648de700 1 -- 172.21.15.23:0/2441122394 shutdown_connections 2020-11-20T11:45:02.776 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.771+0000 7ff7648de700 1 -- 172.21.15.23:0/2441122394 wait complete. 2020-11-20T11:45:02.789 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.773+0000 7ff7648de700 1 Processor -- start 2020-11-20T11:45:02.790 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.773+0000 7ff7648de700 1 -- start start 2020-11-20T11:45:02.791 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.774+0000 7ff7648de700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff7600a8110 0x7ff76013e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:02.791 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.774+0000 7ff7648de700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7600a8b30 0x7ff76013c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:02.792 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.774+0000 7ff7648de700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7600675f0 0x7ff760136d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:02.792 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.774+0000 7ff7648de700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76013eac0 con 0x7ff7600a8b30 2020-11-20T11:45:02.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.774+0000 7ff7648de700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76013e700 con 0x7ff7600675f0 2020-11-20T11:45:02.795 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.774+0000 7ff7648de700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff760067150 con 0x7ff7600a8110 2020-11-20T11:45:02.795 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.774+0000 7ff75f59e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7600675f0 0x7ff760136d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:02.796 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.774+0000 7ff75f59e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7600675f0 0x7ff760136d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:38668/0 (socket says 172.21.15.23:38668) 2020-11-20T11:45:02.797 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.774+0000 7ff75f59e700 1 -- 172.21.15.23:0/4028786479 learned_addr learned my addr 172.21.15.23:0/4028786479 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:45:02.798 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.775+0000 7ff75f59e700 1 -- 172.21.15.23:0/4028786479 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff7600a8110 msgr2=0x7ff76013e060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:45:02.799 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.775+0000 7ff75f59e700 1 --2- 172.21.15.23:0/4028786479 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff7600a8110 0x7ff76013e060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:02.799 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.775+0000 7ff75f59e700 1 -- 172.21.15.23:0/4028786479 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7600a8b30 msgr2=0x7ff76013c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:02.800 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.775+0000 7ff75f59e700 1 --2- 172.21.15.23:0/4028786479 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7600a8b30 0x7ff76013c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:02.802 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.775+0000 7ff75f59e700 1 -- 172.21.15.23:0/4028786479 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff760067a70 con 0x7ff7600675f0 2020-11-20T11:45:02.802 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.775+0000 7ff75f59e700 1 --2- 172.21.15.23:0/4028786479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7600675f0 0x7ff760136d80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7ff75002b670 tx=0x7ff750006d80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:02.802 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.776+0000 7ff746ffd700 1 -- 172.21.15.23:0/4028786479 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75002f510 con 0x7ff7600675f0 2020-11-20T11:45:02.803 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.776+0000 7ff7648de700 1 -- 172.21.15.23:0/4028786479 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7601385f0 con 0x7ff7600675f0 2020-11-20T11:45:02.803 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.776+0000 7ff7648de700 1 -- 172.21.15.23:0/4028786479 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff760064ea0 con 0x7ff7600675f0 2020-11-20T11:45:02.804 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.776+0000 7ff746ffd700 1 -- 172.21.15.23:0/4028786479 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff75002f670 con 0x7ff7600675f0 2020-11-20T11:45:02.804 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.776+0000 7ff746ffd700 1 -- 172.21.15.23:0/4028786479 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75002fa30 con 0x7ff7600675f0 2020-11-20T11:45:02.805 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.778+0000 7ff746ffd700 1 -- 172.21.15.23:0/4028786479 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff750033420 con 0x7ff7600675f0 2020-11-20T11:45:02.805 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.779+0000 7ff746ffd700 1 --2- 172.21.15.23:0/4028786479 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff73c056db0 0x7ff73c059270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:02.805 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.779+0000 7ff746ffd700 1 -- 172.21.15.23:0/4028786479 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7ff750091b70 con 0x7ff7600675f0 2020-11-20T11:45:02.805 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.786+0000 7ff7648de700 1 -- 172.21.15.23:0/4028786479 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff76013e550 con 0x7ff7600675f0 2020-11-20T11:45:02.805 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.786+0000 7ff75ed9d700 1 --2- 172.21.15.23:0/4028786479 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff73c056db0 0x7ff73c059270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:02.806 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.790+0000 7ff75ed9d700 1 --2- 172.21.15.23:0/4028786479 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff73c056db0 0x7ff73c059270 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7ff75401ada0 tx=0x7ff7600a5bc0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:02.807 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:02.798+0000 7ff746ffd700 1 -- 172.21.15.23:0/4028786479 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ff750066d70 con 0x7ff7600675f0 2020-11-20T11:45:03.266 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:03.263+0000 7ff7648de700 1 -- 172.21.15.23:0/4028786479 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff760066f90 con 0x7ff7600675f0 2020-11-20T11:45:03.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:03.264+0000 7ff746ffd700 1 -- 172.21.15.23:0/4028786479 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1479 (secure 0 0 0) 0x7ff75006a390 con 0x7ff7600675f0 2020-11-20T11:45:03.267 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":116,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:45:03.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:03.265+0000 7ff7648de700 1 -- 172.21.15.23:0/4028786479 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff73c056db0 msgr2=0x7ff73c059270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:03.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:03.265+0000 7ff7648de700 1 --2- 172.21.15.23:0/4028786479 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff73c056db0 0x7ff73c059270 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7ff75401ada0 tx=0x7ff7600a5bc0).stop 2020-11-20T11:45:03.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:03.265+0000 7ff7648de700 1 -- 172.21.15.23:0/4028786479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7600675f0 msgr2=0x7ff760136d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:03.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:03.265+0000 7ff7648de700 1 --2- 172.21.15.23:0/4028786479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7600675f0 0x7ff760136d80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7ff75002b670 tx=0x7ff750006d80).stop 2020-11-20T11:45:03.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:03.265+0000 7ff7648de700 1 -- 172.21.15.23:0/4028786479 shutdown_connections 2020-11-20T11:45:03.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:03.265+0000 7ff7648de700 1 --2- 172.21.15.23:0/4028786479 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff7600675f0 0x7ff760136d80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:03.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:03.265+0000 7ff7648de700 1 --2- 172.21.15.23:0/4028786479 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff73c056db0 0x7ff73c059270 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:03.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:03.265+0000 7ff7648de700 1 --2- 172.21.15.23:0/4028786479 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff7600a8110 0x7ff76013e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:03.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:03.265+0000 7ff7648de700 1 --2- 172.21.15.23:0/4028786479 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7600a8b30 0x7ff76013c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:03.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:03.265+0000 7ff7648de700 1 -- 172.21.15.23:0/4028786479 >> 172.21.15.23:0/4028786479 conn(0x7ff760061850 msgr2=0x7ff7600a6b50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:03.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:03.265+0000 7ff7648de700 1 -- 172.21.15.23:0/4028786479 shutdown_connections 2020-11-20T11:45:03.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:03.265+0000 7ff7648de700 1 -- 172.21.15.23:0/4028786479 wait complete. 2020-11-20T11:45:06.293 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:45:06.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:06.643+0000 7efca7dd7700 1 Processor -- start 2020-11-20T11:45:07.177 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:06.644+0000 7efca7dd7700 1 -- start start 2020-11-20T11:45:07.178 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:06.645+0000 7efca7dd7700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7efca0060a80 0x7efca0067c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:07.178 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:06.645+0000 7efca7dd7700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efca00aa1e0 0x7efca013fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:07.178 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:06.645+0000 7efca7dd7700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca00a2e00 con 0x7efca00a9d80 2020-11-20T11:45:07.178 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:06.645+0000 7efca7dd7700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efca00ab100 con 0x7efca0060a80 2020-11-20T11:45:07.179 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:06.645+0000 7efca7dd7700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efca0068100 con 0x7efca00aa1e0 2020-11-20T11:45:07.179 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:06.645+0000 7efca6374700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efca00aa1e0 msgr2=0x7efca013fd20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:45:07.179 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:06.645+0000 7efca6374700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efca00aa1e0 0x7efca013fd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:45:07.179 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:06.645+0000 7efca5b73700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7efca0060a80 0x7efca0067c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:07.179 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:06.645+0000 7efca5b73700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7efca0060a80 0x7efca0067c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45778/0 (socket says 172.21.15.23:45778) 2020-11-20T11:45:07.180 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:06.645+0000 7efca5b73700 1 -- 172.21.15.23:0/1988392771 learned_addr learned my addr 172.21.15.23:0/1988392771 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:45:07.180 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:06.648+0000 7efca5b73700 1 -- 172.21.15.23:0/1988392771 >> v2:172.21.15.23:3300/0 conn(0x7efca0060a80 msgr2=0x7efca0067c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:45:07.180 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:06.648+0000 7efca5b73700 1 -- 172.21.15.23:0/1988392771 >> v2:172.21.15.23:3300/0 conn(0x7efca0060a80 msgr2=0x7efca0067c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:45:07.180 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:06.648+0000 7efca5b73700 1 --2- 172.21.15.23:0/1988392771 >> v2:172.21.15.23:3300/0 conn(0x7efca0060a80 0x7efca0067c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:45:07.180 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:06.648+0000 7efca5b73700 1 --2- 172.21.15.23:0/1988392771 >> v2:172.21.15.23:3300/0 conn(0x7efca0060a80 0x7efca0067c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:45:07.181 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:06.846+0000 7efca6374700 1 -- 172.21.15.23:0/1988392771 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efca00aa1e0 msgr2=0x7efca013fd20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:45:07.181 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:06.846+0000 7efca6374700 1 --2- 172.21.15.23:0/1988392771 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efca00aa1e0 0x7efca013fd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:45:07.181 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:06.849+0000 7efca5b73700 1 --2- 172.21.15.23:0/1988392771 >> v2:172.21.15.23:3300/0 conn(0x7efca0060a80 0x7efca0067c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:07.181 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:06.849+0000 7efca5b73700 1 -- 172.21.15.23:0/1988392771 >> v2:172.21.15.23:3300/0 conn(0x7efca0060a80 msgr2=0x7efca0067c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:45:07.181 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:06.849+0000 7efca5b73700 1 -- 172.21.15.23:0/1988392771 >> v2:172.21.15.23:3300/0 conn(0x7efca0060a80 msgr2=0x7efca0067c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:45:07.182 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:06.849+0000 7efca5b73700 1 --2- 172.21.15.23:0/1988392771 >> v2:172.21.15.23:3300/0 conn(0x7efca0060a80 0x7efca0067c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:45:07.182 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:06.850+0000 7efca5b73700 1 --2- 172.21.15.23:0/1988392771 >> v2:172.21.15.23:3300/0 conn(0x7efca0060a80 0x7efca0067c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:45:07.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:07.247+0000 7efca6374700 1 -- 172.21.15.23:0/1988392771 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efca00aa1e0 msgr2=0x7efca013fd20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:45:07.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:07.247+0000 7efca6374700 1 --2- 172.21.15.23:0/1988392771 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efca00aa1e0 0x7efca013fd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:45:07.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:07.251+0000 7efca5b73700 1 --2- 172.21.15.23:0/1988392771 >> v2:172.21.15.23:3300/0 conn(0x7efca0060a80 0x7efca0067c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:07.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:07.251+0000 7efca5b73700 1 -- 172.21.15.23:0/1988392771 >> v2:172.21.15.23:3300/0 conn(0x7efca0060a80 msgr2=0x7efca0067c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:45:07.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:07.251+0000 7efca5b73700 1 -- 172.21.15.23:0/1988392771 >> v2:172.21.15.23:3300/0 conn(0x7efca0060a80 msgr2=0x7efca0067c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:45:07.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:07.251+0000 7efca5b73700 1 --2- 172.21.15.23:0/1988392771 >> v2:172.21.15.23:3300/0 conn(0x7efca0060a80 0x7efca0067c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:45:07.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:07.251+0000 7efca5b73700 1 --2- 172.21.15.23:0/1988392771 >> v2:172.21.15.23:3300/0 conn(0x7efca0060a80 0x7efca0067c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:45:07.584 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2020-11-20T11:45:08.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:08.048+0000 7efca6374700 1 -- 172.21.15.23:0/1988392771 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efca00aa1e0 msgr2=0x7efca013fd20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:45:08.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:08.048+0000 7efca6374700 1 --2- 172.21.15.23:0/1988392771 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efca00aa1e0 0x7efca013fd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:45:08.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:08.052+0000 7efca5b73700 1 --2- 172.21.15.23:0/1988392771 >> v2:172.21.15.23:3300/0 conn(0x7efca0060a80 0x7efca0067c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:08.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:08.052+0000 7efca5b73700 1 -- 172.21.15.23:0/1988392771 >> v2:172.21.15.23:3300/0 conn(0x7efca0060a80 msgr2=0x7efca0067c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:45:08.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:08.052+0000 7efca5b73700 1 -- 172.21.15.23:0/1988392771 >> v2:172.21.15.23:3300/0 conn(0x7efca0060a80 msgr2=0x7efca0067c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:45:08.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:08.052+0000 7efca5b73700 1 --2- 172.21.15.23:0/1988392771 >> v2:172.21.15.23:3300/0 conn(0x7efca0060a80 0x7efca0067c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:45:08.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:08.052+0000 7efca5b73700 1 --2- 172.21.15.23:0/1988392771 >> v2:172.21.15.23:3300/0 conn(0x7efca0060a80 0x7efca0067c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:45:09.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:09.645+0000 7efc8f7fe700 1 -- 172.21.15.23:0/1988392771 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efca00aa1e0 msgr2=0x7efca013fd20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:45:09.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:09.645+0000 7efc8f7fe700 1 --2- 172.21.15.23:0/1988392771 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efca00aa1e0 0x7efca013fd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:09.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:09.645+0000 7efc8f7fe700 1 -- 172.21.15.23:0/1988392771 >> v2:172.21.15.23:3300/0 conn(0x7efca0060a80 msgr2=0x7efca0067c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:45:09.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:09.645+0000 7efc8f7fe700 1 --2- 172.21.15.23:0/1988392771 >> v2:172.21.15.23:3300/0 conn(0x7efca0060a80 0x7efca0067c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:09.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:09.645+0000 7efc8f7fe700 1 -- 172.21.15.23:0/1988392771 >> v1:172.21.15.23:6789/0 conn(0x7efca00a9d80 legacy=0x7efca013f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:09.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:09.645+0000 7efc8f7fe700 1 --2- 172.21.15.23:0/1988392771 >> v2:172.21.15.155:3300/0 conn(0x7efc800070c0 0x7efc80009580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:09.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:09.645+0000 7efc8f7fe700 1 -- 172.21.15.23:0/1988392771 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca013f1e0 con 0x7efc80003410 2020-11-20T11:45:09.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:09.645+0000 7efc8f7fe700 1 -- 172.21.15.23:0/1988392771 --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efca00ab100 con 0x7efc800070c0 2020-11-20T11:45:09.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:09.645+0000 7efca5372700 1 --2- 172.21.15.23:0/1988392771 >> v2:172.21.15.155:3300/0 conn(0x7efc800070c0 0x7efc80009580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:09.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:09.645+0000 7efca5372700 1 -- 172.21.15.23:0/1988392771 >> v2:172.21.15.155:3300/0 conn(0x7efc800070c0 msgr2=0x7efc80009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:45:09.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:09.645+0000 7efca5372700 1 -- 172.21.15.23:0/1988392771 >> v2:172.21.15.155:3300/0 conn(0x7efc800070c0 msgr2=0x7efc80009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:45:09.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:09.645+0000 7efca5372700 1 --2- 172.21.15.23:0/1988392771 >> v2:172.21.15.155:3300/0 conn(0x7efc800070c0 0x7efc80009580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:45:09.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:09.646+0000 7efca5372700 1 --2- 172.21.15.23:0/1988392771 >> v2:172.21.15.155:3300/0 conn(0x7efc800070c0 0x7efc80009580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:45:09.850 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:09.847+0000 7efca5372700 1 --2- 172.21.15.23:0/1988392771 >> v2:172.21.15.155:3300/0 conn(0x7efc800070c0 0x7efc80009580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:09.851 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:09.847+0000 7efca5372700 1 -- 172.21.15.23:0/1988392771 >> v2:172.21.15.155:3300/0 conn(0x7efc800070c0 msgr2=0x7efc80009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:45:09.851 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:09.847+0000 7efca5372700 1 -- 172.21.15.23:0/1988392771 >> v2:172.21.15.155:3300/0 conn(0x7efc800070c0 msgr2=0x7efc80009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:45:09.852 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:09.847+0000 7efca5372700 1 --2- 172.21.15.23:0/1988392771 >> v2:172.21.15.155:3300/0 conn(0x7efc800070c0 0x7efc80009580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:45:09.853 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:09.847+0000 7efca5372700 1 --2- 172.21.15.23:0/1988392771 >> v2:172.21.15.155:3300/0 conn(0x7efc800070c0 0x7efc80009580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:45:10.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.248+0000 7efca5372700 1 --2- 172.21.15.23:0/1988392771 >> v2:172.21.15.155:3300/0 conn(0x7efc800070c0 0x7efc80009580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:10.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.248+0000 7efca5372700 1 -- 172.21.15.23:0/1988392771 >> v2:172.21.15.155:3300/0 conn(0x7efc800070c0 msgr2=0x7efc80009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:45:10.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.248+0000 7efca5372700 1 -- 172.21.15.23:0/1988392771 >> v2:172.21.15.155:3300/0 conn(0x7efc800070c0 msgr2=0x7efc80009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:45:10.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.248+0000 7efca5372700 1 --2- 172.21.15.23:0/1988392771 >> v2:172.21.15.155:3300/0 conn(0x7efc800070c0 0x7efc80009580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:45:10.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.248+0000 7efca5372700 1 --2- 172.21.15.23:0/1988392771 >> v2:172.21.15.155:3300/0 conn(0x7efc800070c0 0x7efc80009580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:45:10.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.338+0000 7efca4b71700 1 -- 172.21.15.23:0/1988392771 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4120779349 0 0) 0x7efca013f1e0 con 0x7efc80003410 2020-11-20T11:45:10.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.338+0000 7efca4b71700 1 -- 172.21.15.23:0/1988392771 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc88003280 con 0x7efc80003410 2020-11-20T11:45:10.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.343+0000 7efca4b71700 1 -- 172.21.15.23:0/1988392771 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3488412105 0 0) 0x7efc88003280 con 0x7efc80003410 2020-11-20T11:45:10.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.343+0000 7efca4b71700 1 -- 172.21.15.23:0/1988392771 >> v2:172.21.15.155:3300/0 conn(0x7efc800070c0 msgr2=0x7efc80009580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:45:10.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.343+0000 7efca4b71700 1 --2- 172.21.15.23:0/1988392771 >> v2:172.21.15.155:3300/0 conn(0x7efc800070c0 0x7efc80009580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:10.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.343+0000 7efca4b71700 1 -- 172.21.15.23:0/1988392771 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc8000ad00 con 0x7efc80003410 2020-11-20T11:45:10.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.343+0000 7efca4b71700 1 -- 172.21.15.23:0/1988392771 <== mon.1 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7efc9c00d3b0 con 0x7efc80003410 2020-11-20T11:45:10.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.343+0000 7efca4b71700 1 -- 172.21.15.23:0/1988392771 >> v1:172.21.15.155:6789/0 conn(0x7efc80003410 legacy=0x7efc80005890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:10.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.343+0000 7efca6374700 1 -- 172.21.15.23:0/1988392771 reap_dead start 2020-11-20T11:45:10.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.343+0000 7efca4b71700 1 --2- 172.21.15.23:0/1988392771 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efc88003770 0x7efc88003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:10.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.343+0000 7efca4b71700 1 --2- 172.21.15.23:0/1988392771 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc88004080 0x7efc88004560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:10.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.343+0000 7efca4b71700 1 --2- 172.21.15.23:0/1988392771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc88004c00 0x7efc8800b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:10.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.343+0000 7efca4b71700 1 -- 172.21.15.23:0/1988392771 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca00ab100 con 0x7efc88004c00 2020-11-20T11:45:10.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.343+0000 7efca4b71700 1 -- 172.21.15.23:0/1988392771 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc880052a0 con 0x7efc88004080 2020-11-20T11:45:10.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.343+0000 7efca4b71700 1 -- 172.21.15.23:0/1988392771 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc8800bc60 con 0x7efc88003770 2020-11-20T11:45:10.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.343+0000 7efca5b73700 1 -- 172.21.15.23:0/1988392771 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efc88003770 msgr2=0x7efc88003b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:45:10.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.343+0000 7efca5b73700 1 --2- 172.21.15.23:0/1988392771 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efc88003770 0x7efc88003b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:45:10.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.343+0000 7efca5372700 1 --2- 172.21.15.23:0/1988392771 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc88004080 0x7efc88004560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:10.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.343+0000 7efca6374700 1 --2- 172.21.15.23:0/1988392771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc88004c00 0x7efc8800b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:10.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.343+0000 7efca6374700 1 -- 172.21.15.23:0/1988392771 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efc88003770 msgr2=0x7efc88003b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:45:10.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.343+0000 7efca6374700 1 --2- 172.21.15.23:0/1988392771 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efc88003770 0x7efc88003b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:10.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.343+0000 7efca6374700 1 -- 172.21.15.23:0/1988392771 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc88004080 msgr2=0x7efc88004560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:45:10.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.343+0000 7efca6374700 1 --2- 172.21.15.23:0/1988392771 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc88004080 0x7efc88004560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:10.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.343+0000 7efca6374700 1 -- 172.21.15.23:0/1988392771 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc88003280 con 0x7efc88004c00 2020-11-20T11:45:10.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.344+0000 7efca6374700 1 --2- 172.21.15.23:0/1988392771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc88004c00 0x7efc8800b770 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7efca00af170 tx=0x7efc98009640).ready entity=mon.0 client_cookie=3c91f034301b0e1b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:10.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.344+0000 7efca4b71700 1 -- 172.21.15.23:0/1988392771 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9800d440 con 0x7efc88004c00 2020-11-20T11:45:10.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.344+0000 7efca4b71700 1 -- 172.21.15.23:0/1988392771 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc9800d5a0 con 0x7efc88004c00 2020-11-20T11:45:10.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.344+0000 7efca4b71700 1 -- 172.21.15.23:0/1988392771 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9800d960 con 0x7efc88004c00 2020-11-20T11:45:10.355 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.345+0000 7efca7dd7700 1 -- 172.21.15.23:0/1988392771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc88004c00 msgr2=0x7efc8800b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:10.355 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.345+0000 7efca7dd7700 1 --2- 172.21.15.23:0/1988392771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc88004c00 0x7efc8800b770 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7efca00af170 tx=0x7efc98009640).stop 2020-11-20T11:45:10.355 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.345+0000 7efca7dd7700 1 -- 172.21.15.23:0/1988392771 shutdown_connections 2020-11-20T11:45:10.355 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.345+0000 7efca7dd7700 1 --2- 172.21.15.23:0/1988392771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc88004c00 0x7efc8800b770 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:10.356 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.345+0000 7efca7dd7700 1 --2- 172.21.15.23:0/1988392771 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc88004080 0x7efc88004560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:10.356 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.345+0000 7efca7dd7700 1 --2- 172.21.15.23:0/1988392771 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efc88003770 0x7efc88003b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:10.356 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.345+0000 7efca7dd7700 1 -- 172.21.15.23:0/1988392771 >> 172.21.15.23:0/1988392771 conn(0x7efca0066730 msgr2=0x7efca00601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:10.356 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.345+0000 7efca7dd7700 1 -- 172.21.15.23:0/1988392771 shutdown_connections 2020-11-20T11:45:10.356 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.345+0000 7efca7dd7700 1 -- 172.21.15.23:0/1988392771 wait complete. 2020-11-20T11:45:10.357 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.346+0000 7efca7dd7700 1 Processor -- start 2020-11-20T11:45:10.357 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.346+0000 7efca7dd7700 1 -- start start 2020-11-20T11:45:10.357 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.347+0000 7efca7dd7700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc88003770 0x7efca00a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:10.357 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.347+0000 7efca7dd7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc88004080 0x7efca0137cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:10.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.347+0000 7efca7dd7700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efc88004c00 0x7efca00a4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:10.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.347+0000 7efca7dd7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca01386a0 con 0x7efc88004080 2020-11-20T11:45:10.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.347+0000 7efca7dd7700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca0138940 con 0x7efc88003770 2020-11-20T11:45:10.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.347+0000 7efca7dd7700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efca0138be0 con 0x7efc88004c00 2020-11-20T11:45:10.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.347+0000 7efca5372700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc88004080 0x7efca0137cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:10.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.347+0000 7efca5b73700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc88003770 0x7efca00a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:10.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.347+0000 7efca5b73700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc88003770 0x7efca00a4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:38706/0 (socket says 172.21.15.23:38706) 2020-11-20T11:45:10.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.347+0000 7efca5b73700 1 -- 172.21.15.23:0/3920610867 learned_addr learned my addr 172.21.15.23:0/3920610867 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:45:10.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.347+0000 7efca6374700 1 -- 172.21.15.23:0/3920610867 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efc88004c00 msgr2=0x7efca00a4d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:45:10.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.347+0000 7efca6374700 1 --2- 172.21.15.23:0/3920610867 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efc88004c00 0x7efca00a4d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:45:10.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.348+0000 7efca5b73700 1 -- 172.21.15.23:0/3920610867 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efc88004c00 msgr2=0x7efca00a4d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:45:10.360 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.348+0000 7efca5b73700 1 --2- 172.21.15.23:0/3920610867 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efc88004c00 0x7efca00a4d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:10.360 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.348+0000 7efca5b73700 1 -- 172.21.15.23:0/3920610867 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc88004080 msgr2=0x7efca0137cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:10.360 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.348+0000 7efca5b73700 1 --2- 172.21.15.23:0/3920610867 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc88004080 0x7efca0137cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:10.360 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.348+0000 7efca5b73700 1 -- 172.21.15.23:0/3920610867 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca00a6b00 con 0x7efc88003770 2020-11-20T11:45:10.360 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.348+0000 7efca5b73700 1 --2- 172.21.15.23:0/3920610867 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc88003770 0x7efca00a4010 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7efc9c012980 tx=0x7efc9c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:10.361 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.348+0000 7efc8dffb700 1 -- 172.21.15.23:0/3920610867 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9c0108f0 con 0x7efc88003770 2020-11-20T11:45:10.361 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.348+0000 7efc8dffb700 1 -- 172.21.15.23:0/3920610867 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc9c010a50 con 0x7efc88003770 2020-11-20T11:45:10.361 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.348+0000 7efca7dd7700 1 -- 172.21.15.23:0/3920610867 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efca00a4500 con 0x7efc88003770 2020-11-20T11:45:10.361 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.349+0000 7efc8dffb700 1 -- 172.21.15.23:0/3920610867 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9c010e10 con 0x7efc88003770 2020-11-20T11:45:10.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.349+0000 7efca7dd7700 1 -- 172.21.15.23:0/3920610867 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efca01383e0 con 0x7efc88003770 2020-11-20T11:45:10.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.350+0000 7efc8dffb700 1 -- 172.21.15.23:0/3920610867 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7efc9c025070 con 0x7efc88003770 2020-11-20T11:45:10.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.350+0000 7efca7dd7700 1 -- 172.21.15.23:0/3920610867 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efca0044710 con 0x7efc88003770 2020-11-20T11:45:10.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.350+0000 7efc8dffb700 1 --2- 172.21.15.23:0/3920610867 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efc78056d10 0x7efc780591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:10.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.350+0000 7efc8dffb700 1 -- 172.21.15.23:0/3920610867 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7efc9c070950 con 0x7efc88003770 2020-11-20T11:45:10.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.350+0000 7efca5372700 1 --2- 172.21.15.23:0/3920610867 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efc78056d10 0x7efc780591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:10.363 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.351+0000 7efca5372700 1 --2- 172.21.15.23:0/3920610867 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efc78056d10 0x7efc780591d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7efc90007fd0 tx=0x7efc90009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:10.363 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.355+0000 7efc8dffb700 1 -- 172.21.15.23:0/3920610867 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7efc9c045c40 con 0x7efc88003770 2020-11-20T11:45:10.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.643+0000 7efca7dd7700 1 -- 172.21.15.23:0/3920610867 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7efca00a4a70 con 0x7efc88003770 2020-11-20T11:45:10.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.644+0000 7efc8dffb700 1 -- 172.21.15.23:0/3920610867 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1472 (secure 0 0 0) 0x7efca00a4a70 con 0x7efc88003770 2020-11-20T11:45:10.647 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":120,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:45:10.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.645+0000 7efca7dd7700 1 -- 172.21.15.23:0/3920610867 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efc78056d10 msgr2=0x7efc780591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:10.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.645+0000 7efca7dd7700 1 --2- 172.21.15.23:0/3920610867 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efc78056d10 0x7efc780591d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7efc90007fd0 tx=0x7efc90009040).stop 2020-11-20T11:45:10.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.645+0000 7efca7dd7700 1 -- 172.21.15.23:0/3920610867 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc88003770 msgr2=0x7efca00a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:10.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.645+0000 7efca7dd7700 1 --2- 172.21.15.23:0/3920610867 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc88003770 0x7efca00a4010 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7efc9c012980 tx=0x7efc9c008040).stop 2020-11-20T11:45:10.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.645+0000 7efca7dd7700 1 -- 172.21.15.23:0/3920610867 shutdown_connections 2020-11-20T11:45:10.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.645+0000 7efca7dd7700 1 --2- 172.21.15.23:0/3920610867 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efc78056d10 0x7efc780591d0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:10.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.645+0000 7efca7dd7700 1 --2- 172.21.15.23:0/3920610867 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efc88004c00 0x7efca00a4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:10.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.645+0000 7efca7dd7700 1 --2- 172.21.15.23:0/3920610867 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efc88003770 0x7efca00a4010 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:10.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.645+0000 7efca7dd7700 1 --2- 172.21.15.23:0/3920610867 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efc88004080 0x7efca0137cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:10.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.645+0000 7efca7dd7700 1 -- 172.21.15.23:0/3920610867 >> 172.21.15.23:0/3920610867 conn(0x7efca0066730 msgr2=0x7efca00ad0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:10.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.646+0000 7efca7dd7700 1 -- 172.21.15.23:0/3920610867 shutdown_connections 2020-11-20T11:45:10.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.646+0000 7efca7dd7700 1 -- 172.21.15.23:0/3920610867 wait complete. 2020-11-20T11:45:10.660 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2020-11-20T11:45:10.660 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:45:10.808 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.805+0000 7f1d187a4700 1 Processor -- start 2020-11-20T11:45:10.808 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.805+0000 7f1d187a4700 1 -- start start 2020-11-20T11:45:10.811 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.805+0000 7f1d187a4700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1d10061380 0x7f1d100aaac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:10.811 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.805+0000 7f1d187a4700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1d100a9dd0 0x7f1d100a3a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:10.811 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.805+0000 7f1d187a4700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f1d100a3f80 0x7f1d1013e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:10.811 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.805+0000 7f1d187a4700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d100aafb0 con 0x7f1d10061380 2020-11-20T11:45:10.812 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.805+0000 7f1d187a4700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d100a3050 con 0x7f1d100a3f80 2020-11-20T11:45:10.812 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.805+0000 7f1d187a4700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d100a32f0 con 0x7f1d100a9dd0 2020-11-20T11:45:10.812 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.806+0000 7f1d16540700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1d10061380 0x7f1d100aaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:10.812 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.806+0000 7f1d15d3f700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1d100a9dd0 msgr2=0x7f1d100a3a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:45:10.812 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.806+0000 7f1d15d3f700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1d100a9dd0 0x7f1d100a3a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:45:10.813 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.806+0000 7f1d16540700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1d10061380 0x7f1d100aaac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45818/0 (socket says 172.21.15.23:45818) 2020-11-20T11:45:10.813 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.806+0000 7f1d16540700 1 -- 172.21.15.23:0/1871484865 learned_addr learned my addr 172.21.15.23:0/1871484865 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:45:10.813 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.806+0000 7f1d16d41700 1 --2- 172.21.15.23:0/1871484865 >> v2:172.21.15.155:3300/0 conn(0x7f1d100a3f80 0x7f1d1013e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:10.813 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.806+0000 7f1d16540700 1 -- 172.21.15.23:0/1871484865 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1d100a9dd0 msgr2=0x7f1d100a3a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:45:10.814 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.806+0000 7f1d16540700 1 --2- 172.21.15.23:0/1871484865 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1d100a9dd0 0x7f1d100a3a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:10.814 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.806+0000 7f1d16540700 1 -- 172.21.15.23:0/1871484865 >> v2:172.21.15.155:3300/0 conn(0x7f1d100a3f80 msgr2=0x7f1d1013e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:45:10.814 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.806+0000 7f1d16540700 1 --2- 172.21.15.23:0/1871484865 >> v2:172.21.15.155:3300/0 conn(0x7f1d100a3f80 0x7f1d1013e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:10.814 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.806+0000 7f1d16540700 1 -- 172.21.15.23:0/1871484865 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d100a3590 con 0x7f1d10061380 2020-11-20T11:45:10.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.806+0000 7f1d16540700 1 --2- 172.21.15.23:0/1871484865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d10061380 0x7f1d100aaac0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f1d100a2f30 tx=0x7f1d0c006ef0).ready entity=mon.0 client_cookie=8b5c499d025af8e0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:10.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.806+0000 7f1d1553e700 1 -- 172.21.15.23:0/1871484865 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c00b860 con 0x7f1d10061380 2020-11-20T11:45:10.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.806+0000 7f1d1553e700 1 -- 172.21.15.23:0/1871484865 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d0c000e90 con 0x7f1d10061380 2020-11-20T11:45:10.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.806+0000 7f1d1553e700 1 -- 172.21.15.23:0/1871484865 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c00bbd0 con 0x7f1d10061380 2020-11-20T11:45:10.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.807+0000 7f1d187a4700 1 -- 172.21.15.23:0/1871484865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d10061380 msgr2=0x7f1d100aaac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:10.816 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.807+0000 7f1d187a4700 1 --2- 172.21.15.23:0/1871484865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d10061380 0x7f1d100aaac0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f1d100a2f30 tx=0x7f1d0c006ef0).stop 2020-11-20T11:45:10.816 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.807+0000 7f1d187a4700 1 -- 172.21.15.23:0/1871484865 shutdown_connections 2020-11-20T11:45:10.816 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.807+0000 7f1d187a4700 1 --2- 172.21.15.23:0/1871484865 >> v2:172.21.15.155:3300/0 conn(0x7f1d100a3f80 0x7f1d1013e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:10.816 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.807+0000 7f1d187a4700 1 --2- 172.21.15.23:0/1871484865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d10061380 0x7f1d100aaac0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:10.816 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.807+0000 7f1d187a4700 1 --2- 172.21.15.23:0/1871484865 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1d100a9dd0 0x7f1d100a3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:10.817 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.807+0000 7f1d187a4700 1 -- 172.21.15.23:0/1871484865 >> 172.21.15.23:0/1871484865 conn(0x7f1d100ae280 msgr2=0x7f1d100aa290 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:10.817 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.807+0000 7f1d187a4700 1 -- 172.21.15.23:0/1871484865 shutdown_connections 2020-11-20T11:45:10.817 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.807+0000 7f1d187a4700 1 -- 172.21.15.23:0/1871484865 wait complete. 2020-11-20T11:45:10.817 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.808+0000 7f1d187a4700 1 Processor -- start 2020-11-20T11:45:10.817 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.808+0000 7f1d187a4700 1 -- start start 2020-11-20T11:45:10.823 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.808+0000 7f1d187a4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d10061380 0x7f1d100a74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:10.824 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.808+0000 7f1d187a4700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1d100a3f80 0x7f1d100a68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:10.824 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.808+0000 7f1d187a4700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1d100a9dd0 0x7f1d1014e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:10.824 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.808+0000 7f1d187a4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d100a6db0 con 0x7f1d10061380 2020-11-20T11:45:10.824 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.808+0000 7f1d187a4700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d100ac410 con 0x7f1d100a3f80 2020-11-20T11:45:10.825 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.808+0000 7f1d187a4700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1014d950 con 0x7f1d100a9dd0 2020-11-20T11:45:10.825 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.809+0000 7f1d16540700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d10061380 0x7f1d100a74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:10.825 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.809+0000 7f1d16d41700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1d100a9dd0 msgr2=0x7f1d1014e020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:45:10.825 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.809+0000 7f1d16540700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d10061380 0x7f1d100a74e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45824/0 (socket says 172.21.15.23:45824) 2020-11-20T11:45:10.825 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.809+0000 7f1d16d41700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1d100a9dd0 0x7f1d1014e020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:45:10.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.809+0000 7f1d16540700 1 -- 172.21.15.23:0/240982046 learned_addr learned my addr 172.21.15.23:0/240982046 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:45:10.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.809+0000 7f1d15d3f700 1 --2- 172.21.15.23:0/240982046 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1d100a3f80 0x7f1d100a68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:10.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.809+0000 7f1d16540700 1 -- 172.21.15.23:0/240982046 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1d100a9dd0 msgr2=0x7f1d1014e020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:45:10.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.809+0000 7f1d16540700 1 --2- 172.21.15.23:0/240982046 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1d100a9dd0 0x7f1d1014e020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:10.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.809+0000 7f1d16540700 1 -- 172.21.15.23:0/240982046 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1d100a3f80 msgr2=0x7f1d100a68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:10.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.809+0000 7f1d16540700 1 --2- 172.21.15.23:0/240982046 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1d100a3f80 0x7f1d100a68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:10.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.809+0000 7f1d16540700 1 -- 172.21.15.23:0/240982046 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d100ab9a0 con 0x7f1d10061380 2020-11-20T11:45:10.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.809+0000 7f1d16540700 1 --2- 172.21.15.23:0/240982046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d10061380 0x7f1d100a74e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f1d0c00b3c0 tx=0x7f1d0c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:10.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.809+0000 7f1cfe7fc700 1 -- 172.21.15.23:0/240982046 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c011be0 con 0x7f1d10061380 2020-11-20T11:45:10.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.809+0000 7f1cfe7fc700 1 -- 172.21.15.23:0/240982046 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d0c011d40 con 0x7f1d10061380 2020-11-20T11:45:10.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.809+0000 7f1cfe7fc700 1 -- 172.21.15.23:0/240982046 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c01d6b0 con 0x7f1d10061380 2020-11-20T11:45:10.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.809+0000 7f1d187a4700 1 -- 172.21.15.23:0/240982046 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d100a6fd0 con 0x7f1d10061380 2020-11-20T11:45:10.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.809+0000 7f1d187a4700 1 -- 172.21.15.23:0/240982046 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d100a5340 con 0x7f1d10061380 2020-11-20T11:45:10.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.810+0000 7f1d187a4700 1 --2- 172.21.15.23:0/240982046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d1014e510 0x7f1d1012fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:10.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.810+0000 7f1d187a4700 1 -- 172.21.15.23:0/240982046 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1005ca80 con 0x7f1d1014e510 2020-11-20T11:45:10.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.810+0000 7f1d15d3f700 1 --2- 172.21.15.23:0/240982046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d1014e510 0x7f1d1012fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:10.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.810+0000 7f1cfe7fc700 1 -- 172.21.15.23:0/240982046 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1d0c017070 con 0x7f1d10061380 2020-11-20T11:45:10.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.810+0000 7f1d15d3f700 1 -- 172.21.15.23:0/240982046 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1d10061e10 con 0x7f1d1014e510 2020-11-20T11:45:10.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.810+0000 7f1d15d3f700 1 --2- 172.21.15.23:0/240982046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d1014e510 0x7f1d1012fb10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f1d10141730 tx=0x7f1d00006e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:10.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.811+0000 7f1cfe7fc700 1 --2- 172.21.15.23:0/240982046 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1ce8056d10 0x7f1ce80591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:10.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.811+0000 7f1cfe7fc700 1 -- 172.21.15.23:0/240982046 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f1d0c071e10 con 0x7f1d10061380 2020-11-20T11:45:10.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.811+0000 7f1cfe7fc700 1 -- 172.21.15.23:0/240982046 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0000e600 con 0x7f1d1014e510 2020-11-20T11:45:10.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.811+0000 7f1cfe7fc700 1 -- 172.21.15.23:0/240982046 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f1d0000e760 con 0x7f1d1014e510 2020-11-20T11:45:10.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.811+0000 7f1cfe7fc700 1 -- 172.21.15.23:0/240982046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d1014e510 msgr2=0x7f1d1012fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:10.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.811+0000 7f1cfe7fc700 1 --2- 172.21.15.23:0/240982046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d1014e510 0x7f1d1012fb10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f1d10141730 tx=0x7f1d00006e10).stop 2020-11-20T11:45:10.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.811+0000 7f1cfe7fc700 1 -- 172.21.15.23:0/240982046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d1014e510 msgr2=0x7f1d1012fb10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:45:10.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.811+0000 7f1cfe7fc700 1 --2- 172.21.15.23:0/240982046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d1014e510 0x7f1d1012fb10 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:10.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.811+0000 7f1d16d41700 1 --2- 172.21.15.23:0/240982046 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1ce8056d10 0x7f1ce80591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:10.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.811+0000 7f1d16d41700 1 --2- 172.21.15.23:0/240982046 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1ce8056d10 0x7f1ce80591d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f1d080099f0 tx=0x7f1d08008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:10.832 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.817+0000 7f1d187a4700 1 --2- 172.21.15.23:0/240982046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d10130000 0x7f1d1014a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:10.832 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.817+0000 7f1d187a4700 1 -- 172.21.15.23:0/240982046 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d10130410 con 0x7f1d10130000 2020-11-20T11:45:10.832 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.818+0000 7f1d15d3f700 1 --2- 172.21.15.23:0/240982046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d10130000 0x7f1d1014a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:10.832 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.818+0000 7f1d15d3f700 1 -- 172.21.15.23:0/240982046 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1d10044fa0 con 0x7f1d10130000 2020-11-20T11:45:10.832 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.818+0000 7f1d15d3f700 1 --2- 172.21.15.23:0/240982046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d10130000 0x7f1d1014a3e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f1d00015000 tx=0x7f1d00016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:10.833 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.818+0000 7f1cfe7fc700 1 -- 172.21.15.23:0/240982046 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0001a350 con 0x7f1d10130000 2020-11-20T11:45:10.833 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.818+0000 7f1cfe7fc700 1 -- 172.21.15.23:0/240982046 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4605 (secure 0 0 0) 0x7f1d0000e3f0 con 0x7f1d10130000 2020-11-20T11:45:10.833 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.818+0000 7f1cfe7fc700 1 -- 172.21.15.23:0/240982046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d10130000 msgr2=0x7f1d1014a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:10.833 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.818+0000 7f1cfe7fc700 1 --2- 172.21.15.23:0/240982046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d10130000 0x7f1d1014a3e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f1d00015000 tx=0x7f1d00016000).stop 2020-11-20T11:45:10.834 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.818+0000 7f1cfe7fc700 1 -- 172.21.15.23:0/240982046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d10130000 msgr2=0x7f1d1014a3e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:45:10.834 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.818+0000 7f1cfe7fc700 1 --2- 172.21.15.23:0/240982046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d10130000 0x7f1d1014a3e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:10.834 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.819+0000 7f1d187a4700 1 -- 172.21.15.23:0/240982046 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1ce8056d10 msgr2=0x7f1ce80591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:10.834 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.819+0000 7f1d187a4700 1 --2- 172.21.15.23:0/240982046 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1ce8056d10 0x7f1ce80591d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f1d080099f0 tx=0x7f1d08008040).stop 2020-11-20T11:45:10.834 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.819+0000 7f1d187a4700 1 -- 172.21.15.23:0/240982046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d10061380 msgr2=0x7f1d100a74e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:10.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.819+0000 7f1d187a4700 1 --2- 172.21.15.23:0/240982046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d10061380 0x7f1d100a74e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f1d0c00b3c0 tx=0x7f1d0c016000).stop 2020-11-20T11:45:10.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.819+0000 7f1d16d41700 1 -- 172.21.15.23:0/240982046 reap_dead start 2020-11-20T11:45:10.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.819+0000 7f1d16d41700 1 -- 172.21.15.23:0/240982046 reap_dead start 2020-11-20T11:45:10.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.819+0000 7f1d187a4700 1 -- 172.21.15.23:0/240982046 shutdown_connections 2020-11-20T11:45:10.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.819+0000 7f1d187a4700 1 -- 172.21.15.23:0/240982046 >> 172.21.15.23:0/240982046 conn(0x7f1d100ae280 msgr2=0x7f1d101377a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:10.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.819+0000 7f1d187a4700 1 -- 172.21.15.23:0/240982046 shutdown_connections 2020-11-20T11:45:10.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:10.819+0000 7f1d187a4700 1 -- 172.21.15.23:0/240982046 wait complete. 2020-11-20T11:45:10.836 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:45:10.836 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:45:10.836 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:45:10.837 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:45:10.837 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 120, 2020-11-20T11:45:10.837 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:45:10.837 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:45:10.838 INFO:teuthology.orchestra.run.smithi023.stdout: 1 2020-11-20T11:45:10.838 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:10.838 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 0, 2020-11-20T11:45:10.838 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:45:10.838 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:45:10.839 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:45:10.839 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:45:10.839 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:45:10.839 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:45:10.839 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:45:10.840 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:45:10.840 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:45:10.840 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:45:10.840 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:45:10.840 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:10.841 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:45:10.841 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:45:10.841 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:45:10.841 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:45:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:45:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:45:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:45:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:45:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:45:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:45:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:45:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:45:10.844 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:45:10.844 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:45:10.844 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:45:10.844 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:45:10.844 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:45:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:45:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:45:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:45:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:45:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:45:10.846 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:45:10.846 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:45:10.846 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:45:10.846 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:45:10.846 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:45:10.847 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:45:10.847 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:45:10.847 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:45:10.847 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:45:10.847 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:45:10.848 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:45:10.848 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:10.848 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:45:10.848 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:10.848 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:45:10.849 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:10.849 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:45:10.849 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:45:10.849 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:45:10.849 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:45:10.850 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:10.850 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:45:10.850 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:45:10.850 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:10.850 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:45:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:45:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:10.852 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:10.852 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:10.852 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:45:10.852 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:45:10.852 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:45:10.853 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:45:10.853 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:45:10.853 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:10.853 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:10.853 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:45:10.854 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:45:10.854 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:45:10.854 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:45:10.854 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:10.854 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:45:10.855 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:45:10.855 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:10.855 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:10.855 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:10.855 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:45:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:45:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:45:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:45:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:45:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:45:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:45:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:45:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:45:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:45:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:45:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:45:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:45:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:45:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:45:10.861 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:10.861 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:10.861 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:10.861 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:10.861 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:45:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:45:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:45:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:45:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:45:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:10.863 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:10.863 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:10.863 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:45:10.863 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:45:10.863 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:10.864 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:45:10.864 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:45:10.864 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:45:10.864 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:10.864 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:10.865 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:45:10.865 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:10.865 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:45:10.865 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:45:10.865 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:45:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:45:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:45:10.867 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:45:10.867 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:45:10.867 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:10.867 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:10.868 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:45:10.868 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:10.868 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:45:10.868 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:45:10.868 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:45:10.869 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:10.869 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:10.869 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:10.869 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:45:10.869 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:45:10.871 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:45:11.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.205+0000 7f68280e6700 1 Processor -- start 2020-11-20T11:45:11.381 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.205+0000 7f68280e6700 1 -- start start 2020-11-20T11:45:11.382 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.206+0000 7f68280e6700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68200ad7c0 0x7f68200a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:11.382 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.206+0000 7f68280e6700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6820063fe0 0x7f68200a67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:11.383 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.206+0000 7f68280e6700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6820061850 0x7f6820061ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:11.383 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.206+0000 7f68280e6700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68200a6cc0 con 0x7f6820061850 2020-11-20T11:45:11.383 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.206+0000 7f68280e6700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68200a3cc0 con 0x7f6820063fe0 2020-11-20T11:45:11.384 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.206+0000 7f68280e6700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68200a4640 con 0x7f68200ad7c0 2020-11-20T11:45:11.384 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.207+0000 7f6825e82700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68200ad7c0 msgr2=0x7f68200a62e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:45:11.384 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.207+0000 7f6825e82700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68200ad7c0 0x7f68200a62e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:45:11.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.208+0000 7f6825681700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6820063fe0 0x7f68200a67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:11.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.208+0000 7f6825681700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6820063fe0 0x7f68200a67d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:38732/0 (socket says 172.21.15.23:38732) 2020-11-20T11:45:11.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.208+0000 7f6825681700 1 -- 172.21.15.23:0/3238257410 learned_addr learned my addr 172.21.15.23:0/3238257410 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:45:11.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.209+0000 7f6826683700 1 --2- 172.21.15.23:0/3238257410 >> v2:172.21.15.23:3300/0 conn(0x7f6820061850 0x7f6820061ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:11.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.209+0000 7f6825681700 1 -- 172.21.15.23:0/3238257410 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68200ad7c0 msgr2=0x7f68200a62e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:45:11.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.209+0000 7f6825681700 1 --2- 172.21.15.23:0/3238257410 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68200ad7c0 0x7f68200a62e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:11.394 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.209+0000 7f6825681700 1 -- 172.21.15.23:0/3238257410 >> v2:172.21.15.23:3300/0 conn(0x7f6820061850 msgr2=0x7f6820061ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:45:11.394 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.209+0000 7f6825681700 1 --2- 172.21.15.23:0/3238257410 >> v2:172.21.15.23:3300/0 conn(0x7f6820061850 0x7f6820061ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:11.394 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.209+0000 7f6825681700 1 -- 172.21.15.23:0/3238257410 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68200a5ba0 con 0x7f6820063fe0 2020-11-20T11:45:11.395 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.209+0000 7f6825681700 1 --2- 172.21.15.23:0/3238257410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6820063fe0 0x7f68200a67d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f68200615e0 tx=0x7f68180092d0).ready entity=mon.1 client_cookie=49565b60e3ba5edd server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:11.395 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.210+0000 7f6824e80700 1 -- 172.21.15.23:0/3238257410 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681800bbf0 con 0x7f6820063fe0 2020-11-20T11:45:11.395 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.210+0000 7f6824e80700 1 -- 172.21.15.23:0/3238257410 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f681800bd50 con 0x7f6820063fe0 2020-11-20T11:45:11.395 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.210+0000 7f6824e80700 1 -- 172.21.15.23:0/3238257410 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6818011610 con 0x7f6820063fe0 2020-11-20T11:45:11.396 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.211+0000 7f68280e6700 1 -- 172.21.15.23:0/3238257410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6820063fe0 msgr2=0x7f68200a67d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:11.396 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.211+0000 7f68280e6700 1 --2- 172.21.15.23:0/3238257410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6820063fe0 0x7f68200a67d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f68200615e0 tx=0x7f68180092d0).stop 2020-11-20T11:45:11.396 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.228+0000 7f68280e6700 1 -- 172.21.15.23:0/3238257410 shutdown_connections 2020-11-20T11:45:11.397 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.228+0000 7f68280e6700 1 --2- 172.21.15.23:0/3238257410 >> v2:172.21.15.23:3300/0 conn(0x7f6820061850 0x7f6820061ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:11.397 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.228+0000 7f68280e6700 1 --2- 172.21.15.23:0/3238257410 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68200ad7c0 0x7f68200a62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:11.397 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.228+0000 7f68280e6700 1 --2- 172.21.15.23:0/3238257410 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6820063fe0 0x7f68200a67d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:11.398 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.228+0000 7f68280e6700 1 -- 172.21.15.23:0/3238257410 >> 172.21.15.23:0/3238257410 conn(0x7f6820060c80 msgr2=0x7f68200acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:11.398 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.230+0000 7f68280e6700 1 -- 172.21.15.23:0/3238257410 shutdown_connections 2020-11-20T11:45:11.398 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.230+0000 7f68280e6700 1 -- 172.21.15.23:0/3238257410 wait complete. 2020-11-20T11:45:11.399 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.231+0000 7f68280e6700 1 Processor -- start 2020-11-20T11:45:11.399 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.231+0000 7f68280e6700 1 -- start start 2020-11-20T11:45:11.399 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.232+0000 7f68280e6700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6820061850 0x7f68201423d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:11.400 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.232+0000 7f68280e6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6820063fe0 0x7f682014e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:11.400 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.232+0000 7f68280e6700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68200ad7c0 0x7f682014eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:11.400 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.232+0000 7f68280e6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f682014f090 con 0x7f6820063fe0 2020-11-20T11:45:11.401 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.232+0000 7f68280e6700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68200676b0 con 0x7f6820061850 2020-11-20T11:45:11.401 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.232+0000 7f68280e6700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6820067950 con 0x7f68200ad7c0 2020-11-20T11:45:11.401 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.233+0000 7f6825681700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6820063fe0 0x7f682014e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:11.402 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.233+0000 7f6826683700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68200ad7c0 msgr2=0x7f682014eba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:45:11.402 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.233+0000 7f6825681700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6820063fe0 0x7f682014e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45842/0 (socket says 172.21.15.23:45842) 2020-11-20T11:45:11.402 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.233+0000 7f6825681700 1 -- 172.21.15.23:0/4072584377 learned_addr learned my addr 172.21.15.23:0/4072584377 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:45:11.402 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.233+0000 7f6826683700 1 --2- 172.21.15.23:0/4072584377 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68200ad7c0 0x7f682014eba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:45:11.403 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.233+0000 7f6825681700 1 -- 172.21.15.23:0/4072584377 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68200ad7c0 msgr2=0x7f682014eba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:45:11.403 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.233+0000 7f6825681700 1 --2- 172.21.15.23:0/4072584377 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68200ad7c0 0x7f682014eba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:11.403 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.233+0000 7f6825681700 1 -- 172.21.15.23:0/4072584377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6820061850 msgr2=0x7f68201423d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:11.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.233+0000 7f6825681700 1 --2- 172.21.15.23:0/4072584377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6820061850 0x7f68201423d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:11.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.233+0000 7f6825681700 1 -- 172.21.15.23:0/4072584377 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6820067bf0 con 0x7f6820063fe0 2020-11-20T11:45:11.404 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.234+0000 7f6825681700 1 --2- 172.21.15.23:0/4072584377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6820063fe0 0x7f682014e6b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f6818009310 tx=0x7f6818016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:11.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.234+0000 7f6815ffb700 1 -- 172.21.15.23:0/4072584377 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681801c680 con 0x7f6820063fe0 2020-11-20T11:45:11.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.234+0000 7f6815ffb700 1 -- 172.21.15.23:0/4072584377 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f681801c7e0 con 0x7f6820063fe0 2020-11-20T11:45:11.405 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.234+0000 7f6815ffb700 1 -- 172.21.15.23:0/4072584377 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681801cb80 con 0x7f6820063fe0 2020-11-20T11:45:11.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.234+0000 7f68280e6700 1 -- 172.21.15.23:0/4072584377 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6820141f10 con 0x7f6820063fe0 2020-11-20T11:45:11.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.234+0000 7f68280e6700 1 -- 172.21.15.23:0/4072584377 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68200673e0 con 0x7f6820063fe0 2020-11-20T11:45:11.406 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.235+0000 7f68280e6700 1 --2- 172.21.15.23:0/4072584377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6820144270 0x7f6820139ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:11.407 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.236+0000 7f68280e6700 1 -- 172.21.15.23:0/4072584377 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68200a3e30 con 0x7f6820144270 2020-11-20T11:45:11.407 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.236+0000 7f6825e82700 1 --2- 172.21.15.23:0/4072584377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6820144270 0x7f6820139ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:11.407 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.237+0000 7f6825e82700 1 -- 172.21.15.23:0/4072584377 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6820067fb0 con 0x7f6820144270 2020-11-20T11:45:11.407 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.237+0000 7f6815ffb700 1 -- 172.21.15.23:0/4072584377 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6818017070 con 0x7f6820063fe0 2020-11-20T11:45:11.408 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.237+0000 7f6825e82700 1 --2- 172.21.15.23:0/4072584377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6820144270 0x7f6820139ea0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f681c003690 tx=0x7f681c006e10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:11.408 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.237+0000 7f6815ffb700 1 --2- 172.21.15.23:0/4072584377 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6800056d60 0x7f6800059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:11.408 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.237+0000 7f6815ffb700 1 -- 172.21.15.23:0/4072584377 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f6818072050 con 0x7f6820063fe0 2020-11-20T11:45:11.409 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.237+0000 7f6815ffb700 1 -- 172.21.15.23:0/4072584377 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681c006170 con 0x7f6820144270 2020-11-20T11:45:11.409 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.237+0000 7f6826683700 1 --2- 172.21.15.23:0/4072584377 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6800056d60 0x7f6800059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:11.409 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.238+0000 7f6815ffb700 1 -- 172.21.15.23:0/4072584377 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f681c0062d0 con 0x7f6820144270 2020-11-20T11:45:11.410 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.238+0000 7f6815ffb700 1 -- 172.21.15.23:0/4072584377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6820144270 msgr2=0x7f6820139ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:11.410 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.238+0000 7f6815ffb700 1 --2- 172.21.15.23:0/4072584377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6820144270 0x7f6820139ea0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f681c003690 tx=0x7f681c006e10).stop 2020-11-20T11:45:11.410 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.238+0000 7f6815ffb700 1 -- 172.21.15.23:0/4072584377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6820144270 msgr2=0x7f6820139ea0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:45:11.410 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.238+0000 7f6815ffb700 1 --2- 172.21.15.23:0/4072584377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6820144270 0x7f6820139ea0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:11.411 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.238+0000 7f6826683700 1 --2- 172.21.15.23:0/4072584377 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6800056d60 0x7f6800059220 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f6820143cb0 tx=0x7f6810008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:11.411 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.257+0000 7f68280e6700 1 --2- 172.21.15.23:0/4072584377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f682013a390 0x7f6820151c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:11.411 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.257+0000 7f68280e6700 1 -- 172.21.15.23:0/4072584377 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6820061650 con 0x7f682013a390 2020-11-20T11:45:11.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.257+0000 7f6825e82700 1 --2- 172.21.15.23:0/4072584377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f682013a390 0x7f6820151c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:11.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.258+0000 7f6825e82700 1 -- 172.21.15.23:0/4072584377 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6820067d80 con 0x7f682013a390 2020-11-20T11:45:11.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.258+0000 7f6825e82700 1 --2- 172.21.15.23:0/4072584377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f682013a390 0x7f6820151c80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f681c003690 tx=0x7f681c013000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:11.413 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.259+0000 7f6815ffb700 1 -- 172.21.15.23:0/4072584377 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681c013d30 con 0x7f682013a390 2020-11-20T11:45:11.413 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.259+0000 7f6815ffb700 1 -- 172.21.15.23:0/4072584377 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4773 (secure 0 0 0) 0x7f681c013e90 con 0x7f682013a390 2020-11-20T11:45:11.413 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.259+0000 7f6815ffb700 1 -- 172.21.15.23:0/4072584377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f682013a390 msgr2=0x7f6820151c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:11.414 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.259+0000 7f6815ffb700 1 --2- 172.21.15.23:0/4072584377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f682013a390 0x7f6820151c80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f681c003690 tx=0x7f681c013000).stop 2020-11-20T11:45:11.414 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.259+0000 7f6815ffb700 1 -- 172.21.15.23:0/4072584377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f682013a390 msgr2=0x7f6820151c80 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:45:11.414 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.259+0000 7f6815ffb700 1 --2- 172.21.15.23:0/4072584377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f682013a390 0x7f6820151c80 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0x7f681c003690 tx=0x7f681c013000).stop 2020-11-20T11:45:11.415 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.259+0000 7f68280e6700 1 -- 172.21.15.23:0/4072584377 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6800056d60 msgr2=0x7f6800059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:11.415 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.259+0000 7f68280e6700 1 --2- 172.21.15.23:0/4072584377 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6800056d60 0x7f6800059220 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f6820143cb0 tx=0x7f6810008040).stop 2020-11-20T11:45:11.415 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.259+0000 7f6826683700 1 -- 172.21.15.23:0/4072584377 reap_dead start 2020-11-20T11:45:11.415 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.259+0000 7f68280e6700 1 -- 172.21.15.23:0/4072584377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6820063fe0 msgr2=0x7f682014e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:11.416 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.259+0000 7f68280e6700 1 --2- 172.21.15.23:0/4072584377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6820063fe0 0x7f682014e6b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f6818009310 tx=0x7f6818016000).stop 2020-11-20T11:45:11.416 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.260+0000 7f68280e6700 1 -- 172.21.15.23:0/4072584377 shutdown_connections 2020-11-20T11:45:11.416 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.260+0000 7f68280e6700 1 --2- 172.21.15.23:0/4072584377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6820063fe0 0x7f682014e6b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:11.417 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.260+0000 7f68280e6700 1 -- 172.21.15.23:0/4072584377 >> 172.21.15.23:0/4072584377 conn(0x7f6820060c80 msgr2=0x7f68200a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:11.417 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.261+0000 7f68280e6700 1 -- 172.21.15.23:0/4072584377 shutdown_connections 2020-11-20T11:45:11.417 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:11.261+0000 7f68280e6700 1 -- 172.21.15.23:0/4072584377 wait complete. 2020-11-20T11:45:11.418 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:45:11.418 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:45:11.418 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:45:11.419 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:45:11.419 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 120, 2020-11-20T11:45:11.419 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:45:11.420 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:45:11.420 INFO:teuthology.orchestra.run.smithi023.stdout: 1 2020-11-20T11:45:11.420 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:11.421 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 0, 2020-11-20T11:45:11.421 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:45:11.421 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:45:11.422 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:45:11.422 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:45:11.422 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:45:11.422 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:45:11.423 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:45:11.423 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:45:11.423 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:45:11.424 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:45:11.424 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:45:11.424 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:11.425 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:45:11.425 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:45:11.425 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:45:11.426 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:45:11.426 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:45:11.427 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:45:11.427 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:45:11.427 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:45:11.428 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:45:11.428 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:45:11.428 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:11.429 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:11.429 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:45:11.429 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:45:11.430 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:45:11.430 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:45:11.430 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:45:11.431 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:45:11.431 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:45:11.432 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:45:11.432 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:45:11.432 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:45:11.433 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:45:11.433 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:45:11.433 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:45:11.434 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:45:11.434 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:45:11.434 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:45:11.435 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:45:11.435 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:45:11.436 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:45:11.436 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:45:11.436 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:45:11.436 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:45:11.437 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:45:11.437 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:11.438 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:45:11.438 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:11.438 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:45:11.439 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:11.439 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:45:11.439 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:45:11.440 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:45:11.440 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:45:11.440 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:11.441 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:45:11.441 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:45:11.441 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:11.442 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:11.442 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:11.442 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:45:11.443 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:45:11.443 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:11.444 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:11.444 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:11.444 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:11.445 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:45:11.445 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:45:11.445 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:45:11.446 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:45:11.446 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:45:11.446 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:11.447 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:11.447 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:45:11.447 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:45:11.448 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:45:11.448 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:45:11.448 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:11.449 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:45:11.449 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:45:11.449 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:11.450 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:11.450 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:11.450 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:45:11.451 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:45:11.451 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:11.451 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:11.452 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:11.452 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:11.452 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:45:11.453 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:45:11.453 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:45:11.453 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:45:11.454 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:45:11.454 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:11.454 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:11.455 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:45:11.455 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:45:11.455 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:45:11.456 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:45:11.456 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:11.456 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:45:11.457 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:45:11.457 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:11.457 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:11.458 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:11.458 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:45:11.458 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:45:11.459 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:11.462 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:11.462 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:11.462 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:11.463 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:45:11.463 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:45:11.463 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:45:11.463 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:45:11.464 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:45:11.464 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:11.464 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:11.465 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:11.465 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:45:11.465 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:45:11.466 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:11.466 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:45:11.466 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:45:11.466 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:45:11.467 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:11.467 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:11.467 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:45:11.467 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:11.467 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:45:11.468 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:45:11.468 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:45:11.468 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:11.468 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:11.468 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:45:11.469 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:11.469 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:45:11.469 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:45:11.469 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:45:11.470 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:11.470 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:11.470 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:45:11.470 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:11.470 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:45:11.471 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:45:11.471 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:45:11.471 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:11.471 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:11.472 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:45:11.472 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:11.472 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:45:11.472 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:45:11.473 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:45:11.473 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:11.473 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:11.473 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:11.474 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:45:11.474 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:45:11.474 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2020-11-20T11:45:31.475 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2020-11-20T11:45:31.476 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2020-11-20T11:45:31.476 INFO:tasks.ceph.mon.c:Restarting daemon 2020-11-20T11:45:31.476 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2020-11-20T11:45:31.479 INFO:tasks.ceph.mon.c:Started 2020-11-20T11:45:31.479 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:45:31.480 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:45:31.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.679+0000 7fac00657700 1 Processor -- start 2020-11-20T11:45:31.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.679+0000 7fac00657700 1 -- start start 2020-11-20T11:45:31.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.680+0000 7fac00657700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fabf80aaab0 0x7fabf813ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:31.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.680+0000 7fac00657700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fabf80aae90 0x7fabf81381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:31.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.680+0000 7fac00657700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fabf813aeb0 0x7fabf80a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:31.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.680+0000 7fac00657700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabf813e600 con 0x7fabf813aeb0 2020-11-20T11:45:31.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.680+0000 7fac00657700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabf813e8a0 con 0x7fabf80aae90 2020-11-20T11:45:31.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.680+0000 7fac00657700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabf80ab6d0 con 0x7fabf80aaab0 2020-11-20T11:45:31.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.681+0000 7fabfe3f3700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fabf80aaab0 msgr2=0x7fabf813ed40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:45:31.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.681+0000 7fabfe3f3700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fabf80aaab0 0x7fabf813ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:45:31.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.681+0000 7fabfebf4700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fabf813aeb0 0x7fabf80a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:31.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.681+0000 7fabfebf4700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fabf813aeb0 0x7fabf80a8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45858/0 (socket says 172.21.15.23:45858) 2020-11-20T11:45:31.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.681+0000 7fabfebf4700 1 -- 172.21.15.23:0/2421852437 learned_addr learned my addr 172.21.15.23:0/2421852437 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:45:31.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.681+0000 7fabfdbf2700 1 --2- 172.21.15.23:0/2421852437 >> v2:172.21.15.155:3300/0 conn(0x7fabf80aae90 0x7fabf81381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:31.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.681+0000 7fabfebf4700 1 -- 172.21.15.23:0/2421852437 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fabf80aaab0 msgr2=0x7fabf813ed40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:45:31.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.681+0000 7fabfebf4700 1 --2- 172.21.15.23:0/2421852437 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fabf80aaab0 0x7fabf813ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:31.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.681+0000 7fabfebf4700 1 -- 172.21.15.23:0/2421852437 >> v2:172.21.15.155:3300/0 conn(0x7fabf80aae90 msgr2=0x7fabf81381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:45:31.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.681+0000 7fabfebf4700 1 --2- 172.21.15.23:0/2421852437 >> v2:172.21.15.155:3300/0 conn(0x7fabf80aae90 0x7fabf81381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:31.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.681+0000 7fabfebf4700 1 -- 172.21.15.23:0/2421852437 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf813f230 con 0x7fabf813aeb0 2020-11-20T11:45:31.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.681+0000 7fabfebf4700 1 --2- 172.21.15.23:0/2421852437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabf813aeb0 0x7fabf80a8c60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fabf8062e20 tx=0x7fabf00092d0).ready entity=mon.0 client_cookie=e7a31ded6d7aeee4 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:31.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.684+0000 7fabfd3f1700 1 -- 172.21.15.23:0/2421852437 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf000bb10 con 0x7fabf813aeb0 2020-11-20T11:45:31.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.684+0000 7fabfd3f1700 1 -- 172.21.15.23:0/2421852437 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf000bc70 con 0x7fabf813aeb0 2020-11-20T11:45:31.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.684+0000 7fabfd3f1700 1 -- 172.21.15.23:0/2421852437 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf0011400 con 0x7fabf813aeb0 2020-11-20T11:45:31.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.684+0000 7fac00657700 1 -- 172.21.15.23:0/2421852437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabf813aeb0 msgr2=0x7fabf80a8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:31.689 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.684+0000 7fac00657700 1 --2- 172.21.15.23:0/2421852437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabf813aeb0 0x7fabf80a8c60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fabf8062e20 tx=0x7fabf00092d0).stop 2020-11-20T11:45:31.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.687+0000 7fac00657700 1 -- 172.21.15.23:0/2421852437 shutdown_connections 2020-11-20T11:45:31.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.687+0000 7fac00657700 1 --2- 172.21.15.23:0/2421852437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabf813aeb0 0x7fabf80a8c60 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:31.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.687+0000 7fac00657700 1 --2- 172.21.15.23:0/2421852437 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fabf80aaab0 0x7fabf813ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:31.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.687+0000 7fac00657700 1 --2- 172.21.15.23:0/2421852437 >> v2:172.21.15.155:3300/0 conn(0x7fabf80aae90 0x7fabf81381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:31.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.687+0000 7fac00657700 1 -- 172.21.15.23:0/2421852437 >> 172.21.15.23:0/2421852437 conn(0x7fabf80678e0 msgr2=0x7fabf8067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:31.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.687+0000 7fac00657700 1 -- 172.21.15.23:0/2421852437 shutdown_connections 2020-11-20T11:45:31.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.687+0000 7fac00657700 1 -- 172.21.15.23:0/2421852437 wait complete. 2020-11-20T11:45:31.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.688+0000 7fac00657700 1 Processor -- start 2020-11-20T11:45:31.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.689+0000 7fac00657700 1 -- start start 2020-11-20T11:45:31.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.689+0000 7fac00657700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabf80aaab0 0x7fabf8140d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:31.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.689+0000 7fac00657700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fabf80aae90 0x7fabf80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:31.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.689+0000 7fac00657700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fabf80a6440 0x7fabf80a6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:31.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.689+0000 7fac00657700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf80a5a10 con 0x7fabf80aaab0 2020-11-20T11:45:31.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.689+0000 7fac00657700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf80a5fa0 con 0x7fabf80aae90 2020-11-20T11:45:31.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.689+0000 7fac00657700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabf8149ff0 con 0x7fabf80a6440 2020-11-20T11:45:31.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.689+0000 7fabfe3f3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabf80aaab0 0x7fabf8140d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:31.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.689+0000 7fabfebf4700 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fabf80a6440 msgr2=0x7fabf80a6860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3301/0 2020-11-20T11:45:31.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.689+0000 7fabfebf4700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fabf80a6440 0x7fabf80a6860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:45:31.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.689+0000 7fabfe3f3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabf80aaab0 0x7fabf8140d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45860/0 (socket says 172.21.15.23:45860) 2020-11-20T11:45:31.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.689+0000 7fabfe3f3700 1 -- 172.21.15.23:0/3653718897 learned_addr learned my addr 172.21.15.23:0/3653718897 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:45:31.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.689+0000 7fabfdbf2700 1 --2- 172.21.15.23:0/3653718897 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fabf80aae90 0x7fabf80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:31.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.690+0000 7fabfe3f3700 1 -- 172.21.15.23:0/3653718897 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fabf80a6440 msgr2=0x7fabf80a6860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:45:31.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.690+0000 7fabfe3f3700 1 --2- 172.21.15.23:0/3653718897 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fabf80a6440 0x7fabf80a6860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:31.699 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.690+0000 7fabfe3f3700 1 -- 172.21.15.23:0/3653718897 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fabf80aae90 msgr2=0x7fabf80a5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:31.699 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.690+0000 7fabfe3f3700 1 --2- 172.21.15.23:0/3653718897 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fabf80aae90 0x7fabf80a5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:31.699 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.690+0000 7fabfe3f3700 1 -- 172.21.15.23:0/3653718897 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf80a5660 con 0x7fabf80aaab0 2020-11-20T11:45:31.700 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.690+0000 7fabfe3f3700 1 --2- 172.21.15.23:0/3653718897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabf80aaab0 0x7fabf8140d10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fabe8009790 tx=0x7fabe8006ea0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:31.700 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.690+0000 7fabe67fc700 1 -- 172.21.15.23:0/3653718897 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe8006170 con 0x7fabf80aaab0 2020-11-20T11:45:31.700 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.690+0000 7fabe67fc700 1 -- 172.21.15.23:0/3653718897 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe80062d0 con 0x7fabf80aaab0 2020-11-20T11:45:31.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.690+0000 7fac00657700 1 -- 172.21.15.23:0/3653718897 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabf80a80d0 con 0x7fabf80aaab0 2020-11-20T11:45:31.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.690+0000 7fac00657700 1 -- 172.21.15.23:0/3653718897 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabf814a6d0 con 0x7fabf80aaab0 2020-11-20T11:45:31.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.690+0000 7fabe67fc700 1 -- 172.21.15.23:0/3653718897 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe8003920 con 0x7fabf80aaab0 2020-11-20T11:45:31.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.691+0000 7fac00657700 1 -- 172.21.15.23:0/3653718897 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabf805c7d0 con 0x7fabf80aaab0 2020-11-20T11:45:31.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.691+0000 7fabe67fc700 1 -- 172.21.15.23:0/3653718897 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fabe8017030 con 0x7fabf80aaab0 2020-11-20T11:45:31.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.692+0000 7fabe67fc700 1 --2- 172.21.15.23:0/3653718897 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fabd0056d10 0x7fabd00591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:31.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.692+0000 7fabe67fc700 1 -- 172.21.15.23:0/3653718897 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fabe806f320 con 0x7fabf80aaab0 2020-11-20T11:45:31.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.692+0000 7fabfdbf2700 1 --2- 172.21.15.23:0/3653718897 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fabd0056d10 0x7fabd00591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:31.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.692+0000 7fabfdbf2700 1 --2- 172.21.15.23:0/3653718897 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fabd0056d10 0x7fabd00591d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fabf80b18e0 tx=0x7fabf4006e30).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:31.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.696+0000 7fabe67fc700 1 -- 172.21.15.23:0/3653718897 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fabe80445b0 con 0x7fabf80aaab0 2020-11-20T11:45:31.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.943+0000 7fac00657700 1 -- 172.21.15.23:0/3653718897 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fabf80a5850 con 0x7fabf80aaab0 2020-11-20T11:45:31.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.943+0000 7fabe67fc700 1 -- 172.21.15.23:0/3653718897 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1473 (secure 0 0 0) 0x7fabe8047bd0 con 0x7fabf80aaab0 2020-11-20T11:45:31.947 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":120,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:45:31.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.945+0000 7fac00657700 1 -- 172.21.15.23:0/3653718897 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fabd0056d10 msgr2=0x7fabd00591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:31.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.945+0000 7fac00657700 1 --2- 172.21.15.23:0/3653718897 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fabd0056d10 0x7fabd00591d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fabf80b18e0 tx=0x7fabf4006e30).stop 2020-11-20T11:45:31.949 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.945+0000 7fac00657700 1 -- 172.21.15.23:0/3653718897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabf80aaab0 msgr2=0x7fabf8140d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:31.949 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.945+0000 7fac00657700 1 --2- 172.21.15.23:0/3653718897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabf80aaab0 0x7fabf8140d10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fabe8009790 tx=0x7fabe8006ea0).stop 2020-11-20T11:45:31.949 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.945+0000 7fac00657700 1 -- 172.21.15.23:0/3653718897 shutdown_connections 2020-11-20T11:45:31.949 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.945+0000 7fac00657700 1 --2- 172.21.15.23:0/3653718897 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fabd0056d10 0x7fabd00591d0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:31.949 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.945+0000 7fac00657700 1 --2- 172.21.15.23:0/3653718897 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fabf80a6440 0x7fabf80a6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:31.950 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.945+0000 7fac00657700 1 --2- 172.21.15.23:0/3653718897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabf80aaab0 0x7fabf8140d10 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:31.950 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.945+0000 7fac00657700 1 --2- 172.21.15.23:0/3653718897 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fabf80aae90 0x7fabf80a5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:31.950 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.945+0000 7fac00657700 1 -- 172.21.15.23:0/3653718897 >> 172.21.15.23:0/3653718897 conn(0x7fabf80678e0 msgr2=0x7fabf813a020 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:31.950 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.945+0000 7fac00657700 1 -- 172.21.15.23:0/3653718897 shutdown_connections 2020-11-20T11:45:31.950 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:31.945+0000 7fac00657700 1 -- 172.21.15.23:0/3653718897 wait complete. 2020-11-20T11:45:34.966 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:45:35.438 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:35.431+0000 7f25181bf700 1 Processor -- start 2020-11-20T11:45:35.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:35.432+0000 7f25181bf700 1 -- start start 2020-11-20T11:45:35.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:35.432+0000 7f25181bf700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f25100aae90 0x7f25101381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:35.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:35.432+0000 7f25181bf700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f251013aeb0 0x7f25100a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:35.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:35.432+0000 7f25181bf700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f251013e600 con 0x7f25100aaab0 2020-11-20T11:45:35.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:35.432+0000 7f25181bf700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f251013e8a0 con 0x7f25100aae90 2020-11-20T11:45:35.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:35.432+0000 7f25181bf700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25100ab6d0 con 0x7f251013aeb0 2020-11-20T11:45:35.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:35.432+0000 7f251575a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f25100aae90 0x7f25101381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:35.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:35.432+0000 7f2515f5b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f25100aaab0 0x7f251013ed40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:60118/0 (socket says 172.21.15.23:60118) 2020-11-20T11:45:35.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:35.432+0000 7f2515f5b700 1 -- 172.21.15.23:0/384651333 learned_addr learned my addr 172.21.15.23:0/384651333 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:45:35.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:35.432+0000 7f251575a700 1 -- 172.21.15.23:0/384651333 >> v2:172.21.15.23:3300/0 conn(0x7f25100aae90 msgr2=0x7f25101381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:45:35.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:35.432+0000 7f251575a700 1 -- 172.21.15.23:0/384651333 >> v2:172.21.15.23:3300/0 conn(0x7f25100aae90 msgr2=0x7f25101381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:45:35.442 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:35.432+0000 7f251575a700 1 --2- 172.21.15.23:0/384651333 >> v2:172.21.15.23:3300/0 conn(0x7f25100aae90 0x7f25101381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:45:35.442 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:35.433+0000 7f251575a700 1 --2- 172.21.15.23:0/384651333 >> v2:172.21.15.23:3300/0 conn(0x7f25100aae90 0x7f25101381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:45:35.442 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:35.433+0000 7f251675c700 1 --2- 172.21.15.23:0/384651333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f251013aeb0 0x7f25100a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:35.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:35.433+0000 7f251675c700 1 -- 172.21.15.23:0/384651333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f251013aeb0 msgr2=0x7f25100a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:45:35.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:35.433+0000 7f251675c700 1 -- 172.21.15.23:0/384651333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f251013aeb0 msgr2=0x7f25100a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:45:35.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:35.433+0000 7f251675c700 1 --2- 172.21.15.23:0/384651333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f251013aeb0 0x7f25100a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:45:35.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:35.433+0000 7f251675c700 1 --2- 172.21.15.23:0/384651333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f251013aeb0 0x7f25100a8c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:45:35.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:35.634+0000 7f251575a700 1 --2- 172.21.15.23:0/384651333 >> v2:172.21.15.23:3300/0 conn(0x7f25100aae90 0x7f25101381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:35.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:35.634+0000 7f251675c700 1 --2- 172.21.15.23:0/384651333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f251013aeb0 0x7f25100a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:35.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:35.634+0000 7f251575a700 1 -- 172.21.15.23:0/384651333 >> v2:172.21.15.23:3300/0 conn(0x7f25100aae90 msgr2=0x7f25101381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:45:35.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:35.634+0000 7f251575a700 1 -- 172.21.15.23:0/384651333 >> v2:172.21.15.23:3300/0 conn(0x7f25100aae90 msgr2=0x7f25101381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:45:35.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:35.634+0000 7f251575a700 1 --2- 172.21.15.23:0/384651333 >> v2:172.21.15.23:3300/0 conn(0x7f25100aae90 0x7f25101381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:45:35.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:35.634+0000 7f251575a700 1 --2- 172.21.15.23:0/384651333 >> v2:172.21.15.23:3300/0 conn(0x7f25100aae90 0x7f25101381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:45:35.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:35.634+0000 7f251675c700 1 -- 172.21.15.23:0/384651333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f251013aeb0 msgr2=0x7f25100a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:45:35.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:35.634+0000 7f251675c700 1 -- 172.21.15.23:0/384651333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f251013aeb0 msgr2=0x7f25100a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:45:35.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:35.634+0000 7f251675c700 1 --2- 172.21.15.23:0/384651333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f251013aeb0 0x7f25100a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:45:35.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:35.634+0000 7f251675c700 1 --2- 172.21.15.23:0/384651333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f251013aeb0 0x7f25100a8c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:45:36.038 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:36.035+0000 7f251575a700 1 --2- 172.21.15.23:0/384651333 >> v2:172.21.15.23:3300/0 conn(0x7f25100aae90 0x7f25101381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:36.039 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:36.035+0000 7f251575a700 1 -- 172.21.15.23:0/384651333 >> v2:172.21.15.23:3300/0 conn(0x7f25100aae90 msgr2=0x7f25101381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:45:36.039 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:36.035+0000 7f251575a700 1 -- 172.21.15.23:0/384651333 >> v2:172.21.15.23:3300/0 conn(0x7f25100aae90 msgr2=0x7f25101381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:45:36.039 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:36.035+0000 7f251575a700 1 --2- 172.21.15.23:0/384651333 >> v2:172.21.15.23:3300/0 conn(0x7f25100aae90 0x7f25101381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:45:36.039 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:36.035+0000 7f251675c700 1 --2- 172.21.15.23:0/384651333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f251013aeb0 0x7f25100a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:36.040 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:36.035+0000 7f251575a700 1 --2- 172.21.15.23:0/384651333 >> v2:172.21.15.23:3300/0 conn(0x7f25100aae90 0x7f25101381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:45:36.040 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:36.035+0000 7f251675c700 1 -- 172.21.15.23:0/384651333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f251013aeb0 msgr2=0x7f25100a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:45:36.040 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:36.035+0000 7f251675c700 1 -- 172.21.15.23:0/384651333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f251013aeb0 msgr2=0x7f25100a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:45:36.040 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:36.035+0000 7f251675c700 1 --2- 172.21.15.23:0/384651333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f251013aeb0 0x7f25100a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:45:36.040 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:36.035+0000 7f251675c700 1 --2- 172.21.15.23:0/384651333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f251013aeb0 0x7f25100a8c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:45:36.839 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:36.836+0000 7f251575a700 1 --2- 172.21.15.23:0/384651333 >> v2:172.21.15.23:3300/0 conn(0x7f25100aae90 0x7f25101381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:36.839 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:36.836+0000 7f251575a700 1 -- 172.21.15.23:0/384651333 >> v2:172.21.15.23:3300/0 conn(0x7f25100aae90 msgr2=0x7f25101381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:45:36.840 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:36.836+0000 7f251575a700 1 -- 172.21.15.23:0/384651333 >> v2:172.21.15.23:3300/0 conn(0x7f25100aae90 msgr2=0x7f25101381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:45:36.840 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:36.836+0000 7f251575a700 1 --2- 172.21.15.23:0/384651333 >> v2:172.21.15.23:3300/0 conn(0x7f25100aae90 0x7f25101381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:45:36.840 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:36.836+0000 7f251575a700 1 --2- 172.21.15.23:0/384651333 >> v2:172.21.15.23:3300/0 conn(0x7f25100aae90 0x7f25101381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:45:36.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:36.837+0000 7f251675c700 1 --2- 172.21.15.23:0/384651333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f251013aeb0 0x7f25100a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:36.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:36.838+0000 7f251675c700 1 -- 172.21.15.23:0/384651333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f251013aeb0 msgr2=0x7f25100a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:45:36.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:36.838+0000 7f251675c700 1 -- 172.21.15.23:0/384651333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f251013aeb0 msgr2=0x7f25100a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:45:36.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:36.838+0000 7f251675c700 1 --2- 172.21.15.23:0/384651333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f251013aeb0 0x7f25100a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:45:36.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:36.838+0000 7f251675c700 1 --2- 172.21.15.23:0/384651333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f251013aeb0 0x7f25100a8c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:45:38.320 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.317+0000 7f2514f59700 1 -- 172.21.15.23:0/384651333 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 592380010 0 0) 0x7f251013e600 con 0x7f25100aaab0 2020-11-20T11:45:38.320 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.317+0000 7f2514f59700 1 -- 172.21.15.23:0/384651333 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24f8003280 con 0x7f25100aaab0 2020-11-20T11:45:38.329 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.326+0000 7f2514f59700 1 -- 172.21.15.23:0/384651333 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2078916216 0 0) 0x7f24f8003280 con 0x7f25100aaab0 2020-11-20T11:45:38.330 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.326+0000 7f2514f59700 1 -- 172.21.15.23:0/384651333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f251013aeb0 msgr2=0x7f25100a8c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:45:38.330 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.326+0000 7f2514f59700 1 --2- 172.21.15.23:0/384651333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f251013aeb0 0x7f25100a8c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:38.330 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.326+0000 7f2514f59700 1 -- 172.21.15.23:0/384651333 >> v2:172.21.15.23:3300/0 conn(0x7f25100aae90 msgr2=0x7f25101381a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:45:38.330 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.326+0000 7f2514f59700 1 --2- 172.21.15.23:0/384651333 >> v2:172.21.15.23:3300/0 conn(0x7f25100aae90 0x7f25101381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:38.331 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.326+0000 7f2514f59700 1 -- 172.21.15.23:0/384651333 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2510138690 con 0x7f25100aaab0 2020-11-20T11:45:38.331 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.326+0000 7f2514f59700 1 -- 172.21.15.23:0/384651333 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f250c001520 con 0x7f25100aaab0 2020-11-20T11:45:38.331 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.326+0000 7f2514f59700 1 -- 172.21.15.23:0/384651333 >> v1:172.21.15.23:6789/0 conn(0x7f25100aaab0 legacy=0x7f251013ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:38.331 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.326+0000 7f2514f59700 1 --2- 172.21.15.23:0/384651333 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24f80037f0 0x7f24f8003c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:38.332 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.326+0000 7f2514f59700 1 --2- 172.21.15.23:0/384651333 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24f8004520 0x7f24f80049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:38.333 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.326+0000 7f2514f59700 1 --2- 172.21.15.23:0/384651333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f24f8004e90 0x7f24f800b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:38.334 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.326+0000 7f2514f59700 1 -- 172.21.15.23:0/384651333 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251013e8a0 con 0x7f24f80037f0 2020-11-20T11:45:38.334 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.326+0000 7f2514f59700 1 -- 172.21.15.23:0/384651333 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25100ab6d0 con 0x7f24f8004520 2020-11-20T11:45:38.334 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.326+0000 7f2514f59700 1 -- 172.21.15.23:0/384651333 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24f80052d0 con 0x7f24f8004e90 2020-11-20T11:45:38.334 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.327+0000 7f251675c700 1 --2- 172.21.15.23:0/384651333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f24f8004e90 0x7f24f800b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:38.334 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.327+0000 7f2515f5b700 1 --2- 172.21.15.23:0/384651333 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24f80037f0 0x7f24f8003c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:38.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.327+0000 7f2515f5b700 1 -- 172.21.15.23:0/384651333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f24f8004e90 msgr2=0x7f24f800b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:45:38.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.327+0000 7f2515f5b700 1 --2- 172.21.15.23:0/384651333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f24f8004e90 0x7f24f800b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:38.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.327+0000 7f2515f5b700 1 -- 172.21.15.23:0/384651333 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24f8004520 msgr2=0x7f24f80049a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:45:38.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.327+0000 7f2515f5b700 1 --2- 172.21.15.23:0/384651333 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24f8004520 0x7f24f80049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:38.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.327+0000 7f2515f5b700 1 -- 172.21.15.23:0/384651333 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2510138690 con 0x7f24f80037f0 2020-11-20T11:45:38.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.327+0000 7f251675c700 1 --2- 172.21.15.23:0/384651333 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f24f8004e90 0x7f24f800b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:45:38.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.328+0000 7f2515f5b700 1 --2- 172.21.15.23:0/384651333 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24f80037f0 0x7f24f8003c10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f250c00c490 tx=0x7f250c007810).ready entity=mon.0 client_cookie=43435503e5f8a91a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:38.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.329+0000 7f2514f59700 1 -- 172.21.15.23:0/384651333 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250c00ce50 con 0x7f24f80037f0 2020-11-20T11:45:38.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.329+0000 7f2514f59700 1 -- 172.21.15.23:0/384651333 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250c0072e0 con 0x7f24f80037f0 2020-11-20T11:45:38.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.329+0000 7f2514f59700 1 -- 172.21.15.23:0/384651333 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250c005880 con 0x7f24f80037f0 2020-11-20T11:45:38.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.330+0000 7f25181bf700 1 -- 172.21.15.23:0/384651333 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24f80037f0 msgr2=0x7f24f8003c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:38.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.330+0000 7f25181bf700 1 --2- 172.21.15.23:0/384651333 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24f80037f0 0x7f24f8003c10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f250c00c490 tx=0x7f250c007810).stop 2020-11-20T11:45:38.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.330+0000 7f251675c700 1 -- 172.21.15.23:0/384651333 reap_dead start 2020-11-20T11:45:38.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.330+0000 7f25181bf700 1 -- 172.21.15.23:0/384651333 shutdown_connections 2020-11-20T11:45:38.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.330+0000 7f25181bf700 1 -- 172.21.15.23:0/384651333 >> 172.21.15.23:0/384651333 conn(0x7f25100678e0 msgr2=0x7f2510067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:38.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.330+0000 7f25181bf700 1 -- 172.21.15.23:0/384651333 shutdown_connections 2020-11-20T11:45:38.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.330+0000 7f25181bf700 1 -- 172.21.15.23:0/384651333 wait complete. 2020-11-20T11:45:38.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.332+0000 7f25181bf700 1 Processor -- start 2020-11-20T11:45:38.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.332+0000 7f25181bf700 1 -- start start 2020-11-20T11:45:38.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.333+0000 7f25181bf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24f80037f0 0x7f25100a77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:38.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.333+0000 7f25181bf700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25100a5f30 0x7f251013d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:38.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.333+0000 7f25181bf700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f251013a980 0x7f2510134070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:38.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.333+0000 7f25181bf700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25100a6560 con 0x7f24f80037f0 2020-11-20T11:45:38.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.333+0000 7f25181bf700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25100a6d90 con 0x7f251013a980 2020-11-20T11:45:38.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.333+0000 7f25181bf700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25100a5170 con 0x7f25100a5f30 2020-11-20T11:45:38.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.333+0000 7f251675c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f251013a980 0x7f2510134070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:38.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.333+0000 7f251675c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f251013a980 0x7f2510134070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:38792/0 (socket says 172.21.15.23:38792) 2020-11-20T11:45:38.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.333+0000 7f251675c700 1 -- 172.21.15.23:0/1831191384 learned_addr learned my addr 172.21.15.23:0/1831191384 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:45:38.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.333+0000 7f2515f5b700 1 --2- 172.21.15.23:0/1831191384 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24f80037f0 0x7f25100a77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:38.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.333+0000 7f251575a700 1 --2- 172.21.15.23:0/1831191384 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25100a5f30 0x7f251013d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:38.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.334+0000 7f251575a700 1 -- 172.21.15.23:0/1831191384 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25100a5f30 msgr2=0x7f251013d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2020-11-20T11:45:38.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.334+0000 7f251575a700 1 -- 172.21.15.23:0/1831191384 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25100a5f30 msgr2=0x7f251013d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:45:38.345 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.334+0000 7f251575a700 1 --2- 172.21.15.23:0/1831191384 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25100a5f30 0x7f251013d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:45:38.345 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.334+0000 7f251575a700 1 --2- 172.21.15.23:0/1831191384 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25100a5f30 0x7f251013d6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:45:38.345 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.334+0000 7f251675c700 1 -- 172.21.15.23:0/1831191384 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25100a5f30 msgr2=0x7f251013d6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:45:38.345 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.334+0000 7f251675c700 1 --2- 172.21.15.23:0/1831191384 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25100a5f30 0x7f251013d6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:38.345 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.334+0000 7f251675c700 1 -- 172.21.15.23:0/1831191384 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24f80037f0 msgr2=0x7f25100a77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:38.346 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.334+0000 7f251675c700 1 --2- 172.21.15.23:0/1831191384 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24f80037f0 0x7f25100a77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:38.346 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.334+0000 7f251675c700 1 -- 172.21.15.23:0/1831191384 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25100a53a0 con 0x7f251013a980 2020-11-20T11:45:38.346 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.334+0000 7f251675c700 1 --2- 172.21.15.23:0/1831191384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f251013a980 0x7f2510134070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f2508009100 tx=0x7f2508008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:38.346 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.334+0000 7f24fdffb700 1 -- 172.21.15.23:0/1831191384 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2508015ec0 con 0x7f251013a980 2020-11-20T11:45:38.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.334+0000 7f25181bf700 1 -- 172.21.15.23:0/1831191384 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25100a5640 con 0x7f251013a980 2020-11-20T11:45:38.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.335+0000 7f25181bf700 1 -- 172.21.15.23:0/1831191384 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f251013ade0 con 0x7f251013a980 2020-11-20T11:45:38.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.335+0000 7f24fdffb700 1 -- 172.21.15.23:0/1831191384 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250800c070 con 0x7f251013a980 2020-11-20T11:45:38.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.335+0000 7f24fdffb700 1 -- 172.21.15.23:0/1831191384 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2508018450 con 0x7f251013a980 2020-11-20T11:45:38.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.336+0000 7f25181bf700 1 -- 172.21.15.23:0/1831191384 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f251013e3e0 con 0x7f251013a980 2020-11-20T11:45:38.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.337+0000 7f24fdffb700 1 -- 172.21.15.23:0/1831191384 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2508029070 con 0x7f251013a980 2020-11-20T11:45:38.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.337+0000 7f24fdffb700 1 --2- 172.21.15.23:0/1831191384 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f24e8056d90 0x7f24e8059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:38.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.337+0000 7f24fdffb700 1 -- 172.21.15.23:0/1831191384 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f2508070fb0 con 0x7f251013a980 2020-11-20T11:45:38.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.341+0000 7f2515f5b700 1 --2- 172.21.15.23:0/1831191384 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f24e8056d90 0x7f24e8059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:38.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.343+0000 7f2515f5b700 1 --2- 172.21.15.23:0/1831191384 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f24e8056d90 0x7f24e8059250 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f250c0045a0 tx=0x7f250c015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:38.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:38.345+0000 7f24fdffb700 1 -- 172.21.15.23:0/1831191384 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2508046190 con 0x7f251013a980 2020-11-20T11:45:39.029 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:39.026+0000 7f25181bf700 1 -- 172.21.15.23:0/1831191384 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f251005c7b0 con 0x7f251013a980 2020-11-20T11:45:39.030 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:39.026+0000 7f24fdffb700 1 -- 172.21.15.23:0/1831191384 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1472 (secure 0 0 0) 0x7f2508027070 con 0x7f251013a980 2020-11-20T11:45:39.036 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":124,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:45:39.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:39.043+0000 7f25181bf700 1 -- 172.21.15.23:0/1831191384 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f24e8056d90 msgr2=0x7f24e8059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:39.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:39.043+0000 7f25181bf700 1 --2- 172.21.15.23:0/1831191384 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f24e8056d90 0x7f24e8059250 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f250c0045a0 tx=0x7f250c015000).stop 2020-11-20T11:45:39.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:39.044+0000 7f25181bf700 1 -- 172.21.15.23:0/1831191384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f251013a980 msgr2=0x7f2510134070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:39.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:39.044+0000 7f25181bf700 1 --2- 172.21.15.23:0/1831191384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f251013a980 0x7f2510134070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f2508009100 tx=0x7f2508008040).stop 2020-11-20T11:45:39.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:39.044+0000 7f25181bf700 1 -- 172.21.15.23:0/1831191384 shutdown_connections 2020-11-20T11:45:39.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:39.044+0000 7f25181bf700 1 --2- 172.21.15.23:0/1831191384 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f251013a980 0x7f2510134070 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:39.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:39.044+0000 7f25181bf700 1 --2- 172.21.15.23:0/1831191384 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24f80037f0 0x7f25100a77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:39.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:39.044+0000 7f25181bf700 1 --2- 172.21.15.23:0/1831191384 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f24e8056d90 0x7f24e8059250 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:39.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:39.044+0000 7f25181bf700 1 --2- 172.21.15.23:0/1831191384 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f25100a5f30 0x7f251013d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:39.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:39.045+0000 7f25181bf700 1 -- 172.21.15.23:0/1831191384 >> 172.21.15.23:0/1831191384 conn(0x7f25100678e0 msgr2=0x7f2510136850 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:39.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:39.045+0000 7f25181bf700 1 -- 172.21.15.23:0/1831191384 shutdown_connections 2020-11-20T11:45:39.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:39.045+0000 7f25181bf700 1 -- 172.21.15.23:0/1831191384 wait complete. 2020-11-20T11:45:42.079 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:45:42.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.215+0000 7fc366ec7700 1 Processor -- start 2020-11-20T11:45:42.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.215+0000 7fc366ec7700 1 -- start start 2020-11-20T11:45:42.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.216+0000 7fc366ec7700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc3600aaab0 0x7fc36013ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:42.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.216+0000 7fc366ec7700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3601388b0 con 0x7fc3600aae90 2020-11-20T11:45:42.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.216+0000 7fc366ec7700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc36013e600 con 0x7fc36013aeb0 2020-11-20T11:45:42.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.216+0000 7fc366ec7700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc36013e8a0 con 0x7fc3600aaab0 2020-11-20T11:45:42.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.216+0000 7fc35ffff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fc3600aae90 0x7fc3601381a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:60152/0 (socket says 172.21.15.23:60152) 2020-11-20T11:45:42.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.216+0000 7fc35ffff700 1 -- 172.21.15.23:0/1537152748 learned_addr learned my addr 172.21.15.23:0/1537152748 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:45:42.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.216+0000 7fc364c63700 1 --2- 172.21.15.23:0/1537152748 >> v2:172.21.15.155:3300/0 conn(0x7fc3600aaab0 0x7fc36013ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:42.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.216+0000 7fc35f7fe700 1 -- 172.21.15.23:0/1537152748 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3875701545 0 0) 0x7fc3601388b0 con 0x7fc3600aae90 2020-11-20T11:45:42.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.216+0000 7fc35f7fe700 1 -- 172.21.15.23:0/1537152748 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc34c003280 con 0x7fc3600aae90 2020-11-20T11:45:42.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.217+0000 7fc364c63700 1 -- 172.21.15.23:0/1537152748 >> v1:172.21.15.155:6789/0 conn(0x7fc36013aeb0 legacy=0x7fc3600a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:42.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.217+0000 7fc364c63700 1 -- 172.21.15.23:0/1537152748 >> v1:172.21.15.23:6789/0 conn(0x7fc3600aae90 legacy=0x7fc3601381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:42.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.217+0000 7fc364c63700 1 -- 172.21.15.23:0/1537152748 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3600ab6d0 con 0x7fc3600aaab0 2020-11-20T11:45:42.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.217+0000 7fc364c63700 1 --2- 172.21.15.23:0/1537152748 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3600aaab0 0x7fc36013ed40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fc354006fd0 tx=0x7fc354006e30).ready entity=mon.1 client_cookie=5e46de28842c5882 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:42.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.217+0000 7fc35f7fe700 1 -- 172.21.15.23:0/1537152748 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3540097e0 con 0x7fc3600aaab0 2020-11-20T11:45:42.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.217+0000 7fc35f7fe700 1 -- 172.21.15.23:0/1537152748 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc354000e90 con 0x7fc3600aaab0 2020-11-20T11:45:42.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.217+0000 7fc35f7fe700 1 -- 172.21.15.23:0/1537152748 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3540097e0 con 0x7fc3600aaab0 2020-11-20T11:45:42.225 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.219+0000 7fc366ec7700 1 -- 172.21.15.23:0/1537152748 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3600aaab0 msgr2=0x7fc36013ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:42.225 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.219+0000 7fc366ec7700 1 --2- 172.21.15.23:0/1537152748 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3600aaab0 0x7fc36013ed40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fc354006fd0 tx=0x7fc354006e30).stop 2020-11-20T11:45:42.225 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.220+0000 7fc366ec7700 1 -- 172.21.15.23:0/1537152748 shutdown_connections 2020-11-20T11:45:42.226 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.220+0000 7fc366ec7700 1 --2- 172.21.15.23:0/1537152748 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3600aaab0 0x7fc36013ed40 secure :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0x7fc354006fd0 tx=0x7fc354006e30).stop 2020-11-20T11:45:42.226 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.220+0000 7fc366ec7700 1 -- 172.21.15.23:0/1537152748 >> 172.21.15.23:0/1537152748 conn(0x7fc3600678e0 msgr2=0x7fc360067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:42.226 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.221+0000 7fc366ec7700 1 -- 172.21.15.23:0/1537152748 shutdown_connections 2020-11-20T11:45:42.227 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.223+0000 7fc366ec7700 1 -- 172.21.15.23:0/1537152748 wait complete. 2020-11-20T11:45:42.227 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.224+0000 7fc366ec7700 1 Processor -- start 2020-11-20T11:45:42.228 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.224+0000 7fc366ec7700 1 -- start start 2020-11-20T11:45:42.230 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.224+0000 7fc366ec7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3600aae90 0x7fc360135e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:42.230 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.224+0000 7fc366ec7700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3600a63a0 0x7fc36014df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:42.230 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.224+0000 7fc366ec7700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc36014e450 0x7fc360130480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:42.230 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.224+0000 7fc366ec7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36014e8d0 con 0x7fc3600aae90 2020-11-20T11:45:42.231 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.224+0000 7fc366ec7700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36013d750 con 0x7fc3600a63a0 2020-11-20T11:45:42.231 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.224+0000 7fc366ec7700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc36013d9f0 con 0x7fc36014e450 2020-11-20T11:45:42.231 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.224+0000 7fc364c63700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3600aae90 0x7fc360135e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:42.231 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.225+0000 7fc364c63700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3600aae90 0x7fc360135e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45908/0 (socket says 172.21.15.23:45908) 2020-11-20T11:45:42.231 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.225+0000 7fc364c63700 1 -- 172.21.15.23:0/1797543051 learned_addr learned my addr 172.21.15.23:0/1797543051 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:45:42.232 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.225+0000 7fc35ffff700 1 --2- 172.21.15.23:0/1797543051 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3600a63a0 0x7fc36014df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:42.232 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.225+0000 7fc365464700 1 --2- 172.21.15.23:0/1797543051 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc36014e450 0x7fc360130480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:42.232 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.225+0000 7fc364c63700 1 -- 172.21.15.23:0/1797543051 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc36014e450 msgr2=0x7fc360130480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:42.232 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.225+0000 7fc364c63700 1 --2- 172.21.15.23:0/1797543051 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc36014e450 0x7fc360130480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:42.232 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.225+0000 7fc364c63700 1 -- 172.21.15.23:0/1797543051 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3600a63a0 msgr2=0x7fc36014df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:42.233 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.225+0000 7fc364c63700 1 --2- 172.21.15.23:0/1797543051 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3600a63a0 0x7fc36014df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:42.233 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.225+0000 7fc364c63700 1 -- 172.21.15.23:0/1797543051 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36013dc90 con 0x7fc3600aae90 2020-11-20T11:45:42.233 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.225+0000 7fc364c63700 1 --2- 172.21.15.23:0/1797543051 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3600aae90 0x7fc360135e00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fc354006c90 tx=0x7fc354015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:42.233 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.225+0000 7fc35cff9700 1 -- 172.21.15.23:0/1797543051 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc354011df0 con 0x7fc3600aae90 2020-11-20T11:45:42.233 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.225+0000 7fc35cff9700 1 -- 172.21.15.23:0/1797543051 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35401e460 con 0x7fc3600aae90 2020-11-20T11:45:42.234 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.225+0000 7fc366ec7700 1 -- 172.21.15.23:0/1797543051 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc36013d3d0 con 0x7fc3600aae90 2020-11-20T11:45:42.234 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.225+0000 7fc35cff9700 1 -- 172.21.15.23:0/1797543051 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35401e800 con 0x7fc3600aae90 2020-11-20T11:45:42.234 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.225+0000 7fc366ec7700 1 -- 172.21.15.23:0/1797543051 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3600b0740 con 0x7fc3600aae90 2020-11-20T11:45:42.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.226+0000 7fc366ec7700 1 -- 172.21.15.23:0/1797543051 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc36005c7d0 con 0x7fc3600aae90 2020-11-20T11:45:42.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.227+0000 7fc35cff9700 1 -- 172.21.15.23:0/1797543051 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc354025030 con 0x7fc3600aae90 2020-11-20T11:45:42.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.227+0000 7fc35cff9700 1 --2- 172.21.15.23:0/1797543051 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc338056d60 0x7fc338059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:42.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.227+0000 7fc35cff9700 1 -- 172.21.15.23:0/1797543051 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fc354071fe0 con 0x7fc3600aae90 2020-11-20T11:45:42.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.228+0000 7fc35ffff700 1 --2- 172.21.15.23:0/1797543051 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc338056d60 0x7fc338059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:42.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.228+0000 7fc35ffff700 1 --2- 172.21.15.23:0/1797543051 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc338056d60 0x7fc338059220 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fc3600a5920 tx=0x7fc350009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:42.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.231+0000 7fc35cff9700 1 -- 172.21.15.23:0/1797543051 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc354047350 con 0x7fc3600aae90 2020-11-20T11:45:42.534 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.530+0000 7fc366ec7700 1 -- 172.21.15.23:0/1797543051 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc36013d5c0 con 0x7fc3600aae90 2020-11-20T11:45:42.534 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.530+0000 7fc35cff9700 1 -- 172.21.15.23:0/1797543051 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7fc354023030 con 0x7fc3600aae90 2020-11-20T11:45:42.535 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":126,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:45:42.536 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.531+0000 7fc366ec7700 1 -- 172.21.15.23:0/1797543051 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc338056d60 msgr2=0x7fc338059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:42.536 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.531+0000 7fc366ec7700 1 --2- 172.21.15.23:0/1797543051 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc338056d60 0x7fc338059220 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fc3600a5920 tx=0x7fc350009040).stop 2020-11-20T11:45:42.536 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.531+0000 7fc366ec7700 1 -- 172.21.15.23:0/1797543051 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3600aae90 msgr2=0x7fc360135e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:42.536 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.531+0000 7fc366ec7700 1 --2- 172.21.15.23:0/1797543051 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3600aae90 0x7fc360135e00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fc354006c90 tx=0x7fc354015000).stop 2020-11-20T11:45:42.537 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.532+0000 7fc366ec7700 1 -- 172.21.15.23:0/1797543051 shutdown_connections 2020-11-20T11:45:42.537 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.532+0000 7fc366ec7700 1 --2- 172.21.15.23:0/1797543051 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc338056d60 0x7fc338059220 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:42.537 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.532+0000 7fc366ec7700 1 --2- 172.21.15.23:0/1797543051 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc36014e450 0x7fc360130480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:42.537 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.532+0000 7fc366ec7700 1 --2- 172.21.15.23:0/1797543051 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3600aae90 0x7fc360135e00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:42.538 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.532+0000 7fc366ec7700 1 --2- 172.21.15.23:0/1797543051 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc3600a63a0 0x7fc36014df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:42.538 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.532+0000 7fc366ec7700 1 -- 172.21.15.23:0/1797543051 >> 172.21.15.23:0/1797543051 conn(0x7fc3600678e0 msgr2=0x7fc36013a0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:42.538 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.532+0000 7fc366ec7700 1 -- 172.21.15.23:0/1797543051 shutdown_connections 2020-11-20T11:45:42.538 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.532+0000 7fc366ec7700 1 -- 172.21.15.23:0/1797543051 wait complete. 2020-11-20T11:45:42.547 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:45:42.548 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:45:42.801 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.796+0000 7f538ccfe700 1 Processor -- start 2020-11-20T11:45:42.801 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.796+0000 7f538ccfe700 1 -- start start 2020-11-20T11:45:42.817 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.798+0000 7f538ccfe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53880ad7c0 0x7f53880a3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:42.818 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.798+0000 7f538ccfe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5388138420 0x7f53880a9280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:42.818 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.798+0000 7f538ccfe700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53880a3860 con 0x7f53880a8e00 2020-11-20T11:45:42.818 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.798+0000 7f538ccfe700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53881388b0 con 0x7f5388138420 2020-11-20T11:45:42.818 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.798+0000 7f538ccfe700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53880a9770 con 0x7f53880ad7c0 2020-11-20T11:45:42.819 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.802+0000 7f5385d9b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f53880a8e00 0x7f5388137d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:60166/0 (socket says 172.21.15.23:60166) 2020-11-20T11:45:42.819 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.802+0000 7f5385d9b700 1 -- 172.21.15.23:0/1218254475 learned_addr learned my addr 172.21.15.23:0/1218254475 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:45:42.819 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.802+0000 7f538659c700 1 --2- 172.21.15.23:0/1218254475 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53880ad7c0 0x7f53880a3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:42.819 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.803+0000 7f538559a700 1 -- 172.21.15.23:0/1218254475 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1141531487 0 0) 0x7f53880a3860 con 0x7f53880a8e00 2020-11-20T11:45:42.820 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.803+0000 7f538559a700 1 -- 172.21.15.23:0/1218254475 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f536c003280 con 0x7f53880a8e00 2020-11-20T11:45:42.820 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.804+0000 7f538659c700 1 -- 172.21.15.23:0/1218254475 >> v2:172.21.15.155:3300/0 conn(0x7f5388138420 msgr2=0x7f53880a9280 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2020-11-20T11:45:42.820 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.804+0000 7f538659c700 1 --2- 172.21.15.23:0/1218254475 >> v2:172.21.15.155:3300/0 conn(0x7f5388138420 0x7f53880a9280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:42.820 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.804+0000 7f538659c700 1 -- 172.21.15.23:0/1218254475 >> v1:172.21.15.23:6789/0 conn(0x7f53880a8e00 legacy=0x7f5388137d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:42.820 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.804+0000 7f538659c700 1 -- 172.21.15.23:0/1218254475 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53880a3320 con 0x7f53880ad7c0 2020-11-20T11:45:42.822 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.805+0000 7f538659c700 1 --2- 172.21.15.23:0/1218254475 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53880ad7c0 0x7f53880a3a10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f5378006fd0 tx=0x7f5378006e30).ready entity=mon.2 client_cookie=bb60a3b39db73b6d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:42.823 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.807+0000 7f538559a700 1 -- 172.21.15.23:0/1218254475 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53780097e0 con 0x7f53880ad7c0 2020-11-20T11:45:42.823 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.807+0000 7f538559a700 1 -- 172.21.15.23:0/1218254475 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5378000e90 con 0x7f53880ad7c0 2020-11-20T11:45:42.823 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.808+0000 7f538ccfe700 1 -- 172.21.15.23:0/1218254475 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53880ad7c0 msgr2=0x7f53880a3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:42.823 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.808+0000 7f538ccfe700 1 --2- 172.21.15.23:0/1218254475 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53880ad7c0 0x7f53880a3a10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f5378006fd0 tx=0x7f5378006e30).stop 2020-11-20T11:45:42.824 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.812+0000 7f538ccfe700 1 -- 172.21.15.23:0/1218254475 shutdown_connections 2020-11-20T11:45:42.824 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.812+0000 7f538ccfe700 1 --2- 172.21.15.23:0/1218254475 >> v2:172.21.15.155:3300/0 conn(0x7f5388138420 0x7f53880a9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:42.824 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.812+0000 7f538ccfe700 1 --2- 172.21.15.23:0/1218254475 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53880ad7c0 0x7f53880a3a10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:42.825 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.812+0000 7f538ccfe700 1 -- 172.21.15.23:0/1218254475 >> 172.21.15.23:0/1218254475 conn(0x7f5388060c80 msgr2=0x7f53880acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:42.825 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.820+0000 7f538ccfe700 1 -- 172.21.15.23:0/1218254475 shutdown_connections 2020-11-20T11:45:42.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.820+0000 7f538ccfe700 1 -- 172.21.15.23:0/1218254475 wait complete. 2020-11-20T11:45:42.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.821+0000 7f538ccfe700 1 Processor -- start 2020-11-20T11:45:42.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.821+0000 7f538ccfe700 1 -- start start 2020-11-20T11:45:42.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.822+0000 7f538ccfe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53880ad7c0 0x7f53880a6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:42.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.822+0000 7f538ccfe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5388138420 0x7f53880a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:42.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.822+0000 7f538ccfe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f538814e070 0x7f538814e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:42.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.822+0000 7f538ccfe700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53880a6780 con 0x7f5388138420 2020-11-20T11:45:42.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.822+0000 7f538ccfe700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f538814d9a0 con 0x7f538814e070 2020-11-20T11:45:42.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.822+0000 7f538ccfe700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f538813c030 con 0x7f53880ad7c0 2020-11-20T11:45:42.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.823+0000 7f5385d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5388138420 0x7f53880a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:42.832 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.823+0000 7f5385d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5388138420 0x7f53880a6290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45924/0 (socket says 172.21.15.23:45924) 2020-11-20T11:45:42.832 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.823+0000 7f5385d9b700 1 -- 172.21.15.23:0/3884108146 learned_addr learned my addr 172.21.15.23:0/3884108146 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:45:42.832 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.823+0000 7f538659c700 1 --2- 172.21.15.23:0/3884108146 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53880ad7c0 0x7f53880a6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.824+0000 7f538659c700 1 -- 172.21.15.23:0/3884108146 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f538814e070 msgr2=0x7f538814e4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2020-11-20T11:45:42.834 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.824+0000 7f538659c700 1 --2- 172.21.15.23:0/3884108146 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f538814e070 0x7f538814e4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.824+0000 7f538659c700 1 -- 172.21.15.23:0/3884108146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5388138420 msgr2=0x7f53880a6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.824+0000 7f538659c700 1 --2- 172.21.15.23:0/3884108146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5388138420 0x7f53880a6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.824+0000 7f538659c700 1 -- 172.21.15.23:0/3884108146 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5388064ef0 con 0x7f53880ad7c0 2020-11-20T11:45:42.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.824+0000 7f5385d9b700 1 --2- 172.21.15.23:0/3884108146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5388138420 0x7f53880a6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:45:42.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.824+0000 7f538659c700 1 --2- 172.21.15.23:0/3884108146 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53880ad7c0 0x7f53880a6eb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f53880615e0 tx=0x7f5378015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:42.837 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.830+0000 7f53767fc700 1 -- 172.21.15.23:0/3884108146 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5378011b40 con 0x7f53880ad7c0 2020-11-20T11:45:42.837 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.830+0000 7f538ccfe700 1 -- 172.21.15.23:0/3884108146 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53880a69a0 con 0x7f53880ad7c0 2020-11-20T11:45:42.838 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.830+0000 7f538ccfe700 1 -- 172.21.15.23:0/3884108146 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53880a45f0 con 0x7f53880ad7c0 2020-11-20T11:45:42.838 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.830+0000 7f53767fc700 1 -- 172.21.15.23:0/3884108146 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5378011ca0 con 0x7f53880ad7c0 2020-11-20T11:45:42.838 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.830+0000 7f53767fc700 1 -- 172.21.15.23:0/3884108146 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f537801e670 con 0x7f53880ad7c0 2020-11-20T11:45:42.838 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.831+0000 7f538ccfe700 1 --2- 172.21.15.23:0/3884108146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5388067160 0x7f5388067540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:42.839 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.831+0000 7f538ccfe700 1 -- 172.21.15.23:0/3884108146 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5388061650 con 0x7f5388067160 2020-11-20T11:45:42.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.832+0000 7f5385d9b700 1 --2- 172.21.15.23:0/3884108146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5388067160 0x7f5388067540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:42.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.832+0000 7f5385d9b700 1 -- 172.21.15.23:0/3884108146 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f538814dd90 con 0x7f5388067160 2020-11-20T11:45:42.845 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.832+0000 7f5385d9b700 1 --2- 172.21.15.23:0/3884108146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5388067160 0x7f5388067540 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f538813b7c0 tx=0x7f5380009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:42.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.842+0000 7f53767fc700 1 -- 172.21.15.23:0/3884108146 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538000de50 con 0x7f5388067160 2020-11-20T11:45:42.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.842+0000 7f53767fc700 1 -- 172.21.15.23:0/3884108146 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5378025030 con 0x7f53880ad7c0 2020-11-20T11:45:42.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.842+0000 7f53767fc700 1 --2- 172.21.15.23:0/3884108146 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f535c056e50 0x7f535c059310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:42.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.842+0000 7f53767fc700 1 -- 172.21.15.23:0/3884108146 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f5380005990 con 0x7f5388067160 2020-11-20T11:45:42.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.842+0000 7f53767fc700 1 -- 172.21.15.23:0/3884108146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5388067160 msgr2=0x7f5388067540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:42.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.842+0000 7f53767fc700 1 --2- 172.21.15.23:0/3884108146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5388067160 0x7f5388067540 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f538813b7c0 tx=0x7f5380009040).stop 2020-11-20T11:45:42.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.843+0000 7f53767fc700 1 -- 172.21.15.23:0/3884108146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5388067160 msgr2=0x7f5388067540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:45:42.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.843+0000 7f53767fc700 1 --2- 172.21.15.23:0/3884108146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5388067160 0x7f5388067540 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:42.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.843+0000 7f53767fc700 1 -- 172.21.15.23:0/3884108146 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f5378071550 con 0x7f53880ad7c0 2020-11-20T11:45:42.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.844+0000 7f5386d9d700 1 --2- 172.21.15.23:0/3884108146 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f535c056e50 0x7f535c059310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:42.850 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.844+0000 7f5386d9d700 1 --2- 172.21.15.23:0/3884108146 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f535c056e50 0x7f535c059310 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f537c006fd0 tx=0x7f537c006970).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:42.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.853+0000 7f538ccfe700 1 --2- 172.21.15.23:0/3884108146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5388067a30 0x7f5388132430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:42.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.853+0000 7f538ccfe700 1 -- 172.21.15.23:0/3884108146 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f538805c520 con 0x7f5388067a30 2020-11-20T11:45:42.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.853+0000 7f5385d9b700 1 --2- 172.21.15.23:0/3884108146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5388067a30 0x7f5388132430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:42.859 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.854+0000 7f5385d9b700 1 -- 172.21.15.23:0/3884108146 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5388044710 con 0x7f5388067a30 2020-11-20T11:45:42.859 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.854+0000 7f5385d9b700 1 --2- 172.21.15.23:0/3884108146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5388067a30 0x7f5388132430 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f538813b7c0 tx=0x7f5380018000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:42.859 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.854+0000 7f53767fc700 1 -- 172.21.15.23:0/3884108146 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53800146e0 con 0x7f5388067a30 2020-11-20T11:45:42.860 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.855+0000 7f53767fc700 1 -- 172.21.15.23:0/3884108146 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4616 (secure 0 0 0) 0x7f5380005b00 con 0x7f5388067a30 2020-11-20T11:45:42.860 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.855+0000 7f53767fc700 1 -- 172.21.15.23:0/3884108146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5388067a30 msgr2=0x7f5388132430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:42.860 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.855+0000 7f53767fc700 1 --2- 172.21.15.23:0/3884108146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5388067a30 0x7f5388132430 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f538813b7c0 tx=0x7f5380018000).stop 2020-11-20T11:45:42.860 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.855+0000 7f53767fc700 1 -- 172.21.15.23:0/3884108146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5388067a30 msgr2=0x7f5388132430 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:45:42.861 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:45:42.861 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:45:42.861 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:45:42.862 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:45:42.862 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 126, 2020-11-20T11:45:42.862 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:45:42.862 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:45:42.862 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:45:42.863 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:45:42.863 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:42.863 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 3, 2020-11-20T11:45:42.863 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:45:42.863 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:45:42.864 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:45:42.864 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:45:42.864 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:45:42.864 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:45:42.864 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:45:42.865 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:45:42.865 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:45:42.865 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:45:42.865 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:45:42.866 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:42.866 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:45:42.866 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:45:42.866 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:45:42.866 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:45:42.867 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:45:42.867 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:45:42.867 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:45:42.867 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:45:42.867 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:45:42.868 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:45:42.868 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:42.868 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:42.868 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:45:42.868 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:45:42.869 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:45:42.869 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:45:42.869 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:45:42.869 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:45:42.869 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:45:42.870 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:45:42.871 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:45:42.871 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:45:42.871 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:45:42.872 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:45:42.872 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:45:42.872 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:45:42.872 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:45:42.872 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:45:42.873 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:45:42.873 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:45:42.873 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:45:42.873 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:45:42.874 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:45:42.874 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:45:42.874 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:45:42.874 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:42.874 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:45:42.875 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:42.875 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:45:42.875 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:42.875 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:45:42.875 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:45:42.876 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:45:42.876 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:45:42.876 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:42.876 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:45:42.876 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:45:42.877 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:42.877 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:42.877 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:42.877 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:45:42.877 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:45:42.878 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:42.878 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:42.878 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:42.878 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:42.878 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:45:42.879 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:45:42.879 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:45:42.879 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:45:42.879 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:45:42.879 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:42.880 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:42.880 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:45:42.880 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:45:42.881 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:45:42.881 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:45:42.882 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:42.882 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:45:42.882 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:45:42.883 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:42.883 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:42.883 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:42.884 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:45:42.884 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:45:42.884 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:42.885 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:42.885 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:42.885 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:42.885 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:45:42.885 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:45:42.886 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:45:42.886 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:45:42.886 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:45:42.886 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:42.887 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:42.887 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:45:42.887 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:45:42.887 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:45:42.887 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:45:42.888 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:42.888 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:45:42.888 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:45:42.888 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:42.888 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:42.889 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:42.889 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:45:42.889 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:45:42.889 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:42.890 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:42.890 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:42.890 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:42.891 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:45:42.891 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:45:42.891 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:45:42.891 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:45:42.893 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:45:42.893 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:42.894 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:42.894 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:42.894 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:45:42.894 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:45:42.895 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:42.895 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:45:42.895 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:45:42.895 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:45:42.896 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:42.896 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:42.896 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:45:42.896 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:42.897 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:45:42.897 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:45:42.897 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:45:42.898 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:42.898 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:42.898 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:45:42.898 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:42.899 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:45:42.899 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:45:42.899 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:45:42.900 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:42.900 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:42.900 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:45:42.900 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:42.901 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:45:42.901 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:45:42.901 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:45:42.901 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:42.902 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:42.902 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:42.902 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:45:42.903 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:45:42.903 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.856+0000 7f53767fc700 1 --2- 172.21.15.23:0/3884108146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5388067a30 0x7f5388132430 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:42.905 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.864+0000 7f538ccfe700 1 -- 172.21.15.23:0/3884108146 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f535c056e50 msgr2=0x7f535c059310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:42.905 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.864+0000 7f538ccfe700 1 --2- 172.21.15.23:0/3884108146 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f535c056e50 0x7f535c059310 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f537c006fd0 tx=0x7f537c006970).stop 2020-11-20T11:45:42.906 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.864+0000 7f538ccfe700 1 -- 172.21.15.23:0/3884108146 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53880ad7c0 msgr2=0x7f53880a6eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:42.906 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.864+0000 7f538ccfe700 1 --2- 172.21.15.23:0/3884108146 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53880ad7c0 0x7f53880a6eb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f53880615e0 tx=0x7f5378015000).stop 2020-11-20T11:45:42.906 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.865+0000 7f5386d9d700 1 -- 172.21.15.23:0/3884108146 reap_dead start 2020-11-20T11:45:42.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.865+0000 7f5386d9d700 1 -- 172.21.15.23:0/3884108146 reap_dead start 2020-11-20T11:45:42.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.874+0000 7f538ccfe700 1 -- 172.21.15.23:0/3884108146 shutdown_connections 2020-11-20T11:45:42.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.874+0000 7f538ccfe700 1 -- 172.21.15.23:0/3884108146 >> 172.21.15.23:0/3884108146 conn(0x7f5388060c80 msgr2=0x7f53880a3720 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:42.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.878+0000 7f538ccfe700 1 -- 172.21.15.23:0/3884108146 shutdown_connections 2020-11-20T11:45:42.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:42.878+0000 7f538ccfe700 1 -- 172.21.15.23:0/3884108146 wait complete. 2020-11-20T11:45:42.918 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:45:43.197 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.189+0000 7fc9c8ae1700 1 Processor -- start 2020-11-20T11:45:43.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.189+0000 7fc9c8ae1700 1 -- start start 2020-11-20T11:45:43.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.192+0000 7fc9c8ae1700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc9c4065140 0x7fc9c40a8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:43.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.192+0000 7fc9c8ae1700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc9c40a8e50 0x7fc9c4151b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:43.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.192+0000 7fc9c8ae1700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c40a92e0 con 0x7fc9c4064d60 2020-11-20T11:45:43.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.192+0000 7fc9c8ae1700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9c40a7f80 con 0x7fc9c4065140 2020-11-20T11:45:43.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.192+0000 7fc9c8ae1700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9c40a8220 con 0x7fc9c40a8e50 2020-11-20T11:45:43.203 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.192+0000 7fc9c1d9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc9c4065140 0x7fc9c40a8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:43.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.192+0000 7fc9c1d9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc9c4065140 0x7fc9c40a8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45934/0 (socket says 172.21.15.23:45934) 2020-11-20T11:45:43.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.192+0000 7fc9c1d9b700 1 -- 172.21.15.23:0/3684908037 learned_addr learned my addr 172.21.15.23:0/3684908037 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:45:43.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.192+0000 7fc9c2d9d700 1 --2- 172.21.15.23:0/3684908037 >> v2:172.21.15.155:3300/0 conn(0x7fc9c40a8e50 0x7fc9c4151b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:43.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.193+0000 7fc9c1d9b700 1 -- 172.21.15.23:0/3684908037 >> v2:172.21.15.155:3300/0 conn(0x7fc9c40a8e50 msgr2=0x7fc9c4151b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:45:43.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.193+0000 7fc9c1d9b700 1 --2- 172.21.15.23:0/3684908037 >> v2:172.21.15.155:3300/0 conn(0x7fc9c40a8e50 0x7fc9c4151b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:43.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.193+0000 7fc9c1d9b700 1 -- 172.21.15.23:0/3684908037 >> v1:172.21.15.23:6789/0 conn(0x7fc9c4064d60 legacy=0x7fc9c40a7750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:45:43.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.193+0000 7fc9c1d9b700 1 -- 172.21.15.23:0/3684908037 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c40a84c0 con 0x7fc9c4065140 2020-11-20T11:45:43.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.193+0000 7fc9c1d9b700 1 --2- 172.21.15.23:0/3684908037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9c4065140 0x7fc9c40a8960 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fc9c40aac80 tx=0x7fc9bc00d040).ready entity=mon.0 client_cookie=9a738e894759fa38 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:43.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.193+0000 7fc9c159a700 1 -- 172.21.15.23:0/3684908037 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc00cdd0 con 0x7fc9c4065140 2020-11-20T11:45:43.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.193+0000 7fc9c159a700 1 -- 172.21.15.23:0/3684908037 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9bc0158c0 con 0x7fc9c4065140 2020-11-20T11:45:43.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.194+0000 7fc9c8ae1700 1 -- 172.21.15.23:0/3684908037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9c4065140 msgr2=0x7fc9c40a8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:43.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.194+0000 7fc9c8ae1700 1 --2- 172.21.15.23:0/3684908037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9c4065140 0x7fc9c40a8960 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fc9c40aac80 tx=0x7fc9bc00d040).stop 2020-11-20T11:45:43.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.195+0000 7fc9c8ae1700 1 -- 172.21.15.23:0/3684908037 shutdown_connections 2020-11-20T11:45:43.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.195+0000 7fc9c8ae1700 1 --2- 172.21.15.23:0/3684908037 >> v2:172.21.15.155:3300/0 conn(0x7fc9c40a8e50 0x7fc9c4151b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:43.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.195+0000 7fc9c8ae1700 1 --2- 172.21.15.23:0/3684908037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9c4065140 0x7fc9c40a8960 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:43.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.195+0000 7fc9c8ae1700 1 -- 172.21.15.23:0/3684908037 >> 172.21.15.23:0/3684908037 conn(0x7fc9c40601b0 msgr2=0x7fc9c40605c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:43.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.195+0000 7fc9c8ae1700 1 -- 172.21.15.23:0/3684908037 shutdown_connections 2020-11-20T11:45:43.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.195+0000 7fc9c8ae1700 1 -- 172.21.15.23:0/3684908037 wait complete. 2020-11-20T11:45:43.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.198+0000 7fc9c8ae1700 1 Processor -- start 2020-11-20T11:45:43.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.198+0000 7fc9c8ae1700 1 -- start start 2020-11-20T11:45:43.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.199+0000 7fc9c8ae1700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc9c4064d60 0x7fc9c40a6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:43.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.199+0000 7fc9c8ae1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9c4065140 0x7fc9c409f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:43.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.199+0000 7fc9c8ae1700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9c40a8e50 0x7fc9c409fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:43.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.199+0000 7fc9c8ae1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c40a02e0 con 0x7fc9c4065140 2020-11-20T11:45:43.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.199+0000 7fc9c8ae1700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c40a14c0 con 0x7fc9c40a8e50 2020-11-20T11:45:43.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.199+0000 7fc9c8ae1700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c40a2be0 con 0x7fc9c4064d60 2020-11-20T11:45:43.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.199+0000 7fc9c1d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9c4065140 0x7fc9c409f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:43.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.199+0000 7fc9c1d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9c4065140 0x7fc9c409f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45942/0 (socket says 172.21.15.23:45942) 2020-11-20T11:45:43.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.199+0000 7fc9c1d9b700 1 -- 172.21.15.23:0/4292564016 learned_addr learned my addr 172.21.15.23:0/4292564016 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:45:43.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.200+0000 7fc9c1d9b700 1 -- 172.21.15.23:0/4292564016 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc9c4064d60 msgr2=0x7fc9c40a6bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2020-11-20T11:45:43.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.200+0000 7fc9c1d9b700 1 --2- 172.21.15.23:0/4292564016 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc9c4064d60 0x7fc9c40a6bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:43.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.200+0000 7fc9c1d9b700 1 -- 172.21.15.23:0/4292564016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9c40a8e50 msgr2=0x7fc9c409fdf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2020-11-20T11:45:43.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.200+0000 7fc9c1d9b700 1 --2- 172.21.15.23:0/4292564016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9c40a8e50 0x7fc9c409fdf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:43.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.200+0000 7fc9c1d9b700 1 -- 172.21.15.23:0/4292564016 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c40a2e80 con 0x7fc9c4065140 2020-11-20T11:45:43.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.200+0000 7fc9c1d9b700 1 --2- 172.21.15.23:0/4292564016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9c4065140 0x7fc9c409f900 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fc9bc00ca90 tx=0x7fc9bc01d000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:43.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.202+0000 7fc9b27fc700 1 -- 172.21.15.23:0/4292564016 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc021030 con 0x7fc9c4065140 2020-11-20T11:45:43.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.202+0000 7fc9c8ae1700 1 -- 172.21.15.23:0/4292564016 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9c40a2860 con 0x7fc9c4065140 2020-11-20T11:45:43.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.202+0000 7fc9c8ae1700 1 -- 172.21.15.23:0/4292564016 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c40a0a80 con 0x7fc9c4065140 2020-11-20T11:45:43.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.202+0000 7fc9b27fc700 1 -- 172.21.15.23:0/4292564016 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9bc022040 con 0x7fc9c4065140 2020-11-20T11:45:43.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.202+0000 7fc9b27fc700 1 -- 172.21.15.23:0/4292564016 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc026400 con 0x7fc9c4065140 2020-11-20T11:45:43.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.205+0000 7fc9b27fc700 1 -- 172.21.15.23:0/4292564016 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc9bc0267f0 con 0x7fc9c4065140 2020-11-20T11:45:43.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.205+0000 7fc9b27fc700 1 --2- 172.21.15.23:0/4292564016 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc9a4056f80 0x7fc9a4059440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:43.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.205+0000 7fc9b27fc700 1 -- 172.21.15.23:0/4292564016 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fc9bc07a0b0 con 0x7fc9c4065140 2020-11-20T11:45:43.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.206+0000 7fc9c259c700 1 --2- 172.21.15.23:0/4292564016 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc9a4056f80 0x7fc9a4059440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:43.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.207+0000 7fc9c259c700 1 --2- 172.21.15.23:0/4292564016 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc9a4056f80 0x7fc9a4059440 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fc9b4006fd0 tx=0x7fc9b4006970).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:43.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.210+0000 7fc9c8ae1700 1 --2- 172.21.15.23:0/4292564016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9c40611c0 0x7fc9c409d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:43.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.210+0000 7fc9c8ae1700 1 -- 172.21.15.23:0/4292564016 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c40a2a80 con 0x7fc9c40611c0 2020-11-20T11:45:43.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.210+0000 7fc9c2d9d700 1 --2- 172.21.15.23:0/4292564016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9c40611c0 0x7fc9c409d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:43.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.211+0000 7fc9c2d9d700 1 -- 172.21.15.23:0/4292564016 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc9c4061650 con 0x7fc9c40611c0 2020-11-20T11:45:43.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.211+0000 7fc9c2d9d700 1 --2- 172.21.15.23:0/4292564016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9c40611c0 0x7fc9c409d4a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fc9b800b900 tx=0x7fc9b800f040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:43.225 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.212+0000 7fc9b27fc700 1 -- 172.21.15.23:0/4292564016 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b800bea0 con 0x7fc9c40611c0 2020-11-20T11:45:43.225 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.212+0000 7fc9b27fc700 1 -- 172.21.15.23:0/4292564016 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fc9b800f630 con 0x7fc9c40611c0 2020-11-20T11:45:43.225 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.212+0000 7fc9b27fc700 1 -- 172.21.15.23:0/4292564016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9c40611c0 msgr2=0x7fc9c409d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:43.226 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.212+0000 7fc9b27fc700 1 --2- 172.21.15.23:0/4292564016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9c40611c0 0x7fc9c409d4a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fc9b800b900 tx=0x7fc9b800f040).stop 2020-11-20T11:45:43.226 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.212+0000 7fc9b27fc700 1 -- 172.21.15.23:0/4292564016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9c40611c0 msgr2=0x7fc9c409d4a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:45:43.226 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.212+0000 7fc9b27fc700 1 --2- 172.21.15.23:0/4292564016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9c40611c0 0x7fc9c409d4a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:43.233 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.227+0000 7fc9c8ae1700 1 --2- 172.21.15.23:0/4292564016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9c409d990 0x7fc9c409dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:43.234 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.228+0000 7fc9c8ae1700 1 -- 172.21.15.23:0/4292564016 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c40a0cf0 con 0x7fc9c409d990 2020-11-20T11:45:43.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.232+0000 7fc9c2d9d700 1 --2- 172.21.15.23:0/4292564016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9c409d990 0x7fc9c409dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:43.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.236+0000 7fc9c2d9d700 1 -- 172.21.15.23:0/4292564016 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc9c4057dc0 con 0x7fc9c409d990 2020-11-20T11:45:43.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.236+0000 7fc9c2d9d700 1 --2- 172.21.15.23:0/4292564016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9c409d990 0x7fc9c409dda0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fc9b8007fd0 tx=0x7fc9b8017000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:43.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.237+0000 7fc9b27fc700 1 -- 172.21.15.23:0/4292564016 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b8017e40 con 0x7fc9c409d990 2020-11-20T11:45:43.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.237+0000 7fc9b27fc700 1 -- 172.21.15.23:0/4292564016 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4784 (secure 0 0 0) 0x7fc9b8000e90 con 0x7fc9c409d990 2020-11-20T11:45:43.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.237+0000 7fc9b27fc700 1 -- 172.21.15.23:0/4292564016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9c409d990 msgr2=0x7fc9c409dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:43.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.237+0000 7fc9b27fc700 1 --2- 172.21.15.23:0/4292564016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9c409d990 0x7fc9c409dda0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fc9b8007fd0 tx=0x7fc9b8017000).stop 2020-11-20T11:45:43.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.237+0000 7fc9b27fc700 1 -- 172.21.15.23:0/4292564016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9c409d990 msgr2=0x7fc9c409dda0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:45:43.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.237+0000 7fc9b27fc700 1 --2- 172.21.15.23:0/4292564016 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9c409d990 0x7fc9c409dda0 secure :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0x7fc9b8007fd0 tx=0x7fc9b8017000).stop 2020-11-20T11:45:43.245 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:45:43.246 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:45:43.246 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:45:43.246 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:45:43.247 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 126, 2020-11-20T11:45:43.247 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:45:43.247 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:45:43.248 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:45:43.248 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:45:43.248 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:43.249 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 3, 2020-11-20T11:45:43.249 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:45:43.249 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:45:43.250 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:45:43.250 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:45:43.250 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:45:43.251 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:45:43.251 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:45:43.251 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:45:43.252 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:45:43.252 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:45:43.252 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:45:43.253 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:43.253 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:45:43.254 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:45:43.254 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:45:43.254 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:45:43.255 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:45:43.255 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:45:43.255 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:45:43.256 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:45:43.256 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:45:43.256 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:45:43.257 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:43.257 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:43.258 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:45:43.258 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:45:43.258 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:45:43.258 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:45:43.259 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:45:43.259 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:45:43.259 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:45:43.260 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:45:43.260 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:45:43.260 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:45:43.260 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:45:43.261 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:45:43.261 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:45:43.261 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:45:43.262 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:45:43.262 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:45:43.262 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:45:43.263 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:45:43.263 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:45:43.263 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:45:43.264 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:45:43.264 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:45:43.265 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:45:43.265 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:43.265 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:45:43.266 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:43.266 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:45:43.266 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:43.267 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:45:43.267 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:45:43.268 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:45:43.268 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:45:43.268 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:43.269 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:45:43.269 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:45:43.270 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:43.270 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:43.270 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:43.270 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:45:43.270 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:45:43.271 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:43.271 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:43.271 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:43.271 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:43.271 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:45:43.272 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:45:43.272 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:45:43.272 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:45:43.272 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:45:43.272 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:43.272 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:43.273 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:45:43.273 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:45:43.273 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:45:43.273 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:45:43.273 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:43.274 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:45:43.274 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:45:43.274 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:43.274 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:43.274 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:43.275 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:45:43.275 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:45:43.275 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:43.275 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:43.275 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:43.276 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:43.276 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:45:43.276 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:45:43.276 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:45:43.276 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:45:43.277 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:45:43.277 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:43.277 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:43.277 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:45:43.277 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:45:43.278 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:45:43.278 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:45:43.278 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:43.278 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:45:43.278 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:45:43.278 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:43.279 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:43.279 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:43.279 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:45:43.279 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:45:43.279 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:43.280 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:43.280 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:43.280 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:43.280 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:45:43.280 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:45:43.281 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:45:43.281 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:45:43.281 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:45:43.281 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:43.281 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:43.282 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:43.282 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:45:43.282 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:45:43.282 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:43.282 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:45:43.283 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:45:43.283 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:45:43.283 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:43.283 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:43.283 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:45:43.283 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:43.284 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:45:43.284 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:45:43.284 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:45:43.284 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:43.284 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:43.285 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:45:43.285 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:43.285 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:45:43.285 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:45:43.285 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:45:43.286 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:43.286 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:43.286 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:45:43.286 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:43.286 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:45:43.287 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:45:43.287 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:45:43.287 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:43.287 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:43.287 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:45:43.288 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:43.288 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:45:43.288 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:45:43.288 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:45:43.288 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:43.288 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:43.289 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:43.289 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:45:43.289 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:45:43.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.248+0000 7fc9c8ae1700 1 -- 172.21.15.23:0/4292564016 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc9a4056f80 msgr2=0x7fc9a4059440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:43.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.248+0000 7fc9c8ae1700 1 --2- 172.21.15.23:0/4292564016 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc9a4056f80 0x7fc9a4059440 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fc9b4006fd0 tx=0x7fc9b4006970).stop 2020-11-20T11:45:43.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.248+0000 7fc9c8ae1700 1 -- 172.21.15.23:0/4292564016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9c4065140 msgr2=0x7fc9c409f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:43.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.248+0000 7fc9c8ae1700 1 --2- 172.21.15.23:0/4292564016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9c4065140 0x7fc9c409f900 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fc9bc00ca90 tx=0x7fc9bc01d000).stop 2020-11-20T11:45:43.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.248+0000 7fc9c2d9d700 1 -- 172.21.15.23:0/4292564016 reap_dead start 2020-11-20T11:45:43.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.249+0000 7fc9c8ae1700 1 -- 172.21.15.23:0/4292564016 shutdown_connections 2020-11-20T11:45:43.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.249+0000 7fc9c8ae1700 1 -- 172.21.15.23:0/4292564016 >> 172.21.15.23:0/4292564016 conn(0x7fc9c40601b0 msgr2=0x7fc9c40a86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:43.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.250+0000 7fc9c8ae1700 1 -- 172.21.15.23:0/4292564016 shutdown_connections 2020-11-20T11:45:43.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.250+0000 7fc9c8ae1700 1 -- 172.21.15.23:0/4292564016 wait complete. 2020-11-20T11:45:43.293 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:45:43.625 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.619+0000 7f123bfff700 1 Processor -- start 2020-11-20T11:45:43.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.620+0000 7f123bfff700 1 -- start start 2020-11-20T11:45:43.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.620+0000 7f123bfff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f123c0acca0 0x7f123c0a7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:43.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.620+0000 7f123bfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f123c0ad080 0x7f123c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:43.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.620+0000 7f123bfff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f123c14ac70 0x7f123c14b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:43.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.620+0000 7f123bfff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f123c0a5a30 con 0x7f123c0ad080 2020-11-20T11:45:43.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.620+0000 7f123bfff700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f123c0a7880 con 0x7f123c0acca0 2020-11-20T11:45:43.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.620+0000 7f123bfff700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f123c0a7b20 con 0x7f123c14ac70 2020-11-20T11:45:43.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.621+0000 7f123a7fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f123c0ad080 0x7f123c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:43.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.621+0000 7f123a7fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f123c0ad080 0x7f123c0a5240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45950/0 (socket says 172.21.15.23:45950) 2020-11-20T11:45:43.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.621+0000 7f123a7fc700 1 -- 172.21.15.23:0/622633739 learned_addr learned my addr 172.21.15.23:0/622633739 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:45:43.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.621+0000 7f123affd700 1 --2- 172.21.15.23:0/622633739 >> v2:172.21.15.155:3300/0 conn(0x7f123c0acca0 0x7f123c0a7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:43.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.621+0000 7f123b7fe700 1 --2- 172.21.15.23:0/622633739 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f123c14ac70 0x7f123c14b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:43.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.621+0000 7f123affd700 1 -- 172.21.15.23:0/622633739 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f123c14ac70 msgr2=0x7f123c14b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:45:43.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.621+0000 7f123affd700 1 --2- 172.21.15.23:0/622633739 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f123c14ac70 0x7f123c14b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:43.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.621+0000 7f123affd700 1 -- 172.21.15.23:0/622633739 >> v2:172.21.15.23:3300/0 conn(0x7f123c0ad080 msgr2=0x7f123c0a5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:45:43.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.621+0000 7f123affd700 1 --2- 172.21.15.23:0/622633739 >> v2:172.21.15.23:3300/0 conn(0x7f123c0ad080 0x7f123c0a5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:43.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.621+0000 7f123affd700 1 -- 172.21.15.23:0/622633739 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f123c0a84b0 con 0x7f123c0acca0 2020-11-20T11:45:43.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.623+0000 7f123affd700 1 --2- 172.21.15.23:0/622633739 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f123c0acca0 0x7f123c0a7fc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f123c062870 tx=0x7f122c006e30).ready entity=mon.1 client_cookie=c6db5939ba139396 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:43.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.623+0000 7f1239ffb700 1 -- 172.21.15.23:0/622633739 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122c0097b0 con 0x7f123c0acca0 2020-11-20T11:45:43.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.623+0000 7f1239ffb700 1 -- 172.21.15.23:0/622633739 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f122c000e90 con 0x7f123c0acca0 2020-11-20T11:45:43.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.623+0000 7f1239ffb700 1 -- 172.21.15.23:0/622633739 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122c009b20 con 0x7f123c0acca0 2020-11-20T11:45:43.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.624+0000 7f123bfff700 1 -- 172.21.15.23:0/622633739 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f123c0acca0 msgr2=0x7f123c0a7fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:43.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.624+0000 7f123bfff700 1 --2- 172.21.15.23:0/622633739 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f123c0acca0 0x7f123c0a7fc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f123c062870 tx=0x7f122c006e30).stop 2020-11-20T11:45:43.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.624+0000 7f123bfff700 1 -- 172.21.15.23:0/622633739 shutdown_connections 2020-11-20T11:45:43.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.624+0000 7f123bfff700 1 --2- 172.21.15.23:0/622633739 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f123c14ac70 0x7f123c14b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:43.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.624+0000 7f123bfff700 1 --2- 172.21.15.23:0/622633739 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f123c0acca0 0x7f123c0a7fc0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:43.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.624+0000 7f123bfff700 1 --2- 172.21.15.23:0/622633739 >> v2:172.21.15.23:3300/0 conn(0x7f123c0ad080 0x7f123c0a5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:43.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.624+0000 7f123bfff700 1 -- 172.21.15.23:0/622633739 >> 172.21.15.23:0/622633739 conn(0x7f123c061850 msgr2=0x7f123c061c60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:43.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.630+0000 7f123bfff700 1 -- 172.21.15.23:0/622633739 shutdown_connections 2020-11-20T11:45:43.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.634+0000 7f123bfff700 1 -- 172.21.15.23:0/622633739 wait complete. 2020-11-20T11:45:43.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.635+0000 7f123bfff700 1 Processor -- start 2020-11-20T11:45:43.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.635+0000 7f123bfff700 1 -- start start 2020-11-20T11:45:43.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.636+0000 7f123bfff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f123c0acca0 0x7f123c0a2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:43.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.636+0000 7f123bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f123c14ac70 0x7f123c09fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:43.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.636+0000 7f123bfff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f123c14c660 0x7f123c09dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:43.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.636+0000 7f123bfff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123c0a0480 con 0x7f123c14ac70 2020-11-20T11:45:43.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.636+0000 7f123bfff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123c09d540 con 0x7f123c0acca0 2020-11-20T11:45:43.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.636+0000 7f123bfff700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f123c09d7e0 con 0x7f123c14c660 2020-11-20T11:45:43.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.636+0000 7f123b7fe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f123c14c660 0x7f123c09dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:43.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.636+0000 7f123b7fe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f123c14c660 0x7f123c09dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:56198/0 (socket says 172.21.15.23:56198) 2020-11-20T11:45:43.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.636+0000 7f123b7fe700 1 -- 172.21.15.23:0/3029111595 learned_addr learned my addr 172.21.15.23:0/3029111595 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:45:43.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.637+0000 7f123b7fe700 1 -- 172.21.15.23:0/3029111595 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f123c0acca0 msgr2=0x7f123c0a2cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2020-11-20T11:45:43.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.637+0000 7f123b7fe700 1 --2- 172.21.15.23:0/3029111595 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f123c0acca0 0x7f123c0a2cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:43.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.637+0000 7f123b7fe700 1 -- 172.21.15.23:0/3029111595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f123c14ac70 msgr2=0x7f123c09fc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:45:43.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.637+0000 7f123b7fe700 1 --2- 172.21.15.23:0/3029111595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f123c14ac70 0x7f123c09fc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:43.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.637+0000 7f123b7fe700 1 -- 172.21.15.23:0/3029111595 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f123c09e7d0 con 0x7f123c14c660 2020-11-20T11:45:43.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.637+0000 7f123b7fe700 1 --2- 172.21.15.23:0/3029111595 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f123c14c660 0x7f123c09dc80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f123400ebf0 tx=0x7f1234008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:43.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.637+0000 7f1222ffd700 1 -- 172.21.15.23:0/3029111595 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123400c7c0 con 0x7f123c14c660 2020-11-20T11:45:43.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.637+0000 7f1222ffd700 1 -- 172.21.15.23:0/3029111595 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f123400c920 con 0x7f123c14c660 2020-11-20T11:45:43.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.637+0000 7f1222ffd700 1 -- 172.21.15.23:0/3029111595 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123400cca0 con 0x7f123c14c660 2020-11-20T11:45:43.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.637+0000 7f123bfff700 1 -- 172.21.15.23:0/3029111595 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f123c0a0120 con 0x7f123c14c660 2020-11-20T11:45:43.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.637+0000 7f123bfff700 1 -- 172.21.15.23:0/3029111595 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f123c09e3b0 con 0x7f123c14c660 2020-11-20T11:45:43.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.639+0000 7f1222ffd700 1 -- 172.21.15.23:0/3029111595 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1234013640 con 0x7f123c14c660 2020-11-20T11:45:43.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.640+0000 7f1222ffd700 1 --2- 172.21.15.23:0/3029111595 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1214056db0 0x7f1214059270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:43.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.640+0000 7f1222ffd700 1 -- 172.21.15.23:0/3029111595 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f123406fd40 con 0x7f123c14c660 2020-11-20T11:45:43.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.640+0000 7f123affd700 1 --2- 172.21.15.23:0/3029111595 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1214056db0 0x7f1214059270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:43.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.640+0000 7f123affd700 1 --2- 172.21.15.23:0/3029111595 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1214056db0 0x7f1214059270 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f122c00b190 tx=0x7f122c015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:43.656 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.652+0000 7f123bfff700 1 --2- 172.21.15.23:0/3029111595 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1218002a50 0x7f1218004f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:43.656 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.652+0000 7f123bfff700 1 -- 172.21.15.23:0/3029111595 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1218007bb0 con 0x7f1218002a50 2020-11-20T11:45:43.657 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.653+0000 7f123a7fc700 1 --2- 172.21.15.23:0/3029111595 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1218002a50 0x7f1218004f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:43.657 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.653+0000 7f123a7fc700 1 -- 172.21.15.23:0/3029111595 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1218007d70 con 0x7f1218002a50 2020-11-20T11:45:43.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.653+0000 7f123a7fc700 1 --2- 172.21.15.23:0/3029111595 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1218002a50 0x7f1218004f10 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f123c14a2d0 tx=0x7f1230006d60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:43.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.653+0000 7f1222ffd700 1 -- 172.21.15.23:0/3029111595 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1230010040 con 0x7f1218002a50 2020-11-20T11:45:43.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.654+0000 7f1222ffd700 1 -- 172.21.15.23:0/3029111595 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f1230000e90 con 0x7f1218002a50 2020-11-20T11:45:43.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.654+0000 7f1222ffd700 1 -- 172.21.15.23:0/3029111595 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1218002a50 msgr2=0x7f1218004f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:43.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.654+0000 7f1222ffd700 1 --2- 172.21.15.23:0/3029111595 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1218002a50 0x7f1218004f10 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f123c14a2d0 tx=0x7f1230006d60).stop 2020-11-20T11:45:43.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.654+0000 7f1222ffd700 1 -- 172.21.15.23:0/3029111595 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1218002a50 msgr2=0x7f1218004f10 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:45:43.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.654+0000 7f1222ffd700 1 --2- 172.21.15.23:0/3029111595 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1218002a50 0x7f1218004f10 secure :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0x7f123c14a2d0 tx=0x7f1230006d60).stop 2020-11-20T11:45:43.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.664+0000 7f123bfff700 1 --2- 172.21.15.23:0/3029111595 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1218006bc0 0x7f1218007050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:43.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.665+0000 7f123bfff700 1 -- 172.21.15.23:0/3029111595 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1218007660 con 0x7f1218006bc0 2020-11-20T11:45:43.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.669+0000 7f123a7fc700 1 --2- 172.21.15.23:0/3029111595 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1218006bc0 0x7f1218007050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:43.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.672+0000 7f123a7fc700 1 -- 172.21.15.23:0/3029111595 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1218007820 con 0x7f1218006bc0 2020-11-20T11:45:43.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.672+0000 7f123a7fc700 1 --2- 172.21.15.23:0/3029111595 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1218006bc0 0x7f1218007050 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f1230016000 tx=0x7f1230017000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:43.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.672+0000 7f1222ffd700 1 -- 172.21.15.23:0/3029111595 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1230014720 con 0x7f1218006bc0 2020-11-20T11:45:43.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.672+0000 7f1222ffd700 1 -- 172.21.15.23:0/3029111595 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4274 (secure 0 0 0) 0x7f1230002970 con 0x7f1218006bc0 2020-11-20T11:45:43.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.673+0000 7f1222ffd700 1 -- 172.21.15.23:0/3029111595 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1218006bc0 msgr2=0x7f1218007050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:43.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.673+0000 7f1222ffd700 1 --2- 172.21.15.23:0/3029111595 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1218006bc0 0x7f1218007050 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f1230016000 tx=0x7f1230017000).stop 2020-11-20T11:45:43.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.673+0000 7f1222ffd700 1 -- 172.21.15.23:0/3029111595 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1218006bc0 msgr2=0x7f1218007050 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:45:43.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.673+0000 7f1222ffd700 1 --2- 172.21.15.23:0/3029111595 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1218006bc0 0x7f1218007050 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0x7f1230016000 tx=0x7f1230017000).stop 2020-11-20T11:45:43.681 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:45:43.682 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:45:43.682 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:45:43.682 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:45:43.682 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 126, 2020-11-20T11:45:43.682 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:45:43.683 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:45:43.683 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:45:43.683 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:45:43.683 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:43.684 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 4, 2020-11-20T11:45:43.684 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:45:43.684 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:45:43.684 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:45:43.684 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:45:43.685 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:45:43.685 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:45:43.685 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:45:43.685 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:45:43.686 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:45:43.686 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:45:43.686 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:45:43.686 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:43.686 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:45:43.687 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:45:43.687 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:45:43.687 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:45:43.687 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:45:43.687 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:45:43.688 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:45:43.688 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:45:43.688 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:45:43.688 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:45:43.688 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:43.689 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:43.689 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:45:43.689 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:45:43.689 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:45:43.690 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:45:43.690 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:45:43.690 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:45:43.690 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:45:43.690 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:45:43.691 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:45:43.691 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:45:43.691 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:45:43.691 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:45:43.691 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:45:43.692 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:45:43.692 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:45:43.692 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:45:43.692 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:45:43.692 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:45:43.693 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:45:43.693 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:45:43.693 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:45:43.693 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:45:43.694 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:45:43.694 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:43.694 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:45:43.694 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:43.694 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:45:43.695 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:43.695 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:45:43.695 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:45:43.696 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:45:43.696 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:45:43.696 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:43.696 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:45:43.697 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:45:43.697 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:43.697 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:43.697 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:43.697 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:45:43.698 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:45:43.698 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:43.698 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:43.698 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:43.698 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:43.699 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:45:43.699 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:45:43.699 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:45:43.699 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:45:43.699 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:45:43.700 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:43.700 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:43.700 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:45:43.700 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:45:43.700 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:45:43.701 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:45:43.701 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:43.701 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:45:43.701 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:45:43.701 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:43.702 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:43.702 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:43.702 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:45:43.702 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:45:43.703 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:43.703 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:43.703 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:43.703 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:43.704 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:45:43.704 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:45:43.704 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:45:43.704 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:45:43.704 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:45:43.705 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:43.705 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:43.705 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:45:43.705 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:45:43.705 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:45:43.706 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:45:43.706 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:43.706 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:45:43.706 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:45:43.707 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:43.707 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:43.707 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:43.709 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:45:43.709 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:45:43.709 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:43.709 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:43.710 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:43.710 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:43.710 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:45:43.710 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:45:43.710 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:45:43.711 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:45:43.711 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:45:43.711 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:43.711 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:43.712 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:43.712 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:45:43.712 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:45:43.712 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:43.712 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:45:43.713 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:45:43.713 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:45:43.713 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:43.713 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:43.713 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:45:43.714 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:43.714 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:45:43.714 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:45:43.714 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:45:43.714 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:43.715 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:43.715 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:43.715 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:45:43.715 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:45:43.717 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.675+0000 7f123bfff700 1 -- 172.21.15.23:0/3029111595 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1214056db0 msgr2=0x7f1214059270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:43.717 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.675+0000 7f123bfff700 1 --2- 172.21.15.23:0/3029111595 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1214056db0 0x7f1214059270 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f122c00b190 tx=0x7f122c015000).stop 2020-11-20T11:45:43.717 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.675+0000 7f123b7fe700 1 -- 172.21.15.23:0/3029111595 reap_dead start 2020-11-20T11:45:43.717 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.675+0000 7f123bfff700 1 -- 172.21.15.23:0/3029111595 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f123c14c660 msgr2=0x7f123c09dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:43.718 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.676+0000 7f123bfff700 1 --2- 172.21.15.23:0/3029111595 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f123c14c660 0x7f123c09dc80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f123400ebf0 tx=0x7f1234008040).stop 2020-11-20T11:45:43.718 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.676+0000 7f123bfff700 1 -- 172.21.15.23:0/3029111595 shutdown_connections 2020-11-20T11:45:43.718 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.676+0000 7f123bfff700 1 --2- 172.21.15.23:0/3029111595 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f123c14c660 0x7f123c09dc80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:43.718 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.676+0000 7f123bfff700 1 -- 172.21.15.23:0/3029111595 >> 172.21.15.23:0/3029111595 conn(0x7f123c061850 msgr2=0x7f123c0a9690 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:43.718 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.676+0000 7f123bfff700 1 -- 172.21.15.23:0/3029111595 shutdown_connections 2020-11-20T11:45:43.719 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.676+0000 7f123bfff700 1 -- 172.21.15.23:0/3029111595 wait complete. 2020-11-20T11:45:43.719 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2020-11-20T11:45:43.719 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2020-11-20T11:45:43.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.875+0000 7f7f83b0c700 1 Processor -- start 2020-11-20T11:45:43.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.876+0000 7f7f83b0c700 1 -- start start 2020-11-20T11:45:43.887 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.876+0000 7f7f83b0c700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f7f7c0aaab0 0x7f7f7c13ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:43.887 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.876+0000 7f7f83b0c700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f7c1388b0 con 0x7f7f7c0aae90 2020-11-20T11:45:43.887 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.876+0000 7f7f83b0c700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f7c13e600 con 0x7f7f7c13aeb0 2020-11-20T11:45:43.887 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.876+0000 7f7f83b0c700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f7c13e8a0 con 0x7f7f7c0aaab0 2020-11-20T11:45:43.888 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.876+0000 7f7f810a7700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f7f7c0aae90 0x7f7f7c1381a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:60216/0 (socket says 172.21.15.23:60216) 2020-11-20T11:45:43.888 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.876+0000 7f7f810a7700 1 -- 172.21.15.23:0/1803401348 learned_addr learned my addr 172.21.15.23:0/1803401348 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:45:43.888 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.877+0000 7f7f818a8700 1 --2- 172.21.15.23:0/1803401348 >> v2:172.21.15.155:3300/0 conn(0x7f7f7c0aaab0 0x7f7f7c13ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:43.888 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.877+0000 7f7f808a6700 1 -- 172.21.15.23:0/1803401348 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243156496 0 0) 0x7f7f7c1388b0 con 0x7f7f7c0aae90 2020-11-20T11:45:43.888 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.877+0000 7f7f808a6700 1 -- 172.21.15.23:0/1803401348 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f64003280 con 0x7f7f7c0aae90 2020-11-20T11:45:43.889 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.877+0000 7f7f808a6700 1 -- 172.21.15.23:0/1803401348 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2515839408 0 0) 0x7f7f7c13e600 con 0x7f7f7c13aeb0 2020-11-20T11:45:43.889 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.877+0000 7f7f808a6700 1 -- 172.21.15.23:0/1803401348 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f7c1388b0 con 0x7f7f7c13aeb0 2020-11-20T11:45:43.889 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.877+0000 7f7f808a6700 1 -- 172.21.15.23:0/1803401348 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 85220746 0 0) 0x7f7f64003280 con 0x7f7f7c0aae90 2020-11-20T11:45:43.889 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.877+0000 7f7f808a6700 1 -- 172.21.15.23:0/1803401348 >> v2:172.21.15.155:3300/0 conn(0x7f7f7c0aaab0 msgr2=0x7f7f7c13ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:45:43.890 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.877+0000 7f7f808a6700 1 --2- 172.21.15.23:0/1803401348 >> v2:172.21.15.155:3300/0 conn(0x7f7f7c0aaab0 0x7f7f7c13ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:43.890 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.877+0000 7f7f808a6700 1 -- 172.21.15.23:0/1803401348 >> v1:172.21.15.155:6789/0 conn(0x7f7f7c13aeb0 legacy=0x7f7f7c0a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:43.890 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.877+0000 7f7f808a6700 1 -- 172.21.15.23:0/1803401348 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f7c0ab6d0 con 0x7f7f7c0aae90 2020-11-20T11:45:43.890 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.877+0000 7f7f808a6700 1 -- 172.21.15.23:0/1803401348 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f7f78001520 con 0x7f7f7c0aae90 2020-11-20T11:45:43.890 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.877+0000 7f7f808a6700 1 -- 172.21.15.23:0/1803401348 >> v1:172.21.15.23:6789/0 conn(0x7f7f7c0aae90 legacy=0x7f7f7c1381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:43.891 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.877+0000 7f7f808a6700 1 --2- 172.21.15.23:0/1803401348 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7f64003900 0x7f7f64003d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:43.891 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.877+0000 7f7f808a6700 1 --2- 172.21.15.23:0/1803401348 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f64004520 0x7f7f640049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:43.891 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.877+0000 7f7f808a6700 1 --2- 172.21.15.23:0/1803401348 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f64004e90 0x7f7f6400b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:43.891 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.877+0000 7f7f808a6700 1 -- 172.21.15.23:0/1803401348 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7c13e8a0 con 0x7f7f64004e90 2020-11-20T11:45:43.891 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.877+0000 7f7f808a6700 1 -- 172.21.15.23:0/1803401348 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6400bc60 con 0x7f7f64004520 2020-11-20T11:45:43.892 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.877+0000 7f7f808a6700 1 -- 172.21.15.23:0/1803401348 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6400bda0 con 0x7f7f64003900 2020-11-20T11:45:43.892 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.877+0000 7f7f820a9700 1 --2- 172.21.15.23:0/1803401348 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f64004e90 0x7f7f6400b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:43.892 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.877+0000 7f7f818a8700 1 --2- 172.21.15.23:0/1803401348 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7f64003900 0x7f7f64003d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:43.892 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.877+0000 7f7f810a7700 1 --2- 172.21.15.23:0/1803401348 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f64004520 0x7f7f640049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:43.892 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.878+0000 7f7f820a9700 1 -- 172.21.15.23:0/1803401348 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7f64003900 msgr2=0x7f7f64003d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:45:43.893 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.878+0000 7f7f820a9700 1 --2- 172.21.15.23:0/1803401348 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7f64003900 0x7f7f64003d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:43.893 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.878+0000 7f7f820a9700 1 -- 172.21.15.23:0/1803401348 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f64004520 msgr2=0x7f7f640049a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:45:43.893 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.878+0000 7f7f820a9700 1 --2- 172.21.15.23:0/1803401348 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f64004520 0x7f7f640049a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:43.893 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.878+0000 7f7f820a9700 1 -- 172.21.15.23:0/1803401348 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f64003280 con 0x7f7f64004e90 2020-11-20T11:45:43.894 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.878+0000 7f7f820a9700 1 -- 172.21.15.23:0/1803401348 reap_dead start 2020-11-20T11:45:43.894 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.878+0000 7f7f820a9700 1 --2- 172.21.15.23:0/1803401348 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f64004e90 0x7f7f6400b770 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f7f74002050 tx=0x7f7f74007f50).ready entity=mon.0 client_cookie=f0ab601343b78924 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:43.894 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.878+0000 7f7f808a6700 1 -- 172.21.15.23:0/1803401348 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7400cb00 con 0x7f7f64004e90 2020-11-20T11:45:43.894 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.878+0000 7f7f808a6700 1 -- 172.21.15.23:0/1803401348 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f7400cc60 con 0x7f7f64004e90 2020-11-20T11:45:43.895 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.878+0000 7f7f808a6700 1 -- 172.21.15.23:0/1803401348 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f740068b0 con 0x7f7f64004e90 2020-11-20T11:45:43.895 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.878+0000 7f7f83b0c700 1 -- 172.21.15.23:0/1803401348 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f64004e90 msgr2=0x7f7f6400b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:43.895 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.878+0000 7f7f83b0c700 1 --2- 172.21.15.23:0/1803401348 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f64004e90 0x7f7f6400b770 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f7f74002050 tx=0x7f7f74007f50).stop 2020-11-20T11:45:43.895 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.879+0000 7f7f83b0c700 1 -- 172.21.15.23:0/1803401348 shutdown_connections 2020-11-20T11:45:43.895 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.879+0000 7f7f83b0c700 1 --2- 172.21.15.23:0/1803401348 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f64004e90 0x7f7f6400b770 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:43.896 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.879+0000 7f7f83b0c700 1 -- 172.21.15.23:0/1803401348 >> 172.21.15.23:0/1803401348 conn(0x7f7f7c0678e0 msgr2=0x7f7f7c067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:43.896 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.879+0000 7f7f83b0c700 1 -- 172.21.15.23:0/1803401348 shutdown_connections 2020-11-20T11:45:43.896 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.879+0000 7f7f83b0c700 1 -- 172.21.15.23:0/1803401348 wait complete. 2020-11-20T11:45:43.896 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.880+0000 7f7f83b0c700 1 Processor -- start 2020-11-20T11:45:43.896 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.880+0000 7f7f83b0c700 1 -- start start 2020-11-20T11:45:43.897 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.880+0000 7f7f83b0c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f64004e90 0x7f7f7c064490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:43.897 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.880+0000 7f7f83b0c700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7f7c064ca0 0x7f7f7c0a7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:43.897 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.880+0000 7f7f83b0c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f7c151d90 0x7f7f7c1521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:43.897 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.880+0000 7f7f83b0c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7c13c080 con 0x7f7f64004e90 2020-11-20T11:45:43.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.880+0000 7f7f83b0c700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7c0b10b0 con 0x7f7f7c151d90 2020-11-20T11:45:43.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.880+0000 7f7f83b0c700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7c0b0e10 con 0x7f7f7c064ca0 2020-11-20T11:45:43.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.880+0000 7f7f818a8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f64004e90 0x7f7f7c064490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:43.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.880+0000 7f7f818a8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f64004e90 0x7f7f7c064490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45976/0 (socket says 172.21.15.23:45976) 2020-11-20T11:45:43.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.880+0000 7f7f818a8700 1 -- 172.21.15.23:0/2820204354 learned_addr learned my addr 172.21.15.23:0/2820204354 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:45:43.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.881+0000 7f7f810a7700 1 --2- 172.21.15.23:0/2820204354 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7f7c064ca0 0x7f7f7c0a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:43.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.881+0000 7f7f820a9700 1 --2- 172.21.15.23:0/2820204354 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f7c151d90 0x7f7f7c1521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:43.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.881+0000 7f7f818a8700 1 -- 172.21.15.23:0/2820204354 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7f7c064ca0 msgr2=0x7f7f7c0a7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:43.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.881+0000 7f7f818a8700 1 --2- 172.21.15.23:0/2820204354 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7f7c064ca0 0x7f7f7c0a7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:43.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.881+0000 7f7f818a8700 1 -- 172.21.15.23:0/2820204354 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f7c151d90 msgr2=0x7f7f7c1521b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:43.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.881+0000 7f7f818a8700 1 --2- 172.21.15.23:0/2820204354 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f7c151d90 0x7f7f7c1521b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:43.900 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.881+0000 7f7f818a8700 1 -- 172.21.15.23:0/2820204354 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f7c0a74b0 con 0x7f7f64004e90 2020-11-20T11:45:43.900 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.881+0000 7f7f818a8700 1 --2- 172.21.15.23:0/2820204354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f64004e90 0x7f7f7c064490 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f7f6c0068a0 tx=0x7f7f6c006e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:43.900 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.881+0000 7f7f69ffb700 1 -- 172.21.15.23:0/2820204354 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f6c018070 con 0x7f7f64004e90 2020-11-20T11:45:43.900 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.881+0000 7f7f69ffb700 1 -- 172.21.15.23:0/2820204354 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f6c000e90 con 0x7f7f64004e90 2020-11-20T11:45:43.900 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.881+0000 7f7f69ffb700 1 -- 172.21.15.23:0/2820204354 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f6c00fb50 con 0x7f7f64004e90 2020-11-20T11:45:43.901 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.881+0000 7f7f83b0c700 1 -- 172.21.15.23:0/2820204354 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f7c064980 con 0x7f7f64004e90 2020-11-20T11:45:43.901 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.881+0000 7f7f83b0c700 1 -- 172.21.15.23:0/2820204354 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f7c0a5610 con 0x7f7f64004e90 2020-11-20T11:45:43.901 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.882+0000 7f7f83b0c700 1 -- 172.21.15.23:0/2820204354 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f7c05c7d0 con 0x7f7f64004e90 2020-11-20T11:45:43.901 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.882+0000 7f7f69ffb700 1 -- 172.21.15.23:0/2820204354 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7f6c014440 con 0x7f7f64004e90 2020-11-20T11:45:43.901 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.883+0000 7f7f69ffb700 1 --2- 172.21.15.23:0/2820204354 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7f54056d60 0x7f7f54059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:43.902 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.883+0000 7f7f69ffb700 1 -- 172.21.15.23:0/2820204354 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f7f6c070550 con 0x7f7f64004e90 2020-11-20T11:45:43.902 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.883+0000 7f7f810a7700 1 --2- 172.21.15.23:0/2820204354 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7f54056d60 0x7f7f54059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:43.903 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.883+0000 7f7f810a7700 1 --2- 172.21.15.23:0/2820204354 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7f54056d60 0x7f7f54059220 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f7f7c0a6c20 tx=0x7f7f78007790).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:43.903 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:43.887+0000 7f7f69ffb700 1 -- 172.21.15.23:0/2820204354 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7f6c0457e0 con 0x7f7f64004e90 2020-11-20T11:45:44.167 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:44.163+0000 7f7f83b0c700 1 -- 172.21.15.23:0/2820204354 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f7f7c064b10 con 0x7f7f64004e90 2020-11-20T11:45:44.544 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:44.164+0000 7f7f69ffb700 1 -- 172.21.15.23:0/2820204354 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f7f6c048e00 con 0x7f7f64004e90 2020-11-20T11:45:44.545 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:44.168+0000 7f7f83b0c700 1 -- 172.21.15.23:0/2820204354 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7f54056d60 msgr2=0x7f7f54059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:44.545 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:44.168+0000 7f7f83b0c700 1 --2- 172.21.15.23:0/2820204354 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7f54056d60 0x7f7f54059220 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f7f7c0a6c20 tx=0x7f7f78007790).stop 2020-11-20T11:45:44.545 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:44.168+0000 7f7f83b0c700 1 -- 172.21.15.23:0/2820204354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f64004e90 msgr2=0x7f7f7c064490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:44.545 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:44.168+0000 7f7f83b0c700 1 --2- 172.21.15.23:0/2820204354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f64004e90 0x7f7f7c064490 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f7f6c0068a0 tx=0x7f7f6c006e30).stop 2020-11-20T11:45:44.546 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:44.168+0000 7f7f83b0c700 1 -- 172.21.15.23:0/2820204354 shutdown_connections 2020-11-20T11:45:44.546 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:44.168+0000 7f7f83b0c700 1 --2- 172.21.15.23:0/2820204354 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7f7c151d90 0x7f7f7c1521b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:44.546 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:44.168+0000 7f7f83b0c700 1 --2- 172.21.15.23:0/2820204354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7f64004e90 0x7f7f7c064490 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:44.546 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:44.168+0000 7f7f83b0c700 1 --2- 172.21.15.23:0/2820204354 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7f54056d60 0x7f7f54059220 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:44.547 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:44.168+0000 7f7f83b0c700 1 --2- 172.21.15.23:0/2820204354 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7f7c064ca0 0x7f7f7c0a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:44.547 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:44.168+0000 7f7f83b0c700 1 -- 172.21.15.23:0/2820204354 >> 172.21.15.23:0/2820204354 conn(0x7f7f7c0678e0 msgr2=0x7f7f7c0b1360 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:44.547 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:44.168+0000 7f7f83b0c700 1 -- 172.21.15.23:0/2820204354 shutdown_connections 2020-11-20T11:45:44.547 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:44.168+0000 7f7f83b0c700 1 -- 172.21.15.23:0/2820204354 wait complete. 2020-11-20T11:45:44.547 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2020-11-20T11:45:54.549 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:45:54.549 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:45:54.913 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.907+0000 7fbaed90b700 1 Processor -- start 2020-11-20T11:45:54.914 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.907+0000 7fbaed90b700 1 -- start start 2020-11-20T11:45:54.914 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.907+0000 7fbaed90b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbae80a2080 0x7fbae80aef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:54.915 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.907+0000 7fbaed90b700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbae80aaab0 con 0x7fbae80a2460 2020-11-20T11:45:54.915 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.907+0000 7fbaed90b700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbae80aad50 con 0x7fbae8067a10 2020-11-20T11:45:54.915 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.907+0000 7fbaed90b700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae813aa50 con 0x7fbae80a2080 2020-11-20T11:45:54.915 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.908+0000 7fbae7fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fbae80a2460 0x7fbae8137df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:60234/0 (socket says 172.21.15.23:60234) 2020-11-20T11:45:54.916 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.908+0000 7fbae7fff700 1 -- 172.21.15.23:0/1102169422 learned_addr learned my addr 172.21.15.23:0/1102169422 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:45:54.916 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.908+0000 7fbaec909700 1 --2- 172.21.15.23:0/1102169422 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbae80a2080 0x7fbae80aef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:54.916 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.908+0000 7fbae77fe700 1 -- 172.21.15.23:0/1102169422 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2646193996 0 0) 0x7fbae80aaab0 con 0x7fbae80a2460 2020-11-20T11:45:54.916 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.908+0000 7fbae77fe700 1 -- 172.21.15.23:0/1102169422 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbacc003280 con 0x7fbae80a2460 2020-11-20T11:45:54.917 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.908+0000 7fbae77fe700 1 -- 172.21.15.23:0/1102169422 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048751335 0 0) 0x7fbae80aad50 con 0x7fbae8067a10 2020-11-20T11:45:54.917 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.908+0000 7fbae77fe700 1 -- 172.21.15.23:0/1102169422 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbae80aaab0 con 0x7fbae8067a10 2020-11-20T11:45:54.917 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.908+0000 7fbaec909700 1 -- 172.21.15.23:0/1102169422 >> v1:172.21.15.155:6789/0 conn(0x7fbae8067a10 legacy=0x7fbae81453c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:54.917 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.908+0000 7fbaec909700 1 -- 172.21.15.23:0/1102169422 >> v1:172.21.15.23:6789/0 conn(0x7fbae80a2460 legacy=0x7fbae8137df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:54.917 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.908+0000 7fbaec909700 1 -- 172.21.15.23:0/1102169422 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae813b920 con 0x7fbae80a2080 2020-11-20T11:45:54.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.908+0000 7fbaec909700 1 --2- 172.21.15.23:0/1102169422 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbae80a2080 0x7fbae80aef10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fbadc00ba00 tx=0x7fbadc006dd0).ready entity=mon.2 client_cookie=9e2fe99f3e689469 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:54.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.908+0000 7fbae77fe700 1 -- 172.21.15.23:0/1102169422 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbadc006030 con 0x7fbae80a2080 2020-11-20T11:45:54.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.908+0000 7fbae77fe700 1 -- 172.21.15.23:0/1102169422 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbadc006190 con 0x7fbae80a2080 2020-11-20T11:45:54.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.908+0000 7fbae77fe700 1 -- 172.21.15.23:0/1102169422 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbadc006510 con 0x7fbae80a2080 2020-11-20T11:45:54.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.909+0000 7fbaed90b700 1 -- 172.21.15.23:0/1102169422 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbae80a2080 msgr2=0x7fbae80aef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:54.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.909+0000 7fbaed90b700 1 --2- 172.21.15.23:0/1102169422 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbae80a2080 0x7fbae80aef10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fbadc00ba00 tx=0x7fbadc006dd0).stop 2020-11-20T11:45:54.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.909+0000 7fbaed90b700 1 -- 172.21.15.23:0/1102169422 shutdown_connections 2020-11-20T11:45:54.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.909+0000 7fbaed90b700 1 --2- 172.21.15.23:0/1102169422 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbae80a2080 0x7fbae80aef10 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:54.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.909+0000 7fbaed90b700 1 -- 172.21.15.23:0/1102169422 >> 172.21.15.23:0/1102169422 conn(0x7fbae8063980 msgr2=0x7fbae8063d90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:54.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.909+0000 7fbaed90b700 1 -- 172.21.15.23:0/1102169422 shutdown_connections 2020-11-20T11:45:54.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.909+0000 7fbaed90b700 1 -- 172.21.15.23:0/1102169422 wait complete. 2020-11-20T11:45:54.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.910+0000 7fbaed90b700 1 Processor -- start 2020-11-20T11:45:54.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.910+0000 7fbaed90b700 1 -- start start 2020-11-20T11:45:54.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.911+0000 7fbaed90b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbae8067a10 0x7fbae813c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:54.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.911+0000 7fbaed90b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbae80a2080 0x7fbae80a7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:54.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.911+0000 7fbaed90b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbae80a2460 0x7fbae814d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:54.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.911+0000 7fbaed90b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae80a89e0 con 0x7fbae8067a10 2020-11-20T11:45:54.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.911+0000 7fbaed90b700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae80a6530 con 0x7fbae80a2460 2020-11-20T11:45:54.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.911+0000 7fbaed90b700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae80a67d0 con 0x7fbae80a2080 2020-11-20T11:45:54.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.911+0000 7fbaec909700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbae8067a10 0x7fbae813c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:54.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.911+0000 7fbaec909700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbae8067a10 0x7fbae813c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45990/0 (socket says 172.21.15.23:45990) 2020-11-20T11:45:54.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.911+0000 7fbaec909700 1 -- 172.21.15.23:0/3565608848 learned_addr learned my addr 172.21.15.23:0/3565608848 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:45:54.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.911+0000 7fbae7fff700 1 --2- 172.21.15.23:0/3565608848 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbae80a2080 0x7fbae80a7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:54.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.911+0000 7fbaed10a700 1 --2- 172.21.15.23:0/3565608848 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbae80a2460 0x7fbae814d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:54.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.911+0000 7fbaec909700 1 -- 172.21.15.23:0/3565608848 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbae80a2080 msgr2=0x7fbae80a7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:54.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.911+0000 7fbaec909700 1 --2- 172.21.15.23:0/3565608848 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbae80a2080 0x7fbae80a7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:54.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.911+0000 7fbaec909700 1 -- 172.21.15.23:0/3565608848 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbae80a2460 msgr2=0x7fbae814d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:54.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.911+0000 7fbaec909700 1 --2- 172.21.15.23:0/3565608848 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbae80a2460 0x7fbae814d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:54.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.911+0000 7fbaec909700 1 -- 172.21.15.23:0/3565608848 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae80a6a70 con 0x7fbae8067a10 2020-11-20T11:45:54.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.911+0000 7fbaec909700 1 --2- 172.21.15.23:0/3565608848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbae8067a10 0x7fbae813c150 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fbadc006fd0 tx=0x7fbadc015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:54.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.911+0000 7fbae4ff9700 1 -- 172.21.15.23:0/3565608848 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbadc0064d0 con 0x7fbae8067a10 2020-11-20T11:45:54.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.911+0000 7fbae4ff9700 1 -- 172.21.15.23:0/3565608848 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbadc009ce0 con 0x7fbae8067a10 2020-11-20T11:45:54.928 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.911+0000 7fbaed90b700 1 -- 172.21.15.23:0/3565608848 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbae8137bb0 con 0x7fbae8067a10 2020-11-20T11:45:54.928 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.911+0000 7fbae4ff9700 1 -- 172.21.15.23:0/3565608848 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbadc01e670 con 0x7fbae8067a10 2020-11-20T11:45:54.928 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.911+0000 7fbaed90b700 1 -- 172.21.15.23:0/3565608848 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae80a6f10 con 0x7fbae8067a10 2020-11-20T11:45:54.928 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.912+0000 7fbaca7fc700 1 -- 172.21.15.23:0/3565608848 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbab8004fd0 con 0x7fbae8067a10 2020-11-20T11:45:54.928 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.913+0000 7fbae4ff9700 1 -- 172.21.15.23:0/3565608848 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fbadc025030 con 0x7fbae8067a10 2020-11-20T11:45:54.929 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.913+0000 7fbae4ff9700 1 --2- 172.21.15.23:0/3565608848 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbac0056d60 0x7fbac0059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:54.929 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.913+0000 7fbae4ff9700 1 -- 172.21.15.23:0/3565608848 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fbadc071750 con 0x7fbae8067a10 2020-11-20T11:45:54.929 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.913+0000 7fbae7fff700 1 --2- 172.21.15.23:0/3565608848 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbac0056d60 0x7fbac0059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:54.929 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.914+0000 7fbae7fff700 1 --2- 172.21.15.23:0/3565608848 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbac0056d60 0x7fbac0059220 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7fbae8068420 tx=0x7fbad4009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:54.930 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:54.920+0000 7fbae4ff9700 1 -- 172.21.15.23:0/3565608848 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fbadc046af0 con 0x7fbae8067a10 2020-11-20T11:45:55.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.191+0000 7fbaca7fc700 1 -- 172.21.15.23:0/3565608848 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbab80016a0 con 0x7fbae8067a10 2020-11-20T11:45:55.196 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.192+0000 7fbae4ff9700 1 -- 172.21.15.23:0/3565608848 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1479 (secure 0 0 0) 0x7fbadc023070 con 0x7fbae8067a10 2020-11-20T11:45:55.196 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":126,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:45:55.197 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.193+0000 7fbaca7fc700 1 -- 172.21.15.23:0/3565608848 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbac0056d60 msgr2=0x7fbac0059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:55.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.193+0000 7fbaca7fc700 1 --2- 172.21.15.23:0/3565608848 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbac0056d60 0x7fbac0059220 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7fbae8068420 tx=0x7fbad4009040).stop 2020-11-20T11:45:55.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.193+0000 7fbaca7fc700 1 -- 172.21.15.23:0/3565608848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbae8067a10 msgr2=0x7fbae813c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:55.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.193+0000 7fbaca7fc700 1 --2- 172.21.15.23:0/3565608848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbae8067a10 0x7fbae813c150 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fbadc006fd0 tx=0x7fbadc015000).stop 2020-11-20T11:45:55.198 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.193+0000 7fbaca7fc700 1 -- 172.21.15.23:0/3565608848 shutdown_connections 2020-11-20T11:45:55.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.193+0000 7fbaca7fc700 1 --2- 172.21.15.23:0/3565608848 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbae80a2460 0x7fbae814d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:55.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.193+0000 7fbaca7fc700 1 --2- 172.21.15.23:0/3565608848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbae8067a10 0x7fbae813c150 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:55.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.193+0000 7fbaca7fc700 1 --2- 172.21.15.23:0/3565608848 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fbac0056d60 0x7fbac0059220 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:55.199 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.193+0000 7fbaca7fc700 1 --2- 172.21.15.23:0/3565608848 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbae80a2080 0x7fbae80a7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:55.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.193+0000 7fbaca7fc700 1 -- 172.21.15.23:0/3565608848 >> 172.21.15.23:0/3565608848 conn(0x7fbae8063980 msgr2=0x7fbae8068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:55.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.193+0000 7fbaca7fc700 1 -- 172.21.15.23:0/3565608848 shutdown_connections 2020-11-20T11:45:55.200 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.193+0000 7fbaca7fc700 1 -- 172.21.15.23:0/3565608848 wait complete. 2020-11-20T11:45:55.211 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:45:55.211 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2020-11-20T11:45:55.211 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:45:55.458 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.454+0000 7f891773c700 1 Processor -- start 2020-11-20T11:45:55.461 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.455+0000 7f891773c700 1 -- start start 2020-11-20T11:45:55.462 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.456+0000 7f891773c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8910061a10 0x7f89100a6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:55.462 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.456+0000 7f891773c700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f891013f800 0x7f8910141bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:55.462 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.456+0000 7f891773c700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89100a4970 con 0x7f8910136ea0 2020-11-20T11:45:55.462 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.456+0000 7f891773c700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89100a5ef0 con 0x7f8910061a10 2020-11-20T11:45:55.462 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.456+0000 7f891773c700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89100a5490 con 0x7f891013f800 2020-11-20T11:45:55.463 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.456+0000 7f89154d8700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8910061a10 0x7f89100a6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:55.463 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.456+0000 7f89154d8700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8910061a10 0x7f89100a6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46002/0 (socket says 172.21.15.23:46002) 2020-11-20T11:45:55.463 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.456+0000 7f89154d8700 1 -- 172.21.15.23:0/392922574 learned_addr learned my addr 172.21.15.23:0/392922574 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:45:55.463 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.457+0000 7f89154d8700 1 -- 172.21.15.23:0/392922574 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f891013f800 msgr2=0x7f8910141bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:45:55.463 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.457+0000 7f89154d8700 1 --2- 172.21.15.23:0/392922574 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f891013f800 0x7f8910141bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:55.464 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.457+0000 7f89154d8700 1 -- 172.21.15.23:0/392922574 >> v1:172.21.15.155:6789/0 conn(0x7f8910136ea0 legacy=0x7f89100a8ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2020-11-20T11:45:55.464 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.457+0000 7f89154d8700 1 -- 172.21.15.23:0/392922574 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89100a60e0 con 0x7f8910061a10 2020-11-20T11:45:55.464 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.457+0000 7f89154d8700 1 --2- 172.21.15.23:0/392922574 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8910061a10 0x7f89100a6580 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f890c0076e0 tx=0x7f890c00c330).ready entity=mon.0 client_cookie=4ba5a957a38c0dfc server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:55.464 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.457+0000 7f88fffff700 1 -- 172.21.15.23:0/392922574 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890c00f070 con 0x7f8910061a10 2020-11-20T11:45:55.464 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.458+0000 7f88fffff700 1 -- 172.21.15.23:0/392922574 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f890c000e90 con 0x7f8910061a10 2020-11-20T11:45:55.465 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.458+0000 7f891773c700 1 -- 172.21.15.23:0/392922574 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8910061a10 msgr2=0x7f89100a6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:55.465 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.458+0000 7f891773c700 1 --2- 172.21.15.23:0/392922574 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8910061a10 0x7f89100a6580 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f890c0076e0 tx=0x7f890c00c330).stop 2020-11-20T11:45:55.466 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.459+0000 7f891773c700 1 -- 172.21.15.23:0/392922574 shutdown_connections 2020-11-20T11:45:55.466 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.459+0000 7f891773c700 1 --2- 172.21.15.23:0/392922574 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8910061a10 0x7f89100a6580 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:55.466 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.459+0000 7f891773c700 1 --2- 172.21.15.23:0/392922574 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f891013f800 0x7f8910141bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:55.467 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.459+0000 7f891773c700 1 -- 172.21.15.23:0/392922574 >> 172.21.15.23:0/392922574 conn(0x7f8910060c80 msgr2=0x7f89100acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:55.473 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.470+0000 7f891773c700 1 -- 172.21.15.23:0/392922574 shutdown_connections 2020-11-20T11:45:55.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.488+0000 7f891773c700 1 -- 172.21.15.23:0/392922574 wait complete. 2020-11-20T11:45:55.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.489+0000 7f891773c700 1 Processor -- start 2020-11-20T11:45:55.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.489+0000 7f891773c700 1 -- start start 2020-11-20T11:45:55.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.490+0000 7f891773c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8910061a10 0x7f891009d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:55.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.490+0000 7f891773c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f891013f800 0x7f891009f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:55.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.490+0000 7f891773c700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f89100a0010 0x7f89100a0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:55.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.491+0000 7f891773c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891013f5f0 con 0x7f8910061a10 2020-11-20T11:45:55.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.491+0000 7f891773c700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8910067400 con 0x7f891013f800 2020-11-20T11:45:55.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.491+0000 7f891773c700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89100676a0 con 0x7f89100a0010 2020-11-20T11:45:55.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.491+0000 7f89154d8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8910061a10 0x7f891009d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:55.497 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.491+0000 7f89154d8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8910061a10 0x7f891009d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46004/0 (socket says 172.21.15.23:46004) 2020-11-20T11:45:55.497 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.491+0000 7f89154d8700 1 -- 172.21.15.23:0/2844393400 learned_addr learned my addr 172.21.15.23:0/2844393400 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:45:55.498 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.491+0000 7f89154d8700 1 -- 172.21.15.23:0/2844393400 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f89100a0010 msgr2=0x7f89100a0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:55.498 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.491+0000 7f89154d8700 1 --2- 172.21.15.23:0/2844393400 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f89100a0010 0x7f89100a0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:55.498 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.491+0000 7f89154d8700 1 -- 172.21.15.23:0/2844393400 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f891013f800 msgr2=0x7f891009f5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2020-11-20T11:45:55.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.491+0000 7f89154d8700 1 --2- 172.21.15.23:0/2844393400 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f891013f800 0x7f891009f5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:55.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.491+0000 7f89154d8700 1 -- 172.21.15.23:0/2844393400 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f891009fac0 con 0x7f8910061a10 2020-11-20T11:45:55.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.492+0000 7f89154d8700 1 --2- 172.21.15.23:0/2844393400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8910061a10 0x7f891009d660 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f890c00c370 tx=0x7f89100a6270).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:55.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.495+0000 7f88fd7fa700 1 -- 172.21.15.23:0/2844393400 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890c018070 con 0x7f8910061a10 2020-11-20T11:45:55.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.495+0000 7f891773c700 1 -- 172.21.15.23:0/2844393400 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f891009fcb0 con 0x7f8910061a10 2020-11-20T11:45:55.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.496+0000 7f891773c700 1 -- 172.21.15.23:0/2844393400 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8910152670 con 0x7f8910061a10 2020-11-20T11:45:55.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.496+0000 7f88fd7fa700 1 -- 172.21.15.23:0/2844393400 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f890c019040 con 0x7f8910061a10 2020-11-20T11:45:55.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.496+0000 7f88fd7fa700 1 -- 172.21.15.23:0/2844393400 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890c007ea0 con 0x7f8910061a10 2020-11-20T11:45:55.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.499+0000 7f88fd7fa700 1 -- 172.21.15.23:0/2844393400 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f890c024030 con 0x7f8910061a10 2020-11-20T11:45:55.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.499+0000 7f88fd7fa700 1 --2- 172.21.15.23:0/2844393400 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f88ec056db0 0x7f88ec059270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:55.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.500+0000 7f88fd7fa700 1 -- 172.21.15.23:0/2844393400 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f890c070ce0 con 0x7f8910061a10 2020-11-20T11:45:55.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.500+0000 7f8914cd7700 1 --2- 172.21.15.23:0/2844393400 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f88ec056db0 0x7f88ec059270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:55.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.503+0000 7f891773c700 1 --2- 172.21.15.23:0/2844393400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8910152920 0x7f8910152d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:55.508 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.503+0000 7f891773c700 1 -- 172.21.15.23:0/2844393400 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89100678a0 con 0x7f8910152920 2020-11-20T11:45:55.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.508+0000 7f8915cd9700 1 --2- 172.21.15.23:0/2844393400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8910152920 0x7f8910152d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:55.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.509+0000 7f8915cd9700 1 -- 172.21.15.23:0/2844393400 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f89101380e0 con 0x7f8910152920 2020-11-20T11:45:55.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.509+0000 7f8915cd9700 1 --2- 172.21.15.23:0/2844393400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8910152920 0x7f8910152d00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f89100a5e10 tx=0x7f8900006d20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:55.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.510+0000 7f88fd7fa700 1 -- 172.21.15.23:0/2844393400 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8900002320 con 0x7f8910152920 2020-11-20T11:45:55.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.510+0000 7f8914cd7700 1 --2- 172.21.15.23:0/2844393400 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f88ec056db0 0x7f88ec059270 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f891009ee30 tx=0x7f8908006cb0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:55.518 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.514+0000 7f88fd7fa700 1 -- 172.21.15.23:0/2844393400 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f8900002480 con 0x7f8910152920 2020-11-20T11:45:55.518 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.514+0000 7f88fd7fa700 1 -- 172.21.15.23:0/2844393400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8910152920 msgr2=0x7f8910152d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:55.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.514+0000 7f88fd7fa700 1 --2- 172.21.15.23:0/2844393400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8910152920 0x7f8910152d00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f89100a5e10 tx=0x7f8900006d20).stop 2020-11-20T11:45:55.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.514+0000 7f88fd7fa700 1 -- 172.21.15.23:0/2844393400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8910152920 msgr2=0x7f8910152d00 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:45:55.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.514+0000 7f88fd7fa700 1 --2- 172.21.15.23:0/2844393400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8910152920 0x7f8910152d00 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0x7f89100a5e10 tx=0x7f8900006d20).stop 2020-11-20T11:45:55.535 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.531+0000 7f891773c700 1 --2- 172.21.15.23:0/2844393400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8910133cc0 0x7f89101340d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:55.536 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.531+0000 7f891773c700 1 -- 172.21.15.23:0/2844393400 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8910061650 con 0x7f8910133cc0 2020-11-20T11:45:55.536 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.532+0000 7f8915cd9700 1 --2- 172.21.15.23:0/2844393400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8910133cc0 0x7f89101340d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:55.537 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.533+0000 7f8915cd9700 1 -- 172.21.15.23:0/2844393400 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f891005c520 con 0x7f8910133cc0 2020-11-20T11:45:55.538 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.533+0000 7f8915cd9700 1 --2- 172.21.15.23:0/2844393400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8910133cc0 0x7f89101340d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f8900006fd0 tx=0x7f8900015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:55.541 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.537+0000 7f88fd7fa700 1 -- 172.21.15.23:0/2844393400 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8900012e50 con 0x7f8910133cc0 2020-11-20T11:45:55.541 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.537+0000 7f88fd7fa700 1 -- 172.21.15.23:0/2844393400 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4617 (secure 0 0 0) 0x7f89000113b0 con 0x7f8910133cc0 2020-11-20T11:45:55.541 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.537+0000 7f88fd7fa700 1 -- 172.21.15.23:0/2844393400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8910133cc0 msgr2=0x7f89101340d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:55.541 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.537+0000 7f88fd7fa700 1 --2- 172.21.15.23:0/2844393400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8910133cc0 0x7f89101340d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f8900006fd0 tx=0x7f8900015000).stop 2020-11-20T11:45:55.542 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.537+0000 7f88fd7fa700 1 -- 172.21.15.23:0/2844393400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8910133cc0 msgr2=0x7f89101340d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:45:55.542 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.537+0000 7f88fd7fa700 1 --2- 172.21.15.23:0/2844393400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8910133cc0 0x7f89101340d0 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0x7f8900006fd0 tx=0x7f8900015000).stop 2020-11-20T11:45:55.566 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:45:55.567 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:45:55.567 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:45:55.567 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:45:55.567 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 126, 2020-11-20T11:45:55.568 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:45:55.568 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:45:55.568 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:45:55.568 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:45:55.568 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:55.569 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 16, 2020-11-20T11:45:55.569 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:45:55.569 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:45:55.569 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:45:55.569 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:45:55.570 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:45:55.570 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:45:55.570 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:45:55.570 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:45:55.571 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:45:55.571 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:45:55.571 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:45:55.572 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:55.572 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:45:55.572 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:45:55.572 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:45:55.572 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:45:55.573 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:45:55.573 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:45:55.573 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:45:55.573 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:45:55.574 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:45:55.574 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:45:55.574 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:55.574 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:55.574 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:45:55.575 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:45:55.575 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:45:55.575 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:45:55.575 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:45:55.576 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:45:55.576 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:45:55.576 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:45:55.576 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:45:55.577 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:45:55.577 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:45:55.577 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:45:55.577 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:45:55.577 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:45:55.578 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:45:55.578 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:45:55.579 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:45:55.579 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:45:55.579 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:45:55.579 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:45:55.579 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:45:55.580 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:45:55.580 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:45:55.580 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:55.580 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:45:55.581 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:55.581 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:45:55.581 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:55.582 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:45:55.582 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:45:55.582 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:45:55.582 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:45:55.583 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:55.583 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:45:55.583 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:45:55.583 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:55.583 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:55.584 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:55.584 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:45:55.584 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:45:55.584 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:55.584 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:55.585 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:55.585 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:55.585 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:45:55.585 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:45:55.586 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:45:55.586 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:45:55.586 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:45:55.586 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:55.586 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:55.587 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:45:55.587 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:45:55.587 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:45:55.587 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:45:55.587 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:55.588 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:45:55.588 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:45:55.588 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:55.588 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:55.588 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:55.589 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:45:55.589 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:45:55.589 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:55.589 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:55.590 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:55.590 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:55.590 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:45:55.590 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:45:55.590 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:45:55.591 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:45:55.591 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:45:55.591 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:55.591 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:55.591 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:45:55.592 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:45:55.592 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:45:55.592 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:45:55.592 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:55.592 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:45:55.592 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:45:55.593 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:55.593 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:55.593 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:55.593 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:45:55.593 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:45:55.594 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:55.594 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:55.594 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:55.594 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:55.594 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:45:55.595 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:45:55.595 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:45:55.595 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:45:55.595 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:45:55.595 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:55.596 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:55.596 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:55.596 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:45:55.596 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:45:55.596 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:55.597 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:45:55.597 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:45:55.597 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:45:55.597 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:55.598 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:55.598 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:45:55.598 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:55.598 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:45:55.598 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:45:55.598 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:45:55.599 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:55.599 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:55.599 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:45:55.599 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:55.599 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:45:55.600 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:45:55.600 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:45:55.600 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:55.600 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:55.600 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:45:55.601 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:55.601 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:45:55.601 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:45:55.601 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:45:55.601 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:55.602 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:55.602 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:55.602 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:45:55.602 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:45:55.603 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.581+0000 7f88f6ffd700 1 -- 172.21.15.23:0/2844393400 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f88ec056db0 msgr2=0x7f88ec059270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:55.603 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.581+0000 7f88f6ffd700 1 --2- 172.21.15.23:0/2844393400 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f88ec056db0 0x7f88ec059270 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f891009ee30 tx=0x7f8908006cb0).stop 2020-11-20T11:45:55.604 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.581+0000 7f88f6ffd700 1 -- 172.21.15.23:0/2844393400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8910061a10 msgr2=0x7f891009d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:55.604 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.581+0000 7f88f6ffd700 1 --2- 172.21.15.23:0/2844393400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8910061a10 0x7f891009d660 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f890c00c370 tx=0x7f89100a6270).stop 2020-11-20T11:45:55.604 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.582+0000 7f8915cd9700 1 -- 172.21.15.23:0/2844393400 reap_dead start 2020-11-20T11:45:55.604 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.582+0000 7f88f6ffd700 1 -- 172.21.15.23:0/2844393400 shutdown_connections 2020-11-20T11:45:55.604 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.582+0000 7f88f6ffd700 1 -- 172.21.15.23:0/2844393400 >> 172.21.15.23:0/2844393400 conn(0x7f8910060c80 msgr2=0x7f8910135630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:55.605 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.582+0000 7f88f6ffd700 1 -- 172.21.15.23:0/2844393400 shutdown_connections 2020-11-20T11:45:55.605 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:55.582+0000 7f88f6ffd700 1 -- 172.21.15.23:0/2844393400 wait complete. 2020-11-20T11:45:55.624 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:45:56.027 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.023+0000 7f1eef58a700 1 Processor -- start 2020-11-20T11:45:56.028 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.023+0000 7f1eef58a700 1 -- start start 2020-11-20T11:45:56.033 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.024+0000 7f1eef58a700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f1ee0038b70 0x7f1ee003a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:56.034 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.024+0000 7f1eef58a700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1ee0038f50 0x7f1ee0033a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:56.034 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.024+0000 7f1eef58a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1ee00c79a0 0x7f1ee00c7f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:56.034 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.024+0000 7f1eef58a700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ee0039ac0 con 0x7f1ee00c79a0 2020-11-20T11:45:56.034 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.024+0000 7f1eef58a700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ee00370f0 con 0x7f1ee0038b70 2020-11-20T11:45:56.035 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.024+0000 7f1eef58a700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee0039cb0 con 0x7f1ee0038f50 2020-11-20T11:45:56.035 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.025+0000 7f1eedd87700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1ee0038f50 0x7f1ee0033a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:56.035 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.025+0000 7f1eedd87700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1ee0038f50 0x7f1ee0033a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:56256/0 (socket says 172.21.15.23:56256) 2020-11-20T11:45:56.035 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.025+0000 7f1eedd87700 1 -- 172.21.15.23:0/879638732 learned_addr learned my addr 172.21.15.23:0/879638732 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:45:56.035 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.025+0000 7f1eee588700 1 --2- 172.21.15.23:0/879638732 >> v2:172.21.15.155:3300/0 conn(0x7f1ee0038b70 0x7f1ee003a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:56.036 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.025+0000 7f1eedd87700 1 -- 172.21.15.23:0/879638732 >> v2:172.21.15.155:3300/0 conn(0x7f1ee0038b70 msgr2=0x7f1ee003a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:45:56.036 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.025+0000 7f1eedd87700 1 --2- 172.21.15.23:0/879638732 >> v2:172.21.15.155:3300/0 conn(0x7f1ee0038b70 0x7f1ee003a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:56.036 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.025+0000 7f1eedd87700 1 -- 172.21.15.23:0/879638732 >> v2:172.21.15.23:3300/0 conn(0x7f1ee00c79a0 msgr2=0x7f1ee00c7f10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:45:56.036 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.025+0000 7f1eedd87700 1 --2- 172.21.15.23:0/879638732 >> v2:172.21.15.23:3300/0 conn(0x7f1ee00c79a0 0x7f1ee00c7f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:56.036 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.025+0000 7f1eedd87700 1 -- 172.21.15.23:0/879638732 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee0039f50 con 0x7f1ee0038f50 2020-11-20T11:45:56.037 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.025+0000 7f1eedd87700 1 --2- 172.21.15.23:0/879638732 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1ee0038f50 0x7f1ee0033a30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f1ee805e0d0 tx=0x7f1ee805e740).ready entity=mon.2 client_cookie=b671ee4589df38ec server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:56.037 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.026+0000 7f1eed586700 1 -- 172.21.15.23:0/879638732 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee805cf80 con 0x7f1ee0038f50 2020-11-20T11:45:56.037 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.026+0000 7f1eed586700 1 -- 172.21.15.23:0/879638732 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ee8044cc0 con 0x7f1ee0038f50 2020-11-20T11:45:56.037 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.026+0000 7f1eed586700 1 -- 172.21.15.23:0/879638732 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee805cf80 con 0x7f1ee0038f50 2020-11-20T11:45:56.038 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.030+0000 7f1eef58a700 1 -- 172.21.15.23:0/879638732 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1ee0038f50 msgr2=0x7f1ee0033a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:56.038 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.031+0000 7f1eef58a700 1 --2- 172.21.15.23:0/879638732 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1ee0038f50 0x7f1ee0033a30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f1ee805e0d0 tx=0x7f1ee805e740).stop 2020-11-20T11:45:56.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.043+0000 7f1eef58a700 1 -- 172.21.15.23:0/879638732 shutdown_connections 2020-11-20T11:45:56.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.043+0000 7f1eef58a700 1 --2- 172.21.15.23:0/879638732 >> v2:172.21.15.23:3300/0 conn(0x7f1ee00c79a0 0x7f1ee00c7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:56.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.043+0000 7f1eef58a700 1 --2- 172.21.15.23:0/879638732 >> v2:172.21.15.155:3300/0 conn(0x7f1ee0038b70 0x7f1ee003a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:56.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.043+0000 7f1eef58a700 1 --2- 172.21.15.23:0/879638732 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1ee0038f50 0x7f1ee0033a30 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:56.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.043+0000 7f1eef58a700 1 -- 172.21.15.23:0/879638732 >> 172.21.15.23:0/879638732 conn(0x7f1ee000bcc0 msgr2=0x7f1ee0006a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:56.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.044+0000 7f1eef58a700 1 -- 172.21.15.23:0/879638732 shutdown_connections 2020-11-20T11:45:56.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.044+0000 7f1eef58a700 1 -- 172.21.15.23:0/879638732 wait complete. 2020-11-20T11:45:56.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.046+0000 7f1eef58a700 1 Processor -- start 2020-11-20T11:45:56.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.046+0000 7f1eef58a700 1 -- start start 2020-11-20T11:45:56.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.047+0000 7f1eef58a700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1ee0038b70 0x7f1ee0032560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:56.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.047+0000 7f1eef58a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ee0038f50 0x7f1ee00c3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:56.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.047+0000 7f1eef58a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ee00c79a0 0x7f1ee00de920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:56.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.047+0000 7f1eef58a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee00de450 con 0x7f1ee0038f50 2020-11-20T11:45:56.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.047+0000 7f1eef58a700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee00c4a80 con 0x7f1ee00c79a0 2020-11-20T11:45:56.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.047+0000 7f1eef58a700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee00c4d20 con 0x7f1ee0038b70 2020-11-20T11:45:56.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.047+0000 7f1eedd87700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ee0038f50 0x7f1ee00c3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:56.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.047+0000 7f1eedd87700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ee0038f50 0x7f1ee00c3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46022/0 (socket says 172.21.15.23:46022) 2020-11-20T11:45:56.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.047+0000 7f1eedd87700 1 -- 172.21.15.23:0/1996335109 learned_addr learned my addr 172.21.15.23:0/1996335109 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:45:56.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.048+0000 7f1eedd87700 1 -- 172.21.15.23:0/1996335109 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1ee0038b70 msgr2=0x7f1ee0032560 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2020-11-20T11:45:56.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.048+0000 7f1eedd87700 1 --2- 172.21.15.23:0/1996335109 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1ee0038b70 0x7f1ee0032560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:56.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.048+0000 7f1eedd87700 1 -- 172.21.15.23:0/1996335109 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ee00c79a0 msgr2=0x7f1ee00de920 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2020-11-20T11:45:56.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.048+0000 7f1eedd87700 1 --2- 172.21.15.23:0/1996335109 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ee00c79a0 0x7f1ee00de920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:56.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.048+0000 7f1eedd87700 1 -- 172.21.15.23:0/1996335109 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee00c34f0 con 0x7f1ee0038f50 2020-11-20T11:45:56.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.048+0000 7f1eedd87700 1 --2- 172.21.15.23:0/1996335109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ee0038f50 0x7f1ee00c3990 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f1ee8059ed0 tx=0x7f1ee8065070).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:56.055 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.048+0000 7f1ed67fc700 1 -- 172.21.15.23:0/1996335109 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee805cf80 con 0x7f1ee0038f50 2020-11-20T11:45:56.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.048+0000 7f1eef58a700 1 -- 172.21.15.23:0/1996335109 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ee00cc110 con 0x7f1ee0038f50 2020-11-20T11:45:56.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.048+0000 7f1eef58a700 1 -- 172.21.15.23:0/1996335109 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee00c47b0 con 0x7f1ee0038f50 2020-11-20T11:45:56.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.049+0000 7f1ed67fc700 1 -- 172.21.15.23:0/1996335109 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ee808a480 con 0x7f1ee0038f50 2020-11-20T11:45:56.056 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.049+0000 7f1ed67fc700 1 -- 172.21.15.23:0/1996335109 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee8056ea0 con 0x7f1ee0038f50 2020-11-20T11:45:56.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.051+0000 7f1ed67fc700 1 -- 172.21.15.23:0/1996335109 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1ee808aa50 con 0x7f1ee0038f50 2020-11-20T11:45:56.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.051+0000 7f1ed67fc700 1 --2- 172.21.15.23:0/1996335109 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1ec0056de0 0x7f1ec00592a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:56.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.052+0000 7f1ed67fc700 1 -- 172.21.15.23:0/1996335109 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f1ee80e0e70 con 0x7f1ee0038f50 2020-11-20T11:45:56.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.054+0000 7f1eef58a700 1 --2- 172.21.15.23:0/1996335109 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ee00cdb20 0x7f1ee0003100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:56.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.054+0000 7f1eef58a700 1 -- 172.21.15.23:0/1996335109 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee00035f0 con 0x7f1ee00cdb20 2020-11-20T11:45:56.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.054+0000 7f1eeed89700 1 --2- 172.21.15.23:0/1996335109 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ee00cdb20 0x7f1ee0003100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:56.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.055+0000 7f1eeed89700 1 -- 172.21.15.23:0/1996335109 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1ee00c3680 con 0x7f1ee00cdb20 2020-11-20T11:45:56.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.055+0000 7f1eeed89700 1 --2- 172.21.15.23:0/1996335109 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ee00cdb20 0x7f1ee0003100 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f1ee00c7e10 tx=0x7f1ed8006be0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:56.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.055+0000 7f1ed67fc700 1 -- 172.21.15.23:0/1996335109 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed8009e20 con 0x7f1ee00cdb20 2020-11-20T11:45:56.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.055+0000 7f1ed67fc700 1 -- 172.21.15.23:0/1996335109 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f1ed8002320 con 0x7f1ee00cdb20 2020-11-20T11:45:56.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.055+0000 7f1ed67fc700 1 -- 172.21.15.23:0/1996335109 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ee00cdb20 msgr2=0x7f1ee0003100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:56.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.055+0000 7f1ed67fc700 1 --2- 172.21.15.23:0/1996335109 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ee00cdb20 0x7f1ee0003100 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f1ee00c7e10 tx=0x7f1ed8006be0).stop 2020-11-20T11:45:56.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.055+0000 7f1ed67fc700 1 -- 172.21.15.23:0/1996335109 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ee00cdb20 msgr2=0x7f1ee0003100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:45:56.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.055+0000 7f1ed67fc700 1 --2- 172.21.15.23:0/1996335109 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ee00cdb20 0x7f1ee0003100 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:56.072 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.068+0000 7f1ec7fff700 1 --2- 172.21.15.23:0/1996335109 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ebc002900 0x7f1ebc004da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:56.073 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.068+0000 7f1ec7fff700 1 -- 172.21.15.23:0/1996335109 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc007a40 con 0x7f1ebc002900 2020-11-20T11:45:56.073 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.068+0000 7f1eeed89700 1 --2- 172.21.15.23:0/1996335109 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ebc002900 0x7f1ebc004da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:56.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.069+0000 7f1eeed89700 1 -- 172.21.15.23:0/1996335109 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1ebc007c00 con 0x7f1ebc002900 2020-11-20T11:45:56.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.069+0000 7f1eee588700 1 --2- 172.21.15.23:0/1996335109 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1ec0056de0 0x7f1ec00592a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:56.074 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.070+0000 7f1eeed89700 1 --2- 172.21.15.23:0/1996335109 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ebc002900 0x7f1ebc004da0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f1ee00c7e10 tx=0x7f1ed8015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:56.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.070+0000 7f1ed67fc700 1 -- 172.21.15.23:0/1996335109 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed8012e00 con 0x7f1ebc002900 2020-11-20T11:45:56.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.070+0000 7f1ed67fc700 1 -- 172.21.15.23:0/1996335109 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4785 (secure 0 0 0) 0x7f1ed8002110 con 0x7f1ebc002900 2020-11-20T11:45:56.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.070+0000 7f1ed67fc700 1 -- 172.21.15.23:0/1996335109 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ebc002900 msgr2=0x7f1ebc004da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:56.075 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.070+0000 7f1ed67fc700 1 --2- 172.21.15.23:0/1996335109 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ebc002900 0x7f1ebc004da0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f1ee00c7e10 tx=0x7f1ed8015000).stop 2020-11-20T11:45:56.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.070+0000 7f1ed67fc700 1 -- 172.21.15.23:0/1996335109 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ebc002900 msgr2=0x7f1ebc004da0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:45:56.076 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.070+0000 7f1ed67fc700 1 --2- 172.21.15.23:0/1996335109 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1ebc002900 0x7f1ebc004da0 secure :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0x7f1ee00c7e10 tx=0x7f1ed8015000).stop 2020-11-20T11:45:56.078 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:45:56.080 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:45:56.080 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:45:56.080 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:45:56.080 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 126, 2020-11-20T11:45:56.080 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:45:56.081 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:45:56.082 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:45:56.082 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:45:56.082 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:56.082 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 16, 2020-11-20T11:45:56.083 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:45:56.083 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:45:56.083 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:45:56.083 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:45:56.084 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:45:56.084 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:45:56.084 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:45:56.084 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:45:56.084 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:45:56.084 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:45:56.085 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:45:56.085 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:56.085 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:45:56.085 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:45:56.086 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:45:56.086 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:45:56.086 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:45:56.086 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:45:56.086 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:45:56.087 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:45:56.087 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:45:56.087 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:45:56.087 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:56.087 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:56.087 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:45:56.088 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:45:56.088 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:45:56.088 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:45:56.088 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:45:56.089 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:45:56.089 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:45:56.089 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:45:56.089 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:45:56.089 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:45:56.090 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:45:56.090 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:45:56.090 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:45:56.090 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:45:56.090 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:45:56.091 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:45:56.091 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:45:56.091 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:45:56.091 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:45:56.091 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:45:56.092 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:45:56.092 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:45:56.092 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:45:56.092 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:56.092 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:45:56.093 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:56.093 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:45:56.093 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:56.093 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:45:56.093 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:45:56.094 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:45:56.094 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:45:56.094 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:56.094 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:45:56.094 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:45:56.095 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:56.095 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:56.095 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:56.095 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:45:56.095 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:45:56.096 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:56.096 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:56.096 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:56.096 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:56.096 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:45:56.096 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:45:56.097 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:45:56.097 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:45:56.097 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:45:56.097 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:56.097 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:56.098 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:45:56.098 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:45:56.098 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:45:56.098 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:45:56.098 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:56.099 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:45:56.099 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:45:56.099 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:56.099 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:56.099 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:56.100 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:45:56.100 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:45:56.100 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:56.100 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:56.100 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:56.101 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:56.101 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:45:56.101 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:45:56.101 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:45:56.101 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:45:56.102 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:45:56.102 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:56.102 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:56.102 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:45:56.102 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:45:56.103 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:45:56.103 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:45:56.103 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:56.103 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:45:56.103 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:45:56.104 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:56.104 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:56.104 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:56.104 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:45:56.104 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:45:56.104 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:56.105 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:56.105 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:56.105 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:56.105 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:45:56.106 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:45:56.106 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:45:56.106 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:45:56.106 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:45:56.106 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:56.106 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:56.107 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:56.107 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:45:56.107 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:45:56.107 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:56.108 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:45:56.108 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:45:56.108 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:45:56.108 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:56.108 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:56.109 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:45:56.109 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:56.109 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:45:56.109 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:45:56.109 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:45:56.110 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:56.110 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:56.110 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:45:56.110 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:56.110 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:45:56.111 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:45:56.111 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:45:56.111 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:56.111 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:56.111 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:45:56.112 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:56.112 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:45:56.112 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:45:56.112 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:45:56.112 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:56.113 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:56.113 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:45:56.113 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:56.113 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:45:56.113 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:45:56.114 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:45:56.114 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:56.114 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:56.114 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:56.114 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:45:56.115 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:45:56.115 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.074+0000 7f1ec7fff700 1 -- 172.21.15.23:0/1996335109 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1ec0056de0 msgr2=0x7f1ec00592a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:56.116 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.080+0000 7f1eee588700 1 --2- 172.21.15.23:0/1996335109 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1ec0056de0 0x7f1ec00592a0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f1ee4006fd0 tx=0x7f1ee4006e30).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:56.116 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.080+0000 7f1ec7fff700 1 --2- 172.21.15.23:0/1996335109 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1ec0056de0 0x7f1ec00592a0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f1ee4006fd0 tx=0x7f1ee4006e30).stop 2020-11-20T11:45:56.117 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.080+0000 7f1ec7fff700 1 -- 172.21.15.23:0/1996335109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ee0038f50 msgr2=0x7f1ee00c3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:56.117 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.080+0000 7f1ec7fff700 1 --2- 172.21.15.23:0/1996335109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ee0038f50 0x7f1ee00c3990 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f1ee8059ed0 tx=0x7f1ee8065070).stop 2020-11-20T11:45:56.117 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.080+0000 7f1eeed89700 1 -- 172.21.15.23:0/1996335109 reap_dead start 2020-11-20T11:45:56.117 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.092+0000 7f1ec7fff700 1 -- 172.21.15.23:0/1996335109 shutdown_connections 2020-11-20T11:45:56.118 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.092+0000 7f1ec7fff700 1 --2- 172.21.15.23:0/1996335109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1ee0038f50 0x7f1ee00c3990 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:56.118 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.092+0000 7f1ec7fff700 1 -- 172.21.15.23:0/1996335109 >> 172.21.15.23:0/1996335109 conn(0x7f1ee000bcc0 msgr2=0x7f1ee0006930 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:56.118 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.093+0000 7f1ec7fff700 1 -- 172.21.15.23:0/1996335109 shutdown_connections 2020-11-20T11:45:56.118 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.093+0000 7f1ec7fff700 1 -- 172.21.15.23:0/1996335109 wait complete. 2020-11-20T11:45:56.119 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:45:56.287 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.282+0000 7ff6c5e5b700 1 Processor -- start 2020-11-20T11:45:56.287 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.283+0000 7ff6c5e5b700 1 -- start start 2020-11-20T11:45:56.289 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.283+0000 7ff6c5e5b700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ff6c00ab2a0 0x7ff6c013e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:56.289 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.283+0000 7ff6c5e5b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff6c00ad8d0 0x7ff6c00a8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:56.289 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.283+0000 7ff6c5e5b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6c00a4510 0x7ff6c00a4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:56.289 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.283+0000 7ff6c5e5b700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6c013f4a0 con 0x7ff6c00ad8d0 2020-11-20T11:45:56.289 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.283+0000 7ff6c5e5b700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6c00aa0e0 con 0x7ff6c00ab2a0 2020-11-20T11:45:56.290 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.283+0000 7ff6c5e5b700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c00b1a20 con 0x7ff6c00a4510 2020-11-20T11:45:56.290 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.284+0000 7ff6b7fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff6c00ad8d0 0x7ff6c00a8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:56.290 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.284+0000 7ff6b7fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7ff6c00ad8d0 0x7ff6c00a8fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46032/0 (socket says 172.21.15.23:46032) 2020-11-20T11:45:56.290 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.284+0000 7ff6b7fff700 1 -- 172.21.15.23:0/2490825387 learned_addr learned my addr 172.21.15.23:0/2490825387 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:45:56.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.284+0000 7ff6bf7fe700 1 --2- 172.21.15.23:0/2490825387 >> v2:172.21.15.155:3300/0 conn(0x7ff6c00ab2a0 0x7ff6c013e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:56.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.284+0000 7ff6bffff700 1 --2- 172.21.15.23:0/2490825387 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6c00a4510 0x7ff6c00a4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:56.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.284+0000 7ff6b7fff700 1 -- 172.21.15.23:0/2490825387 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6c00a4510 msgr2=0x7ff6c00a4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:45:56.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.284+0000 7ff6b7fff700 1 --2- 172.21.15.23:0/2490825387 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6c00a4510 0x7ff6c00a4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:56.291 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.284+0000 7ff6b7fff700 1 -- 172.21.15.23:0/2490825387 >> v2:172.21.15.155:3300/0 conn(0x7ff6c00ab2a0 msgr2=0x7ff6c013e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:45:56.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.284+0000 7ff6b7fff700 1 --2- 172.21.15.23:0/2490825387 >> v2:172.21.15.155:3300/0 conn(0x7ff6c00ab2a0 0x7ff6c013e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:56.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.284+0000 7ff6b7fff700 1 -- 172.21.15.23:0/2490825387 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c00a8b90 con 0x7ff6c00ad8d0 2020-11-20T11:45:56.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.284+0000 7ff6b7fff700 1 --2- 172.21.15.23:0/2490825387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6c00ad8d0 0x7ff6c00a8fc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7ff6c0068850 tx=0x7ff6a8006ef0).ready entity=mon.0 client_cookie=22afb1a7eeb2b7cb server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:56.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.285+0000 7ff6beffd700 1 -- 172.21.15.23:0/2490825387 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a800b680 con 0x7ff6c00ad8d0 2020-11-20T11:45:56.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.285+0000 7ff6beffd700 1 -- 172.21.15.23:0/2490825387 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6a8000e90 con 0x7ff6c00ad8d0 2020-11-20T11:45:56.292 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.285+0000 7ff6beffd700 1 -- 172.21.15.23:0/2490825387 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a800b9f0 con 0x7ff6c00ad8d0 2020-11-20T11:45:56.295 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.285+0000 7ff6c5e5b700 1 -- 172.21.15.23:0/2490825387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6c00ad8d0 msgr2=0x7ff6c00a8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:56.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.285+0000 7ff6c5e5b700 1 --2- 172.21.15.23:0/2490825387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6c00ad8d0 0x7ff6c00a8fc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7ff6c0068850 tx=0x7ff6a8006ef0).stop 2020-11-20T11:45:56.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.285+0000 7ff6c5e5b700 1 -- 172.21.15.23:0/2490825387 shutdown_connections 2020-11-20T11:45:56.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.285+0000 7ff6c5e5b700 1 --2- 172.21.15.23:0/2490825387 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6c00a4510 0x7ff6c00a4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:56.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.285+0000 7ff6c5e5b700 1 --2- 172.21.15.23:0/2490825387 >> v2:172.21.15.155:3300/0 conn(0x7ff6c00ab2a0 0x7ff6c013e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:56.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.285+0000 7ff6c5e5b700 1 --2- 172.21.15.23:0/2490825387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6c00ad8d0 0x7ff6c00a8fc0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:56.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.285+0000 7ff6c5e5b700 1 -- 172.21.15.23:0/2490825387 >> 172.21.15.23:0/2490825387 conn(0x7ff6c00683b0 msgr2=0x7ff6c00b1310 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:56.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.286+0000 7ff6c5e5b700 1 -- 172.21.15.23:0/2490825387 shutdown_connections 2020-11-20T11:45:56.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.286+0000 7ff6c5e5b700 1 -- 172.21.15.23:0/2490825387 wait complete. 2020-11-20T11:45:56.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.286+0000 7ff6c5e5b700 1 Processor -- start 2020-11-20T11:45:56.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.287+0000 7ff6c5e5b700 1 -- start start 2020-11-20T11:45:56.298 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.287+0000 7ff6c5e5b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6c00a4510 0x7ff6c00b10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:56.298 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.287+0000 7ff6c5e5b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6c00ab2a0 0x7ff6c009fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:56.298 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.287+0000 7ff6c5e5b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6c00ad8d0 0x7ff6c00a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:56.298 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.287+0000 7ff6c5e5b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c009f400 con 0x7ff6c00a4510 2020-11-20T11:45:56.298 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.287+0000 7ff6c5e5b700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c009f6a0 con 0x7ff6c00ad8d0 2020-11-20T11:45:56.299 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.287+0000 7ff6c5e5b700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c00a0030 con 0x7ff6c00ab2a0 2020-11-20T11:45:56.299 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.287+0000 7ff6bf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6c00a4510 0x7ff6c00b10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:56.299 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.287+0000 7ff6bf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6c00a4510 0x7ff6c00b10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46038/0 (socket says 172.21.15.23:46038) 2020-11-20T11:45:56.299 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.287+0000 7ff6bf7fe700 1 -- 172.21.15.23:0/3359722467 learned_addr learned my addr 172.21.15.23:0/3359722467 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:45:56.299 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.287+0000 7ff6bffff700 1 --2- 172.21.15.23:0/3359722467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6c00ad8d0 0x7ff6c00a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:56.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.287+0000 7ff6b7fff700 1 --2- 172.21.15.23:0/3359722467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6c00ab2a0 0x7ff6c009fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:56.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.288+0000 7ff6bffff700 1 -- 172.21.15.23:0/3359722467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6c00ab2a0 msgr2=0x7ff6c009fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:56.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.288+0000 7ff6bffff700 1 --2- 172.21.15.23:0/3359722467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6c00ab2a0 0x7ff6c009fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:56.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.288+0000 7ff6bffff700 1 -- 172.21.15.23:0/3359722467 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6c00a4510 msgr2=0x7ff6c00b10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:56.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.288+0000 7ff6bffff700 1 --2- 172.21.15.23:0/3359722467 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6c00a4510 0x7ff6c00b10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:56.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.288+0000 7ff6bffff700 1 -- 172.21.15.23:0/3359722467 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c00a2890 con 0x7ff6c00ad8d0 2020-11-20T11:45:56.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.288+0000 7ff6bf7fe700 1 --2- 172.21.15.23:0/3359722467 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6c00a4510 0x7ff6c00b10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:45:56.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.288+0000 7ff6bffff700 1 --2- 172.21.15.23:0/3359722467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6c00ad8d0 0x7ff6c00a23a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7ff6b800ef50 tx=0x7ff6b8009220).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:56.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.288+0000 7ff6b77fe700 1 -- 172.21.15.23:0/3359722467 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b800cab0 con 0x7ff6c00ad8d0 2020-11-20T11:45:56.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.288+0000 7ff6b77fe700 1 -- 172.21.15.23:0/3359722467 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6b800cc10 con 0x7ff6c00ad8d0 2020-11-20T11:45:56.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.288+0000 7ff6b77fe700 1 -- 172.21.15.23:0/3359722467 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b80123f0 con 0x7ff6c00ad8d0 2020-11-20T11:45:56.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.288+0000 7ff6c5e5b700 1 -- 172.21.15.23:0/3359722467 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6c009e730 con 0x7ff6c00ad8d0 2020-11-20T11:45:56.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.288+0000 7ff6c5e5b700 1 -- 172.21.15.23:0/3359722467 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6c014dc80 con 0x7ff6c00ad8d0 2020-11-20T11:45:56.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.289+0000 7ff6c5e5b700 1 --2- 172.21.15.23:0/3359722467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6c00601b0 0x7ff6c014e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:56.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.289+0000 7ff6c5e5b700 1 -- 172.21.15.23:0/3359722467 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c005c260 con 0x7ff6c00601b0 2020-11-20T11:45:56.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.290+0000 7ff6b77fe700 1 -- 172.21.15.23:0/3359722467 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ff6b8022070 con 0x7ff6c00ad8d0 2020-11-20T11:45:56.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.290+0000 7ff6b77fe700 1 --2- 172.21.15.23:0/3359722467 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff69c056d60 0x7ff69c059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:56.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.290+0000 7ff6b77fe700 1 -- 172.21.15.23:0/3359722467 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7ff6b8070830 con 0x7ff6c00ad8d0 2020-11-20T11:45:56.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.291+0000 7ff6b7fff700 1 --2- 172.21.15.23:0/3359722467 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff69c056d60 0x7ff69c059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:56.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.291+0000 7ff6b7fff700 1 --2- 172.21.15.23:0/3359722467 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff69c056d60 0x7ff69c059220 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7ff6a8006f30 tx=0x7ff6a8016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:56.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.293+0000 7ff6bf7fe700 1 --2- 172.21.15.23:0/3359722467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6c00601b0 0x7ff6c014e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:56.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.293+0000 7ff6bf7fe700 1 -- 172.21.15.23:0/3359722467 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff6c00a2ba0 con 0x7ff6c00601b0 2020-11-20T11:45:56.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.293+0000 7ff6bf7fe700 1 --2- 172.21.15.23:0/3359722467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6c00601b0 0x7ff6c014e210 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7ff6b000f740 tx=0x7ff6b0006e30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:56.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.293+0000 7ff6b77fe700 1 -- 172.21.15.23:0/3359722467 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b0014650 con 0x7ff6c00601b0 2020-11-20T11:45:56.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.294+0000 7ff6b77fe700 1 -- 172.21.15.23:0/3359722467 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7ff6b0000e90 con 0x7ff6c00601b0 2020-11-20T11:45:56.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.294+0000 7ff6b77fe700 1 -- 172.21.15.23:0/3359722467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6c00601b0 msgr2=0x7ff6c014e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:56.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.294+0000 7ff6b77fe700 1 --2- 172.21.15.23:0/3359722467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6c00601b0 0x7ff6c014e210 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7ff6b000f740 tx=0x7ff6b0006e30).stop 2020-11-20T11:45:56.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.294+0000 7ff6b77fe700 1 -- 172.21.15.23:0/3359722467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6c00601b0 msgr2=0x7ff6c014e210 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:45:56.308 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.294+0000 7ff6b77fe700 1 --2- 172.21.15.23:0/3359722467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6c00601b0 0x7ff6c014e210 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:56.308 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.300+0000 7ff6c5e5b700 1 --2- 172.21.15.23:0/3359722467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6c014e700 0x7ff6c009d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:56.308 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.300+0000 7ff6c5e5b700 1 -- 172.21.15.23:0/3359722467 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c00af140 con 0x7ff6c014e700 2020-11-20T11:45:56.308 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.301+0000 7ff6bf7fe700 1 --2- 172.21.15.23:0/3359722467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6c014e700 0x7ff6c009d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:56.308 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.301+0000 7ff6bf7fe700 1 -- 172.21.15.23:0/3359722467 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ff6c0060640 con 0x7ff6c014e700 2020-11-20T11:45:56.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.301+0000 7ff6bf7fe700 1 --2- 172.21.15.23:0/3359722467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6c014e700 0x7ff6c009d4a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7ff6b000f740 tx=0x7ff6b001a000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:56.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.301+0000 7ff6b77fe700 1 -- 172.21.15.23:0/3359722467 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b0022dc0 con 0x7ff6c014e700 2020-11-20T11:45:56.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.301+0000 7ff6b77fe700 1 -- 172.21.15.23:0/3359722467 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4275 (secure 0 0 0) 0x7ff6b0027070 con 0x7ff6c014e700 2020-11-20T11:45:56.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.301+0000 7ff6b77fe700 1 -- 172.21.15.23:0/3359722467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6c014e700 msgr2=0x7ff6c009d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:56.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.301+0000 7ff6b77fe700 1 --2- 172.21.15.23:0/3359722467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6c014e700 0x7ff6c009d4a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7ff6b000f740 tx=0x7ff6b001a000).stop 2020-11-20T11:45:56.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.301+0000 7ff6b77fe700 1 -- 172.21.15.23:0/3359722467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6c014e700 msgr2=0x7ff6c009d4a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:45:56.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.301+0000 7ff6b77fe700 1 --2- 172.21.15.23:0/3359722467 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ff6c014e700 0x7ff6c009d4a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:56.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.302+0000 7ff6c5e5b700 1 -- 172.21.15.23:0/3359722467 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff69c056d60 msgr2=0x7ff69c059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:56.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.302+0000 7ff6c5e5b700 1 --2- 172.21.15.23:0/3359722467 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ff69c056d60 0x7ff69c059220 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7ff6a8006f30 tx=0x7ff6a8016000).stop 2020-11-20T11:45:56.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.302+0000 7ff6c5e5b700 1 -- 172.21.15.23:0/3359722467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6c00ad8d0 msgr2=0x7ff6c00a23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:56.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.302+0000 7ff6c5e5b700 1 --2- 172.21.15.23:0/3359722467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6c00ad8d0 0x7ff6c00a23a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7ff6b800ef50 tx=0x7ff6b8009220).stop 2020-11-20T11:45:56.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.302+0000 7ff6bffff700 1 -- 172.21.15.23:0/3359722467 reap_dead start 2020-11-20T11:45:56.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.302+0000 7ff6c5e5b700 1 -- 172.21.15.23:0/3359722467 shutdown_connections 2020-11-20T11:45:56.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.302+0000 7ff6c5e5b700 1 --2- 172.21.15.23:0/3359722467 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ff6c00ad8d0 0x7ff6c00a23a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:56.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.302+0000 7ff6c5e5b700 1 -- 172.21.15.23:0/3359722467 >> 172.21.15.23:0/3359722467 conn(0x7ff6c00683b0 msgr2=0x7ff6c00acb10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:56.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.302+0000 7ff6c5e5b700 1 -- 172.21.15.23:0/3359722467 shutdown_connections 2020-11-20T11:45:56.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.302+0000 7ff6c5e5b700 1 -- 172.21.15.23:0/3359722467 wait complete. 2020-11-20T11:45:56.312 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:45:56.312 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:45:56.313 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:45:56.313 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:45:56.313 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 126, 2020-11-20T11:45:56.313 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:45:56.313 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:45:56.314 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:45:56.314 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:45:56.314 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:56.314 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 16, 2020-11-20T11:45:56.314 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:45:56.315 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:45:56.315 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:45:56.315 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:45:56.315 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:45:56.315 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:45:56.316 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:45:56.316 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:45:56.316 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:45:56.316 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:45:56.316 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:45:56.317 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:56.317 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:45:56.317 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:45:56.317 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:45:56.317 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:45:56.318 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:45:56.318 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:45:56.318 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:45:56.318 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:45:56.318 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:45:56.319 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:45:56.319 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:56.319 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:56.319 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:45:56.319 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:45:56.319 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:45:56.320 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:45:56.320 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:45:56.320 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:45:56.320 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:45:56.320 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:45:56.321 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:45:56.321 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:45:56.321 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:45:56.321 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:45:56.321 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:45:56.322 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:45:56.322 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:45:56.322 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:45:56.322 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:45:56.322 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:45:56.323 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:45:56.323 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:45:56.323 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:45:56.323 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:45:56.323 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:45:56.324 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:56.324 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:45:56.324 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:56.324 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:45:56.324 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:56.325 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:45:56.325 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:45:56.325 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:45:56.325 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:45:56.325 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:56.326 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:45:56.326 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:45:56.326 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:56.326 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:56.326 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:56.327 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:45:56.327 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:45:56.327 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:56.327 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:56.327 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:56.328 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:56.328 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:45:56.328 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:45:56.328 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:45:56.328 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:45:56.329 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:45:56.329 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:56.329 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:56.329 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:45:56.329 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:45:56.330 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:45:56.330 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:45:56.330 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:56.330 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:45:56.331 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:45:56.331 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:56.331 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:56.331 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:56.331 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:45:56.332 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:45:56.332 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:56.332 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:56.332 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:56.332 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:56.333 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:45:56.333 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:45:56.333 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:45:56.333 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:45:56.333 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:45:56.334 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:56.334 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:56.334 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:45:56.334 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:45:56.334 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:45:56.335 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:45:56.335 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:56.335 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:45:56.335 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:45:56.335 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:56.336 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:56.336 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:56.336 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:45:56.336 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:45:56.336 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:45:56.337 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:56.337 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:56.337 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:56.337 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:45:56.337 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:45:56.338 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:45:56.338 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:45:56.338 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:45:56.338 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:56.338 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:56.339 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:56.339 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:45:56.339 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:45:56.339 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:56.340 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:45:56.340 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:45:56.340 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:45:56.340 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:56.340 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:45:56.341 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:45:56.341 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:45:56.341 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:45:56.341 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:45:56.342 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:45:56.342 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:45:56.342 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:45:56.342 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:45:56.342 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:45:56.343 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:45:56.344 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2020-11-20T11:45:56.344 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2020-11-20T11:45:56.344 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2020-11-20T11:45:56.345 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2020-11-20T11:45:56.345 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2020-11-20T11:45:56.345 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T11:45:56.360 INFO:tasks.ceph.mon.b:Stopped 2020-11-20T11:45:56.360 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2020-11-20T11:45:56.360 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:45:56.498 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.494+0000 7f1cf8969700 1 Processor -- start 2020-11-20T11:45:56.499 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.494+0000 7f1cf8969700 1 -- start start 2020-11-20T11:45:56.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.495+0000 7f1cf8969700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1cf40aae90 0x7f1cf41381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:56.501 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.495+0000 7f1cf8969700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1cf413aeb0 0x7f1cf40a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:56.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.495+0000 7f1cf8969700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf413e600 con 0x7f1cf40aaab0 2020-11-20T11:45:56.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.495+0000 7f1cf8969700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cf413e8a0 con 0x7f1cf413aeb0 2020-11-20T11:45:56.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.495+0000 7f1cf8969700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf40ab6d0 con 0x7f1cf40aae90 2020-11-20T11:45:56.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.495+0000 7f1cf259c700 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f1cf40aaab0 legacy=0x7f1cf413ed40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:45:56.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.495+0000 7f1cf2d9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1cf413aeb0 0x7f1cf40a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:56.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.495+0000 7f1cf2d9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f1cf413aeb0 0x7f1cf40a8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46076/0 (socket says 172.21.15.23:46076) 2020-11-20T11:45:56.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.495+0000 7f1cf2d9d700 1 -- 172.21.15.23:0/1127172987 learned_addr learned my addr 172.21.15.23:0/1127172987 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:45:56.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.495+0000 7f1cf1d9b700 1 --2- 172.21.15.23:0/1127172987 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1cf40aae90 0x7f1cf41381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:56.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.496+0000 7f1cf2d9d700 1 -- 172.21.15.23:0/1127172987 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1cf40aae90 msgr2=0x7f1cf41381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:45:56.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.496+0000 7f1cf2d9d700 1 --2- 172.21.15.23:0/1127172987 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1cf40aae90 0x7f1cf41381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:56.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.496+0000 7f1cf2d9d700 1 -- 172.21.15.23:0/1127172987 >> v1:172.21.15.155:6789/0 conn(0x7f1cf40aaab0 legacy=0x7f1cf413ed40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:45:56.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.496+0000 7f1cf2d9d700 1 -- 172.21.15.23:0/1127172987 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf4138690 con 0x7f1cf413aeb0 2020-11-20T11:45:56.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.496+0000 7f1cf2d9d700 1 --2- 172.21.15.23:0/1127172987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1cf413aeb0 0x7f1cf40a8c60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f1cec009d90 tx=0x7f1cec0092d0).ready entity=mon.0 client_cookie=1eafc2cecf7e9df7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:56.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.496+0000 7f1cf159a700 1 -- 172.21.15.23:0/1127172987 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec00bbb0 con 0x7f1cf413aeb0 2020-11-20T11:45:56.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.496+0000 7f1cf159a700 1 -- 172.21.15.23:0/1127172987 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cec00bd10 con 0x7f1cf413aeb0 2020-11-20T11:45:56.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.496+0000 7f1cf159a700 1 -- 172.21.15.23:0/1127172987 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec011610 con 0x7f1cf413aeb0 2020-11-20T11:45:56.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.496+0000 7f1cf8969700 1 -- 172.21.15.23:0/1127172987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1cf413aeb0 msgr2=0x7f1cf40a8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:56.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.496+0000 7f1cf8969700 1 --2- 172.21.15.23:0/1127172987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1cf413aeb0 0x7f1cf40a8c60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f1cec009d90 tx=0x7f1cec0092d0).stop 2020-11-20T11:45:56.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.496+0000 7f1cf8969700 1 -- 172.21.15.23:0/1127172987 shutdown_connections 2020-11-20T11:45:56.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.496+0000 7f1cf8969700 1 --2- 172.21.15.23:0/1127172987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1cf413aeb0 0x7f1cf40a8c60 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:56.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.496+0000 7f1cf8969700 1 --2- 172.21.15.23:0/1127172987 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1cf40aae90 0x7f1cf41381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:56.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.496+0000 7f1cf8969700 1 -- 172.21.15.23:0/1127172987 >> 172.21.15.23:0/1127172987 conn(0x7f1cf40678e0 msgr2=0x7f1cf4067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:56.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.497+0000 7f1cf8969700 1 -- 172.21.15.23:0/1127172987 shutdown_connections 2020-11-20T11:45:56.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.497+0000 7f1cf8969700 1 -- 172.21.15.23:0/1127172987 wait complete. 2020-11-20T11:45:56.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.497+0000 7f1cf8969700 1 Processor -- start 2020-11-20T11:45:56.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.498+0000 7f1cf8969700 1 -- start start 2020-11-20T11:45:56.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.498+0000 7f1cf8969700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1cf40aaab0 0x7f1cf40a6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:56.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.498+0000 7f1cf8969700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1cf40aae90 0x7f1cf413db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:56.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.498+0000 7f1cf8969700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1cf414d660 0x7f1cf40a7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:56.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.498+0000 7f1cf8969700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf4132cd0 con 0x7f1cf40aae90 2020-11-20T11:45:56.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.498+0000 7f1cf8969700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf414a4c0 con 0x7f1cf40aaab0 2020-11-20T11:45:56.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.498+0000 7f1cf8969700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf414a760 con 0x7f1cf414d660 2020-11-20T11:45:56.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.498+0000 7f1cf1d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1cf40aae90 0x7f1cf413db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:56.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.498+0000 7f1cf259c700 1 -- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1cf40aaab0 msgr2=0x7f1cf40a6580 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:45:56.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.498+0000 7f1cf259c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1cf40aaab0 0x7f1cf40a6580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:45:56.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.498+0000 7f1cf1d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1cf40aae90 0x7f1cf413db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46078/0 (socket says 172.21.15.23:46078) 2020-11-20T11:45:56.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.498+0000 7f1cf1d9b700 1 -- 172.21.15.23:0/4103565534 learned_addr learned my addr 172.21.15.23:0/4103565534 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:45:56.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.498+0000 7f1cf2d9d700 1 --2- 172.21.15.23:0/4103565534 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1cf414d660 0x7f1cf40a7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:56.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.498+0000 7f1cf1d9b700 1 -- 172.21.15.23:0/4103565534 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1cf414d660 msgr2=0x7f1cf40a7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:56.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.499+0000 7f1cf1d9b700 1 --2- 172.21.15.23:0/4103565534 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1cf414d660 0x7f1cf40a7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:56.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.499+0000 7f1cf1d9b700 1 -- 172.21.15.23:0/4103565534 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1cf40aaab0 msgr2=0x7f1cf40a6580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:45:56.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.499+0000 7f1cf1d9b700 1 --2- 172.21.15.23:0/4103565534 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1cf40aaab0 0x7f1cf40a6580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:56.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.499+0000 7f1cf1d9b700 1 -- 172.21.15.23:0/4103565534 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf414aa00 con 0x7f1cf40aae90 2020-11-20T11:45:56.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.499+0000 7f1cf1d9b700 1 --2- 172.21.15.23:0/4103565534 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1cf40aae90 0x7f1cf413db10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f1ce400eba0 tx=0x7f1ce4006e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:56.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.499+0000 7f1ce27fc700 1 -- 172.21.15.23:0/4103565534 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce400c920 con 0x7f1cf40aae90 2020-11-20T11:45:56.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.499+0000 7f1ce27fc700 1 -- 172.21.15.23:0/4103565534 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce400ca80 con 0x7f1cf40aae90 2020-11-20T11:45:56.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.499+0000 7f1cf8969700 1 -- 172.21.15.23:0/4103565534 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cf414aea0 con 0x7f1cf40aae90 2020-11-20T11:45:56.515 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.499+0000 7f1ce27fc700 1 -- 172.21.15.23:0/4103565534 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce400ce00 con 0x7f1cf40aae90 2020-11-20T11:45:56.515 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.499+0000 7f1cf8969700 1 -- 172.21.15.23:0/4103565534 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cf414a290 con 0x7f1cf40aae90 2020-11-20T11:45:56.515 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.500+0000 7f1cf8969700 1 -- 172.21.15.23:0/4103565534 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cf405c7d0 con 0x7f1cf40aae90 2020-11-20T11:45:56.515 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.500+0000 7f1ce27fc700 1 -- 172.21.15.23:0/4103565534 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1ce4012600 con 0x7f1cf40aae90 2020-11-20T11:45:56.516 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.500+0000 7f1ce27fc700 1 --2- 172.21.15.23:0/4103565534 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1cc8056d10 0x7f1cc80591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:56.516 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.500+0000 7f1ce27fc700 1 -- 172.21.15.23:0/4103565534 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f1ce4070470 con 0x7f1cf40aae90 2020-11-20T11:45:56.516 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.501+0000 7f1cf259c700 1 --2- 172.21.15.23:0/4103565534 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1cc8056d10 0x7f1cc80591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:56.516 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.501+0000 7f1cf259c700 1 --2- 172.21.15.23:0/4103565534 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1cc8056d10 0x7f1cc80591d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f1ce8006fd0 tx=0x7f1ce8006b20).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:56.516 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.506+0000 7f1ce27fc700 1 -- 172.21.15.23:0/4103565534 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f1ce4045760 con 0x7f1cf40aae90 2020-11-20T11:45:56.790 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.786+0000 7f1cf8969700 1 -- 172.21.15.23:0/4103565534 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1cf414ab90 con 0x7f1cf40aae90 2020-11-20T11:45:56.791 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.786+0000 7f1ce27fc700 1 -- 172.21.15.23:0/4103565534 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1479 (secure 0 0 0) 0x7f1ce4048d80 con 0x7f1cf40aae90 2020-11-20T11:45:56.791 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":126,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:45:56.793 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.789+0000 7f1cf8969700 1 -- 172.21.15.23:0/4103565534 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1cc8056d10 msgr2=0x7f1cc80591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:56.793 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.789+0000 7f1cf8969700 1 --2- 172.21.15.23:0/4103565534 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1cc8056d10 0x7f1cc80591d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f1ce8006fd0 tx=0x7f1ce8006b20).stop 2020-11-20T11:45:56.793 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.789+0000 7f1cf8969700 1 -- 172.21.15.23:0/4103565534 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1cf40aae90 msgr2=0x7f1cf413db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:56.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.789+0000 7f1cf8969700 1 --2- 172.21.15.23:0/4103565534 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1cf40aae90 0x7f1cf413db10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f1ce400eba0 tx=0x7f1ce4006e30).stop 2020-11-20T11:45:56.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.789+0000 7f1cf8969700 1 -- 172.21.15.23:0/4103565534 shutdown_connections 2020-11-20T11:45:56.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.789+0000 7f1cf8969700 1 --2- 172.21.15.23:0/4103565534 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1cc8056d10 0x7f1cc80591d0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:56.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.789+0000 7f1cf8969700 1 --2- 172.21.15.23:0/4103565534 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1cf414d660 0x7f1cf40a7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:56.795 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.789+0000 7f1cf8969700 1 --2- 172.21.15.23:0/4103565534 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1cf40aaab0 0x7f1cf40a6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:56.796 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.789+0000 7f1cf8969700 1 --2- 172.21.15.23:0/4103565534 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1cf40aae90 0x7f1cf413db10 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:56.796 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.789+0000 7f1cf8969700 1 -- 172.21.15.23:0/4103565534 >> 172.21.15.23:0/4103565534 conn(0x7f1cf40678e0 msgr2=0x7f1cf40ac4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:56.796 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.789+0000 7f1cf8969700 1 -- 172.21.15.23:0/4103565534 shutdown_connections 2020-11-20T11:45:56.796 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:56.789+0000 7f1cf8969700 1 -- 172.21.15.23:0/4103565534 wait complete. 2020-11-20T11:45:59.806 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:45:59.950 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.944+0000 7f5e15872700 1 Processor -- start 2020-11-20T11:45:59.951 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.945+0000 7f5e15872700 1 -- start start 2020-11-20T11:45:59.951 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.945+0000 7f5e15872700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e10068260 0x7f5e10065780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:59.951 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.945+0000 7f5e15872700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5e10065c70 0x7f5e10137870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:59.952 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.946+0000 7f5e15872700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5e10137d60 0x7f5e1013a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:59.952 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.946+0000 7f5e15872700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e10134c40 con 0x7f5e10065c70 2020-11-20T11:45:59.952 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.946+0000 7f5e15872700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e100663a0 con 0x7f5e10137d60 2020-11-20T11:45:59.952 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.946+0000 7f5e15872700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e100a9000 con 0x7f5e10068260 2020-11-20T11:45:59.952 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.946+0000 7f5e0f7fe700 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f5e10137d60 msgr2=0x7f5e1013a6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:45:59.953 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.946+0000 7f5e0f7fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5e10137d60 0x7f5e1013a6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:45:59.953 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.946+0000 7f5e0effd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e10068260 0x7f5e10065780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:59.953 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.946+0000 7f5e0effd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e10068260 0x7f5e10065780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:56336/0 (socket says 172.21.15.23:56336) 2020-11-20T11:45:59.953 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.946+0000 7f5e0effd700 1 -- 172.21.15.23:0/1806829013 learned_addr learned my addr 172.21.15.23:0/1806829013 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:45:59.954 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.946+0000 7f5e0effd700 1 -- 172.21.15.23:0/1806829013 >> v2:172.21.15.155:3300/0 conn(0x7f5e10137d60 msgr2=0x7f5e1013a6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:45:59.954 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.946+0000 7f5e0effd700 1 --2- 172.21.15.23:0/1806829013 >> v2:172.21.15.155:3300/0 conn(0x7f5e10137d60 0x7f5e1013a6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:59.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.946+0000 7f5e0effd700 1 -- 172.21.15.23:0/1806829013 >> v2:172.21.15.23:3300/0 conn(0x7f5e10065c70 msgr2=0x7f5e10137870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:45:59.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.946+0000 7f5e0effd700 1 --2- 172.21.15.23:0/1806829013 >> v2:172.21.15.23:3300/0 conn(0x7f5e10065c70 0x7f5e10137870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:59.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.946+0000 7f5e0effd700 1 -- 172.21.15.23:0/1806829013 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e10137170 con 0x7f5e10068260 2020-11-20T11:45:59.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.946+0000 7f5e0effd700 1 --2- 172.21.15.23:0/1806829013 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e10068260 0x7f5e10065780 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f5e100a8f70 tx=0x7f5e00006ef0).ready entity=mon.2 client_cookie=53959e65394f162d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:59.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.949+0000 7f5e0dffb700 1 -- 172.21.15.23:0/1806829013 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e00017070 con 0x7f5e10068260 2020-11-20T11:45:59.956 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.949+0000 7f5e0dffb700 1 -- 172.21.15.23:0/1806829013 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e00000e90 con 0x7f5e10068260 2020-11-20T11:45:59.956 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.949+0000 7f5e15872700 1 -- 172.21.15.23:0/1806829013 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e10068260 msgr2=0x7f5e10065780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:59.956 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.949+0000 7f5e15872700 1 --2- 172.21.15.23:0/1806829013 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e10068260 0x7f5e10065780 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f5e100a8f70 tx=0x7f5e00006ef0).stop 2020-11-20T11:45:59.957 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.952+0000 7f5e15872700 1 -- 172.21.15.23:0/1806829013 shutdown_connections 2020-11-20T11:45:59.957 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.952+0000 7f5e15872700 1 --2- 172.21.15.23:0/1806829013 >> v2:172.21.15.155:3300/0 conn(0x7f5e10137d60 0x7f5e1013a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:59.957 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.952+0000 7f5e15872700 1 --2- 172.21.15.23:0/1806829013 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e10068260 0x7f5e10065780 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:59.957 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.952+0000 7f5e15872700 1 --2- 172.21.15.23:0/1806829013 >> v2:172.21.15.23:3300/0 conn(0x7f5e10065c70 0x7f5e10137870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:59.958 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.952+0000 7f5e15872700 1 -- 172.21.15.23:0/1806829013 >> 172.21.15.23:0/1806829013 conn(0x7f5e100ad7a0 msgr2=0x7f5e100adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:45:59.958 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.952+0000 7f5e15872700 1 -- 172.21.15.23:0/1806829013 shutdown_connections 2020-11-20T11:45:59.958 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.952+0000 7f5e15872700 1 -- 172.21.15.23:0/1806829013 wait complete. 2020-11-20T11:45:59.960 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.954+0000 7f5e15872700 1 Processor -- start 2020-11-20T11:45:59.960 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.954+0000 7f5e15872700 1 -- start start 2020-11-20T11:45:59.960 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.955+0000 7f5e15872700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e10068260 0x7f5e101312a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:59.961 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.955+0000 7f5e15872700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e10137d60 0x7f5e100ac740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:59.961 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.955+0000 7f5e15872700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e1014e710 0x7f5e100a70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:59.961 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.955+0000 7f5e15872700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1012f0e0 con 0x7f5e10137d60 2020-11-20T11:45:59.961 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.955+0000 7f5e15872700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e10151ff0 con 0x7f5e1014e710 2020-11-20T11:45:59.961 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.955+0000 7f5e15872700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e10152290 con 0x7f5e10068260 2020-11-20T11:45:59.962 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.955+0000 7f5e0f7fe700 1 -- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e1014e710 msgr2=0x7f5e100a70e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:45:59.962 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.955+0000 7f5e0e7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e10137d60 0x7f5e100ac740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:59.962 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.955+0000 7f5e0f7fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e1014e710 0x7f5e100a70e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:45:59.962 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.955+0000 7f5e0effd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e10068260 0x7f5e101312a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:59.962 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.955+0000 7f5e0e7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e10137d60 0x7f5e100ac740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46102/0 (socket says 172.21.15.23:46102) 2020-11-20T11:45:59.963 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.955+0000 7f5e0e7fc700 1 -- 172.21.15.23:0/3642840286 learned_addr learned my addr 172.21.15.23:0/3642840286 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:45:59.963 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.955+0000 7f5e0effd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e10068260 0x7f5e101312a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:56342/0 (socket says 172.21.15.23:56342) 2020-11-20T11:45:59.963 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.955+0000 7f5e0e7fc700 1 -- 172.21.15.23:0/3642840286 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e10068260 msgr2=0x7f5e101312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:45:59.963 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.955+0000 7f5e0e7fc700 1 --2- 172.21.15.23:0/3642840286 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e10068260 0x7f5e101312a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:45:59.964 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.955+0000 7f5e0e7fc700 1 -- 172.21.15.23:0/3642840286 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e1014e710 msgr2=0x7f5e100a70e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:45:59.964 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.955+0000 7f5e0e7fc700 1 --2- 172.21.15.23:0/3642840286 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e1014e710 0x7f5e100a70e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:45:59.964 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.955+0000 7f5e0e7fc700 1 -- 172.21.15.23:0/3642840286 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e10152530 con 0x7f5e10137d60 2020-11-20T11:45:59.964 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.955+0000 7f5e0effd700 1 --2- 172.21.15.23:0/3642840286 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e10068260 0x7f5e101312a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:45:59.964 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.956+0000 7f5e0e7fc700 1 --2- 172.21.15.23:0/3642840286 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e10137d60 0x7f5e100ac740 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f5e0800bae0 tx=0x7f5e080091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:59.965 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.956+0000 7f5df6ffd700 1 -- 172.21.15.23:0/3642840286 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0800c7a0 con 0x7f5e10137d60 2020-11-20T11:45:59.965 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.956+0000 7f5df6ffd700 1 -- 172.21.15.23:0/3642840286 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e0800c900 con 0x7f5e10137d60 2020-11-20T11:45:59.965 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.956+0000 7f5e15872700 1 -- 172.21.15.23:0/3642840286 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e10151ce0 con 0x7f5e10137d60 2020-11-20T11:45:59.965 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.956+0000 7f5df6ffd700 1 -- 172.21.15.23:0/3642840286 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0800ca90 con 0x7f5e10137d60 2020-11-20T11:45:59.965 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.956+0000 7f5e15872700 1 -- 172.21.15.23:0/3642840286 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e101529d0 con 0x7f5e10137d60 2020-11-20T11:45:59.966 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.957+0000 7f5e15872700 1 -- 172.21.15.23:0/3642840286 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e101526c0 con 0x7f5e10137d60 2020-11-20T11:45:59.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.957+0000 7f5df6ffd700 1 -- 172.21.15.23:0/3642840286 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5e0801c070 con 0x7f5e10137d60 2020-11-20T11:45:59.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.957+0000 7f5df6ffd700 1 --2- 172.21.15.23:0/3642840286 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5dec056d10 0x7f5dec0591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:45:59.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.957+0000 7f5df6ffd700 1 -- 172.21.15.23:0/3642840286 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f5e0806f710 con 0x7f5e10137d60 2020-11-20T11:45:59.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.958+0000 7f5e0effd700 1 --2- 172.21.15.23:0/3642840286 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5dec056d10 0x7f5dec0591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:45:59.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.958+0000 7f5e0effd700 1 --2- 172.21.15.23:0/3642840286 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5dec056d10 0x7f5dec0591d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f5e00006f30 tx=0x7f5e00019000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:45:59.968 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:45:59.961+0000 7f5df6ffd700 1 -- 172.21.15.23:0/3642840286 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5e08044a00 con 0x7f5e10137d60 2020-11-20T11:46:00.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:00.231+0000 7f5e15872700 1 -- 172.21.15.23:0/3642840286 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5e10062fe0 con 0x7f5e10137d60 2020-11-20T11:46:00.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:00.231+0000 7f5df6ffd700 1 -- 172.21.15.23:0/3642840286 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1479 (secure 0 0 0) 0x7f5e0801a070 con 0x7f5e10137d60 2020-11-20T11:46:00.236 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":126,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:46:00.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:00.234+0000 7f5e15872700 1 -- 172.21.15.23:0/3642840286 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5dec056d10 msgr2=0x7f5dec0591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:00.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:00.234+0000 7f5e15872700 1 --2- 172.21.15.23:0/3642840286 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5dec056d10 0x7f5dec0591d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f5e00006f30 tx=0x7f5e00019000).stop 2020-11-20T11:46:00.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:00.234+0000 7f5e15872700 1 -- 172.21.15.23:0/3642840286 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e10137d60 msgr2=0x7f5e100ac740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:00.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:00.234+0000 7f5e15872700 1 --2- 172.21.15.23:0/3642840286 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e10137d60 0x7f5e100ac740 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f5e0800bae0 tx=0x7f5e080091d0).stop 2020-11-20T11:46:00.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:00.234+0000 7f5e15872700 1 -- 172.21.15.23:0/3642840286 shutdown_connections 2020-11-20T11:46:00.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:00.234+0000 7f5e15872700 1 --2- 172.21.15.23:0/3642840286 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e1014e710 0x7f5e100a70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:00.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:00.234+0000 7f5e15872700 1 --2- 172.21.15.23:0/3642840286 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5dec056d10 0x7f5dec0591d0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:00.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:00.234+0000 7f5e15872700 1 --2- 172.21.15.23:0/3642840286 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e10068260 0x7f5e101312a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:00.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:00.234+0000 7f5e15872700 1 --2- 172.21.15.23:0/3642840286 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e10137d60 0x7f5e100ac740 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:00.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:00.234+0000 7f5e15872700 1 -- 172.21.15.23:0/3642840286 >> 172.21.15.23:0/3642840286 conn(0x7f5e100ad7a0 msgr2=0x7f5e10135970 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:00.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:00.234+0000 7f5e15872700 1 -- 172.21.15.23:0/3642840286 shutdown_connections 2020-11-20T11:46:00.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:00.234+0000 7f5e15872700 1 -- 172.21.15.23:0/3642840286 wait complete. 2020-11-20T11:46:02.687 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:02.659+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4659.0:46439 4.2s0 4:5ef0fbf2:::10000000378.00000001:head [write 2871296~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:03.255 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:46:03.602 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.589+0000 7f9d7ed03700 1 Processor -- start 2020-11-20T11:46:03.603 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.589+0000 7f9d7ed03700 1 -- start start 2020-11-20T11:46:03.604 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.590+0000 7f9d7ed03700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f9d780a41a0 0x7f9d780aaac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:03.604 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.590+0000 7f9d7ed03700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d780aafb0 con 0x7f9d78061380 2020-11-20T11:46:03.604 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.590+0000 7f9d7ed03700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d780a3050 con 0x7f9d780a9dd0 2020-11-20T11:46:03.605 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.590+0000 7f9d7ed03700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d780a32f0 con 0x7f9d780a41a0 2020-11-20T11:46:03.605 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.590+0000 7f9d7d2a0700 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f9d780a9dd0 legacy=0x7f9d781378c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:46:03.606 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.590+0000 7f9d77fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f9d780a41a0 0x7f9d780aaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:03.606 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.590+0000 7f9d7ca9f700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f9d78061380 0x7f9d780a3a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:60362/0 (socket says 172.21.15.23:60362) 2020-11-20T11:46:03.607 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.590+0000 7f9d7ca9f700 1 -- 172.21.15.23:0/2578222739 learned_addr learned my addr 172.21.15.23:0/2578222739 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:46:03.607 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.590+0000 7f9d77fff700 1 -- 172.21.15.23:0/2578222739 >> v1:172.21.15.155:6789/0 conn(0x7f9d780a9dd0 legacy=0x7f9d781378c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:46:03.607 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.590+0000 7f9d77fff700 1 -- 172.21.15.23:0/2578222739 >> v1:172.21.15.23:6789/0 conn(0x7f9d78061380 legacy=0x7f9d780a3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:03.608 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.590+0000 7f9d77fff700 1 -- 172.21.15.23:0/2578222739 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d780a3590 con 0x7f9d780a41a0 2020-11-20T11:46:03.608 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.591+0000 7f9d77fff700 1 --2- 172.21.15.23:0/2578222739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d780a41a0 0x7f9d780aaac0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f9d6400ba00 tx=0x7f9d64006e30).ready entity=mon.0 client_cookie=ce1842ccfd359a26 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:03.608 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.591+0000 7f9d777fe700 1 -- 172.21.15.23:0/2578222739 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d64009760 con 0x7f9d780a41a0 2020-11-20T11:46:03.609 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.591+0000 7f9d777fe700 1 -- 172.21.15.23:0/2578222739 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d64000e90 con 0x7f9d780a41a0 2020-11-20T11:46:03.609 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.591+0000 7f9d777fe700 1 -- 172.21.15.23:0/2578222739 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d64009ad0 con 0x7f9d780a41a0 2020-11-20T11:46:03.609 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.591+0000 7f9d7ed03700 1 -- 172.21.15.23:0/2578222739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d780a41a0 msgr2=0x7f9d780aaac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:03.610 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.591+0000 7f9d7ed03700 1 --2- 172.21.15.23:0/2578222739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d780a41a0 0x7f9d780aaac0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f9d6400ba00 tx=0x7f9d64006e30).stop 2020-11-20T11:46:03.610 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.591+0000 7f9d7ed03700 1 -- 172.21.15.23:0/2578222739 shutdown_connections 2020-11-20T11:46:03.611 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.591+0000 7f9d7ed03700 1 --2- 172.21.15.23:0/2578222739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d780a41a0 0x7f9d780aaac0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:03.611 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.591+0000 7f9d7ed03700 1 -- 172.21.15.23:0/2578222739 >> 172.21.15.23:0/2578222739 conn(0x7f9d780ae280 msgr2=0x7f9d780aa290 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:03.611 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.591+0000 7f9d7ed03700 1 -- 172.21.15.23:0/2578222739 shutdown_connections 2020-11-20T11:46:03.612 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.591+0000 7f9d7ed03700 1 -- 172.21.15.23:0/2578222739 wait complete. 2020-11-20T11:46:03.613 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.592+0000 7f9d7ed03700 1 Processor -- start 2020-11-20T11:46:03.613 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.592+0000 7f9d7ed03700 1 -- start start 2020-11-20T11:46:03.613 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.593+0000 7f9d7ed03700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d78061380 0x7f9d78136a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:03.614 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.593+0000 7f9d7ed03700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d780a41a0 0x7f9d78066540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:03.614 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.593+0000 7f9d7ed03700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d780a9dd0 0x7f9d78066a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:03.614 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.593+0000 7f9d7ed03700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d78066f20 con 0x7f9d78061380 2020-11-20T11:46:03.615 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.593+0000 7f9d7ed03700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d780679b0 con 0x7f9d780a9dd0 2020-11-20T11:46:03.615 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.593+0000 7f9d7ed03700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d78067c50 con 0x7f9d780a41a0 2020-11-20T11:46:03.615 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.593+0000 7f9d7d2a0700 1 -- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d780a9dd0 msgr2=0x7f9d78066a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:46:03.616 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.593+0000 7f9d7ca9f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d78061380 0x7f9d78136a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:03.616 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.593+0000 7f9d7d2a0700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d780a9dd0 0x7f9d78066a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:46:03.617 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.593+0000 7f9d7ca9f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d78061380 0x7f9d78136a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46118/0 (socket says 172.21.15.23:46118) 2020-11-20T11:46:03.617 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.593+0000 7f9d7ca9f700 1 -- 172.21.15.23:0/3992223914 learned_addr learned my addr 172.21.15.23:0/3992223914 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:46:03.618 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.593+0000 7f9d77fff700 1 --2- 172.21.15.23:0/3992223914 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d780a41a0 0x7f9d78066540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:03.618 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.593+0000 7f9d7ca9f700 1 -- 172.21.15.23:0/3992223914 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d780a41a0 msgr2=0x7f9d78066540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:03.618 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.593+0000 7f9d7ca9f700 1 --2- 172.21.15.23:0/3992223914 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d780a41a0 0x7f9d78066540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:03.619 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.593+0000 7f9d7ca9f700 1 -- 172.21.15.23:0/3992223914 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d780a9dd0 msgr2=0x7f9d78066a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:46:03.619 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.593+0000 7f9d7ca9f700 1 --2- 172.21.15.23:0/3992223914 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d780a9dd0 0x7f9d78066a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:03.619 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.593+0000 7f9d7ca9f700 1 -- 172.21.15.23:0/3992223914 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d78067ef0 con 0x7f9d78061380 2020-11-20T11:46:03.620 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.593+0000 7f9d77fff700 1 --2- 172.21.15.23:0/3992223914 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d780a41a0 0x7f9d78066540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:46:03.620 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.594+0000 7f9d7ca9f700 1 --2- 172.21.15.23:0/3992223914 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d78061380 0x7f9d78136a50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f9d6c00c710 tx=0x7f9d6c007ce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:03.620 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.594+0000 7f9d74ff9700 1 -- 172.21.15.23:0/3992223914 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c007320 con 0x7f9d78061380 2020-11-20T11:46:03.621 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.594+0000 7f9d74ff9700 1 -- 172.21.15.23:0/3992223914 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d6c007480 con 0x7f9d78061380 2020-11-20T11:46:03.621 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.594+0000 7f9d74ff9700 1 -- 172.21.15.23:0/3992223914 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c005840 con 0x7f9d78061380 2020-11-20T11:46:03.621 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.594+0000 7f9d7ed03700 1 -- 172.21.15.23:0/3992223914 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d780658a0 con 0x7f9d78061380 2020-11-20T11:46:03.622 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.594+0000 7f9d7ed03700 1 -- 172.21.15.23:0/3992223914 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d7813d410 con 0x7f9d78061380 2020-11-20T11:46:03.622 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.595+0000 7f9d7ed03700 1 -- 172.21.15.23:0/3992223914 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d78061e10 con 0x7f9d78061380 2020-11-20T11:46:03.622 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.595+0000 7f9d74ff9700 1 -- 172.21.15.23:0/3992223914 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9d6c010030 con 0x7f9d78061380 2020-11-20T11:46:03.622 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.595+0000 7f9d74ff9700 1 --2- 172.21.15.23:0/3992223914 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9d50056d10 0x7f9d500591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:03.623 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.595+0000 7f9d74ff9700 1 -- 172.21.15.23:0/3992223914 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f9d6c06fc80 con 0x7f9d78061380 2020-11-20T11:46:03.623 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.596+0000 7f9d77fff700 1 --2- 172.21.15.23:0/3992223914 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9d50056d10 0x7f9d500591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:03.623 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.596+0000 7f9d77fff700 1 --2- 172.21.15.23:0/3992223914 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9d50056d10 0x7f9d500591d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f9d64006e70 tx=0x7f9d64015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:03.624 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.602+0000 7f9d74ff9700 1 -- 172.21.15.23:0/3992223914 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f9d6c044f70 con 0x7f9d78061380 2020-11-20T11:46:03.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.959+0000 7f9d7ed03700 1 -- 172.21.15.23:0/3992223914 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f9d780677c0 con 0x7f9d78061380 2020-11-20T11:46:03.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.959+0000 7f9d74ff9700 1 -- 172.21.15.23:0/3992223914 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1479 (secure 0 0 0) 0x7f9d6c048590 con 0x7f9d78061380 2020-11-20T11:46:03.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.961+0000 7f9d7ed03700 1 -- 172.21.15.23:0/3992223914 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9d50056d10 msgr2=0x7f9d500591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:03.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.961+0000 7f9d7ed03700 1 --2- 172.21.15.23:0/3992223914 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9d50056d10 0x7f9d500591d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f9d64006e70 tx=0x7f9d64015000).stop 2020-11-20T11:46:03.968 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.961+0000 7f9d7ed03700 1 -- 172.21.15.23:0/3992223914 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d78061380 msgr2=0x7f9d78136a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:03.968 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.961+0000 7f9d7ed03700 1 --2- 172.21.15.23:0/3992223914 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d78061380 0x7f9d78136a50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f9d6c00c710 tx=0x7f9d6c007ce0).stop 2020-11-20T11:46:03.968 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.961+0000 7f9d7ed03700 1 -- 172.21.15.23:0/3992223914 shutdown_connections 2020-11-20T11:46:03.968 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.961+0000 7f9d7ed03700 1 --2- 172.21.15.23:0/3992223914 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9d780a9dd0 0x7f9d78066a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:03.969 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.961+0000 7f9d7ed03700 1 --2- 172.21.15.23:0/3992223914 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d78061380 0x7f9d78136a50 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:03.969 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.961+0000 7f9d7ed03700 1 --2- 172.21.15.23:0/3992223914 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9d50056d10 0x7f9d500591d0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:03.969 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.961+0000 7f9d7ed03700 1 --2- 172.21.15.23:0/3992223914 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9d780a41a0 0x7f9d78066540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:03.969 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.961+0000 7f9d7ed03700 1 -- 172.21.15.23:0/3992223914 >> 172.21.15.23:0/3992223914 conn(0x7f9d780ae280 msgr2=0x7f9d780a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:03.969 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.961+0000 7f9d7ed03700 1 -- 172.21.15.23:0/3992223914 shutdown_connections 2020-11-20T11:46:03.970 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:03.961+0000 7f9d7ed03700 1 -- 172.21.15.23:0/3992223914 wait complete. 2020-11-20T11:46:03.970 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":126,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:46:04.760 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:04.733+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4659.0:46998 4.2s0 4:5f6d9015:::10000000382.0000000f:head [write 856064~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:05.857 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:05.817+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4659.0:46998 4.2s0 4:5f6d9015:::10000000382.0000000f:head [write 856064~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:06.848 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:06.829+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4659.0:46998 4.2s0 4:5f6d9015:::10000000382.0000000f:head [write 856064~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:06.979 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:46:07.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.136+0000 7f0ffda68700 1 Processor -- start 2020-11-20T11:46:07.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.136+0000 7f0ffda68700 1 -- start start 2020-11-20T11:46:07.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.137+0000 7f0ffda68700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0ff80676a0 0x7f0ff80a89c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:07.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.137+0000 7f0ffda68700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ff8065e50 0x7f0ff80a8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:07.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.137+0000 7f0ffda68700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ff80aad30 0x7f0ff80ab1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:07.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.137+0000 7f0ffda68700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ff8067d10 con 0x7f0ff80676a0 2020-11-20T11:46:07.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.137+0000 7f0ffda68700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ff8066700 con 0x7f0ff80aad30 2020-11-20T11:46:07.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.137+0000 7f0ffda68700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff80a93a0 con 0x7f0ff8065e50 2020-11-20T11:46:07.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.137+0000 7f0ffca66700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0ff80676a0 0x7f0ff80a89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:07.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.137+0000 7f0ffca66700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0ff80676a0 0x7f0ff80a89c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46126/0 (socket says 172.21.15.23:46126) 2020-11-20T11:46:07.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.137+0000 7f0ffca66700 1 -- 172.21.15.23:0/3646697495 learned_addr learned my addr 172.21.15.23:0/3646697495 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:46:07.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.137+0000 7f0ffd267700 1 -- 172.21.15.23:0/3646697495 >> v2:172.21.15.155:3300/0 conn(0x7f0ff80aad30 msgr2=0x7f0ff80ab1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:46:07.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.137+0000 7f0ffd267700 1 --2- 172.21.15.23:0/3646697495 >> v2:172.21.15.155:3300/0 conn(0x7f0ff80aad30 0x7f0ff80ab1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:46:07.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.137+0000 7f0ff7fff700 1 --2- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ff8065e50 0x7f0ff80a8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:07.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.137+0000 7f0ffca66700 1 -- 172.21.15.23:0/3646697495 >> v2:172.21.15.23:3300/0 conn(0x7f0ff80676a0 msgr2=0x7f0ff80a89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:46:07.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.137+0000 7f0ffca66700 1 -- 172.21.15.23:0/3646697495 >> v2:172.21.15.23:3300/0 conn(0x7f0ff80676a0 msgr2=0x7f0ff80a89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:46:07.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.137+0000 7f0ffca66700 1 --2- 172.21.15.23:0/3646697495 >> v2:172.21.15.23:3300/0 conn(0x7f0ff80676a0 0x7f0ff80a89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:46:07.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.137+0000 7f0ffca66700 1 --2- 172.21.15.23:0/3646697495 >> v2:172.21.15.23:3300/0 conn(0x7f0ff80676a0 0x7f0ff80a89c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:46:07.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.137+0000 7f0ff7fff700 1 -- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ff8065e50 msgr2=0x7f0ff80a8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:46:07.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.137+0000 7f0ff7fff700 1 -- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ff8065e50 msgr2=0x7f0ff80a8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:46:07.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.137+0000 7f0ff7fff700 1 --2- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ff8065e50 0x7f0ff80a8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:46:07.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.137+0000 7f0ff7fff700 1 --2- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ff8065e50 0x7f0ff80a8eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:46:07.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.338+0000 7f0ffd267700 1 -- 172.21.15.23:0/3646697495 >> v2:172.21.15.155:3300/0 conn(0x7f0ff80aad30 msgr2=0x7f0ff80ab1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:46:07.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.338+0000 7f0ffd267700 1 --2- 172.21.15.23:0/3646697495 >> v2:172.21.15.155:3300/0 conn(0x7f0ff80aad30 0x7f0ff80ab1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:46:07.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.338+0000 7f0ffca66700 1 --2- 172.21.15.23:0/3646697495 >> v2:172.21.15.23:3300/0 conn(0x7f0ff80676a0 0x7f0ff80a89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:07.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.338+0000 7f0ffca66700 1 -- 172.21.15.23:0/3646697495 >> v2:172.21.15.23:3300/0 conn(0x7f0ff80676a0 msgr2=0x7f0ff80a89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:46:07.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.338+0000 7f0ffca66700 1 -- 172.21.15.23:0/3646697495 >> v2:172.21.15.23:3300/0 conn(0x7f0ff80676a0 msgr2=0x7f0ff80a89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:46:07.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.338+0000 7f0ffca66700 1 --2- 172.21.15.23:0/3646697495 >> v2:172.21.15.23:3300/0 conn(0x7f0ff80676a0 0x7f0ff80a89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:46:07.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.338+0000 7f0ffca66700 1 --2- 172.21.15.23:0/3646697495 >> v2:172.21.15.23:3300/0 conn(0x7f0ff80676a0 0x7f0ff80a89c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:46:07.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.338+0000 7f0ff7fff700 1 --2- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ff8065e50 0x7f0ff80a8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:07.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.338+0000 7f0ff7fff700 1 -- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ff8065e50 msgr2=0x7f0ff80a8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:46:07.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.338+0000 7f0ff7fff700 1 -- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ff8065e50 msgr2=0x7f0ff80a8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:46:07.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.338+0000 7f0ff7fff700 1 --2- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ff8065e50 0x7f0ff80a8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:46:07.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.338+0000 7f0ff7fff700 1 --2- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ff8065e50 0x7f0ff80a8eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:46:07.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.739+0000 7f0ffd267700 1 -- 172.21.15.23:0/3646697495 >> v2:172.21.15.155:3300/0 conn(0x7f0ff80aad30 msgr2=0x7f0ff80ab1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:46:07.744 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.739+0000 7f0ffd267700 1 --2- 172.21.15.23:0/3646697495 >> v2:172.21.15.155:3300/0 conn(0x7f0ff80aad30 0x7f0ff80ab1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:46:07.744 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.739+0000 7f0ffca66700 1 --2- 172.21.15.23:0/3646697495 >> v2:172.21.15.23:3300/0 conn(0x7f0ff80676a0 0x7f0ff80a89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:07.744 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.739+0000 7f0ff7fff700 1 --2- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ff8065e50 0x7f0ff80a8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:07.744 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.739+0000 7f0ffca66700 1 -- 172.21.15.23:0/3646697495 >> v2:172.21.15.23:3300/0 conn(0x7f0ff80676a0 msgr2=0x7f0ff80a89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:46:07.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.739+0000 7f0ffca66700 1 -- 172.21.15.23:0/3646697495 >> v2:172.21.15.23:3300/0 conn(0x7f0ff80676a0 msgr2=0x7f0ff80a89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:46:07.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.739+0000 7f0ffca66700 1 --2- 172.21.15.23:0/3646697495 >> v2:172.21.15.23:3300/0 conn(0x7f0ff80676a0 0x7f0ff80a89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:46:07.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.739+0000 7f0ffca66700 1 --2- 172.21.15.23:0/3646697495 >> v2:172.21.15.23:3300/0 conn(0x7f0ff80676a0 0x7f0ff80a89c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:46:07.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.739+0000 7f0ff7fff700 1 -- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ff8065e50 msgr2=0x7f0ff80a8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:46:07.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.739+0000 7f0ff7fff700 1 -- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ff8065e50 msgr2=0x7f0ff80a8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:46:07.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.739+0000 7f0ff7fff700 1 --2- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ff8065e50 0x7f0ff80a8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:46:07.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:07.739+0000 7f0ff7fff700 1 --2- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ff8065e50 0x7f0ff80a8eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:46:07.906 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:07.898+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4659.0:47292 4.2s0 4:4cf9516a:::100000003b6.0000000f:head [write 925696~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:08.544 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:08.540+0000 7f0ffca66700 1 --2- 172.21.15.23:0/3646697495 >> v2:172.21.15.23:3300/0 conn(0x7f0ff80676a0 0x7f0ff80a89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:08.544 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:08.540+0000 7f0ffca66700 1 -- 172.21.15.23:0/3646697495 >> v2:172.21.15.23:3300/0 conn(0x7f0ff80676a0 msgr2=0x7f0ff80a89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:46:08.545 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:08.540+0000 7f0ffca66700 1 -- 172.21.15.23:0/3646697495 >> v2:172.21.15.23:3300/0 conn(0x7f0ff80676a0 msgr2=0x7f0ff80a89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:46:08.545 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:08.540+0000 7f0ffca66700 1 --2- 172.21.15.23:0/3646697495 >> v2:172.21.15.23:3300/0 conn(0x7f0ff80676a0 0x7f0ff80a89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:46:08.545 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:08.540+0000 7f0ffca66700 1 --2- 172.21.15.23:0/3646697495 >> v2:172.21.15.23:3300/0 conn(0x7f0ff80676a0 0x7f0ff80a89c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:46:08.546 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:08.541+0000 7f0ffd267700 1 -- 172.21.15.23:0/3646697495 >> v2:172.21.15.155:3300/0 conn(0x7f0ff80aad30 msgr2=0x7f0ff80ab1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:46:08.546 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:08.541+0000 7f0ffd267700 1 --2- 172.21.15.23:0/3646697495 >> v2:172.21.15.155:3300/0 conn(0x7f0ff80aad30 0x7f0ff80ab1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:46:08.547 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:08.541+0000 7f0ff7fff700 1 --2- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ff8065e50 0x7f0ff80a8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:08.548 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:08.542+0000 7f0ff7fff700 1 -- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ff8065e50 msgr2=0x7f0ff80a8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:46:08.548 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:08.542+0000 7f0ff7fff700 1 -- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ff8065e50 msgr2=0x7f0ff80a8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:46:08.548 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:08.542+0000 7f0ff7fff700 1 --2- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ff8065e50 0x7f0ff80a8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:46:08.549 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:08.542+0000 7f0ff7fff700 1 --2- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ff8065e50 0x7f0ff80a8eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:46:08.926 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:08.908+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4659.0:47503 4.2s0 4:5b1218e7:::1000000037e.00000003:head [write 2736128~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:09.972 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:09.963+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4659.0:47503 4.2s0 4:5b1218e7:::1000000037e.00000003:head [write 2736128~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:10.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.137+0000 7f0ff67fc700 1 -- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ff8065e50 msgr2=0x7f0ff80a8eb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:46:10.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.137+0000 7f0ff67fc700 1 --2- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ff8065e50 0x7f0ff80a8eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:10.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.137+0000 7f0ff67fc700 1 -- 172.21.15.23:0/3646697495 >> v2:172.21.15.155:3300/0 conn(0x7f0ff80aad30 msgr2=0x7f0ff80ab1b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:46:10.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.137+0000 7f0ff67fc700 1 --2- 172.21.15.23:0/3646697495 >> v2:172.21.15.155:3300/0 conn(0x7f0ff80aad30 0x7f0ff80ab1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:10.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.137+0000 7f0ff67fc700 1 -- 172.21.15.23:0/3646697495 >> v2:172.21.15.23:3300/0 conn(0x7f0ff80676a0 msgr2=0x7f0ff80a89c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:46:10.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.137+0000 7f0ff67fc700 1 --2- 172.21.15.23:0/3646697495 >> v2:172.21.15.23:3300/0 conn(0x7f0ff80676a0 0x7f0ff80a89c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:10.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.137+0000 7f0ff67fc700 1 -- 172.21.15.23:0/3646697495 --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff8138360 con 0x7f0fe0007850 2020-11-20T11:46:10.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.137+0000 7f0ff67fc700 1 -- 172.21.15.23:0/3646697495 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fe0010230 con 0x7f0fe000b500 2020-11-20T11:46:10.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.137+0000 7f0ff7fff700 1 -- 172.21.15.23:0/3646697495 >> v1:172.21.15.155:6789/0 conn(0x7f0fe000b500 legacy=0x7f0fe000d9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:46:10.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.338+0000 7f0ff7fff700 1 -- 172.21.15.23:0/3646697495 >> v1:172.21.15.155:6789/0 conn(0x7f0fe000b500 legacy=0x7f0fe000d9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:46:10.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.739+0000 7f0ff7fff700 1 -- 172.21.15.23:0/3646697495 >> v1:172.21.15.155:6789/0 conn(0x7f0fe000b500 legacy=0x7f0fe000d9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:46:10.973 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.967+0000 7f0ff77fe700 1 -- 172.21.15.23:0/3646697495 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3537591283 0 0) 0x7f0ff8138360 con 0x7f0fe0007850 2020-11-20T11:46:10.973 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.967+0000 7f0ff77fe700 1 -- 172.21.15.23:0/3646697495 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fdc003280 con 0x7f0fe0007850 2020-11-20T11:46:10.973 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.967+0000 7f0ff77fe700 1 -- 172.21.15.23:0/3646697495 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3548176186 0 0) 0x7f0fdc003280 con 0x7f0fe0007850 2020-11-20T11:46:10.974 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.967+0000 7f0ff77fe700 1 -- 172.21.15.23:0/3646697495 >> v1:172.21.15.155:6789/0 conn(0x7f0fe000b500 legacy=0x7f0fe000d9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:46:10.974 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.967+0000 7f0ff77fe700 1 -- 172.21.15.23:0/3646697495 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe00114b0 con 0x7f0fe0007850 2020-11-20T11:46:10.974 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.967+0000 7f0ff77fe700 1 -- 172.21.15.23:0/3646697495 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f0fec00f5c0 con 0x7f0fe0007850 2020-11-20T11:46:10.974 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.967+0000 7f0ff77fe700 1 -- 172.21.15.23:0/3646697495 >> v1:172.21.15.23:6789/0 conn(0x7f0fe0007850 legacy=0x7f0fe0009cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:10.975 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.967+0000 7f0ffd267700 1 -- 172.21.15.23:0/3646697495 reap_dead start 2020-11-20T11:46:10.975 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.967+0000 7f0ff77fe700 1 --2- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fdc0037a0 0x7f0fdc003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:10.975 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.968+0000 7f0ff77fe700 1 --2- 172.21.15.23:0/3646697495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fdc004070 0x7f0fdc004540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:10.975 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.968+0000 7f0ff77fe700 1 --2- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0fdc004be0 0x7f0fdc00b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:10.976 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.968+0000 7f0ff77fe700 1 -- 172.21.15.23:0/3646697495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fdc005250 con 0x7f0fdc004070 2020-11-20T11:46:10.976 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.968+0000 7f0ff77fe700 1 -- 172.21.15.23:0/3646697495 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fdc0053c0 con 0x7f0fdc0037a0 2020-11-20T11:46:10.976 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.968+0000 7f0ff77fe700 1 -- 172.21.15.23:0/3646697495 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fdc00bcb0 con 0x7f0fdc004be0 2020-11-20T11:46:10.977 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.968+0000 7f0ffca66700 1 -- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fdc0037a0 msgr2=0x7f0fdc003b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:46:10.977 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.968+0000 7f0ffca66700 1 --2- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fdc0037a0 0x7f0fdc003b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:46:10.977 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.968+0000 7f0ff7fff700 1 --2- 172.21.15.23:0/3646697495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fdc004070 0x7f0fdc004540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:10.978 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.968+0000 7f0ffd267700 1 --2- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0fdc004be0 0x7f0fdc00b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:10.978 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.968+0000 7f0ff7fff700 1 -- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0fdc004be0 msgr2=0x7f0fdc00b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:46:10.978 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.968+0000 7f0ff7fff700 1 --2- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0fdc004be0 0x7f0fdc00b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:10.978 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.968+0000 7f0ff7fff700 1 -- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fdc0037a0 msgr2=0x7f0fdc003b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:46:10.978 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.968+0000 7f0ff7fff700 1 --2- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fdc0037a0 0x7f0fdc003b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:10.979 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.968+0000 7f0ff7fff700 1 -- 172.21.15.23:0/3646697495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fdc003280 con 0x7f0fdc004070 2020-11-20T11:46:10.979 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.968+0000 7f0ff7fff700 1 --2- 172.21.15.23:0/3646697495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fdc004070 0x7f0fdc004540 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f0fe400dc90 tx=0x7f0fe4008040).ready entity=mon.0 client_cookie=d4f7f4b571e907e8 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:10.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.969+0000 7f0ff77fe700 1 -- 172.21.15.23:0/3646697495 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe400dea0 con 0x7f0fdc004070 2020-11-20T11:46:10.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.969+0000 7f0ff77fe700 1 -- 172.21.15.23:0/3646697495 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe4012da0 con 0x7f0fdc004070 2020-11-20T11:46:10.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.970+0000 7f0ff77fe700 1 -- 172.21.15.23:0/3646697495 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe4015660 con 0x7f0fdc004070 2020-11-20T11:46:10.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.970+0000 7f0ffda68700 1 -- 172.21.15.23:0/3646697495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fdc004070 msgr2=0x7f0fdc004540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:10.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.970+0000 7f0ffda68700 1 --2- 172.21.15.23:0/3646697495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fdc004070 0x7f0fdc004540 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f0fe400dc90 tx=0x7f0fe4008040).stop 2020-11-20T11:46:10.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.970+0000 7f0ffda68700 1 -- 172.21.15.23:0/3646697495 shutdown_connections 2020-11-20T11:46:10.983 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.970+0000 7f0ffda68700 1 --2- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0fdc004be0 0x7f0fdc00b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:10.983 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.970+0000 7f0ffda68700 1 --2- 172.21.15.23:0/3646697495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fdc004070 0x7f0fdc004540 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:10.983 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.970+0000 7f0ffda68700 1 --2- 172.21.15.23:0/3646697495 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fdc0037a0 0x7f0fdc003b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:10.983 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.970+0000 7f0ffda68700 1 -- 172.21.15.23:0/3646697495 >> 172.21.15.23:0/3646697495 conn(0x7f0ff80612c0 msgr2=0x7f0ff80616d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:10.983 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.971+0000 7f0ffda68700 1 -- 172.21.15.23:0/3646697495 shutdown_connections 2020-11-20T11:46:10.984 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.971+0000 7f0ffda68700 1 -- 172.21.15.23:0/3646697495 wait complete. 2020-11-20T11:46:10.984 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.972+0000 7f0ffda68700 1 Processor -- start 2020-11-20T11:46:10.984 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.972+0000 7f0ffda68700 1 -- start start 2020-11-20T11:46:10.984 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.973+0000 7f0ffda68700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fdc0037a0 0x7f0ff81322e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:10.985 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.973+0000 7f0ffda68700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fdc004070 0x7f0ff8132d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:10.985 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.973+0000 7f0ffda68700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0fdc004be0 0x7f0ff8133280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:10.985 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.973+0000 7f0ffda68700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff8138ca0 con 0x7f0fdc0037a0 2020-11-20T11:46:10.985 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.973+0000 7f0ffda68700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff8133770 con 0x7f0fdc004070 2020-11-20T11:46:10.985 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.973+0000 7f0ffda68700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff813bd80 con 0x7f0fdc004be0 2020-11-20T11:46:10.986 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.973+0000 7f0ffca66700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fdc0037a0 0x7f0ff81322e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:10.986 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.973+0000 7f0ffca66700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fdc0037a0 0x7f0ff81322e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46164/0 (socket says 172.21.15.23:46164) 2020-11-20T11:46:10.986 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.973+0000 7f0ffca66700 1 -- 172.21.15.23:0/2461198645 learned_addr learned my addr 172.21.15.23:0/2461198645 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:46:10.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.973+0000 7f0ff7fff700 1 -- 172.21.15.23:0/2461198645 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fdc004070 msgr2=0x7f0ff8132d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:46:10.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.973+0000 7f0ff7fff700 1 --2- 172.21.15.23:0/2461198645 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fdc004070 0x7f0ff8132d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:46:10.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.973+0000 7f0ffd267700 1 --2- 172.21.15.23:0/2461198645 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0fdc004be0 0x7f0ff8133280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:10.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.974+0000 7f0ffd267700 1 -- 172.21.15.23:0/2461198645 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fdc004070 msgr2=0x7f0ff8132d90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:46:10.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.974+0000 7f0ffd267700 1 --2- 172.21.15.23:0/2461198645 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fdc004070 0x7f0ff8132d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:10.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.974+0000 7f0ffd267700 1 -- 172.21.15.23:0/2461198645 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fdc0037a0 msgr2=0x7f0ff81322e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:10.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.974+0000 7f0ffd267700 1 --2- 172.21.15.23:0/2461198645 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fdc0037a0 0x7f0ff81322e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:10.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.974+0000 7f0ffd267700 1 -- 172.21.15.23:0/2461198645 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff813a3a0 con 0x7f0fdc004be0 2020-11-20T11:46:10.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.974+0000 7f0ffd267700 1 --2- 172.21.15.23:0/2461198645 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0fdc004be0 0x7f0ff8133280 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f0ff80622a0 tx=0x7f0ff0003ca0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:10.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.974+0000 7f0ff4ff9700 1 -- 172.21.15.23:0/2461198645 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff000be50 con 0x7f0fdc004be0 2020-11-20T11:46:10.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.974+0000 7f0ff4ff9700 1 -- 172.21.15.23:0/2461198645 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff0012070 con 0x7f0fdc004be0 2020-11-20T11:46:10.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.974+0000 7f0ff4ff9700 1 -- 172.21.15.23:0/2461198645 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff000e450 con 0x7f0fdc004be0 2020-11-20T11:46:10.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.974+0000 7f0ffda68700 1 -- 172.21.15.23:0/2461198645 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ff813a640 con 0x7f0fdc004be0 2020-11-20T11:46:10.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.974+0000 7f0ffda68700 1 -- 172.21.15.23:0/2461198645 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff80a5200 con 0x7f0fdc004be0 2020-11-20T11:46:10.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.976+0000 7f0ff4ff9700 1 -- 172.21.15.23:0/2461198645 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0ff0003d70 con 0x7f0fdc004be0 2020-11-20T11:46:10.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.976+0000 7f0ff4ff9700 1 --2- 172.21.15.23:0/2461198645 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0fcc056d10 0x7f0fcc0591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:10.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.976+0000 7f0ff4ff9700 1 -- 172.21.15.23:0/2461198645 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f0ff006ff90 con 0x7f0fdc004be0 2020-11-20T11:46:10.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.977+0000 7f0ffca66700 1 --2- 172.21.15.23:0/2461198645 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0fcc056d10 0x7f0fcc0591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:10.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.977+0000 7f0ffca66700 1 --2- 172.21.15.23:0/2461198645 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0fcc056d10 0x7f0fcc0591d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f0fec006b50 tx=0x7f0fec006ac0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:10.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.977+0000 7f0ffda68700 1 -- 172.21.15.23:0/2461198645 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ff8062760 con 0x7f0fdc004be0 2020-11-20T11:46:10.992 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:10.982+0000 7f0ff4ff9700 1 -- 172.21.15.23:0/2461198645 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0ff0048d30 con 0x7f0fdc004be0 2020-11-20T11:46:11.037 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:10.997+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4659.0:48041 4.2s0 4:56d4cd0f:::10000000393.00000002:head [write 3596288~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:11.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.260+0000 7f0ffda68700 1 -- 172.21.15.23:0/2461198645 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0ff8057dc0 con 0x7f0fdc004be0 2020-11-20T11:46:11.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.261+0000 7f0ff4ff9700 1 -- 172.21.15.23:0/2461198645 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1472 (secure 0 0 0) 0x7f0ff00094e0 con 0x7f0fdc004be0 2020-11-20T11:46:11.265 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":130,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:46:11.266 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.262+0000 7f0ffda68700 1 -- 172.21.15.23:0/2461198645 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0fcc056d10 msgr2=0x7f0fcc0591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:11.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.262+0000 7f0ffda68700 1 --2- 172.21.15.23:0/2461198645 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0fcc056d10 0x7f0fcc0591d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f0fec006b50 tx=0x7f0fec006ac0).stop 2020-11-20T11:46:11.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.262+0000 7f0ffda68700 1 -- 172.21.15.23:0/2461198645 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0fdc004be0 msgr2=0x7f0ff8133280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:11.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.262+0000 7f0ffda68700 1 --2- 172.21.15.23:0/2461198645 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0fdc004be0 0x7f0ff8133280 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f0ff80622a0 tx=0x7f0ff0003ca0).stop 2020-11-20T11:46:11.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.262+0000 7f0ffda68700 1 -- 172.21.15.23:0/2461198645 shutdown_connections 2020-11-20T11:46:11.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.262+0000 7f0ffda68700 1 --2- 172.21.15.23:0/2461198645 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0fcc056d10 0x7f0fcc0591d0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:11.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.262+0000 7f0ffda68700 1 --2- 172.21.15.23:0/2461198645 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0fdc004be0 0x7f0ff8133280 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:11.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.262+0000 7f0ffda68700 1 --2- 172.21.15.23:0/2461198645 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0fdc0037a0 0x7f0ff81322e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:11.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.262+0000 7f0ffda68700 1 --2- 172.21.15.23:0/2461198645 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0fdc004070 0x7f0ff8132d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:11.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.262+0000 7f0ffda68700 1 -- 172.21.15.23:0/2461198645 >> 172.21.15.23:0/2461198645 conn(0x7f0ff80612c0 msgr2=0x7f0ff80616a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:11.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.262+0000 7f0ffda68700 1 -- 172.21.15.23:0/2461198645 shutdown_connections 2020-11-20T11:46:11.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.262+0000 7f0ffda68700 1 -- 172.21.15.23:0/2461198645 wait complete. 2020-11-20T11:46:11.280 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2020-11-20T11:46:11.281 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:46:11.426 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.422+0000 7f509215b700 1 Processor -- start 2020-11-20T11:46:11.427 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.422+0000 7f509215b700 1 -- start start 2020-11-20T11:46:11.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.423+0000 7f509215b700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f508c0a4c20 0x7f508c13e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:11.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.423+0000 7f509215b700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f508c0b1a20 con 0x7f508c0ab2a0 2020-11-20T11:46:11.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.423+0000 7f509215b700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f508c0a8b90 con 0x7f508c0ad8d0 2020-11-20T11:46:11.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.423+0000 7f509215b700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f508c13f4a0 con 0x7f508c0a4c20 2020-11-20T11:46:11.432 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.423+0000 7f508bfff700 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f508c0a4c20 msgr2=0x7f508c13e570 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:46:11.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.423+0000 7f508affd700 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f508c0ad8d0 legacy=0x7f508c1438a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:46:11.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.423+0000 7f508bfff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f508c0a4c20 0x7f508c13e570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:46:11.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.423+0000 7f508b7fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f508c0ab2a0 0x7f508c0a4510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:60422/0 (socket says 172.21.15.23:60422) 2020-11-20T11:46:11.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.423+0000 7f508b7fe700 1 -- 172.21.15.23:0/958076303 learned_addr learned my addr 172.21.15.23:0/958076303 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:46:11.433 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.423+0000 7f508a7fc700 1 -- 172.21.15.23:0/958076303 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3010610822 0 0) 0x7f508c0b1a20 con 0x7f508c0ab2a0 2020-11-20T11:46:11.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.423+0000 7f508a7fc700 1 -- 172.21.15.23:0/958076303 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5078003280 con 0x7f508c0ab2a0 2020-11-20T11:46:11.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.423+0000 7f508a7fc700 1 -- 172.21.15.23:0/958076303 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1090811748 0 0) 0x7f5078003280 con 0x7f508c0ab2a0 2020-11-20T11:46:11.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.423+0000 7f508a7fc700 1 -- 172.21.15.23:0/958076303 >> v2:172.21.15.155:3300/0 conn(0x7f508c0a4c20 msgr2=0x7f508c13e570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:46:11.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.423+0000 7f508a7fc700 1 --2- 172.21.15.23:0/958076303 >> v2:172.21.15.155:3300/0 conn(0x7f508c0a4c20 0x7f508c13e570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:11.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.423+0000 7f508a7fc700 1 -- 172.21.15.23:0/958076303 >> v1:172.21.15.155:6789/0 conn(0x7f508c0ad8d0 legacy=0x7f508c1438a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:46:11.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.423+0000 7f508a7fc700 1 -- 172.21.15.23:0/958076303 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f508c0b1c50 con 0x7f508c0ab2a0 2020-11-20T11:46:11.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.423+0000 7f508a7fc700 1 -- 172.21.15.23:0/958076303 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f507c001520 con 0x7f508c0ab2a0 2020-11-20T11:46:11.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.423+0000 7f508a7fc700 1 -- 172.21.15.23:0/958076303 >> v1:172.21.15.23:6789/0 conn(0x7f508c0ab2a0 legacy=0x7f508c0a4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:11.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.423+0000 7f508a7fc700 1 --2- 172.21.15.23:0/958076303 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5078003770 0x7f5078003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:11.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.424+0000 7f508a7fc700 1 --2- 172.21.15.23:0/958076303 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50780040d0 0x7f50780045a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:11.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.424+0000 7f508a7fc700 1 --2- 172.21.15.23:0/958076303 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5078004ba0 0x7f507800b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:11.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.424+0000 7f508a7fc700 1 -- 172.21.15.23:0/958076303 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5078005170 con 0x7f5078004ba0 2020-11-20T11:46:11.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.424+0000 7f508a7fc700 1 -- 172.21.15.23:0/958076303 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50780052e0 con 0x7f5078003770 2020-11-20T11:46:11.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.424+0000 7f508a7fc700 1 -- 172.21.15.23:0/958076303 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f507800bc60 con 0x7f50780040d0 2020-11-20T11:46:11.437 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.424+0000 7f508bfff700 1 --2- 172.21.15.23:0/958076303 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5078004ba0 0x7f507800b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:11.437 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.424+0000 7f508b7fe700 1 -- 172.21.15.23:0/958076303 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5078003770 msgr2=0x7f5078003b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:46:11.437 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.424+0000 7f508b7fe700 1 --2- 172.21.15.23:0/958076303 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5078003770 0x7f5078003b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:46:11.437 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.424+0000 7f508affd700 1 --2- 172.21.15.23:0/958076303 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50780040d0 0x7f50780045a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:11.438 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.424+0000 7f508bfff700 1 -- 172.21.15.23:0/958076303 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50780040d0 msgr2=0x7f50780045a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:46:11.438 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.424+0000 7f508bfff700 1 --2- 172.21.15.23:0/958076303 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50780040d0 0x7f50780045a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:11.438 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.424+0000 7f508bfff700 1 -- 172.21.15.23:0/958076303 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5078003770 msgr2=0x7f5078003b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:46:11.438 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.424+0000 7f508bfff700 1 --2- 172.21.15.23:0/958076303 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5078003770 0x7f5078003b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:11.438 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.424+0000 7f508bfff700 1 -- 172.21.15.23:0/958076303 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5078003280 con 0x7f5078004ba0 2020-11-20T11:46:11.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.424+0000 7f508affd700 1 --2- 172.21.15.23:0/958076303 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f50780040d0 0x7f50780045a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:46:11.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.424+0000 7f508bfff700 1 -- 172.21.15.23:0/958076303 reap_dead start 2020-11-20T11:46:11.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.424+0000 7f508bfff700 1 --2- 172.21.15.23:0/958076303 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5078004ba0 0x7f507800b770 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f5084009d40 tx=0x7f50840092b0).ready entity=mon.0 client_cookie=4e24ec3c7b9095be server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:11.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.424+0000 7f508a7fc700 1 -- 172.21.15.23:0/958076303 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5084003ca0 con 0x7f5078004ba0 2020-11-20T11:46:11.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.424+0000 7f508a7fc700 1 -- 172.21.15.23:0/958076303 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5084003e20 con 0x7f5078004ba0 2020-11-20T11:46:11.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.424+0000 7f508a7fc700 1 -- 172.21.15.23:0/958076303 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508400bbf0 con 0x7f5078004ba0 2020-11-20T11:46:11.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.425+0000 7f509215b700 1 -- 172.21.15.23:0/958076303 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5078004ba0 msgr2=0x7f507800b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:11.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.425+0000 7f509215b700 1 --2- 172.21.15.23:0/958076303 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5078004ba0 0x7f507800b770 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f5084009d40 tx=0x7f50840092b0).stop 2020-11-20T11:46:11.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.425+0000 7f509215b700 1 -- 172.21.15.23:0/958076303 shutdown_connections 2020-11-20T11:46:11.440 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.425+0000 7f509215b700 1 --2- 172.21.15.23:0/958076303 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5078004ba0 0x7f507800b770 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:11.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.425+0000 7f509215b700 1 -- 172.21.15.23:0/958076303 >> 172.21.15.23:0/958076303 conn(0x7f508c0683b0 msgr2=0x7f508c0b1310 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:11.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.425+0000 7f509215b700 1 -- 172.21.15.23:0/958076303 shutdown_connections 2020-11-20T11:46:11.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.425+0000 7f509215b700 1 -- 172.21.15.23:0/958076303 wait complete. 2020-11-20T11:46:11.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.426+0000 7f509215b700 1 Processor -- start 2020-11-20T11:46:11.441 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.426+0000 7f509215b700 1 -- start start 2020-11-20T11:46:11.442 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.426+0000 7f509215b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5078004ba0 0x7f508c09f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:11.442 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.426+0000 7f509215b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f508c0a29f0 0x7f508c0a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:11.442 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.426+0000 7f509215b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f508c14a500 0x7f508c14a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:11.442 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.427+0000 7f509215b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508c0a1470 con 0x7f508c14a500 2020-11-20T11:46:11.442 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.427+0000 7f509215b700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508c0a2e70 con 0x7f508c0a29f0 2020-11-20T11:46:11.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.427+0000 7f509215b700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f508c09fdb0 con 0x7f5078004ba0 2020-11-20T11:46:11.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.427+0000 7f508bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f508c14a500 0x7f508c14a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:11.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.427+0000 7f508affd700 1 -- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f508c0a29f0 msgr2=0x7f508c0a0f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:46:11.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.427+0000 7f508bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f508c14a500 0x7f508c14a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46188/0 (socket says 172.21.15.23:46188) 2020-11-20T11:46:11.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.427+0000 7f508bfff700 1 -- 172.21.15.23:0/2275662777 learned_addr learned my addr 172.21.15.23:0/2275662777 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:46:11.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.427+0000 7f508affd700 1 --2- 172.21.15.23:0/2275662777 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f508c0a29f0 0x7f508c0a0f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:46:11.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.427+0000 7f508b7fe700 1 --2- 172.21.15.23:0/2275662777 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5078004ba0 0x7f508c09f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:11.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.427+0000 7f508bfff700 1 -- 172.21.15.23:0/2275662777 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5078004ba0 msgr2=0x7f508c09f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:11.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.427+0000 7f508bfff700 1 --2- 172.21.15.23:0/2275662777 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5078004ba0 0x7f508c09f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:11.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.427+0000 7f508bfff700 1 -- 172.21.15.23:0/2275662777 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f508c0a29f0 msgr2=0x7f508c0a0f80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:46:11.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.427+0000 7f508bfff700 1 --2- 172.21.15.23:0/2275662777 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f508c0a29f0 0x7f508c0a0f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:11.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.427+0000 7f508bfff700 1 -- 172.21.15.23:0/2275662777 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f508c0a0050 con 0x7f508c14a500 2020-11-20T11:46:11.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.427+0000 7f508bfff700 1 --2- 172.21.15.23:0/2275662777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f508c14a500 0x7f508c14a960 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f508c0b2230 tx=0x7f5084015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:11.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.427+0000 7f506b7fe700 1 -- 172.21.15.23:0/2275662777 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5084005680 con 0x7f508c14a500 2020-11-20T11:46:11.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.427+0000 7f506b7fe700 1 -- 172.21.15.23:0/2275662777 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50840057e0 con 0x7f508c14a500 2020-11-20T11:46:11.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.427+0000 7f509215b700 1 -- 172.21.15.23:0/2275662777 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f508c0a02f0 con 0x7f508c14a500 2020-11-20T11:46:11.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.427+0000 7f506b7fe700 1 -- 172.21.15.23:0/2275662777 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508401c640 con 0x7f508c14a500 2020-11-20T11:46:11.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.427+0000 7f509215b700 1 -- 172.21.15.23:0/2275662777 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f508c14d980 con 0x7f508c14a500 2020-11-20T11:46:11.448 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.428+0000 7f509215b700 1 --2- 172.21.15.23:0/2275662777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f508c0601b0 0x7f508c14dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:11.448 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.428+0000 7f509215b700 1 -- 172.21.15.23:0/2275662777 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508c13c330 con 0x7f508c0601b0 2020-11-20T11:46:11.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.428+0000 7f508b7fe700 1 --2- 172.21.15.23:0/2275662777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f508c0601b0 0x7f508c14dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:11.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.428+0000 7f506b7fe700 1 -- 172.21.15.23:0/2275662777 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f508401a040 con 0x7f508c14a500 2020-11-20T11:46:11.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.429+0000 7f508b7fe700 1 -- 172.21.15.23:0/2275662777 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f508c060640 con 0x7f508c0601b0 2020-11-20T11:46:11.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.429+0000 7f508b7fe700 1 --2- 172.21.15.23:0/2275662777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f508c0601b0 0x7f508c14dbd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f507c00e7b0 tx=0x7f508c0ab070).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:11.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.429+0000 7f506b7fe700 1 --2- 172.21.15.23:0/2275662777 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5064056d10 0x7f50640591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:11.450 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.429+0000 7f506b7fe700 1 -- 172.21.15.23:0/2275662777 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f5084071870 con 0x7f508c14a500 2020-11-20T11:46:11.450 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.429+0000 7f506b7fe700 1 -- 172.21.15.23:0/2275662777 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507c00c5b0 con 0x7f508c0601b0 2020-11-20T11:46:11.450 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.429+0000 7f508affd700 1 --2- 172.21.15.23:0/2275662777 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5064056d10 0x7f50640591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:11.450 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.429+0000 7f506b7fe700 1 -- 172.21.15.23:0/2275662777 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f507c00c710 con 0x7f508c0601b0 2020-11-20T11:46:11.450 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.429+0000 7f506b7fe700 1 -- 172.21.15.23:0/2275662777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f508c0601b0 msgr2=0x7f508c14dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:11.451 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.429+0000 7f506b7fe700 1 --2- 172.21.15.23:0/2275662777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f508c0601b0 0x7f508c14dbd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f507c00e7b0 tx=0x7f508c0ab070).stop 2020-11-20T11:46:11.451 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.429+0000 7f506b7fe700 1 -- 172.21.15.23:0/2275662777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f508c0601b0 msgr2=0x7f508c14dbd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:46:11.451 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.429+0000 7f506b7fe700 1 --2- 172.21.15.23:0/2275662777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f508c0601b0 0x7f508c14dbd0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:11.451 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.430+0000 7f508affd700 1 --2- 172.21.15.23:0/2275662777 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5064056d10 0x7f50640591d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f508c1420d0 tx=0x7f5080008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:11.451 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.436+0000 7f509215b700 1 --2- 172.21.15.23:0/2275662777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f508c14e0c0 0x7f508c14e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:11.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.436+0000 7f509215b700 1 -- 172.21.15.23:0/2275662777 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508c05c280 con 0x7f508c14e0c0 2020-11-20T11:46:11.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.436+0000 7f508b7fe700 1 --2- 172.21.15.23:0/2275662777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f508c14e0c0 0x7f508c14e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:11.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.436+0000 7f508b7fe700 1 -- 172.21.15.23:0/2275662777 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f508c0a0480 con 0x7f508c14e0c0 2020-11-20T11:46:11.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.437+0000 7f508b7fe700 1 --2- 172.21.15.23:0/2275662777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f508c14e0c0 0x7f508c14e4d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f507c015000 tx=0x7f507c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:11.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.437+0000 7f506b7fe700 1 -- 172.21.15.23:0/2275662777 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507c014cd0 con 0x7f508c14e0c0 2020-11-20T11:46:11.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.437+0000 7f506b7fe700 1 -- 172.21.15.23:0/2275662777 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4775 (secure 0 0 0) 0x7f507c014e30 con 0x7f508c14e0c0 2020-11-20T11:46:11.453 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.437+0000 7f506b7fe700 1 -- 172.21.15.23:0/2275662777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f508c14e0c0 msgr2=0x7f508c14e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:11.453 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.437+0000 7f506b7fe700 1 --2- 172.21.15.23:0/2275662777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f508c14e0c0 0x7f508c14e4d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f507c015000 tx=0x7f507c016000).stop 2020-11-20T11:46:11.453 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.437+0000 7f506b7fe700 1 -- 172.21.15.23:0/2275662777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f508c14e0c0 msgr2=0x7f508c14e4d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:46:11.453 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.437+0000 7f506b7fe700 1 --2- 172.21.15.23:0/2275662777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f508c14e0c0 0x7f508c14e4d0 secure :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0x7f507c015000 tx=0x7f507c016000).stop 2020-11-20T11:46:11.454 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:46:11.454 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:46:11.454 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:46:11.454 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:46:11.454 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 130, 2020-11-20T11:46:11.455 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:46:11.455 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:46:11.455 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:46:11.455 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:11.455 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 0, 2020-11-20T11:46:11.456 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:46:11.456 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:46:11.456 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:46:11.456 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:46:11.456 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:46:11.457 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:46:11.457 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:46:11.457 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:46:11.457 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:46:11.457 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:46:11.458 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:46:11.458 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:11.458 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:46:11.458 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:46:11.458 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:46:11.459 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:46:11.459 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:46:11.459 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:46:11.459 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:46:11.459 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:46:11.460 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:46:11.460 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:46:11.460 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:11.460 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:11.460 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:46:11.461 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:46:11.461 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:46:11.461 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:46:11.461 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:46:11.461 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:46:11.462 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:46:11.462 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:46:11.462 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:46:11.462 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:46:11.462 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:46:11.463 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:46:11.463 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:46:11.463 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:46:11.463 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:46:11.463 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:46:11.464 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:46:11.464 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:46:11.464 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:46:11.464 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:46:11.464 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:46:11.465 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:46:11.465 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:46:11.465 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:11.465 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:46:11.465 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:11.466 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:46:11.466 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:11.466 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:46:11.466 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:46:11.466 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:46:11.467 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:46:11.467 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:11.467 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:46:11.467 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:46:11.467 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:11.468 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:11.468 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:11.468 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:46:11.468 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:46:11.468 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:11.469 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:11.469 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:11.469 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:11.469 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:46:11.469 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:46:11.470 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:46:11.470 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:46:11.470 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:46:11.470 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:11.470 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:11.471 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:46:11.471 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:46:11.471 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:46:11.471 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:46:11.471 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:11.471 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:46:11.472 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:46:11.472 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:11.472 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:11.472 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:11.472 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:46:11.473 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:46:11.473 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:11.473 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:11.473 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:11.473 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:11.474 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:46:11.474 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:46:11.474 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:46:11.474 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:46:11.474 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:46:11.475 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:11.475 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:11.475 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:46:11.475 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:46:11.475 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:46:11.476 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:46:11.476 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:11.476 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:46:11.476 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:46:11.476 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:11.477 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:11.477 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:11.477 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:46:11.477 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:46:11.477 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:11.478 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:11.478 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:11.478 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:11.478 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:46:11.478 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:46:11.479 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:46:11.479 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:46:11.479 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:46:11.479 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:11.479 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:11.480 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:11.480 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:46:11.480 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:46:11.480 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:11.480 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:11.481 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:11.481 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:46:11.481 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:11.481 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:11.481 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:46:11.482 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:11.482 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:11.482 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:11.482 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:46:11.482 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:11.482 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:11.483 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:46:11.483 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:11.483 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:11.483 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:11.483 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:46:11.484 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:11.484 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:11.484 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:46:11.484 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:11.484 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:11.485 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:11.485 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 4 2020-11-20T11:46:11.485 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:11.485 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:11.485 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:46:11.486 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:11.486 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:11.486 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:11.486 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:46:11.486 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:11.487 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:11.487 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:11.487 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:46:11.487 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:46:11.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.445+0000 7f509215b700 1 -- 172.21.15.23:0/2275662777 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5064056d10 msgr2=0x7f50640591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:11.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.445+0000 7f509215b700 1 --2- 172.21.15.23:0/2275662777 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5064056d10 0x7f50640591d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f508c1420d0 tx=0x7f5080008040).stop 2020-11-20T11:46:11.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.445+0000 7f509215b700 1 -- 172.21.15.23:0/2275662777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f508c14a500 msgr2=0x7f508c14a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:11.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.445+0000 7f509215b700 1 --2- 172.21.15.23:0/2275662777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f508c14a500 0x7f508c14a960 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f508c0b2230 tx=0x7f5084015000).stop 2020-11-20T11:46:11.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.445+0000 7f508bfff700 1 -- 172.21.15.23:0/2275662777 reap_dead start 2020-11-20T11:46:11.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.445+0000 7f508bfff700 1 -- 172.21.15.23:0/2275662777 reap_dead start 2020-11-20T11:46:11.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.445+0000 7f509215b700 1 -- 172.21.15.23:0/2275662777 shutdown_connections 2020-11-20T11:46:11.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.445+0000 7f509215b700 1 -- 172.21.15.23:0/2275662777 >> 172.21.15.23:0/2275662777 conn(0x7f508c0683b0 msgr2=0x7f508c0accc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:11.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.445+0000 7f509215b700 1 -- 172.21.15.23:0/2275662777 shutdown_connections 2020-11-20T11:46:11.493 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.445+0000 7f509215b700 1 -- 172.21.15.23:0/2275662777 wait complete. 2020-11-20T11:46:11.494 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:46:11.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.636+0000 7f311000a700 1 Processor -- start 2020-11-20T11:46:11.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.636+0000 7f311000a700 1 -- start start 2020-11-20T11:46:11.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.636+0000 7f311000a700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3108061b90 0x7f31080a92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:11.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.636+0000 7f311000a700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31080a97d0 0x7f310813b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:11.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.636+0000 7f311000a700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31080a9c50 con 0x7f310813b700 2020-11-20T11:46:11.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.636+0000 7f311000a700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3108067190 con 0x7f3108061b90 2020-11-20T11:46:11.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.636+0000 7f311000a700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3108067430 con 0x7f31080a97d0 2020-11-20T11:46:11.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.636+0000 7f310dda6700 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f3108061b90 msgr2=0x7f31080a92e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:46:11.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.636+0000 7f310dda6700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3108061b90 0x7f31080a92e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:46:11.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.636+0000 7f310e5a7700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f310813b700 0x7f3108137e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:60446/0 (socket says 172.21.15.23:60446) 2020-11-20T11:46:11.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.636+0000 7f310e5a7700 1 -- 172.21.15.23:0/155156317 learned_addr learned my addr 172.21.15.23:0/155156317 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:46:11.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.636+0000 7f310d5a5700 1 --2- 172.21.15.23:0/155156317 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31080a97d0 0x7f310813b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:11.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.637+0000 7f310cda4700 1 -- 172.21.15.23:0/155156317 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626223016 0 0) 0x7f31080a9c50 con 0x7f310813b700 2020-11-20T11:46:11.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.637+0000 7f310cda4700 1 -- 172.21.15.23:0/155156317 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30f0003280 con 0x7f310813b700 2020-11-20T11:46:11.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.637+0000 7f310d5a5700 1 -- 172.21.15.23:0/155156317 >> v2:172.21.15.155:3300/0 conn(0x7f3108061b90 msgr2=0x7f31080a92e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:46:11.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.637+0000 7f310d5a5700 1 --2- 172.21.15.23:0/155156317 >> v2:172.21.15.155:3300/0 conn(0x7f3108061b90 0x7f31080a92e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:11.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.637+0000 7f310d5a5700 1 -- 172.21.15.23:0/155156317 >> v1:172.21.15.23:6789/0 conn(0x7f310813b700 legacy=0x7f3108137e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:11.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.637+0000 7f310d5a5700 1 -- 172.21.15.23:0/155156317 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3108067860 con 0x7f31080a97d0 2020-11-20T11:46:11.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.637+0000 7f310d5a5700 1 --2- 172.21.15.23:0/155156317 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31080a97d0 0x7f310813b210 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f30f8006fd0 tx=0x7f30f8006e30).ready entity=mon.2 client_cookie=432e11c27e68f53b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:11.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.637+0000 7f310cda4700 1 -- 172.21.15.23:0/155156317 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f80097e0 con 0x7f31080a97d0 2020-11-20T11:46:11.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.637+0000 7f310cda4700 1 -- 172.21.15.23:0/155156317 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30f8000e90 con 0x7f31080a97d0 2020-11-20T11:46:11.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.637+0000 7f310cda4700 1 -- 172.21.15.23:0/155156317 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f8009b50 con 0x7f31080a97d0 2020-11-20T11:46:11.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.638+0000 7f311000a700 1 -- 172.21.15.23:0/155156317 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31080a97d0 msgr2=0x7f310813b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:11.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.638+0000 7f311000a700 1 --2- 172.21.15.23:0/155156317 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31080a97d0 0x7f310813b210 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f30f8006fd0 tx=0x7f30f8006e30).stop 2020-11-20T11:46:11.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.638+0000 7f311000a700 1 -- 172.21.15.23:0/155156317 shutdown_connections 2020-11-20T11:46:11.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.638+0000 7f311000a700 1 --2- 172.21.15.23:0/155156317 >> v2:172.21.15.155:3300/0 conn(0x7f3108061b90 0x7f31080a92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:11.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.638+0000 7f311000a700 1 --2- 172.21.15.23:0/155156317 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31080a97d0 0x7f310813b210 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:11.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.638+0000 7f311000a700 1 -- 172.21.15.23:0/155156317 >> 172.21.15.23:0/155156317 conn(0x7f3108063640 msgr2=0x7f31080612c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:11.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.638+0000 7f311000a700 1 -- 172.21.15.23:0/155156317 shutdown_connections 2020-11-20T11:46:11.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.638+0000 7f311000a700 1 -- 172.21.15.23:0/155156317 wait complete. 2020-11-20T11:46:11.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.639+0000 7f311000a700 1 Processor -- start 2020-11-20T11:46:11.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.639+0000 7f311000a700 1 -- start start 2020-11-20T11:46:11.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.639+0000 7f311000a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3108061b90 0x7f310812f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:11.652 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.640+0000 7f311000a700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31080a97d0 0x7f31080ad7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:11.652 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.640+0000 7f311000a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f310813b700 0x7f31080abe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:11.652 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.640+0000 7f311000a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31080ad350 con 0x7f310813b700 2020-11-20T11:46:11.652 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.640+0000 7f311000a700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31080a6130 con 0x7f3108061b90 2020-11-20T11:46:11.653 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.640+0000 7f311000a700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31080a63d0 con 0x7f31080a97d0 2020-11-20T11:46:11.653 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.640+0000 7f310dda6700 1 -- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3108061b90 msgr2=0x7f310812f510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:46:11.653 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.640+0000 7f310dda6700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3108061b90 0x7f310812f510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:46:11.653 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.640+0000 7f310d5a5700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31080a97d0 0x7f31080ad7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:11.653 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.640+0000 7f310d5a5700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31080a97d0 0x7f31080ad7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:56444/0 (socket says 172.21.15.23:56444) 2020-11-20T11:46:11.654 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.640+0000 7f310d5a5700 1 -- 172.21.15.23:0/781142406 learned_addr learned my addr 172.21.15.23:0/781142406 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:46:11.654 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.640+0000 7f310e5a7700 1 --2- 172.21.15.23:0/781142406 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f310813b700 0x7f31080abe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:11.654 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.640+0000 7f310d5a5700 1 -- 172.21.15.23:0/781142406 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3108061b90 msgr2=0x7f310812f510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:46:11.654 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.640+0000 7f310d5a5700 1 --2- 172.21.15.23:0/781142406 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3108061b90 0x7f310812f510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:11.654 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.640+0000 7f310d5a5700 1 -- 172.21.15.23:0/781142406 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f310813b700 msgr2=0x7f31080abe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:11.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.640+0000 7f310d5a5700 1 --2- 172.21.15.23:0/781142406 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f310813b700 0x7f31080abe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:11.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.640+0000 7f310d5a5700 1 -- 172.21.15.23:0/781142406 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310812ab50 con 0x7f31080a97d0 2020-11-20T11:46:11.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.640+0000 7f310d5a5700 1 --2- 172.21.15.23:0/781142406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31080a97d0 0x7f31080ad7b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f30f8006e70 tx=0x7f30f8015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:11.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.640+0000 7f30f5ffb700 1 -- 172.21.15.23:0/781142406 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f8011cc0 con 0x7f31080a97d0 2020-11-20T11:46:11.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.640+0000 7f30f5ffb700 1 -- 172.21.15.23:0/781142406 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30f8011e20 con 0x7f31080a97d0 2020-11-20T11:46:11.656 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.640+0000 7f311000a700 1 -- 172.21.15.23:0/781142406 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31080a5e20 con 0x7f31080a97d0 2020-11-20T11:46:11.656 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.640+0000 7f30f5ffb700 1 -- 172.21.15.23:0/781142406 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f801e670 con 0x7f31080a97d0 2020-11-20T11:46:11.656 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.641+0000 7f311000a700 1 -- 172.21.15.23:0/781142406 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31080a1e30 con 0x7f31080a97d0 2020-11-20T11:46:11.656 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.641+0000 7f311000a700 1 --2- 172.21.15.23:0/781142406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f310805c070 0x7f31080ac320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:11.656 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.641+0000 7f311000a700 1 -- 172.21.15.23:0/781142406 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f310812fa00 con 0x7f310805c070 2020-11-20T11:46:11.657 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.642+0000 7f310dda6700 1 --2- 172.21.15.23:0/781142406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f310805c070 0x7f31080ac320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:11.657 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.642+0000 7f30f5ffb700 1 -- 172.21.15.23:0/781142406 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f30f8025030 con 0x7f31080a97d0 2020-11-20T11:46:11.657 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.642+0000 7f310dda6700 1 -- 172.21.15.23:0/781142406 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f31080ad1d0 con 0x7f310805c070 2020-11-20T11:46:11.657 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.642+0000 7f310dda6700 1 --2- 172.21.15.23:0/781142406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f310805c070 0x7f31080ac320 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f310400bfd0 tx=0x7f3104006ea0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:11.657 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.642+0000 7f30f5ffb700 1 --2- 172.21.15.23:0/781142406 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f30e0056d10 0x7f30e00591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:11.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.642+0000 7f30f5ffb700 1 -- 172.21.15.23:0/781142406 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f30f8071590 con 0x7f31080a97d0 2020-11-20T11:46:11.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.642+0000 7f310e5a7700 1 --2- 172.21.15.23:0/781142406 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f30e0056d10 0x7f30e00591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:11.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.643+0000 7f30f5ffb700 1 -- 172.21.15.23:0/781142406 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3104005320 con 0x7f310805c070 2020-11-20T11:46:11.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.643+0000 7f30f5ffb700 1 -- 172.21.15.23:0/781142406 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f3104005480 con 0x7f310805c070 2020-11-20T11:46:11.658 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.643+0000 7f30f5ffb700 1 -- 172.21.15.23:0/781142406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f310805c070 msgr2=0x7f31080ac320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:11.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.643+0000 7f30f5ffb700 1 --2- 172.21.15.23:0/781142406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f310805c070 0x7f31080ac320 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f310400bfd0 tx=0x7f3104006ea0).stop 2020-11-20T11:46:11.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.643+0000 7f30f5ffb700 1 -- 172.21.15.23:0/781142406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f310805c070 msgr2=0x7f31080ac320 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:46:11.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.643+0000 7f30f5ffb700 1 --2- 172.21.15.23:0/781142406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f310805c070 0x7f31080ac320 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:11.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.643+0000 7f310e5a7700 1 --2- 172.21.15.23:0/781142406 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f30e0056d10 0x7f30e00591d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f31000013f0 tx=0x7f3100009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:11.661 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.649+0000 7f311000a700 1 --2- 172.21.15.23:0/781142406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31080622d0 0x7f31080ac810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:11.661 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.649+0000 7f311000a700 1 -- 172.21.15.23:0/781142406 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31080acd00 con 0x7f31080622d0 2020-11-20T11:46:11.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.650+0000 7f310dda6700 1 --2- 172.21.15.23:0/781142406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31080622d0 0x7f31080ac810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:11.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.650+0000 7f310dda6700 1 -- 172.21.15.23:0/781142406 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f31080ad4c0 con 0x7f31080622d0 2020-11-20T11:46:11.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.650+0000 7f310dda6700 1 --2- 172.21.15.23:0/781142406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31080622d0 0x7f31080ac810 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f310400bfd0 tx=0x7f3104015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:11.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.650+0000 7f30f5ffb700 1 -- 172.21.15.23:0/781142406 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3104012c60 con 0x7f31080622d0 2020-11-20T11:46:11.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.651+0000 7f30f5ffb700 1 -- 172.21.15.23:0/781142406 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4773 (secure 0 0 0) 0x7f3104005110 con 0x7f31080622d0 2020-11-20T11:46:11.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.651+0000 7f30f5ffb700 1 -- 172.21.15.23:0/781142406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31080622d0 msgr2=0x7f31080ac810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:11.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.651+0000 7f30f5ffb700 1 --2- 172.21.15.23:0/781142406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31080622d0 0x7f31080ac810 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f310400bfd0 tx=0x7f3104015000).stop 2020-11-20T11:46:11.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.651+0000 7f30f5ffb700 1 -- 172.21.15.23:0/781142406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31080622d0 msgr2=0x7f31080ac810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:46:11.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.651+0000 7f30f5ffb700 1 --2- 172.21.15.23:0/781142406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31080622d0 0x7f31080ac810 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:11.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.651+0000 7f311000a700 1 -- 172.21.15.23:0/781142406 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f30e0056d10 msgr2=0x7f30e00591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:11.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.651+0000 7f311000a700 1 --2- 172.21.15.23:0/781142406 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f30e0056d10 0x7f30e00591d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f31000013f0 tx=0x7f3100009040).stop 2020-11-20T11:46:11.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.651+0000 7f311000a700 1 -- 172.21.15.23:0/781142406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31080a97d0 msgr2=0x7f31080ad7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:11.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.651+0000 7f311000a700 1 --2- 172.21.15.23:0/781142406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31080a97d0 0x7f31080ad7b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f30f8006e70 tx=0x7f30f8015000).stop 2020-11-20T11:46:11.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.651+0000 7f310e5a7700 1 -- 172.21.15.23:0/781142406 reap_dead start 2020-11-20T11:46:11.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.651+0000 7f311000a700 1 -- 172.21.15.23:0/781142406 shutdown_connections 2020-11-20T11:46:11.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.651+0000 7f311000a700 1 --2- 172.21.15.23:0/781142406 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f31080a97d0 0x7f31080ad7b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:11.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.651+0000 7f311000a700 1 -- 172.21.15.23:0/781142406 >> 172.21.15.23:0/781142406 conn(0x7f3108063640 msgr2=0x7f310812cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:11.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.651+0000 7f311000a700 1 -- 172.21.15.23:0/781142406 shutdown_connections 2020-11-20T11:46:11.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:11.651+0000 7f311000a700 1 -- 172.21.15.23:0/781142406 wait complete. 2020-11-20T11:46:11.665 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:46:11.666 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:46:11.666 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:46:11.666 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:46:11.666 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 130, 2020-11-20T11:46:11.667 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:46:11.667 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:46:11.667 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:46:11.667 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:11.667 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 0, 2020-11-20T11:46:11.668 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:46:11.668 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:46:11.668 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:46:11.668 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:46:11.668 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:46:11.669 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:46:11.669 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:46:11.669 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:46:11.669 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:46:11.669 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:46:11.670 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:46:11.670 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:11.670 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:46:11.670 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:46:11.670 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:46:11.671 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:46:11.671 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:46:11.671 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:46:11.671 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:46:11.671 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:46:11.671 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:46:11.672 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:46:11.672 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:11.672 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:11.672 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:46:11.672 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:46:11.673 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:46:11.673 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:46:11.673 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:46:11.673 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:46:11.673 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:46:11.674 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:46:11.674 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:46:11.674 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:46:11.674 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:46:11.674 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:46:11.675 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:46:11.675 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:46:11.675 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:46:11.675 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:46:11.675 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:46:11.676 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:46:11.676 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:46:11.676 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:46:11.676 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:46:11.676 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:46:11.676 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:46:11.677 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:11.677 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:46:11.677 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:11.677 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:46:11.677 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:11.678 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:46:11.678 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:46:11.678 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:46:11.678 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:46:11.678 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:11.679 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:46:11.679 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:46:11.679 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:11.679 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:11.679 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:11.680 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:46:11.680 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:46:11.680 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:11.680 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:11.680 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:11.681 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:11.681 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:46:11.681 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:46:11.681 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:46:11.681 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:46:11.682 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:46:11.682 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:11.682 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:11.682 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:46:11.682 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:46:11.682 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:46:11.683 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:46:11.683 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:11.683 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:46:11.683 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:46:11.683 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:11.684 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:11.684 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:11.684 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:46:11.684 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:46:11.684 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:11.685 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:11.685 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:11.685 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:11.685 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:46:11.685 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:46:11.686 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:46:11.686 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:46:11.686 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:46:11.686 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:11.686 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:11.686 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:46:11.687 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:46:11.687 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:46:11.687 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:46:11.687 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:11.687 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:46:11.688 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:46:11.688 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:11.688 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:11.688 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:11.688 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:46:11.689 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:46:11.689 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:11.689 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:11.689 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:11.689 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:11.690 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:46:11.690 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:46:11.690 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:46:11.690 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:46:11.690 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:46:11.691 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:11.691 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:11.691 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:11.691 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:46:11.691 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:46:11.692 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:11.692 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:11.692 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:11.692 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:46:11.692 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:11.693 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:11.693 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:46:11.693 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:11.693 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:11.693 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:11.694 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:46:11.694 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:11.694 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:11.694 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:46:11.694 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:11.695 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:11.695 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:11.695 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:46:11.695 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:11.695 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:11.695 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:46:11.696 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:11.696 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:11.696 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:11.696 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 4 2020-11-20T11:46:11.696 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:11.697 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:11.697 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:46:11.697 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:11.697 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:11.697 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:11.698 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:46:11.698 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:11.698 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:11.698 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:11.698 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:46:11.699 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:46:11.700 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2020-11-20T11:46:12.107 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:12.047+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4659.0:48041 4.2s0 4:56d4cd0f:::10000000393.00000002:head [write 3596288~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:13.140 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:13.117+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4659.0:48041 4.2s0 4:56d4cd0f:::10000000393.00000002:head [write 3596288~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:14.178 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:14.139+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4659.0:48401 4.2s0 4:49e75ffb:::100000003a1.00000004:head [write 503808~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:15.115 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:15.108+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4659.0:48401 4.2s0 4:49e75ffb:::100000003a1.00000004:head [write 503808~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:16.214 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:16.151+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4659.0:48462 4.2s0 4:562b1d3e:::100000003a1.00000012:head [write 1773568~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:17.209 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:17.175+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4659.0:48612 4.2s0 4:568ccee9:::100000003a4.0000000b:head [write 688128~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:18.192 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:18.149+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.4659.0:48857 4.2s0 4:46210aa0:::10000000353.00000002:head [write 36864~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:19.237 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:19.193+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4659.0:48857 4.2s0 4:46210aa0:::10000000353.00000002:head [write 36864~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:20.267 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:20.220+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4659.0:49198 4.2s0 4:4ae2880c:::10000000364.0000000b:head [write 3571712~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:21.261 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:21.231+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 23 slow ops, oldest is osd_op(client.4659.0:49198 4.2s0 4:4ae2880c:::10000000364.0000000b:head [write 3571712~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:22.265 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:22.231+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 25 slow ops, oldest is osd_op(client.4659.0:49764 4.2s0 4:5da044e0:::10000000364.00000018:head [write 2269184~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:23.231 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:23.228+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 29 slow ops, oldest is osd_op(client.4659.0:49764 4.2s0 4:5da044e0:::10000000364.00000018:head [write 2269184~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:24.300 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:24.263+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 33 slow ops, oldest is osd_op(client.4659.0:49764 4.2s0 4:5da044e0:::10000000364.00000018:head [write 2269184~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:25.317 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:25.313+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 39 slow ops, oldest is osd_op(client.4659.0:49764 4.2s0 4:5da044e0:::10000000364.00000018:head [write 2269184~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:26.377 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:26.374+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 45 slow ops, oldest is osd_op(client.4659.0:49999 4.2s0 4:542fa72b:::100000003c2.00000000:head [write 872448~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:27.441 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:27.435+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.4659.0:50248 4.2s0 4:44ade8ed:::100000003c8.00000002:head [write 69632~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:28.533 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:28.502+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4659.0:50248 4.2s0 4:44ade8ed:::100000003c8.00000002:head [write 69632~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:29.511 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:29.499+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 49 slow ops, oldest is osd_op(client.4659.0:50602 4.2s0 4:543b65c4:::10000000364.00000008:head [write 4014080~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:30.590 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:30.548+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 50 slow ops, oldest is osd_op(client.4659.0:50602 4.2s0 4:543b65c4:::10000000364.00000008:head [write 4014080~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:31.604 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:31.597+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 56 slow ops, oldest is osd_op(client.4659.0:51089 4.2s0 4:4c96ca10:::1000000034b.0000000d:head [write 1781760~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:31.701 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2020-11-20T11:46:31.701 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2020-11-20T11:46:31.701 INFO:tasks.ceph.mon.b:Restarting daemon 2020-11-20T11:46:31.702 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2020-11-20T11:46:31.704 INFO:tasks.ceph.mon.b:Started 2020-11-20T11:46:31.704 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:46:31.705 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:46:32.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.133+0000 7fe3ff784700 1 Processor -- start 2020-11-20T11:46:32.136 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.133+0000 7fe3ff784700 1 -- start start 2020-11-20T11:46:32.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.134+0000 7fe3ff784700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe3f80aae90 0x7fe3f81381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:32.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.134+0000 7fe3ff784700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe3f813aeb0 0x7fe3f80a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:32.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.134+0000 7fe3ff784700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f813e600 con 0x7fe3f80aaab0 2020-11-20T11:46:32.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.134+0000 7fe3ff784700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3f813e8a0 con 0x7fe3f813aeb0 2020-11-20T11:46:32.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.134+0000 7fe3ff784700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3f80ab6d0 con 0x7fe3f80aae90 2020-11-20T11:46:32.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.134+0000 7fe3fcd1f700 1 -- >> v2:172.21.15.155:3300/0 conn(0x7fe3f80aae90 msgr2=0x7fe3f81381a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:46:32.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.134+0000 7fe3fd520700 1 -- >> v1:172.21.15.155:6789/0 conn(0x7fe3f80aaab0 legacy=0x7fe3f813ed40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:46:32.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.134+0000 7fe3fcd1f700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe3f80aae90 0x7fe3f81381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:46:32.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.134+0000 7fe3fdd21700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe3f813aeb0 0x7fe3f80a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:32.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.134+0000 7fe3fdd21700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe3f813aeb0 0x7fe3f80a8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46220/0 (socket says 172.21.15.23:46220) 2020-11-20T11:46:32.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.134+0000 7fe3fdd21700 1 -- 172.21.15.23:0/810721464 learned_addr learned my addr 172.21.15.23:0/810721464 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:46:32.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.134+0000 7fe3fdd21700 1 -- 172.21.15.23:0/810721464 >> v2:172.21.15.155:3300/0 conn(0x7fe3f80aae90 msgr2=0x7fe3f81381a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:46:32.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.134+0000 7fe3fdd21700 1 --2- 172.21.15.23:0/810721464 >> v2:172.21.15.155:3300/0 conn(0x7fe3f80aae90 0x7fe3f81381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:32.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.134+0000 7fe3fdd21700 1 -- 172.21.15.23:0/810721464 >> v1:172.21.15.155:6789/0 conn(0x7fe3f80aaab0 legacy=0x7fe3f813ed40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:46:32.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.134+0000 7fe3fdd21700 1 -- 172.21.15.23:0/810721464 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f8138690 con 0x7fe3f813aeb0 2020-11-20T11:46:32.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.134+0000 7fe3fdd21700 1 --2- 172.21.15.23:0/810721464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3f813aeb0 0x7fe3f80a8c60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fe3f80ab930 tx=0x7fe3f00092d0).ready entity=mon.0 client_cookie=60b628b0ab7bbff7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:32.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.134+0000 7fe3e7fff700 1 -- 172.21.15.23:0/810721464 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f000bbb0 con 0x7fe3f813aeb0 2020-11-20T11:46:32.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.134+0000 7fe3e7fff700 1 -- 172.21.15.23:0/810721464 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f000bd10 con 0x7fe3f813aeb0 2020-11-20T11:46:32.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.134+0000 7fe3e7fff700 1 -- 172.21.15.23:0/810721464 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0011610 con 0x7fe3f813aeb0 2020-11-20T11:46:32.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.135+0000 7fe3ff784700 1 -- 172.21.15.23:0/810721464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3f813aeb0 msgr2=0x7fe3f80a8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:32.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.135+0000 7fe3ff784700 1 --2- 172.21.15.23:0/810721464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3f813aeb0 0x7fe3f80a8c60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fe3f80ab930 tx=0x7fe3f00092d0).stop 2020-11-20T11:46:32.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.135+0000 7fe3ff784700 1 -- 172.21.15.23:0/810721464 shutdown_connections 2020-11-20T11:46:32.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.135+0000 7fe3ff784700 1 --2- 172.21.15.23:0/810721464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3f813aeb0 0x7fe3f80a8c60 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:32.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.135+0000 7fe3ff784700 1 --2- 172.21.15.23:0/810721464 >> v2:172.21.15.155:3300/0 conn(0x7fe3f80aae90 0x7fe3f81381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:32.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.135+0000 7fe3ff784700 1 -- 172.21.15.23:0/810721464 >> 172.21.15.23:0/810721464 conn(0x7fe3f80678e0 msgr2=0x7fe3f8067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:32.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.135+0000 7fe3ff784700 1 -- 172.21.15.23:0/810721464 shutdown_connections 2020-11-20T11:46:32.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.135+0000 7fe3ff784700 1 -- 172.21.15.23:0/810721464 wait complete. 2020-11-20T11:46:32.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.136+0000 7fe3ff784700 1 Processor -- start 2020-11-20T11:46:32.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.136+0000 7fe3ff784700 1 -- start start 2020-11-20T11:46:32.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.137+0000 7fe3ff784700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3f80aaab0 0x7fe3f80a7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:32.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.137+0000 7fe3ff784700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe3f80aae90 0x7fe3f80a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:32.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.137+0000 7fe3ff784700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3f809e6b0 0x7fe3f80a4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:32.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.137+0000 7fe3ff784700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f80a7300 con 0x7fe3f80aaab0 2020-11-20T11:46:32.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.137+0000 7fe3ff784700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f809eb50 con 0x7fe3f809e6b0 2020-11-20T11:46:32.148 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.137+0000 7fe3ff784700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f809edf0 con 0x7fe3f80aae90 2020-11-20T11:46:32.148 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.137+0000 7fe3fdd21700 1 -- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3f809e6b0 msgr2=0x7fe3f80a4410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:46:32.148 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.137+0000 7fe3fdd21700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3f809e6b0 0x7fe3f80a4410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:46:32.148 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.137+0000 7fe3fd520700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3f80aaab0 0x7fe3f80a7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:32.149 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.137+0000 7fe3fd520700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3f80aaab0 0x7fe3f80a7630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46222/0 (socket says 172.21.15.23:46222) 2020-11-20T11:46:32.149 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.137+0000 7fe3fd520700 1 -- 172.21.15.23:0/488097437 learned_addr learned my addr 172.21.15.23:0/488097437 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:46:32.149 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.137+0000 7fe3fcd1f700 1 --2- 172.21.15.23:0/488097437 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe3f80aae90 0x7fe3f80a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:32.149 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.137+0000 7fe3fd520700 1 -- 172.21.15.23:0/488097437 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe3f80aae90 msgr2=0x7fe3f80a6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:32.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.137+0000 7fe3fd520700 1 --2- 172.21.15.23:0/488097437 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe3f80aae90 0x7fe3f80a6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:32.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.137+0000 7fe3fd520700 1 -- 172.21.15.23:0/488097437 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3f809e6b0 msgr2=0x7fe3f80a4410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:46:32.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.137+0000 7fe3fd520700 1 --2- 172.21.15.23:0/488097437 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3f809e6b0 0x7fe3f80a4410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:32.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.137+0000 7fe3fd520700 1 -- 172.21.15.23:0/488097437 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f814a0e0 con 0x7fe3f80aaab0 2020-11-20T11:46:32.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.137+0000 7fe3fd520700 1 --2- 172.21.15.23:0/488097437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3f80aaab0 0x7fe3f80a7630 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fe3f4009710 tx=0x7fe3f4006e50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:32.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.137+0000 7fe3e57fa700 1 -- 172.21.15.23:0/488097437 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f40050d0 con 0x7fe3f80aaab0 2020-11-20T11:46:32.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.137+0000 7fe3e57fa700 1 -- 172.21.15.23:0/488097437 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f4005250 con 0x7fe3f80aaab0 2020-11-20T11:46:32.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.137+0000 7fe3ff784700 1 -- 172.21.15.23:0/488097437 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3f80a5330 con 0x7fe3f80aaab0 2020-11-20T11:46:32.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.137+0000 7fe3e57fa700 1 -- 172.21.15.23:0/488097437 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f4003850 con 0x7fe3f80aaab0 2020-11-20T11:46:32.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.138+0000 7fe3ff784700 1 -- 172.21.15.23:0/488097437 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3f80b1080 con 0x7fe3f80aaab0 2020-11-20T11:46:32.152 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.138+0000 7fe3ff784700 1 -- 172.21.15.23:0/488097437 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3f805c7d0 con 0x7fe3f80aaab0 2020-11-20T11:46:32.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.139+0000 7fe3e57fa700 1 -- 172.21.15.23:0/488097437 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe3f400f070 con 0x7fe3f80aaab0 2020-11-20T11:46:32.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.139+0000 7fe3e57fa700 1 --2- 172.21.15.23:0/488097437 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe3d0056d10 0x7fe3d00591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:32.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.139+0000 7fe3e57fa700 1 -- 172.21.15.23:0/488097437 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fe3f406f840 con 0x7fe3f80aaab0 2020-11-20T11:46:32.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.139+0000 7fe3fcd1f700 1 --2- 172.21.15.23:0/488097437 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe3d0056d10 0x7fe3d00591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:32.153 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.140+0000 7fe3fcd1f700 1 --2- 172.21.15.23:0/488097437 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe3d0056d10 0x7fe3d00591d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fe3e8006fd0 tx=0x7fe3e8006e10).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:32.154 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.144+0000 7fe3e57fa700 1 -- 172.21.15.23:0/488097437 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe3f4044b30 con 0x7fe3f80aaab0 2020-11-20T11:46:32.409 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.406+0000 7fe3ff784700 1 -- 172.21.15.23:0/488097437 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe3f8044710 con 0x7fe3f80aaab0 2020-11-20T11:46:32.410 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.407+0000 7fe3e57fa700 1 -- 172.21.15.23:0/488097437 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1473 (secure 0 0 0) 0x7fe3f4048150 con 0x7fe3f80aaab0 2020-11-20T11:46:32.410 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":130,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:46:32.411 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.408+0000 7fe3ff784700 1 -- 172.21.15.23:0/488097437 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe3d0056d10 msgr2=0x7fe3d00591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:32.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.408+0000 7fe3ff784700 1 --2- 172.21.15.23:0/488097437 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe3d0056d10 0x7fe3d00591d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fe3e8006fd0 tx=0x7fe3e8006e10).stop 2020-11-20T11:46:32.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.408+0000 7fe3ff784700 1 -- 172.21.15.23:0/488097437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3f80aaab0 msgr2=0x7fe3f80a7630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:32.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.408+0000 7fe3ff784700 1 --2- 172.21.15.23:0/488097437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3f80aaab0 0x7fe3f80a7630 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fe3f4009710 tx=0x7fe3f4006e50).stop 2020-11-20T11:46:32.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.408+0000 7fe3ff784700 1 -- 172.21.15.23:0/488097437 shutdown_connections 2020-11-20T11:46:32.412 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.408+0000 7fe3ff784700 1 --2- 172.21.15.23:0/488097437 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe3f809e6b0 0x7fe3f80a4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:32.413 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.408+0000 7fe3ff784700 1 --2- 172.21.15.23:0/488097437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3f80aaab0 0x7fe3f80a7630 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:32.413 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.408+0000 7fe3ff784700 1 --2- 172.21.15.23:0/488097437 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe3d0056d10 0x7fe3d00591d0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:32.413 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.408+0000 7fe3ff784700 1 --2- 172.21.15.23:0/488097437 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe3f80aae90 0x7fe3f80a6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:32.413 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.408+0000 7fe3ff784700 1 -- 172.21.15.23:0/488097437 >> 172.21.15.23:0/488097437 conn(0x7fe3f80678e0 msgr2=0x7fe3f8068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:32.414 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.408+0000 7fe3ff784700 1 -- 172.21.15.23:0/488097437 shutdown_connections 2020-11-20T11:46:32.414 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:32.408+0000 7fe3ff784700 1 -- 172.21.15.23:0/488097437 wait complete. 2020-11-20T11:46:32.636 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:32.610+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 62 slow ops, oldest is osd_op(client.4659.0:51089 4.2s0 4:4c96ca10:::1000000034b.0000000d:head [write 1781760~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:33.651 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:33.629+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 62 slow ops, oldest is osd_op(client.4659.0:51089 4.2s0 4:4c96ca10:::1000000034b.0000000d:head [write 1781760~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:34.708 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:34.666+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 61 slow ops, oldest is osd_op(client.4659.0:51089 4.2s0 4:4c96ca10:::1000000034b.0000000d:head [write 1781760~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:35.426 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:46:35.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.574+0000 7f0598fd7700 1 Processor -- start 2020-11-20T11:46:35.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.574+0000 7f0598fd7700 1 -- start start 2020-11-20T11:46:35.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.575+0000 7f0598fd7700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f05940aaab0 0x7f059413ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:35.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.575+0000 7f0598fd7700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f059413aeb0 0x7f05940a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:35.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.575+0000 7f0598fd7700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05941388b0 con 0x7f05940aae90 2020-11-20T11:46:35.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.575+0000 7f0598fd7700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f059413e600 con 0x7f05940aaab0 2020-11-20T11:46:35.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.575+0000 7f0598fd7700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f059413e8a0 con 0x7f059413aeb0 2020-11-20T11:46:35.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.575+0000 7f059259c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f05940aaab0 0x7f059413ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:35.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.575+0000 7f0591d9b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f05940aae90 0x7f05941381a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:60482/0 (socket says 172.21.15.23:60482) 2020-11-20T11:46:35.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.575+0000 7f0591d9b700 1 -- 172.21.15.23:0/2617309429 learned_addr learned my addr 172.21.15.23:0/2617309429 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:46:35.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.575+0000 7f0592d9d700 1 --2- 172.21.15.23:0/2617309429 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f059413aeb0 0x7f05940a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:35.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.576+0000 7f059259c700 1 -- 172.21.15.23:0/2617309429 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f059413aeb0 msgr2=0x7f05940a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:46:35.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.576+0000 7f059259c700 1 --2- 172.21.15.23:0/2617309429 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f059413aeb0 0x7f05940a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:35.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.576+0000 7f059259c700 1 -- 172.21.15.23:0/2617309429 >> v1:172.21.15.23:6789/0 conn(0x7f05940aae90 legacy=0x7f05941381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:35.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.576+0000 7f059259c700 1 -- 172.21.15.23:0/2617309429 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05940ab6d0 con 0x7f05940aaab0 2020-11-20T11:46:35.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.576+0000 7f059159a700 1 -- 172.21.15.23:0/2617309429 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3959783266 0 0) 0x7f05941388b0 con 0x7f05940aae90 2020-11-20T11:46:35.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.576+0000 7f059259c700 1 --2- 172.21.15.23:0/2617309429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05940aaab0 0x7f059413ed40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f0584006fd0 tx=0x7f0584006e30).ready entity=mon.0 client_cookie=86d55673a4ea6dbc server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:35.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.576+0000 7f059159a700 1 -- 172.21.15.23:0/2617309429 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05840097e0 con 0x7f05940aaab0 2020-11-20T11:46:35.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.576+0000 7f059159a700 1 -- 172.21.15.23:0/2617309429 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0584000e90 con 0x7f05940aaab0 2020-11-20T11:46:35.584 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.576+0000 7f059159a700 1 -- 172.21.15.23:0/2617309429 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0584009b50 con 0x7f05940aaab0 2020-11-20T11:46:35.585 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.577+0000 7f0598fd7700 1 -- 172.21.15.23:0/2617309429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05940aaab0 msgr2=0x7f059413ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:35.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.577+0000 7f0598fd7700 1 --2- 172.21.15.23:0/2617309429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05940aaab0 0x7f059413ed40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f0584006fd0 tx=0x7f0584006e30).stop 2020-11-20T11:46:35.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.577+0000 7f0598fd7700 1 -- 172.21.15.23:0/2617309429 shutdown_connections 2020-11-20T11:46:35.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.577+0000 7f0598fd7700 1 --2- 172.21.15.23:0/2617309429 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f059413aeb0 0x7f05940a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:35.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.577+0000 7f0598fd7700 1 --2- 172.21.15.23:0/2617309429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05940aaab0 0x7f059413ed40 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:35.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.577+0000 7f0598fd7700 1 -- 172.21.15.23:0/2617309429 >> 172.21.15.23:0/2617309429 conn(0x7f05940678e0 msgr2=0x7f0594067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:35.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.582+0000 7f0598fd7700 1 -- 172.21.15.23:0/2617309429 shutdown_connections 2020-11-20T11:46:35.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.582+0000 7f0598fd7700 1 -- 172.21.15.23:0/2617309429 wait complete. 2020-11-20T11:46:35.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.583+0000 7f0598fd7700 1 Processor -- start 2020-11-20T11:46:35.593 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.583+0000 7f0598fd7700 1 -- start start 2020-11-20T11:46:35.593 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.584+0000 7f0598fd7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05940aaab0 0x7f05940a70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:35.593 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.584+0000 7f0598fd7700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05940aae90 0x7f05941356a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:35.594 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.584+0000 7f0598fd7700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0594135b90 0x7f059413d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:35.594 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.584+0000 7f0598fd7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059413e320 con 0x7f05940aaab0 2020-11-20T11:46:35.595 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.584+0000 7f0598fd7700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0594136010 con 0x7f05940aae90 2020-11-20T11:46:35.595 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.584+0000 7f0598fd7700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05940a5620 con 0x7f0594135b90 2020-11-20T11:46:35.595 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.584+0000 7f059259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05940aaab0 0x7f05940a70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:35.596 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.584+0000 7f059259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05940aaab0 0x7f05940a70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46236/0 (socket says 172.21.15.23:46236) 2020-11-20T11:46:35.596 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.584+0000 7f059259c700 1 -- 172.21.15.23:0/4021018829 learned_addr learned my addr 172.21.15.23:0/4021018829 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:46:35.597 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.584+0000 7f0591d9b700 1 --2- 172.21.15.23:0/4021018829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05940aae90 0x7f05941356a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:35.597 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.584+0000 7f0592d9d700 1 --2- 172.21.15.23:0/4021018829 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0594135b90 0x7f059413d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:35.597 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.584+0000 7f0591d9b700 1 -- 172.21.15.23:0/4021018829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05940aae90 msgr2=0x7f05941356a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2020-11-20T11:46:35.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.584+0000 7f0591d9b700 1 -- 172.21.15.23:0/4021018829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05940aae90 msgr2=0x7f05941356a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:46:35.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.584+0000 7f0591d9b700 1 --2- 172.21.15.23:0/4021018829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05940aae90 0x7f05941356a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:46:35.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.584+0000 7f0591d9b700 1 --2- 172.21.15.23:0/4021018829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05940aae90 0x7f05941356a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:46:35.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.585+0000 7f0592d9d700 1 -- 172.21.15.23:0/4021018829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05940aae90 msgr2=0x7f05941356a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:46:35.598 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.585+0000 7f0592d9d700 1 --2- 172.21.15.23:0/4021018829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05940aae90 0x7f05941356a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:35.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.585+0000 7f0592d9d700 1 -- 172.21.15.23:0/4021018829 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05940aaab0 msgr2=0x7f05940a70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:35.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.585+0000 7f0592d9d700 1 --2- 172.21.15.23:0/4021018829 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05940aaab0 0x7f05940a70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:35.599 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.585+0000 7f0592d9d700 1 -- 172.21.15.23:0/4021018829 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05940a58b0 con 0x7f0594135b90 2020-11-20T11:46:35.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.585+0000 7f059259c700 1 --2- 172.21.15.23:0/4021018829 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05940aaab0 0x7f05940a70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:46:35.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.585+0000 7f0592d9d700 1 --2- 172.21.15.23:0/4021018829 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0594135b90 0x7f059413d4a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f058c00ec20 tx=0x7f058c009220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:35.600 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.585+0000 7f05827fc700 1 -- 172.21.15.23:0/4021018829 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c00c920 con 0x7f0594135b90 2020-11-20T11:46:35.601 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.585+0000 7f05827fc700 1 -- 172.21.15.23:0/4021018829 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f058c00ca80 con 0x7f0594135b90 2020-11-20T11:46:35.601 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.585+0000 7f0598fd7700 1 -- 172.21.15.23:0/4021018829 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05940a7de0 con 0x7f0594135b90 2020-11-20T11:46:35.602 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.585+0000 7f05827fc700 1 -- 172.21.15.23:0/4021018829 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c00ce20 con 0x7f0594135b90 2020-11-20T11:46:35.602 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.585+0000 7f0598fd7700 1 -- 172.21.15.23:0/4021018829 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05940a78d0 con 0x7f0594135b90 2020-11-20T11:46:35.603 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.587+0000 7f05827fc700 1 -- 172.21.15.23:0/4021018829 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f058c022070 con 0x7f0594135b90 2020-11-20T11:46:35.604 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.587+0000 7f05827fc700 1 --2- 172.21.15.23:0/4021018829 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0568056d60 0x7f0568059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:35.604 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.587+0000 7f05827fc700 1 -- 172.21.15.23:0/4021018829 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f058c0706c0 con 0x7f0594135b90 2020-11-20T11:46:35.604 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.587+0000 7f056ffff700 1 -- 172.21.15.23:0/4021018829 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f059405c7d0 con 0x7f0594135b90 2020-11-20T11:46:35.605 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.587+0000 7f059259c700 1 --2- 172.21.15.23:0/4021018829 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0568056d60 0x7f0568059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:35.605 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.588+0000 7f059259c700 1 --2- 172.21.15.23:0/4021018829 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0568056d60 0x7f0568059220 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f0584009fd0 tx=0x7f0584015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:35.606 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.593+0000 7f05827fc700 1 -- 172.21.15.23:0/4021018829 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f058c016070 con 0x7f0594135b90 2020-11-20T11:46:35.763 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:35.722+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 66 slow ops, oldest is osd_op(client.4659.0:51418 4.2s0 4:439f8c72:::1000000034d.00000010:head [write 1150976~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:35.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.866+0000 7f056ffff700 1 -- 172.21.15.23:0/4021018829 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f05940a5a40 con 0x7f0594135b90 2020-11-20T11:46:35.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.866+0000 7f05827fc700 1 -- 172.21.15.23:0/4021018829 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1473 (secure 0 0 0) 0x7f058c029080 con 0x7f0594135b90 2020-11-20T11:46:35.870 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":130,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:46:35.871 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.868+0000 7f056ffff700 1 -- 172.21.15.23:0/4021018829 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0568056d60 msgr2=0x7f0568059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:35.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.868+0000 7f056ffff700 1 --2- 172.21.15.23:0/4021018829 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0568056d60 0x7f0568059220 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f0584009fd0 tx=0x7f0584015000).stop 2020-11-20T11:46:35.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.868+0000 7f056ffff700 1 -- 172.21.15.23:0/4021018829 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0594135b90 msgr2=0x7f059413d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:35.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.868+0000 7f056ffff700 1 --2- 172.21.15.23:0/4021018829 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0594135b90 0x7f059413d4a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f058c00ec20 tx=0x7f058c009220).stop 2020-11-20T11:46:35.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.868+0000 7f056ffff700 1 -- 172.21.15.23:0/4021018829 shutdown_connections 2020-11-20T11:46:35.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.868+0000 7f056ffff700 1 --2- 172.21.15.23:0/4021018829 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0568056d60 0x7f0568059220 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:35.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.868+0000 7f056ffff700 1 --2- 172.21.15.23:0/4021018829 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0594135b90 0x7f059413d4a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:35.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.868+0000 7f056ffff700 1 --2- 172.21.15.23:0/4021018829 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05940aaab0 0x7f05940a70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:35.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.868+0000 7f056ffff700 1 --2- 172.21.15.23:0/4021018829 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05940aae90 0x7f05941356a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:35.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.868+0000 7f056ffff700 1 -- 172.21.15.23:0/4021018829 >> 172.21.15.23:0/4021018829 conn(0x7f05940678e0 msgr2=0x7f0594136a40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:35.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.869+0000 7f056ffff700 1 -- 172.21.15.23:0/4021018829 shutdown_connections 2020-11-20T11:46:35.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:35.869+0000 7f056ffff700 1 -- 172.21.15.23:0/4021018829 wait complete. 2020-11-20T11:46:36.804 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:36.766+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 73 slow ops, oldest is osd_op(client.4659.0:51418 4.2s0 4:439f8c72:::1000000034d.00000010:head [write 1150976~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:37.769 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:37.760+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 73 slow ops, oldest is osd_op(client.4659.0:51517 4.2s0 4:5ef0fbf2:::10000000378.00000001:head [write 3317760~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:38.851 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:38.830+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 77 slow ops, oldest is osd_op(client.4659.0:51713 4.2s0 4:5c05d725:::1000000036d.00000011:head [write 741376~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:38.888 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:46:39.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.648+0000 7fb85fd16700 1 Processor -- start 2020-11-20T11:46:39.654 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.648+0000 7fb85fd16700 1 -- start start 2020-11-20T11:46:39.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.649+0000 7fb85fd16700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb8580aa0a0 0x7fb858139700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:39.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.649+0000 7fb85fd16700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8580aabe0 con 0x7fb858060ca0 2020-11-20T11:46:39.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.649+0000 7fb85fd16700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8580aae80 con 0x7fb8580608c0 2020-11-20T11:46:39.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.649+0000 7fb85fd16700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8580ab300 con 0x7fb8580aa0a0 2020-11-20T11:46:39.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.649+0000 7fb85dab2700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fb8580608c0 0x7fb8580a6d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:38470/0 (socket says 172.21.15.23:38470) 2020-11-20T11:46:39.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.649+0000 7fb85dab2700 1 -- 172.21.15.23:0/1253202133 learned_addr learned my addr 172.21.15.23:0/1253202133 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:46:39.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.649+0000 7fb85e2b3700 1 --2- 172.21.15.23:0/1253202133 >> v2:172.21.15.23:3300/0 conn(0x7fb8580aa0a0 0x7fb858139700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:39.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.649+0000 7fb85cab0700 1 -- 172.21.15.23:0/1253202133 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 129216289 0 0) 0x7fb8580aae80 con 0x7fb8580608c0 2020-11-20T11:46:39.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.649+0000 7fb85cab0700 1 -- 172.21.15.23:0/1253202133 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb83c003280 con 0x7fb8580608c0 2020-11-20T11:46:39.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.650+0000 7fb85cab0700 1 -- 172.21.15.23:0/1253202133 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4100011961 0 0) 0x7fb8580aabe0 con 0x7fb858060ca0 2020-11-20T11:46:39.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.650+0000 7fb85cab0700 1 -- 172.21.15.23:0/1253202133 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8580aae80 con 0x7fb858060ca0 2020-11-20T11:46:39.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.650+0000 7fb85e2b3700 1 -- 172.21.15.23:0/1253202133 >> v1:172.21.15.155:6789/0 conn(0x7fb8580608c0 legacy=0x7fb8580a6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:39.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.650+0000 7fb85e2b3700 1 -- 172.21.15.23:0/1253202133 >> v1:172.21.15.23:6789/0 conn(0x7fb858060ca0 legacy=0x7fb858138ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:39.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.650+0000 7fb85e2b3700 1 -- 172.21.15.23:0/1253202133 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8580a26c0 con 0x7fb8580aa0a0 2020-11-20T11:46:39.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.650+0000 7fb85e2b3700 1 --2- 172.21.15.23:0/1253202133 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8580aa0a0 0x7fb858139700 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fb850009ce0 tx=0x7fb850009220).ready entity=mon.0 client_cookie=aae4f0410d217246 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:39.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.650+0000 7fb85cab0700 1 -- 172.21.15.23:0/1253202133 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85000b990 con 0x7fb8580aa0a0 2020-11-20T11:46:39.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.650+0000 7fb85cab0700 1 -- 172.21.15.23:0/1253202133 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb85000baf0 con 0x7fb8580aa0a0 2020-11-20T11:46:39.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.650+0000 7fb85cab0700 1 -- 172.21.15.23:0/1253202133 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85000be70 con 0x7fb8580aa0a0 2020-11-20T11:46:39.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.651+0000 7fb85fd16700 1 -- 172.21.15.23:0/1253202133 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8580aa0a0 msgr2=0x7fb858139700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:39.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.651+0000 7fb85fd16700 1 --2- 172.21.15.23:0/1253202133 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8580aa0a0 0x7fb858139700 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fb850009ce0 tx=0x7fb850009220).stop 2020-11-20T11:46:39.675 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.651+0000 7fb85fd16700 1 -- 172.21.15.23:0/1253202133 shutdown_connections 2020-11-20T11:46:39.675 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.651+0000 7fb85fd16700 1 --2- 172.21.15.23:0/1253202133 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8580aa0a0 0x7fb858139700 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:39.675 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.651+0000 7fb85fd16700 1 -- 172.21.15.23:0/1253202133 >> 172.21.15.23:0/1253202133 conn(0x7fb858066750 msgr2=0x7fb8580601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:39.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.652+0000 7fb85fd16700 1 -- 172.21.15.23:0/1253202133 shutdown_connections 2020-11-20T11:46:39.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.653+0000 7fb85fd16700 1 -- 172.21.15.23:0/1253202133 wait complete. 2020-11-20T11:46:39.676 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.654+0000 7fb85fd16700 1 Processor -- start 2020-11-20T11:46:39.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.654+0000 7fb85fd16700 1 -- start start 2020-11-20T11:46:39.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.655+0000 7fb85fd16700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8580608c0 0x7fb858138f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:39.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.655+0000 7fb85fd16700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb858060ca0 0x7fb85809d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:39.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.655+0000 7fb85fd16700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8580aa0a0 0x7fb85812bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:39.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.655+0000 7fb85fd16700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85809f790 con 0x7fb858060ca0 2020-11-20T11:46:39.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.655+0000 7fb85fd16700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85809f980 con 0x7fb8580aa0a0 2020-11-20T11:46:39.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.655+0000 7fb85fd16700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb85809fc20 con 0x7fb8580608c0 2020-11-20T11:46:39.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.655+0000 7fb85dab2700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8580608c0 0x7fb858138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:39.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.655+0000 7fb85e2b3700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8580aa0a0 0x7fb85812bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:39.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.655+0000 7fb85e2b3700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8580aa0a0 0x7fb85812bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:39152/0 (socket says 172.21.15.23:39152) 2020-11-20T11:46:39.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.655+0000 7fb85e2b3700 1 -- 172.21.15.23:0/4133722759 learned_addr learned my addr 172.21.15.23:0/4133722759 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:46:39.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.655+0000 7fb85e2b3700 1 -- 172.21.15.23:0/4133722759 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8580608c0 msgr2=0x7fb858138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:39.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.655+0000 7fb85e2b3700 1 --2- 172.21.15.23:0/4133722759 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8580608c0 0x7fb858138f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:39.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.656+0000 7fb85e2b3700 1 -- 172.21.15.23:0/4133722759 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb858060ca0 msgr2=0x7fb85809d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:39.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.656+0000 7fb85e2b3700 1 --2- 172.21.15.23:0/4133722759 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb858060ca0 0x7fb85809d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:39.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.656+0000 7fb85e2b3700 1 -- 172.21.15.23:0/4133722759 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb85809fec0 con 0x7fb8580aa0a0 2020-11-20T11:46:39.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.656+0000 7fb85e2b3700 1 --2- 172.21.15.23:0/4133722759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8580aa0a0 0x7fb85812bba0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fb8500090b0 tx=0x7fb850016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:39.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.656+0000 7fb84dffb700 1 -- 172.21.15.23:0/4133722759 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb850011e50 con 0x7fb8580aa0a0 2020-11-20T11:46:39.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.656+0000 7fb84dffb700 1 -- 172.21.15.23:0/4133722759 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb850010a50 con 0x7fb8580aa0a0 2020-11-20T11:46:39.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.656+0000 7fb85fd16700 1 -- 172.21.15.23:0/4133722759 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb85809d990 con 0x7fb8580aa0a0 2020-11-20T11:46:39.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.656+0000 7fb85fd16700 1 -- 172.21.15.23:0/4133722759 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb85812c2d0 con 0x7fb8580aa0a0 2020-11-20T11:46:39.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.656+0000 7fb84dffb700 1 -- 172.21.15.23:0/4133722759 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb850010bc0 con 0x7fb8580aa0a0 2020-11-20T11:46:39.684 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.657+0000 7fb85fd16700 1 -- 172.21.15.23:0/4133722759 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8580601b0 con 0x7fb8580aa0a0 2020-11-20T11:46:39.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.658+0000 7fb84dffb700 1 -- 172.21.15.23:0/4133722759 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb850017070 con 0x7fb8580aa0a0 2020-11-20T11:46:39.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.658+0000 7fb84dffb700 1 --2- 172.21.15.23:0/4133722759 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb838056d60 0x7fb838059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:39.685 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.658+0000 7fb84dffb700 1 -- 172.21.15.23:0/4133722759 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fb850072a40 con 0x7fb8580aa0a0 2020-11-20T11:46:39.686 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.658+0000 7fb85dab2700 1 --2- 172.21.15.23:0/4133722759 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb838056d60 0x7fb838059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:39.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.659+0000 7fb85dab2700 1 --2- 172.21.15.23:0/4133722759 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb838056d60 0x7fb838059220 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fb854007fd0 tx=0x7fb854009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:39.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.662+0000 7fb84dffb700 1 -- 172.21.15.23:0/4133722759 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fb850047cb0 con 0x7fb8580aa0a0 2020-11-20T11:46:39.880 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:39.820+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4659.0:51845 4.2s0 4:5ef0fbf2:::10000000378.00000001:head [write 2260992~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:39.934 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.931+0000 7fb85fd16700 1 -- 172.21.15.23:0/4133722759 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb8580a0050 con 0x7fb8580aa0a0 2020-11-20T11:46:39.934 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.931+0000 7fb84dffb700 1 -- 172.21.15.23:0/4133722759 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7fb8580a0050 con 0x7fb8580aa0a0 2020-11-20T11:46:39.935 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":132,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:46:39.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.934+0000 7fb85fd16700 1 -- 172.21.15.23:0/4133722759 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb838056d60 msgr2=0x7fb838059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:39.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.934+0000 7fb85fd16700 1 --2- 172.21.15.23:0/4133722759 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb838056d60 0x7fb838059220 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fb854007fd0 tx=0x7fb854009040).stop 2020-11-20T11:46:39.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.934+0000 7fb85fd16700 1 -- 172.21.15.23:0/4133722759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8580aa0a0 msgr2=0x7fb85812bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:39.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.934+0000 7fb85fd16700 1 --2- 172.21.15.23:0/4133722759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8580aa0a0 0x7fb85812bba0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fb8500090b0 tx=0x7fb850016000).stop 2020-11-20T11:46:39.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.934+0000 7fb85fd16700 1 -- 172.21.15.23:0/4133722759 shutdown_connections 2020-11-20T11:46:39.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.934+0000 7fb85fd16700 1 --2- 172.21.15.23:0/4133722759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb8580aa0a0 0x7fb85812bba0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:39.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.934+0000 7fb85fd16700 1 --2- 172.21.15.23:0/4133722759 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb838056d60 0x7fb838059220 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:39.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.934+0000 7fb85fd16700 1 --2- 172.21.15.23:0/4133722759 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb8580608c0 0x7fb858138f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:39.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.934+0000 7fb85fd16700 1 --2- 172.21.15.23:0/4133722759 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb858060ca0 0x7fb85809d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:39.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.934+0000 7fb85fd16700 1 -- 172.21.15.23:0/4133722759 >> 172.21.15.23:0/4133722759 conn(0x7fb858066750 msgr2=0x7fb8580ae100 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:39.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.934+0000 7fb85fd16700 1 -- 172.21.15.23:0/4133722759 shutdown_connections 2020-11-20T11:46:39.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:39.934+0000 7fb85fd16700 1 -- 172.21.15.23:0/4133722759 wait complete. 2020-11-20T11:46:39.949 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:46:39.949 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:46:40.091 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.088+0000 7f9195abb700 1 Processor -- start 2020-11-20T11:46:40.091 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.088+0000 7f9195abb700 1 -- start start 2020-11-20T11:46:40.093 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.089+0000 7f9195abb700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f91900a6e10 0x7f9190061850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:40.093 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.089+0000 7f9195abb700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9190062ad0 con 0x7f9190137870 2020-11-20T11:46:40.093 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.089+0000 7f9195abb700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f919013b580 con 0x7f9190137410 2020-11-20T11:46:40.094 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.089+0000 7f9195abb700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91900a61f0 con 0x7f91900a6e10 2020-11-20T11:46:40.094 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.089+0000 7f918ffff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f9190137870 0x7f919013ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:60510/0 (socket says 172.21.15.23:60510) 2020-11-20T11:46:40.094 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.089+0000 7f918ffff700 1 -- 172.21.15.23:0/264439627 learned_addr learned my addr 172.21.15.23:0/264439627 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:46:40.094 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.089+0000 7f918f7fe700 1 --2- 172.21.15.23:0/264439627 >> v2:172.21.15.155:3300/0 conn(0x7f91900a6e10 0x7f9190061850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:40.094 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.089+0000 7f918e7fc700 1 -- 172.21.15.23:0/264439627 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2999258600 0 0) 0x7f9190062ad0 con 0x7f9190137870 2020-11-20T11:46:40.095 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.089+0000 7f918e7fc700 1 -- 172.21.15.23:0/264439627 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f917c003280 con 0x7f9190137870 2020-11-20T11:46:40.095 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.089+0000 7f918f7fe700 1 -- 172.21.15.23:0/264439627 >> v1:172.21.15.155:6789/0 conn(0x7f9190137410 legacy=0x7f919013a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:40.095 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.089+0000 7f918e7fc700 1 -- 172.21.15.23:0/264439627 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2043003715 0 0) 0x7f917c003280 con 0x7f9190137870 2020-11-20T11:46:40.095 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.089+0000 7f918f7fe700 1 -- 172.21.15.23:0/264439627 >> v1:172.21.15.23:6789/0 conn(0x7f9190137870 legacy=0x7f919013ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:40.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.089+0000 7f918f7fe700 1 -- 172.21.15.23:0/264439627 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91900a63e0 con 0x7f91900a6e10 2020-11-20T11:46:40.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.090+0000 7f918f7fe700 1 --2- 172.21.15.23:0/264439627 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f91900a6e10 0x7f9190061850 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f9180006fd0 tx=0x7f9180006e30).ready entity=mon.1 client_cookie=c6a2ff4a53c405a1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:40.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.095+0000 7f918e7fc700 1 -- 172.21.15.23:0/264439627 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91880018a0 con 0x7f91900a6e10 2020-11-20T11:46:40.099 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.095+0000 7f918e7fc700 1 -- 172.21.15.23:0/264439627 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9180000e90 con 0x7f91900a6e10 2020-11-20T11:46:40.099 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.095+0000 7f918e7fc700 1 -- 172.21.15.23:0/264439627 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91800096e0 con 0x7f91900a6e10 2020-11-20T11:46:40.099 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.096+0000 7f9195abb700 1 -- 172.21.15.23:0/264439627 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f91900a6e10 msgr2=0x7f9190061850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:40.099 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.096+0000 7f9195abb700 1 --2- 172.21.15.23:0/264439627 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f91900a6e10 0x7f9190061850 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f9180006fd0 tx=0x7f9180006e30).stop 2020-11-20T11:46:40.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.097+0000 7f9195abb700 1 -- 172.21.15.23:0/264439627 shutdown_connections 2020-11-20T11:46:40.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.097+0000 7f9195abb700 1 --2- 172.21.15.23:0/264439627 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f91900a6e10 0x7f9190061850 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:40.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.097+0000 7f9195abb700 1 -- 172.21.15.23:0/264439627 >> 172.21.15.23:0/264439627 conn(0x7f9190060c80 msgr2=0x7f91900acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:40.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.099+0000 7f9195abb700 1 -- 172.21.15.23:0/264439627 shutdown_connections 2020-11-20T11:46:40.102 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.099+0000 7f9195abb700 1 -- 172.21.15.23:0/264439627 wait complete. 2020-11-20T11:46:40.103 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.100+0000 7f9195abb700 1 Processor -- start 2020-11-20T11:46:40.104 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.100+0000 7f9195abb700 1 -- start start 2020-11-20T11:46:40.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.101+0000 7f9195abb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91900a6e10 0x7f91900a28e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:40.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.101+0000 7f9195abb700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9190137410 0x7f9190139740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:40.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.101+0000 7f9195abb700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9190137870 0x7f919014aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:40.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.101+0000 7f9195abb700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9190135030 con 0x7f91900a6e10 2020-11-20T11:46:40.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.101+0000 7f9195abb700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919014a3f0 con 0x7f9190137870 2020-11-20T11:46:40.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.101+0000 7f9195abb700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91900a3220 con 0x7f9190137410 2020-11-20T11:46:40.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.101+0000 7f918f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91900a6e10 0x7f91900a28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:40.107 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.101+0000 7f918f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91900a6e10 0x7f91900a28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46266/0 (socket says 172.21.15.23:46266) 2020-11-20T11:46:40.107 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.101+0000 7f918f7fe700 1 -- 172.21.15.23:0/1442517744 learned_addr learned my addr 172.21.15.23:0/1442517744 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:46:40.107 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.101+0000 7f918ffff700 1 --2- 172.21.15.23:0/1442517744 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9190137870 0x7f919014aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:40.107 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.101+0000 7f918effd700 1 --2- 172.21.15.23:0/1442517744 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9190137410 0x7f9190139740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:40.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.102+0000 7f918effd700 1 -- 172.21.15.23:0/1442517744 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9190137870 msgr2=0x7f919014aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:40.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.102+0000 7f918effd700 1 --2- 172.21.15.23:0/1442517744 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9190137870 0x7f919014aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:40.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.102+0000 7f918effd700 1 -- 172.21.15.23:0/1442517744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91900a6e10 msgr2=0x7f91900a28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:40.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.102+0000 7f918effd700 1 --2- 172.21.15.23:0/1442517744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91900a6e10 0x7f91900a28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:40.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.102+0000 7f918effd700 1 -- 172.21.15.23:0/1442517744 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91900a2f80 con 0x7f9190137410 2020-11-20T11:46:40.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.102+0000 7f918ffff700 1 --2- 172.21.15.23:0/1442517744 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9190137870 0x7f919014aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:46:40.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.102+0000 7f918effd700 1 --2- 172.21.15.23:0/1442517744 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9190137410 0x7f9190139740 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f918400c710 tx=0x7f9184007b50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:40.110 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.102+0000 7f91777fe700 1 -- 172.21.15.23:0/1442517744 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918400e4a0 con 0x7f9190137410 2020-11-20T11:46:40.110 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.102+0000 7f91777fe700 1 -- 172.21.15.23:0/1442517744 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f918400e600 con 0x7f9190137410 2020-11-20T11:46:40.110 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.102+0000 7f9195abb700 1 -- 172.21.15.23:0/1442517744 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f919014a5e0 con 0x7f9190137410 2020-11-20T11:46:40.110 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.102+0000 7f9195abb700 1 -- 172.21.15.23:0/1442517744 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91900ac0c0 con 0x7f9190137410 2020-11-20T11:46:40.111 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.102+0000 7f91777fe700 1 -- 172.21.15.23:0/1442517744 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918400e980 con 0x7f9190137410 2020-11-20T11:46:40.111 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.104+0000 7f91777fe700 1 -- 172.21.15.23:0/1442517744 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9184019030 con 0x7f9190137410 2020-11-20T11:46:40.112 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.104+0000 7f91777fe700 1 --2- 172.21.15.23:0/1442517744 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9164056d90 0x7f9164059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:40.112 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.104+0000 7f91777fe700 1 -- 172.21.15.23:0/1442517744 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f918406f530 con 0x7f9190137410 2020-11-20T11:46:40.112 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.104+0000 7f918f7fe700 1 --2- 172.21.15.23:0/1442517744 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9164056d90 0x7f9164059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:40.112 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.105+0000 7f9195abb700 1 --2- 172.21.15.23:0/1442517744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f919014b090 0x7f919014de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:40.113 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.105+0000 7f9195abb700 1 -- 172.21.15.23:0/1442517744 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9190137ec0 con 0x7f919014b090 2020-11-20T11:46:40.113 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.105+0000 7f918f7fe700 1 --2- 172.21.15.23:0/1442517744 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9164056d90 0x7f9164059250 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f9180006ab0 tx=0x7f91900a6570).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:40.114 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.111+0000 7f918ffff700 1 --2- 172.21.15.23:0/1442517744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f919014b090 0x7f919014de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:40.115 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.112+0000 7f918ffff700 1 -- 172.21.15.23:0/1442517744 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f919005c520 con 0x7f919014b090 2020-11-20T11:46:40.115 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.112+0000 7f918ffff700 1 --2- 172.21.15.23:0/1442517744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f919014b090 0x7f919014de70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f918800f400 tx=0x7f9188001940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:40.115 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.113+0000 7f91777fe700 1 -- 172.21.15.23:0/1442517744 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9188012350 con 0x7f919014b090 2020-11-20T11:46:40.116 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.113+0000 7f91777fe700 1 -- 172.21.15.23:0/1442517744 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f91880124b0 con 0x7f919014b090 2020-11-20T11:46:40.116 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.113+0000 7f91777fe700 1 -- 172.21.15.23:0/1442517744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f919014b090 msgr2=0x7f919014de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:40.116 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.113+0000 7f91777fe700 1 --2- 172.21.15.23:0/1442517744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f919014b090 0x7f919014de70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f918800f400 tx=0x7f9188001940).stop 2020-11-20T11:46:40.116 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.113+0000 7f91777fe700 1 -- 172.21.15.23:0/1442517744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f919014b090 msgr2=0x7f919014de70 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:46:40.117 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.113+0000 7f91777fe700 1 --2- 172.21.15.23:0/1442517744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f919014b090 0x7f919014de70 secure :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0x7f918800f400 tx=0x7f9188001940).stop 2020-11-20T11:46:40.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.126+0000 7f9195abb700 1 --2- 172.21.15.23:0/1442517744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f919014e360 0x7f91901351f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:40.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.126+0000 7f9195abb700 1 -- 172.21.15.23:0/1442517744 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9190137030 con 0x7f919014e360 2020-11-20T11:46:40.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.126+0000 7f918ffff700 1 --2- 172.21.15.23:0/1442517744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f919014e360 0x7f91901351f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:40.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.126+0000 7f918ffff700 1 -- 172.21.15.23:0/1442517744 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9190044710 con 0x7f919014e360 2020-11-20T11:46:40.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.126+0000 7f918ffff700 1 --2- 172.21.15.23:0/1442517744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f919014e360 0x7f91901351f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f9188016000 tx=0x7f9188017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:40.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.126+0000 7f91777fe700 1 -- 172.21.15.23:0/1442517744 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9188015740 con 0x7f919014e360 2020-11-20T11:46:40.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.127+0000 7f91777fe700 1 -- 172.21.15.23:0/1442517744 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4786 (secure 0 0 0) 0x7f918800fd90 con 0x7f919014e360 2020-11-20T11:46:40.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.127+0000 7f91777fe700 1 -- 172.21.15.23:0/1442517744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f919014e360 msgr2=0x7f91901351f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:40.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.127+0000 7f91777fe700 1 --2- 172.21.15.23:0/1442517744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f919014e360 0x7f91901351f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f9188016000 tx=0x7f9188017000).stop 2020-11-20T11:46:40.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.127+0000 7f91777fe700 1 -- 172.21.15.23:0/1442517744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f919014e360 msgr2=0x7f91901351f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:46:40.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.127+0000 7f91777fe700 1 --2- 172.21.15.23:0/1442517744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f919014e360 0x7f91901351f0 secure :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0x7f9188016000 tx=0x7f9188017000).stop 2020-11-20T11:46:40.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.127+0000 7f9195abb700 1 -- 172.21.15.23:0/1442517744 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9164056d90 msgr2=0x7f9164059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:40.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.127+0000 7f9195abb700 1 --2- 172.21.15.23:0/1442517744 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9164056d90 0x7f9164059250 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f9180006ab0 tx=0x7f91900a6570).stop 2020-11-20T11:46:40.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.127+0000 7f9195abb700 1 -- 172.21.15.23:0/1442517744 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9190137410 msgr2=0x7f9190139740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:40.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.127+0000 7f9195abb700 1 --2- 172.21.15.23:0/1442517744 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9190137410 0x7f9190139740 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f918400c710 tx=0x7f9184007b50).stop 2020-11-20T11:46:40.137 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.127+0000 7f918ffff700 1 -- 172.21.15.23:0/1442517744 reap_dead start 2020-11-20T11:46:40.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.127+0000 7f9195abb700 1 -- 172.21.15.23:0/1442517744 shutdown_connections 2020-11-20T11:46:40.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.127+0000 7f9195abb700 1 --2- 172.21.15.23:0/1442517744 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9190137410 0x7f9190139740 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:40.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.127+0000 7f9195abb700 1 -- 172.21.15.23:0/1442517744 >> 172.21.15.23:0/1442517744 conn(0x7f9190060c80 msgr2=0x7f919013a880 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:40.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.127+0000 7f9195abb700 1 -- 172.21.15.23:0/1442517744 shutdown_connections 2020-11-20T11:46:40.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.127+0000 7f9195abb700 1 -- 172.21.15.23:0/1442517744 wait complete. 2020-11-20T11:46:40.140 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:46:40.140 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:46:40.141 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:46:40.141 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:46:40.142 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 132, 2020-11-20T11:46:40.142 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:46:40.143 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:46:40.143 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:46:40.143 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:46:40.144 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:40.144 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 2, 2020-11-20T11:46:40.145 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:46:40.146 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:46:40.146 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:46:40.147 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:46:40.147 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:46:40.147 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:46:40.148 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:46:40.148 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:46:40.149 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:46:40.149 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:46:40.149 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:46:40.150 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:40.150 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:46:40.151 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:46:40.151 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:46:40.151 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:46:40.152 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:46:40.152 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:46:40.152 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:46:40.153 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:46:40.153 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:46:40.154 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:46:40.154 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:40.154 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.155 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:46:40.155 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:46:40.156 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:46:40.157 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:46:40.157 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:46:40.157 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:46:40.158 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:46:40.158 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:46:40.158 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:46:40.158 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:46:40.158 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:46:40.159 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:46:40.159 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:46:40.159 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:46:40.159 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:46:40.160 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:46:40.160 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:46:40.160 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:46:40.160 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:46:40.160 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:46:40.161 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:46:40.161 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:46:40.161 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:46:40.161 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:40.161 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:46:40.162 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.162 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:46:40.162 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.162 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:46:40.162 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:46:40.163 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:46:40.163 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:46:40.163 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.163 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:46:40.163 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:46:40.164 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:40.164 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.164 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.164 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:46:40.164 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:46:40.165 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:40.165 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:40.165 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:40.165 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.166 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:46:40.166 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:46:40.167 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:46:40.167 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:46:40.168 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:46:40.168 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.168 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.169 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:46:40.169 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:46:40.170 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:46:40.170 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:46:40.171 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.171 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:46:40.172 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:46:40.172 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:40.172 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.172 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.172 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:46:40.173 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:46:40.173 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:40.173 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:40.173 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:40.173 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.174 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:46:40.174 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:46:40.174 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:46:40.174 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:46:40.174 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:46:40.175 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.175 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.175 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:46:40.175 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:46:40.176 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:46:40.176 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:46:40.176 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.176 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:46:40.176 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:46:40.176 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:40.177 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.177 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.177 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:46:40.177 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:46:40.178 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:40.178 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:40.178 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:40.178 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.178 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:46:40.179 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:46:40.179 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:46:40.179 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:46:40.179 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:46:40.179 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:40.180 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:40.180 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.180 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:46:40.180 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:46:40.180 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.181 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:40.181 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:40.181 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:46:40.181 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:40.181 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:40.182 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:46:40.182 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.182 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:40.182 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:40.182 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:46:40.183 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:40.183 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:40.183 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:46:40.183 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.183 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:40.184 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:40.184 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:46:40.184 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:40.184 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:40.184 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:46:40.185 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.185 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:40.185 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:40.185 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:46:40.185 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:40.186 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:40.186 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:46:40.186 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.186 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:40.186 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:40.187 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:46:40.187 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:40.187 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:40.187 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.187 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:46:40.188 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:46:40.189 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:46:40.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.338+0000 7fd5ca579700 1 Processor -- start 2020-11-20T11:46:40.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.338+0000 7fd5ca579700 1 -- start start 2020-11-20T11:46:40.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.339+0000 7fd5ca579700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5c40aaab0 0x7fd5c413ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:40.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.339+0000 7fd5ca579700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5c41388b0 con 0x7fd5c413aeb0 2020-11-20T11:46:40.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.339+0000 7fd5ca579700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5c413e600 con 0x7fd5c40aae90 2020-11-20T11:46:40.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.339+0000 7fd5ca579700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c413e8a0 con 0x7fd5c40aaab0 2020-11-20T11:46:40.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.339+0000 7fd5c8b16700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fd5c413aeb0 0x7fd5c40a8c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:60532/0 (socket says 172.21.15.23:60532) 2020-11-20T11:46:40.345 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.339+0000 7fd5c8b16700 1 -- 172.21.15.23:0/3404013401 learned_addr learned my addr 172.21.15.23:0/3404013401 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:46:40.345 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.339+0000 7fd5c3fff700 1 --2- 172.21.15.23:0/3404013401 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5c40aaab0 0x7fd5c413ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:40.345 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.339+0000 7fd5c2ffd700 1 -- 172.21.15.23:0/3404013401 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2774787325 0 0) 0x7fd5c41388b0 con 0x7fd5c413aeb0 2020-11-20T11:46:40.345 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.339+0000 7fd5c2ffd700 1 -- 172.21.15.23:0/3404013401 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5b0003280 con 0x7fd5c413aeb0 2020-11-20T11:46:40.346 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.339+0000 7fd5c3fff700 1 -- 172.21.15.23:0/3404013401 >> v1:172.21.15.155:6789/0 conn(0x7fd5c40aae90 legacy=0x7fd5c41381a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2020-11-20T11:46:40.346 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.339+0000 7fd5c3fff700 1 -- 172.21.15.23:0/3404013401 >> v1:172.21.15.23:6789/0 conn(0x7fd5c413aeb0 legacy=0x7fd5c40a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:40.346 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.339+0000 7fd5c3fff700 1 -- 172.21.15.23:0/3404013401 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c40ab6d0 con 0x7fd5c40aaab0 2020-11-20T11:46:40.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.340+0000 7fd5c3fff700 1 --2- 172.21.15.23:0/3404013401 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5c40aaab0 0x7fd5c413ed40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fd5b400ba00 tx=0x7fd5b4006e30).ready entity=mon.2 client_cookie=9fc8b45de452ac9d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:40.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.340+0000 7fd5c2ffd700 1 -- 172.21.15.23:0/3404013401 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b40097a0 con 0x7fd5c40aaab0 2020-11-20T11:46:40.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.340+0000 7fd5c2ffd700 1 -- 172.21.15.23:0/3404013401 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5b4000e90 con 0x7fd5c40aaab0 2020-11-20T11:46:40.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.340+0000 7fd5c2ffd700 1 -- 172.21.15.23:0/3404013401 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b4009b10 con 0x7fd5c40aaab0 2020-11-20T11:46:40.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.341+0000 7fd5ca579700 1 -- 172.21.15.23:0/3404013401 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5c40aaab0 msgr2=0x7fd5c413ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:40.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.341+0000 7fd5ca579700 1 --2- 172.21.15.23:0/3404013401 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5c40aaab0 0x7fd5c413ed40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fd5b400ba00 tx=0x7fd5b4006e30).stop 2020-11-20T11:46:40.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.345+0000 7fd5ca579700 1 -- 172.21.15.23:0/3404013401 shutdown_connections 2020-11-20T11:46:40.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.345+0000 7fd5ca579700 1 --2- 172.21.15.23:0/3404013401 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5c40aaab0 0x7fd5c413ed40 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:40.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.345+0000 7fd5ca579700 1 -- 172.21.15.23:0/3404013401 >> 172.21.15.23:0/3404013401 conn(0x7fd5c40678e0 msgr2=0x7fd5c4067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:40.356 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.353+0000 7fd5ca579700 1 -- 172.21.15.23:0/3404013401 shutdown_connections 2020-11-20T11:46:40.356 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.353+0000 7fd5ca579700 1 -- 172.21.15.23:0/3404013401 wait complete. 2020-11-20T11:46:40.357 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.354+0000 7fd5ca579700 1 Processor -- start 2020-11-20T11:46:40.357 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.354+0000 7fd5ca579700 1 -- start start 2020-11-20T11:46:40.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.355+0000 7fd5ca579700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5c40aaab0 0x7fd5c413dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:40.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.355+0000 7fd5ca579700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5c40aae90 0x7fd5c409dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:40.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.355+0000 7fd5ca579700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5c40a0160 0x7fd5c414abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:40.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.355+0000 7fd5ca579700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c409e810 con 0x7fd5c40a0160 2020-11-20T11:46:40.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.355+0000 7fd5ca579700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c414a480 con 0x7fd5c40aaab0 2020-11-20T11:46:40.360 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.355+0000 7fd5ca579700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c414a720 con 0x7fd5c40aae90 2020-11-20T11:46:40.360 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.355+0000 7fd5c37fe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5c40aae90 0x7fd5c409dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:40.360 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.355+0000 7fd5c8b16700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5c40a0160 0x7fd5c414abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:40.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.356+0000 7fd5c3fff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5c40aaab0 0x7fd5c413dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:40.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.356+0000 7fd5c8b16700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5c40a0160 0x7fd5c414abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46286/0 (socket says 172.21.15.23:46286) 2020-11-20T11:46:40.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.356+0000 7fd5c37fe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5c40aae90 0x7fd5c409dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:56524/0 (socket says 172.21.15.23:56524) 2020-11-20T11:46:40.364 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.356+0000 7fd5c8b16700 1 -- 172.21.15.23:0/659615278 learned_addr learned my addr 172.21.15.23:0/659615278 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:46:40.367 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.356+0000 7fd5c3fff700 1 -- 172.21.15.23:0/659615278 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5c40aae90 msgr2=0x7fd5c409dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:40.368 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.356+0000 7fd5c3fff700 1 --2- 172.21.15.23:0/659615278 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5c40aae90 0x7fd5c409dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:40.368 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.356+0000 7fd5c3fff700 1 -- 172.21.15.23:0/659615278 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5c40a0160 msgr2=0x7fd5c414abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:40.368 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.356+0000 7fd5c3fff700 1 --2- 172.21.15.23:0/659615278 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5c40a0160 0x7fd5c414abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:40.369 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.356+0000 7fd5c3fff700 1 -- 172.21.15.23:0/659615278 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c409e4d0 con 0x7fd5c40aaab0 2020-11-20T11:46:40.369 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.356+0000 7fd5c37fe700 1 --2- 172.21.15.23:0/659615278 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5c40aae90 0x7fd5c409dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:46:40.369 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.356+0000 7fd5c3fff700 1 --2- 172.21.15.23:0/659615278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5c40aaab0 0x7fd5c413dd30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fd5b4012000 tx=0x7fd5b4015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:40.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.356+0000 7fd5a3fff700 1 -- 172.21.15.23:0/659615278 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b4011b00 con 0x7fd5c40aaab0 2020-11-20T11:46:40.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.356+0000 7fd5a3fff700 1 -- 172.21.15.23:0/659615278 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5b4011c60 con 0x7fd5c40aaab0 2020-11-20T11:46:40.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.356+0000 7fd5a3fff700 1 -- 172.21.15.23:0/659615278 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b401d450 con 0x7fd5c40aaab0 2020-11-20T11:46:40.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.356+0000 7fd5ca579700 1 -- 172.21.15.23:0/659615278 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5c414d8b0 con 0x7fd5c40aaab0 2020-11-20T11:46:40.370 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.357+0000 7fd5ca579700 1 -- 172.21.15.23:0/659615278 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5c414de60 con 0x7fd5c40aaab0 2020-11-20T11:46:40.371 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.357+0000 7fd5ca579700 1 --2- 172.21.15.23:0/659615278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5c413aeb0 0x7fd5c414e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:40.371 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.357+0000 7fd5ca579700 1 -- 172.21.15.23:0/659615278 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c409e6c0 con 0x7fd5c413aeb0 2020-11-20T11:46:40.371 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.358+0000 7fd5c37fe700 1 --2- 172.21.15.23:0/659615278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5c413aeb0 0x7fd5c414e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:40.371 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.358+0000 7fd5a3fff700 1 -- 172.21.15.23:0/659615278 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd5b400f070 con 0x7fd5c40aaab0 2020-11-20T11:46:40.371 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.358+0000 7fd5c37fe700 1 -- 172.21.15.23:0/659615278 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd5c414a300 con 0x7fd5c413aeb0 2020-11-20T11:46:40.372 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.358+0000 7fd5c37fe700 1 --2- 172.21.15.23:0/659615278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5c413aeb0 0x7fd5c414e050 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fd5c409f420 tx=0x7fd5b8008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:40.372 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.358+0000 7fd5a3fff700 1 --2- 172.21.15.23:0/659615278 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd59c056d90 0x7fd59c059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:40.372 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.358+0000 7fd5a3fff700 1 -- 172.21.15.23:0/659615278 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fd5b4071b60 con 0x7fd5c40aaab0 2020-11-20T11:46:40.372 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.358+0000 7fd5a3fff700 1 -- 172.21.15.23:0/659615278 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b800cc40 con 0x7fd5c413aeb0 2020-11-20T11:46:40.372 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.358+0000 7fd5c8b16700 1 --2- 172.21.15.23:0/659615278 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd59c056d90 0x7fd59c059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:40.373 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.358+0000 7fd5a3fff700 1 -- 172.21.15.23:0/659615278 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fd5b800cda0 con 0x7fd5c413aeb0 2020-11-20T11:46:40.373 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.358+0000 7fd5a3fff700 1 -- 172.21.15.23:0/659615278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5c413aeb0 msgr2=0x7fd5c414e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:40.373 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.358+0000 7fd5a3fff700 1 --2- 172.21.15.23:0/659615278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5c413aeb0 0x7fd5c414e050 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fd5c409f420 tx=0x7fd5b8008040).stop 2020-11-20T11:46:40.374 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.358+0000 7fd5a3fff700 1 -- 172.21.15.23:0/659615278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5c413aeb0 msgr2=0x7fd5c414e050 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:46:40.374 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.358+0000 7fd5a3fff700 1 --2- 172.21.15.23:0/659615278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5c413aeb0 0x7fd5c414e050 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:40.374 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.359+0000 7fd5c8b16700 1 --2- 172.21.15.23:0/659615278 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd59c056d90 0x7fd59c059250 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7fd5c40a0580 tx=0x7fd5bc009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:40.376 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.365+0000 7fd5ca579700 1 --2- 172.21.15.23:0/659615278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5c414e540 0x7fd5c40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:40.376 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.365+0000 7fd5ca579700 1 -- 172.21.15.23:0/659615278 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c414a5c0 con 0x7fd5c414e540 2020-11-20T11:46:40.376 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.366+0000 7fd5c37fe700 1 --2- 172.21.15.23:0/659615278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5c414e540 0x7fd5c40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:40.376 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.366+0000 7fd5c37fe700 1 -- 172.21.15.23:0/659615278 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd5c414da40 con 0x7fd5c414e540 2020-11-20T11:46:40.377 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.366+0000 7fd5c37fe700 1 --2- 172.21.15.23:0/659615278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5c414e540 0x7fd5c40a5170 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fd5c41481f0 tx=0x7fd5b8015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:40.377 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.367+0000 7fd5a3fff700 1 -- 172.21.15.23:0/659615278 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b80140d0 con 0x7fd5c414e540 2020-11-20T11:46:40.378 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.367+0000 7fd5a3fff700 1 -- 172.21.15.23:0/659615278 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4274 (secure 0 0 0) 0x7fd5b800ca30 con 0x7fd5c414e540 2020-11-20T11:46:40.378 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.367+0000 7fd5a3fff700 1 -- 172.21.15.23:0/659615278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5c414e540 msgr2=0x7fd5c40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:40.378 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.367+0000 7fd5a3fff700 1 --2- 172.21.15.23:0/659615278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5c414e540 0x7fd5c40a5170 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fd5c41481f0 tx=0x7fd5b8015000).stop 2020-11-20T11:46:40.378 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.367+0000 7fd5a3fff700 1 -- 172.21.15.23:0/659615278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5c414e540 msgr2=0x7fd5c40a5170 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:46:40.379 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.367+0000 7fd5a3fff700 1 --2- 172.21.15.23:0/659615278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5c414e540 0x7fd5c40a5170 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0x7fd5c41481f0 tx=0x7fd5b8015000).stop 2020-11-20T11:46:40.379 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.367+0000 7fd5ca579700 1 -- 172.21.15.23:0/659615278 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd59c056d90 msgr2=0x7fd59c059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:40.379 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.367+0000 7fd5ca579700 1 --2- 172.21.15.23:0/659615278 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd59c056d90 0x7fd59c059250 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7fd5c40a0580 tx=0x7fd5bc009040).stop 2020-11-20T11:46:40.379 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.367+0000 7fd5ca579700 1 -- 172.21.15.23:0/659615278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5c40aaab0 msgr2=0x7fd5c413dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:40.379 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.367+0000 7fd5ca579700 1 --2- 172.21.15.23:0/659615278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5c40aaab0 0x7fd5c413dd30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fd5b4012000 tx=0x7fd5b4015000).stop 2020-11-20T11:46:40.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.367+0000 7fd5c8b16700 1 -- 172.21.15.23:0/659615278 reap_dead start 2020-11-20T11:46:40.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.368+0000 7fd5c8b16700 1 -- 172.21.15.23:0/659615278 reap_dead start 2020-11-20T11:46:40.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.368+0000 7fd5ca579700 1 -- 172.21.15.23:0/659615278 shutdown_connections 2020-11-20T11:46:40.381 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.368+0000 7fd5ca579700 1 -- 172.21.15.23:0/659615278 >> 172.21.15.23:0/659615278 conn(0x7fd5c40678e0 msgr2=0x7fd5c41383b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:40.381 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.368+0000 7fd5ca579700 1 -- 172.21.15.23:0/659615278 shutdown_connections 2020-11-20T11:46:40.381 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.368+0000 7fd5ca579700 1 -- 172.21.15.23:0/659615278 wait complete. 2020-11-20T11:46:40.381 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:46:40.382 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:46:40.382 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:46:40.382 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:46:40.382 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 132, 2020-11-20T11:46:40.382 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:46:40.383 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:46:40.383 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:46:40.383 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:46:40.383 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:40.384 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 3, 2020-11-20T11:46:40.384 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:46:40.384 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:46:40.384 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:46:40.384 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:46:40.384 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:46:40.385 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:46:40.385 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:46:40.385 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:46:40.385 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:46:40.386 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:46:40.386 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:46:40.386 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:40.386 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:46:40.386 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:46:40.387 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:46:40.387 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:46:40.387 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:46:40.387 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:46:40.387 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:46:40.388 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:46:40.388 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:46:40.388 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:46:40.388 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:40.388 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.388 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:46:40.389 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:46:40.389 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:46:40.389 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:46:40.389 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:46:40.390 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:46:40.390 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:46:40.390 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:46:40.391 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:46:40.391 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:46:40.391 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:46:40.391 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:46:40.391 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:46:40.392 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:46:40.392 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:46:40.392 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:46:40.392 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:46:40.392 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:46:40.393 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:46:40.393 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:46:40.393 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:46:40.393 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:46:40.393 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:46:40.394 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:40.394 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:46:40.394 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.394 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:46:40.394 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.395 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:46:40.395 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:46:40.395 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:46:40.395 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:46:40.395 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.396 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:46:40.396 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:46:40.396 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:40.396 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.396 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.397 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:46:40.397 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:46:40.397 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:40.397 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:40.398 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:40.398 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.398 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:46:40.398 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:46:40.398 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:46:40.399 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:46:40.399 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:46:40.399 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.399 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.399 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:46:40.400 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:46:40.400 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:46:40.400 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:46:40.400 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.400 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:46:40.401 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:46:40.401 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:40.401 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.401 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.402 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:46:40.402 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:46:40.402 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:40.402 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:40.402 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:40.403 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.403 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:46:40.403 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:46:40.403 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:46:40.403 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:46:40.404 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:46:40.404 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.404 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.404 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:46:40.404 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:46:40.405 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:46:40.405 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:46:40.405 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.405 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:46:40.405 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:46:40.406 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:40.406 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.406 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.406 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:46:40.406 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:46:40.407 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:40.407 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:40.407 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:40.407 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.407 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:46:40.408 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:46:40.408 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:46:40.408 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:46:40.408 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:46:40.408 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:40.409 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:40.409 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.409 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:46:40.409 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:46:40.409 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.410 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:40.410 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:40.410 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:46:40.410 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:40.410 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:40.411 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:46:40.411 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.411 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:40.411 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:40.411 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:46:40.412 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:40.412 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:40.412 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.412 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:46:40.412 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:46:40.414 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:46:40.556 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.553+0000 7f03a8daa700 1 Processor -- start 2020-11-20T11:46:40.557 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.553+0000 7f03a8daa700 1 -- start start 2020-11-20T11:46:40.559 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.554+0000 7f03a8daa700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f03a40ad600 0x7f03a40658b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:40.560 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.554+0000 7f03a8daa700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f03a40ad9e0 0x7f03a413c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:40.560 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.554+0000 7f03a8daa700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03a40a8b90 con 0x7f03a413cd00 2020-11-20T11:46:40.560 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.554+0000 7f03a8daa700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03a4065da0 con 0x7f03a40ad9e0 2020-11-20T11:46:40.560 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.554+0000 7f03a8daa700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03a40a83b0 con 0x7f03a40ad600 2020-11-20T11:46:40.560 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.554+0000 7f03a1d9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f03a40ad9e0 0x7f03a413c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:40.561 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.554+0000 7f03a1d9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f03a40ad9e0 0x7f03a413c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46294/0 (socket says 172.21.15.23:46294) 2020-11-20T11:46:40.561 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.554+0000 7f03a1d9b700 1 -- 172.21.15.23:0/4037765789 learned_addr learned my addr 172.21.15.23:0/4037765789 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:46:40.561 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.554+0000 7f03a259c700 1 --2- 172.21.15.23:0/4037765789 >> v2:172.21.15.155:3300/0 conn(0x7f03a40ad600 0x7f03a40658b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:40.561 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.555+0000 7f03a1d9b700 1 -- 172.21.15.23:0/4037765789 >> v2:172.21.15.155:3300/0 conn(0x7f03a40ad600 msgr2=0x7f03a40658b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:46:40.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.555+0000 7f03a1d9b700 1 --2- 172.21.15.23:0/4037765789 >> v2:172.21.15.155:3300/0 conn(0x7f03a40ad600 0x7f03a40658b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:40.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.555+0000 7f03a1d9b700 1 -- 172.21.15.23:0/4037765789 >> v1:172.21.15.155:6789/0 conn(0x7f03a413cd00 legacy=0x7f03a40a6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:40.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.555+0000 7f03a1d9b700 1 -- 172.21.15.23:0/4037765789 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a40a8650 con 0x7f03a40ad9e0 2020-11-20T11:46:40.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.555+0000 7f03a1d9b700 1 --2- 172.21.15.23:0/4037765789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03a40ad9e0 0x7f03a413c810 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f03a40610d0 tx=0x7f0394006ef0).ready entity=mon.0 client_cookie=7d9e1722b58b6237 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:40.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.555+0000 7f03a159a700 1 -- 172.21.15.23:0/4037765789 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039400b860 con 0x7f03a40ad9e0 2020-11-20T11:46:40.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.555+0000 7f03a159a700 1 -- 172.21.15.23:0/4037765789 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0394000e90 con 0x7f03a40ad9e0 2020-11-20T11:46:40.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.555+0000 7f03a159a700 1 -- 172.21.15.23:0/4037765789 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039400bbd0 con 0x7f03a40ad9e0 2020-11-20T11:46:40.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.555+0000 7f03a8daa700 1 -- 172.21.15.23:0/4037765789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03a40ad9e0 msgr2=0x7f03a413c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:40.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.555+0000 7f03a8daa700 1 --2- 172.21.15.23:0/4037765789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03a40ad9e0 0x7f03a413c810 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f03a40610d0 tx=0x7f0394006ef0).stop 2020-11-20T11:46:40.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.556+0000 7f03a8daa700 1 -- 172.21.15.23:0/4037765789 shutdown_connections 2020-11-20T11:46:40.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.556+0000 7f03a8daa700 1 --2- 172.21.15.23:0/4037765789 >> v2:172.21.15.155:3300/0 conn(0x7f03a40ad600 0x7f03a40658b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:40.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.556+0000 7f03a8daa700 1 --2- 172.21.15.23:0/4037765789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03a40ad9e0 0x7f03a413c810 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:40.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.556+0000 7f03a8daa700 1 -- 172.21.15.23:0/4037765789 >> 172.21.15.23:0/4037765789 conn(0x7f03a4060c80 msgr2=0x7f03a40acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:40.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.556+0000 7f03a8daa700 1 -- 172.21.15.23:0/4037765789 shutdown_connections 2020-11-20T11:46:40.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.556+0000 7f03a8daa700 1 -- 172.21.15.23:0/4037765789 wait complete. 2020-11-20T11:46:40.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.557+0000 7f03a8daa700 1 Processor -- start 2020-11-20T11:46:40.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.557+0000 7f03a8daa700 1 -- start start 2020-11-20T11:46:40.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.557+0000 7f03a8daa700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03a40ad600 0x7f03a40ac060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:40.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.557+0000 7f03a8daa700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03a40ad9e0 0x7f03a4134860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:40.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.557+0000 7f03a8daa700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f03a413cd00 0x7f03a40a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:40.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.557+0000 7f03a8daa700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a40a2090 con 0x7f03a40ad600 2020-11-20T11:46:40.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.557+0000 7f03a8daa700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a40a2330 con 0x7f03a40ad9e0 2020-11-20T11:46:40.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.557+0000 7f03a8daa700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03a40a25d0 con 0x7f03a413cd00 2020-11-20T11:46:40.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.557+0000 7f03a259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03a40ad600 0x7f03a40ac060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:40.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.557+0000 7f03a259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03a40ad600 0x7f03a40ac060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46300/0 (socket says 172.21.15.23:46300) 2020-11-20T11:46:40.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.557+0000 7f03a259c700 1 -- 172.21.15.23:0/1409157930 learned_addr learned my addr 172.21.15.23:0/1409157930 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:46:40.570 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.557+0000 7f03a1d9b700 1 --2- 172.21.15.23:0/1409157930 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03a40ad9e0 0x7f03a4134860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:40.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.557+0000 7f03a2d9d700 1 --2- 172.21.15.23:0/1409157930 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f03a413cd00 0x7f03a40a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:40.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.558+0000 7f03a259c700 1 -- 172.21.15.23:0/1409157930 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f03a413cd00 msgr2=0x7f03a40a4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:40.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.558+0000 7f03a259c700 1 --2- 172.21.15.23:0/1409157930 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f03a413cd00 0x7f03a40a4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:40.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.558+0000 7f03a259c700 1 -- 172.21.15.23:0/1409157930 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03a40ad9e0 msgr2=0x7f03a4134860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:40.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.558+0000 7f03a259c700 1 --2- 172.21.15.23:0/1409157930 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03a40ad9e0 0x7f03a4134860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:40.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.558+0000 7f03a259c700 1 -- 172.21.15.23:0/1409157930 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a40acb30 con 0x7f03a40ad600 2020-11-20T11:46:40.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.558+0000 7f03a1d9b700 1 --2- 172.21.15.23:0/1409157930 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f03a40ad9e0 0x7f03a4134860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:46:40.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.558+0000 7f03a259c700 1 --2- 172.21.15.23:0/1409157930 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03a40ad600 0x7f03a40ac060 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f039800eba0 tx=0x7f0398006e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:40.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.558+0000 7f03927fc700 1 -- 172.21.15.23:0/1409157930 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039800c850 con 0x7f03a40ad600 2020-11-20T11:46:40.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.558+0000 7f03927fc700 1 -- 172.21.15.23:0/1409157930 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f039800c9b0 con 0x7f03a40ad600 2020-11-20T11:46:40.573 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.558+0000 7f03a8daa700 1 -- 172.21.15.23:0/1409157930 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03a4134d50 con 0x7f03a40ad600 2020-11-20T11:46:40.573 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.558+0000 7f03927fc700 1 -- 172.21.15.23:0/1409157930 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039800cd30 con 0x7f03a40ad600 2020-11-20T11:46:40.573 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.558+0000 7f03a8daa700 1 -- 172.21.15.23:0/1409157930 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03a40a5460 con 0x7f03a40ad600 2020-11-20T11:46:40.573 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.559+0000 7f03a8daa700 1 --2- 172.21.15.23:0/1409157930 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f03a4132070 0x7f03a4132450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:40.573 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.559+0000 7f03a8daa700 1 -- 172.21.15.23:0/1409157930 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03a40a21d0 con 0x7f03a4132070 2020-11-20T11:46:40.574 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.559+0000 7f03a1d9b700 1 --2- 172.21.15.23:0/1409157930 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f03a4132070 0x7f03a4132450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:40.574 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.559+0000 7f03927fc700 1 -- 172.21.15.23:0/1409157930 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0398012600 con 0x7f03a40ad600 2020-11-20T11:46:40.574 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.560+0000 7f03a1d9b700 1 -- 172.21.15.23:0/1409157930 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f03a40a1f10 con 0x7f03a4132070 2020-11-20T11:46:40.574 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.560+0000 7f03927fc700 1 --2- 172.21.15.23:0/1409157930 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0378056d60 0x7f0378059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:40.574 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.560+0000 7f03927fc700 1 -- 172.21.15.23:0/1409157930 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f0398070340 con 0x7f03a40ad600 2020-11-20T11:46:40.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.560+0000 7f03a1d9b700 1 --2- 172.21.15.23:0/1409157930 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f03a4132070 0x7f03a4132450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f03a40a3eb0 tx=0x7f0394016000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:40.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.560+0000 7f03a2d9d700 1 --2- 172.21.15.23:0/1409157930 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0378056d60 0x7f0378059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:40.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.560+0000 7f03927fc700 1 -- 172.21.15.23:0/1409157930 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039401c440 con 0x7f03a4132070 2020-11-20T11:46:40.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.560+0000 7f03927fc700 1 -- 172.21.15.23:0/1409157930 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f039401c5a0 con 0x7f03a4132070 2020-11-20T11:46:40.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.560+0000 7f03927fc700 1 -- 172.21.15.23:0/1409157930 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f03a4132070 msgr2=0x7f03a4132450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:40.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.560+0000 7f03927fc700 1 --2- 172.21.15.23:0/1409157930 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f03a4132070 0x7f03a4132450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f03a40a3eb0 tx=0x7f0394016000).stop 2020-11-20T11:46:40.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.560+0000 7f03a2d9d700 1 --2- 172.21.15.23:0/1409157930 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0378056d60 0x7f0378059220 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f03a414a2a0 tx=0x7f039c009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:40.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.560+0000 7f03927fc700 1 -- 172.21.15.23:0/1409157930 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f03a4132070 msgr2=0x7f03a4132450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:46:40.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.560+0000 7f03927fc700 1 --2- 172.21.15.23:0/1409157930 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f03a4132070 0x7f03a4132450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:40.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.571+0000 7f03a8daa700 1 --2- 172.21.15.23:0/1409157930 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f03a4132940 0x7f03a414d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:40.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.571+0000 7f03a8daa700 1 -- 172.21.15.23:0/1409157930 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03a40a2790 con 0x7f03a4132940 2020-11-20T11:46:40.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.571+0000 7f03a1d9b700 1 --2- 172.21.15.23:0/1409157930 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f03a4132940 0x7f03a414d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:40.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.572+0000 7f03a1d9b700 1 -- 172.21.15.23:0/1409157930 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f03a405c520 con 0x7f03a4132940 2020-11-20T11:46:40.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.572+0000 7f03a1d9b700 1 --2- 172.21.15.23:0/1409157930 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f03a4132940 0x7f03a414d830 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f03a4148180 tx=0x7f039401f040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:40.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.572+0000 7f03927fc700 1 -- 172.21.15.23:0/1409157930 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0394027cd0 con 0x7f03a4132940 2020-11-20T11:46:40.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.572+0000 7f03927fc700 1 -- 172.21.15.23:0/1409157930 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4784 (secure 0 0 0) 0x7f0394011cf0 con 0x7f03a4132940 2020-11-20T11:46:40.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.572+0000 7f03927fc700 1 -- 172.21.15.23:0/1409157930 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f03a4132940 msgr2=0x7f03a414d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:40.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.572+0000 7f03927fc700 1 --2- 172.21.15.23:0/1409157930 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f03a4132940 0x7f03a414d830 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f03a4148180 tx=0x7f039401f040).stop 2020-11-20T11:46:40.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.572+0000 7f03927fc700 1 -- 172.21.15.23:0/1409157930 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f03a4132940 msgr2=0x7f03a414d830 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:46:40.580 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.572+0000 7f03927fc700 1 --2- 172.21.15.23:0/1409157930 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f03a4132940 0x7f03a414d830 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0x7f03a4148180 tx=0x7f039401f040).stop 2020-11-20T11:46:40.580 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:46:40.580 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:46:40.580 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:46:40.581 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:46:40.581 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 132, 2020-11-20T11:46:40.581 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:46:40.581 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:46:40.582 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:46:40.582 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:46:40.582 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:40.582 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 3, 2020-11-20T11:46:40.582 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:46:40.583 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:46:40.583 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:46:40.583 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:46:40.583 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:46:40.583 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:46:40.584 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:46:40.584 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:46:40.584 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:46:40.584 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:46:40.584 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:46:40.585 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:40.585 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:46:40.585 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:46:40.585 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:46:40.585 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:46:40.586 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:46:40.586 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:46:40.587 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:46:40.587 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:46:40.587 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:46:40.587 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:46:40.587 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:40.588 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.588 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:46:40.588 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:46:40.588 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:46:40.588 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:46:40.589 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:46:40.589 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:46:40.589 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:46:40.589 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:46:40.590 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:46:40.590 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:46:40.590 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:46:40.590 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:46:40.590 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:46:40.591 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:46:40.591 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:46:40.591 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:46:40.591 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:46:40.591 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:46:40.592 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:46:40.592 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:46:40.592 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:46:40.592 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:46:40.592 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:46:40.593 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:40.593 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:46:40.593 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.593 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:46:40.593 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.594 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:46:40.594 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:46:40.594 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:46:40.594 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:46:40.594 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.595 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:46:40.595 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:46:40.595 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:40.595 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.595 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.596 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:46:40.596 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:46:40.596 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:40.596 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:40.596 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:40.597 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.597 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:46:40.597 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:46:40.597 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:46:40.598 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:46:40.598 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:46:40.598 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.598 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.599 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:46:40.599 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:46:40.599 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:46:40.599 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:46:40.599 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.600 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:46:40.600 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:46:40.600 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:40.600 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.600 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.601 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:46:40.601 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:46:40.601 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:40.601 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:40.601 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:40.602 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.602 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:46:40.602 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:46:40.602 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:46:40.603 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:46:40.603 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:46:40.603 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.603 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.603 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:46:40.604 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:46:40.604 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:46:40.604 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:46:40.605 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.605 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:46:40.606 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:46:40.606 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:40.606 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.607 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.607 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:46:40.608 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:46:40.608 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:40.608 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:40.609 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:40.609 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.610 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:46:40.610 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:46:40.610 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:46:40.611 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:46:40.611 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:46:40.611 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:40.612 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:40.612 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.613 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:46:40.613 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:46:40.614 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.614 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:40.614 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:40.615 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:46:40.615 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:40.615 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:40.616 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:46:40.616 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.617 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:40.617 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:40.617 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:46:40.618 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:40.618 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:40.619 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:46:40.619 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.619 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:40.620 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:40.620 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:46:40.621 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:40.621 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:40.622 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:46:40.622 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.622 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:40.623 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:40.623 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:46:40.624 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:40.624 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:40.624 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:46:40.625 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:40.625 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:40.626 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:40.626 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:46:40.626 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:40.627 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:40.627 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:40.627 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:46:40.628 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:46:40.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.575+0000 7f03a8daa700 1 -- 172.21.15.23:0/1409157930 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0378056d60 msgr2=0x7f0378059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:40.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.575+0000 7f03a8daa700 1 --2- 172.21.15.23:0/1409157930 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0378056d60 0x7f0378059220 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f03a414a2a0 tx=0x7f039c009040).stop 2020-11-20T11:46:40.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.575+0000 7f03a8daa700 1 -- 172.21.15.23:0/1409157930 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03a40ad600 msgr2=0x7f03a40ac060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:40.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.575+0000 7f03a8daa700 1 --2- 172.21.15.23:0/1409157930 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03a40ad600 0x7f03a40ac060 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f039800eba0 tx=0x7f0398006e30).stop 2020-11-20T11:46:40.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.575+0000 7f03a2d9d700 1 -- 172.21.15.23:0/1409157930 reap_dead start 2020-11-20T11:46:40.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.575+0000 7f03a2d9d700 1 -- 172.21.15.23:0/1409157930 reap_dead start 2020-11-20T11:46:40.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.575+0000 7f03a8daa700 1 -- 172.21.15.23:0/1409157930 shutdown_connections 2020-11-20T11:46:40.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.575+0000 7f03a8daa700 1 -- 172.21.15.23:0/1409157930 >> 172.21.15.23:0/1409157930 conn(0x7f03a4060c80 msgr2=0x7f03a4138010 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:40.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.575+0000 7f03a8daa700 1 -- 172.21.15.23:0/1409157930 shutdown_connections 2020-11-20T11:46:40.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.576+0000 7f03a8daa700 1 -- 172.21.15.23:0/1409157930 wait complete. 2020-11-20T11:46:40.635 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2020-11-20T11:46:40.635 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2020-11-20T11:46:40.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.772+0000 7f1179c68700 1 Processor -- start 2020-11-20T11:46:40.776 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.772+0000 7f1179c68700 1 -- start start 2020-11-20T11:46:40.778 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.773+0000 7f1179c68700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f11740ae760 0x7f117413e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:40.778 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.773+0000 7f1179c68700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11740aeb40 0x7f1174137ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:40.778 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.773+0000 7f1179c68700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11740661b0 con 0x7f1174065100 2020-11-20T11:46:40.778 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.773+0000 7f1179c68700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f117413ad60 con 0x7f11740ae760 2020-11-20T11:46:40.779 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.773+0000 7f1179c68700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1174066c70 con 0x7f11740aeb40 2020-11-20T11:46:40.779 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.773+0000 7f11737fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f11740ae760 0x7f117413e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:40.779 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.773+0000 7f11737fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f11740ae760 0x7f117413e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46312/0 (socket says 172.21.15.23:46312) 2020-11-20T11:46:40.779 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.773+0000 7f11737fe700 1 -- 172.21.15.23:0/1067189244 learned_addr learned my addr 172.21.15.23:0/1067189244 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:46:40.779 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.773+0000 7f1172ffd700 1 --2- 172.21.15.23:0/1067189244 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11740aeb40 0x7f1174137ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:40.780 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.773+0000 7f11737fe700 1 -- 172.21.15.23:0/1067189244 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11740aeb40 msgr2=0x7f1174137ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:46:40.780 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.773+0000 7f11737fe700 1 --2- 172.21.15.23:0/1067189244 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11740aeb40 0x7f1174137ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:40.780 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.773+0000 7f11737fe700 1 -- 172.21.15.23:0/1067189244 >> v1:172.21.15.155:6789/0 conn(0x7f1174065100 legacy=0x7f1174062b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:40.780 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.773+0000 7f11737fe700 1 -- 172.21.15.23:0/1067189244 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117413e470 con 0x7f11740ae760 2020-11-20T11:46:40.780 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.774+0000 7f11737fe700 1 --2- 172.21.15.23:0/1067189244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11740ae760 0x7f117413e910 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f1164009a10 tx=0x7f1164006ef0).ready entity=mon.0 client_cookie=cc78d3026eeaa873 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:40.781 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.774+0000 7f11727fc700 1 -- 172.21.15.23:0/1067189244 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116400b860 con 0x7f11740ae760 2020-11-20T11:46:40.781 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.774+0000 7f11727fc700 1 -- 172.21.15.23:0/1067189244 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1164000e90 con 0x7f11740ae760 2020-11-20T11:46:40.781 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.774+0000 7f11727fc700 1 -- 172.21.15.23:0/1067189244 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116400bbd0 con 0x7f11740ae760 2020-11-20T11:46:40.781 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.774+0000 7f1179c68700 1 -- 172.21.15.23:0/1067189244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11740ae760 msgr2=0x7f117413e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:40.781 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.774+0000 7f1179c68700 1 --2- 172.21.15.23:0/1067189244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11740ae760 0x7f117413e910 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f1164009a10 tx=0x7f1164006ef0).stop 2020-11-20T11:46:40.782 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.774+0000 7f1179c68700 1 -- 172.21.15.23:0/1067189244 shutdown_connections 2020-11-20T11:46:40.782 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.774+0000 7f1179c68700 1 --2- 172.21.15.23:0/1067189244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11740ae760 0x7f117413e910 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:40.782 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.774+0000 7f1179c68700 1 --2- 172.21.15.23:0/1067189244 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11740aeb40 0x7f1174137ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:40.782 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.774+0000 7f1179c68700 1 -- 172.21.15.23:0/1067189244 >> 172.21.15.23:0/1067189244 conn(0x7f1174061d80 msgr2=0x7f11740ae050 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:40.782 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.774+0000 7f1179c68700 1 -- 172.21.15.23:0/1067189244 shutdown_connections 2020-11-20T11:46:40.783 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.774+0000 7f1179c68700 1 -- 172.21.15.23:0/1067189244 wait complete. 2020-11-20T11:46:40.785 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.775+0000 7f1179c68700 1 Processor -- start 2020-11-20T11:46:40.785 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.775+0000 7f1179c68700 1 -- start start 2020-11-20T11:46:40.785 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.776+0000 7f1179c68700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1174065100 0x7f11740ad930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:40.786 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.776+0000 7f1179c68700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11740ae760 0x7f117413c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:40.786 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.776+0000 7f1179c68700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11740aeb40 0x7f1174133110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:40.786 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.776+0000 7f1179c68700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117413c5d0 con 0x7f11740ae760 2020-11-20T11:46:40.786 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.776+0000 7f1179c68700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1174136b90 con 0x7f1174065100 2020-11-20T11:46:40.787 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.776+0000 7f1179c68700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1174136e30 con 0x7f11740aeb40 2020-11-20T11:46:40.787 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.776+0000 7f1172ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11740ae760 0x7f117413c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:40.787 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.776+0000 7f1172ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11740ae760 0x7f117413c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46322/0 (socket says 172.21.15.23:46322) 2020-11-20T11:46:40.787 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.776+0000 7f1172ffd700 1 -- 172.21.15.23:0/1718177205 learned_addr learned my addr 172.21.15.23:0/1718177205 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:46:40.787 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.776+0000 7f11737fe700 1 --2- 172.21.15.23:0/1718177205 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1174065100 0x7f11740ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:40.788 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.776+0000 7f1173fff700 1 --2- 172.21.15.23:0/1718177205 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11740aeb40 0x7f1174133110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:40.788 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.776+0000 7f1172ffd700 1 -- 172.21.15.23:0/1718177205 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11740aeb40 msgr2=0x7f1174133110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:40.788 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.776+0000 7f1172ffd700 1 --2- 172.21.15.23:0/1718177205 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11740aeb40 0x7f1174133110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:40.788 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.776+0000 7f1172ffd700 1 -- 172.21.15.23:0/1718177205 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1174065100 msgr2=0x7f11740ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:40.788 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.776+0000 7f1172ffd700 1 --2- 172.21.15.23:0/1718177205 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1174065100 0x7f11740ad930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:40.788 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.776+0000 7f1172ffd700 1 -- 172.21.15.23:0/1718177205 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11740ace20 con 0x7f11740ae760 2020-11-20T11:46:40.789 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.777+0000 7f1172ffd700 1 --2- 172.21.15.23:0/1718177205 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11740ae760 0x7f117413c0e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f116800ebd0 tx=0x7f1168006e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:40.789 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.777+0000 7f11537fe700 1 -- 172.21.15.23:0/1718177205 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116800c850 con 0x7f11740ae760 2020-11-20T11:46:40.789 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.777+0000 7f11537fe700 1 -- 172.21.15.23:0/1718177205 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f116800c9b0 con 0x7f11740ae760 2020-11-20T11:46:40.789 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.777+0000 7f1179c68700 1 -- 172.21.15.23:0/1718177205 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11741367d0 con 0x7f11740ae760 2020-11-20T11:46:40.789 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.777+0000 7f11537fe700 1 -- 172.21.15.23:0/1718177205 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116800cd30 con 0x7f11740ae760 2020-11-20T11:46:40.790 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.777+0000 7f1179c68700 1 -- 172.21.15.23:0/1718177205 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f117413d1f0 con 0x7f11740ae760 2020-11-20T11:46:40.790 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.778+0000 7f1179c68700 1 -- 172.21.15.23:0/1718177205 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1174062760 con 0x7f11740ae760 2020-11-20T11:46:40.790 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.778+0000 7f11537fe700 1 -- 172.21.15.23:0/1718177205 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f1168012600 con 0x7f11740ae760 2020-11-20T11:46:40.790 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.778+0000 7f11537fe700 1 --2- 172.21.15.23:0/1718177205 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f114c056d60 0x7f114c059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:40.790 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.778+0000 7f11537fe700 1 -- 172.21.15.23:0/1718177205 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f11680702e0 con 0x7f11740ae760 2020-11-20T11:46:40.791 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.778+0000 7f11737fe700 1 --2- 172.21.15.23:0/1718177205 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f114c056d60 0x7f114c059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:40.791 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.779+0000 7f11737fe700 1 --2- 172.21.15.23:0/1718177205 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f114c056d60 0x7f114c059220 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f117413e280 tx=0x7f1164016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:40.791 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:40.782+0000 7f11537fe700 1 -- 172.21.15.23:0/1718177205 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f11680455d0 con 0x7f11740ae760 2020-11-20T11:46:40.905 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:40.882+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 86 slow ops, oldest is osd_op(client.4659.0:52502 4.2s0 4:568ccee9:::100000003a4.0000000b:head [write 3776512~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:41.084 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:41.081+0000 7f1179c68700 1 -- 172.21.15.23:0/1718177205 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f117413d470 con 0x7f11740ae760 2020-11-20T11:46:41.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:41.082+0000 7f11537fe700 1 -- 172.21.15.23:0/1718177205 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f1168048bf0 con 0x7f11740ae760 2020-11-20T11:46:41.087 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:41.084+0000 7f1179c68700 1 -- 172.21.15.23:0/1718177205 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f114c056d60 msgr2=0x7f114c059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:41.087 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:41.084+0000 7f1179c68700 1 --2- 172.21.15.23:0/1718177205 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f114c056d60 0x7f114c059220 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f117413e280 tx=0x7f1164016000).stop 2020-11-20T11:46:41.087 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:41.084+0000 7f1179c68700 1 -- 172.21.15.23:0/1718177205 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11740ae760 msgr2=0x7f117413c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:41.087 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:41.084+0000 7f1179c68700 1 --2- 172.21.15.23:0/1718177205 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11740ae760 0x7f117413c0e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f116800ebd0 tx=0x7f1168006e30).stop 2020-11-20T11:46:41.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:41.084+0000 7f1179c68700 1 -- 172.21.15.23:0/1718177205 shutdown_connections 2020-11-20T11:46:41.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:41.084+0000 7f1179c68700 1 --2- 172.21.15.23:0/1718177205 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f114c056d60 0x7f114c059220 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:41.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:41.084+0000 7f1179c68700 1 --2- 172.21.15.23:0/1718177205 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f11740aeb40 0x7f1174133110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:41.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:41.084+0000 7f1179c68700 1 --2- 172.21.15.23:0/1718177205 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1174065100 0x7f11740ad930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:41.089 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:41.084+0000 7f1179c68700 1 --2- 172.21.15.23:0/1718177205 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11740ae760 0x7f117413c0e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:41.089 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:41.084+0000 7f1179c68700 1 -- 172.21.15.23:0/1718177205 >> 172.21.15.23:0/1718177205 conn(0x7f1174061d80 msgr2=0x7f11740a8920 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:41.089 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:41.084+0000 7f1179c68700 1 -- 172.21.15.23:0/1718177205 shutdown_connections 2020-11-20T11:46:41.089 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:41.085+0000 7f1179c68700 1 -- 172.21.15.23:0/1718177205 wait complete. 2020-11-20T11:46:41.100 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2020-11-20T11:46:41.950 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:41.914+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 93 slow ops, oldest is osd_op(client.4659.0:52502 4.2s0 4:568ccee9:::100000003a4.0000000b:head [write 3776512~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:42.969 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:42.926+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4659.0:52502 4.2s0 4:568ccee9:::100000003a4.0000000b:head [write 3776512~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:43.981 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:43.977+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 96 slow ops, oldest is osd_op(client.4659.0:52502 4.2s0 4:568ccee9:::100000003a4.0000000b:head [write 3776512~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:45.019 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:45.007+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 99 slow ops, oldest is osd_op(client.4659.0:52502 4.2s0 4:568ccee9:::100000003a4.0000000b:head [write 3776512~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:46.037 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:46.006+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 100 slow ops, oldest is osd_op(client.4659.0:53229 4.2s0 4:449dcbc8:::100000003b6.00000013:head [write 3899392~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:47.037 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:47.009+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.4659.0:53229 4.2s0 4:449dcbc8:::100000003b6.00000013:head [write 3899392~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:48.119 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:48.068+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 108 slow ops, oldest is osd_op(client.4659.0:53229 4.2s0 4:449dcbc8:::100000003b6.00000013:head [write 3899392~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:49.111 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:49.047+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 110 slow ops, oldest is osd_op(client.4659.0:53229 4.2s0 4:449dcbc8:::100000003b6.00000013:head [write 3899392~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:50.092 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:50.030+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 111 slow ops, oldest is osd_op(client.4659.0:53229 4.2s0 4:449dcbc8:::100000003b6.00000013:head [write 3899392~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:51.101 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:46:51.102 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:46:51.131 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:51.093+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 114 slow ops, oldest is osd_op(client.4659.0:53229 4.2s0 4:449dcbc8:::100000003b6.00000013:head [write 3899392~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:51.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.379+0000 7f7b7d6a5700 1 Processor -- start 2020-11-20T11:46:51.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.379+0000 7f7b7d6a5700 1 -- start start 2020-11-20T11:46:51.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.380+0000 7f7b7d6a5700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f7b780a6e10 0x7f7b78061850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:51.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.380+0000 7f7b7d6a5700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b78062ad0 con 0x7f7b78137870 2020-11-20T11:46:51.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.380+0000 7f7b7d6a5700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b7813b580 con 0x7f7b78137410 2020-11-20T11:46:51.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.380+0000 7f7b7d6a5700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b780a61f0 con 0x7f7b780a6e10 2020-11-20T11:46:51.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.380+0000 7f7b76ffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f7b780a6e10 0x7f7b78061850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:51.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.380+0000 7f7b777fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f7b78137870 0x7f7b7813ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:60578/0 (socket says 172.21.15.23:60578) 2020-11-20T11:46:51.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.380+0000 7f7b777fe700 1 -- 172.21.15.23:0/3011613374 learned_addr learned my addr 172.21.15.23:0/3011613374 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:46:51.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.380+0000 7f7b76ffd700 1 -- 172.21.15.23:0/3011613374 >> v1:172.21.15.155:6789/0 conn(0x7f7b78137410 legacy=0x7f7b7813a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:51.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.380+0000 7f7b76ffd700 1 -- 172.21.15.23:0/3011613374 >> v1:172.21.15.23:6789/0 conn(0x7f7b78137870 legacy=0x7f7b7813ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:51.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.380+0000 7f7b75ffb700 1 -- 172.21.15.23:0/3011613374 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694972046 0 0) 0x7f7b78062ad0 con 0x7f7b78137870 2020-11-20T11:46:51.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.380+0000 7f7b76ffd700 1 -- 172.21.15.23:0/3011613374 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b780a63e0 con 0x7f7b780a6e10 2020-11-20T11:46:51.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.380+0000 7f7b76ffd700 1 --2- 172.21.15.23:0/3011613374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b780a6e10 0x7f7b78061850 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f7b68006fd0 tx=0x7f7b68006e30).ready entity=mon.0 client_cookie=297f7aedbac1bf04 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:51.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.380+0000 7f7b75ffb700 1 -- 172.21.15.23:0/3011613374 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b680097e0 con 0x7f7b780a6e10 2020-11-20T11:46:51.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.380+0000 7f7b75ffb700 1 -- 172.21.15.23:0/3011613374 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b68000e90 con 0x7f7b780a6e10 2020-11-20T11:46:51.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.380+0000 7f7b75ffb700 1 -- 172.21.15.23:0/3011613374 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b68009b50 con 0x7f7b780a6e10 2020-11-20T11:46:51.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.381+0000 7f7b7d6a5700 1 -- 172.21.15.23:0/3011613374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b780a6e10 msgr2=0x7f7b78061850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:51.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.381+0000 7f7b7d6a5700 1 --2- 172.21.15.23:0/3011613374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b780a6e10 0x7f7b78061850 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f7b68006fd0 tx=0x7f7b68006e30).stop 2020-11-20T11:46:51.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.381+0000 7f7b7d6a5700 1 -- 172.21.15.23:0/3011613374 shutdown_connections 2020-11-20T11:46:51.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.381+0000 7f7b7d6a5700 1 --2- 172.21.15.23:0/3011613374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b780a6e10 0x7f7b78061850 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:51.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.381+0000 7f7b7d6a5700 1 -- 172.21.15.23:0/3011613374 >> 172.21.15.23:0/3011613374 conn(0x7f7b78060c80 msgr2=0x7f7b780acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:51.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.381+0000 7f7b7d6a5700 1 -- 172.21.15.23:0/3011613374 shutdown_connections 2020-11-20T11:46:51.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.381+0000 7f7b7d6a5700 1 -- 172.21.15.23:0/3011613374 wait complete. 2020-11-20T11:46:51.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.382+0000 7f7b7d6a5700 1 Processor -- start 2020-11-20T11:46:51.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.382+0000 7f7b7d6a5700 1 -- start start 2020-11-20T11:46:51.394 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.383+0000 7f7b7d6a5700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b780a6e10 0x7f7b780ad900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:51.394 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.383+0000 7f7b7d6a5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b78137410 0x7f7b780acc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:51.394 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.383+0000 7f7b7d6a5700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b78137870 0x7f7b7812e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:51.395 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.383+0000 7f7b7d6a5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b780ab080 con 0x7f7b78137410 2020-11-20T11:46:51.395 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.383+0000 7f7b7d6a5700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b780ab320 con 0x7f7b78137870 2020-11-20T11:46:51.395 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.383+0000 7f7b7d6a5700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b780ab5c0 con 0x7f7b780a6e10 2020-11-20T11:46:51.395 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.383+0000 7f7b767fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b78137410 0x7f7b780acc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:51.395 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.383+0000 7f7b76ffd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b780a6e10 0x7f7b780ad900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:51.396 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.383+0000 7f7b767fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b78137410 0x7f7b780acc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46336/0 (socket says 172.21.15.23:46336) 2020-11-20T11:46:51.396 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.383+0000 7f7b767fc700 1 -- 172.21.15.23:0/2907666023 learned_addr learned my addr 172.21.15.23:0/2907666023 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:46:51.396 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.383+0000 7f7b777fe700 1 --2- 172.21.15.23:0/2907666023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b78137870 0x7f7b7812e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:51.396 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.383+0000 7f7b767fc700 1 -- 172.21.15.23:0/2907666023 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b780a6e10 msgr2=0x7f7b780ad900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:51.396 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.383+0000 7f7b767fc700 1 --2- 172.21.15.23:0/2907666023 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b780a6e10 0x7f7b780ad900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:51.397 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.383+0000 7f7b767fc700 1 -- 172.21.15.23:0/2907666023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b78137870 msgr2=0x7f7b7812e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:51.397 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.383+0000 7f7b767fc700 1 --2- 172.21.15.23:0/2907666023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b78137870 0x7f7b7812e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:51.397 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.383+0000 7f7b767fc700 1 -- 172.21.15.23:0/2907666023 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b780a1ad0 con 0x7f7b78137410 2020-11-20T11:46:51.397 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.383+0000 7f7b767fc700 1 --2- 172.21.15.23:0/2907666023 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b78137410 0x7f7b780acc10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f7b6c00c710 tx=0x7f7b6c007c90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:51.397 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.383+0000 7f7b5effd700 1 -- 172.21.15.23:0/2907666023 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c007020 con 0x7f7b78137410 2020-11-20T11:46:51.398 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.383+0000 7f7b5effd700 1 -- 172.21.15.23:0/2907666023 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b6c001e90 con 0x7f7b78137410 2020-11-20T11:46:51.398 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.383+0000 7f7b7d6a5700 1 -- 172.21.15.23:0/2907666023 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b780aad00 con 0x7f7b78137410 2020-11-20T11:46:51.398 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.383+0000 7f7b5effd700 1 -- 172.21.15.23:0/2907666023 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c0073b0 con 0x7f7b78137410 2020-11-20T11:46:51.398 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.384+0000 7f7b7d6a5700 1 -- 172.21.15.23:0/2907666023 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b7812e8c0 con 0x7f7b78137410 2020-11-20T11:46:51.399 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.384+0000 7f7b7d6a5700 1 -- 172.21.15.23:0/2907666023 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b780aaef0 con 0x7f7b78137410 2020-11-20T11:46:51.399 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.385+0000 7f7b5effd700 1 -- 172.21.15.23:0/2907666023 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7b6c01a060 con 0x7f7b78137410 2020-11-20T11:46:51.400 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.385+0000 7f7b5effd700 1 --2- 172.21.15.23:0/2907666023 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7b54056d60 0x7f7b54059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:51.400 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.385+0000 7f7b5effd700 1 -- 172.21.15.23:0/2907666023 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f7b6c06f800 con 0x7f7b78137410 2020-11-20T11:46:51.400 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.385+0000 7f7b76ffd700 1 --2- 172.21.15.23:0/2907666023 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7b54056d60 0x7f7b54059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:51.400 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.386+0000 7f7b76ffd700 1 --2- 172.21.15.23:0/2907666023 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7b54056d60 0x7f7b54059220 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f7b6800b150 tx=0x7f7b68015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:51.401 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.389+0000 7f7b5effd700 1 -- 172.21.15.23:0/2907666023 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7b6c0449e0 con 0x7f7b78137410 2020-11-20T11:46:51.673 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.670+0000 7f7b7d6a5700 1 -- 172.21.15.23:0/2907666023 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7b7812eb40 con 0x7f7b78137410 2020-11-20T11:46:51.674 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.671+0000 7f7b5effd700 1 -- 172.21.15.23:0/2907666023 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1479 (secure 0 0 0) 0x7f7b7812eb40 con 0x7f7b78137410 2020-11-20T11:46:51.674 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":132,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:46:51.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.674+0000 7f7b7d6a5700 1 -- 172.21.15.23:0/2907666023 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7b54056d60 msgr2=0x7f7b54059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:51.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.674+0000 7f7b7d6a5700 1 --2- 172.21.15.23:0/2907666023 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7b54056d60 0x7f7b54059220 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f7b6800b150 tx=0x7f7b68015000).stop 2020-11-20T11:46:51.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.674+0000 7f7b7d6a5700 1 -- 172.21.15.23:0/2907666023 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b78137410 msgr2=0x7f7b780acc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:51.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.674+0000 7f7b7d6a5700 1 --2- 172.21.15.23:0/2907666023 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b78137410 0x7f7b780acc10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f7b6c00c710 tx=0x7f7b6c007c90).stop 2020-11-20T11:46:51.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.674+0000 7f7b7d6a5700 1 -- 172.21.15.23:0/2907666023 shutdown_connections 2020-11-20T11:46:51.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.674+0000 7f7b7d6a5700 1 --2- 172.21.15.23:0/2907666023 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7b78137870 0x7f7b7812e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:51.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.674+0000 7f7b7d6a5700 1 --2- 172.21.15.23:0/2907666023 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7b54056d60 0x7f7b54059220 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:51.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.674+0000 7f7b7d6a5700 1 --2- 172.21.15.23:0/2907666023 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7b780a6e10 0x7f7b780ad900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:51.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.674+0000 7f7b7d6a5700 1 --2- 172.21.15.23:0/2907666023 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b78137410 0x7f7b780acc10 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:51.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.674+0000 7f7b7d6a5700 1 -- 172.21.15.23:0/2907666023 >> 172.21.15.23:0/2907666023 conn(0x7f7b78060c80 msgr2=0x7f7b78134240 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:51.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.674+0000 7f7b7d6a5700 1 -- 172.21.15.23:0/2907666023 shutdown_connections 2020-11-20T11:46:51.680 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.674+0000 7f7b7d6a5700 1 -- 172.21.15.23:0/2907666023 wait complete. 2020-11-20T11:46:51.690 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:46:51.690 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2020-11-20T11:46:51.691 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:46:51.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.866+0000 7f4b02e6e700 1 Processor -- start 2020-11-20T11:46:51.870 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.867+0000 7f4b02e6e700 1 -- start start 2020-11-20T11:46:51.871 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.867+0000 7f4b02e6e700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f4afc0aaab0 0x7f4afc13ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:51.871 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.867+0000 7f4b02e6e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4afc0aae90 0x7f4afc1381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:51.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.867+0000 7f4b02e6e700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4afc13e600 con 0x7f4afc13aeb0 2020-11-20T11:46:51.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.867+0000 7f4b02e6e700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4afc13e8a0 con 0x7f4afc0aaab0 2020-11-20T11:46:51.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.867+0000 7f4b02e6e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4afc0ab6d0 con 0x7f4afc0aae90 2020-11-20T11:46:51.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.867+0000 7f4b00c0a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f4afc0aaab0 0x7f4afc13ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:51.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.867+0000 7f4b0140b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f4afc13aeb0 0x7f4afc0a8c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:60590/0 (socket says 172.21.15.23:60590) 2020-11-20T11:46:51.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.867+0000 7f4b0140b700 1 -- 172.21.15.23:0/4028775861 learned_addr learned my addr 172.21.15.23:0/4028775861 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:46:51.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.868+0000 7f4afbfff700 1 --2- 172.21.15.23:0/4028775861 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4afc0aae90 0x7f4afc1381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:51.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.868+0000 7f4b00c0a700 1 -- 172.21.15.23:0/4028775861 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4afc0aae90 msgr2=0x7f4afc1381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:46:51.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.868+0000 7f4b00c0a700 1 --2- 172.21.15.23:0/4028775861 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4afc0aae90 0x7f4afc1381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:51.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.868+0000 7f4b00c0a700 1 -- 172.21.15.23:0/4028775861 >> v1:172.21.15.23:6789/0 conn(0x7f4afc13aeb0 legacy=0x7f4afc0a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:51.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.868+0000 7f4b00c0a700 1 -- 172.21.15.23:0/4028775861 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4afc0a9370 con 0x7f4afc0aaab0 2020-11-20T11:46:51.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.868+0000 7f4b00c0a700 1 --2- 172.21.15.23:0/4028775861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4afc0aaab0 0x7f4afc13ed40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f4af0006fd0 tx=0x7f4af0006e30).ready entity=mon.0 client_cookie=9879de196fe3a7d1 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:51.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.871+0000 7f4afb7fe700 1 -- 172.21.15.23:0/4028775861 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af000bec0 con 0x7f4afc0aaab0 2020-11-20T11:46:51.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.871+0000 7f4afb7fe700 1 -- 172.21.15.23:0/4028775861 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af0000e90 con 0x7f4afc0aaab0 2020-11-20T11:46:51.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.871+0000 7f4afb7fe700 1 -- 172.21.15.23:0/4028775861 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af00098f0 con 0x7f4afc0aaab0 2020-11-20T11:46:51.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.871+0000 7f4b02e6e700 1 -- 172.21.15.23:0/4028775861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4afc0aaab0 msgr2=0x7f4afc13ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:51.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.871+0000 7f4b02e6e700 1 --2- 172.21.15.23:0/4028775861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4afc0aaab0 0x7f4afc13ed40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f4af0006fd0 tx=0x7f4af0006e30).stop 2020-11-20T11:46:51.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.872+0000 7f4b02e6e700 1 -- 172.21.15.23:0/4028775861 shutdown_connections 2020-11-20T11:46:51.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.872+0000 7f4b02e6e700 1 --2- 172.21.15.23:0/4028775861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4afc0aaab0 0x7f4afc13ed40 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:51.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.872+0000 7f4b02e6e700 1 --2- 172.21.15.23:0/4028775861 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4afc0aae90 0x7f4afc1381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:51.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.872+0000 7f4b02e6e700 1 -- 172.21.15.23:0/4028775861 >> 172.21.15.23:0/4028775861 conn(0x7f4afc0678e0 msgr2=0x7f4afc067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:51.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.872+0000 7f4b02e6e700 1 -- 172.21.15.23:0/4028775861 shutdown_connections 2020-11-20T11:46:51.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.872+0000 7f4b02e6e700 1 -- 172.21.15.23:0/4028775861 wait complete. 2020-11-20T11:46:51.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.873+0000 7f4b02e6e700 1 Processor -- start 2020-11-20T11:46:51.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.873+0000 7f4b02e6e700 1 -- start start 2020-11-20T11:46:51.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.874+0000 7f4b02e6e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4afc0aaab0 0x7f4afc140800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:51.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.874+0000 7f4b02e6e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4afc0aae90 0x7f4afc0abbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:51.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.874+0000 7f4b02e6e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4afc0a5720 0x7f4afc13c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:51.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.875+0000 7f4b00c0a700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4afc0aaab0 0x7f4afc140800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:51.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.875+0000 7f4afbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4afc0aae90 0x7f4afc0abbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:51.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.875+0000 7f4afbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4afc0aae90 0x7f4afc0abbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46348/0 (socket says 172.21.15.23:46348) 2020-11-20T11:46:51.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.875+0000 7f4b00c0a700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4afc0aaab0 0x7f4afc140800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:56586/0 (socket says 172.21.15.23:56586) 2020-11-20T11:46:51.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.875+0000 7f4b0140b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4afc0a5720 0x7f4afc13c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:51.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.875+0000 7f4afbfff700 1 -- 172.21.15.23:0/3130246960 learned_addr learned my addr 172.21.15.23:0/3130246960 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:46:51.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.875+0000 7f4b02e6e700 1 -- 172.21.15.23:0/3130246960 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4afc0ac410 con 0x7f4afc0aae90 2020-11-20T11:46:51.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.875+0000 7f4b02e6e700 1 -- 172.21.15.23:0/3130246960 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4afc141370 con 0x7f4afc0a5720 2020-11-20T11:46:51.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.875+0000 7f4b02e6e700 1 -- 172.21.15.23:0/3130246960 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4afc140e30 con 0x7f4afc0aaab0 2020-11-20T11:46:51.882 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.876+0000 7f4afbfff700 1 -- 172.21.15.23:0/3130246960 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4afc0aaab0 msgr2=0x7f4afc140800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:51.882 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.876+0000 7f4afbfff700 1 --2- 172.21.15.23:0/3130246960 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4afc0aaab0 0x7f4afc140800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:51.882 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.876+0000 7f4afbfff700 1 -- 172.21.15.23:0/3130246960 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4afc0a5720 msgr2=0x7f4afc13c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:51.883 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.876+0000 7f4afbfff700 1 --2- 172.21.15.23:0/3130246960 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4afc0a5720 0x7f4afc13c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:51.883 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.876+0000 7f4afbfff700 1 -- 172.21.15.23:0/3130246960 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4afc1410d0 con 0x7f4afc0aae90 2020-11-20T11:46:51.883 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.876+0000 7f4afbfff700 1 --2- 172.21.15.23:0/3130246960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4afc0aae90 0x7f4afc0abbc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f4ae800eba0 tx=0x7f4ae8006e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:51.883 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.878+0000 7f4b00c0a700 1 --2- 172.21.15.23:0/3130246960 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4afc0aaab0 0x7f4afc140800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2020-11-20T11:46:51.884 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.880+0000 7f4af8ff9700 1 -- 172.21.15.23:0/3130246960 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae800c9b0 con 0x7f4afc0aae90 2020-11-20T11:46:51.884 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.880+0000 7f4af8ff9700 1 -- 172.21.15.23:0/3130246960 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ae800cb10 con 0x7f4afc0aae90 2020-11-20T11:46:51.885 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.880+0000 7f4b02e6e700 1 -- 172.21.15.23:0/3130246960 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4afc13cce0 con 0x7f4afc0aae90 2020-11-20T11:46:51.885 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.880+0000 7f4af8ff9700 1 -- 172.21.15.23:0/3130246960 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae800ce90 con 0x7f4afc0aae90 2020-11-20T11:46:51.885 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.880+0000 7f4b02e6e700 1 -- 172.21.15.23:0/3130246960 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4afc1335b0 con 0x7f4afc0aae90 2020-11-20T11:46:51.885 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.881+0000 7f4b02e6e700 1 --2- 172.21.15.23:0/3130246960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4afc13aeb0 0x7f4afc133890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:51.885 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.881+0000 7f4b02e6e700 1 -- 172.21.15.23:0/3130246960 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4afc140cf0 con 0x7f4afc13aeb0 2020-11-20T11:46:51.886 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.881+0000 7f4b00c0a700 1 --2- 172.21.15.23:0/3130246960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4afc13aeb0 0x7f4afc133890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:51.887 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.882+0000 7f4b00c0a700 1 -- 172.21.15.23:0/3130246960 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4afc05c7d0 con 0x7f4afc13aeb0 2020-11-20T11:46:51.887 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.882+0000 7f4b00c0a700 1 --2- 172.21.15.23:0/3130246960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4afc13aeb0 0x7f4afc133890 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f4af0006fd0 tx=0x7f4af0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:51.887 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.882+0000 7f4af8ff9700 1 -- 172.21.15.23:0/3130246960 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af0011860 con 0x7f4afc13aeb0 2020-11-20T11:46:51.888 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.882+0000 7f4af8ff9700 1 -- 172.21.15.23:0/3130246960 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4ae8012600 con 0x7f4afc0aae90 2020-11-20T11:46:51.888 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.883+0000 7f4af8ff9700 1 --2- 172.21.15.23:0/3130246960 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4ad0056e50 0x7f4ad0059310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:51.888 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.883+0000 7f4af8ff9700 1 -- 172.21.15.23:0/3130246960 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f4af00119c0 con 0x7f4afc13aeb0 2020-11-20T11:46:51.888 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.883+0000 7f4af8ff9700 1 -- 172.21.15.23:0/3130246960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4afc13aeb0 msgr2=0x7f4afc133890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:51.888 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.883+0000 7f4af8ff9700 1 --2- 172.21.15.23:0/3130246960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4afc13aeb0 0x7f4afc133890 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f4af0006fd0 tx=0x7f4af0015000).stop 2020-11-20T11:46:51.889 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.883+0000 7f4af8ff9700 1 -- 172.21.15.23:0/3130246960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4afc13aeb0 msgr2=0x7f4afc133890 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:46:51.889 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.883+0000 7f4af8ff9700 1 --2- 172.21.15.23:0/3130246960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4afc13aeb0 0x7f4afc133890 secure :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0x7f4af0006fd0 tx=0x7f4af0015000).stop 2020-11-20T11:46:51.889 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.883+0000 7f4af8ff9700 1 -- 172.21.15.23:0/3130246960 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f4ae806fcc0 con 0x7f4afc0aae90 2020-11-20T11:46:51.889 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.883+0000 7f4b0140b700 1 --2- 172.21.15.23:0/3130246960 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4ad0056e50 0x7f4ad0059310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:51.890 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.883+0000 7f4b0140b700 1 --2- 172.21.15.23:0/3130246960 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4ad0056e50 0x7f4ad0059310 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f4afc0a5ba0 tx=0x7f4af4009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:51.893 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.890+0000 7f4b02e6e700 1 --2- 172.21.15.23:0/3130246960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4afc133d80 0x7f4afc14d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:51.893 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.890+0000 7f4b02e6e700 1 -- 172.21.15.23:0/3130246960 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4afc13c4e0 con 0x7f4afc133d80 2020-11-20T11:46:51.893 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.890+0000 7f4b00c0a700 1 --2- 172.21.15.23:0/3130246960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4afc133d80 0x7f4afc14d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:51.895 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.890+0000 7f4b00c0a700 1 -- 172.21.15.23:0/3130246960 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4afc044710 con 0x7f4afc133d80 2020-11-20T11:46:51.895 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.891+0000 7f4b00c0a700 1 --2- 172.21.15.23:0/3130246960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4afc133d80 0x7f4afc14d820 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f4afc13d250 tx=0x7f4af001f000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:51.895 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.891+0000 7f4af8ff9700 1 -- 172.21.15.23:0/3130246960 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af001e710 con 0x7f4afc133d80 2020-11-20T11:46:51.896 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.891+0000 7f4af8ff9700 1 -- 172.21.15.23:0/3130246960 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4787 (secure 0 0 0) 0x7f4af0009840 con 0x7f4afc133d80 2020-11-20T11:46:51.896 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.891+0000 7f4af8ff9700 1 -- 172.21.15.23:0/3130246960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4afc133d80 msgr2=0x7f4afc14d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:51.896 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.891+0000 7f4af8ff9700 1 --2- 172.21.15.23:0/3130246960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4afc133d80 0x7f4afc14d820 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f4afc13d250 tx=0x7f4af001f000).stop 2020-11-20T11:46:51.896 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.891+0000 7f4af8ff9700 1 -- 172.21.15.23:0/3130246960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4afc133d80 msgr2=0x7f4afc14d820 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:46:51.897 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.891+0000 7f4af8ff9700 1 --2- 172.21.15.23:0/3130246960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4afc133d80 0x7f4afc14d820 secure :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0x7f4afc13d250 tx=0x7f4af001f000).stop 2020-11-20T11:46:51.897 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.891+0000 7f4b02e6e700 1 -- 172.21.15.23:0/3130246960 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4ad0056e50 msgr2=0x7f4ad0059310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:51.897 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.891+0000 7f4b02e6e700 1 --2- 172.21.15.23:0/3130246960 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4ad0056e50 0x7f4ad0059310 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f4afc0a5ba0 tx=0x7f4af4009040).stop 2020-11-20T11:46:51.897 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.891+0000 7f4b02e6e700 1 -- 172.21.15.23:0/3130246960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4afc0aae90 msgr2=0x7f4afc0abbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:51.897 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.891+0000 7f4b02e6e700 1 --2- 172.21.15.23:0/3130246960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4afc0aae90 0x7f4afc0abbc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f4ae800eba0 tx=0x7f4ae8006e30).stop 2020-11-20T11:46:51.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.891+0000 7f4b0140b700 1 -- 172.21.15.23:0/3130246960 reap_dead start 2020-11-20T11:46:51.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.891+0000 7f4b02e6e700 1 -- 172.21.15.23:0/3130246960 shutdown_connections 2020-11-20T11:46:51.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.891+0000 7f4b02e6e700 1 --2- 172.21.15.23:0/3130246960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4afc0aae90 0x7f4afc0abbc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:51.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.891+0000 7f4b02e6e700 1 -- 172.21.15.23:0/3130246960 >> 172.21.15.23:0/3130246960 conn(0x7f4afc0678e0 msgr2=0x7f4afc1364f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:51.898 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.892+0000 7f4b02e6e700 1 -- 172.21.15.23:0/3130246960 shutdown_connections 2020-11-20T11:46:51.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:51.892+0000 7f4b02e6e700 1 -- 172.21.15.23:0/3130246960 wait complete. 2020-11-20T11:46:51.899 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:46:51.899 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:46:51.899 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:46:51.900 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:46:51.900 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 132, 2020-11-20T11:46:51.900 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:46:51.900 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:46:51.900 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:46:51.901 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:46:51.901 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:51.901 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 14, 2020-11-20T11:46:51.901 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:46:51.902 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:46:51.902 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:46:51.902 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:46:51.902 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:46:51.902 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:46:51.903 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:46:51.903 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:46:51.903 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:46:51.904 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:46:51.904 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:46:51.904 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:51.904 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:46:51.904 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:46:51.905 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:46:51.905 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:46:51.905 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:46:51.905 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:46:51.905 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:46:51.906 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:46:51.906 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:46:51.906 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:46:51.906 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:51.906 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:51.907 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:46:51.907 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:46:51.907 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:46:51.907 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:46:51.907 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:46:51.908 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:46:51.908 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:46:51.908 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:46:51.908 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:46:51.908 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:46:51.909 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:46:51.909 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:46:51.909 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:46:51.909 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:46:51.909 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:46:51.910 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:46:51.910 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:46:51.910 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:46:51.910 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:46:51.910 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:46:51.911 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:46:51.911 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:46:51.911 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:46:51.911 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:51.911 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:46:51.912 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:51.912 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:46:51.912 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:51.912 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:46:51.912 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:46:51.913 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:46:51.913 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:46:51.913 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:51.913 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:46:51.913 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:46:51.914 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:51.914 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:51.914 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:51.914 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:46:51.915 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:46:51.915 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:51.915 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:51.915 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:51.915 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:51.916 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:46:51.916 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:46:51.916 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:46:51.916 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:46:51.916 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:46:51.917 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:51.917 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:51.917 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:46:51.917 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:46:51.917 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:46:51.918 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:46:51.918 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:51.918 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:46:51.918 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:46:51.918 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:51.919 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:51.919 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:51.919 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:46:51.919 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:46:51.919 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:51.920 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:51.920 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:51.920 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:51.920 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:46:51.920 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:46:51.921 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:46:51.921 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:46:51.921 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:46:51.921 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:51.921 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:51.922 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:46:51.922 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:46:51.922 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:46:51.922 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:46:51.922 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:51.923 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:46:51.923 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:46:51.923 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:51.923 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:51.923 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:51.924 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:46:51.924 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:46:51.924 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:51.924 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:51.924 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:51.925 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:51.925 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:46:51.925 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:46:51.925 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:46:51.925 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:46:51.926 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:46:51.926 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:51.926 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:51.926 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:51.926 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:46:51.927 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:46:51.927 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:51.927 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:51.927 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:51.928 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:46:51.928 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:51.928 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:51.928 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:46:51.928 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:51.929 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:51.929 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:51.929 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:46:51.929 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:51.930 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:51.930 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:46:51.930 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:51.930 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:51.930 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:51.931 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:46:51.931 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:51.931 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:51.931 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:46:51.931 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:51.931 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:51.932 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:51.932 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 4 2020-11-20T11:46:51.932 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:51.932 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:51.932 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:46:51.933 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:51.933 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:51.933 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:51.933 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:46:51.933 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:51.934 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:51.934 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:51.934 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:46:51.934 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:46:51.936 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:46:52.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.082+0000 7f45fa0bc700 1 Processor -- start 2020-11-20T11:46:52.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.082+0000 7f45fa0bc700 1 -- start start 2020-11-20T11:46:52.087 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.082+0000 7f45fa0bc700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f45f40ad7e0 0x7f45f40a6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:52.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.082+0000 7f45fa0bc700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f45f40a8fb0 0x7f45f413ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:52.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.082+0000 7f45fa0bc700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45f40a6080 con 0x7f45f40a5be0 2020-11-20T11:46:52.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.082+0000 7f45fa0bc700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45f40a6320 con 0x7f45f40ad7e0 2020-11-20T11:46:52.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.082+0000 7f45fa0bc700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f40adbc0 con 0x7f45f40a8fb0 2020-11-20T11:46:52.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.083+0000 7f45f88b9700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f45f40a5be0 0x7f45f413a520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:60608/0 (socket says 172.21.15.23:60608) 2020-11-20T11:46:52.089 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.083+0000 7f45f88b9700 1 -- 172.21.15.23:0/1912525607 learned_addr learned my addr 172.21.15.23:0/1912525607 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:46:52.089 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.083+0000 7f45f90ba700 1 --2- 172.21.15.23:0/1912525607 >> v2:172.21.15.155:3300/0 conn(0x7f45f40ad7e0 0x7f45f40a6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:52.089 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.083+0000 7f45f98bb700 1 --2- 172.21.15.23:0/1912525607 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f45f40a8fb0 0x7f45f413ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:52.089 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.083+0000 7f45e3fff700 1 -- 172.21.15.23:0/1912525607 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2711785169 0 0) 0x7f45f40a6080 con 0x7f45f40a5be0 2020-11-20T11:46:52.089 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.083+0000 7f45e3fff700 1 -- 172.21.15.23:0/1912525607 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45dc004390 con 0x7f45f40a5be0 2020-11-20T11:46:52.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.083+0000 7f45f90ba700 1 -- 172.21.15.23:0/1912525607 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f45f40a8fb0 msgr2=0x7f45f413ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:46:52.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.083+0000 7f45f90ba700 1 --2- 172.21.15.23:0/1912525607 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f45f40a8fb0 0x7f45f413ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:52.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.083+0000 7f45f90ba700 1 -- 172.21.15.23:0/1912525607 >> v1:172.21.15.23:6789/0 conn(0x7f45f40a5be0 legacy=0x7f45f413a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:52.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.083+0000 7f45f90ba700 1 -- 172.21.15.23:0/1912525607 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f40a6620 con 0x7f45f40ad7e0 2020-11-20T11:46:52.091 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.083+0000 7f45f90ba700 1 --2- 172.21.15.23:0/1912525607 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45f40ad7e0 0x7f45f40a6ae0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f45e4006fd0 tx=0x7f45e4006e30).ready entity=mon.1 client_cookie=42cacdc115ed187 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:52.091 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.083+0000 7f45e3fff700 1 -- 172.21.15.23:0/1912525607 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e4009630 con 0x7f45f40ad7e0 2020-11-20T11:46:52.091 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.083+0000 7f45e3fff700 1 -- 172.21.15.23:0/1912525607 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e4000e90 con 0x7f45f40ad7e0 2020-11-20T11:46:52.091 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.083+0000 7f45e3fff700 1 -- 172.21.15.23:0/1912525607 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e40099a0 con 0x7f45f40ad7e0 2020-11-20T11:46:52.092 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.084+0000 7f45fa0bc700 1 -- 172.21.15.23:0/1912525607 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45f40ad7e0 msgr2=0x7f45f40a6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:52.092 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.084+0000 7f45fa0bc700 1 --2- 172.21.15.23:0/1912525607 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45f40ad7e0 0x7f45f40a6ae0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f45e4006fd0 tx=0x7f45e4006e30).stop 2020-11-20T11:46:52.092 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.084+0000 7f45fa0bc700 1 -- 172.21.15.23:0/1912525607 shutdown_connections 2020-11-20T11:46:52.092 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.084+0000 7f45fa0bc700 1 --2- 172.21.15.23:0/1912525607 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f45f40a8fb0 0x7f45f413ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:52.092 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.084+0000 7f45fa0bc700 1 --2- 172.21.15.23:0/1912525607 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45f40ad7e0 0x7f45f40a6ae0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:52.093 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.084+0000 7f45fa0bc700 1 -- 172.21.15.23:0/1912525607 >> 172.21.15.23:0/1912525607 conn(0x7f45f4061850 msgr2=0x7f45f4061c60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:52.093 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.084+0000 7f45fa0bc700 1 -- 172.21.15.23:0/1912525607 shutdown_connections 2020-11-20T11:46:52.093 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.084+0000 7f45fa0bc700 1 -- 172.21.15.23:0/1912525607 wait complete. 2020-11-20T11:46:52.095 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.085+0000 7f45fa0bc700 1 Processor -- start 2020-11-20T11:46:52.095 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.085+0000 7f45fa0bc700 1 -- start start 2020-11-20T11:46:52.095 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.086+0000 7f45fa0bc700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f45f40a5be0 0x7f45f40a3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:52.095 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.086+0000 7f45fa0bc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45f40a8fb0 0x7f45f4139a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:52.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.086+0000 7f45fa0bc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f45f40ad7e0 0x7f45f4067770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:52.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.086+0000 7f45fa0bc700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f4067c60 con 0x7f45f40ad7e0 2020-11-20T11:46:52.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.086+0000 7f45fa0bc700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f4137470 con 0x7f45f40a8fb0 2020-11-20T11:46:52.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.086+0000 7f45fa0bc700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f4137710 con 0x7f45f40a5be0 2020-11-20T11:46:52.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.086+0000 7f45f98bb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f45f40ad7e0 0x7f45f4067770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:52.097 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.086+0000 7f45f98bb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f45f40ad7e0 0x7f45f4067770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46368/0 (socket says 172.21.15.23:46368) 2020-11-20T11:46:52.097 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.086+0000 7f45f98bb700 1 -- 172.21.15.23:0/1418664865 learned_addr learned my addr 172.21.15.23:0/1418664865 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:46:52.097 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.086+0000 7f45f88b9700 1 --2- 172.21.15.23:0/1418664865 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45f40a8fb0 0x7f45f4139a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:52.097 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.086+0000 7f45f90ba700 1 --2- 172.21.15.23:0/1418664865 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f45f40a5be0 0x7f45f40a3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:52.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.087+0000 7f45f98bb700 1 -- 172.21.15.23:0/1418664865 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f45f40a5be0 msgr2=0x7f45f40a3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:52.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.087+0000 7f45f98bb700 1 --2- 172.21.15.23:0/1418664865 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f45f40a5be0 0x7f45f40a3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:52.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.087+0000 7f45f98bb700 1 -- 172.21.15.23:0/1418664865 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45f40a8fb0 msgr2=0x7f45f4139a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:52.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.087+0000 7f45f98bb700 1 --2- 172.21.15.23:0/1418664865 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45f40a8fb0 0x7f45f4139a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:52.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.087+0000 7f45f98bb700 1 -- 172.21.15.23:0/1418664865 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f4136db0 con 0x7f45f40ad7e0 2020-11-20T11:46:52.099 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.087+0000 7f45f98bb700 1 --2- 172.21.15.23:0/1418664865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f45f40ad7e0 0x7f45f4067770 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f45ec00ef50 tx=0x7f45ec009220).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:52.099 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.087+0000 7f45e17fa700 1 -- 172.21.15.23:0/1418664865 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ec00cc10 con 0x7f45f40ad7e0 2020-11-20T11:46:52.099 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.087+0000 7f45fa0bc700 1 -- 172.21.15.23:0/1418664865 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45f4134fd0 con 0x7f45f40ad7e0 2020-11-20T11:46:52.099 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.087+0000 7f45e17fa700 1 -- 172.21.15.23:0/1418664865 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45ec00cd70 con 0x7f45f40ad7e0 2020-11-20T11:46:52.099 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.087+0000 7f45e17fa700 1 -- 172.21.15.23:0/1418664865 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ec00cee0 con 0x7f45f40ad7e0 2020-11-20T11:46:52.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.087+0000 7f45fa0bc700 1 -- 172.21.15.23:0/1418664865 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45f413c9a0 con 0x7f45f40ad7e0 2020-11-20T11:46:52.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.088+0000 7f45fa0bc700 1 --2- 172.21.15.23:0/1418664865 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45f413cc80 0x7f45f413d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:52.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.088+0000 7f45fa0bc700 1 -- 172.21.15.23:0/1418664865 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f4135dd0 con 0x7f45f413cc80 2020-11-20T11:46:52.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.088+0000 7f45e17fa700 1 -- 172.21.15.23:0/1418664865 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f45ec022070 con 0x7f45f40ad7e0 2020-11-20T11:46:52.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.089+0000 7f45e17fa700 1 --2- 172.21.15.23:0/1418664865 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f45cc056d60 0x7f45cc059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:52.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.089+0000 7f45e17fa700 1 -- 172.21.15.23:0/1418664865 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f45ec0708e0 con 0x7f45f40ad7e0 2020-11-20T11:46:52.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.089+0000 7f45f88b9700 1 --2- 172.21.15.23:0/1418664865 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f45cc056d60 0x7f45cc059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:52.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.089+0000 7f45f90ba700 1 --2- 172.21.15.23:0/1418664865 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45f413cc80 0x7f45f413d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:52.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.089+0000 7f45f90ba700 1 -- 172.21.15.23:0/1418664865 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f45f41372f0 con 0x7f45f413cc80 2020-11-20T11:46:52.102 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.089+0000 7f45f88b9700 1 --2- 172.21.15.23:0/1418664865 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f45cc056d60 0x7f45cc059220 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f45f4065070 tx=0x7f45f0009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:52.102 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.089+0000 7f45f90ba700 1 --2- 172.21.15.23:0/1418664865 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45f413cc80 0x7f45f413d060 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f45e4006fd0 tx=0x7f45e4015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:52.102 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.090+0000 7f45e17fa700 1 -- 172.21.15.23:0/1418664865 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e4011b20 con 0x7f45f413cc80 2020-11-20T11:46:52.102 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.090+0000 7f45e17fa700 1 -- 172.21.15.23:0/1418664865 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f45e4011c80 con 0x7f45f413cc80 2020-11-20T11:46:52.102 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.090+0000 7f45e17fa700 1 -- 172.21.15.23:0/1418664865 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45f413cc80 msgr2=0x7f45f413d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:52.103 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.090+0000 7f45e17fa700 1 --2- 172.21.15.23:0/1418664865 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45f413cc80 0x7f45f413d060 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f45e4006fd0 tx=0x7f45e4015000).stop 2020-11-20T11:46:52.103 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.090+0000 7f45e17fa700 1 -- 172.21.15.23:0/1418664865 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45f413cc80 msgr2=0x7f45f413d060 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:46:52.103 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.090+0000 7f45e17fa700 1 --2- 172.21.15.23:0/1418664865 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45f413cc80 0x7f45f413d060 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:52.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.102+0000 7f45fa0bc700 1 --2- 172.21.15.23:0/1418664865 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45f413d550 0x7f45f4159f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:52.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.102+0000 7f45fa0bc700 1 -- 172.21.15.23:0/1418664865 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f4057dc0 con 0x7f45f413d550 2020-11-20T11:46:52.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.102+0000 7f45f90ba700 1 --2- 172.21.15.23:0/1418664865 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45f413d550 0x7f45f4159f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:52.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.103+0000 7f45f90ba700 1 -- 172.21.15.23:0/1418664865 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f45f4044710 con 0x7f45f413d550 2020-11-20T11:46:52.107 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.103+0000 7f45f90ba700 1 --2- 172.21.15.23:0/1418664865 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45f413d550 0x7f45f4159f90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f45e40093c0 tx=0x7f45e4021070).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:52.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.103+0000 7f45e17fa700 1 -- 172.21.15.23:0/1418664865 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e4029950 con 0x7f45f413d550 2020-11-20T11:46:52.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.103+0000 7f45e17fa700 1 -- 172.21.15.23:0/1418664865 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4275 (secure 0 0 0) 0x7f45e4011890 con 0x7f45f413d550 2020-11-20T11:46:52.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.103+0000 7f45e17fa700 1 -- 172.21.15.23:0/1418664865 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45f413d550 msgr2=0x7f45f4159f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:52.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.103+0000 7f45e17fa700 1 --2- 172.21.15.23:0/1418664865 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45f413d550 0x7f45f4159f90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f45e40093c0 tx=0x7f45e4021070).stop 2020-11-20T11:46:52.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.103+0000 7f45e17fa700 1 -- 172.21.15.23:0/1418664865 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45f413d550 msgr2=0x7f45f4159f90 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:46:52.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.103+0000 7f45e17fa700 1 --2- 172.21.15.23:0/1418664865 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f45f413d550 0x7f45f4159f90 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0x7f45e40093c0 tx=0x7f45e4021070).stop 2020-11-20T11:46:52.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.104+0000 7f45fa0bc700 1 -- 172.21.15.23:0/1418664865 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f45cc056d60 msgr2=0x7f45cc059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:52.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.104+0000 7f45fa0bc700 1 --2- 172.21.15.23:0/1418664865 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f45cc056d60 0x7f45cc059220 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f45f4065070 tx=0x7f45f0009040).stop 2020-11-20T11:46:52.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.104+0000 7f45fa0bc700 1 -- 172.21.15.23:0/1418664865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f45f40ad7e0 msgr2=0x7f45f4067770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:52.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.104+0000 7f45fa0bc700 1 --2- 172.21.15.23:0/1418664865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f45f40ad7e0 0x7f45f4067770 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f45ec00ef50 tx=0x7f45ec009220).stop 2020-11-20T11:46:52.110 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.104+0000 7f45f98bb700 1 -- 172.21.15.23:0/1418664865 reap_dead start 2020-11-20T11:46:52.110 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.104+0000 7f45f98bb700 1 -- 172.21.15.23:0/1418664865 reap_dead start 2020-11-20T11:46:52.110 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.104+0000 7f45fa0bc700 1 -- 172.21.15.23:0/1418664865 shutdown_connections 2020-11-20T11:46:52.110 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.104+0000 7f45fa0bc700 1 -- 172.21.15.23:0/1418664865 >> 172.21.15.23:0/1418664865 conn(0x7f45f4061850 msgr2=0x7f45f4061c30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:52.110 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.105+0000 7f45fa0bc700 1 -- 172.21.15.23:0/1418664865 shutdown_connections 2020-11-20T11:46:52.111 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.105+0000 7f45fa0bc700 1 -- 172.21.15.23:0/1418664865 wait complete. 2020-11-20T11:46:52.111 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:46:52.111 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:46:52.111 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:46:52.112 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:46:52.112 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 132, 2020-11-20T11:46:52.112 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:46:52.116 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:46:52.116 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:46:52.117 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:46:52.117 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:52.117 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 14, 2020-11-20T11:46:52.117 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:46:52.118 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:46:52.118 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:46:52.118 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:46:52.118 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:46:52.118 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:46:52.119 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:46:52.119 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:46:52.119 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:46:52.119 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:46:52.119 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:46:52.120 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:52.120 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:46:52.120 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:46:52.120 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:46:52.120 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:46:52.121 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:46:52.121 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:46:52.121 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:46:52.121 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:46:52.121 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:46:52.122 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:46:52.122 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:52.122 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:52.122 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:46:52.122 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:46:52.123 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:46:52.123 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:46:52.123 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:46:52.123 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:46:52.123 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:46:52.124 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:46:52.124 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:46:52.124 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:46:52.124 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:46:52.124 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:46:52.125 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:46:52.125 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:46:52.125 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:46:52.125 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:46:52.125 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:46:52.126 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:46:52.126 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:46:52.126 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:46:52.126 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:46:52.126 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:46:52.127 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:46:52.127 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:52.127 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:46:52.127 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:52.127 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:46:52.128 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:52.128 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:46:52.128 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:46:52.128 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:46:52.128 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:46:52.128 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:52.129 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:46:52.129 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:46:52.129 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:52.129 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:52.130 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:52.130 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:46:52.130 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:46:52.130 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:52.130 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:52.131 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:52.131 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:52.131 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:46:52.131 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:46:52.131 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:46:52.132 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:46:52.132 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:46:52.132 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:52.132 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:52.132 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:46:52.133 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:46:52.133 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:46:52.133 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:46:52.133 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:52.133 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:46:52.134 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:46:52.134 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:52.134 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:52.134 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:52.134 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:46:52.135 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:46:52.135 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:52.135 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:52.135 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:52.135 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:52.136 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:46:52.136 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:46:52.136 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:46:52.136 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:46:52.136 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:46:52.137 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:52.137 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:52.137 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:46:52.137 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:46:52.137 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:46:52.138 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:46:52.138 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:52.138 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:46:52.138 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:46:52.139 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:52.139 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:52.139 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:52.140 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:46:52.140 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:46:52.140 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:52.140 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:52.140 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:52.141 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:52.141 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:46:52.141 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:46:52.141 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:46:52.141 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:46:52.142 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:46:52.142 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:52.142 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:52.142 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:52.142 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:46:52.142 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:46:52.143 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:52.143 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:52.143 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:52.143 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:46:52.144 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:52.144 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:52.144 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:46:52.144 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:52.144 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:52.145 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:52.145 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:46:52.145 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:52.145 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:52.145 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:52.146 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:46:52.146 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:46:52.147 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:46:52.221 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:52.216+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 117 slow ops, oldest is osd_op(client.4659.0:53336 4.2s0 4:52da5114:::100000003c5.00000000:head [write 2568192~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:52.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.294+0000 7f9479f7f700 1 Processor -- start 2020-11-20T11:46:52.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.294+0000 7f9479f7f700 1 -- start start 2020-11-20T11:46:52.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.295+0000 7f9479f7f700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94740a2080 0x7f94740aef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:52.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.295+0000 7f9479f7f700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f94740a2460 0x7f9474137df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:52.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.295+0000 7f9479f7f700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94740aaab0 con 0x7f9474067a10 2020-11-20T11:46:52.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.295+0000 7f9479f7f700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f947413aa50 con 0x7f94740a2460 2020-11-20T11:46:52.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.295+0000 7f9479f7f700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94740aad50 con 0x7f94740a2080 2020-11-20T11:46:52.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.295+0000 7f9473fff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f94740a2460 0x7f9474137df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:52.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.295+0000 7f9478f7d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94740a2080 0x7f94740aef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:52.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.295+0000 7f9473fff700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f94740a2460 0x7f9474137df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:39274/0 (socket says 172.21.15.23:39274) 2020-11-20T11:46:52.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.295+0000 7f9473fff700 1 -- 172.21.15.23:0/2985711428 learned_addr learned my addr 172.21.15.23:0/2985711428 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:46:52.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.295+0000 7f9478f7d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94740a2080 0x7f94740aef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:56616/0 (socket says 172.21.15.23:56616) 2020-11-20T11:46:52.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.295+0000 7f947977e700 1 --1- 172.21.15.23:0/2985711428 >> v1:172.21.15.155:6789/0 conn(0x7f9474067a10 0x7f947413e020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:38602/0 (socket says 172.21.15.23:38602) 2020-11-20T11:46:52.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.295+0000 7f9473fff700 1 -- 172.21.15.23:0/2985711428 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94740a2080 msgr2=0x7f94740aef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:46:52.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.295+0000 7f9473fff700 1 --2- 172.21.15.23:0/2985711428 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94740a2080 0x7f94740aef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:52.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.295+0000 7f9473fff700 1 -- 172.21.15.23:0/2985711428 >> v1:172.21.15.155:6789/0 conn(0x7f9474067a10 legacy=0x7f947413e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:52.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.295+0000 7f9473fff700 1 -- 172.21.15.23:0/2985711428 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f947413b920 con 0x7f94740a2460 2020-11-20T11:46:52.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.295+0000 7f9473fff700 1 --2- 172.21.15.23:0/2985711428 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94740a2460 0x7f9474137df0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f9460009a10 tx=0x7f9460006ef0).ready entity=mon.1 client_cookie=f1e52a1bb0ce08b5 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:52.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.296+0000 7f94737fe700 1 -- 172.21.15.23:0/2985711428 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946000b860 con 0x7f94740a2460 2020-11-20T11:46:52.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.296+0000 7f94737fe700 1 -- 172.21.15.23:0/2985711428 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9460000e90 con 0x7f94740a2460 2020-11-20T11:46:52.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.296+0000 7f94737fe700 1 -- 172.21.15.23:0/2985711428 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946000bbd0 con 0x7f94740a2460 2020-11-20T11:46:52.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.296+0000 7f9479f7f700 1 -- 172.21.15.23:0/2985711428 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94740a2460 msgr2=0x7f9474137df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:52.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.296+0000 7f9479f7f700 1 --2- 172.21.15.23:0/2985711428 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94740a2460 0x7f9474137df0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f9460009a10 tx=0x7f9460006ef0).stop 2020-11-20T11:46:52.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.296+0000 7f9479f7f700 1 -- 172.21.15.23:0/2985711428 shutdown_connections 2020-11-20T11:46:52.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.296+0000 7f9479f7f700 1 --2- 172.21.15.23:0/2985711428 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94740a2080 0x7f94740aef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:52.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.296+0000 7f9479f7f700 1 --2- 172.21.15.23:0/2985711428 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94740a2460 0x7f9474137df0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:52.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.296+0000 7f9479f7f700 1 -- 172.21.15.23:0/2985711428 >> 172.21.15.23:0/2985711428 conn(0x7f9474063980 msgr2=0x7f9474063d90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:52.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.296+0000 7f9479f7f700 1 -- 172.21.15.23:0/2985711428 shutdown_connections 2020-11-20T11:46:52.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.296+0000 7f9479f7f700 1 -- 172.21.15.23:0/2985711428 wait complete. 2020-11-20T11:46:52.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.297+0000 7f9479f7f700 1 Processor -- start 2020-11-20T11:46:52.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.297+0000 7f9479f7f700 1 -- start start 2020-11-20T11:46:52.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.298+0000 7f9479f7f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9474067a10 0x7f9474061cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:52.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.298+0000 7f9479f7f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94740a2080 0x7f94740a80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:52.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.298+0000 7f9479f7f700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94740a2460 0x7f94740a85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:52.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.298+0000 7f9479f7f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94741386c0 con 0x7f94740a2080 2020-11-20T11:46:52.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.298+0000 7f9479f7f700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947414a2a0 con 0x7f9474067a10 2020-11-20T11:46:52.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.298+0000 7f9479f7f700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94740a4940 con 0x7f94740a2460 2020-11-20T11:46:52.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.298+0000 7f9473fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94740a2080 0x7f94740a80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:52.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.298+0000 7f947977e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94740a2460 0x7f94740a85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:52.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.298+0000 7f9478f7d700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9474067a10 0x7f9474061cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:52.313 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.298+0000 7f9473fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94740a2080 0x7f94740a80e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46384/0 (socket says 172.21.15.23:46384) 2020-11-20T11:46:52.313 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.298+0000 7f9473fff700 1 -- 172.21.15.23:0/3610407477 learned_addr learned my addr 172.21.15.23:0/3610407477 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:46:52.314 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.298+0000 7f9473fff700 1 -- 172.21.15.23:0/3610407477 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94740a2460 msgr2=0x7f94740a85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:52.314 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.298+0000 7f9473fff700 1 --2- 172.21.15.23:0/3610407477 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94740a2460 0x7f94740a85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:52.314 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.298+0000 7f9473fff700 1 -- 172.21.15.23:0/3610407477 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9474067a10 msgr2=0x7f9474061cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:52.315 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.298+0000 7f9473fff700 1 --2- 172.21.15.23:0/3610407477 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9474067a10 0x7f9474061cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:52.315 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.298+0000 7f9473fff700 1 -- 172.21.15.23:0/3610407477 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94740a4be0 con 0x7f94740a2080 2020-11-20T11:46:52.315 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.298+0000 7f947977e700 1 --2- 172.21.15.23:0/3610407477 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94740a2460 0x7f94740a85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:46:52.316 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.298+0000 7f9478f7d700 1 --2- 172.21.15.23:0/3610407477 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9474067a10 0x7f9474061cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:46:52.316 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.299+0000 7f9473fff700 1 --2- 172.21.15.23:0/3610407477 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94740a2080 0x7f94740a80e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f9474064520 tx=0x7f9460016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:52.317 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.299+0000 7f9470ff9700 1 -- 172.21.15.23:0/3610407477 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94600119f0 con 0x7f94740a2080 2020-11-20T11:46:52.317 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.299+0000 7f9470ff9700 1 -- 172.21.15.23:0/3610407477 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9460011b50 con 0x7f94740a2080 2020-11-20T11:46:52.317 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.299+0000 7f9479f7f700 1 -- 172.21.15.23:0/3610407477 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94740a7c40 con 0x7f94740a2080 2020-11-20T11:46:52.318 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.299+0000 7f9470ff9700 1 -- 172.21.15.23:0/3610407477 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946001d450 con 0x7f94740a2080 2020-11-20T11:46:52.320 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.299+0000 7f9479f7f700 1 -- 172.21.15.23:0/3610407477 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94740a2da0 con 0x7f94740a2080 2020-11-20T11:46:52.320 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.300+0000 7f9479f7f700 1 --2- 172.21.15.23:0/3610407477 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94740a43c0 0x7f9474132fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:52.321 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.300+0000 7f9479f7f700 1 -- 172.21.15.23:0/3610407477 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9474133d30 con 0x7f94740a43c0 2020-11-20T11:46:52.321 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.300+0000 7f9478f7d700 1 --2- 172.21.15.23:0/3610407477 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94740a43c0 0x7f9474132fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:52.321 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.300+0000 7f9470ff9700 1 -- 172.21.15.23:0/3610407477 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9460023020 con 0x7f94740a2080 2020-11-20T11:46:52.322 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.300+0000 7f9478f7d700 1 -- 172.21.15.23:0/3610407477 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f94740a47a0 con 0x7f94740a43c0 2020-11-20T11:46:52.322 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.300+0000 7f9478f7d700 1 --2- 172.21.15.23:0/3610407477 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94740a43c0 0x7f9474132fd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f946800f470 tx=0x7f9468006dd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:52.322 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.300+0000 7f9470ff9700 1 --2- 172.21.15.23:0/3610407477 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f944c056d60 0x7f944c059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:52.323 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.300+0000 7f9470ff9700 1 -- 172.21.15.23:0/3610407477 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f9460071c50 con 0x7f94740a2080 2020-11-20T11:46:52.323 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.301+0000 7f947977e700 1 --2- 172.21.15.23:0/3610407477 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f944c056d60 0x7f944c059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:52.324 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.301+0000 7f9470ff9700 1 -- 172.21.15.23:0/3610407477 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9468014440 con 0x7f94740a43c0 2020-11-20T11:46:52.324 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.301+0000 7f9470ff9700 1 -- 172.21.15.23:0/3610407477 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f94680145a0 con 0x7f94740a43c0 2020-11-20T11:46:52.324 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.301+0000 7f9470ff9700 1 -- 172.21.15.23:0/3610407477 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94740a43c0 msgr2=0x7f9474132fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:52.325 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.301+0000 7f9470ff9700 1 --2- 172.21.15.23:0/3610407477 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94740a43c0 0x7f9474132fd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f946800f470 tx=0x7f9468006dd0).stop 2020-11-20T11:46:52.325 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.301+0000 7f9470ff9700 1 -- 172.21.15.23:0/3610407477 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94740a43c0 msgr2=0x7f9474132fd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:46:52.326 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.301+0000 7f9470ff9700 1 --2- 172.21.15.23:0/3610407477 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94740a43c0 0x7f9474132fd0 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0x7f946800f470 tx=0x7f9468006dd0).stop 2020-11-20T11:46:52.326 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.301+0000 7f947977e700 1 --2- 172.21.15.23:0/3610407477 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f944c056d60 0x7f944c059220 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f94740a8ac0 tx=0x7f946c007ec0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:52.329 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.307+0000 7f9479f7f700 1 --2- 172.21.15.23:0/3610407477 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94741334c0 0x7f94740a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:52.329 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.307+0000 7f9479f7f700 1 -- 172.21.15.23:0/3610407477 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94741338d0 con 0x7f94741334c0 2020-11-20T11:46:52.329 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.308+0000 7f9478f7d700 1 --2- 172.21.15.23:0/3610407477 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94741334c0 0x7f94740a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:52.329 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.308+0000 7f9478f7d700 1 -- 172.21.15.23:0/3610407477 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f94740a2ff0 con 0x7f94741334c0 2020-11-20T11:46:52.330 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.308+0000 7f9478f7d700 1 --2- 172.21.15.23:0/3610407477 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94741334c0 0x7f94740a6300 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f9468018000 tx=0x7f946801b000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:52.330 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.308+0000 7f9470ff9700 1 -- 172.21.15.23:0/3610407477 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946801de10 con 0x7f94741334c0 2020-11-20T11:46:52.330 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.308+0000 7f9470ff9700 1 -- 172.21.15.23:0/3610407477 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4785 (secure 0 0 0) 0x7f9468024650 con 0x7f94741334c0 2020-11-20T11:46:52.330 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.308+0000 7f9470ff9700 1 -- 172.21.15.23:0/3610407477 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94741334c0 msgr2=0x7f94740a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:52.331 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.308+0000 7f9470ff9700 1 --2- 172.21.15.23:0/3610407477 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94741334c0 0x7f94740a6300 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f9468018000 tx=0x7f946801b000).stop 2020-11-20T11:46:52.331 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.308+0000 7f9470ff9700 1 -- 172.21.15.23:0/3610407477 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94741334c0 msgr2=0x7f94740a6300 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:46:52.331 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.308+0000 7f9470ff9700 1 --2- 172.21.15.23:0/3610407477 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94741334c0 0x7f94740a6300 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:52.332 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.309+0000 7f9479f7f700 1 -- 172.21.15.23:0/3610407477 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f944c056d60 msgr2=0x7f944c059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:52.332 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.309+0000 7f9479f7f700 1 --2- 172.21.15.23:0/3610407477 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f944c056d60 0x7f944c059220 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f94740a8ac0 tx=0x7f946c007ec0).stop 2020-11-20T11:46:52.332 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.309+0000 7f9479f7f700 1 -- 172.21.15.23:0/3610407477 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94740a2080 msgr2=0x7f94740a80e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:52.333 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.309+0000 7f9479f7f700 1 --2- 172.21.15.23:0/3610407477 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f94740a2080 0x7f94740a80e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f9474064520 tx=0x7f9460016000).stop 2020-11-20T11:46:52.333 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.309+0000 7f947977e700 1 -- 172.21.15.23:0/3610407477 reap_dead start 2020-11-20T11:46:52.334 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.309+0000 7f947977e700 1 -- 172.21.15.23:0/3610407477 reap_dead start 2020-11-20T11:46:52.334 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.309+0000 7f9479f7f700 1 -- 172.21.15.23:0/3610407477 shutdown_connections 2020-11-20T11:46:52.334 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.309+0000 7f9479f7f700 1 -- 172.21.15.23:0/3610407477 >> 172.21.15.23:0/3610407477 conn(0x7f9474063980 msgr2=0x7f9474068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:52.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.309+0000 7f9479f7f700 1 -- 172.21.15.23:0/3610407477 shutdown_connections 2020-11-20T11:46:52.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.309+0000 7f9479f7f700 1 -- 172.21.15.23:0/3610407477 wait complete. 2020-11-20T11:46:52.336 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:46:52.336 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:46:52.336 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:46:52.337 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:46:52.337 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 132, 2020-11-20T11:46:52.337 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:46:52.337 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:46:52.338 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:46:52.338 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:46:52.338 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:52.339 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 15, 2020-11-20T11:46:52.339 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:46:52.339 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:46:52.340 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:46:52.340 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:46:52.340 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:46:52.341 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:46:52.341 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:46:52.342 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:46:52.345 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:46:52.346 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:46:52.346 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:46:52.347 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:52.347 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:46:52.347 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:46:52.348 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:46:52.348 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:46:52.349 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:46:52.349 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:46:52.349 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:46:52.350 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:46:52.350 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:46:52.350 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:46:52.351 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:52.351 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:52.352 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:46:52.352 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:46:52.352 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:46:52.353 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:46:52.353 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:46:52.354 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:46:52.354 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:46:52.354 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:46:52.354 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:46:52.355 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:46:52.355 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:46:52.355 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:46:52.355 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:46:52.355 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:46:52.356 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:46:52.356 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:46:52.356 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:46:52.357 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:46:52.357 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:46:52.357 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:46:52.358 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:46:52.358 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:46:52.358 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:46:52.359 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:52.359 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:46:52.359 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:52.360 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:46:52.360 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:52.360 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:46:52.361 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:46:52.361 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:46:52.361 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:46:52.361 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:52.361 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:46:52.362 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:46:52.362 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:52.362 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:52.362 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:52.362 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:46:52.363 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:46:52.363 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:52.363 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:52.363 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:52.364 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:52.364 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:46:52.364 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:46:52.364 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:46:52.364 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:46:52.365 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:46:52.365 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:52.365 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:52.365 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:46:52.366 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:46:52.366 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:46:52.366 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:46:52.366 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:52.367 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:46:52.367 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:46:52.367 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:52.367 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:52.368 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:52.368 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:46:52.368 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:46:52.369 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:52.369 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:52.369 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:52.370 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:52.370 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:46:52.370 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:46:52.371 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:46:52.371 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:46:52.372 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:46:52.372 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:52.372 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:52.372 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:46:52.372 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:46:52.373 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:46:52.373 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:46:52.373 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:52.373 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:46:52.373 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:46:52.374 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:52.374 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:52.374 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:52.374 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:46:52.374 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:46:52.375 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:46:52.375 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:52.375 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:52.375 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:52.376 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:46:52.376 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:46:52.377 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:46:52.377 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:46:52.377 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:46:52.378 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:52.378 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:52.379 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:52.379 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:46:52.379 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:46:52.380 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:52.380 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:52.380 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:52.381 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:46:52.381 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:52.382 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:52.382 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:46:52.382 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:52.382 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:52.382 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:52.383 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:46:52.383 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:52.383 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:52.383 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:46:52.384 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:52.384 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:52.384 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:52.384 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:46:52.385 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:52.385 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:52.386 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:46:52.386 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:52.386 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:52.386 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:52.387 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:46:52.387 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:52.387 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:46:52.387 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:46:52.387 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:46:52.388 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:46:52.388 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:46:52.388 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:46:52.388 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:46:52.388 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:46:52.389 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:46:52.389 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:46:52.389 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:46:52.390 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2020-11-20T11:46:52.390 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2020-11-20T11:46:52.390 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2020-11-20T11:46:52.390 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2020-11-20T11:46:52.391 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2020-11-20T11:46:52.391 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T11:46:52.400 INFO:tasks.ceph.mon.b:Stopped 2020-11-20T11:46:52.400 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2020-11-20T11:46:52.400 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:46:52.536 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.533+0000 7fa6c359e700 1 Processor -- start 2020-11-20T11:46:52.536 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.533+0000 7fa6c359e700 1 -- start start 2020-11-20T11:46:52.537 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.534+0000 7fa6c359e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa6c40a2460 0x7fa6c4137df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:52.537 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.534+0000 7fa6c359e700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa6c4067a10 0x7fa6c4067ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:52.537 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.534+0000 7fa6c359e700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c40aaab0 con 0x7fa6c40a2080 2020-11-20T11:46:52.538 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.534+0000 7fa6c359e700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c413aa50 con 0x7fa6c4067a10 2020-11-20T11:46:52.538 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.534+0000 7fa6c359e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c40aad50 con 0x7fa6c40a2460 2020-11-20T11:46:52.539 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.535+0000 7fa6c1d9b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa6c40a2460 0x7fa6c4137df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:52.539 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.535+0000 7fa6c1d9b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa6c40a2460 0x7fa6c4137df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:56634/0 (socket says 172.21.15.23:56634) 2020-11-20T11:46:52.540 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.535+0000 7fa6c1d9b700 1 -- 172.21.15.23:0/751288241 learned_addr learned my addr 172.21.15.23:0/751288241 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:46:52.541 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.535+0000 7fa6c1d9b700 1 -- 172.21.15.23:0/751288241 >> v2:172.21.15.23:3300/0 conn(0x7fa6c4067a10 msgr2=0x7fa6c4067ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:46:52.542 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.535+0000 7fa6c1d9b700 1 --2- 172.21.15.23:0/751288241 >> v2:172.21.15.23:3300/0 conn(0x7fa6c4067a10 0x7fa6c4067ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:52.542 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.535+0000 7fa6c1d9b700 1 -- 172.21.15.23:0/751288241 >> v1:172.21.15.155:6789/0 conn(0x7fa6c40a2080 legacy=0x7fa6c40aef10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:46:52.542 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.535+0000 7fa6c1d9b700 1 -- 172.21.15.23:0/751288241 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c413b920 con 0x7fa6c40a2460 2020-11-20T11:46:52.543 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.535+0000 7fa6c1d9b700 1 --2- 172.21.15.23:0/751288241 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa6c40a2460 0x7fa6c4137df0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fa6b4009a10 tx=0x7fa6b4006ef0).ready entity=mon.2 client_cookie=68026ffb5003827c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:52.543 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.536+0000 7fa6c159a700 1 -- 172.21.15.23:0/751288241 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b400b820 con 0x7fa6c40a2460 2020-11-20T11:46:52.543 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.536+0000 7fa6c159a700 1 -- 172.21.15.23:0/751288241 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6b4000e90 con 0x7fa6c40a2460 2020-11-20T11:46:52.544 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.536+0000 7fa6c159a700 1 -- 172.21.15.23:0/751288241 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b400bb90 con 0x7fa6c40a2460 2020-11-20T11:46:52.544 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.536+0000 7fa6c359e700 1 -- 172.21.15.23:0/751288241 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa6c40a2460 msgr2=0x7fa6c4137df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:52.545 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.536+0000 7fa6c359e700 1 --2- 172.21.15.23:0/751288241 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa6c40a2460 0x7fa6c4137df0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fa6b4009a10 tx=0x7fa6b4006ef0).stop 2020-11-20T11:46:52.546 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.538+0000 7fa6c359e700 1 -- 172.21.15.23:0/751288241 shutdown_connections 2020-11-20T11:46:52.547 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.538+0000 7fa6c359e700 1 --2- 172.21.15.23:0/751288241 >> v2:172.21.15.23:3300/0 conn(0x7fa6c4067a10 0x7fa6c4067ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:52.547 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.538+0000 7fa6c359e700 1 --2- 172.21.15.23:0/751288241 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa6c40a2460 0x7fa6c4137df0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:52.548 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.538+0000 7fa6c359e700 1 -- 172.21.15.23:0/751288241 >> 172.21.15.23:0/751288241 conn(0x7fa6c4063980 msgr2=0x7fa6c4063d90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:52.548 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.538+0000 7fa6c359e700 1 -- 172.21.15.23:0/751288241 shutdown_connections 2020-11-20T11:46:52.549 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.538+0000 7fa6c359e700 1 -- 172.21.15.23:0/751288241 wait complete. 2020-11-20T11:46:52.549 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.540+0000 7fa6c359e700 1 Processor -- start 2020-11-20T11:46:52.550 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.543+0000 7fa6c359e700 1 -- start start 2020-11-20T11:46:52.551 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.544+0000 7fa6c359e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6c4067a10 0x7fa6c40a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:52.551 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.544+0000 7fa6c359e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6c40a2080 0x7fa6c40ae6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:52.552 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.544+0000 7fa6c359e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa6c40a2460 0x7fa6c413c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:52.552 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.544+0000 7fa6c359e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c413bfd0 con 0x7fa6c4067a10 2020-11-20T11:46:52.552 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.544+0000 7fa6c359e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c413c270 con 0x7fa6c40a2080 2020-11-20T11:46:52.553 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.544+0000 7fa6c359e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c41383a0 con 0x7fa6c40a2460 2020-11-20T11:46:52.553 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.545+0000 7fa6c1d9b700 1 -- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6c40a2080 msgr2=0x7fa6c40ae6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:46:52.554 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.545+0000 7fa6c1d9b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6c40a2080 0x7fa6c40ae6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:46:52.556 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.548+0000 7fa6c259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6c4067a10 0x7fa6c40a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:52.556 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.548+0000 7fa6c259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6c4067a10 0x7fa6c40a4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46396/0 (socket says 172.21.15.23:46396) 2020-11-20T11:46:52.557 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.548+0000 7fa6c259c700 1 -- 172.21.15.23:0/1913266485 learned_addr learned my addr 172.21.15.23:0/1913266485 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:46:52.557 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.549+0000 7fa6c259c700 1 -- 172.21.15.23:0/1913266485 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa6c40a2460 msgr2=0x7fa6c413c710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:46:52.558 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.549+0000 7fa6c259c700 1 --2- 172.21.15.23:0/1913266485 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa6c40a2460 0x7fa6c413c710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:52.558 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.549+0000 7fa6c259c700 1 -- 172.21.15.23:0/1913266485 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6c40a2080 msgr2=0x7fa6c40ae6f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:46:52.559 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.549+0000 7fa6c259c700 1 --2- 172.21.15.23:0/1913266485 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6c40a2080 0x7fa6c40ae6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:52.559 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.549+0000 7fa6c259c700 1 -- 172.21.15.23:0/1913266485 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c413e4f0 con 0x7fa6c4067a10 2020-11-20T11:46:52.559 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.549+0000 7fa6c259c700 1 --2- 172.21.15.23:0/1913266485 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6c4067a10 0x7fa6c40a4d60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fa6b8009760 tx=0x7fa6b8006e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:52.560 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.551+0000 7fa6b27fc700 1 -- 172.21.15.23:0/1913266485 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b800d5b0 con 0x7fa6c4067a10 2020-11-20T11:46:52.560 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.552+0000 7fa6b27fc700 1 -- 172.21.15.23:0/1913266485 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6b800d710 con 0x7fa6c4067a10 2020-11-20T11:46:52.560 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.552+0000 7fa6c359e700 1 -- 172.21.15.23:0/1913266485 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c40a41a0 con 0x7fa6c4067a10 2020-11-20T11:46:52.561 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.552+0000 7fa6b27fc700 1 -- 172.21.15.23:0/1913266485 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b800da90 con 0x7fa6c4067a10 2020-11-20T11:46:52.561 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.552+0000 7fa6c359e700 1 -- 172.21.15.23:0/1913266485 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c40a2da0 con 0x7fa6c4067a10 2020-11-20T11:46:52.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.553+0000 7fa6c359e700 1 -- 172.21.15.23:0/1913266485 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c4057dc0 con 0x7fa6c4067a10 2020-11-20T11:46:52.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.555+0000 7fa6b27fc700 1 -- 172.21.15.23:0/1913266485 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa6b8017030 con 0x7fa6c4067a10 2020-11-20T11:46:52.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.556+0000 7fa6b27fc700 1 --2- 172.21.15.23:0/1913266485 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa698056d60 0x7fa698059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:52.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.556+0000 7fa6b27fc700 1 -- 172.21.15.23:0/1913266485 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fa6b806f850 con 0x7fa6c4067a10 2020-11-20T11:46:52.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.556+0000 7fa6c1d9b700 1 --2- 172.21.15.23:0/1913266485 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa698056d60 0x7fa698059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:52.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.556+0000 7fa6c1d9b700 1 --2- 172.21.15.23:0/1913266485 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa698056d60 0x7fa698059220 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fa6c40a5da0 tx=0x7fa6b4016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:52.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.558+0000 7fa6b27fc700 1 -- 172.21.15.23:0/1913266485 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa6b8044bc0 con 0x7fa6c4067a10 2020-11-20T11:46:52.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.824+0000 7fa6c359e700 1 -- 172.21.15.23:0/1913266485 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa6c40a5dd0 con 0x7fa6c4067a10 2020-11-20T11:46:52.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.825+0000 7fa6b27fc700 1 -- 172.21.15.23:0/1913266485 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1479 (secure 0 0 0) 0x7fa6b8015030 con 0x7fa6c4067a10 2020-11-20T11:46:52.828 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":132,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:46:52.829 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.826+0000 7fa6c359e700 1 -- 172.21.15.23:0/1913266485 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa698056d60 msgr2=0x7fa698059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:52.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.826+0000 7fa6c359e700 1 --2- 172.21.15.23:0/1913266485 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa698056d60 0x7fa698059220 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fa6c40a5da0 tx=0x7fa6b4016000).stop 2020-11-20T11:46:52.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.826+0000 7fa6c359e700 1 -- 172.21.15.23:0/1913266485 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6c4067a10 msgr2=0x7fa6c40a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:52.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.826+0000 7fa6c359e700 1 --2- 172.21.15.23:0/1913266485 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6c4067a10 0x7fa6c40a4d60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fa6b8009760 tx=0x7fa6b8006e20).stop 2020-11-20T11:46:52.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.826+0000 7fa6c359e700 1 -- 172.21.15.23:0/1913266485 shutdown_connections 2020-11-20T11:46:52.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.826+0000 7fa6c359e700 1 --2- 172.21.15.23:0/1913266485 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa698056d60 0x7fa698059220 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:52.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.826+0000 7fa6c359e700 1 --2- 172.21.15.23:0/1913266485 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa6c40a2460 0x7fa6c413c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:52.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.826+0000 7fa6c359e700 1 --2- 172.21.15.23:0/1913266485 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6c4067a10 0x7fa6c40a4d60 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:52.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.826+0000 7fa6c359e700 1 --2- 172.21.15.23:0/1913266485 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6c40a2080 0x7fa6c40ae6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:52.832 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.826+0000 7fa6c359e700 1 -- 172.21.15.23:0/1913266485 >> 172.21.15.23:0/1913266485 conn(0x7fa6c4063980 msgr2=0x7fa6c4068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:52.832 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.826+0000 7fa6c359e700 1 -- 172.21.15.23:0/1913266485 shutdown_connections 2020-11-20T11:46:52.832 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:52.826+0000 7fa6c359e700 1 -- 172.21.15.23:0/1913266485 wait complete. 2020-11-20T11:46:53.262 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:53.202+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 115 slow ops, oldest is osd_op(client.4659.0:53403 4.2s0 4:507539af:::10000000350.00000016:head [write 503808~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:54.318 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:54.297+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 116 slow ops, oldest is osd_op(client.4659.0:53687 4.2s0 4:4ae2880c:::10000000364.0000000b:head [write 1916928~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:55.370 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:55.321+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 118 slow ops, oldest is osd_op(client.4659.0:53687 4.2s0 4:4ae2880c:::10000000364.0000000b:head [write 1916928~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:55.842 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:46:55.993 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.989+0000 7fa61289a700 1 Processor -- start 2020-11-20T11:46:55.994 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.989+0000 7fa61289a700 1 -- start start 2020-11-20T11:46:55.994 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.990+0000 7fa61289a700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa60c0aaab0 0x7fa60c13ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:55.994 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.990+0000 7fa61289a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa60c0aae90 0x7fa60c1381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:55.995 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.990+0000 7fa61289a700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa60c13e600 con 0x7fa60c13aeb0 2020-11-20T11:46:55.995 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.990+0000 7fa61289a700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa60c13e8a0 con 0x7fa60c0aae90 2020-11-20T11:46:55.995 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.990+0000 7fa61289a700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa60c0ab6d0 con 0x7fa60c0aaab0 2020-11-20T11:46:55.998 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.990+0000 7fa610e37700 1 -- >> v1:172.21.15.155:6789/0 conn(0x7fa60c13aeb0 legacy=0x7fa60c0a8c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:46:55.999 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.990+0000 7fa60b7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa60c0aae90 0x7fa60c1381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:55.999 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.990+0000 7fa60b7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa60c0aae90 0x7fa60c1381a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46412/0 (socket says 172.21.15.23:46412) 2020-11-20T11:46:55.999 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.990+0000 7fa60b7fe700 1 -- 172.21.15.23:0/2976278876 learned_addr learned my addr 172.21.15.23:0/2976278876 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:46:55.999 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.990+0000 7fa60bfff700 1 --2- 172.21.15.23:0/2976278876 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa60c0aaab0 0x7fa60c13ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:56.000 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.991+0000 7fa60b7fe700 1 -- 172.21.15.23:0/2976278876 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa60c0aaab0 msgr2=0x7fa60c13ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:46:56.000 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.991+0000 7fa60b7fe700 1 --2- 172.21.15.23:0/2976278876 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa60c0aaab0 0x7fa60c13ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:56.000 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.991+0000 7fa60b7fe700 1 -- 172.21.15.23:0/2976278876 >> v1:172.21.15.155:6789/0 conn(0x7fa60c13aeb0 legacy=0x7fa60c0a8c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:46:56.000 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.991+0000 7fa60b7fe700 1 -- 172.21.15.23:0/2976278876 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa60c0a9370 con 0x7fa60c0aae90 2020-11-20T11:46:56.000 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.991+0000 7fa60b7fe700 1 --2- 172.21.15.23:0/2976278876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa60c0aae90 0x7fa60c1381a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fa5fc009a10 tx=0x7fa5fc006ef0).ready entity=mon.0 client_cookie=6b3ac7d5134178ce server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:56.001 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.991+0000 7fa60affd700 1 -- 172.21.15.23:0/2976278876 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc00b820 con 0x7fa60c0aae90 2020-11-20T11:46:56.001 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.991+0000 7fa60affd700 1 -- 172.21.15.23:0/2976278876 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5fc000e90 con 0x7fa60c0aae90 2020-11-20T11:46:56.001 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.991+0000 7fa60affd700 1 -- 172.21.15.23:0/2976278876 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc00bb90 con 0x7fa60c0aae90 2020-11-20T11:46:56.001 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.992+0000 7fa61289a700 1 -- 172.21.15.23:0/2976278876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa60c0aae90 msgr2=0x7fa60c1381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:56.001 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.992+0000 7fa61289a700 1 --2- 172.21.15.23:0/2976278876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa60c0aae90 0x7fa60c1381a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fa5fc009a10 tx=0x7fa5fc006ef0).stop 2020-11-20T11:46:56.002 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.992+0000 7fa61289a700 1 -- 172.21.15.23:0/2976278876 shutdown_connections 2020-11-20T11:46:56.002 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.992+0000 7fa61289a700 1 --2- 172.21.15.23:0/2976278876 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa60c0aaab0 0x7fa60c13ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:56.002 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.992+0000 7fa61289a700 1 --2- 172.21.15.23:0/2976278876 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa60c0aae90 0x7fa60c1381a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:56.002 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.992+0000 7fa61289a700 1 -- 172.21.15.23:0/2976278876 >> 172.21.15.23:0/2976278876 conn(0x7fa60c0678e0 msgr2=0x7fa60c067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:56.003 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.992+0000 7fa61289a700 1 -- 172.21.15.23:0/2976278876 shutdown_connections 2020-11-20T11:46:56.003 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.992+0000 7fa61289a700 1 -- 172.21.15.23:0/2976278876 wait complete. 2020-11-20T11:46:56.003 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.993+0000 7fa61289a700 1 Processor -- start 2020-11-20T11:46:56.003 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.994+0000 7fa61289a700 1 -- start start 2020-11-20T11:46:56.003 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.994+0000 7fa61289a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa60c0aaab0 0x7fa60c0ad620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:56.004 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.994+0000 7fa61289a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa60c0aae90 0x7fa60c13d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:56.004 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.994+0000 7fa61289a700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa60c14da30 0x7fa60c14de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:56.004 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.994+0000 7fa61289a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60c0b0b70 con 0x7fa60c0aaab0 2020-11-20T11:46:56.004 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.994+0000 7fa61289a700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60c0b0e10 con 0x7fa60c0aae90 2020-11-20T11:46:56.005 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.994+0000 7fa61289a700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa60c0b10b0 con 0x7fa60c14da30 2020-11-20T11:46:56.005 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.995+0000 7fa60bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa60c0aaab0 0x7fa60c0ad620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:56.005 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.995+0000 7fa60b7fe700 1 -- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa60c0aae90 msgr2=0x7fa60c13d730 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:46:56.005 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.995+0000 7fa60bfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa60c0aaab0 0x7fa60c0ad620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46418/0 (socket says 172.21.15.23:46418) 2020-11-20T11:46:56.005 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.995+0000 7fa60bfff700 1 -- 172.21.15.23:0/1166529434 learned_addr learned my addr 172.21.15.23:0/1166529434 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:46:56.006 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.995+0000 7fa60b7fe700 1 --2- 172.21.15.23:0/1166529434 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa60c0aae90 0x7fa60c13d730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:46:56.006 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.995+0000 7fa610e37700 1 --2- 172.21.15.23:0/1166529434 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa60c14da30 0x7fa60c14de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:56.006 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.995+0000 7fa60bfff700 1 -- 172.21.15.23:0/1166529434 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa60c14da30 msgr2=0x7fa60c14de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:56.006 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.995+0000 7fa60bfff700 1 --2- 172.21.15.23:0/1166529434 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa60c14da30 0x7fa60c14de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:56.006 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.995+0000 7fa60bfff700 1 -- 172.21.15.23:0/1166529434 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa60c0aae90 msgr2=0x7fa60c13d730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:46:56.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.995+0000 7fa60bfff700 1 --2- 172.21.15.23:0/1166529434 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa60c0aae90 0x7fa60c13d730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:56.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.995+0000 7fa60bfff700 1 -- 172.21.15.23:0/1166529434 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa60c14e380 con 0x7fa60c0aaab0 2020-11-20T11:46:56.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.995+0000 7fa60bfff700 1 --2- 172.21.15.23:0/1166529434 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa60c0aaab0 0x7fa60c0ad620 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fa60000eba0 tx=0x7fa600006e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:56.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.995+0000 7fa5ebfff700 1 -- 172.21.15.23:0/1166529434 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60000c920 con 0x7fa60c0aaab0 2020-11-20T11:46:56.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.995+0000 7fa5ebfff700 1 -- 172.21.15.23:0/1166529434 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa60000ca80 con 0x7fa60c0aaab0 2020-11-20T11:46:56.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.995+0000 7fa5ebfff700 1 -- 172.21.15.23:0/1166529434 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60000ce00 con 0x7fa60c0aaab0 2020-11-20T11:46:56.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.995+0000 7fa61289a700 1 -- 172.21.15.23:0/1166529434 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa60c0b07f0 con 0x7fa60c0aaab0 2020-11-20T11:46:56.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.995+0000 7fa61289a700 1 -- 172.21.15.23:0/1166529434 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa60c131590 con 0x7fa60c0aaab0 2020-11-20T11:46:56.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.996+0000 7fa61289a700 1 -- 172.21.15.23:0/1166529434 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa60c0b09e0 con 0x7fa60c0aaab0 2020-11-20T11:46:56.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.997+0000 7fa5ebfff700 1 -- 172.21.15.23:0/1166529434 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa600012600 con 0x7fa60c0aaab0 2020-11-20T11:46:56.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.997+0000 7fa5ebfff700 1 --2- 172.21.15.23:0/1166529434 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa5e4056d40 0x7fa5e4059200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:56.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.997+0000 7fa5ebfff700 1 -- 172.21.15.23:0/1166529434 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fa6000703f0 con 0x7fa60c0aaab0 2020-11-20T11:46:56.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.997+0000 7fa60b7fe700 1 --2- 172.21.15.23:0/1166529434 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa5e4056d40 0x7fa5e4059200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:56.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:55.998+0000 7fa60b7fe700 1 --2- 172.21.15.23:0/1166529434 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa5e4056d40 0x7fa5e4059200 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fa60c0aa7e0 tx=0x7fa5fc016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:56.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:56.001+0000 7fa5ebfff700 1 -- 172.21.15.23:0/1166529434 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa6000456e0 con 0x7fa60c0aaab0 2020-11-20T11:46:56.343 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:56.340+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 120 slow ops, oldest is osd_op(client.4659.0:54271 4.2s0 4:49e75ffb:::100000003a1.00000004:head [write 3026944~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:56.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:56.656+0000 7fa61289a700 1 -- 172.21.15.23:0/1166529434 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa60c05c7d0 con 0x7fa60c0aaab0 2020-11-20T11:46:56.659 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:56.656+0000 7fa5ebfff700 1 -- 172.21.15.23:0/1166529434 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1479 (secure 0 0 0) 0x7fa600048d00 con 0x7fa60c0aaab0 2020-11-20T11:46:56.660 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":132,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:46:56.661 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:56.658+0000 7fa61289a700 1 -- 172.21.15.23:0/1166529434 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa5e4056d40 msgr2=0x7fa5e4059200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:56.661 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:56.658+0000 7fa61289a700 1 --2- 172.21.15.23:0/1166529434 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa5e4056d40 0x7fa5e4059200 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fa60c0aa7e0 tx=0x7fa5fc016000).stop 2020-11-20T11:46:56.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:56.658+0000 7fa61289a700 1 -- 172.21.15.23:0/1166529434 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa60c0aaab0 msgr2=0x7fa60c0ad620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:56.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:56.658+0000 7fa61289a700 1 --2- 172.21.15.23:0/1166529434 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa60c0aaab0 0x7fa60c0ad620 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fa60000eba0 tx=0x7fa600006e30).stop 2020-11-20T11:46:56.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:56.667+0000 7fa61289a700 1 -- 172.21.15.23:0/1166529434 shutdown_connections 2020-11-20T11:46:56.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:56.667+0000 7fa61289a700 1 --2- 172.21.15.23:0/1166529434 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa5e4056d40 0x7fa5e4059200 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:56.667+0000 7fa61289a700 1 --2- 172.21.15.23:0/1166529434 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa60c14da30 0x7fa60c14de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:56.667+0000 7fa61289a700 1 --2- 172.21.15.23:0/1166529434 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa60c0aaab0 0x7fa60c0ad620 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:56.667+0000 7fa61289a700 1 --2- 172.21.15.23:0/1166529434 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa60c0aae90 0x7fa60c13d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:56.667+0000 7fa61289a700 1 -- 172.21.15.23:0/1166529434 >> 172.21.15.23:0/1166529434 conn(0x7fa60c0678e0 msgr2=0x7fa60c13a750 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:56.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:56.667+0000 7fa61289a700 1 -- 172.21.15.23:0/1166529434 shutdown_connections 2020-11-20T11:46:56.672 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:56.667+0000 7fa61289a700 1 -- 172.21.15.23:0/1166529434 wait complete. 2020-11-20T11:46:57.400 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:57.366+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 121 slow ops, oldest is osd_op(client.4659.0:54271 4.2s0 4:49e75ffb:::100000003a1.00000004:head [write 3026944~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:58.433 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:58.379+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 123 slow ops, oldest is osd_op(client.4659.0:54271 4.2s0 4:49e75ffb:::100000003a1.00000004:head [write 3026944~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:59.412 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:46:59.365+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 126 slow ops, oldest is osd_op(client.4659.0:54271 4.2s0 4:49e75ffb:::100000003a1.00000004:head [write 3026944~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:46:59.692 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:46:59.899 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.894+0000 7f8c963ec700 1 Processor -- start 2020-11-20T11:46:59.902 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.894+0000 7f8c963ec700 1 -- start start 2020-11-20T11:46:59.902 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.895+0000 7f8c963ec700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f8c900ad080 0x7f8c900a5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:59.902 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.895+0000 7f8c963ec700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8c900a9740 0x7f8c900a9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:59.903 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.895+0000 7f8c963ec700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c900a5a30 con 0x7f8c900acca0 2020-11-20T11:46:59.903 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.895+0000 7f8c963ec700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c900a7880 con 0x7f8c900ad080 2020-11-20T11:46:59.903 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.895+0000 7f8c963ec700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c900a7b20 con 0x7f8c900a9740 2020-11-20T11:46:59.904 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.895+0000 7f8c94be9700 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f8c900ad080 msgr2=0x7f8c900a5240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:46:59.904 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.895+0000 7f8c94be9700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f8c900ad080 0x7f8c900a5240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:46:59.904 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.895+0000 7f8c953ea700 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f8c900acca0 legacy=0x7f8c900a7fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:46:59.906 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.902+0000 7f8c95beb700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8c900a9740 0x7f8c900a9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:59.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.902+0000 7f8c95beb700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8c900a9740 0x7f8c900a9bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:56672/0 (socket says 172.21.15.23:56672) 2020-11-20T11:46:59.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.902+0000 7f8c95beb700 1 -- 172.21.15.23:0/2434040297 learned_addr learned my addr 172.21.15.23:0/2434040297 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:46:59.907 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.902+0000 7f8c95beb700 1 -- 172.21.15.23:0/2434040297 >> v2:172.21.15.155:3300/0 conn(0x7f8c900ad080 msgr2=0x7f8c900a5240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:46:59.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.902+0000 7f8c95beb700 1 --2- 172.21.15.23:0/2434040297 >> v2:172.21.15.155:3300/0 conn(0x7f8c900ad080 0x7f8c900a5240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:59.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.902+0000 7f8c95beb700 1 -- 172.21.15.23:0/2434040297 >> v1:172.21.15.155:6789/0 conn(0x7f8c900acca0 legacy=0x7f8c900a7fc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:46:59.908 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.902+0000 7f8c95beb700 1 -- 172.21.15.23:0/2434040297 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c900a5730 con 0x7f8c900a9740 2020-11-20T11:46:59.909 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.902+0000 7f8c95beb700 1 --2- 172.21.15.23:0/2434040297 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8c900a9740 0x7f8c900a9bf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f8c80008eb0 tx=0x7f8c800083f0).ready entity=mon.2 client_cookie=5668eb98aae4c476 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:59.910 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.903+0000 7f8c87fff700 1 -- 172.21.15.23:0/2434040297 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c80011290 con 0x7f8c900a9740 2020-11-20T11:46:59.910 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.903+0000 7f8c87fff700 1 -- 172.21.15.23:0/2434040297 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c80000e90 con 0x7f8c900a9740 2020-11-20T11:46:59.912 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.903+0000 7f8c87fff700 1 -- 172.21.15.23:0/2434040297 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c80009df0 con 0x7f8c900a9740 2020-11-20T11:46:59.912 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.904+0000 7f8c963ec700 1 -- 172.21.15.23:0/2434040297 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8c900a9740 msgr2=0x7f8c900a9bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:46:59.913 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.904+0000 7f8c963ec700 1 --2- 172.21.15.23:0/2434040297 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8c900a9740 0x7f8c900a9bf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f8c80008eb0 tx=0x7f8c800083f0).stop 2020-11-20T11:46:59.914 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.904+0000 7f8c963ec700 1 -- 172.21.15.23:0/2434040297 shutdown_connections 2020-11-20T11:46:59.914 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.904+0000 7f8c963ec700 1 --2- 172.21.15.23:0/2434040297 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8c900a9740 0x7f8c900a9bf0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:46:59.914 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.904+0000 7f8c963ec700 1 --2- 172.21.15.23:0/2434040297 >> v2:172.21.15.155:3300/0 conn(0x7f8c900ad080 0x7f8c900a5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:59.915 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.904+0000 7f8c963ec700 1 -- 172.21.15.23:0/2434040297 >> 172.21.15.23:0/2434040297 conn(0x7f8c90061850 msgr2=0x7f8c90061c60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:46:59.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.907+0000 7f8c963ec700 1 -- 172.21.15.23:0/2434040297 shutdown_connections 2020-11-20T11:46:59.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.907+0000 7f8c963ec700 1 -- 172.21.15.23:0/2434040297 wait complete. 2020-11-20T11:46:59.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.908+0000 7f8c963ec700 1 Processor -- start 2020-11-20T11:46:59.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.908+0000 7f8c963ec700 1 -- start start 2020-11-20T11:46:59.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.909+0000 7f8c963ec700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c900a9740 0x7f8c900a21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:59.928 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.910+0000 7f8c963ec700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c900acca0 0x7f8c900a0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:59.928 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.910+0000 7f8c963ec700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8c900ad080 0x7f8c90159fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:59.928 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.910+0000 7f8c963ec700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c900a1780 con 0x7f8c900acca0 2020-11-20T11:46:59.929 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.910+0000 7f8c963ec700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c900a1a20 con 0x7f8c900a9740 2020-11-20T11:46:59.929 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.910+0000 7f8c963ec700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c900a13d0 con 0x7f8c900ad080 2020-11-20T11:46:59.930 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.910+0000 7f8c953ea700 1 -- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c900a9740 msgr2=0x7f8c900a21c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:46:59.930 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.910+0000 7f8c953ea700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c900a9740 0x7f8c900a21c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:46:59.930 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.910+0000 7f8c94be9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c900acca0 0x7f8c900a0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:59.931 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.910+0000 7f8c94be9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c900acca0 0x7f8c900a0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46438/0 (socket says 172.21.15.23:46438) 2020-11-20T11:46:59.931 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.910+0000 7f8c94be9700 1 -- 172.21.15.23:0/3256228463 learned_addr learned my addr 172.21.15.23:0/3256228463 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:46:59.931 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.910+0000 7f8c94be9700 1 -- 172.21.15.23:0/3256228463 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8c900ad080 msgr2=0x7f8c90159fe0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2020-11-20T11:46:59.932 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.910+0000 7f8c94be9700 1 --2- 172.21.15.23:0/3256228463 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8c900ad080 0x7f8c90159fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:59.932 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.910+0000 7f8c94be9700 1 -- 172.21.15.23:0/3256228463 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c900a9740 msgr2=0x7f8c900a21c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:46:59.932 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.910+0000 7f8c94be9700 1 --2- 172.21.15.23:0/3256228463 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c900a9740 0x7f8c900a21c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:46:59.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.910+0000 7f8c94be9700 1 -- 172.21.15.23:0/3256228463 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c900a28a0 con 0x7f8c900acca0 2020-11-20T11:46:59.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.911+0000 7f8c94be9700 1 --2- 172.21.15.23:0/3256228463 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c900acca0 0x7f8c900a0ee0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f8c8800bb60 tx=0x7f8c88009290).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:46:59.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.912+0000 7f8c857fa700 1 -- 172.21.15.23:0/3256228463 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c88003ca0 con 0x7f8c900acca0 2020-11-20T11:46:59.934 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.912+0000 7f8c857fa700 1 -- 172.21.15.23:0/3256228463 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c88003e00 con 0x7f8c900acca0 2020-11-20T11:46:59.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.912+0000 7f8c963ec700 1 -- 172.21.15.23:0/3256228463 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c9015a4d0 con 0x7f8c900acca0 2020-11-20T11:46:59.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.912+0000 7f8c963ec700 1 -- 172.21.15.23:0/3256228463 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c90147fc0 con 0x7f8c900acca0 2020-11-20T11:46:59.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.915+0000 7f8c857fa700 1 -- 172.21.15.23:0/3256228463 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8800c7d0 con 0x7f8c900acca0 2020-11-20T11:46:59.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.915+0000 7f8c857fa700 1 -- 172.21.15.23:0/3256228463 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8c8800cb40 con 0x7f8c900acca0 2020-11-20T11:46:59.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.916+0000 7f8c857fa700 1 --2- 172.21.15.23:0/3256228463 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8c70056d60 0x7f8c70059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:46:59.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.918+0000 7f8c963ec700 1 -- 172.21.15.23:0/3256228463 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c900a2a90 con 0x7f8c900acca0 2020-11-20T11:46:59.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.923+0000 7f8c953ea700 1 --2- 172.21.15.23:0/3256228463 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8c70056d60 0x7f8c70059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:46:59.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.927+0000 7f8c857fa700 1 -- 172.21.15.23:0/3256228463 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f8c88011020 con 0x7f8c900acca0 2020-11-20T11:46:59.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.927+0000 7f8c857fa700 1 -- 172.21.15.23:0/3256228463 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8c900a2a90 con 0x7f8c900acca0 2020-11-20T11:46:59.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:46:59.927+0000 7f8c953ea700 1 --2- 172.21.15.23:0/3256228463 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8c70056d60 0x7f8c70059220 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f8c8c003610 tx=0x7f8c8c006ad0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:00.437 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:00.431+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4659.0:54594 4.2s0 4:57b2ba66:::100000003c9.0000000b:head [write 315392~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:00.716 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:00.713+0000 7f8c963ec700 1 -- 172.21.15.23:0/3256228463 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8c900a1590 con 0x7f8c900acca0 2020-11-20T11:47:00.717 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:00.714+0000 7f8c857fa700 1 -- 172.21.15.23:0/3256228463 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1479 (secure 0 0 0) 0x7f8c88044fc0 con 0x7f8c900acca0 2020-11-20T11:47:00.717 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":132,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":23,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:47:00.719 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:00.716+0000 7f8c963ec700 1 -- 172.21.15.23:0/3256228463 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8c70056d60 msgr2=0x7f8c70059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:00.719 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:00.716+0000 7f8c963ec700 1 --2- 172.21.15.23:0/3256228463 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8c70056d60 0x7f8c70059220 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f8c8c003610 tx=0x7f8c8c006ad0).stop 2020-11-20T11:47:00.719 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:00.716+0000 7f8c963ec700 1 -- 172.21.15.23:0/3256228463 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c900acca0 msgr2=0x7f8c900a0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:00.720 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:00.716+0000 7f8c963ec700 1 --2- 172.21.15.23:0/3256228463 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c900acca0 0x7f8c900a0ee0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f8c8800bb60 tx=0x7f8c88009290).stop 2020-11-20T11:47:00.720 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:00.716+0000 7f8c963ec700 1 -- 172.21.15.23:0/3256228463 shutdown_connections 2020-11-20T11:47:00.720 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:00.716+0000 7f8c963ec700 1 --2- 172.21.15.23:0/3256228463 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8c70056d60 0x7f8c70059220 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:00.720 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:00.716+0000 7f8c963ec700 1 --2- 172.21.15.23:0/3256228463 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8c900ad080 0x7f8c90159fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:47:00.720 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:00.716+0000 7f8c963ec700 1 --2- 172.21.15.23:0/3256228463 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8c900a9740 0x7f8c900a21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:47:00.721 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:00.716+0000 7f8c963ec700 1 --2- 172.21.15.23:0/3256228463 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c900acca0 0x7f8c900a0ee0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:00.721 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:00.716+0000 7f8c963ec700 1 -- 172.21.15.23:0/3256228463 >> 172.21.15.23:0/3256228463 conn(0x7f8c90061850 msgr2=0x7f8c900a7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:47:00.721 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:00.716+0000 7f8c963ec700 1 -- 172.21.15.23:0/3256228463 shutdown_connections 2020-11-20T11:47:00.721 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:00.716+0000 7f8c963ec700 1 -- 172.21.15.23:0/3256228463 wait complete. 2020-11-20T11:47:01.466 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:01.433+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 132 slow ops, oldest is osd_op(client.4659.0:54594 4.2s0 4:57b2ba66:::100000003c9.0000000b:head [write 315392~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:02.505 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:02.441+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 136 slow ops, oldest is osd_op(client.4659.0:54974 4.2s0 4:58320a1e:::10000000396.00000009:head [write 1908736~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:03.607 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:03.575+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 141 slow ops, oldest is osd_op(client.4659.0:54974 4.2s0 4:58320a1e:::10000000396.00000009:head [write 1908736~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:03.755 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:47:03.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:03.915+0000 7f2805626700 1 Processor -- start 2020-11-20T11:47:03.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:03.915+0000 7f2805626700 1 -- start start 2020-11-20T11:47:03.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:03.915+0000 7f2805626700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f28000ad7e0 0x7f28000a6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:03.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:03.915+0000 7f2805626700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f280013b340 con 0x7f28000a8fb0 2020-11-20T11:47:03.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:03.915+0000 7f2805626700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28000a6080 con 0x7f28000a5be0 2020-11-20T11:47:03.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:03.915+0000 7f2805626700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28000a6320 con 0x7f28000ad7e0 2020-11-20T11:47:03.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:03.915+0000 7f27fffff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f28000ad7e0 0x7f28000a6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:03.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:03.915+0000 7f27ff7fe700 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f28000a5be0 legacy=0x7f280013a520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:47:03.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:03.915+0000 7f27fffff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f28000ad7e0 0x7f28000a6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46442/0 (socket says 172.21.15.23:46442) 2020-11-20T11:47:03.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:03.915+0000 7f27fffff700 1 -- 172.21.15.23:0/453341916 learned_addr learned my addr 172.21.15.23:0/453341916 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:47:03.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:03.916+0000 7f27fffff700 1 -- 172.21.15.23:0/453341916 >> v2:172.21.15.23:3300/0 conn(0x7f28000ad7e0 msgr2=0x7f28000a6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:47:03.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:03.916+0000 7f27fffff700 1 -- 172.21.15.23:0/453341916 >> v2:172.21.15.23:3300/0 conn(0x7f28000ad7e0 msgr2=0x7f28000a6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:47:03.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:03.916+0000 7f27fffff700 1 --2- 172.21.15.23:0/453341916 >> v2:172.21.15.23:3300/0 conn(0x7f28000ad7e0 0x7f28000a6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:47:03.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:03.916+0000 7f27fffff700 1 --2- 172.21.15.23:0/453341916 >> v2:172.21.15.23:3300/0 conn(0x7f28000ad7e0 0x7f28000a6ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:47:04.119 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:04.116+0000 7f27ff7fe700 1 -- 172.21.15.23:0/453341916 >> v1:172.21.15.155:6789/0 conn(0x7f28000a5be0 legacy=0x7f280013a520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:47:04.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:04.117+0000 7f27fffff700 1 --2- 172.21.15.23:0/453341916 >> v2:172.21.15.23:3300/0 conn(0x7f28000ad7e0 0x7f28000a6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:04.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:04.117+0000 7f27fffff700 1 -- 172.21.15.23:0/453341916 >> v2:172.21.15.23:3300/0 conn(0x7f28000ad7e0 msgr2=0x7f28000a6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:47:04.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:04.117+0000 7f27fffff700 1 -- 172.21.15.23:0/453341916 >> v2:172.21.15.23:3300/0 conn(0x7f28000ad7e0 msgr2=0x7f28000a6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:47:04.121 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:04.117+0000 7f27fffff700 1 --2- 172.21.15.23:0/453341916 >> v2:172.21.15.23:3300/0 conn(0x7f28000ad7e0 0x7f28000a6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:47:04.121 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:04.117+0000 7f27fffff700 1 --2- 172.21.15.23:0/453341916 >> v2:172.21.15.23:3300/0 conn(0x7f28000ad7e0 0x7f28000a6ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:47:04.520 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:04.517+0000 7f27ff7fe700 1 -- 172.21.15.23:0/453341916 >> v1:172.21.15.155:6789/0 conn(0x7f28000a5be0 legacy=0x7f280013a520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:47:04.521 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:04.518+0000 7f27fffff700 1 --2- 172.21.15.23:0/453341916 >> v2:172.21.15.23:3300/0 conn(0x7f28000ad7e0 0x7f28000a6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:04.521 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:04.518+0000 7f27fffff700 1 -- 172.21.15.23:0/453341916 >> v2:172.21.15.23:3300/0 conn(0x7f28000ad7e0 msgr2=0x7f28000a6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:47:04.521 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:04.518+0000 7f27fffff700 1 -- 172.21.15.23:0/453341916 >> v2:172.21.15.23:3300/0 conn(0x7f28000ad7e0 msgr2=0x7f28000a6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:47:04.521 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:04.518+0000 7f27fffff700 1 --2- 172.21.15.23:0/453341916 >> v2:172.21.15.23:3300/0 conn(0x7f28000ad7e0 0x7f28000a6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:47:04.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:04.518+0000 7f27fffff700 1 --2- 172.21.15.23:0/453341916 >> v2:172.21.15.23:3300/0 conn(0x7f28000ad7e0 0x7f28000a6ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:47:04.618 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:04.552+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4659.0:54974 4.2s0 4:58320a1e:::10000000396.00000009:head [write 1908736~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:05.322 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:05.319+0000 7f27fffff700 1 --2- 172.21.15.23:0/453341916 >> v2:172.21.15.23:3300/0 conn(0x7f28000ad7e0 0x7f28000a6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:05.323 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:05.319+0000 7f27fffff700 1 -- 172.21.15.23:0/453341916 >> v2:172.21.15.23:3300/0 conn(0x7f28000ad7e0 msgr2=0x7f28000a6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:47:05.323 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:05.319+0000 7f27fffff700 1 -- 172.21.15.23:0/453341916 >> v2:172.21.15.23:3300/0 conn(0x7f28000ad7e0 msgr2=0x7f28000a6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:47:05.323 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:05.319+0000 7f27fffff700 1 --2- 172.21.15.23:0/453341916 >> v2:172.21.15.23:3300/0 conn(0x7f28000ad7e0 0x7f28000a6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:47:05.324 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:05.319+0000 7f27fffff700 1 --2- 172.21.15.23:0/453341916 >> v2:172.21.15.23:3300/0 conn(0x7f28000ad7e0 0x7f28000a6ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:47:05.330 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:05.323+0000 7f27ff7fe700 1 -- 172.21.15.23:0/453341916 >> v1:172.21.15.155:6789/0 conn(0x7f28000a5be0 legacy=0x7f280013a520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:47:05.630 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:05.625+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 151 slow ops, oldest is osd_op(client.4659.0:54993 4.2s0 4:548adc15:::10000000360.00000002:head [write 274432~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:06.715 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:06.684+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 153 slow ops, oldest is osd_op(client.4659.0:55152 4.2s0 4:5daf17f2:::100000003a4.00000003:head [write 2457600~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:06.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:06.915+0000 7f27fdffb700 1 -- 172.21.15.23:0/453341916 >> v2:172.21.15.23:3300/0 conn(0x7f28000ad7e0 msgr2=0x7f28000a6ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:47:06.919 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:06.915+0000 7f27fdffb700 1 --2- 172.21.15.23:0/453341916 >> v2:172.21.15.23:3300/0 conn(0x7f28000ad7e0 0x7f28000a6ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:06.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:06.915+0000 7f27fdffb700 1 -- 172.21.15.23:0/453341916 >> v1:172.21.15.155:6789/0 conn(0x7f28000a5be0 legacy=0x7f280013a520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:47:06.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:06.915+0000 7f27fdffb700 1 -- 172.21.15.23:0/453341916 >> v1:172.21.15.23:6789/0 conn(0x7f28000a8fb0 legacy=0x7f280013ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:06.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:06.915+0000 7f27fdffb700 1 --2- 172.21.15.23:0/453341916 >> v2:172.21.15.155:3300/0 conn(0x7f27e00033f0 0x7f27e0005870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:06.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:06.915+0000 7f27fdffb700 1 --2- 172.21.15.23:0/453341916 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27e0005e10 0x7f27e00082d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:06.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:06.915+0000 7f27fdffb700 1 -- 172.21.15.23:0/453341916 --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28000a6320 con 0x7f27e00033f0 2020-11-20T11:47:06.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:06.915+0000 7f27fdffb700 1 -- 172.21.15.23:0/453341916 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27e00089b0 con 0x7f27e0005e10 2020-11-20T11:47:06.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:06.916+0000 7f27fffff700 1 -- 172.21.15.23:0/453341916 >> v2:172.21.15.155:3300/0 conn(0x7f27e00033f0 msgr2=0x7f27e0005870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:47:06.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:06.916+0000 7f27fffff700 1 --2- 172.21.15.23:0/453341916 >> v2:172.21.15.155:3300/0 conn(0x7f27e00033f0 0x7f27e0005870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:47:06.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:06.916+0000 7f27ff7fe700 1 --2- 172.21.15.23:0/453341916 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27e0005e10 0x7f27e00082d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:06.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:06.916+0000 7f27ff7fe700 1 -- 172.21.15.23:0/453341916 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27e0005e10 msgr2=0x7f27e00082d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:47:06.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:06.916+0000 7f27ff7fe700 1 -- 172.21.15.23:0/453341916 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27e0005e10 msgr2=0x7f27e00082d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:47:06.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:06.916+0000 7f27ff7fe700 1 --2- 172.21.15.23:0/453341916 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27e0005e10 0x7f27e00082d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:47:06.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:06.916+0000 7f27ff7fe700 1 --2- 172.21.15.23:0/453341916 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27e0005e10 0x7f27e00082d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:47:07.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.117+0000 7f27fffff700 1 -- 172.21.15.23:0/453341916 >> v2:172.21.15.155:3300/0 conn(0x7f27e00033f0 msgr2=0x7f27e0005870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:47:07.121 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.117+0000 7f27fffff700 1 --2- 172.21.15.23:0/453341916 >> v2:172.21.15.155:3300/0 conn(0x7f27e00033f0 0x7f27e0005870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:47:07.121 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.117+0000 7f27ff7fe700 1 --2- 172.21.15.23:0/453341916 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27e0005e10 0x7f27e00082d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:07.121 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.117+0000 7f27ff7fe700 1 -- 172.21.15.23:0/453341916 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27e0005e10 msgr2=0x7f27e00082d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:47:07.121 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.117+0000 7f27ff7fe700 1 -- 172.21.15.23:0/453341916 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27e0005e10 msgr2=0x7f27e00082d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:47:07.122 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.117+0000 7f27ff7fe700 1 --2- 172.21.15.23:0/453341916 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27e0005e10 0x7f27e00082d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:47:07.122 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.117+0000 7f27ff7fe700 1 --2- 172.21.15.23:0/453341916 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27e0005e10 0x7f27e00082d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:47:07.521 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.518+0000 7f27fffff700 1 -- 172.21.15.23:0/453341916 >> v2:172.21.15.155:3300/0 conn(0x7f27e00033f0 msgr2=0x7f27e0005870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:47:07.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.518+0000 7f27fffff700 1 --2- 172.21.15.23:0/453341916 >> v2:172.21.15.155:3300/0 conn(0x7f27e00033f0 0x7f27e0005870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:47:07.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.518+0000 7f27ff7fe700 1 --2- 172.21.15.23:0/453341916 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27e0005e10 0x7f27e00082d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:07.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.518+0000 7f27ff7fe700 1 -- 172.21.15.23:0/453341916 >> v2:172.21.15.155:3300/0 conn(0x7f27e00033f0 msgr2=0x7f27e0005870 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:47:07.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.518+0000 7f27ff7fe700 1 --2- 172.21.15.23:0/453341916 >> v2:172.21.15.155:3300/0 conn(0x7f27e00033f0 0x7f27e0005870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:47:07.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.518+0000 7f27ff7fe700 1 -- 172.21.15.23:0/453341916 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28000adbc0 con 0x7f27e0005e10 2020-11-20T11:47:07.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.519+0000 7f27ff7fe700 1 --2- 172.21.15.23:0/453341916 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27e0005e10 0x7f27e00082d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f27f0006fd0 tx=0x7f27f0008040).ready entity=mon.2 client_cookie=f3f2a2d62d7a6039 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:07.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.519+0000 7f27feffd700 1 -- 172.21.15.23:0/453341916 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f000f8b0 con 0x7f27e0005e10 2020-11-20T11:47:07.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.519+0000 7f27feffd700 1 -- 172.21.15.23:0/453341916 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27f000fa10 con 0x7f27e0005e10 2020-11-20T11:47:07.525 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.519+0000 7f27feffd700 1 -- 172.21.15.23:0/453341916 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f000fd90 con 0x7f27e0005e10 2020-11-20T11:47:07.525 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.520+0000 7f2805626700 1 -- 172.21.15.23:0/453341916 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27e0005e10 msgr2=0x7f27e00082d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:07.525 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.520+0000 7f2805626700 1 --2- 172.21.15.23:0/453341916 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27e0005e10 0x7f27e00082d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f27f0006fd0 tx=0x7f27f0008040).stop 2020-11-20T11:47:07.525 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.520+0000 7f2804e25700 1 -- 172.21.15.23:0/453341916 reap_dead start 2020-11-20T11:47:07.526 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.520+0000 7f2805626700 1 -- 172.21.15.23:0/453341916 shutdown_connections 2020-11-20T11:47:07.526 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.520+0000 7f2805626700 1 -- 172.21.15.23:0/453341916 >> 172.21.15.23:0/453341916 conn(0x7f2800061850 msgr2=0x7f2800061c60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:47:07.526 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.520+0000 7f2805626700 1 -- 172.21.15.23:0/453341916 shutdown_connections 2020-11-20T11:47:07.526 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.520+0000 7f2805626700 1 -- 172.21.15.23:0/453341916 wait complete. 2020-11-20T11:47:07.529 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.521+0000 7f2805626700 1 Processor -- start 2020-11-20T11:47:07.530 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.521+0000 7f2805626700 1 -- start start 2020-11-20T11:47:07.530 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.522+0000 7f2805626700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27e0005e10 0x7f28000670f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:07.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.522+0000 7f2805626700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28000a3440 0x7f28000a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:07.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.522+0000 7f2805626700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28000a1770 0x7f280014b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:07.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.522+0000 7f2805626700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2800139d90 con 0x7f28000a3440 2020-11-20T11:47:07.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.522+0000 7f2805626700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f280012c310 con 0x7f28000a1770 2020-11-20T11:47:07.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.522+0000 7f2805626700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f280012c5b0 con 0x7f27e0005e10 2020-11-20T11:47:07.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.523+0000 7f27ff7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28000a3440 0x7f28000a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:07.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.523+0000 7f2804e25700 1 -- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28000a1770 msgr2=0x7f280014b6e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:47:07.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.523+0000 7f2804e25700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28000a1770 0x7f280014b6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:47:07.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.523+0000 7f27ff7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28000a3440 0x7f28000a1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46474/0 (socket says 172.21.15.23:46474) 2020-11-20T11:47:07.533 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.523+0000 7f27ff7fe700 1 -- 172.21.15.23:0/2083191759 learned_addr learned my addr 172.21.15.23:0/2083191759 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:47:07.533 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.523+0000 7f27fffff700 1 --2- 172.21.15.23:0/2083191759 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27e0005e10 0x7f28000670f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:07.533 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.523+0000 7f27ff7fe700 1 -- 172.21.15.23:0/2083191759 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27e0005e10 msgr2=0x7f28000670f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:07.534 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.523+0000 7f27ff7fe700 1 --2- 172.21.15.23:0/2083191759 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27e0005e10 0x7f28000670f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:07.534 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.523+0000 7f27ff7fe700 1 -- 172.21.15.23:0/2083191759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28000a1770 msgr2=0x7f280014b6e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:47:07.534 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.523+0000 7f27ff7fe700 1 --2- 172.21.15.23:0/2083191759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28000a1770 0x7f280014b6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:47:07.534 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.523+0000 7f27ff7fe700 1 -- 172.21.15.23:0/2083191759 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f280012c850 con 0x7f28000a3440 2020-11-20T11:47:07.534 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.523+0000 7f27ff7fe700 1 --2- 172.21.15.23:0/2083191759 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28000a3440 0x7f28000a1280 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f27e0008b40 tx=0x7f27f0008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:07.535 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.523+0000 7f27dffff700 1 -- 172.21.15.23:0/2083191759 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f0005920 con 0x7f28000a3440 2020-11-20T11:47:07.536 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.523+0000 7f27dffff700 1 -- 172.21.15.23:0/2083191759 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27f0022070 con 0x7f28000a3440 2020-11-20T11:47:07.536 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.523+0000 7f2805626700 1 -- 172.21.15.23:0/2083191759 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28000a0d40 con 0x7f28000a3440 2020-11-20T11:47:07.537 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.523+0000 7f27dffff700 1 -- 172.21.15.23:0/2083191759 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f00036c0 con 0x7f28000a3440 2020-11-20T11:47:07.537 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.523+0000 7f2805626700 1 -- 172.21.15.23:0/2083191759 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f280014bd90 con 0x7f28000a3440 2020-11-20T11:47:07.537 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.524+0000 7f2805626700 1 -- 172.21.15.23:0/2083191759 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28000a10f0 con 0x7f28000a3440 2020-11-20T11:47:07.537 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.524+0000 7f27dffff700 1 -- 172.21.15.23:0/2083191759 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f27f002e070 con 0x7f28000a3440 2020-11-20T11:47:07.538 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.525+0000 7f27dffff700 1 --2- 172.21.15.23:0/2083191759 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f27d8056d10 0x7f27d80591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:07.538 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.525+0000 7f27dffff700 1 -- 172.21.15.23:0/2083191759 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f27f0072060 con 0x7f28000a3440 2020-11-20T11:47:07.538 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.525+0000 7f27fffff700 1 --2- 172.21.15.23:0/2083191759 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f27d8056d10 0x7f27d80591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:07.538 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.525+0000 7f27fffff700 1 --2- 172.21.15.23:0/2083191759 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f27d8056d10 0x7f27d80591d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f27f40069e0 tx=0x7f27f4008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:07.539 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.529+0000 7f27dffff700 1 -- 172.21.15.23:0/2083191759 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f27f0047350 con 0x7f28000a3440 2020-11-20T11:47:07.771 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:07.731+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 154 slow ops, oldest is osd_op(client.4659.0:55505 4.2s0 4:4c361bf0:::1000000034d.00000009:head [write 3477504~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:07.811 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.808+0000 7f2805626700 1 -- 172.21.15.23:0/2083191759 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2800057dc0 con 0x7f28000a3440 2020-11-20T11:47:07.811 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.808+0000 7f27dffff700 1 -- 172.21.15.23:0/2083191759 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1472 (secure 0 0 0) 0x7f27f0013e60 con 0x7f28000a3440 2020-11-20T11:47:07.812 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":136,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:47:07.813 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.810+0000 7f2805626700 1 -- 172.21.15.23:0/2083191759 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f27d8056d10 msgr2=0x7f27d80591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:07.813 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.810+0000 7f2805626700 1 --2- 172.21.15.23:0/2083191759 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f27d8056d10 0x7f27d80591d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f27f40069e0 tx=0x7f27f4008040).stop 2020-11-20T11:47:07.813 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.810+0000 7f2805626700 1 -- 172.21.15.23:0/2083191759 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28000a3440 msgr2=0x7f28000a1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:07.814 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.810+0000 7f2805626700 1 --2- 172.21.15.23:0/2083191759 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28000a3440 0x7f28000a1280 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f27e0008b40 tx=0x7f27f0008040).stop 2020-11-20T11:47:07.814 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.810+0000 7f2805626700 1 -- 172.21.15.23:0/2083191759 shutdown_connections 2020-11-20T11:47:07.814 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.810+0000 7f2805626700 1 --2- 172.21.15.23:0/2083191759 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f28000a1770 0x7f280014b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:47:07.814 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.810+0000 7f2805626700 1 --2- 172.21.15.23:0/2083191759 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f27d8056d10 0x7f27d80591d0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:07.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.810+0000 7f2805626700 1 --2- 172.21.15.23:0/2083191759 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f27e0005e10 0x7f28000670f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:07.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.810+0000 7f2805626700 1 --2- 172.21.15.23:0/2083191759 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f28000a3440 0x7f28000a1280 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:07.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.810+0000 7f2805626700 1 -- 172.21.15.23:0/2083191759 >> 172.21.15.23:0/2083191759 conn(0x7f2800061850 msgr2=0x7f2800134f40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:47:07.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.810+0000 7f2805626700 1 -- 172.21.15.23:0/2083191759 shutdown_connections 2020-11-20T11:47:07.815 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:07.810+0000 7f2805626700 1 -- 172.21.15.23:0/2083191759 wait complete. 2020-11-20T11:47:07.826 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2020-11-20T11:47:07.826 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:47:08.786 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:08.766+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 155 slow ops, oldest is osd_op(client.4659.0:55505 4.2s0 4:4c361bf0:::1000000034d.00000009:head [write 3477504~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:09.755 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:09.722+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 158 slow ops, oldest is osd_op(client.4659.0:55505 4.2s0 4:4c361bf0:::1000000034d.00000009:head [write 3477504~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:10.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.246+0000 7f4803a32700 1 Processor -- start 2020-11-20T11:47:10.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.246+0000 7f4803a32700 1 -- start start 2020-11-20T11:47:10.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.247+0000 7f4803a32700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f47fc0aaab0 0x7f47fc13ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:10.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.247+0000 7f4803a32700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f47fc0aae90 0x7f47fc1381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:10.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.247+0000 7f4803a32700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47fc13e600 con 0x7f47fc13aeb0 2020-11-20T11:47:10.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.247+0000 7f4803a32700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47fc13e8a0 con 0x7f47fc0aaab0 2020-11-20T11:47:10.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.247+0000 7f4803a32700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47fc0ab6d0 con 0x7f47fc0aae90 2020-11-20T11:47:10.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.247+0000 7f4801fcf700 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f47fc13aeb0 legacy=0x7f47fc0a8c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2020-11-20T11:47:10.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.247+0000 7f4800fcd700 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f47fc0aae90 msgr2=0x7f47fc1381a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:47:10.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.247+0000 7f4800fcd700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f47fc0aae90 0x7f47fc1381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:47:10.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.247+0000 7f48017ce700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f47fc0aaab0 0x7f47fc13ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:10.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.247+0000 7f48017ce700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f47fc0aaab0 0x7f47fc13ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46482/0 (socket says 172.21.15.23:46482) 2020-11-20T11:47:10.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.247+0000 7f48017ce700 1 -- 172.21.15.23:0/2225581667 learned_addr learned my addr 172.21.15.23:0/2225581667 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:47:10.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.248+0000 7f48017ce700 1 -- 172.21.15.23:0/2225581667 >> v2:172.21.15.155:3300/0 conn(0x7f47fc0aae90 msgr2=0x7f47fc1381a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:47:10.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.248+0000 7f48017ce700 1 --2- 172.21.15.23:0/2225581667 >> v2:172.21.15.155:3300/0 conn(0x7f47fc0aae90 0x7f47fc1381a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:47:10.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.248+0000 7f48017ce700 1 -- 172.21.15.23:0/2225581667 >> v1:172.21.15.155:6789/0 conn(0x7f47fc13aeb0 legacy=0x7f47fc0a8c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:47:10.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.248+0000 7f48017ce700 1 -- 172.21.15.23:0/2225581667 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47fc0a9370 con 0x7f47fc0aaab0 2020-11-20T11:47:10.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.248+0000 7f48017ce700 1 --2- 172.21.15.23:0/2225581667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47fc0aaab0 0x7f47fc13ed40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f47fc0ab930 tx=0x7f47f8006ef0).ready entity=mon.0 client_cookie=2193c7f6d40b967e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:10.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.248+0000 7f47ebfff700 1 -- 172.21.15.23:0/2225581667 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f800b820 con 0x7f47fc0aaab0 2020-11-20T11:47:10.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.249+0000 7f47ebfff700 1 -- 172.21.15.23:0/2225581667 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f8000e90 con 0x7f47fc0aaab0 2020-11-20T11:47:10.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.249+0000 7f47ebfff700 1 -- 172.21.15.23:0/2225581667 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f800bb90 con 0x7f47fc0aaab0 2020-11-20T11:47:10.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.249+0000 7f4803a32700 1 -- 172.21.15.23:0/2225581667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47fc0aaab0 msgr2=0x7f47fc13ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:10.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.249+0000 7f4803a32700 1 --2- 172.21.15.23:0/2225581667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47fc0aaab0 0x7f47fc13ed40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f47fc0ab930 tx=0x7f47f8006ef0).stop 2020-11-20T11:47:10.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.249+0000 7f4803a32700 1 -- 172.21.15.23:0/2225581667 shutdown_connections 2020-11-20T11:47:10.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.249+0000 7f4803a32700 1 --2- 172.21.15.23:0/2225581667 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47fc0aaab0 0x7f47fc13ed40 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:10.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.249+0000 7f4803a32700 1 --2- 172.21.15.23:0/2225581667 >> v2:172.21.15.155:3300/0 conn(0x7f47fc0aae90 0x7f47fc1381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:47:10.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.249+0000 7f4803a32700 1 -- 172.21.15.23:0/2225581667 >> 172.21.15.23:0/2225581667 conn(0x7f47fc0678e0 msgr2=0x7f47fc067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:47:10.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.250+0000 7f4803a32700 1 -- 172.21.15.23:0/2225581667 shutdown_connections 2020-11-20T11:47:10.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.250+0000 7f4803a32700 1 -- 172.21.15.23:0/2225581667 wait complete. 2020-11-20T11:47:10.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.251+0000 7f4803a32700 1 Processor -- start 2020-11-20T11:47:10.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.251+0000 7f4803a32700 1 -- start start 2020-11-20T11:47:10.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.251+0000 7f4803a32700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f47fc0aae90 0x7f47fc0b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:10.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.252+0000 7f4803a32700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47fc0a5980 0x7f47fc130880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:10.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.252+0000 7f4803a32700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47fc13c870 0x7f47fc13ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:10.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.252+0000 7f4803a32700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47fc13d1e0 con 0x7f47fc0a5980 2020-11-20T11:47:10.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.252+0000 7f4803a32700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47fc135e80 con 0x7f47fc13c870 2020-11-20T11:47:10.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.252+0000 7f4803a32700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47fc136120 con 0x7f47fc0aae90 2020-11-20T11:47:10.265 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.252+0000 7f4801fcf700 1 -- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47fc13c870 msgr2=0x7f47fc13ccf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:47:10.266 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.252+0000 7f4801fcf700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47fc13c870 0x7f47fc13ccf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:47:10.266 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.252+0000 7f4800fcd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47fc0a5980 0x7f47fc130880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:10.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.252+0000 7f4800fcd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47fc0a5980 0x7f47fc130880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46488/0 (socket says 172.21.15.23:46488) 2020-11-20T11:47:10.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.252+0000 7f4800fcd700 1 -- 172.21.15.23:0/508690377 learned_addr learned my addr 172.21.15.23:0/508690377 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:47:10.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.252+0000 7f48017ce700 1 --2- 172.21.15.23:0/508690377 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f47fc0aae90 0x7f47fc0b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:10.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.252+0000 7f4800fcd700 1 -- 172.21.15.23:0/508690377 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f47fc0aae90 msgr2=0x7f47fc0b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:10.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.252+0000 7f4800fcd700 1 --2- 172.21.15.23:0/508690377 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f47fc0aae90 0x7f47fc0b03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:10.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.252+0000 7f4800fcd700 1 -- 172.21.15.23:0/508690377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47fc13c870 msgr2=0x7f47fc13ccf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:47:10.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.252+0000 7f4800fcd700 1 --2- 172.21.15.23:0/508690377 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f47fc13c870 0x7f47fc13ccf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:47:10.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.252+0000 7f4800fcd700 1 -- 172.21.15.23:0/508690377 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47fc1363c0 con 0x7f47fc0a5980 2020-11-20T11:47:10.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.252+0000 7f48017ce700 1 --2- 172.21.15.23:0/508690377 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f47fc0aae90 0x7f47fc0b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:47:10.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.252+0000 7f4800fcd700 1 --2- 172.21.15.23:0/508690377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47fc0a5980 0x7f47fc130880 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f47ec009790 tx=0x7f47ec006ea0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:10.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.252+0000 7f47e97fa700 1 -- 172.21.15.23:0/508690377 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec006170 con 0x7f47fc0a5980 2020-11-20T11:47:10.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.253+0000 7f47e97fa700 1 -- 172.21.15.23:0/508690377 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47ec0062d0 con 0x7f47fc0a5980 2020-11-20T11:47:10.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.253+0000 7f4803a32700 1 -- 172.21.15.23:0/508690377 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47fc0b0b20 con 0x7f47fc0a5980 2020-11-20T11:47:10.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.253+0000 7f47e97fa700 1 -- 172.21.15.23:0/508690377 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec003920 con 0x7f47fc0a5980 2020-11-20T11:47:10.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.253+0000 7f4803a32700 1 -- 172.21.15.23:0/508690377 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47fc135c10 con 0x7f47fc0a5980 2020-11-20T11:47:10.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.253+0000 7f4803a32700 1 --2- 172.21.15.23:0/508690377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47fc13aeb0 0x7f47fc134590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:10.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.253+0000 7f4803a32700 1 -- 172.21.15.23:0/508690377 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47fc05c7d0 con 0x7f47fc13aeb0 2020-11-20T11:47:10.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.254+0000 7f48017ce700 1 --2- 172.21.15.23:0/508690377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47fc13aeb0 0x7f47fc134590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:10.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.254+0000 7f48017ce700 1 -- 172.21.15.23:0/508690377 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47fc0a8050 con 0x7f47fc13aeb0 2020-11-20T11:47:10.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.254+0000 7f48017ce700 1 --2- 172.21.15.23:0/508690377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47fc13aeb0 0x7f47fc134590 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f47fc0ab930 tx=0x7f47f8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:10.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.254+0000 7f47e97fa700 1 -- 172.21.15.23:0/508690377 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f8011610 con 0x7f47fc13aeb0 2020-11-20T11:47:10.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.254+0000 7f47e97fa700 1 -- 172.21.15.23:0/508690377 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f47f8011770 con 0x7f47fc13aeb0 2020-11-20T11:47:10.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.254+0000 7f47e97fa700 1 -- 172.21.15.23:0/508690377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47fc13aeb0 msgr2=0x7f47fc134590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:10.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.254+0000 7f47e97fa700 1 --2- 172.21.15.23:0/508690377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47fc13aeb0 0x7f47fc134590 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f47fc0ab930 tx=0x7f47f8016000).stop 2020-11-20T11:47:10.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.254+0000 7f47e97fa700 1 -- 172.21.15.23:0/508690377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47fc13aeb0 msgr2=0x7f47fc134590 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:47:10.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.254+0000 7f47e97fa700 1 --2- 172.21.15.23:0/508690377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47fc13aeb0 0x7f47fc134590 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:10.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.255+0000 7f47e97fa700 1 -- 172.21.15.23:0/508690377 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f47ec003d10 con 0x7f47fc0a5980 2020-11-20T11:47:10.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.255+0000 7f47e97fa700 1 --2- 172.21.15.23:0/508690377 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f47d4057420 0x7f47d40598e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:10.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.255+0000 7f47e97fa700 1 -- 172.21.15.23:0/508690377 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f47ec06fe90 con 0x7f47fc0a5980 2020-11-20T11:47:10.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.256+0000 7f48017ce700 1 --2- 172.21.15.23:0/508690377 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f47d4057420 0x7f47d40598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:10.275 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.256+0000 7f48017ce700 1 --2- 172.21.15.23:0/508690377 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f47d4057420 0x7f47d40598e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f47fc05c910 tx=0x7f47f801f040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:10.275 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.262+0000 7f4803a32700 1 --2- 172.21.15.23:0/508690377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47fc0601b0 0x7f47fc134a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:10.275 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.262+0000 7f4803a32700 1 -- 172.21.15.23:0/508690377 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47fc044710 con 0x7f47fc0601b0 2020-11-20T11:47:10.276 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.262+0000 7f4801fcf700 1 --2- 172.21.15.23:0/508690377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47fc0601b0 0x7f47fc134a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:10.276 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.263+0000 7f4801fcf700 1 -- 172.21.15.23:0/508690377 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f47fc05c280 con 0x7f47fc0601b0 2020-11-20T11:47:10.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.263+0000 7f4801fcf700 1 --2- 172.21.15.23:0/508690377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47fc0601b0 0x7f47fc134a80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f47fc0af230 tx=0x7f47f4009220).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:10.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.263+0000 7f47e97fa700 1 -- 172.21.15.23:0/508690377 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f4003cf0 con 0x7f47fc0601b0 2020-11-20T11:47:10.278 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.263+0000 7f47e97fa700 1 -- 172.21.15.23:0/508690377 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4775 (secure 0 0 0) 0x7f47f4003e50 con 0x7f47fc0601b0 2020-11-20T11:47:10.278 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.263+0000 7f47e97fa700 1 -- 172.21.15.23:0/508690377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47fc0601b0 msgr2=0x7f47fc134a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:10.278 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.263+0000 7f47e97fa700 1 --2- 172.21.15.23:0/508690377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47fc0601b0 0x7f47fc134a80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f47fc0af230 tx=0x7f47f4009220).stop 2020-11-20T11:47:10.279 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.263+0000 7f47e97fa700 1 -- 172.21.15.23:0/508690377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47fc0601b0 msgr2=0x7f47fc134a80 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:47:10.279 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.263+0000 7f47e97fa700 1 --2- 172.21.15.23:0/508690377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47fc0601b0 0x7f47fc134a80 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:10.279 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.263+0000 7f4803a32700 1 -- 172.21.15.23:0/508690377 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f47d4057420 msgr2=0x7f47d40598e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:10.279 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.264+0000 7f4803a32700 1 --2- 172.21.15.23:0/508690377 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f47d4057420 0x7f47d40598e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f47fc05c910 tx=0x7f47f801f040).stop 2020-11-20T11:47:10.280 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.264+0000 7f4803a32700 1 -- 172.21.15.23:0/508690377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47fc0a5980 msgr2=0x7f47fc130880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:10.281 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.264+0000 7f4803a32700 1 --2- 172.21.15.23:0/508690377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f47fc0a5980 0x7f47fc130880 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f47ec009790 tx=0x7f47ec006ea0).stop 2020-11-20T11:47:10.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.264+0000 7f4801fcf700 1 -- 172.21.15.23:0/508690377 reap_dead start 2020-11-20T11:47:10.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.264+0000 7f4801fcf700 1 -- 172.21.15.23:0/508690377 reap_dead start 2020-11-20T11:47:10.282 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.264+0000 7f4803a32700 1 -- 172.21.15.23:0/508690377 shutdown_connections 2020-11-20T11:47:10.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.264+0000 7f4803a32700 1 -- 172.21.15.23:0/508690377 >> 172.21.15.23:0/508690377 conn(0x7f47fc0678e0 msgr2=0x7f47fc133b20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:47:10.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.264+0000 7f4803a32700 1 -- 172.21.15.23:0/508690377 shutdown_connections 2020-11-20T11:47:10.283 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.264+0000 7f4803a32700 1 -- 172.21.15.23:0/508690377 wait complete. 2020-11-20T11:47:10.283 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:47:10.283 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:47:10.284 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:47:10.285 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:47:10.285 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 136, 2020-11-20T11:47:10.285 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:47:10.285 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:47:10.286 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:47:10.286 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:10.286 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 2, 2020-11-20T11:47:10.286 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:47:10.287 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:47:10.287 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:47:10.287 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:47:10.287 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:47:10.287 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:47:10.288 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:47:10.288 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:47:10.288 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:47:10.288 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:47:10.288 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:47:10.289 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:10.289 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:47:10.289 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:47:10.289 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:47:10.289 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:47:10.290 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:47:10.290 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:47:10.290 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:47:10.290 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:47:10.290 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:47:10.291 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:47:10.291 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:10.291 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:10.291 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:47:10.291 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:47:10.292 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:47:10.292 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:47:10.292 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:47:10.292 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:47:10.292 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:47:10.293 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:47:10.293 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:47:10.293 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:47:10.293 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:47:10.294 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:47:10.295 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:47:10.295 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:47:10.296 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:47:10.296 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:47:10.296 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:47:10.296 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:47:10.296 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:47:10.297 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:47:10.297 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:47:10.297 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:47:10.297 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:47:10.297 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:10.298 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:47:10.298 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:10.298 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:47:10.298 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:10.298 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:47:10.299 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:47:10.299 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:47:10.299 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:47:10.299 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:10.299 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:47:10.300 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:47:10.300 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:10.300 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:10.300 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:10.300 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:47:10.301 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:47:10.301 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:10.301 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:10.301 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:10.301 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:10.302 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:47:10.302 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:47:10.302 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:47:10.302 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:47:10.302 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:47:10.303 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:10.303 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:10.303 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:47:10.303 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:47:10.303 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:47:10.304 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:47:10.304 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:10.304 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:47:10.304 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:47:10.304 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:10.305 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:10.305 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:10.305 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:47:10.305 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:47:10.305 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:10.306 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:10.306 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:10.306 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:10.306 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:47:10.306 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:47:10.307 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:47:10.307 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:47:10.307 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:47:10.307 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:10.307 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:10.308 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:47:10.308 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:47:10.308 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:47:10.308 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:47:10.308 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:10.309 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:47:10.309 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:47:10.309 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:10.309 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:10.309 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:10.311 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:47:10.311 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:47:10.311 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:10.311 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:10.312 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:10.312 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:10.312 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:47:10.312 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:47:10.312 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:47:10.313 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:47:10.313 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:47:10.313 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:10.313 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:10.313 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:10.314 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:47:10.314 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:47:10.314 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:10.314 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:10.314 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:10.315 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:47:10.315 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:10.315 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:10.315 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:47:10.315 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:10.316 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:10.316 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:10.316 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:47:10.316 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:10.316 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:10.318 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:47:10.318 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:10.318 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:10.318 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:10.318 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:47:10.319 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:10.319 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:10.319 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:47:10.319 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:10.319 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:10.320 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:10.320 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 4 2020-11-20T11:47:10.320 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:10.320 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:10.320 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:47:10.321 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:10.321 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:10.321 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:10.321 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:47:10.321 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:10.322 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:10.322 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:10.322 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:47:10.322 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:47:10.323 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:47:10.472 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.465+0000 7f4c6425e700 1 Processor -- start 2020-11-20T11:47:10.473 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.466+0000 7f4c6425e700 1 -- start start 2020-11-20T11:47:10.473 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.466+0000 7f4c6425e700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f4c5c0a6e10 0x7f4c5c061850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:10.473 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.466+0000 7f4c6425e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c5c137410 0x7f4c5c137870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:10.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.466+0000 7f4c6425e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4c5c13a760 0x7f4c5c13abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:10.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.466+0000 7f4c6425e700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c5c0a61f0 con 0x7f4c5c0a6e10 2020-11-20T11:47:10.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.466+0000 7f4c6425e700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c5c062ad0 con 0x7f4c5c13a760 2020-11-20T11:47:10.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.466+0000 7f4c6425e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c0a63e0 con 0x7f4c5c137410 2020-11-20T11:47:10.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.466+0000 7f4c627fb700 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f4c5c13a760 msgr2=0x7f4c5c13abd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:47:10.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.466+0000 7f4c61ffa700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f4c5c0a6e10 0x7f4c5c061850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:10.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.466+0000 7f4c627fb700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4c5c13a760 0x7f4c5c13abd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:47:10.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.466+0000 7f4c61ffa700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f4c5c0a6e10 0x7f4c5c061850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46500/0 (socket says 172.21.15.23:46500) 2020-11-20T11:47:10.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.466+0000 7f4c61ffa700 1 -- 172.21.15.23:0/3031030511 learned_addr learned my addr 172.21.15.23:0/3031030511 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:47:10.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.467+0000 7f4c617f9700 1 --2- 172.21.15.23:0/3031030511 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c5c137410 0x7f4c5c137870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:10.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.467+0000 7f4c61ffa700 1 -- 172.21.15.23:0/3031030511 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c5c137410 msgr2=0x7f4c5c137870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:47:10.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.467+0000 7f4c61ffa700 1 --2- 172.21.15.23:0/3031030511 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c5c137410 0x7f4c5c137870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:10.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.467+0000 7f4c61ffa700 1 -- 172.21.15.23:0/3031030511 >> v2:172.21.15.155:3300/0 conn(0x7f4c5c13a760 msgr2=0x7f4c5c13abd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:47:10.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.467+0000 7f4c61ffa700 1 --2- 172.21.15.23:0/3031030511 >> v2:172.21.15.155:3300/0 conn(0x7f4c5c13a760 0x7f4c5c13abd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:47:10.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.467+0000 7f4c61ffa700 1 -- 172.21.15.23:0/3031030511 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5c0a6680 con 0x7f4c5c0a6e10 2020-11-20T11:47:10.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.467+0000 7f4c61ffa700 1 --2- 172.21.15.23:0/3031030511 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c5c0a6e10 0x7f4c5c061850 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f4c5c0a46a0 tx=0x7f4c4c006ef0).ready entity=mon.0 client_cookie=757fcd3cdcf94b67 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:10.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.467+0000 7f4c60ff8700 1 -- 172.21.15.23:0/3031030511 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c00b860 con 0x7f4c5c0a6e10 2020-11-20T11:47:10.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.467+0000 7f4c60ff8700 1 -- 172.21.15.23:0/3031030511 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4c000e90 con 0x7f4c5c0a6e10 2020-11-20T11:47:10.477 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.467+0000 7f4c60ff8700 1 -- 172.21.15.23:0/3031030511 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c00bbd0 con 0x7f4c5c0a6e10 2020-11-20T11:47:10.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.468+0000 7f4c6425e700 1 -- 172.21.15.23:0/3031030511 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c5c0a6e10 msgr2=0x7f4c5c061850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:10.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.468+0000 7f4c6425e700 1 --2- 172.21.15.23:0/3031030511 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c5c0a6e10 0x7f4c5c061850 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f4c5c0a46a0 tx=0x7f4c4c006ef0).stop 2020-11-20T11:47:10.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.468+0000 7f4c6425e700 1 -- 172.21.15.23:0/3031030511 shutdown_connections 2020-11-20T11:47:10.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.468+0000 7f4c6425e700 1 --2- 172.21.15.23:0/3031030511 >> v2:172.21.15.155:3300/0 conn(0x7f4c5c13a760 0x7f4c5c13abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:47:10.478 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.468+0000 7f4c6425e700 1 --2- 172.21.15.23:0/3031030511 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c5c0a6e10 0x7f4c5c061850 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:10.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.468+0000 7f4c6425e700 1 --2- 172.21.15.23:0/3031030511 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c5c137410 0x7f4c5c137870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:10.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.468+0000 7f4c6425e700 1 -- 172.21.15.23:0/3031030511 >> 172.21.15.23:0/3031030511 conn(0x7f4c5c060c80 msgr2=0x7f4c5c0acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:47:10.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.468+0000 7f4c6425e700 1 -- 172.21.15.23:0/3031030511 shutdown_connections 2020-11-20T11:47:10.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.468+0000 7f4c6425e700 1 -- 172.21.15.23:0/3031030511 wait complete. 2020-11-20T11:47:10.479 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.469+0000 7f4c6425e700 1 Processor -- start 2020-11-20T11:47:10.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.469+0000 7f4c6425e700 1 -- start start 2020-11-20T11:47:10.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.469+0000 7f4c6425e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4c5c0a6e10 0x7f4c5c0660e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:10.480 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.469+0000 7f4c6425e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c5c137410 0x7f4c5c0aa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:10.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.469+0000 7f4c6425e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c5c13a760 0x7f4c5c14e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:10.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.469+0000 7f4c6425e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c14a5d0 con 0x7f4c5c13a760 2020-11-20T11:47:10.483 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.469+0000 7f4c6425e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c141540 con 0x7f4c5c0a6e10 2020-11-20T11:47:10.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.469+0000 7f4c6425e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c066780 con 0x7f4c5c137410 2020-11-20T11:47:10.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.469+0000 7f4c61ffa700 1 -- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4c5c0a6e10 msgr2=0x7f4c5c0660e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:47:10.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.470+0000 7f4c61ffa700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4c5c0a6e10 0x7f4c5c0660e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:47:10.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.470+0000 7f4c627fb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c5c13a760 0x7f4c5c14e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:10.484 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.470+0000 7f4c627fb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c5c13a760 0x7f4c5c14e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46510/0 (socket says 172.21.15.23:46510) 2020-11-20T11:47:10.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.470+0000 7f4c627fb700 1 -- 172.21.15.23:0/576935862 learned_addr learned my addr 172.21.15.23:0/576935862 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:47:10.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.470+0000 7f4c617f9700 1 --2- 172.21.15.23:0/576935862 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c5c137410 0x7f4c5c0aa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:10.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.470+0000 7f4c627fb700 1 -- 172.21.15.23:0/576935862 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c5c137410 msgr2=0x7f4c5c0aa1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:10.485 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.470+0000 7f4c627fb700 1 --2- 172.21.15.23:0/576935862 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c5c137410 0x7f4c5c0aa1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:10.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.470+0000 7f4c627fb700 1 -- 172.21.15.23:0/576935862 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4c5c0a6e10 msgr2=0x7f4c5c0660e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:47:10.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.470+0000 7f4c627fb700 1 --2- 172.21.15.23:0/576935862 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4c5c0a6e10 0x7f4c5c0660e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:47:10.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.470+0000 7f4c627fb700 1 -- 172.21.15.23:0/576935862 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5c14dca0 con 0x7f4c5c13a760 2020-11-20T11:47:10.486 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.470+0000 7f4c627fb700 1 --2- 172.21.15.23:0/576935862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c5c13a760 0x7f4c5c14e140 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f4c5400bab0 tx=0x7f4c540091a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:10.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.470+0000 7f4c49ffb700 1 -- 172.21.15.23:0/576935862 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c54003c50 con 0x7f4c5c13a760 2020-11-20T11:47:10.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.470+0000 7f4c49ffb700 1 -- 172.21.15.23:0/576935862 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c54003dd0 con 0x7f4c5c13a760 2020-11-20T11:47:10.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.470+0000 7f4c49ffb700 1 -- 172.21.15.23:0/576935862 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c54006170 con 0x7f4c5c13a760 2020-11-20T11:47:10.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.470+0000 7f4c6425e700 1 -- 172.21.15.23:0/576935862 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c5c0ad9e0 con 0x7f4c5c13a760 2020-11-20T11:47:10.487 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.470+0000 7f4c6425e700 1 -- 172.21.15.23:0/576935862 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c5c14e870 con 0x7f4c5c13a760 2020-11-20T11:47:10.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.471+0000 7f4c6425e700 1 --2- 172.21.15.23:0/576935862 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c5c14ab80 0x7f4c5c14af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:10.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.471+0000 7f4c6425e700 1 -- 172.21.15.23:0/576935862 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c0665d0 con 0x7f4c5c14ab80 2020-11-20T11:47:10.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.471+0000 7f4c61ffa700 1 --2- 172.21.15.23:0/576935862 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c5c14ab80 0x7f4c5c14af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:10.488 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.471+0000 7f4c49ffb700 1 -- 172.21.15.23:0/576935862 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f4c54006520 con 0x7f4c5c13a760 2020-11-20T11:47:10.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.471+0000 7f4c61ffa700 1 -- 172.21.15.23:0/576935862 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c5c066940 con 0x7f4c5c14ab80 2020-11-20T11:47:10.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.472+0000 7f4c49ffb700 1 --2- 172.21.15.23:0/576935862 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4c34056d10 0x7f4c340591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:10.490 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.472+0000 7f4c61ffa700 1 --2- 172.21.15.23:0/576935862 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c5c14ab80 0x7f4c5c14af60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f4c5c0a46a0 tx=0x7f4c4c016000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:10.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.472+0000 7f4c49ffb700 1 -- 172.21.15.23:0/576935862 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f4c54011020 con 0x7f4c5c13a760 2020-11-20T11:47:10.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.472+0000 7f4c49ffb700 1 -- 172.21.15.23:0/576935862 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c011aa0 con 0x7f4c5c14ab80 2020-11-20T11:47:10.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.472+0000 7f4c617f9700 1 --2- 172.21.15.23:0/576935862 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4c34056d10 0x7f4c340591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:10.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.472+0000 7f4c49ffb700 1 -- 172.21.15.23:0/576935862 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f4c4c011c00 con 0x7f4c5c14ab80 2020-11-20T11:47:10.491 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.472+0000 7f4c49ffb700 1 -- 172.21.15.23:0/576935862 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c5c14ab80 msgr2=0x7f4c5c14af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:10.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.472+0000 7f4c49ffb700 1 --2- 172.21.15.23:0/576935862 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c5c14ab80 0x7f4c5c14af60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f4c5c0a46a0 tx=0x7f4c4c016000).stop 2020-11-20T11:47:10.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.472+0000 7f4c49ffb700 1 -- 172.21.15.23:0/576935862 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c5c14ab80 msgr2=0x7f4c5c14af60 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:47:10.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.472+0000 7f4c49ffb700 1 --2- 172.21.15.23:0/576935862 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c5c14ab80 0x7f4c5c14af60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:10.492 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.472+0000 7f4c617f9700 1 --2- 172.21.15.23:0/576935862 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4c34056d10 0x7f4c340591d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f4c5c0aa960 tx=0x7f4c58006e30).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:10.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.482+0000 7f4c6425e700 1 --2- 172.21.15.23:0/576935862 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c5c13ff20 0x7f4c5c140300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:10.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.482+0000 7f4c6425e700 1 -- 172.21.15.23:0/576935862 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5c136530 con 0x7f4c5c13ff20 2020-11-20T11:47:10.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.482+0000 7f4c61ffa700 1 --2- 172.21.15.23:0/576935862 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c5c13ff20 0x7f4c5c140300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:10.494 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.482+0000 7f4c61ffa700 1 -- 172.21.15.23:0/576935862 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4c5c136b50 con 0x7f4c5c13ff20 2020-11-20T11:47:10.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.483+0000 7f4c61ffa700 1 --2- 172.21.15.23:0/576935862 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c5c13ff20 0x7f4c5c140300 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f4c5c14a7f0 tx=0x7f4c4c01e000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:10.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.483+0000 7f4c49ffb700 1 -- 172.21.15.23:0/576935862 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c01d830 con 0x7f4c5c13ff20 2020-11-20T11:47:10.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.483+0000 7f4c49ffb700 1 -- 172.21.15.23:0/576935862 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4773 (secure 0 0 0) 0x7f4c4c011890 con 0x7f4c5c13ff20 2020-11-20T11:47:10.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.483+0000 7f4c49ffb700 1 -- 172.21.15.23:0/576935862 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c5c13ff20 msgr2=0x7f4c5c140300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:10.495 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.483+0000 7f4c49ffb700 1 --2- 172.21.15.23:0/576935862 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c5c13ff20 0x7f4c5c140300 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f4c5c14a7f0 tx=0x7f4c4c01e000).stop 2020-11-20T11:47:10.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.483+0000 7f4c49ffb700 1 -- 172.21.15.23:0/576935862 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c5c13ff20 msgr2=0x7f4c5c140300 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:47:10.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.483+0000 7f4c49ffb700 1 --2- 172.21.15.23:0/576935862 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4c5c13ff20 0x7f4c5c140300 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:10.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.484+0000 7f4c6425e700 1 -- 172.21.15.23:0/576935862 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4c34056d10 msgr2=0x7f4c340591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:10.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.484+0000 7f4c6425e700 1 --2- 172.21.15.23:0/576935862 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f4c34056d10 0x7f4c340591d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f4c5c0aa960 tx=0x7f4c58006e30).stop 2020-11-20T11:47:10.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.484+0000 7f4c627fb700 1 -- 172.21.15.23:0/576935862 reap_dead start 2020-11-20T11:47:10.497 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.484+0000 7f4c6425e700 1 -- 172.21.15.23:0/576935862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c5c13a760 msgr2=0x7f4c5c14e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:10.497 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.484+0000 7f4c6425e700 1 --2- 172.21.15.23:0/576935862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c5c13a760 0x7f4c5c14e140 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f4c5400bab0 tx=0x7f4c540091a0).stop 2020-11-20T11:47:10.497 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.484+0000 7f4c6425e700 1 -- 172.21.15.23:0/576935862 shutdown_connections 2020-11-20T11:47:10.498 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.484+0000 7f4c6425e700 1 --2- 172.21.15.23:0/576935862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4c5c13a760 0x7f4c5c14e140 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:10.498 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.484+0000 7f4c6425e700 1 -- 172.21.15.23:0/576935862 >> 172.21.15.23:0/576935862 conn(0x7f4c5c060c80 msgr2=0x7f4c5c061850 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:47:10.498 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.486+0000 7f4c6425e700 1 -- 172.21.15.23:0/576935862 shutdown_connections 2020-11-20T11:47:10.498 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:10.487+0000 7f4c6425e700 1 -- 172.21.15.23:0/576935862 wait complete. 2020-11-20T11:47:10.499 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:47:10.499 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:47:10.499 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:47:10.499 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:47:10.500 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 136, 2020-11-20T11:47:10.500 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:47:10.500 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:47:10.500 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:47:10.500 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:10.500 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 3, 2020-11-20T11:47:10.501 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:47:10.501 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:47:10.501 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:47:10.501 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:47:10.501 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:47:10.502 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:47:10.502 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:47:10.502 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:47:10.502 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:47:10.502 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:47:10.503 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:47:10.503 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:10.503 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:47:10.503 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:47:10.503 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:47:10.504 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:47:10.504 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:47:10.504 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:47:10.504 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:47:10.504 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:47:10.505 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:47:10.505 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:47:10.505 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:10.505 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:10.506 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:47:10.506 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:47:10.506 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:47:10.507 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:47:10.507 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:47:10.507 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:47:10.507 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:47:10.507 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:47:10.508 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:47:10.508 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:47:10.508 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:47:10.508 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:47:10.508 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:47:10.509 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:47:10.509 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:47:10.509 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:47:10.509 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:47:10.509 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:47:10.510 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:47:10.510 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:47:10.510 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:47:10.510 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:47:10.510 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:47:10.511 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:10.511 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:47:10.511 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:10.511 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:47:10.511 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:10.512 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:47:10.512 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:47:10.512 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:47:10.512 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:47:10.512 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:10.513 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:47:10.513 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:47:10.513 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:10.513 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:10.513 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:10.514 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:47:10.514 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:47:10.514 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:10.514 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:10.514 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:10.515 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:10.515 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:47:10.515 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:47:10.515 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:47:10.515 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:47:10.516 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:47:10.517 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:10.517 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:10.517 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:47:10.517 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:47:10.518 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:47:10.518 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:47:10.518 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:10.518 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:47:10.518 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:47:10.519 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:10.519 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:10.519 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:10.519 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:47:10.519 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:47:10.519 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:10.520 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:10.520 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:10.520 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:10.520 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:47:10.520 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:47:10.521 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:47:10.521 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:47:10.521 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:47:10.521 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:10.521 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:10.522 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:47:10.522 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:47:10.522 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:47:10.522 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:47:10.523 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:10.523 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:47:10.523 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:47:10.523 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:10.524 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:10.524 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:10.524 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:47:10.524 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:47:10.525 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:10.526 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:10.526 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:10.526 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:10.526 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:47:10.527 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:47:10.527 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:47:10.527 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:47:10.527 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:47:10.528 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:10.528 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:10.528 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:10.528 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:47:10.528 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:47:10.529 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:10.529 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:10.529 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:10.529 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:47:10.530 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:10.530 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:10.530 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:47:10.530 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:10.530 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:10.530 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:10.531 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:47:10.531 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:10.531 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:10.531 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:47:10.531 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:10.532 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:10.532 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:10.532 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:47:10.532 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:10.532 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:10.533 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:47:10.533 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:10.533 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:10.533 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:10.534 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:47:10.534 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:10.534 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:10.534 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:47:10.534 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:10.536 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:10.536 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:10.541 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:47:10.542 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:10.542 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:10.542 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:10.542 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:47:10.542 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:47:10.544 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2020-11-20T11:47:10.836 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:10.784+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 158 slow ops, oldest is osd_op(client.4659.0:55529 4.2s0 4:49e75ffb:::100000003a1.00000004:head [write 925696~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:11.839 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:11.788+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 157 slow ops, oldest is osd_op(client.4659.0:55823 4.2s0 4:59b0b385:::1000000037e.00000010:head [write 438272~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:12.819 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:12.773+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 156 slow ops, oldest is osd_op(client.4659.0:55823 4.2s0 4:59b0b385:::1000000037e.00000010:head [write 438272~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:13.805 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:13.766+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 159 slow ops, oldest is osd_op(client.4659.0:56819 4.2s0 4:59b0b385:::1000000037e.00000010:head [write 495616~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:14.860 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:14.839+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 161 slow ops, oldest is osd_op(client.4659.0:56819 4.2s0 4:59b0b385:::1000000037e.00000010:head [write 495616~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:15.883 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:15.837+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 166 slow ops, oldest is osd_op(client.4659.0:56819 4.2s0 4:59b0b385:::1000000037e.00000010:head [write 495616~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:16.942 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:16.908+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 167 slow ops, oldest is osd_op(client.4659.0:56819 4.2s0 4:59b0b385:::1000000037e.00000010:head [write 495616~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:17.951 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:17.899+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 170 slow ops, oldest is osd_op(client.4659.0:56819 4.2s0 4:59b0b385:::1000000037e.00000010:head [write 495616~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:18.934 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:18.905+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 172 slow ops, oldest is osd_op(client.4659.0:56819 4.2s0 4:59b0b385:::1000000037e.00000010:head [write 495616~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:19.955 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:19.882+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 174 slow ops, oldest is osd_op(client.4659.0:56819 4.2s0 4:59b0b385:::1000000037e.00000010:head [write 495616~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:20.899 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:20.880+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 176 slow ops, oldest is osd_op(client.4659.0:56819 4.2s0 4:59b0b385:::1000000037e.00000010:head [write 495616~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:21.989 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:21.954+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 175 slow ops, oldest is osd_op(client.4659.0:56898 4.2s0 4:5ef0fbf2:::10000000378.00000001:head [write 3473408~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:23.065 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:23.029+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4659.0:57109 4.2s0 4:5bc0f152:::100000003c2.00000013:head [write 1474560~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:24.104 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:24.076+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 178 slow ops, oldest is osd_op(client.4659.0:57298 4.2s0 4:59881595:::10000000350.0000000b:head [write 471040~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:25.192 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:25.171+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 181 slow ops, oldest is osd_op(client.4659.0:57298 4.2s0 4:59881595:::10000000350.0000000b:head [write 471040~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:26.214 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:26.207+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 181 slow ops, oldest is osd_op(client.4659.0:57344 4.2s0 4:5f7ad767:::100000003c9.00000017:head [write 3899392~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:27.307 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:27.256+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 180 slow ops, oldest is osd_op(client.4659.0:57600 4.2s0 4:57e37626:::1000000036d.00000002:head [write 1282048~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:28.298 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:28.258+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 182 slow ops, oldest is osd_op(client.4659.0:57600 4.2s0 4:57e37626:::1000000036d.00000002:head [write 1282048~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:29.340 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:29.300+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 179 slow ops, oldest is osd_op(client.4659.0:57600 4.2s0 4:57e37626:::1000000036d.00000002:head [write 1282048~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:30.428 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:30.361+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 181 slow ops, oldest is osd_op(client.4659.0:58003 4.2s0 4:5a5d3f90:::10000000378.0000000a:head [write 2797568~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:30.545 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2020-11-20T11:47:30.545 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2020-11-20T11:47:30.545 INFO:tasks.ceph.mon.b:Restarting daemon 2020-11-20T11:47:30.546 INFO:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2020-11-20T11:47:30.548 INFO:tasks.ceph.mon.b:Started 2020-11-20T11:47:30.548 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:47:30.549 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:47:31.229 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.226+0000 7f808a0de700 1 Processor -- start 2020-11-20T11:47:31.230 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.226+0000 7f808a0de700 1 -- start start 2020-11-20T11:47:31.232 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.227+0000 7f808a0de700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f80840aaab0 0x7f808413ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:31.232 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.227+0000 7f808a0de700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f80840aae90 0x7f80841381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:31.232 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.227+0000 7f808a0de700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f808413e600 con 0x7f808413aeb0 2020-11-20T11:47:31.233 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.227+0000 7f808a0de700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f808413e8a0 con 0x7f80840aaab0 2020-11-20T11:47:31.233 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.227+0000 7f808a0de700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80840ab6d0 con 0x7f80840aae90 2020-11-20T11:47:31.233 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.227+0000 7f80837fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f80840aaab0 0x7f808413ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:31.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.227+0000 7f8083fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f808413aeb0 0x7f80840a8c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:60784/0 (socket says 172.21.15.23:60784) 2020-11-20T11:47:31.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.227+0000 7f8083fff700 1 -- 172.21.15.23:0/2868548169 learned_addr learned my addr 172.21.15.23:0/2868548169 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:47:31.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.227+0000 7f80837fe700 1 --2- 172.21.15.23:0/2868548169 >> v2:172.21.15.23:3300/0 conn(0x7f80840aaab0 0x7f808413ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46530/0 (socket says 172.21.15.23:46530) 2020-11-20T11:47:31.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.228+0000 7f8082ffd700 1 --2- 172.21.15.23:0/2868548169 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f80840aae90 0x7f80841381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:31.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.228+0000 7f80837fe700 1 -- 172.21.15.23:0/2868548169 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f80840aae90 msgr2=0x7f80841381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:47:31.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.228+0000 7f80837fe700 1 --2- 172.21.15.23:0/2868548169 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f80840aae90 0x7f80841381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:31.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.228+0000 7f80837fe700 1 -- 172.21.15.23:0/2868548169 >> v1:172.21.15.23:6789/0 conn(0x7f808413aeb0 legacy=0x7f80840a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:31.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.228+0000 7f80837fe700 1 -- 172.21.15.23:0/2868548169 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80840a9370 con 0x7f80840aaab0 2020-11-20T11:47:31.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.228+0000 7f80827fc700 1 -- 172.21.15.23:0/2868548169 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1130524736 0 0) 0x7f808413e600 con 0x7f808413aeb0 2020-11-20T11:47:31.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.228+0000 7f8082ffd700 1 --2- 172.21.15.23:0/2868548169 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f80840aae90 0x7f80841381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:47:31.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.228+0000 7f80837fe700 1 --2- 172.21.15.23:0/2868548169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80840aaab0 0x7f808413ed40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f8078006fd0 tx=0x7f8078006e30).ready entity=mon.0 client_cookie=5f75e288eb00ff1e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:31.238 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.228+0000 7f80827fc700 1 -- 172.21.15.23:0/2868548169 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80780097e0 con 0x7f80840aaab0 2020-11-20T11:47:31.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.228+0000 7f80827fc700 1 -- 172.21.15.23:0/2868548169 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8078000e90 con 0x7f80840aaab0 2020-11-20T11:47:31.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.228+0000 7f80827fc700 1 -- 172.21.15.23:0/2868548169 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8078009b50 con 0x7f80840aaab0 2020-11-20T11:47:31.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.228+0000 7f808a0de700 1 -- 172.21.15.23:0/2868548169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80840aaab0 msgr2=0x7f808413ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:31.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.228+0000 7f808a0de700 1 --2- 172.21.15.23:0/2868548169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80840aaab0 0x7f808413ed40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f8078006fd0 tx=0x7f8078006e30).stop 2020-11-20T11:47:31.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.228+0000 7f808a0de700 1 -- 172.21.15.23:0/2868548169 shutdown_connections 2020-11-20T11:47:31.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.228+0000 7f808a0de700 1 --2- 172.21.15.23:0/2868548169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80840aaab0 0x7f808413ed40 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:31.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.228+0000 7f808a0de700 1 --2- 172.21.15.23:0/2868548169 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f80840aae90 0x7f80841381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:31.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.228+0000 7f808a0de700 1 -- 172.21.15.23:0/2868548169 >> 172.21.15.23:0/2868548169 conn(0x7f80840678e0 msgr2=0x7f8084067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:47:31.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.228+0000 7f808a0de700 1 -- 172.21.15.23:0/2868548169 shutdown_connections 2020-11-20T11:47:31.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.228+0000 7f808a0de700 1 -- 172.21.15.23:0/2868548169 wait complete. 2020-11-20T11:47:31.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.230+0000 7f808a0de700 1 Processor -- start 2020-11-20T11:47:31.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.230+0000 7f808a0de700 1 -- start start 2020-11-20T11:47:31.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.230+0000 7f808a0de700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f80840aaab0 0x7f80840abed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:31.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.230+0000 7f808a0de700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f80840aae90 0x7f808413c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:31.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.230+0000 7f808a0de700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80840a5330 0x7f808414df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:31.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.230+0000 7f808a0de700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80840a5790 con 0x7f80840a5330 2020-11-20T11:47:31.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.230+0000 7f808a0de700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f808413cc30 con 0x7f80840aaab0 2020-11-20T11:47:31.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.230+0000 7f808a0de700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f808413ced0 con 0x7f80840aae90 2020-11-20T11:47:31.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.230+0000 7f8083fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80840a5330 0x7f808414df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:31.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.230+0000 7f80837fe700 1 -- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f80840aaab0 msgr2=0x7f80840abed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.155:3300/0 2020-11-20T11:47:31.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.230+0000 7f80837fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f80840aaab0 0x7f80840abed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:47:31.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.230+0000 7f8083fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80840a5330 0x7f808414df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46540/0 (socket says 172.21.15.23:46540) 2020-11-20T11:47:31.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.230+0000 7f8083fff700 1 -- 172.21.15.23:0/4108489272 learned_addr learned my addr 172.21.15.23:0/4108489272 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:47:31.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.230+0000 7f8082ffd700 1 --2- 172.21.15.23:0/4108489272 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f80840aae90 0x7f808413c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:31.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.231+0000 7f8083fff700 1 -- 172.21.15.23:0/4108489272 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f80840aae90 msgr2=0x7f808413c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:31.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.231+0000 7f8083fff700 1 --2- 172.21.15.23:0/4108489272 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f80840aae90 0x7f808413c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:31.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.231+0000 7f8083fff700 1 -- 172.21.15.23:0/4108489272 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f80840aaab0 msgr2=0x7f80840abed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:47:31.246 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.231+0000 7f8083fff700 1 --2- 172.21.15.23:0/4108489272 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f80840aaab0 0x7f80840abed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:47:31.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.231+0000 7f8083fff700 1 -- 172.21.15.23:0/4108489272 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f808414e400 con 0x7f80840a5330 2020-11-20T11:47:31.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.231+0000 7f8083fff700 1 --2- 172.21.15.23:0/4108489272 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80840a5330 0x7f808414df10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f807c00cae0 tx=0x7f807c001360).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:31.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.231+0000 7f80637fe700 1 -- 172.21.15.23:0/4108489272 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807c0020d0 con 0x7f80840a5330 2020-11-20T11:47:31.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.231+0000 7f80637fe700 1 -- 172.21.15.23:0/4108489272 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f807c005bc0 con 0x7f80840a5330 2020-11-20T11:47:31.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.231+0000 7f808a0de700 1 -- 172.21.15.23:0/4108489272 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f808414e5f0 con 0x7f80840a5330 2020-11-20T11:47:31.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.231+0000 7f80637fe700 1 -- 172.21.15.23:0/4108489272 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807c007110 con 0x7f80840a5330 2020-11-20T11:47:31.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.231+0000 7f808a0de700 1 -- 172.21.15.23:0/4108489272 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80840b08d0 con 0x7f80840a5330 2020-11-20T11:47:31.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.232+0000 7f808a0de700 1 -- 172.21.15.23:0/4108489272 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f808413ca80 con 0x7f80840a5330 2020-11-20T11:47:31.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.232+0000 7f80637fe700 1 -- 172.21.15.23:0/4108489272 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f807c010030 con 0x7f80840a5330 2020-11-20T11:47:31.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.232+0000 7f80637fe700 1 --2- 172.21.15.23:0/4108489272 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f805c056d10 0x7f805c0591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:31.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.233+0000 7f80637fe700 1 -- 172.21.15.23:0/4108489272 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f807c06fea0 con 0x7f80840a5330 2020-11-20T11:47:31.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.233+0000 7f80837fe700 1 --2- 172.21.15.23:0/4108489272 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f805c056d10 0x7f805c0591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:31.254 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.233+0000 7f80837fe700 1 --2- 172.21.15.23:0/4108489272 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f805c056d10 0x7f805c0591d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f807800b190 tx=0x7f8078015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:31.255 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.237+0000 7f80637fe700 1 -- 172.21.15.23:0/4108489272 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f807c045190 con 0x7f80840a5330 2020-11-20T11:47:31.437 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:31.414+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 183 slow ops, oldest is osd_op(client.4659.0:58003 4.2s0 4:5a5d3f90:::10000000378.0000000a:head [write 2797568~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:31.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.501+0000 7f808a0de700 1 -- 172.21.15.23:0/4108489272 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8084139590 con 0x7f80840a5330 2020-11-20T11:47:31.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.501+0000 7f80637fe700 1 -- 172.21.15.23:0/4108489272 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1473 (secure 0 0 0) 0x7f807c0487b0 con 0x7f80840a5330 2020-11-20T11:47:31.504 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":136,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:47:31.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.503+0000 7f808a0de700 1 -- 172.21.15.23:0/4108489272 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f805c056d10 msgr2=0x7f805c0591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:31.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.503+0000 7f808a0de700 1 --2- 172.21.15.23:0/4108489272 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f805c056d10 0x7f805c0591d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f807800b190 tx=0x7f8078015000).stop 2020-11-20T11:47:31.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.503+0000 7f808a0de700 1 -- 172.21.15.23:0/4108489272 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80840a5330 msgr2=0x7f808414df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:31.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.503+0000 7f808a0de700 1 --2- 172.21.15.23:0/4108489272 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80840a5330 0x7f808414df10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f807c00cae0 tx=0x7f807c001360).stop 2020-11-20T11:47:31.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.503+0000 7f808a0de700 1 -- 172.21.15.23:0/4108489272 shutdown_connections 2020-11-20T11:47:31.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.503+0000 7f808a0de700 1 --2- 172.21.15.23:0/4108489272 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80840a5330 0x7f808414df10 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:31.508 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.503+0000 7f808a0de700 1 --2- 172.21.15.23:0/4108489272 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f80840aaab0 0x7f80840abed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:47:31.508 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.503+0000 7f808a0de700 1 --2- 172.21.15.23:0/4108489272 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f805c056d10 0x7f805c0591d0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:31.508 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.503+0000 7f808a0de700 1 --2- 172.21.15.23:0/4108489272 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f80840aae90 0x7f808413c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:31.508 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.503+0000 7f808a0de700 1 -- 172.21.15.23:0/4108489272 >> 172.21.15.23:0/4108489272 conn(0x7f80840678e0 msgr2=0x7f808413eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:47:31.509 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.503+0000 7f808a0de700 1 -- 172.21.15.23:0/4108489272 shutdown_connections 2020-11-20T11:47:31.509 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:31.503+0000 7f808a0de700 1 -- 172.21.15.23:0/4108489272 wait complete. 2020-11-20T11:47:32.498 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:32.492+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 188 slow ops, oldest is osd_op(client.4659.0:58003 4.2s0 4:5a5d3f90:::10000000378.0000000a:head [write 2797568~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:33.557 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:33.524+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 193 slow ops, oldest is osd_op(client.4659.0:58398 4.2s0 4:56f73163:::10000000360.0000000d:head [write 618496~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:34.519 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:47:34.630 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:34.609+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 196 slow ops, oldest is osd_op(client.4659.0:58398 4.2s0 4:56f73163:::10000000360.0000000d:head [write 618496~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:34.700 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.697+0000 7f24cdfe9700 1 Processor -- start 2020-11-20T11:47:34.701 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.697+0000 7f24cdfe9700 1 -- start start 2020-11-20T11:47:34.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.698+0000 7f24cdfe9700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f24c80aaab0 0x7f24c813ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:34.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.698+0000 7f24cdfe9700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f24c80aae90 0x7f24c81381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:34.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.698+0000 7f24cdfe9700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c813e600 con 0x7f24c813aeb0 2020-11-20T11:47:34.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.698+0000 7f24cdfe9700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24c813e8a0 con 0x7f24c80aaab0 2020-11-20T11:47:34.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.698+0000 7f24cdfe9700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24c80ab6d0 con 0x7f24c80aae90 2020-11-20T11:47:34.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.698+0000 7f24c7fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f24c813aeb0 0x7f24c80a8c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:60798/0 (socket says 172.21.15.23:60798) 2020-11-20T11:47:34.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.698+0000 7f24c7fff700 1 -- 172.21.15.23:0/2252898682 learned_addr learned my addr 172.21.15.23:0/2252898682 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:47:34.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.698+0000 7f24c77fe700 1 --2- 172.21.15.23:0/2252898682 >> v2:172.21.15.155:3300/0 conn(0x7f24c80aaab0 0x7f24c813ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:34.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.698+0000 7f24c6ffd700 1 --2- 172.21.15.23:0/2252898682 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f24c80aae90 0x7f24c81381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:34.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.698+0000 7f24c77fe700 1 -- 172.21.15.23:0/2252898682 >> v2:172.21.15.155:3300/0 conn(0x7f24c80aaab0 msgr2=0x7f24c813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:47:34.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.698+0000 7f24c77fe700 1 -- 172.21.15.23:0/2252898682 >> v2:172.21.15.155:3300/0 conn(0x7f24c80aaab0 msgr2=0x7f24c813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:47:34.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.698+0000 7f24c77fe700 1 --2- 172.21.15.23:0/2252898682 >> v2:172.21.15.155:3300/0 conn(0x7f24c80aaab0 0x7f24c813ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:47:34.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.698+0000 7f24c77fe700 1 --2- 172.21.15.23:0/2252898682 >> v2:172.21.15.155:3300/0 conn(0x7f24c80aaab0 0x7f24c813ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:47:34.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.698+0000 7f24c67fc700 1 -- 172.21.15.23:0/2252898682 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918448885 0 0) 0x7f24c813e600 con 0x7f24c813aeb0 2020-11-20T11:47:34.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.698+0000 7f24c67fc700 1 -- 172.21.15.23:0/2252898682 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24a8003280 con 0x7f24c813aeb0 2020-11-20T11:47:34.707 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.698+0000 7f24c6ffd700 1 -- 172.21.15.23:0/2252898682 >> v2:172.21.15.155:3300/0 conn(0x7f24c80aaab0 msgr2=0x7f24c813ed40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:47:34.707 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.698+0000 7f24c6ffd700 1 --2- 172.21.15.23:0/2252898682 >> v2:172.21.15.155:3300/0 conn(0x7f24c80aaab0 0x7f24c813ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:34.707 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.698+0000 7f24c6ffd700 1 -- 172.21.15.23:0/2252898682 >> v1:172.21.15.23:6789/0 conn(0x7f24c813aeb0 legacy=0x7f24c80a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:34.707 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.698+0000 7f24c6ffd700 1 -- 172.21.15.23:0/2252898682 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c80a9370 con 0x7f24c80aae90 2020-11-20T11:47:34.707 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.699+0000 7f24c6ffd700 1 --2- 172.21.15.23:0/2252898682 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f24c80aae90 0x7f24c81381a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f24b8006fd0 tx=0x7f24b8006e30).ready entity=mon.2 client_cookie=e60d9debf7c32855 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:34.708 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.699+0000 7f24c67fc700 1 -- 172.21.15.23:0/2252898682 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b80097e0 con 0x7f24c80aae90 2020-11-20T11:47:34.708 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.699+0000 7f24c67fc700 1 -- 172.21.15.23:0/2252898682 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24b8000e90 con 0x7f24c80aae90 2020-11-20T11:47:34.708 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.699+0000 7f24c67fc700 1 -- 172.21.15.23:0/2252898682 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b8009b50 con 0x7f24c80aae90 2020-11-20T11:47:34.708 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.699+0000 7f24cdfe9700 1 -- 172.21.15.23:0/2252898682 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f24c80aae90 msgr2=0x7f24c81381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:34.709 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.699+0000 7f24cdfe9700 1 --2- 172.21.15.23:0/2252898682 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f24c80aae90 0x7f24c81381a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f24b8006fd0 tx=0x7f24b8006e30).stop 2020-11-20T11:47:34.709 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.699+0000 7f24cdfe9700 1 -- 172.21.15.23:0/2252898682 shutdown_connections 2020-11-20T11:47:34.709 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.699+0000 7f24cdfe9700 1 --2- 172.21.15.23:0/2252898682 >> v2:172.21.15.155:3300/0 conn(0x7f24c80aaab0 0x7f24c813ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:34.709 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.699+0000 7f24cdfe9700 1 --2- 172.21.15.23:0/2252898682 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f24c80aae90 0x7f24c81381a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:34.709 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.699+0000 7f24cdfe9700 1 -- 172.21.15.23:0/2252898682 >> 172.21.15.23:0/2252898682 conn(0x7f24c80678e0 msgr2=0x7f24c8067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:47:34.710 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.699+0000 7f24cdfe9700 1 -- 172.21.15.23:0/2252898682 shutdown_connections 2020-11-20T11:47:34.710 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.699+0000 7f24cdfe9700 1 -- 172.21.15.23:0/2252898682 wait complete. 2020-11-20T11:47:34.710 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.701+0000 7f24cdfe9700 1 Processor -- start 2020-11-20T11:47:34.710 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.701+0000 7f24cdfe9700 1 -- start start 2020-11-20T11:47:34.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.701+0000 7f24cdfe9700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24c80aaab0 0x7f24c8135930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:34.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.701+0000 7f24cdfe9700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f24c80aae90 0x7f24c813c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:34.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.701+0000 7f24cdfe9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24c813c7a0 0x7f24c814a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:34.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.701+0000 7f24cdfe9700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c813d720 con 0x7f24c813c7a0 2020-11-20T11:47:34.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.701+0000 7f24cdfe9700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c80a8300 con 0x7f24c80aaab0 2020-11-20T11:47:34.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.701+0000 7f24cdfe9700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24c8134ff0 con 0x7f24c80aae90 2020-11-20T11:47:34.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.701+0000 7f24c7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24c813c7a0 0x7f24c814a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:34.714 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.701+0000 7f24c7fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24c813c7a0 0x7f24c814a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46556/0 (socket says 172.21.15.23:46556) 2020-11-20T11:47:34.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.701+0000 7f24c7fff700 1 -- 172.21.15.23:0/4108399615 learned_addr learned my addr 172.21.15.23:0/4108399615 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:47:34.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.702+0000 7f24c6ffd700 1 --2- 172.21.15.23:0/4108399615 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f24c80aae90 0x7f24c813c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:34.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.702+0000 7f24c77fe700 1 --2- 172.21.15.23:0/4108399615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24c80aaab0 0x7f24c8135930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:34.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.702+0000 7f24c77fe700 1 -- 172.21.15.23:0/4108399615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24c80aaab0 msgr2=0x7f24c8135930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2020-11-20T11:47:34.715 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.702+0000 7f24c77fe700 1 -- 172.21.15.23:0/4108399615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24c80aaab0 msgr2=0x7f24c8135930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:47:34.716 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.702+0000 7f24c77fe700 1 --2- 172.21.15.23:0/4108399615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24c80aaab0 0x7f24c8135930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:47:34.716 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.702+0000 7f24c77fe700 1 --2- 172.21.15.23:0/4108399615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24c80aaab0 0x7f24c8135930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:47:34.716 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.702+0000 7f24c7fff700 1 -- 172.21.15.23:0/4108399615 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f24c80aae90 msgr2=0x7f24c813c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:34.716 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.702+0000 7f24c7fff700 1 --2- 172.21.15.23:0/4108399615 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f24c80aae90 0x7f24c813c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:34.717 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.702+0000 7f24c7fff700 1 -- 172.21.15.23:0/4108399615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24c80aaab0 msgr2=0x7f24c8135930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:47:34.717 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.702+0000 7f24c7fff700 1 --2- 172.21.15.23:0/4108399615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24c80aaab0 0x7f24c8135930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:34.717 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.702+0000 7f24c7fff700 1 -- 172.21.15.23:0/4108399615 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c8135490 con 0x7f24c813c7a0 2020-11-20T11:47:34.717 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.702+0000 7f24c7fff700 1 --2- 172.21.15.23:0/4108399615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24c813c7a0 0x7f24c814a110 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f24c0001460 tx=0x7f24c0007e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:34.718 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.702+0000 7f24bf7fe700 1 -- 172.21.15.23:0/4108399615 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c0005bc0 con 0x7f24c813c7a0 2020-11-20T11:47:34.718 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.702+0000 7f24bf7fe700 1 -- 172.21.15.23:0/4108399615 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c0005d20 con 0x7f24c813c7a0 2020-11-20T11:47:34.718 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.702+0000 7f24cdfe9700 1 -- 172.21.15.23:0/4108399615 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24c813cc00 con 0x7f24c813c7a0 2020-11-20T11:47:34.719 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.702+0000 7f24bf7fe700 1 -- 172.21.15.23:0/4108399615 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c0007320 con 0x7f24c813c7a0 2020-11-20T11:47:34.719 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.702+0000 7f24cdfe9700 1 -- 172.21.15.23:0/4108399615 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24c8134400 con 0x7f24c813c7a0 2020-11-20T11:47:34.719 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.703+0000 7f24cdfe9700 1 -- 172.21.15.23:0/4108399615 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24c805c7d0 con 0x7f24c813c7a0 2020-11-20T11:47:34.719 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.703+0000 7f24bf7fe700 1 -- 172.21.15.23:0/4108399615 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f24c0010030 con 0x7f24c813c7a0 2020-11-20T11:47:34.720 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.704+0000 7f24bf7fe700 1 --2- 172.21.15.23:0/4108399615 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f24a4056d60 0x7f24a4059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:34.720 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.704+0000 7f24bf7fe700 1 -- 172.21.15.23:0/4108399615 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f24c006ffd0 con 0x7f24c813c7a0 2020-11-20T11:47:34.720 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.704+0000 7f24c77fe700 1 --2- 172.21.15.23:0/4108399615 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f24a4056d60 0x7f24a4059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:34.721 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.704+0000 7f24c77fe700 1 --2- 172.21.15.23:0/4108399615 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f24a4056d60 0x7f24a4059220 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f24b0006fd0 tx=0x7f24b0008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:34.721 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:34.707+0000 7f24bf7fe700 1 -- 172.21.15.23:0/4108399615 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f24c00452c0 con 0x7f24c813c7a0 2020-11-20T11:47:35.028 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.025+0000 7f24cdfe9700 1 -- 172.21.15.23:0/4108399615 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f24c8135620 con 0x7f24c813c7a0 2020-11-20T11:47:35.029 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.025+0000 7f24bf7fe700 1 -- 172.21.15.23:0/4108399615 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7f24c00488e0 con 0x7f24c813c7a0 2020-11-20T11:47:35.029 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:47:35.030 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.027+0000 7f24cdfe9700 1 -- 172.21.15.23:0/4108399615 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f24a4056d60 msgr2=0x7f24a4059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:35.031 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.027+0000 7f24cdfe9700 1 --2- 172.21.15.23:0/4108399615 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f24a4056d60 0x7f24a4059220 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f24b0006fd0 tx=0x7f24b0008040).stop 2020-11-20T11:47:35.031 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.027+0000 7f24cdfe9700 1 -- 172.21.15.23:0/4108399615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24c813c7a0 msgr2=0x7f24c814a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:35.032 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.027+0000 7f24cdfe9700 1 --2- 172.21.15.23:0/4108399615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24c813c7a0 0x7f24c814a110 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f24c0001460 tx=0x7f24c0007e00).stop 2020-11-20T11:47:35.032 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.027+0000 7f24cdfe9700 1 -- 172.21.15.23:0/4108399615 shutdown_connections 2020-11-20T11:47:35.032 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.027+0000 7f24cdfe9700 1 --2- 172.21.15.23:0/4108399615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f24c813c7a0 0x7f24c814a110 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:35.032 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.027+0000 7f24cdfe9700 1 --2- 172.21.15.23:0/4108399615 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f24c80aaab0 0x7f24c8135930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:35.033 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.027+0000 7f24cdfe9700 1 --2- 172.21.15.23:0/4108399615 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f24a4056d60 0x7f24a4059220 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:35.033 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.027+0000 7f24cdfe9700 1 --2- 172.21.15.23:0/4108399615 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f24c80aae90 0x7f24c813c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:35.033 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.027+0000 7f24cdfe9700 1 -- 172.21.15.23:0/4108399615 >> 172.21.15.23:0/4108399615 conn(0x7f24c80678e0 msgr2=0x7f24c8139d40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:47:35.034 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.027+0000 7f24cdfe9700 1 -- 172.21.15.23:0/4108399615 shutdown_connections 2020-11-20T11:47:35.034 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.027+0000 7f24cdfe9700 1 -- 172.21.15.23:0/4108399615 wait complete. 2020-11-20T11:47:35.044 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:47:35.044 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:47:35.188 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.185+0000 7fd47dc29700 1 Processor -- start 2020-11-20T11:47:35.188 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.185+0000 7fd47dc29700 1 -- start start 2020-11-20T11:47:35.189 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.185+0000 7fd47dc29700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd478137870 0x7fd4781344d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.190 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.185+0000 7fd47dc29700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd478062ad0 con 0x7fd478137410 2020-11-20T11:47:35.190 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.185+0000 7fd47dc29700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4780a63e0 con 0x7fd4780a6e10 2020-11-20T11:47:35.190 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.185+0000 7fd47dc29700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4780a61f0 con 0x7fd478137870 2020-11-20T11:47:35.190 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.186+0000 7fd4777fe700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fd4780a6e10 0x7fd478061850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:38784/0 (socket says 172.21.15.23:38784) 2020-11-20T11:47:35.191 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.186+0000 7fd4777fe700 1 -- 172.21.15.23:0/2579980025 learned_addr learned my addr 172.21.15.23:0/2579980025 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:47:35.191 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.186+0000 7fd477fff700 1 --2- 172.21.15.23:0/2579980025 >> v2:172.21.15.155:3300/0 conn(0x7fd478137870 0x7fd4781344d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:35.191 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.185+0000 7fd476ffd700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fd478137410 0x7fd478133dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:60812/0 (socket says 172.21.15.23:60812) 2020-11-20T11:47:35.191 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.186+0000 7fd477fff700 1 -- 172.21.15.23:0/2579980025 >> v2:172.21.15.155:3300/0 conn(0x7fd478137870 msgr2=0x7fd4781344d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:47:35.191 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.186+0000 7fd477fff700 1 -- 172.21.15.23:0/2579980025 >> v2:172.21.15.155:3300/0 conn(0x7fd478137870 msgr2=0x7fd4781344d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:47:35.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.186+0000 7fd477fff700 1 --2- 172.21.15.23:0/2579980025 >> v2:172.21.15.155:3300/0 conn(0x7fd478137870 0x7fd4781344d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:47:35.192 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.186+0000 7fd477fff700 1 --2- 172.21.15.23:0/2579980025 >> v2:172.21.15.155:3300/0 conn(0x7fd478137870 0x7fd4781344d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:47:35.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.187+0000 7fd4767fc700 1 -- 172.21.15.23:0/2579980025 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153598910 0 0) 0x7fd478062ad0 con 0x7fd478137410 2020-11-20T11:47:35.193 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.187+0000 7fd4767fc700 1 -- 172.21.15.23:0/2579980025 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd464003280 con 0x7fd478137410 2020-11-20T11:47:35.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.187+0000 7fd4767fc700 1 -- 172.21.15.23:0/2579980025 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1964026242 0 0) 0x7fd464003280 con 0x7fd478137410 2020-11-20T11:47:35.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.187+0000 7fd4767fc700 1 -- 172.21.15.23:0/2579980025 >> v2:172.21.15.155:3300/0 conn(0x7fd478137870 msgr2=0x7fd4781344d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:47:35.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.187+0000 7fd4767fc700 1 --2- 172.21.15.23:0/2579980025 >> v2:172.21.15.155:3300/0 conn(0x7fd478137870 0x7fd4781344d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:35.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.187+0000 7fd4767fc700 1 -- 172.21.15.23:0/2579980025 >> v1:172.21.15.155:6789/0 conn(0x7fd4780a6e10 legacy=0x7fd478061850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:35.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.187+0000 7fd4767fc700 1 -- 172.21.15.23:0/2579980025 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4780a6680 con 0x7fd478137410 2020-11-20T11:47:35.205 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.187+0000 7fd4767fc700 1 -- 172.21.15.23:0/2579980025 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7fd46c001520 con 0x7fd478137410 2020-11-20T11:47:35.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.187+0000 7fd4767fc700 1 -- 172.21.15.23:0/2579980025 >> v1:172.21.15.23:6789/0 conn(0x7fd478137410 legacy=0x7fd478133dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:35.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.187+0000 7fd4767fc700 1 --2- 172.21.15.23:0/2579980025 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd464003770 0x7fd464003bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.187+0000 7fd4767fc700 1 --2- 172.21.15.23:0/2579980025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd464004520 0x7fd4640049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.187+0000 7fd4767fc700 1 --2- 172.21.15.23:0/2579980025 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd464004e90 0x7fd46400b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.187+0000 7fd4767fc700 1 -- 172.21.15.23:0/2579980025 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd464005370 con 0x7fd464004520 2020-11-20T11:47:35.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.187+0000 7fd4767fc700 1 -- 172.21.15.23:0/2579980025 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4640042c0 con 0x7fd464004e90 2020-11-20T11:47:35.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.187+0000 7fd4767fc700 1 -- 172.21.15.23:0/2579980025 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd464014570 con 0x7fd464003770 2020-11-20T11:47:35.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.188+0000 7fd476ffd700 1 --2- 172.21.15.23:0/2579980025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd464004520 0x7fd4640049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:35.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.188+0000 7fd477fff700 1 --2- 172.21.15.23:0/2579980025 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd464004e90 0x7fd46400b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:35.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.188+0000 7fd4777fe700 1 --2- 172.21.15.23:0/2579980025 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd464003770 0x7fd464003bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:35.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.188+0000 7fd476ffd700 1 -- 172.21.15.23:0/2579980025 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd464003770 msgr2=0x7fd464003bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:47:35.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.188+0000 7fd476ffd700 1 --2- 172.21.15.23:0/2579980025 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd464003770 0x7fd464003bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:35.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.188+0000 7fd476ffd700 1 -- 172.21.15.23:0/2579980025 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd464004e90 msgr2=0x7fd46400b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:47:35.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.188+0000 7fd476ffd700 1 --2- 172.21.15.23:0/2579980025 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd464004e90 0x7fd46400b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:35.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.188+0000 7fd477fff700 1 -- 172.21.15.23:0/2579980025 reap_dead start 2020-11-20T11:47:35.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.188+0000 7fd476ffd700 1 -- 172.21.15.23:0/2579980025 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd464003280 con 0x7fd464004520 2020-11-20T11:47:35.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.188+0000 7fd4777fe700 1 --2- 172.21.15.23:0/2579980025 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd464003770 0x7fd464003bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:47:35.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.188+0000 7fd476ffd700 1 --2- 172.21.15.23:0/2579980025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd464004520 0x7fd4640049a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fd46c01d450 tx=0x7fd4780a6810).ready entity=mon.0 client_cookie=8a3a421d44ae4cb6 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:35.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.193+0000 7fd4767fc700 1 -- 172.21.15.23:0/2579980025 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46c01de30 con 0x7fd464004520 2020-11-20T11:47:35.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.193+0000 7fd4767fc700 1 -- 172.21.15.23:0/2579980025 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd46c0072e0 con 0x7fd464004520 2020-11-20T11:47:35.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.193+0000 7fd4767fc700 1 -- 172.21.15.23:0/2579980025 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46c0057c0 con 0x7fd464004520 2020-11-20T11:47:35.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.194+0000 7fd47dc29700 1 -- 172.21.15.23:0/2579980025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd464004520 msgr2=0x7fd4640049a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:35.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.194+0000 7fd47dc29700 1 --2- 172.21.15.23:0/2579980025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd464004520 0x7fd4640049a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fd46c01d450 tx=0x7fd4780a6810).stop 2020-11-20T11:47:35.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.194+0000 7fd47dc29700 1 -- 172.21.15.23:0/2579980025 shutdown_connections 2020-11-20T11:47:35.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.194+0000 7fd47dc29700 1 --2- 172.21.15.23:0/2579980025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd464004520 0x7fd4640049a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:35.211 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.194+0000 7fd47dc29700 1 -- 172.21.15.23:0/2579980025 >> 172.21.15.23:0/2579980025 conn(0x7fd478060c80 msgr2=0x7fd4780acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:47:35.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.196+0000 7fd47dc29700 1 -- 172.21.15.23:0/2579980025 shutdown_connections 2020-11-20T11:47:35.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.196+0000 7fd47dc29700 1 -- 172.21.15.23:0/2579980025 wait complete. 2020-11-20T11:47:35.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.197+0000 7fd47dc29700 1 Processor -- start 2020-11-20T11:47:35.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.197+0000 7fd47dc29700 1 -- start start 2020-11-20T11:47:35.212 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.198+0000 7fd47dc29700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd464004520 0x7fd47809e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.198+0000 7fd47dc29700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd478068010 0x7fd478189bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.198+0000 7fd47dc29700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd47809faf0 0x7fd478132d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.199+0000 7fd47dc29700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47818ab40 con 0x7fd47809faf0 2020-11-20T11:47:35.213 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.199+0000 7fd47dc29700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47809da50 con 0x7fd478068010 2020-11-20T11:47:35.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.199+0000 7fd47dc29700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd47809dcf0 con 0x7fd464004520 2020-11-20T11:47:35.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.200+0000 7fd4777fe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd464004520 0x7fd47809e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:35.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.201+0000 7fd4777fe700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd464004520 0x7fd47809e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:56812/0 (socket says 172.21.15.23:56812) 2020-11-20T11:47:35.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.201+0000 7fd4777fe700 1 -- 172.21.15.23:0/3774498064 learned_addr learned my addr 172.21.15.23:0/3774498064 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:47:35.214 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.201+0000 7fd4777fe700 1 -- 172.21.15.23:0/3774498064 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd478068010 msgr2=0x7fd478189bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:47:35.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.201+0000 7fd4777fe700 1 --2- 172.21.15.23:0/3774498064 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd478068010 0x7fd478189bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:47:35.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.201+0000 7fd4777fe700 1 -- 172.21.15.23:0/3774498064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd47809faf0 msgr2=0x7fd478132d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:35.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.201+0000 7fd4777fe700 1 --2- 172.21.15.23:0/3774498064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd47809faf0 0x7fd478132d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:47:35.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.201+0000 7fd4777fe700 1 -- 172.21.15.23:0/3774498064 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd47809df90 con 0x7fd464004520 2020-11-20T11:47:35.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.201+0000 7fd4777fe700 1 --2- 172.21.15.23:0/3774498064 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd464004520 0x7fd47809e430 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fd46800dad0 tx=0x7fd468009040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:35.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.202+0000 7fd45f7fe700 1 -- 172.21.15.23:0/3774498064 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd468005bc0 con 0x7fd464004520 2020-11-20T11:47:35.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.202+0000 7fd47dc29700 1 -- 172.21.15.23:0/3774498064 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd47818a100 con 0x7fd464004520 2020-11-20T11:47:35.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.202+0000 7fd47dc29700 1 -- 172.21.15.23:0/3774498064 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd47809ff30 con 0x7fd464004520 2020-11-20T11:47:35.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.202+0000 7fd45f7fe700 1 -- 172.21.15.23:0/3774498064 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd468005d20 con 0x7fd464004520 2020-11-20T11:47:35.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.202+0000 7fd45f7fe700 1 -- 172.21.15.23:0/3774498064 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd468007320 con 0x7fd464004520 2020-11-20T11:47:35.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.203+0000 7fd47dc29700 1 --2- 172.21.15.23:0/3774498064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd478133210 0x7fd47818d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.204+0000 7fd47dc29700 1 -- 172.21.15.23:0/3774498064 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47809e190 con 0x7fd478133210 2020-11-20T11:47:35.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.204+0000 7fd45f7fe700 1 -- 172.21.15.23:0/3774498064 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd468007710 con 0x7fd464004520 2020-11-20T11:47:35.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.204+0000 7fd476ffd700 1 --2- 172.21.15.23:0/3774498064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd478133210 0x7fd47818d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:35.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.204+0000 7fd476ffd700 1 -- 172.21.15.23:0/3774498064 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd47805c520 con 0x7fd478133210 2020-11-20T11:47:35.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.204+0000 7fd476ffd700 1 --2- 172.21.15.23:0/3774498064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd478133210 0x7fd47818d100 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fd46c01d450 tx=0x7fd4780a6810).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:35.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.204+0000 7fd45f7fe700 1 --2- 172.21.15.23:0/3774498064 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd44c056da0 0x7fd44c059260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.204+0000 7fd45f7fe700 1 -- 172.21.15.23:0/3774498064 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fd46806fd90 con 0x7fd464004520 2020-11-20T11:47:35.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.205+0000 7fd45f7fe700 1 -- 172.21.15.23:0/3774498064 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46c00e070 con 0x7fd478133210 2020-11-20T11:47:35.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.205+0000 7fd45f7fe700 1 -- 172.21.15.23:0/3774498064 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fd46c0011d0 con 0x7fd478133210 2020-11-20T11:47:35.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.205+0000 7fd45f7fe700 1 -- 172.21.15.23:0/3774498064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd478133210 msgr2=0x7fd47818d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:35.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.205+0000 7fd45f7fe700 1 --2- 172.21.15.23:0/3774498064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd478133210 0x7fd47818d100 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fd46c01d450 tx=0x7fd4780a6810).stop 2020-11-20T11:47:35.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.205+0000 7fd45f7fe700 1 -- 172.21.15.23:0/3774498064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd478133210 msgr2=0x7fd47818d100 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:47:35.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.205+0000 7fd45f7fe700 1 --2- 172.21.15.23:0/3774498064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd478133210 0x7fd47818d100 secure :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0x7fd46c01d450 tx=0x7fd4780a6810).stop 2020-11-20T11:47:35.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.209+0000 7fd477fff700 1 --2- 172.21.15.23:0/3774498064 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd44c056da0 0x7fd44c059260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:35.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.209+0000 7fd477fff700 1 --2- 172.21.15.23:0/3774498064 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd44c056da0 0x7fd44c059260 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7fd478062c60 tx=0x7fd470008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:35.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.218+0000 7fd47dc29700 1 --2- 172.21.15.23:0/3774498064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd47818d5f0 0x7fd47818da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.218+0000 7fd47dc29700 1 -- 172.21.15.23:0/3774498064 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47809d710 con 0x7fd47818d5f0 2020-11-20T11:47:35.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.218+0000 7fd476ffd700 1 --2- 172.21.15.23:0/3774498064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd47818d5f0 0x7fd47818da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:35.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.218+0000 7fd476ffd700 1 -- 172.21.15.23:0/3774498064 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd478044710 con 0x7fd47818d5f0 2020-11-20T11:47:35.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.218+0000 7fd476ffd700 1 --2- 172.21.15.23:0/3774498064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd47818d5f0 0x7fd47818da00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fd46c008000 tx=0x7fd46c00e000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:35.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.219+0000 7fd45f7fe700 1 -- 172.21.15.23:0/3774498064 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46c014720 con 0x7fd47818d5f0 2020-11-20T11:47:35.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.219+0000 7fd45f7fe700 1 -- 172.21.15.23:0/3774498064 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4786 (secure 0 0 0) 0x7fd46c01d030 con 0x7fd47818d5f0 2020-11-20T11:47:35.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.219+0000 7fd45f7fe700 1 -- 172.21.15.23:0/3774498064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd47818d5f0 msgr2=0x7fd47818da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:35.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.219+0000 7fd45f7fe700 1 --2- 172.21.15.23:0/3774498064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd47818d5f0 0x7fd47818da00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fd46c008000 tx=0x7fd46c00e000).stop 2020-11-20T11:47:35.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.219+0000 7fd45f7fe700 1 -- 172.21.15.23:0/3774498064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd47818d5f0 msgr2=0x7fd47818da00 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:47:35.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.219+0000 7fd45f7fe700 1 --2- 172.21.15.23:0/3774498064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd47818d5f0 0x7fd47818da00 secure :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0x7fd46c008000 tx=0x7fd46c00e000).stop 2020-11-20T11:47:35.225 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:47:35.226 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:47:35.226 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:47:35.226 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:47:35.226 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 138, 2020-11-20T11:47:35.226 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:47:35.227 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:47:35.227 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:47:35.227 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:47:35.227 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:35.227 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 1, 2020-11-20T11:47:35.228 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:47:35.228 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:47:35.228 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:47:35.228 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:47:35.228 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:47:35.229 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:47:35.229 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:47:35.229 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:47:35.229 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:47:35.229 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:47:35.230 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:47:35.230 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:35.230 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:47:35.230 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:47:35.230 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:47:35.231 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:47:35.231 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:47:35.231 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:47:35.231 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:47:35.231 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:47:35.232 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:47:35.232 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:47:35.232 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:35.232 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.232 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:47:35.232 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:47:35.233 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:47:35.233 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:47:35.233 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:47:35.233 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:47:35.234 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:47:35.234 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:47:35.234 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:47:35.234 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:47:35.234 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:47:35.234 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:47:35.235 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:47:35.235 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:47:35.235 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:47:35.235 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:47:35.236 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:47:35.236 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:47:35.236 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:47:35.236 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:47:35.236 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:47:35.236 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:47:35.237 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:47:35.237 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:35.237 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:47:35.237 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.237 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:47:35.238 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.238 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:47:35.238 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:47:35.238 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:47:35.238 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:47:35.239 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.239 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:47:35.239 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:47:35.239 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:35.239 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.240 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.240 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:47:35.240 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:47:35.240 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:35.240 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:35.241 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:35.241 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.241 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:47:35.241 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:47:35.241 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:47:35.242 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:47:35.242 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:47:35.242 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.242 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.242 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:47:35.243 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:47:35.243 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:47:35.243 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:47:35.243 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.243 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:47:35.244 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:47:35.244 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:35.244 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.244 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.244 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:47:35.245 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:47:35.245 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:35.245 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:35.245 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:35.245 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.246 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:47:35.246 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:47:35.246 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:47:35.246 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:47:35.246 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:47:35.247 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.247 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.247 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:47:35.247 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:47:35.248 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:47:35.248 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:47:35.248 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.248 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:47:35.248 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:47:35.249 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:35.249 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.249 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.249 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:47:35.249 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:47:35.250 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:35.250 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:35.250 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:35.250 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.250 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:47:35.251 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:47:35.251 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:47:35.251 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:47:35.251 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:47:35.252 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:35.252 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:35.252 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.252 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:47:35.252 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:47:35.253 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.253 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:35.253 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:35.253 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:47:35.253 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:35.254 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:35.254 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:47:35.254 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.254 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:35.254 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:35.255 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:47:35.255 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:35.255 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:35.255 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:47:35.255 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.255 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:35.256 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:35.256 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:47:35.256 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:35.256 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:35.256 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:47:35.257 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.257 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:35.257 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:35.257 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:47:35.257 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:35.258 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:35.258 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:47:35.258 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.258 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:35.258 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:35.259 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:47:35.259 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:35.259 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:35.259 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.259 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:47:35.260 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:47:35.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.220+0000 7fd47dc29700 1 -- 172.21.15.23:0/3774498064 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd44c056da0 msgr2=0x7fd44c059260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:35.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.220+0000 7fd47dc29700 1 --2- 172.21.15.23:0/3774498064 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd44c056da0 0x7fd44c059260 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7fd478062c60 tx=0x7fd470008040).stop 2020-11-20T11:47:35.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.220+0000 7fd47dc29700 1 -- 172.21.15.23:0/3774498064 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd464004520 msgr2=0x7fd47809e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:35.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.220+0000 7fd47dc29700 1 --2- 172.21.15.23:0/3774498064 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd464004520 0x7fd47809e430 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fd46800dad0 tx=0x7fd468009040).stop 2020-11-20T11:47:35.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.220+0000 7fd477fff700 1 -- 172.21.15.23:0/3774498064 reap_dead start 2020-11-20T11:47:35.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.220+0000 7fd477fff700 1 -- 172.21.15.23:0/3774498064 reap_dead start 2020-11-20T11:47:35.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.222+0000 7fd47dc29700 1 -- 172.21.15.23:0/3774498064 shutdown_connections 2020-11-20T11:47:35.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.222+0000 7fd47dc29700 1 -- 172.21.15.23:0/3774498064 >> 172.21.15.23:0/3774498064 conn(0x7fd478060c80 msgr2=0x7fd478061850 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:47:35.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.222+0000 7fd47dc29700 1 -- 172.21.15.23:0/3774498064 shutdown_connections 2020-11-20T11:47:35.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.222+0000 7fd47dc29700 1 -- 172.21.15.23:0/3774498064 wait complete. 2020-11-20T11:47:35.263 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:47:35.431 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.428+0000 7f08adcdf700 1 Processor -- start 2020-11-20T11:47:35.431 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.428+0000 7f08adcdf700 1 -- start start 2020-11-20T11:47:35.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.429+0000 7f08adcdf700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f08a80a2080 0x7f08a80aef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.429+0000 7f08adcdf700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f08a8067a10 0x7f08a8067e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.429+0000 7f08adcdf700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a80aaab0 con 0x7f08a80a2460 2020-11-20T11:47:35.434 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.429+0000 7f08adcdf700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08a813aa50 con 0x7f08a80a2080 2020-11-20T11:47:35.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.429+0000 7f08adcdf700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08a80aad50 con 0x7f08a8067a10 2020-11-20T11:47:35.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.429+0000 7f08accdd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f08a80a2080 0x7f08a80aef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:35.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.429+0000 7f08accdd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f08a80a2080 0x7f08a80aef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46582/0 (socket says 172.21.15.23:46582) 2020-11-20T11:47:35.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.429+0000 7f08accdd700 1 -- 172.21.15.23:0/3754621313 learned_addr learned my addr 172.21.15.23:0/3754621313 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:47:35.435 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.429+0000 7f08ad4de700 1 --2- 172.21.15.23:0/3754621313 >> v2:172.21.15.155:3300/0 conn(0x7f08a8067a10 0x7f08a8067e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:35.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.429+0000 7f08accdd700 1 -- 172.21.15.23:0/3754621313 >> v2:172.21.15.155:3300/0 conn(0x7f08a8067a10 msgr2=0x7f08a8067e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:47:35.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.429+0000 7f08accdd700 1 --2- 172.21.15.23:0/3754621313 >> v2:172.21.15.155:3300/0 conn(0x7f08a8067a10 0x7f08a8067e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:35.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.429+0000 7f08accdd700 1 -- 172.21.15.23:0/3754621313 >> v1:172.21.15.155:6789/0 conn(0x7f08a80a2460 legacy=0x7f08a8137df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:35.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.429+0000 7f08accdd700 1 -- 172.21.15.23:0/3754621313 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a813b920 con 0x7f08a80a2080 2020-11-20T11:47:35.436 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.429+0000 7f08accdd700 1 --2- 172.21.15.23:0/3754621313 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08a80a2080 0x7f08a80aef10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f08a8063640 tx=0x7f089c006e90).ready entity=mon.0 client_cookie=4bc7ade615b64ad0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:35.437 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.430+0000 7f08a77fe700 1 -- 172.21.15.23:0/3754621313 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089c006030 con 0x7f08a80a2080 2020-11-20T11:47:35.437 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.430+0000 7f08a77fe700 1 -- 172.21.15.23:0/3754621313 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f089c006190 con 0x7f08a80a2080 2020-11-20T11:47:35.437 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.430+0000 7f08a77fe700 1 -- 172.21.15.23:0/3754621313 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089c006510 con 0x7f08a80a2080 2020-11-20T11:47:35.437 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.430+0000 7f08adcdf700 1 -- 172.21.15.23:0/3754621313 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08a80a2080 msgr2=0x7f08a80aef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:35.437 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.430+0000 7f08adcdf700 1 --2- 172.21.15.23:0/3754621313 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08a80a2080 0x7f08a80aef10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f08a8063640 tx=0x7f089c006e90).stop 2020-11-20T11:47:35.438 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.430+0000 7f08adcdf700 1 -- 172.21.15.23:0/3754621313 shutdown_connections 2020-11-20T11:47:35.438 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.430+0000 7f08adcdf700 1 --2- 172.21.15.23:0/3754621313 >> v2:172.21.15.155:3300/0 conn(0x7f08a8067a10 0x7f08a8067e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:35.438 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.430+0000 7f08adcdf700 1 --2- 172.21.15.23:0/3754621313 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08a80a2080 0x7f08a80aef10 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:35.438 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.430+0000 7f08adcdf700 1 -- 172.21.15.23:0/3754621313 >> 172.21.15.23:0/3754621313 conn(0x7f08a8063980 msgr2=0x7f08a8063d90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:47:35.438 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.430+0000 7f08adcdf700 1 -- 172.21.15.23:0/3754621313 shutdown_connections 2020-11-20T11:47:35.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.430+0000 7f08adcdf700 1 -- 172.21.15.23:0/3754621313 wait complete. 2020-11-20T11:47:35.439 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.431+0000 7f08adcdf700 1 Processor -- start 2020-11-20T11:47:35.442 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.431+0000 7f08adcdf700 1 -- start start 2020-11-20T11:47:35.442 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.432+0000 7f08adcdf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08a8067a10 0x7f08a80ae7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.442 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.432+0000 7f08adcdf700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08a80a2080 0x7f08a80a4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.432+0000 7f08adcdf700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08a80a2460 0x7f08a814d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.432+0000 7f08adcdf700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a80a57e0 con 0x7f08a8067a10 2020-11-20T11:47:35.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.432+0000 7f08adcdf700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a80a5490 con 0x7f08a80a2460 2020-11-20T11:47:35.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.432+0000 7f08adcdf700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08a80a30d0 con 0x7f08a80a2080 2020-11-20T11:47:35.443 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.432+0000 7f08accdd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08a8067a10 0x7f08a80ae7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:35.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.432+0000 7f08accdd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08a8067a10 0x7f08a80ae7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46588/0 (socket says 172.21.15.23:46588) 2020-11-20T11:47:35.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.432+0000 7f08accdd700 1 -- 172.21.15.23:0/2541169399 learned_addr learned my addr 172.21.15.23:0/2541169399 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:47:35.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.432+0000 7f08ad4de700 1 --2- 172.21.15.23:0/2541169399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08a80a2460 0x7f08a814d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:35.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.432+0000 7f08a7fff700 1 --2- 172.21.15.23:0/2541169399 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08a80a2080 0x7f08a80a4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:35.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.432+0000 7f08accdd700 1 -- 172.21.15.23:0/2541169399 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08a80a2080 msgr2=0x7f08a80a4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:35.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.432+0000 7f08accdd700 1 --2- 172.21.15.23:0/2541169399 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08a80a2080 0x7f08a80a4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:35.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.432+0000 7f08accdd700 1 -- 172.21.15.23:0/2541169399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08a80a2460 msgr2=0x7f08a814d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:35.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.432+0000 7f08accdd700 1 --2- 172.21.15.23:0/2541169399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08a80a2460 0x7f08a814d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:35.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.432+0000 7f08accdd700 1 -- 172.21.15.23:0/2541169399 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a80a3370 con 0x7f08a8067a10 2020-11-20T11:47:35.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.432+0000 7f08accdd700 1 --2- 172.21.15.23:0/2541169399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08a8067a10 0x7f08a80ae7e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f089c006fd0 tx=0x7f089c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:35.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.433+0000 7f08a4ff9700 1 -- 172.21.15.23:0/2541169399 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089c005920 con 0x7f08a8067a10 2020-11-20T11:47:35.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.433+0000 7f08a4ff9700 1 -- 172.21.15.23:0/2541169399 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f089c005a80 con 0x7f08a8067a10 2020-11-20T11:47:35.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.433+0000 7f08adcdf700 1 -- 172.21.15.23:0/2541169399 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08a80a5190 con 0x7f08a8067a10 2020-11-20T11:47:35.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.433+0000 7f08a4ff9700 1 -- 172.21.15.23:0/2541169399 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089c005e20 con 0x7f08a8067a10 2020-11-20T11:47:35.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.433+0000 7f08adcdf700 1 -- 172.21.15.23:0/2541169399 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08a80a3810 con 0x7f08a8067a10 2020-11-20T11:47:35.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.433+0000 7f08adcdf700 1 --2- 172.21.15.23:0/2541169399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08a8137310 0x7f08a81376f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.433+0000 7f08adcdf700 1 -- 172.21.15.23:0/2541169399 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a80a5350 con 0x7f08a8137310 2020-11-20T11:47:35.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.434+0000 7f08a7fff700 1 --2- 172.21.15.23:0/2541169399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08a8137310 0x7f08a81376f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:35.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.434+0000 7f08a4ff9700 1 -- 172.21.15.23:0/2541169399 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f089c025030 con 0x7f08a8067a10 2020-11-20T11:47:35.448 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.434+0000 7f08a7fff700 1 -- 172.21.15.23:0/2541169399 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f08a8057dc0 con 0x7f08a8137310 2020-11-20T11:47:35.448 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.434+0000 7f08a7fff700 1 --2- 172.21.15.23:0/2541169399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08a8137310 0x7f08a81376f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f08a80615a0 tx=0x7f0894007bd0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:35.448 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.434+0000 7f08a4ff9700 1 --2- 172.21.15.23:0/2541169399 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0880056d90 0x7f0880059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.448 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.434+0000 7f08a4ff9700 1 -- 172.21.15.23:0/2541169399 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f089c0714e0 con 0x7f08a8067a10 2020-11-20T11:47:35.448 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.434+0000 7f08ad4de700 1 --2- 172.21.15.23:0/2541169399 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0880056d90 0x7f0880059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:35.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.434+0000 7f08a4ff9700 1 -- 172.21.15.23:0/2541169399 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0894011590 con 0x7f08a8137310 2020-11-20T11:47:35.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.434+0000 7f08a4ff9700 1 -- 172.21.15.23:0/2541169399 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f08940116f0 con 0x7f08a8137310 2020-11-20T11:47:35.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.434+0000 7f08a4ff9700 1 -- 172.21.15.23:0/2541169399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08a8137310 msgr2=0x7f08a81376f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:35.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.434+0000 7f08a4ff9700 1 --2- 172.21.15.23:0/2541169399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08a8137310 0x7f08a81376f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f08a80615a0 tx=0x7f0894007bd0).stop 2020-11-20T11:47:35.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.435+0000 7f08a4ff9700 1 -- 172.21.15.23:0/2541169399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08a8137310 msgr2=0x7f08a81376f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:47:35.450 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.435+0000 7f08a4ff9700 1 --2- 172.21.15.23:0/2541169399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08a8137310 0x7f08a81376f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:35.450 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.435+0000 7f08ad4de700 1 --2- 172.21.15.23:0/2541169399 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0880056d90 0x7f0880059250 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f08a0009e50 tx=0x7f08a0009220).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:35.451 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.441+0000 7f08adcdf700 1 --2- 172.21.15.23:0/2541169399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08a814a050 0x7f08a814a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.441+0000 7f08adcdf700 1 -- 172.21.15.23:0/2541169399 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a8137be0 con 0x7f08a814a050 2020-11-20T11:47:35.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.442+0000 7f08a7fff700 1 --2- 172.21.15.23:0/2541169399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08a814a050 0x7f08a814a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:35.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.442+0000 7f08a7fff700 1 -- 172.21.15.23:0/2541169399 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f08a8044710 con 0x7f08a814a050 2020-11-20T11:47:35.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.442+0000 7f08a7fff700 1 --2- 172.21.15.23:0/2541169399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08a814a050 0x7f08a814a460 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f0894015000 tx=0x7f0894018000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:35.453 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.442+0000 7f08a4ff9700 1 -- 172.21.15.23:0/2541169399 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0894007240 con 0x7f08a814a050 2020-11-20T11:47:35.453 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.442+0000 7f08a4ff9700 1 -- 172.21.15.23:0/2541169399 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4274 (secure 0 0 0) 0x7f0894011380 con 0x7f08a814a050 2020-11-20T11:47:35.453 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.442+0000 7f08a4ff9700 1 -- 172.21.15.23:0/2541169399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08a814a050 msgr2=0x7f08a814a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:35.453 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.442+0000 7f08a4ff9700 1 --2- 172.21.15.23:0/2541169399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08a814a050 0x7f08a814a460 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f0894015000 tx=0x7f0894018000).stop 2020-11-20T11:47:35.453 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.442+0000 7f08a4ff9700 1 -- 172.21.15.23:0/2541169399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08a814a050 msgr2=0x7f08a814a460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:47:35.454 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.442+0000 7f08a4ff9700 1 --2- 172.21.15.23:0/2541169399 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f08a814a050 0x7f08a814a460 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:35.454 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.443+0000 7f08adcdf700 1 -- 172.21.15.23:0/2541169399 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0880056d90 msgr2=0x7f0880059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:35.454 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.443+0000 7f08adcdf700 1 --2- 172.21.15.23:0/2541169399 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0880056d90 0x7f0880059250 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f08a0009e50 tx=0x7f08a0009220).stop 2020-11-20T11:47:35.454 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.443+0000 7f08adcdf700 1 -- 172.21.15.23:0/2541169399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08a8067a10 msgr2=0x7f08a80ae7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:35.454 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.443+0000 7f08adcdf700 1 --2- 172.21.15.23:0/2541169399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08a8067a10 0x7f08a80ae7e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f089c006fd0 tx=0x7f089c015000).stop 2020-11-20T11:47:35.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.443+0000 7f08ad4de700 1 -- 172.21.15.23:0/2541169399 reap_dead start 2020-11-20T11:47:35.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.443+0000 7f08ad4de700 1 -- 172.21.15.23:0/2541169399 reap_dead start 2020-11-20T11:47:35.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.443+0000 7f08adcdf700 1 -- 172.21.15.23:0/2541169399 shutdown_connections 2020-11-20T11:47:35.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.443+0000 7f08adcdf700 1 -- 172.21.15.23:0/2541169399 >> 172.21.15.23:0/2541169399 conn(0x7f08a8063980 msgr2=0x7f08a8068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:47:35.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.443+0000 7f08adcdf700 1 -- 172.21.15.23:0/2541169399 shutdown_connections 2020-11-20T11:47:35.456 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.443+0000 7f08adcdf700 1 -- 172.21.15.23:0/2541169399 wait complete. 2020-11-20T11:47:35.456 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:47:35.456 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:47:35.456 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:47:35.456 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:47:35.457 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 138, 2020-11-20T11:47:35.457 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:47:35.457 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:47:35.457 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:47:35.458 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:47:35.458 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:35.458 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 1, 2020-11-20T11:47:35.458 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:47:35.458 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:47:35.459 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:47:35.459 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:47:35.459 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:47:35.459 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:47:35.459 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:47:35.459 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:47:35.460 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:47:35.460 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:47:35.460 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:47:35.460 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:35.461 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:47:35.461 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:47:35.461 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:47:35.461 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:47:35.461 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:47:35.462 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:47:35.462 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:47:35.462 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:47:35.462 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:47:35.462 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:47:35.462 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:35.463 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.463 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:47:35.463 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:47:35.463 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:47:35.463 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:47:35.464 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:47:35.464 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:47:35.464 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:47:35.464 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:47:35.464 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:47:35.465 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:47:35.465 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:47:35.465 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:47:35.465 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:47:35.465 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:47:35.466 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:47:35.466 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:47:35.466 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:47:35.466 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:47:35.466 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:47:35.467 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:47:35.467 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:47:35.467 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:47:35.467 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:47:35.467 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:35.467 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:47:35.468 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.468 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:47:35.468 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.468 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:47:35.468 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:47:35.469 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:47:35.469 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:47:35.469 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.469 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:47:35.469 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:47:35.470 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:35.470 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.470 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.470 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:47:35.470 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:47:35.471 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:35.471 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:35.471 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:35.471 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.471 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:47:35.472 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:47:35.472 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:47:35.472 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:47:35.472 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:47:35.472 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.473 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.473 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:47:35.473 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:47:35.473 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:47:35.473 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:47:35.474 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.474 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:47:35.474 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:47:35.474 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:35.474 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.475 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.475 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:47:35.475 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:47:35.475 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:35.475 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:35.476 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:35.476 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.476 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:47:35.476 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:47:35.476 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:47:35.477 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:47:35.477 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:47:35.477 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.477 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.477 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:47:35.478 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:47:35.478 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:47:35.478 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:47:35.478 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.478 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:47:35.479 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:47:35.479 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:35.479 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.479 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.479 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:47:35.480 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:47:35.480 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:35.480 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:35.480 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:35.480 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.481 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:47:35.481 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:47:35.481 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:47:35.481 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:47:35.481 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:47:35.481 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:35.482 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:35.482 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.482 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:47:35.482 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:47:35.482 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.483 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:35.483 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:35.483 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:47:35.483 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:35.483 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:35.484 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:47:35.484 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.484 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:35.484 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:35.484 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:47:35.485 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:35.485 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:35.485 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.485 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:47:35.485 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:47:35.486 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:47:35.624 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.620+0000 7fd6f5705700 1 Processor -- start 2020-11-20T11:47:35.624 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.621+0000 7fd6f5705700 1 -- start start 2020-11-20T11:47:35.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.621+0000 7fd6f5705700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6f00aaab0 0x7fd6f013ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.621+0000 7fd6f5705700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd6f013aeb0 0x7fd6f00a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.621+0000 7fd6f5705700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f01388b0 con 0x7fd6f00aae90 2020-11-20T11:47:35.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.621+0000 7fd6f5705700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6f013e600 con 0x7fd6f013aeb0 2020-11-20T11:47:35.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.621+0000 7fd6f5705700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f013e8a0 con 0x7fd6f00aaab0 2020-11-20T11:47:35.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.622+0000 7fd6eeffd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6f00aaab0 0x7fd6f013ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:35.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.622+0000 7fd6ef7fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd6f013aeb0 0x7fd6f00a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:35.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.622+0000 7fd6ee7fc700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fd6f00aae90 0x7fd6f01381a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:38824/0 (socket says 172.21.15.23:38824) 2020-11-20T11:47:35.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.622+0000 7fd6ee7fc700 1 -- 172.21.15.23:0/1535945133 learned_addr learned my addr 172.21.15.23:0/1535945133 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:47:35.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.622+0000 7fd6ef7fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd6f013aeb0 0x7fd6f00a8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:39500/0 (socket says 172.21.15.23:39500) 2020-11-20T11:47:35.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.622+0000 7fd6eeffd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6f00aaab0 0x7fd6f013ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:56840/0 (socket says 172.21.15.23:56840) 2020-11-20T11:47:35.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.622+0000 7fd6eeffd700 1 -- 172.21.15.23:0/1535945133 >> v2:172.21.15.155:3300/0 conn(0x7fd6f013aeb0 msgr2=0x7fd6f00a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:47:35.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.622+0000 7fd6eeffd700 1 --2- 172.21.15.23:0/1535945133 >> v2:172.21.15.155:3300/0 conn(0x7fd6f013aeb0 0x7fd6f00a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:35.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.622+0000 7fd6eeffd700 1 -- 172.21.15.23:0/1535945133 >> v1:172.21.15.155:6789/0 conn(0x7fd6f00aae90 legacy=0x7fd6f01381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:35.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.622+0000 7fd6eeffd700 1 -- 172.21.15.23:0/1535945133 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f00ab6d0 con 0x7fd6f00aaab0 2020-11-20T11:47:35.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.622+0000 7fd6edffb700 1 -- 172.21.15.23:0/1535945133 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3982707339 0 0) 0x7fd6f01388b0 con 0x7fd6f00aae90 2020-11-20T11:47:35.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.622+0000 7fd6ef7fe700 1 --2- 172.21.15.23:0/1535945133 >> v2:172.21.15.155:3300/0 conn(0x7fd6f013aeb0 0x7fd6f00a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:47:35.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.622+0000 7fd6eeffd700 1 --2- 172.21.15.23:0/1535945133 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6f00aaab0 0x7fd6f013ed40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fd6e0006fd0 tx=0x7fd6e0006e30).ready entity=mon.2 client_cookie=fc7252eef9801b2b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:35.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.622+0000 7fd6edffb700 1 -- 172.21.15.23:0/1535945133 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e00097e0 con 0x7fd6f00aaab0 2020-11-20T11:47:35.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.622+0000 7fd6edffb700 1 -- 172.21.15.23:0/1535945133 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6e0000e90 con 0x7fd6f00aaab0 2020-11-20T11:47:35.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.622+0000 7fd6edffb700 1 -- 172.21.15.23:0/1535945133 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e0009b50 con 0x7fd6f00aaab0 2020-11-20T11:47:35.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.623+0000 7fd6f5705700 1 -- 172.21.15.23:0/1535945133 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6f00aaab0 msgr2=0x7fd6f013ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:35.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.623+0000 7fd6f5705700 1 --2- 172.21.15.23:0/1535945133 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6f00aaab0 0x7fd6f013ed40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fd6e0006fd0 tx=0x7fd6e0006e30).stop 2020-11-20T11:47:35.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.623+0000 7fd6f5705700 1 -- 172.21.15.23:0/1535945133 shutdown_connections 2020-11-20T11:47:35.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.623+0000 7fd6f5705700 1 --2- 172.21.15.23:0/1535945133 >> v2:172.21.15.155:3300/0 conn(0x7fd6f013aeb0 0x7fd6f00a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:35.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.623+0000 7fd6f5705700 1 --2- 172.21.15.23:0/1535945133 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6f00aaab0 0x7fd6f013ed40 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:35.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.623+0000 7fd6f5705700 1 -- 172.21.15.23:0/1535945133 >> 172.21.15.23:0/1535945133 conn(0x7fd6f00678e0 msgr2=0x7fd6f0067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:47:35.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.623+0000 7fd6f5705700 1 -- 172.21.15.23:0/1535945133 shutdown_connections 2020-11-20T11:47:35.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.623+0000 7fd6f5705700 1 -- 172.21.15.23:0/1535945133 wait complete. 2020-11-20T11:47:35.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.624+0000 7fd6f5705700 1 Processor -- start 2020-11-20T11:47:35.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.624+0000 7fd6f5705700 1 -- start start 2020-11-20T11:47:35.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.625+0000 7fd6f5705700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6f00aaab0 0x7fd6f013c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.625+0000 7fd6f5705700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6f00aae90 0x7fd6f00b03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.625+0000 7fd6f5705700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6f00a5640 0x7fd6f014dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.625+0000 7fd6f5705700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f0138730 con 0x7fd6f00a5640 2020-11-20T11:47:35.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.625+0000 7fd6f5705700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f013cc30 con 0x7fd6f00aae90 2020-11-20T11:47:35.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.625+0000 7fd6f5705700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f013ced0 con 0x7fd6f00aaab0 2020-11-20T11:47:35.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.626+0000 7fd6ef7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6f00a5640 0x7fd6f014dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:35.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.626+0000 7fd6ee7fc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6f00aae90 0x7fd6f00b03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:35.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.626+0000 7fd6ef7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6f00a5640 0x7fd6f014dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46610/0 (socket says 172.21.15.23:46610) 2020-11-20T11:47:35.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.626+0000 7fd6ef7fe700 1 -- 172.21.15.23:0/557362973 learned_addr learned my addr 172.21.15.23:0/557362973 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:47:35.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.626+0000 7fd6eeffd700 1 --2- 172.21.15.23:0/557362973 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6f00aaab0 0x7fd6f013c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:35.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.626+0000 7fd6ef7fe700 1 -- 172.21.15.23:0/557362973 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6f00aaab0 msgr2=0x7fd6f013c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:35.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.626+0000 7fd6ef7fe700 1 --2- 172.21.15.23:0/557362973 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6f00aaab0 0x7fd6f013c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:35.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.626+0000 7fd6ef7fe700 1 -- 172.21.15.23:0/557362973 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6f00aae90 msgr2=0x7fd6f00b03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:35.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.626+0000 7fd6ef7fe700 1 --2- 172.21.15.23:0/557362973 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6f00aae90 0x7fd6f00b03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:35.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.626+0000 7fd6ef7fe700 1 -- 172.21.15.23:0/557362973 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f014d7e0 con 0x7fd6f00a5640 2020-11-20T11:47:35.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.626+0000 7fd6ee7fc700 1 --2- 172.21.15.23:0/557362973 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd6f00aae90 0x7fd6f00b03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:47:35.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.626+0000 7fd6ef7fe700 1 --2- 172.21.15.23:0/557362973 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6f00a5640 0x7fd6f014dc80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fd6e800ef50 tx=0x7fd6e8009220).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:35.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.626+0000 7fd6d6ffd700 1 -- 172.21.15.23:0/557362973 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e800cc10 con 0x7fd6f00a5640 2020-11-20T11:47:35.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.626+0000 7fd6d6ffd700 1 -- 172.21.15.23:0/557362973 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6e800cd70 con 0x7fd6f00a5640 2020-11-20T11:47:35.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.626+0000 7fd6d6ffd700 1 -- 172.21.15.23:0/557362973 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8012620 con 0x7fd6f00a5640 2020-11-20T11:47:35.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.626+0000 7fd6f5705700 1 -- 172.21.15.23:0/557362973 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6f014e170 con 0x7fd6f00a5640 2020-11-20T11:47:35.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.626+0000 7fd6f5705700 1 -- 172.21.15.23:0/557362973 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6f01335b0 con 0x7fd6f00a5640 2020-11-20T11:47:35.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.630+0000 7fd6d6ffd700 1 -- 172.21.15.23:0/557362973 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd6e8022070 con 0x7fd6f00a5640 2020-11-20T11:47:35.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.631+0000 7fd6d6ffd700 1 --2- 172.21.15.23:0/557362973 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd6cc056d90 0x7fd6cc059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.631+0000 7fd6d6ffd700 1 -- 172.21.15.23:0/557362973 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fd6e80709f0 con 0x7fd6f00a5640 2020-11-20T11:47:35.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.631+0000 7fd6eeffd700 1 --2- 172.21.15.23:0/557362973 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd6cc056d90 0x7fd6cc059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:35.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.631+0000 7fd6f5705700 1 --2- 172.21.15.23:0/557362973 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6f013aeb0 0x7fd6f0133890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.631+0000 7fd6f5705700 1 -- 172.21.15.23:0/557362973 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f014e660 con 0x7fd6f013aeb0 2020-11-20T11:47:35.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.632+0000 7fd6ee7fc700 1 --2- 172.21.15.23:0/557362973 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6f013aeb0 0x7fd6f0133890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:35.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.632+0000 7fd6eeffd700 1 --2- 172.21.15.23:0/557362973 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd6cc056d90 0x7fd6cc059250 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fd6e0006e70 tx=0x7fd6e0015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:35.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.636+0000 7fd6ee7fc700 1 -- 172.21.15.23:0/557362973 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd6f013ca80 con 0x7fd6f013aeb0 2020-11-20T11:47:35.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.637+0000 7fd6ee7fc700 1 --2- 172.21.15.23:0/557362973 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6f013aeb0 0x7fd6f0133890 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fd6f00b12a0 tx=0x7fd6e4007a60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:35.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.637+0000 7fd6d6ffd700 1 -- 172.21.15.23:0/557362973 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e400fa00 con 0x7fd6f013aeb0 2020-11-20T11:47:35.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.637+0000 7fd6d6ffd700 1 -- 172.21.15.23:0/557362973 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fd6e400fb60 con 0x7fd6f013aeb0 2020-11-20T11:47:35.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.637+0000 7fd6d6ffd700 1 -- 172.21.15.23:0/557362973 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6f013aeb0 msgr2=0x7fd6f0133890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:35.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.637+0000 7fd6d6ffd700 1 --2- 172.21.15.23:0/557362973 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6f013aeb0 0x7fd6f0133890 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fd6f00b12a0 tx=0x7fd6e4007a60).stop 2020-11-20T11:47:35.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.637+0000 7fd6d6ffd700 1 -- 172.21.15.23:0/557362973 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6f013aeb0 msgr2=0x7fd6f0133890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:47:35.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.637+0000 7fd6d6ffd700 1 --2- 172.21.15.23:0/557362973 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6f013aeb0 0x7fd6f0133890 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:35.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.644+0000 7fd6f5705700 1 --2- 172.21.15.23:0/557362973 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6f0133d80 0x7fd6f014a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.644+0000 7fd6f5705700 1 -- 172.21.15.23:0/557362973 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f005c7d0 con 0x7fd6f0133d80 2020-11-20T11:47:35.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.644+0000 7fd6ee7fc700 1 --2- 172.21.15.23:0/557362973 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6f0133d80 0x7fd6f014a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:35.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.645+0000 7fd6ee7fc700 1 -- 172.21.15.23:0/557362973 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd6f014d970 con 0x7fd6f0133d80 2020-11-20T11:47:35.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.645+0000 7fd6ee7fc700 1 --2- 172.21.15.23:0/557362973 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6f0133d80 0x7fd6f014a230 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fd6e4013000 tx=0x7fd6e4019000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:35.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.645+0000 7fd6d6ffd700 1 -- 172.21.15.23:0/557362973 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e40136f0 con 0x7fd6f0133d80 2020-11-20T11:47:35.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.645+0000 7fd6d6ffd700 1 -- 172.21.15.23:0/557362973 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4784 (secure 0 0 0) 0x7fd6e400f7f0 con 0x7fd6f0133d80 2020-11-20T11:47:35.652 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.645+0000 7fd6d6ffd700 1 -- 172.21.15.23:0/557362973 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6f0133d80 msgr2=0x7fd6f014a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:35.652 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.645+0000 7fd6d6ffd700 1 --2- 172.21.15.23:0/557362973 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6f0133d80 0x7fd6f014a230 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fd6e4013000 tx=0x7fd6e4019000).stop 2020-11-20T11:47:35.652 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.645+0000 7fd6d6ffd700 1 -- 172.21.15.23:0/557362973 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6f0133d80 msgr2=0x7fd6f014a230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:47:35.653 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.645+0000 7fd6d6ffd700 1 --2- 172.21.15.23:0/557362973 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd6f0133d80 0x7fd6f014a230 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:35.653 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.646+0000 7fd6f5705700 1 -- 172.21.15.23:0/557362973 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd6cc056d90 msgr2=0x7fd6cc059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:35.654 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.646+0000 7fd6f5705700 1 --2- 172.21.15.23:0/557362973 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd6cc056d90 0x7fd6cc059250 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fd6e0006e70 tx=0x7fd6e0015000).stop 2020-11-20T11:47:35.654 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.646+0000 7fd6f5705700 1 -- 172.21.15.23:0/557362973 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6f00a5640 msgr2=0x7fd6f014dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:35.654 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.646+0000 7fd6f5705700 1 --2- 172.21.15.23:0/557362973 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6f00a5640 0x7fd6f014dc80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fd6e800ef50 tx=0x7fd6e8009220).stop 2020-11-20T11:47:35.654 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.646+0000 7fd6ef7fe700 1 -- 172.21.15.23:0/557362973 reap_dead start 2020-11-20T11:47:35.654 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.646+0000 7fd6f5705700 1 -- 172.21.15.23:0/557362973 shutdown_connections 2020-11-20T11:47:35.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.646+0000 7fd6f5705700 1 --2- 172.21.15.23:0/557362973 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6f00a5640 0x7fd6f014dc80 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:35.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.646+0000 7fd6f5705700 1 -- 172.21.15.23:0/557362973 >> 172.21.15.23:0/557362973 conn(0x7fd6f00678e0 msgr2=0x7fd6f00aba00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:47:35.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.646+0000 7fd6f5705700 1 -- 172.21.15.23:0/557362973 shutdown_connections 2020-11-20T11:47:35.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.646+0000 7fd6f5705700 1 -- 172.21.15.23:0/557362973 wait complete. 2020-11-20T11:47:35.656 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:47:35.656 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:47:35.657 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:47:35.657 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:47:35.657 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 138, 2020-11-20T11:47:35.658 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:47:35.658 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:47:35.658 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:47:35.659 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:47:35.659 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:35.659 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 2, 2020-11-20T11:47:35.659 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:47:35.660 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:47:35.660 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:47:35.660 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:47:35.660 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:47:35.660 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:47:35.661 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:47:35.661 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:47:35.661 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:47:35.662 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:47:35.662 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:47:35.662 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:35.662 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:47:35.663 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:47:35.663 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:47:35.663 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:47:35.663 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:47:35.663 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:47:35.664 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:47:35.664 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:47:35.664 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:47:35.664 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:47:35.664 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:35.665 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.665 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:47:35.665 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:47:35.665 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:47:35.665 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:47:35.666 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:47:35.666 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:47:35.666 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:47:35.666 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:47:35.666 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:47:35.667 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:47:35.667 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:47:35.667 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:47:35.667 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:47:35.667 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:47:35.668 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:47:35.668 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:47:35.668 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:47:35.668 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:47:35.669 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:47:35.669 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:47:35.669 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:47:35.669 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:47:35.669 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:47:35.670 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:35.670 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:47:35.670 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.671 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:47:35.671 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.671 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:47:35.671 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:47:35.672 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:47:35.672 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:47:35.672 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.672 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:47:35.673 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:47:35.673 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:35.673 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.673 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.674 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:47:35.674 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:47:35.674 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:35.674 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:35.675 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:35.675 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.675 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:47:35.675 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:47:35.675 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:47:35.676 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:47:35.676 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:47:35.676 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.676 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.677 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:47:35.677 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:47:35.677 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:47:35.677 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:47:35.677 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.678 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:47:35.678 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:47:35.678 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:35.678 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.678 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.679 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:47:35.679 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:47:35.679 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:35.679 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:35.680 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:35.680 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.680 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:47:35.680 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:47:35.681 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:47:35.681 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:47:35.681 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:47:35.681 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.682 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.682 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:47:35.682 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:47:35.682 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:47:35.683 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:47:35.683 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.683 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:47:35.683 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:47:35.684 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:35.684 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.684 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.684 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:47:35.685 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:47:35.685 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:35.685 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:35.685 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:35.686 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.686 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:47:35.686 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:47:35.686 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:47:35.686 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:47:35.687 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:47:35.687 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:35.687 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:35.687 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.687 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:47:35.688 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:47:35.688 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.688 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:35.688 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:35.688 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:47:35.689 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:35.689 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:35.689 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:47:35.689 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.689 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:35.690 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:35.690 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:47:35.690 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:35.690 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:35.690 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:47:35.691 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.691 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:35.691 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:35.691 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:47:35.692 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:35.692 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:35.692 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:47:35.693 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.693 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:35.693 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:35.694 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:47:35.694 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:35.694 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:35.694 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:47:35.694 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:35.695 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:35.695 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:35.695 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:47:35.695 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:35.696 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:35.696 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:35.696 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:47:35.696 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:47:35.697 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:35.629+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 198 slow ops, oldest is osd_op(client.4659.0:58398 4.2s0 4:56f73163:::10000000360.0000000d:head [write 618496~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:35.698 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2020-11-20T11:47:35.698 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2020-11-20T11:47:35.860 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.857+0000 7f7d47282700 1 Processor -- start 2020-11-20T11:47:35.860 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.857+0000 7f7d47282700 1 -- start start 2020-11-20T11:47:35.863 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.858+0000 7f7d47282700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f7d400a2460 0x7f7d40137df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.864 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.858+0000 7f7d47282700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d400aaab0 con 0x7f7d400a2080 2020-11-20T11:47:35.864 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.858+0000 7f7d47282700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d400aad50 con 0x7f7d40067a10 2020-11-20T11:47:35.864 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.858+0000 7f7d47282700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d4013aa50 con 0x7f7d400a2460 2020-11-20T11:47:35.864 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.858+0000 7f7d46280700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f7d400a2080 0x7f7d400aef10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:60868/0 (socket says 172.21.15.23:60868) 2020-11-20T11:47:35.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.858+0000 7f7d46280700 1 -- 172.21.15.23:0/2643076017 learned_addr learned my addr 172.21.15.23:0/2643076017 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:47:35.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.858+0000 7f7d45a7f700 1 --2- 172.21.15.23:0/2643076017 >> v2:172.21.15.155:3300/0 conn(0x7f7d400a2460 0x7f7d40137df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:35.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.858+0000 7f7d4527e700 1 -- 172.21.15.23:0/2643076017 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 763255137 0 0) 0x7f7d400aaab0 con 0x7f7d400a2080 2020-11-20T11:47:35.865 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.858+0000 7f7d4527e700 1 -- 172.21.15.23:0/2643076017 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d28003280 con 0x7f7d400a2080 2020-11-20T11:47:35.866 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.858+0000 7f7d4527e700 1 -- 172.21.15.23:0/2643076017 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1007346079 0 0) 0x7f7d28003280 con 0x7f7d400a2080 2020-11-20T11:47:35.866 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.858+0000 7f7d4527e700 1 -- 172.21.15.23:0/2643076017 >> v2:172.21.15.155:3300/0 conn(0x7f7d400a2460 msgr2=0x7f7d40137df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:47:35.866 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.858+0000 7f7d4527e700 1 --2- 172.21.15.23:0/2643076017 >> v2:172.21.15.155:3300/0 conn(0x7f7d400a2460 0x7f7d40137df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:35.866 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.858+0000 7f7d4527e700 1 -- 172.21.15.23:0/2643076017 >> v1:172.21.15.155:6789/0 conn(0x7f7d40067a10 legacy=0x7f7d401453c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:35.866 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.858+0000 7f7d4527e700 1 -- 172.21.15.23:0/2643076017 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4013b920 con 0x7f7d400a2080 2020-11-20T11:47:35.867 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.858+0000 7f7d4527e700 1 -- 172.21.15.23:0/2643076017 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f7d3c0016a0 con 0x7f7d400a2080 2020-11-20T11:47:35.867 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.858+0000 7f7d4527e700 1 -- 172.21.15.23:0/2643076017 >> v1:172.21.15.23:6789/0 conn(0x7f7d400a2080 legacy=0x7f7d400aef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:35.867 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.859+0000 7f7d4527e700 1 --2- 172.21.15.23:0/2643076017 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d28003770 0x7f7d28003b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.867 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.859+0000 7f7d4527e700 1 --2- 172.21.15.23:0/2643076017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d28004520 0x7f7d280049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.867 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.859+0000 7f7d4527e700 1 --2- 172.21.15.23:0/2643076017 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d28004e90 0x7f7d2800b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.859+0000 7f7d4527e700 1 -- 172.21.15.23:0/2643076017 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d280042a0 con 0x7f7d28004e90 2020-11-20T11:47:35.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.859+0000 7f7d4527e700 1 -- 172.21.15.23:0/2643076017 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2800bc60 con 0x7f7d28004520 2020-11-20T11:47:35.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.859+0000 7f7d4527e700 1 -- 172.21.15.23:0/2643076017 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2800bda0 con 0x7f7d28003770 2020-11-20T11:47:35.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.859+0000 7f7d46a81700 1 --2- 172.21.15.23:0/2643076017 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d28004e90 0x7f7d2800b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:35.868 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.859+0000 7f7d46280700 1 --2- 172.21.15.23:0/2643076017 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d28003770 0x7f7d28003b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:35.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.859+0000 7f7d46a81700 1 -- 172.21.15.23:0/2643076017 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d28003770 msgr2=0x7f7d28003b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:47:35.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.859+0000 7f7d46a81700 1 --2- 172.21.15.23:0/2643076017 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d28003770 0x7f7d28003b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:35.869 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.859+0000 7f7d46a81700 1 -- 172.21.15.23:0/2643076017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d28004520 msgr2=0x7f7d280049a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:47:35.870 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.859+0000 7f7d46a81700 1 --2- 172.21.15.23:0/2643076017 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d28004520 0x7f7d280049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:47:35.870 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.859+0000 7f7d46a81700 1 -- 172.21.15.23:0/2643076017 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d28003280 con 0x7f7d28004e90 2020-11-20T11:47:35.870 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.859+0000 7f7d46a81700 1 -- 172.21.15.23:0/2643076017 reap_dead start 2020-11-20T11:47:35.870 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.859+0000 7f7d46a81700 1 --2- 172.21.15.23:0/2643076017 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d28004e90 0x7f7d2800b770 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f7d38001f20 tx=0x7f7d38007f50).ready entity=mon.0 client_cookie=db04071cded15687 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:35.871 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.859+0000 7f7d4527e700 1 -- 172.21.15.23:0/2643076017 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3800ca10 con 0x7f7d28004e90 2020-11-20T11:47:35.871 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.859+0000 7f7d4527e700 1 -- 172.21.15.23:0/2643076017 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3800cb70 con 0x7f7d28004e90 2020-11-20T11:47:35.871 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.859+0000 7f7d4527e700 1 -- 172.21.15.23:0/2643076017 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d380068b0 con 0x7f7d28004e90 2020-11-20T11:47:35.871 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.860+0000 7f7d47282700 1 -- 172.21.15.23:0/2643076017 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d28004e90 msgr2=0x7f7d2800b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:35.871 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.860+0000 7f7d47282700 1 --2- 172.21.15.23:0/2643076017 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d28004e90 0x7f7d2800b770 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f7d38001f20 tx=0x7f7d38007f50).stop 2020-11-20T11:47:35.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.860+0000 7f7d47282700 1 -- 172.21.15.23:0/2643076017 shutdown_connections 2020-11-20T11:47:35.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.860+0000 7f7d47282700 1 --2- 172.21.15.23:0/2643076017 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d28004e90 0x7f7d2800b770 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:35.872 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.860+0000 7f7d47282700 1 -- 172.21.15.23:0/2643076017 >> 172.21.15.23:0/2643076017 conn(0x7f7d40063980 msgr2=0x7f7d40063d90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:47:35.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.860+0000 7f7d47282700 1 -- 172.21.15.23:0/2643076017 shutdown_connections 2020-11-20T11:47:35.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.860+0000 7f7d47282700 1 -- 172.21.15.23:0/2643076017 wait complete. 2020-11-20T11:47:35.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.861+0000 7f7d47282700 1 Processor -- start 2020-11-20T11:47:35.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.861+0000 7f7d47282700 1 -- start start 2020-11-20T11:47:35.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.862+0000 7f7d47282700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d28004e90 0x7f7d400a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.862+0000 7f7d47282700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d400a5be0 0x7f7d400a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.862+0000 7f7d47282700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d400a4f20 0x7f7d4013e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.862+0000 7f7d47282700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d400a53a0 con 0x7f7d28004e90 2020-11-20T11:47:35.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.862+0000 7f7d47282700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4014a7c0 con 0x7f7d400a4f20 2020-11-20T11:47:35.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.862+0000 7f7d47282700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4014aa60 con 0x7f7d400a5be0 2020-11-20T11:47:35.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.862+0000 7f7d46280700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d28004e90 0x7f7d400a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:35.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.862+0000 7f7d46280700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d28004e90 0x7f7d400a4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46630/0 (socket says 172.21.15.23:46630) 2020-11-20T11:47:35.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.862+0000 7f7d46280700 1 -- 172.21.15.23:0/2361354929 learned_addr learned my addr 172.21.15.23:0/2361354929 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:47:35.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.862+0000 7f7d46a81700 1 --2- 172.21.15.23:0/2361354929 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d400a4f20 0x7f7d4013e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:35.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.862+0000 7f7d45a7f700 1 --2- 172.21.15.23:0/2361354929 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d400a5be0 0x7f7d400a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:35.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.862+0000 7f7d46280700 1 -- 172.21.15.23:0/2361354929 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d400a5be0 msgr2=0x7f7d400a4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:35.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.862+0000 7f7d46280700 1 --2- 172.21.15.23:0/2361354929 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d400a5be0 0x7f7d400a4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:35.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.862+0000 7f7d46280700 1 -- 172.21.15.23:0/2361354929 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d400a4f20 msgr2=0x7f7d4013e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:35.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.862+0000 7f7d46280700 1 --2- 172.21.15.23:0/2361354929 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d400a4f20 0x7f7d4013e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:35.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.862+0000 7f7d46280700 1 -- 172.21.15.23:0/2361354929 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4014ad00 con 0x7f7d28004e90 2020-11-20T11:47:35.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.862+0000 7f7d46280700 1 --2- 172.21.15.23:0/2361354929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d28004e90 0x7f7d400a4000 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f7d3c007800 tx=0x7f7d3c007830).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:35.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.862+0000 7f7d2e7fc700 1 -- 172.21.15.23:0/2361354929 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c00f930 con 0x7f7d28004e90 2020-11-20T11:47:35.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.862+0000 7f7d2e7fc700 1 -- 172.21.15.23:0/2361354929 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3c00fa90 con 0x7f7d28004e90 2020-11-20T11:47:35.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.862+0000 7f7d47282700 1 -- 172.21.15.23:0/2361354929 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d4014a440 con 0x7f7d28004e90 2020-11-20T11:47:35.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.862+0000 7f7d2e7fc700 1 -- 172.21.15.23:0/2361354929 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c00fe10 con 0x7f7d28004e90 2020-11-20T11:47:35.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.863+0000 7f7d47282700 1 -- 172.21.15.23:0/2361354929 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d4013e580 con 0x7f7d28004e90 2020-11-20T11:47:35.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.864+0000 7f7d47282700 1 -- 172.21.15.23:0/2361354929 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d400a4830 con 0x7f7d28004e90 2020-11-20T11:47:35.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.864+0000 7f7d2e7fc700 1 -- 172.21.15.23:0/2361354929 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f7d3c01b030 con 0x7f7d28004e90 2020-11-20T11:47:35.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.865+0000 7f7d2e7fc700 1 --2- 172.21.15.23:0/2361354929 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7d18056d60 0x7f7d18059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:35.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.865+0000 7f7d2e7fc700 1 -- 172.21.15.23:0/2361354929 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f7d3c06fb70 con 0x7f7d28004e90 2020-11-20T11:47:35.882 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.869+0000 7f7d45a7f700 1 --2- 172.21.15.23:0/2361354929 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7d18056d60 0x7f7d18059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:35.882 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.869+0000 7f7d45a7f700 1 --2- 172.21.15.23:0/2361354929 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7d18056d60 0x7f7d18059220 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f7d4013f090 tx=0x7f7d30006e30).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:35.882 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:35.871+0000 7f7d2e7fc700 1 -- 172.21.15.23:0/2361354929 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f7d3c044f10 con 0x7f7d28004e90 2020-11-20T11:47:36.162 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:36.160+0000 7f7d47282700 1 -- 172.21.15.23:0/2361354929 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f7d40057dc0 con 0x7f7d28004e90 2020-11-20T11:47:36.163 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:36.160+0000 7f7d2e7fc700 1 -- 172.21.15.23:0/2361354929 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f7d3c048530 con 0x7f7d28004e90 2020-11-20T11:47:36.166 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:36.163+0000 7f7d47282700 1 -- 172.21.15.23:0/2361354929 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7d18056d60 msgr2=0x7f7d18059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:36.167 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:36.163+0000 7f7d47282700 1 --2- 172.21.15.23:0/2361354929 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7d18056d60 0x7f7d18059220 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f7d4013f090 tx=0x7f7d30006e30).stop 2020-11-20T11:47:36.167 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:36.163+0000 7f7d47282700 1 -- 172.21.15.23:0/2361354929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d28004e90 msgr2=0x7f7d400a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:36.167 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:36.163+0000 7f7d47282700 1 --2- 172.21.15.23:0/2361354929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d28004e90 0x7f7d400a4000 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f7d3c007800 tx=0x7f7d3c007830).stop 2020-11-20T11:47:36.167 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:36.163+0000 7f7d47282700 1 -- 172.21.15.23:0/2361354929 shutdown_connections 2020-11-20T11:47:36.167 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:36.163+0000 7f7d47282700 1 --2- 172.21.15.23:0/2361354929 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f7d400a4f20 0x7f7d4013e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:36.168 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:36.163+0000 7f7d47282700 1 --2- 172.21.15.23:0/2361354929 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d28004e90 0x7f7d400a4000 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:36.168 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:36.163+0000 7f7d47282700 1 --2- 172.21.15.23:0/2361354929 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7d18056d60 0x7f7d18059220 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:36.168 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:36.163+0000 7f7d47282700 1 --2- 172.21.15.23:0/2361354929 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7d400a5be0 0x7f7d400a4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:36.168 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:36.163+0000 7f7d47282700 1 -- 172.21.15.23:0/2361354929 >> 172.21.15.23:0/2361354929 conn(0x7f7d40063980 msgr2=0x7f7d400a18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:47:36.168 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:36.163+0000 7f7d47282700 1 -- 172.21.15.23:0/2361354929 shutdown_connections 2020-11-20T11:47:36.169 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:36.163+0000 7f7d47282700 1 -- 172.21.15.23:0/2361354929 wait complete. 2020-11-20T11:47:36.191 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2020-11-20T11:47:36.693 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:36.633+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 202 slow ops, oldest is osd_op(client.4659.0:58398 4.2s0 4:56f73163:::10000000360.0000000d:head [write 618496~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:37.797 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:37.755+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 202 slow ops, oldest is osd_op(client.4659.0:58478 4.2s0 4:56d4cd0f:::10000000393.00000002:head [write 978944~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:38.899 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:38.856+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 202 slow ops, oldest is osd_op(client.4659.0:58814 4.2s0 4:4e4a2d9b:::100000003be.0000000d:head [write 2588672~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:39.886 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:39.834+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 208 slow ops, oldest is osd_op(client.4659.0:58814 4.2s0 4:4e4a2d9b:::100000003be.0000000d:head [write 2588672~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:40.939 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:40.900+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 209 slow ops, oldest is osd_op(client.4659.0:59069 4.2s0 4:449dcbc8:::100000003b6.00000013:head [write 1617920~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:42.016 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:41.955+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 214 slow ops, oldest is osd_op(client.4659.0:59069 4.2s0 4:449dcbc8:::100000003b6.00000013:head [write 1617920~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:42.997 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:42.984+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 218 slow ops, oldest is osd_op(client.4659.0:59114 4.2s0 4:528c9a52:::1000000037e.0000000a:head [write 348160~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:44.061 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:44.022+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 220 slow ops, oldest is osd_op(client.4659.0:59539 4.2s0 4:528c9a52:::1000000037e.0000000a:head [write 217088~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:45.047 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:45.023+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 226 slow ops, oldest is osd_op(client.4659.0:59539 4.2s0 4:528c9a52:::1000000037e.0000000a:head [write 217088~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:46.085 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:46.019+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 227 slow ops, oldest is osd_op(client.4659.0:59539 4.2s0 4:528c9a52:::1000000037e.0000000a:head [write 217088~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:46.191 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:47:46.192 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:47:46.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.316+0000 7f5228d6a700 1 Processor -- start 2020-11-20T11:47:46.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.316+0000 7f5228d6a700 1 -- start start 2020-11-20T11:47:46.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.317+0000 7f5228d6a700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f522407f7f0 0x7f52240a5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:46.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.317+0000 7f5228d6a700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f522407fbd0 0x7f52240a5f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:46.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.317+0000 7f5228d6a700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52240a4330 con 0x7f5224063fe0 2020-11-20T11:47:46.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.317+0000 7f5228d6a700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52240a4560 con 0x7f522407fbd0 2020-11-20T11:47:46.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.317+0000 7f5228d6a700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52240ad830 con 0x7f522407f7f0 2020-11-20T11:47:46.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.317+0000 7f5222d9d700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f5224063fe0 0x7f5224061850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:60888/0 (socket says 172.21.15.23:60888) 2020-11-20T11:47:46.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.317+0000 7f5222d9d700 1 -- 172.21.15.23:0/1317708732 learned_addr learned my addr 172.21.15.23:0/1317708732 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:47:46.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.317+0000 7f5221d9b700 1 --2- 172.21.15.23:0/1317708732 >> v2:172.21.15.155:3300/0 conn(0x7f522407fbd0 0x7f52240a5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:46.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.317+0000 7f522259c700 1 --2- 172.21.15.23:0/1317708732 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f522407f7f0 0x7f52240a5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:46.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.317+0000 7f522159a700 1 -- 172.21.15.23:0/1317708732 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2196515441 0 0) 0x7f52240a4330 con 0x7f5224063fe0 2020-11-20T11:47:46.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.332+0000 7f522159a700 1 -- 172.21.15.23:0/1317708732 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5210003260 con 0x7f5224063fe0 2020-11-20T11:47:46.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.332+0000 7f5221d9b700 1 -- 172.21.15.23:0/1317708732 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f522407f7f0 msgr2=0x7f52240a5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:47:46.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.332+0000 7f5221d9b700 1 --2- 172.21.15.23:0/1317708732 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f522407f7f0 0x7f52240a5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:46.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.332+0000 7f5221d9b700 1 -- 172.21.15.23:0/1317708732 >> v1:172.21.15.23:6789/0 conn(0x7f5224063fe0 legacy=0x7f5224061850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:46.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.332+0000 7f522159a700 1 -- 172.21.15.23:0/1317708732 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3636833888 0 0) 0x7f5210003260 con 0x7f5224063fe0 2020-11-20T11:47:46.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.332+0000 7f5221d9b700 1 -- 172.21.15.23:0/1317708732 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52240ad600 con 0x7f522407fbd0 2020-11-20T11:47:46.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.332+0000 7f522259c700 1 --2- 172.21.15.23:0/1317708732 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f522407f7f0 0x7f52240a5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:47:46.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.332+0000 7f5221d9b700 1 --2- 172.21.15.23:0/1317708732 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f522407fbd0 0x7f52240a5f80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f520c006fd0 tx=0x7f520c006e30).ready entity=mon.1 client_cookie=28e7880ecd8d6cf2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:46.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.333+0000 7f522159a700 1 -- 172.21.15.23:0/1317708732 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520c013070 con 0x7f522407fbd0 2020-11-20T11:47:46.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.333+0000 7f522159a700 1 -- 172.21.15.23:0/1317708732 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f520c000e90 con 0x7f522407fbd0 2020-11-20T11:47:46.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.333+0000 7f522159a700 1 -- 172.21.15.23:0/1317708732 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521c0018a0 con 0x7f522407fbd0 2020-11-20T11:47:46.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.333+0000 7f5228d6a700 1 -- 172.21.15.23:0/1317708732 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f522407fbd0 msgr2=0x7f52240a5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:46.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.333+0000 7f5228d6a700 1 --2- 172.21.15.23:0/1317708732 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f522407fbd0 0x7f52240a5f80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f520c006fd0 tx=0x7f520c006e30).stop 2020-11-20T11:47:46.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.339+0000 7f5228d6a700 1 -- 172.21.15.23:0/1317708732 shutdown_connections 2020-11-20T11:47:46.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.339+0000 7f5228d6a700 1 --2- 172.21.15.23:0/1317708732 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f522407f7f0 0x7f52240a5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:46.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.339+0000 7f5228d6a700 1 --2- 172.21.15.23:0/1317708732 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f522407fbd0 0x7f52240a5f80 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:46.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.339+0000 7f5228d6a700 1 -- 172.21.15.23:0/1317708732 >> 172.21.15.23:0/1317708732 conn(0x7f5224060c80 msgr2=0x7f52240acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:47:46.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.339+0000 7f5228d6a700 1 -- 172.21.15.23:0/1317708732 shutdown_connections 2020-11-20T11:47:46.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.339+0000 7f5228d6a700 1 -- 172.21.15.23:0/1317708732 wait complete. 2020-11-20T11:47:46.345 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.341+0000 7f5228d6a700 1 Processor -- start 2020-11-20T11:47:46.345 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.341+0000 7f5228d6a700 1 -- start start 2020-11-20T11:47:46.346 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.342+0000 7f5228d6a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5224063fe0 0x7f522406f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:46.346 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.342+0000 7f5228d6a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f522407f7f0 0x7f52241574c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:46.346 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.342+0000 7f5228d6a700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f522409ec40 0x7f522409f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:46.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.342+0000 7f5228d6a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5224157c80 con 0x7f522407f7f0 2020-11-20T11:47:46.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.342+0000 7f5228d6a700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522409e7a0 con 0x7f5224063fe0 2020-11-20T11:47:46.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.342+0000 7f5228d6a700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52241579b0 con 0x7f522409ec40 2020-11-20T11:47:46.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.342+0000 7f522259c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5224063fe0 0x7f522406f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:46.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.342+0000 7f522259c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5224063fe0 0x7f522406f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:39538/0 (socket says 172.21.15.23:39538) 2020-11-20T11:47:46.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.342+0000 7f522259c700 1 -- 172.21.15.23:0/529276054 learned_addr learned my addr 172.21.15.23:0/529276054 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:47:46.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.342+0000 7f5222d9d700 1 --2- 172.21.15.23:0/529276054 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f522409ec40 0x7f522409f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:46.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.343+0000 7f522259c700 1 -- 172.21.15.23:0/529276054 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f522409ec40 msgr2=0x7f522409f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:46.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.343+0000 7f522259c700 1 --2- 172.21.15.23:0/529276054 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f522409ec40 0x7f522409f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:46.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.343+0000 7f522259c700 1 -- 172.21.15.23:0/529276054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f522407f7f0 msgr2=0x7f52241574c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:46.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.343+0000 7f522259c700 1 --2- 172.21.15.23:0/529276054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f522407f7f0 0x7f52241574c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:47:46.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.343+0000 7f522259c700 1 -- 172.21.15.23:0/529276054 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f522409f570 con 0x7f5224063fe0 2020-11-20T11:47:46.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.343+0000 7f5222d9d700 1 --2- 172.21.15.23:0/529276054 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f522409ec40 0x7f522409f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:47:46.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.343+0000 7f522259c700 1 --2- 172.21.15.23:0/529276054 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5224063fe0 0x7f522406f410 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f521400eeb0 tx=0x7f5214008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:46.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.343+0000 7f521a7fc700 1 -- 172.21.15.23:0/529276054 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521400f040 con 0x7f5224063fe0 2020-11-20T11:47:46.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.343+0000 7f521a7fc700 1 -- 172.21.15.23:0/529276054 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521400cb40 con 0x7f5224063fe0 2020-11-20T11:47:46.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.343+0000 7f521a7fc700 1 -- 172.21.15.23:0/529276054 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521400cee0 con 0x7f5224063fe0 2020-11-20T11:47:46.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.343+0000 7f5228d6a700 1 -- 172.21.15.23:0/529276054 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f522409ff90 con 0x7f5224063fe0 2020-11-20T11:47:46.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.343+0000 7f5228d6a700 1 -- 172.21.15.23:0/529276054 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52240a1fd0 con 0x7f5224063fe0 2020-11-20T11:47:46.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.345+0000 7f521a7fc700 1 -- 172.21.15.23:0/529276054 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5214013600 con 0x7f5224063fe0 2020-11-20T11:47:46.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.345+0000 7f5228d6a700 1 -- 172.21.15.23:0/529276054 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5224061650 con 0x7f5224063fe0 2020-11-20T11:47:46.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.345+0000 7f521a7fc700 1 --2- 172.21.15.23:0/529276054 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5200056de0 0x7f52000592a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:46.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.345+0000 7f521a7fc700 1 -- 172.21.15.23:0/529276054 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f5214070350 con 0x7f5224063fe0 2020-11-20T11:47:46.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.345+0000 7f5221d9b700 1 --2- 172.21.15.23:0/529276054 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5200056de0 0x7f52000592a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:46.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.346+0000 7f5221d9b700 1 --2- 172.21.15.23:0/529276054 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5200056de0 0x7f52000592a0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f52240a6ed0 tx=0x7f520c015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:46.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:46.350+0000 7f521a7fc700 1 -- 172.21.15.23:0/529276054 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f52140456c0 con 0x7f5224063fe0 2020-11-20T11:47:47.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.060+0000 7f5228d6a700 1 -- 172.21.15.23:0/529276054 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5224044710 con 0x7f5224063fe0 2020-11-20T11:47:47.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.060+0000 7f521a7fc700 1 -- 172.21.15.23:0/529276054 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1479 (secure 0 0 0) 0x7f5214048ce0 con 0x7f5224063fe0 2020-11-20T11:47:47.064 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:47:47.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.062+0000 7f5228d6a700 1 -- 172.21.15.23:0/529276054 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5200056de0 msgr2=0x7f52000592a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:47.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.062+0000 7f5228d6a700 1 --2- 172.21.15.23:0/529276054 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5200056de0 0x7f52000592a0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f52240a6ed0 tx=0x7f520c015000).stop 2020-11-20T11:47:47.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.062+0000 7f5228d6a700 1 -- 172.21.15.23:0/529276054 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5224063fe0 msgr2=0x7f522406f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:47.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.062+0000 7f5228d6a700 1 --2- 172.21.15.23:0/529276054 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5224063fe0 0x7f522406f410 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f521400eeb0 tx=0x7f5214008040).stop 2020-11-20T11:47:47.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.062+0000 7f5228d6a700 1 -- 172.21.15.23:0/529276054 shutdown_connections 2020-11-20T11:47:47.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.062+0000 7f5228d6a700 1 --2- 172.21.15.23:0/529276054 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5200056de0 0x7f52000592a0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:47.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.062+0000 7f5228d6a700 1 --2- 172.21.15.23:0/529276054 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f522409ec40 0x7f522409f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:47.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.062+0000 7f5228d6a700 1 --2- 172.21.15.23:0/529276054 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5224063fe0 0x7f522406f410 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:47.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.062+0000 7f5228d6a700 1 --2- 172.21.15.23:0/529276054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f522407f7f0 0x7f52241574c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:47:47.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.062+0000 7f5228d6a700 1 -- 172.21.15.23:0/529276054 >> 172.21.15.23:0/529276054 conn(0x7f5224060c80 msgr2=0x7f52240aad00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:47:47.068 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.063+0000 7f5228d6a700 1 -- 172.21.15.23:0/529276054 shutdown_connections 2020-11-20T11:47:47.068 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.063+0000 7f5228d6a700 1 -- 172.21.15.23:0/529276054 wait complete. 2020-11-20T11:47:47.085 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:47.065+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4659.0:59668 4.2s0 4:5ba85910:::100000003c5.00000007:head [write 2203648~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:47.097 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:47:47.098 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2020-11-20T11:47:47.098 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:47:47.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.300+0000 7fa3b2548700 1 Processor -- start 2020-11-20T11:47:47.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.300+0000 7fa3b2548700 1 -- start start 2020-11-20T11:47:47.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.301+0000 7fa3b2548700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa3ac0ad600 0x7fa3ac0658b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:47.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.301+0000 7fa3b2548700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa3ac0ad9e0 0x7fa3ac13c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:47.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.301+0000 7fa3b2548700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ac0a8b90 con 0x7fa3ac13cd00 2020-11-20T11:47:47.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.301+0000 7fa3b2548700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3ac065da0 con 0x7fa3ac0ad9e0 2020-11-20T11:47:47.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.301+0000 7fa3b2548700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3ac0a83b0 con 0x7fa3ac0ad600 2020-11-20T11:47:47.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.301+0000 7fa3ab7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa3ac0ad9e0 0x7fa3ac13c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:47.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.301+0000 7fa3ab7fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa3ac0ad9e0 0x7fa3ac13c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46650/0 (socket says 172.21.15.23:46650) 2020-11-20T11:47:47.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.301+0000 7fa3ab7fe700 1 -- 172.21.15.23:0/88419450 learned_addr learned my addr 172.21.15.23:0/88419450 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:47:47.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.301+0000 7fa3ab7fe700 1 -- 172.21.15.23:0/88419450 >> v2:172.21.15.155:3300/0 conn(0x7fa3ac0ad600 msgr2=0x7fa3ac0658b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2020-11-20T11:47:47.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.301+0000 7fa3ab7fe700 1 --2- 172.21.15.23:0/88419450 >> v2:172.21.15.155:3300/0 conn(0x7fa3ac0ad600 0x7fa3ac0658b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:47:47.308 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.301+0000 7fa3ab7fe700 1 -- 172.21.15.23:0/88419450 >> v1:172.21.15.155:6789/0 conn(0x7fa3ac13cd00 legacy=0x7fa3ac0a6b80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2020-11-20T11:47:47.308 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.301+0000 7fa3ab7fe700 1 -- 172.21.15.23:0/88419450 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ac0a8650 con 0x7fa3ac0ad9e0 2020-11-20T11:47:47.308 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.302+0000 7fa3ab7fe700 1 --2- 172.21.15.23:0/88419450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3ac0ad9e0 0x7fa3ac13c810 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fa3ac0610d0 tx=0x7fa3a0006ef0).ready entity=mon.0 client_cookie=127951826bc7abd3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:47.308 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.302+0000 7fa3aaffd700 1 -- 172.21.15.23:0/88419450 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a000b820 con 0x7fa3ac0ad9e0 2020-11-20T11:47:47.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.302+0000 7fa3aaffd700 1 -- 172.21.15.23:0/88419450 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a0000e90 con 0x7fa3ac0ad9e0 2020-11-20T11:47:47.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.302+0000 7fa3aaffd700 1 -- 172.21.15.23:0/88419450 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a000bb90 con 0x7fa3ac0ad9e0 2020-11-20T11:47:47.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.302+0000 7fa3b2548700 1 -- 172.21.15.23:0/88419450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3ac0ad9e0 msgr2=0x7fa3ac13c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:47.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.302+0000 7fa3b2548700 1 --2- 172.21.15.23:0/88419450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3ac0ad9e0 0x7fa3ac13c810 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fa3ac0610d0 tx=0x7fa3a0006ef0).stop 2020-11-20T11:47:47.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.302+0000 7fa3b2548700 1 -- 172.21.15.23:0/88419450 shutdown_connections 2020-11-20T11:47:47.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.302+0000 7fa3b2548700 1 --2- 172.21.15.23:0/88419450 >> v2:172.21.15.155:3300/0 conn(0x7fa3ac0ad600 0x7fa3ac0658b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:47:47.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.302+0000 7fa3b2548700 1 --2- 172.21.15.23:0/88419450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3ac0ad9e0 0x7fa3ac13c810 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:47.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.302+0000 7fa3b2548700 1 -- 172.21.15.23:0/88419450 >> 172.21.15.23:0/88419450 conn(0x7fa3ac060c80 msgr2=0x7fa3ac0acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:47:47.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.305+0000 7fa3b2548700 1 -- 172.21.15.23:0/88419450 shutdown_connections 2020-11-20T11:47:47.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.305+0000 7fa3b2548700 1 -- 172.21.15.23:0/88419450 wait complete. 2020-11-20T11:47:47.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.307+0000 7fa3b2548700 1 Processor -- start 2020-11-20T11:47:47.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.307+0000 7fa3b2548700 1 -- start start 2020-11-20T11:47:47.318 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.308+0000 7fa3b2548700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3ac0ad600 0x7fa3ac09d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:47.319 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.308+0000 7fa3b2548700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3ac0ad9e0 0x7fa3ac09f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:47.319 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.308+0000 7fa3b2548700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3ac13cd00 0x7fa3ac0a5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:47.319 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.308+0000 7fa3b2548700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac14ea90 con 0x7fa3ac0ad600 2020-11-20T11:47:47.319 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.308+0000 7fa3b2548700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac137d80 con 0x7fa3ac13cd00 2020-11-20T11:47:47.320 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.308+0000 7fa3b2548700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac09f700 con 0x7fa3ac0ad9e0 2020-11-20T11:47:47.320 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.308+0000 7fa3b0ae5700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3ac13cd00 0x7fa3ac0a5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:47.320 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.308+0000 7fa3b0ae5700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3ac13cd00 0x7fa3ac0a5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:39552/0 (socket says 172.21.15.23:39552) 2020-11-20T11:47:47.321 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.308+0000 7fa3b0ae5700 1 -- 172.21.15.23:0/3105620498 learned_addr learned my addr 172.21.15.23:0/3105620498 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:47:47.321 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.309+0000 7fa3b0ae5700 1 -- 172.21.15.23:0/3105620498 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3ac0ad9e0 msgr2=0x7fa3ac09f060 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2020-11-20T11:47:47.321 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.309+0000 7fa3b0ae5700 1 --2- 172.21.15.23:0/3105620498 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa3ac0ad9e0 0x7fa3ac09f060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:47:47.322 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.309+0000 7fa3b0ae5700 1 -- 172.21.15.23:0/3105620498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3ac0ad600 msgr2=0x7fa3ac09d960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:47:47.322 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.309+0000 7fa3b0ae5700 1 --2- 172.21.15.23:0/3105620498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3ac0ad600 0x7fa3ac09d960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:47:47.322 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.309+0000 7fa3b0ae5700 1 -- 172.21.15.23:0/3105620498 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ac14ecb0 con 0x7fa3ac13cd00 2020-11-20T11:47:47.322 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.309+0000 7fa3b0ae5700 1 --2- 172.21.15.23:0/3105620498 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3ac13cd00 0x7fa3ac0a5b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fa3a400bae0 tx=0x7fa3a40090f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:47.323 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.309+0000 7fa393fff700 1 -- 172.21.15.23:0/3105620498 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a400c7a0 con 0x7fa3ac13cd00 2020-11-20T11:47:47.323 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.309+0000 7fa393fff700 1 -- 172.21.15.23:0/3105620498 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a400c920 con 0x7fa3ac13cd00 2020-11-20T11:47:47.323 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.309+0000 7fa393fff700 1 -- 172.21.15.23:0/3105620498 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a400cad0 con 0x7fa3ac13cd00 2020-11-20T11:47:47.323 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.310+0000 7fa3b2548700 1 -- 172.21.15.23:0/3105620498 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3ac152710 con 0x7fa3ac13cd00 2020-11-20T11:47:47.324 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.310+0000 7fa3b2548700 1 -- 172.21.15.23:0/3105620498 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3ac152c00 con 0x7fa3ac13cd00 2020-11-20T11:47:47.324 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.313+0000 7fa393fff700 1 -- 172.21.15.23:0/3105620498 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa3a401c070 con 0x7fa3ac13cd00 2020-11-20T11:47:47.324 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.313+0000 7fa393fff700 1 --2- 172.21.15.23:0/3105620498 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa388056d60 0x7fa388059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:47.324 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.313+0000 7fa393fff700 1 -- 172.21.15.23:0/3105620498 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fa3a406f820 con 0x7fa3ac13cd00 2020-11-20T11:47:47.325 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.315+0000 7fa3abfff700 1 --2- 172.21.15.23:0/3105620498 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa388056d60 0x7fa388059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:47.326 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.319+0000 7fa3b2548700 1 --2- 172.21.15.23:0/3105620498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa380002a50 0x7fa380004f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:47.326 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.319+0000 7fa3b2548700 1 -- 172.21.15.23:0/3105620498 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa380007bb0 con 0x7fa380002a50 2020-11-20T11:47:47.326 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.320+0000 7fa3ab7fe700 1 --2- 172.21.15.23:0/3105620498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa380002a50 0x7fa380004f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:47.326 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.320+0000 7fa3ab7fe700 1 -- 172.21.15.23:0/3105620498 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa380007d70 con 0x7fa380002a50 2020-11-20T11:47:47.327 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.320+0000 7fa3ab7fe700 1 --2- 172.21.15.23:0/3105620498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa380002a50 0x7fa380004f10 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fa3ac0a3eb0 tx=0x7fa3a0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:47.327 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.320+0000 7fa393fff700 1 -- 172.21.15.23:0/3105620498 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0011b20 con 0x7fa380002a50 2020-11-20T11:47:47.327 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.321+0000 7fa393fff700 1 -- 172.21.15.23:0/3105620498 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fa3a0011c80 con 0x7fa380002a50 2020-11-20T11:47:47.327 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.321+0000 7fa393fff700 1 -- 172.21.15.23:0/3105620498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa380002a50 msgr2=0x7fa380004f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:47.328 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.321+0000 7fa393fff700 1 --2- 172.21.15.23:0/3105620498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa380002a50 0x7fa380004f10 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fa3ac0a3eb0 tx=0x7fa3a0016000).stop 2020-11-20T11:47:47.328 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.321+0000 7fa393fff700 1 -- 172.21.15.23:0/3105620498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa380002a50 msgr2=0x7fa380004f10 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:47:47.328 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.321+0000 7fa393fff700 1 --2- 172.21.15.23:0/3105620498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa380002a50 0x7fa380004f10 secure :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0x7fa3ac0a3eb0 tx=0x7fa3a0016000).stop 2020-11-20T11:47:47.328 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.322+0000 7fa3abfff700 1 --2- 172.21.15.23:0/3105620498 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa388056d60 0x7fa388059220 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fa39c006fd0 tx=0x7fa39c006970).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:47.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.339+0000 7fa3b2548700 1 --2- 172.21.15.23:0/3105620498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa380006c00 0x7fa380007090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:47.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.339+0000 7fa3ab7fe700 1 --2- 172.21.15.23:0/3105620498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa380006c00 0x7fa380007090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:47.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.340+0000 7fa3b2548700 1 -- 172.21.15.23:0/3105620498 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3800076a0 con 0x7fa380006c00 2020-11-20T11:47:47.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.341+0000 7fa3ab7fe700 1 -- 172.21.15.23:0/3105620498 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fa380007860 con 0x7fa380006c00 2020-11-20T11:47:47.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.341+0000 7fa3ab7fe700 1 --2- 172.21.15.23:0/3105620498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa380006c00 0x7fa380007090 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fa380007d40 tx=0x7fa3a001d000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:47.350 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:47:47.350 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:47:47.351 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:47:47.351 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:47:47.351 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 138, 2020-11-20T11:47:47.351 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:47:47.351 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:47:47.352 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:47:47.352 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:47:47.352 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:47.352 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 13, 2020-11-20T11:47:47.352 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:47:47.353 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:47:47.353 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:47:47.353 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:47:47.353 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:47:47.354 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:47:47.354 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:47:47.354 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:47:47.354 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:47:47.354 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:47:47.355 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:47:47.355 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:47.355 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:47:47.355 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:47:47.355 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:47:47.356 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:47:47.356 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:47:47.356 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:47:47.356 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:47:47.356 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:47:47.357 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:47:47.357 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:47:47.357 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:47.357 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.357 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:47:47.358 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:47:47.358 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:47:47.358 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:47:47.358 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:47:47.359 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:47:47.359 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:47:47.359 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:47:47.359 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:47:47.359 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:47:47.360 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:47:47.360 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:47:47.360 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:47:47.360 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:47:47.360 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:47:47.361 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:47:47.361 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:47:47.361 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:47:47.361 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:47:47.361 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:47:47.362 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:47:47.362 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:47:47.362 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:47:47.362 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:47.362 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:47:47.363 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.363 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:47:47.363 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.363 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:47:47.363 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:47:47.364 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:47:47.364 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:47:47.364 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.364 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:47:47.364 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:47:47.365 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:47.365 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.365 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.365 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:47:47.366 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:47:47.366 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:47.366 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:47.366 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:47.366 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.367 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:47:47.367 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:47:47.367 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:47:47.367 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:47:47.367 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:47:47.368 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.368 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.368 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:47:47.368 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:47:47.368 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:47:47.369 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:47:47.369 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.369 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:47:47.369 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:47:47.369 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:47.370 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.370 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.370 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:47:47.370 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:47:47.370 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:47.371 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:47.371 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:47.371 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.371 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:47:47.371 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:47:47.372 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:47:47.372 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:47:47.372 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:47:47.372 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.372 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.373 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:47:47.373 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:47:47.373 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:47:47.373 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:47:47.374 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.374 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:47:47.374 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:47:47.374 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:47.374 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.375 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.375 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:47:47.375 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:47:47.375 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:47.375 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:47.376 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:47.376 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.376 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:47:47.376 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:47:47.376 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:47:47.377 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:47:47.377 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:47:47.377 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:47.377 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:47.377 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.378 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:47:47.378 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:47:47.378 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.378 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:47.378 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:47.379 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:47:47.379 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:47.379 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:47.379 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:47:47.379 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.380 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:47.380 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:47.380 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:47:47.380 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:47.381 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:47.381 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:47:47.381 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.381 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:47.381 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:47.382 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:47:47.382 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:47.382 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:47.382 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:47:47.382 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.383 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:47.383 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:47.383 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:47:47.383 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:47.384 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:47.384 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:47:47.384 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.384 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:47.385 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:47.385 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:47:47.385 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:47.385 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:47.385 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.386 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:47:47.386 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:47:47.387 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.347+0000 7fa393fff700 1 -- 172.21.15.23:0/3105620498 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a00259d0 con 0x7fa380006c00 2020-11-20T11:47:47.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.347+0000 7fa393fff700 1 -- 172.21.15.23:0/3105620498 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4787 (secure 0 0 0) 0x7fa3a0025b30 con 0x7fa380006c00 2020-11-20T11:47:47.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.347+0000 7fa393fff700 1 -- 172.21.15.23:0/3105620498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa380006c00 msgr2=0x7fa380007090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:47.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.347+0000 7fa393fff700 1 --2- 172.21.15.23:0/3105620498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa380006c00 0x7fa380007090 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fa380007d40 tx=0x7fa3a001d000).stop 2020-11-20T11:47:47.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.347+0000 7fa393fff700 1 -- 172.21.15.23:0/3105620498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa380006c00 msgr2=0x7fa380007090 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:47:47.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.347+0000 7fa393fff700 1 --2- 172.21.15.23:0/3105620498 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa380006c00 0x7fa380007090 secure :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0x7fa380007d40 tx=0x7fa3a001d000).stop 2020-11-20T11:47:47.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.349+0000 7fa3b2548700 1 -- 172.21.15.23:0/3105620498 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa388056d60 msgr2=0x7fa388059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:47.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.349+0000 7fa3b2548700 1 --2- 172.21.15.23:0/3105620498 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa388056d60 0x7fa388059220 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fa39c006fd0 tx=0x7fa39c006970).stop 2020-11-20T11:47:47.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.349+0000 7fa3b2548700 1 -- 172.21.15.23:0/3105620498 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3ac13cd00 msgr2=0x7fa3ac0a5b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:47.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.349+0000 7fa3b2548700 1 --2- 172.21.15.23:0/3105620498 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa3ac13cd00 0x7fa3ac0a5b20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fa3a400bae0 tx=0x7fa3a40090f0).stop 2020-11-20T11:47:47.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.352+0000 7fa3b0ae5700 1 -- 172.21.15.23:0/3105620498 reap_dead start 2020-11-20T11:47:47.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.352+0000 7fa3b0ae5700 1 -- 172.21.15.23:0/3105620498 reap_dead start 2020-11-20T11:47:47.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.352+0000 7fa3b2548700 1 -- 172.21.15.23:0/3105620498 shutdown_connections 2020-11-20T11:47:47.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.352+0000 7fa3b2548700 1 -- 172.21.15.23:0/3105620498 >> 172.21.15.23:0/3105620498 conn(0x7fa3ac060c80 msgr2=0x7fa3ac0a9990 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:47:47.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.353+0000 7fa3b2548700 1 -- 172.21.15.23:0/3105620498 shutdown_connections 2020-11-20T11:47:47.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.353+0000 7fa3b2548700 1 -- 172.21.15.23:0/3105620498 wait complete. 2020-11-20T11:47:47.391 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:47:47.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.621+0000 7fc24cdd0700 1 Processor -- start 2020-11-20T11:47:47.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.621+0000 7fc24cdd0700 1 -- start start 2020-11-20T11:47:47.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.623+0000 7fc24cdd0700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc24805bc80 0x7fc248067c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:47.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.623+0000 7fc24cdd0700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc248065780 0x7fc2480601b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:47.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.623+0000 7fc24cdd0700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc2480606a0 0x7fc248060b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:47.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.623+0000 7fc24cdd0700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2480669e0 con 0x7fc2480606a0 2020-11-20T11:47:47.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.623+0000 7fc24cdd0700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc248067800 con 0x7fc248065780 2020-11-20T11:47:47.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.623+0000 7fc24cdd0700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc248061000 con 0x7fc24805bc80 2020-11-20T11:47:47.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.623+0000 7fc246d9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc2480606a0 0x7fc248060b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:47.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.623+0000 7fc246d9d700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fc2480606a0 0x7fc248060b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46668/0 (socket says 172.21.15.23:46668) 2020-11-20T11:47:47.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.623+0000 7fc246d9d700 1 -- 172.21.15.23:0/675485370 learned_addr learned my addr 172.21.15.23:0/675485370 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:47:47.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.623+0000 7fc24659c700 1 --2- 172.21.15.23:0/675485370 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc24805bc80 0x7fc248067c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:47.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.623+0000 7fc246d9d700 1 -- 172.21.15.23:0/675485370 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc24805bc80 msgr2=0x7fc248067c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:47:47.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.623+0000 7fc246d9d700 1 --2- 172.21.15.23:0/675485370 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc24805bc80 0x7fc248067c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:47.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.623+0000 7fc246d9d700 1 -- 172.21.15.23:0/675485370 >> v2:172.21.15.155:3300/0 conn(0x7fc248065780 msgr2=0x7fc2480601b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:47:47.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.623+0000 7fc246d9d700 1 --2- 172.21.15.23:0/675485370 >> v2:172.21.15.155:3300/0 conn(0x7fc248065780 0x7fc2480601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:47:47.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.623+0000 7fc246d9d700 1 -- 172.21.15.23:0/675485370 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2480674a0 con 0x7fc2480606a0 2020-11-20T11:47:47.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.623+0000 7fc246d9d700 1 --2- 172.21.15.23:0/675485370 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc2480606a0 0x7fc248060b10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fc24805c2e0 tx=0x7fc2400092d0).ready entity=mon.0 client_cookie=196da2a773ed16a7 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:47.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.623+0000 7fc24559a700 1 -- 172.21.15.23:0/675485370 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24000bbf0 con 0x7fc2480606a0 2020-11-20T11:47:47.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.624+0000 7fc24559a700 1 -- 172.21.15.23:0/675485370 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc24000bd50 con 0x7fc2480606a0 2020-11-20T11:47:47.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.624+0000 7fc24559a700 1 -- 172.21.15.23:0/675485370 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc240011610 con 0x7fc2480606a0 2020-11-20T11:47:47.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.624+0000 7fc24cdd0700 1 -- 172.21.15.23:0/675485370 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc2480606a0 msgr2=0x7fc248060b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:47.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.624+0000 7fc24cdd0700 1 --2- 172.21.15.23:0/675485370 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc2480606a0 0x7fc248060b10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fc24805c2e0 tx=0x7fc2400092d0).stop 2020-11-20T11:47:47.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.631+0000 7fc24cdd0700 1 -- 172.21.15.23:0/675485370 shutdown_connections 2020-11-20T11:47:47.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.631+0000 7fc24cdd0700 1 --2- 172.21.15.23:0/675485370 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc2480606a0 0x7fc248060b10 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:47.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.631+0000 7fc24cdd0700 1 --2- 172.21.15.23:0/675485370 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc24805bc80 0x7fc248067c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:47.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.631+0000 7fc24cdd0700 1 --2- 172.21.15.23:0/675485370 >> v2:172.21.15.155:3300/0 conn(0x7fc248065780 0x7fc2480601b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:47:47.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.631+0000 7fc24cdd0700 1 -- 172.21.15.23:0/675485370 >> 172.21.15.23:0/675485370 conn(0x7fc2480ad7a0 msgr2=0x7fc2480adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:47:47.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.634+0000 7fc24cdd0700 1 -- 172.21.15.23:0/675485370 shutdown_connections 2020-11-20T11:47:47.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.634+0000 7fc24cdd0700 1 -- 172.21.15.23:0/675485370 wait complete. 2020-11-20T11:47:47.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.636+0000 7fc24cdd0700 1 Processor -- start 2020-11-20T11:47:47.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.636+0000 7fc24cdd0700 1 -- start start 2020-11-20T11:47:47.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.636+0000 7fc24cdd0700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc24805bc80 0x7fc2480a4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:47.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.636+0000 7fc24cdd0700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc2480606a0 0x7fc2480a0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:47.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.636+0000 7fc24cdd0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc248065780 0x7fc24809f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:47.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.637+0000 7fc24659c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc24805bc80 0x7fc2480a4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:47.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.637+0000 7fc245d9b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc2480606a0 0x7fc2480a0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:47.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.637+0000 7fc24659c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc24805bc80 0x7fc2480a4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:39568/0 (socket says 172.21.15.23:39568) 2020-11-20T11:47:47.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.637+0000 7fc245d9b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc2480606a0 0x7fc2480a0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:56914/0 (socket says 172.21.15.23:56914) 2020-11-20T11:47:47.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.637+0000 7fc246d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc248065780 0x7fc24809f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:47.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.637+0000 7fc246d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc248065780 0x7fc24809f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46676/0 (socket says 172.21.15.23:46676) 2020-11-20T11:47:47.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.637+0000 7fc246d9d700 1 -- 172.21.15.23:0/2797744713 learned_addr learned my addr 172.21.15.23:0/2797744713 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:47:47.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.637+0000 7fc24cdd0700 1 -- 172.21.15.23:0/2797744713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc248151590 con 0x7fc248065780 2020-11-20T11:47:47.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.637+0000 7fc24cdd0700 1 -- 172.21.15.23:0/2797744713 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2480a0a90 con 0x7fc24805bc80 2020-11-20T11:47:47.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.637+0000 7fc24cdd0700 1 -- 172.21.15.23:0/2797744713 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2480a0d30 con 0x7fc2480606a0 2020-11-20T11:47:47.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.637+0000 7fc246d9d700 1 -- 172.21.15.23:0/2797744713 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc2480606a0 msgr2=0x7fc2480a0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:47.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.637+0000 7fc246d9d700 1 --2- 172.21.15.23:0/2797744713 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc2480606a0 0x7fc2480a0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:47.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.637+0000 7fc246d9d700 1 -- 172.21.15.23:0/2797744713 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc24805bc80 msgr2=0x7fc2480a4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:47.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.637+0000 7fc246d9d700 1 --2- 172.21.15.23:0/2797744713 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc24805bc80 0x7fc2480a4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:47.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.637+0000 7fc246d9d700 1 -- 172.21.15.23:0/2797744713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc24809f1e0 con 0x7fc248065780 2020-11-20T11:47:47.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.637+0000 7fc245d9b700 1 --2- 172.21.15.23:0/2797744713 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc2480606a0 0x7fc2480a0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:47:47.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.637+0000 7fc246d9d700 1 --2- 172.21.15.23:0/2797744713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc248065780 0x7fc24809f680 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fc2480ae390 tx=0x7fc240016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:47.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.638+0000 7fc22e7fc700 1 -- 172.21.15.23:0/2797744713 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24001c680 con 0x7fc248065780 2020-11-20T11:47:47.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.638+0000 7fc22e7fc700 1 -- 172.21.15.23:0/2797744713 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc24001c7e0 con 0x7fc248065780 2020-11-20T11:47:47.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.638+0000 7fc22e7fc700 1 -- 172.21.15.23:0/2797744713 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24001cb60 con 0x7fc248065780 2020-11-20T11:47:47.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.638+0000 7fc24cdd0700 1 -- 172.21.15.23:0/2797744713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2480a0750 con 0x7fc248065780 2020-11-20T11:47:47.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.638+0000 7fc24cdd0700 1 -- 172.21.15.23:0/2797744713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2481167e0 con 0x7fc248065780 2020-11-20T11:47:47.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.639+0000 7fc24cdd0700 1 --2- 172.21.15.23:0/2797744713 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc24809fb70 0x7fc248151950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:47.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.639+0000 7fc24cdd0700 1 -- 172.21.15.23:0/2797744713 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc248062fe0 con 0x7fc24809fb70 2020-11-20T11:47:47.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.641+0000 7fc24659c700 1 --2- 172.21.15.23:0/2797744713 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc24809fb70 0x7fc248151950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:47.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.641+0000 7fc22e7fc700 1 -- 172.21.15.23:0/2797744713 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc240023030 con 0x7fc248065780 2020-11-20T11:47:47.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.642+0000 7fc22e7fc700 1 --2- 172.21.15.23:0/2797744713 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc224056d60 0x7fc224059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:47.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.642+0000 7fc22e7fc700 1 -- 172.21.15.23:0/2797744713 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fc2400716c0 con 0x7fc248065780 2020-11-20T11:47:47.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.644+0000 7fc24659c700 1 -- 172.21.15.23:0/2797744713 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc24809f370 con 0x7fc24809fb70 2020-11-20T11:47:47.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.646+0000 7fc24659c700 1 --2- 172.21.15.23:0/2797744713 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc24809fb70 0x7fc248151950 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fc238006fd0 tx=0x7fc238006e30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:47.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.646+0000 7fc22e7fc700 1 -- 172.21.15.23:0/2797744713 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc238011e90 con 0x7fc24809fb70 2020-11-20T11:47:47.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.646+0000 7fc22e7fc700 1 -- 172.21.15.23:0/2797744713 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fc238000e90 con 0x7fc24809fb70 2020-11-20T11:47:47.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.646+0000 7fc22e7fc700 1 -- 172.21.15.23:0/2797744713 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc24809fb70 msgr2=0x7fc248151950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:47.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.646+0000 7fc22e7fc700 1 --2- 172.21.15.23:0/2797744713 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc24809fb70 0x7fc248151950 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fc238006fd0 tx=0x7fc238006e30).stop 2020-11-20T11:47:47.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.646+0000 7fc22e7fc700 1 -- 172.21.15.23:0/2797744713 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc24809fb70 msgr2=0x7fc248151950 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:47:47.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.646+0000 7fc22e7fc700 1 --2- 172.21.15.23:0/2797744713 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc24809fb70 0x7fc248151950 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:47.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.646+0000 7fc245d9b700 1 --2- 172.21.15.23:0/2797744713 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc224056d60 0x7fc224059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:47.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.652+0000 7fc245d9b700 1 --2- 172.21.15.23:0/2797744713 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc224056d60 0x7fc224059220 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7fc2480648c0 tx=0x7fc23c008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:47.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.662+0000 7fc24cdd0700 1 --2- 172.21.15.23:0/2797744713 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc248151e40 0x7fc24809d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:47.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.662+0000 7fc24cdd0700 1 -- 172.21.15.23:0/2797744713 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc24809f540 con 0x7fc248151e40 2020-11-20T11:47:47.666 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.662+0000 7fc24659c700 1 --2- 172.21.15.23:0/2797744713 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc248151e40 0x7fc24809d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:47.667 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.662+0000 7fc24659c700 1 -- 172.21.15.23:0/2797744713 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc248044710 con 0x7fc248151e40 2020-11-20T11:47:47.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.663+0000 7fc24659c700 1 --2- 172.21.15.23:0/2797744713 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc248151e40 0x7fc24809d4a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fc23800b190 tx=0x7fc23801a000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:47.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.663+0000 7fc22e7fc700 1 -- 172.21.15.23:0/2797744713 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc238022850 con 0x7fc248151e40 2020-11-20T11:47:47.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.663+0000 7fc22e7fc700 1 -- 172.21.15.23:0/2797744713 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4275 (secure 0 0 0) 0x7fc23800fd60 con 0x7fc248151e40 2020-11-20T11:47:47.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.663+0000 7fc22e7fc700 1 -- 172.21.15.23:0/2797744713 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc248151e40 msgr2=0x7fc24809d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:47.668 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.663+0000 7fc22e7fc700 1 --2- 172.21.15.23:0/2797744713 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc248151e40 0x7fc24809d4a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fc23800b190 tx=0x7fc23801a000).stop 2020-11-20T11:47:47.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.663+0000 7fc22e7fc700 1 -- 172.21.15.23:0/2797744713 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc248151e40 msgr2=0x7fc24809d4a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:47:47.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.663+0000 7fc22e7fc700 1 --2- 172.21.15.23:0/2797744713 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc248151e40 0x7fc24809d4a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:47.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.664+0000 7fc24cdd0700 1 -- 172.21.15.23:0/2797744713 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc224056d60 msgr2=0x7fc224059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:47.669 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.664+0000 7fc24cdd0700 1 --2- 172.21.15.23:0/2797744713 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc224056d60 0x7fc224059220 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7fc2480648c0 tx=0x7fc23c008040).stop 2020-11-20T11:47:47.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.664+0000 7fc246d9d700 1 -- 172.21.15.23:0/2797744713 reap_dead start 2020-11-20T11:47:47.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.664+0000 7fc24cdd0700 1 -- 172.21.15.23:0/2797744713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc248065780 msgr2=0x7fc24809f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:47.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.664+0000 7fc24cdd0700 1 --2- 172.21.15.23:0/2797744713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc248065780 0x7fc24809f680 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fc2480ae390 tx=0x7fc240016000).stop 2020-11-20T11:47:47.670 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.664+0000 7fc24cdd0700 1 -- 172.21.15.23:0/2797744713 shutdown_connections 2020-11-20T11:47:47.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.664+0000 7fc24cdd0700 1 --2- 172.21.15.23:0/2797744713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc248065780 0x7fc24809f680 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:47.671 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.664+0000 7fc24cdd0700 1 -- 172.21.15.23:0/2797744713 >> 172.21.15.23:0/2797744713 conn(0x7fc2480ad7a0 msgr2=0x7fc2480a7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:47:47.671 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:47:47.671 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:47:47.671 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:47:47.672 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:47:47.672 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 138, 2020-11-20T11:47:47.672 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:47:47.672 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:47:47.673 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:47:47.673 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:47:47.673 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:47.673 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 14, 2020-11-20T11:47:47.674 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:47:47.674 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:47:47.674 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:47:47.674 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:47:47.674 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:47:47.675 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:47:47.675 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:47:47.675 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:47:47.675 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:47:47.675 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:47:47.676 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:47:47.676 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:47.676 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:47:47.676 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:47:47.676 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:47:47.677 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:47:47.677 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:47:47.677 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:47:47.677 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:47:47.678 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:47:47.678 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:47:47.678 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:47:47.678 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:47.678 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.679 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:47:47.679 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:47:47.680 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:47:47.680 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:47:47.680 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:47:47.680 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:47:47.681 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:47:47.681 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:47:47.681 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:47:47.681 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:47:47.682 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:47:47.682 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:47:47.682 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:47:47.682 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:47:47.682 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:47:47.683 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:47:47.683 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:47:47.683 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:47:47.683 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:47:47.683 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:47:47.683 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:47:47.684 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:47:47.684 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:47:47.684 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:47.684 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:47:47.685 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.685 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:47:47.685 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.685 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:47:47.685 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:47:47.686 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:47:47.686 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:47:47.686 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.686 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:47:47.686 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:47:47.687 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:47.687 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.687 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.687 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:47:47.687 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:47:47.688 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:47.688 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:47.688 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:47.688 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.688 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:47:47.689 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:47:47.689 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:47:47.689 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:47:47.689 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:47:47.689 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.690 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.690 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:47:47.690 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:47:47.690 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:47:47.690 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:47:47.691 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.691 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:47:47.691 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:47:47.691 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:47.691 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.692 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.692 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:47:47.692 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:47:47.692 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:47.692 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:47.693 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:47.693 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.693 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:47:47.693 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:47:47.694 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:47:47.694 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:47:47.694 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:47:47.695 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.695 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.696 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:47:47.696 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:47:47.696 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:47:47.697 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:47:47.697 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.698 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:47:47.698 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:47:47.698 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:47.699 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.699 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.699 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:47:47.699 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:47:47.699 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:47.700 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:47.700 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:47.700 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.700 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:47:47.700 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:47:47.701 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:47:47.701 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:47:47.701 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:47:47.701 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:47.702 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:47.702 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.702 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:47:47.702 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:47:47.702 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.703 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:47.703 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:47.703 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:47:47.703 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:47.703 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:47.704 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:47:47.704 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.704 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:47.704 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:47.704 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:47:47.705 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:47.705 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:47.705 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.705 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:47:47.705 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:47:47.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.666+0000 7fc24cdd0700 1 -- 172.21.15.23:0/2797744713 shutdown_connections 2020-11-20T11:47:47.707 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.667+0000 7fc24cdd0700 1 -- 172.21.15.23:0/2797744713 wait complete. 2020-11-20T11:47:47.707 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:47:47.863 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.860+0000 7ffa6e4bd700 1 Processor -- start 2020-11-20T11:47:47.863 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.860+0000 7ffa6e4bd700 1 -- start start 2020-11-20T11:47:47.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.860+0000 7ffa6e4bd700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ffa68136ea0 0x7ffa680a8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:47.873 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.861+0000 7ffa6e4bd700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa680a4970 con 0x7ffa68061a10 2020-11-20T11:47:47.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.861+0000 7ffa6e4bd700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa680a60e0 con 0x7ffa680a8fc0 2020-11-20T11:47:47.874 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.861+0000 7ffa6e4bd700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa680a5ef0 con 0x7ffa68136ea0 2020-11-20T11:47:47.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.861+0000 7ffa67fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7ffa68061a10 0x7ffa680a6580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:60934/0 (socket says 172.21.15.23:60934) 2020-11-20T11:47:47.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.861+0000 7ffa67fff700 1 -- 172.21.15.23:0/2312191178 learned_addr learned my addr 172.21.15.23:0/2312191178 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:47:47.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.861+0000 7ffa677fe700 1 --2- 172.21.15.23:0/2312191178 >> v2:172.21.15.155:3300/0 conn(0x7ffa68136ea0 0x7ffa680a8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:47.875 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.861+0000 7ffa66ffd700 1 -- 172.21.15.23:0/2312191178 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 515887480 0 0) 0x7ffa680a4970 con 0x7ffa68061a10 2020-11-20T11:47:47.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.861+0000 7ffa66ffd700 1 -- 172.21.15.23:0/2312191178 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa54003280 con 0x7ffa68061a10 2020-11-20T11:47:47.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.861+0000 7ffa677fe700 1 -- 172.21.15.23:0/2312191178 >> v1:172.21.15.155:6789/0 conn(0x7ffa680a8fc0 legacy=0x7ffa68141810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:47.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.861+0000 7ffa677fe700 1 -- 172.21.15.23:0/2312191178 >> v1:172.21.15.23:6789/0 conn(0x7ffa68061a10 legacy=0x7ffa680a6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:47.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.861+0000 7ffa677fe700 1 -- 172.21.15.23:0/2312191178 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa68061df0 con 0x7ffa68136ea0 2020-11-20T11:47:47.876 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.861+0000 7ffa677fe700 1 --2- 172.21.15.23:0/2312191178 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa68136ea0 0x7ffa680a8ad0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7ffa5c006fd0 tx=0x7ffa5c006e30).ready entity=mon.1 client_cookie=525c39b98033f733 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:47.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.861+0000 7ffa66ffd700 1 -- 172.21.15.23:0/2312191178 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c0097e0 con 0x7ffa68136ea0 2020-11-20T11:47:47.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.861+0000 7ffa66ffd700 1 -- 172.21.15.23:0/2312191178 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa5c000e90 con 0x7ffa68136ea0 2020-11-20T11:47:47.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.861+0000 7ffa66ffd700 1 -- 172.21.15.23:0/2312191178 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c009b50 con 0x7ffa68136ea0 2020-11-20T11:47:47.877 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.862+0000 7ffa6e4bd700 1 -- 172.21.15.23:0/2312191178 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa68136ea0 msgr2=0x7ffa680a8ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:47.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.862+0000 7ffa6e4bd700 1 --2- 172.21.15.23:0/2312191178 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa68136ea0 0x7ffa680a8ad0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7ffa5c006fd0 tx=0x7ffa5c006e30).stop 2020-11-20T11:47:47.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.862+0000 7ffa6e4bd700 1 -- 172.21.15.23:0/2312191178 shutdown_connections 2020-11-20T11:47:47.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.862+0000 7ffa6e4bd700 1 --2- 172.21.15.23:0/2312191178 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa68136ea0 0x7ffa680a8ad0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:47.878 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.862+0000 7ffa6e4bd700 1 -- 172.21.15.23:0/2312191178 >> 172.21.15.23:0/2312191178 conn(0x7ffa68060c80 msgr2=0x7ffa680acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:47:47.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.862+0000 7ffa6e4bd700 1 -- 172.21.15.23:0/2312191178 shutdown_connections 2020-11-20T11:47:47.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.862+0000 7ffa6e4bd700 1 -- 172.21.15.23:0/2312191178 wait complete. 2020-11-20T11:47:47.879 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.863+0000 7ffa6e4bd700 1 Processor -- start 2020-11-20T11:47:47.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.864+0000 7ffa6e4bd700 1 -- start start 2020-11-20T11:47:47.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.864+0000 7ffa6e4bd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa68061a10 0x7ffa6809e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:47.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.864+0000 7ffa6e4bd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa680a8fc0 0x7ffa68066ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:47.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.864+0000 7ffa6e4bd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffa68136ea0 0x7ffa68066130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:47.880 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.864+0000 7ffa6e4bd700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa680672f0 con 0x7ffa68061a10 2020-11-20T11:47:47.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.864+0000 7ffa6e4bd700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa68067590 con 0x7ffa680a8fc0 2020-11-20T11:47:47.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.864+0000 7ffa6e4bd700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa68066fb0 con 0x7ffa68136ea0 2020-11-20T11:47:47.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.864+0000 7ffa67fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa68061a10 0x7ffa6809e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:47.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.864+0000 7ffa67fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa68061a10 0x7ffa6809e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46690/0 (socket says 172.21.15.23:46690) 2020-11-20T11:47:47.881 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.864+0000 7ffa67fff700 1 -- 172.21.15.23:0/14564524 learned_addr learned my addr 172.21.15.23:0/14564524 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:47:47.882 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.864+0000 7ffa677fe700 1 --2- 172.21.15.23:0/14564524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa680a8fc0 0x7ffa68066ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:47.882 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.864+0000 7ffa6ca5a700 1 --2- 172.21.15.23:0/14564524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffa68136ea0 0x7ffa68066130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:47.882 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.865+0000 7ffa67fff700 1 -- 172.21.15.23:0/14564524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffa68136ea0 msgr2=0x7ffa68066130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:47.882 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.865+0000 7ffa67fff700 1 --2- 172.21.15.23:0/14564524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffa68136ea0 0x7ffa68066130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:47.882 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.865+0000 7ffa67fff700 1 -- 172.21.15.23:0/14564524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa680a8fc0 msgr2=0x7ffa68066ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:47.883 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.865+0000 7ffa67fff700 1 --2- 172.21.15.23:0/14564524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa680a8fc0 0x7ffa68066ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:47.883 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.865+0000 7ffa67fff700 1 -- 172.21.15.23:0/14564524 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa68067830 con 0x7ffa68061a10 2020-11-20T11:47:47.883 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.865+0000 7ffa677fe700 1 --2- 172.21.15.23:0/14564524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffa680a8fc0 0x7ffa68066ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:47:47.883 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.865+0000 7ffa67fff700 1 --2- 172.21.15.23:0/14564524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa68061a10 0x7ffa6809e1c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7ffa5800b790 tx=0x7ffa580091a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:47.884 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.865+0000 7ffa47fff700 1 -- 172.21.15.23:0/14564524 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa58006320 con 0x7ffa68061a10 2020-11-20T11:47:47.884 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.865+0000 7ffa47fff700 1 -- 172.21.15.23:0/14564524 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa58000e90 con 0x7ffa68061a10 2020-11-20T11:47:47.884 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.865+0000 7ffa47fff700 1 -- 172.21.15.23:0/14564524 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa58006690 con 0x7ffa68061a10 2020-11-20T11:47:47.884 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.865+0000 7ffa6e4bd700 1 -- 172.21.15.23:0/14564524 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa6813a3d0 con 0x7ffa68061a10 2020-11-20T11:47:47.885 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.865+0000 7ffa6e4bd700 1 -- 172.21.15.23:0/14564524 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa68066860 con 0x7ffa68061a10 2020-11-20T11:47:47.885 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.866+0000 7ffa6e4bd700 1 --2- 172.21.15.23:0/14564524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffa68146260 0x7ffa68146640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:47.885 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.866+0000 7ffa6e4bd700 1 -- 172.21.15.23:0/14564524 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa68067170 con 0x7ffa68146260 2020-11-20T11:47:47.885 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.866+0000 7ffa677fe700 1 --2- 172.21.15.23:0/14564524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffa68146260 0x7ffa68146640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:47.886 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.867+0000 7ffa47fff700 1 -- 172.21.15.23:0/14564524 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ffa5800f070 con 0x7ffa68061a10 2020-11-20T11:47:47.886 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.867+0000 7ffa677fe700 1 -- 172.21.15.23:0/14564524 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa68061650 con 0x7ffa68146260 2020-11-20T11:47:47.886 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.867+0000 7ffa677fe700 1 --2- 172.21.15.23:0/14564524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffa68146260 0x7ffa68146640 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7ffa5c006fd0 tx=0x7ffa5c017000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:47.887 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.867+0000 7ffa47fff700 1 --2- 172.21.15.23:0/14564524 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ffa40056d60 0x7ffa40059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:47.887 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.867+0000 7ffa47fff700 1 -- 172.21.15.23:0/14564524 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7ffa5806fb90 con 0x7ffa68061a10 2020-11-20T11:47:47.887 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.867+0000 7ffa6ca5a700 1 --2- 172.21.15.23:0/14564524 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ffa40056d60 0x7ffa40059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:47.888 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.867+0000 7ffa47fff700 1 -- 172.21.15.23:0/14564524 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c025070 con 0x7ffa68146260 2020-11-20T11:47:47.888 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.867+0000 7ffa47fff700 1 -- 172.21.15.23:0/14564524 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7ffa5c010e70 con 0x7ffa68146260 2020-11-20T11:47:47.888 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.867+0000 7ffa47fff700 1 -- 172.21.15.23:0/14564524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffa68146260 msgr2=0x7ffa68146640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:47.888 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.867+0000 7ffa47fff700 1 --2- 172.21.15.23:0/14564524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffa68146260 0x7ffa68146640 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7ffa5c006fd0 tx=0x7ffa5c017000).stop 2020-11-20T11:47:47.888 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.867+0000 7ffa47fff700 1 -- 172.21.15.23:0/14564524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffa68146260 msgr2=0x7ffa68146640 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:47:47.889 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.867+0000 7ffa47fff700 1 --2- 172.21.15.23:0/14564524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffa68146260 0x7ffa68146640 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:47.889 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.868+0000 7ffa6ca5a700 1 --2- 172.21.15.23:0/14564524 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ffa40056d60 0x7ffa40059220 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7ffa60004290 tx=0x7ffa60007ef0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:47.891 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.881+0000 7ffa6e4bd700 1 --2- 172.21.15.23:0/14564524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffa68146b30 0x7ffa681497e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:47.892 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.881+0000 7ffa6e4bd700 1 -- 172.21.15.23:0/14564524 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa68067430 con 0x7ffa68146b30 2020-11-20T11:47:47.892 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.881+0000 7ffa677fe700 1 --2- 172.21.15.23:0/14564524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffa68146b30 0x7ffa681497e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:47.892 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.881+0000 7ffa677fe700 1 -- 172.21.15.23:0/14564524 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ffa6805c520 con 0x7ffa68146b30 2020-11-20T11:47:47.892 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.881+0000 7ffa677fe700 1 --2- 172.21.15.23:0/14564524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffa68146b30 0x7ffa681497e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7ffa68026f40 tx=0x7ffa5c025000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:47.893 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.882+0000 7ffa47fff700 1 -- 172.21.15.23:0/14564524 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c020c60 con 0x7ffa68146b30 2020-11-20T11:47:47.893 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.882+0000 7ffa47fff700 1 -- 172.21.15.23:0/14564524 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4785 (secure 0 0 0) 0x7ffa5c020450 con 0x7ffa68146b30 2020-11-20T11:47:47.894 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.882+0000 7ffa47fff700 1 -- 172.21.15.23:0/14564524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffa68146b30 msgr2=0x7ffa681497e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:47.894 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.882+0000 7ffa47fff700 1 --2- 172.21.15.23:0/14564524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffa68146b30 0x7ffa681497e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7ffa68026f40 tx=0x7ffa5c025000).stop 2020-11-20T11:47:47.894 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.882+0000 7ffa47fff700 1 -- 172.21.15.23:0/14564524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffa68146b30 msgr2=0x7ffa681497e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:47:47.894 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.882+0000 7ffa47fff700 1 --2- 172.21.15.23:0/14564524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffa68146b30 0x7ffa681497e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:47.895 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.882+0000 7ffa6e4bd700 1 -- 172.21.15.23:0/14564524 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ffa40056d60 msgr2=0x7ffa40059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:47.895 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.882+0000 7ffa6e4bd700 1 --2- 172.21.15.23:0/14564524 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ffa40056d60 0x7ffa40059220 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7ffa60004290 tx=0x7ffa60007ef0).stop 2020-11-20T11:47:47.895 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.882+0000 7ffa6e4bd700 1 -- 172.21.15.23:0/14564524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa68061a10 msgr2=0x7ffa6809e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:47.895 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.882+0000 7ffa6e4bd700 1 --2- 172.21.15.23:0/14564524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa68061a10 0x7ffa6809e1c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7ffa5800b790 tx=0x7ffa580091a0).stop 2020-11-20T11:47:47.895 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.882+0000 7ffa6ca5a700 1 -- 172.21.15.23:0/14564524 reap_dead start 2020-11-20T11:47:47.896 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.882+0000 7ffa6ca5a700 1 -- 172.21.15.23:0/14564524 reap_dead start 2020-11-20T11:47:47.896 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.882+0000 7ffa6e4bd700 1 -- 172.21.15.23:0/14564524 shutdown_connections 2020-11-20T11:47:47.896 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.882+0000 7ffa6e4bd700 1 -- 172.21.15.23:0/14564524 >> 172.21.15.23:0/14564524 conn(0x7ffa68060c80 msgr2=0x7ffa680a77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:47:47.896 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.882+0000 7ffa6e4bd700 1 -- 172.21.15.23:0/14564524 shutdown_connections 2020-11-20T11:47:47.896 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:47.882+0000 7ffa6e4bd700 1 -- 172.21.15.23:0/14564524 wait complete. 2020-11-20T11:47:47.897 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:47:47.897 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:47:47.897 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:47:47.897 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:47:47.898 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 138, 2020-11-20T11:47:47.898 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:47:47.898 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:47:47.898 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:47:47.898 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:47:47.899 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:47.899 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 14, 2020-11-20T11:47:47.899 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:47:47.899 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:47:47.899 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:47:47.900 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:47:47.900 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:47:47.900 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:47:47.900 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:47:47.900 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:47:47.901 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:47:47.901 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:47:47.901 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:47:47.901 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:47.901 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:47:47.902 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:47:47.902 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:47:47.902 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:47:47.902 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:47:47.902 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:47:47.903 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:47:47.903 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:47:47.903 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:47:47.903 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:47:47.904 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:47.904 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.904 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:47:47.904 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:47:47.905 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:47:47.905 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:47:47.905 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:47:47.905 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:47:47.905 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:47:47.906 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:47:47.906 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:47:47.906 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:47:47.906 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:47:47.906 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:47:47.907 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:47:47.907 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:47:47.907 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:47:47.907 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:47:47.907 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:47:47.908 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:47:47.908 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:47:47.908 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:47:47.908 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:47:47.908 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:47:47.909 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:47:47.909 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:47.909 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:47:47.909 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.909 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:47:47.910 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.910 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:47:47.910 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:47:47.910 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:47:47.910 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:47:47.911 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.911 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:47:47.911 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:47:47.911 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:47.911 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.912 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.912 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:47:47.912 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:47:47.912 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:47.912 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:47.913 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:47.913 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.913 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:47:47.913 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:47:47.913 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:47:47.914 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:47:47.914 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:47:47.914 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.914 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.914 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:47:47.914 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:47:47.915 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:47:47.915 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:47:47.915 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.915 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:47:47.915 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:47:47.916 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:47.916 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.916 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.916 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:47:47.917 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:47:47.917 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:47.917 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:47.918 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:47.918 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.918 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:47:47.918 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:47:47.918 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:47:47.919 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:47:47.919 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:47:47.919 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.919 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.919 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:47:47.920 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:47:47.920 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:47:47.920 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:47:47.920 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.920 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:47:47.921 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:47:47.921 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:47.921 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.922 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.922 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:47:47.922 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:47:47.922 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:47:47.922 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:47.923 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:47.923 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.923 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:47:47.923 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:47:47.923 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:47:47.924 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:47:47.924 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:47:47.924 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:47.924 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:47.924 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.925 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:47:47.925 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:47:47.925 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.925 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:47.925 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:47.926 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:47:47.926 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:47.926 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:47.926 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:47:47.926 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.927 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:47.927 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:47.927 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:47:47.927 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:47.928 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:47.928 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:47:47.928 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.928 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:47.928 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:47.929 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:47:47.929 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:47.929 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:47.929 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:47:47.929 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.930 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:47.930 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:47.930 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 3 2020-11-20T11:47:47.930 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:47.930 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:47:47.931 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:47:47.931 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:47:47.931 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:47:47.931 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:47:47.931 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:47:47.932 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:47:47.932 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:47:47.932 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:47:47.932 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:47:47.932 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:47:47.934 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2020-11-20T11:47:47.934 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2020-11-20T11:47:47.934 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2020-11-20T11:47:47.934 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2020-11-20T11:47:47.935 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2020-11-20T11:47:47.935 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T11:47:47.950 INFO:tasks.ceph.mon.a:Stopped 2020-11-20T11:47:47.950 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2020-11-20T11:47:47.951 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:47:48.104 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:48.082+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 231 slow ops, oldest is osd_op(client.4659.0:60021 4.2s0 4:5050cdd3:::10000000378.00000009:head [write 1388544~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:48.293 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.290+0000 7fa25b59e700 1 Processor -- start 2020-11-20T11:47:48.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.291+0000 7fa25b59e700 1 -- start start 2020-11-20T11:47:48.296 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.293+0000 7fa25b59e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa25c0ad1e0 0x7fa25c0a60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:48.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.293+0000 7fa25b59e700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa25c136e00 con 0x7fa25c0ad5c0 2020-11-20T11:47:48.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.293+0000 7fa25b59e700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa25c1370a0 con 0x7fa25c136960 2020-11-20T11:47:48.297 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.293+0000 7fa25b59e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c137340 con 0x7fa25c0ad1e0 2020-11-20T11:47:48.298 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.293+0000 7fa259d9b700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7fa25c0ad5c0 legacy=0x7fa25c133bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2020-11-20T11:47:48.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.297+0000 7fa25a59c700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa25c0ad1e0 0x7fa25c0a60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:48.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.297+0000 7fa25ad9d700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fa25c136960 0x7fa25c1412d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:38960/0 (socket says 172.21.15.23:38960) 2020-11-20T11:47:48.300 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.297+0000 7fa25ad9d700 1 -- 172.21.15.23:0/1138594179 learned_addr learned my addr 172.21.15.23:0/1138594179 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:47:48.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.298+0000 7fa25a59c700 1 -- 172.21.15.23:0/1138594179 >> v1:172.21.15.155:6789/0 conn(0x7fa25c136960 legacy=0x7fa25c1412d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:47:48.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.298+0000 7fa25a59c700 1 -- 172.21.15.23:0/1138594179 >> v1:172.21.15.23:6789/0 conn(0x7fa25c0ad5c0 legacy=0x7fa25c133bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:47:48.301 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.298+0000 7fa25a59c700 1 -- 172.21.15.23:0/1138594179 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25c1377e0 con 0x7fa25c0ad1e0 2020-11-20T11:47:48.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.298+0000 7fa25a59c700 1 --2- 172.21.15.23:0/1138594179 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa25c0ad1e0 0x7fa25c0a60e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fa25000cb10 tx=0x7fa250006e30).ready entity=mon.2 client_cookie=13a76fd65ca16d51 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:48.302 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.298+0000 7fa25959a700 1 -- 172.21.15.23:0/1138594179 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa250009810 con 0x7fa25c0ad1e0 2020-11-20T11:47:48.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.299+0000 7fa25959a700 1 -- 172.21.15.23:0/1138594179 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa250000e90 con 0x7fa25c0ad1e0 2020-11-20T11:47:48.303 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.299+0000 7fa25b59e700 1 -- 172.21.15.23:0/1138594179 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa25c0ad1e0 msgr2=0x7fa25c0a60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:48.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.299+0000 7fa25b59e700 1 --2- 172.21.15.23:0/1138594179 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa25c0ad1e0 0x7fa25c0a60e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fa25000cb10 tx=0x7fa250006e30).stop 2020-11-20T11:47:48.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.300+0000 7fa25b59e700 1 -- 172.21.15.23:0/1138594179 shutdown_connections 2020-11-20T11:47:48.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.300+0000 7fa25b59e700 1 --2- 172.21.15.23:0/1138594179 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa25c0ad1e0 0x7fa25c0a60e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:48.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.300+0000 7fa25b59e700 1 -- 172.21.15.23:0/1138594179 >> 172.21.15.23:0/1138594179 conn(0x7fa25c061850 msgr2=0x7fa25c061c60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:47:48.305 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.301+0000 7fa25b59e700 1 -- 172.21.15.23:0/1138594179 shutdown_connections 2020-11-20T11:47:48.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.301+0000 7fa25b59e700 1 -- 172.21.15.23:0/1138594179 wait complete. 2020-11-20T11:47:48.306 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.302+0000 7fa25b59e700 1 Processor -- start 2020-11-20T11:47:48.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.302+0000 7fa25b59e700 1 -- start start 2020-11-20T11:47:48.307 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.303+0000 7fa25b59e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa25c0ad5c0 0x7fa25c15a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:48.308 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.303+0000 7fa25b59e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa25c136960 0x7fa25c09ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:48.308 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.303+0000 7fa25b59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa25c09f340 0x7fa25c09de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:48.308 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.304+0000 7fa25ad9d700 1 -- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa25c09f340 msgr2=0x7fa25c09de60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:47:48.308 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.304+0000 7fa25ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa25c09f340 0x7fa25c09de60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:47:48.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.304+0000 7fa25a59c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa25c0ad5c0 0x7fa25c15a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:48.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.304+0000 7fa25a59c700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa25c0ad5c0 0x7fa25c15a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:39634/0 (socket says 172.21.15.23:39634) 2020-11-20T11:47:48.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.304+0000 7fa259d9b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa25c136960 0x7fa25c09ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:48.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.304+0000 7fa259d9b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa25c136960 0x7fa25c09ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:56980/0 (socket says 172.21.15.23:56980) 2020-11-20T11:47:48.309 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.304+0000 7fa25b59e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c15ab90 con 0x7fa25c09f340 2020-11-20T11:47:48.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.304+0000 7fa25a59c700 1 -- 172.21.15.23:0/3687399113 learned_addr learned my addr 172.21.15.23:0/3687399113 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:47:48.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.304+0000 7fa25b59e700 1 -- 172.21.15.23:0/3687399113 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c15ae30 con 0x7fa25c0ad5c0 2020-11-20T11:47:48.310 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.304+0000 7fa25b59e700 1 -- 172.21.15.23:0/3687399113 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c09d9c0 con 0x7fa25c136960 2020-11-20T11:47:48.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.305+0000 7fa259d9b700 1 -- 172.21.15.23:0/3687399113 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa25c0ad5c0 msgr2=0x7fa25c15a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:48.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.305+0000 7fa259d9b700 1 --2- 172.21.15.23:0/3687399113 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa25c0ad5c0 0x7fa25c15a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:48.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.305+0000 7fa259d9b700 1 -- 172.21.15.23:0/3687399113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa25c09f340 msgr2=0x7fa25c09de60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:47:48.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.305+0000 7fa259d9b700 1 --2- 172.21.15.23:0/3687399113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa25c09f340 0x7fa25c09de60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:47:48.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.305+0000 7fa259d9b700 1 -- 172.21.15.23:0/3687399113 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25c0a37a0 con 0x7fa25c136960 2020-11-20T11:47:48.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.305+0000 7fa259d9b700 1 --2- 172.21.15.23:0/3687399113 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa25c136960 0x7fa25c09ee50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fa25c05cf20 tx=0x7fa254009250).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:48.313 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.305+0000 7fa24a7fc700 1 -- 172.21.15.23:0/3687399113 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa254003ca0 con 0x7fa25c136960 2020-11-20T11:47:48.313 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.305+0000 7fa25b59e700 1 -- 172.21.15.23:0/3687399113 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa25c09e350 con 0x7fa25c136960 2020-11-20T11:47:48.313 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.305+0000 7fa25b59e700 1 -- 172.21.15.23:0/3687399113 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa25c09e7e0 con 0x7fa25c136960 2020-11-20T11:47:48.313 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.306+0000 7fa24a7fc700 1 -- 172.21.15.23:0/3687399113 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa254010070 con 0x7fa25c136960 2020-11-20T11:47:48.313 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.306+0000 7fa24a7fc700 1 -- 172.21.15.23:0/3687399113 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa254003e00 con 0x7fa25c136960 2020-11-20T11:47:48.314 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.308+0000 7fa24a7fc700 1 -- 172.21.15.23:0/3687399113 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa25401c070 con 0x7fa25c136960 2020-11-20T11:47:48.314 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.308+0000 7fa24a7fc700 1 --2- 172.21.15.23:0/3687399113 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa230056d60 0x7fa230059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:48.314 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.308+0000 7fa24a7fc700 1 -- 172.21.15.23:0/3687399113 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fa25406f2c0 con 0x7fa25c136960 2020-11-20T11:47:48.314 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.308+0000 7fa25a59c700 1 --2- 172.21.15.23:0/3687399113 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa230056d60 0x7fa230059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:48.315 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.312+0000 7fa25a59c700 1 --2- 172.21.15.23:0/3687399113 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa230056d60 0x7fa230059220 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fa250000fd0 tx=0x7fa250017000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:48.316 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.313+0000 7fa25b59e700 1 -- 172.21.15.23:0/3687399113 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa25c15aa00 con 0x7fa25c136960 2020-11-20T11:47:48.324 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.321+0000 7fa24a7fc700 1 -- 172.21.15.23:0/3687399113 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa25401a020 con 0x7fa25c136960 2020-11-20T11:47:48.739 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.735+0000 7fa25b59e700 1 -- 172.21.15.23:0/3687399113 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa25c09db80 con 0x7fa25c136960 2020-11-20T11:47:48.740 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.735+0000 7fa24a7fc700 1 -- 172.21.15.23:0/3687399113 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1479 (secure 0 0 0) 0x7fa254044440 con 0x7fa25c136960 2020-11-20T11:47:48.740 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:47:48.741 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.737+0000 7fa25b59e700 1 -- 172.21.15.23:0/3687399113 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa230056d60 msgr2=0x7fa230059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:48.741 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.737+0000 7fa25b59e700 1 --2- 172.21.15.23:0/3687399113 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa230056d60 0x7fa230059220 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fa250000fd0 tx=0x7fa250017000).stop 2020-11-20T11:47:48.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.737+0000 7fa25b59e700 1 -- 172.21.15.23:0/3687399113 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa25c136960 msgr2=0x7fa25c09ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:48.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.737+0000 7fa25b59e700 1 --2- 172.21.15.23:0/3687399113 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa25c136960 0x7fa25c09ee50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fa25c05cf20 tx=0x7fa254009250).stop 2020-11-20T11:47:48.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.737+0000 7fa25b59e700 1 -- 172.21.15.23:0/3687399113 shutdown_connections 2020-11-20T11:47:48.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.737+0000 7fa25b59e700 1 --2- 172.21.15.23:0/3687399113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa25c09f340 0x7fa25c09de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:47:48.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.737+0000 7fa25b59e700 1 --2- 172.21.15.23:0/3687399113 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa25c0ad5c0 0x7fa25c15a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:48.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.737+0000 7fa25b59e700 1 --2- 172.21.15.23:0/3687399113 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa230056d60 0x7fa230059220 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:48.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.737+0000 7fa25b59e700 1 --2- 172.21.15.23:0/3687399113 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa25c136960 0x7fa25c09ee50 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:48.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.737+0000 7fa25b59e700 1 -- 172.21.15.23:0/3687399113 >> 172.21.15.23:0/3687399113 conn(0x7fa25c061850 msgr2=0x7fa25c135f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:47:48.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.737+0000 7fa25b59e700 1 -- 172.21.15.23:0/3687399113 shutdown_connections 2020-11-20T11:47:48.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:48.737+0000 7fa25b59e700 1 -- 172.21.15.23:0/3687399113 wait complete. 2020-11-20T11:47:49.187 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:49.140+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 235 slow ops, oldest is osd_op(client.4659.0:60021 4.2s0 4:5050cdd3:::10000000378.00000009:head [write 1388544~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:50.233 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:50.213+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.4659.0:60021 4.2s0 4:5050cdd3:::10000000378.00000009:head [write 1388544~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:51.287 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:51.249+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 235 slow ops, oldest is osd_op(client.4659.0:60233 4.2s0 4:48ab2915:::100000003c2.0000000e:head [write 3649536~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:51.755 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:47:52.325 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:52.302+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 234 slow ops, oldest is osd_op(client.4659.0:61014 4.2s0 4:5196b857:::10000000386.0000000c:head [write 3321856~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:52.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:52.568+0000 7efd6aba2700 1 Processor -- start 2020-11-20T11:47:52.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:52.569+0000 7efd6aba2700 1 -- start start 2020-11-20T11:47:52.573 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:52.569+0000 7efd6aba2700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7efd640a2080 0x7efd640aef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:52.573 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:52.569+0000 7efd6aba2700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd640aaab0 con 0x7efd640a2460 2020-11-20T11:47:52.573 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:52.569+0000 7efd6aba2700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd640aad50 con 0x7efd64067a10 2020-11-20T11:47:52.574 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:52.569+0000 7efd6aba2700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd6413aa50 con 0x7efd640a2080 2020-11-20T11:47:52.574 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:52.569+0000 7efd6939f700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7efd640a2460 legacy=0x7efd64137df0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2020-11-20T11:47:52.574 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:52.569+0000 7efd69ba0700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7efd640a2080 msgr2=0x7efd640aef10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:47:52.574 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:52.569+0000 7efd69ba0700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7efd640a2080 0x7efd640aef10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:47:52.574 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:52.570+0000 7efd6a3a1700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7efd64067a10 0x7efd641453c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:38974/0 (socket says 172.21.15.23:38974) 2020-11-20T11:47:52.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:52.570+0000 7efd6a3a1700 1 -- 172.21.15.23:0/1595449254 learned_addr learned my addr 172.21.15.23:0/1595449254 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:47:52.773 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:52.770+0000 7efd69ba0700 1 -- 172.21.15.23:0/1595449254 >> v2:172.21.15.23:3300/0 conn(0x7efd640a2080 msgr2=0x7efd640aef10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:47:52.773 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:52.770+0000 7efd6939f700 1 -- 172.21.15.23:0/1595449254 >> v1:172.21.15.23:6789/0 conn(0x7efd640a2460 legacy=0x7efd64137df0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2020-11-20T11:47:52.774 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:52.770+0000 7efd69ba0700 1 --2- 172.21.15.23:0/1595449254 >> v2:172.21.15.23:3300/0 conn(0x7efd640a2080 0x7efd640aef10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:47:53.174 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:53.171+0000 7efd6939f700 1 -- 172.21.15.23:0/1595449254 >> v1:172.21.15.23:6789/0 conn(0x7efd640a2460 legacy=0x7efd64137df0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2020-11-20T11:47:53.174 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:53.171+0000 7efd69ba0700 1 -- 172.21.15.23:0/1595449254 >> v2:172.21.15.23:3300/0 conn(0x7efd640a2080 msgr2=0x7efd640aef10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:47:53.174 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:53.171+0000 7efd69ba0700 1 --2- 172.21.15.23:0/1595449254 >> v2:172.21.15.23:3300/0 conn(0x7efd640a2080 0x7efd640aef10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:47:53.340 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:53.288+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 239 slow ops, oldest is osd_op(client.4659.0:61014 4.2s0 4:5196b857:::10000000386.0000000c:head [write 3321856~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:53.975 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:53.972+0000 7efd6939f700 1 -- 172.21.15.23:0/1595449254 >> v1:172.21.15.23:6789/0 conn(0x7efd640a2460 legacy=0x7efd64137df0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2020-11-20T11:47:53.975 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:53.972+0000 7efd69ba0700 1 -- 172.21.15.23:0/1595449254 >> v2:172.21.15.23:3300/0 conn(0x7efd640a2080 msgr2=0x7efd640aef10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:47:53.975 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:53.972+0000 7efd69ba0700 1 --2- 172.21.15.23:0/1595449254 >> v2:172.21.15.23:3300/0 conn(0x7efd640a2080 0x7efd640aef10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:47:54.367 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:54.331+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 242 slow ops, oldest is osd_op(client.4659.0:61014 4.2s0 4:5196b857:::10000000386.0000000c:head [write 3321856~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:55.474 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:55.448+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 242 slow ops, oldest is osd_op(client.4659.0:61014 4.2s0 4:5196b857:::10000000386.0000000c:head [write 3321856~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:55.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.569+0000 7efd537fe700 1 -- 172.21.15.23:0/1595449254 >> v2:172.21.15.23:3300/0 conn(0x7efd640a2080 msgr2=0x7efd640aef10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:47:55.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.569+0000 7efd537fe700 1 --2- 172.21.15.23:0/1595449254 >> v2:172.21.15.23:3300/0 conn(0x7efd640a2080 0x7efd640aef10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:47:55.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.569+0000 7efd537fe700 1 -- 172.21.15.23:0/1595449254 >> v1:172.21.15.155:6789/0 conn(0x7efd64067a10 legacy=0x7efd641453c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:55.576 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.569+0000 7efd537fe700 1 -- 172.21.15.23:0/1595449254 >> v1:172.21.15.23:6789/0 conn(0x7efd640a2460 legacy=0x7efd64137df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:47:55.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.569+0000 7efd537fe700 1 --2- 172.21.15.23:0/1595449254 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd440033f0 0x7efd44005840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:55.577 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.569+0000 7efd537fe700 1 --2- 172.21.15.23:0/1595449254 >> v2:172.21.15.155:3300/0 conn(0x7efd44005e80 0x7efd44008320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:55.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.569+0000 7efd537fe700 1 -- 172.21.15.23:0/1595449254 --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd44008960 con 0x7efd44005e80 2020-11-20T11:47:55.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.569+0000 7efd537fe700 1 -- 172.21.15.23:0/1595449254 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd44008ad0 con 0x7efd440033f0 2020-11-20T11:47:55.578 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.570+0000 7efd6939f700 1 --2- 172.21.15.23:0/1595449254 >> v2:172.21.15.155:3300/0 conn(0x7efd44005e80 0x7efd44008320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:55.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.570+0000 7efd69ba0700 1 --2- 172.21.15.23:0/1595449254 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd440033f0 0x7efd44005840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:55.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.570+0000 7efd6939f700 1 -- 172.21.15.23:0/1595449254 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd440033f0 msgr2=0x7efd44005840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:47:55.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.570+0000 7efd6939f700 1 --2- 172.21.15.23:0/1595449254 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd440033f0 0x7efd44005840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:55.579 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.570+0000 7efd6939f700 1 -- 172.21.15.23:0/1595449254 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd6413b920 con 0x7efd44005e80 2020-11-20T11:47:55.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.570+0000 7efd69ba0700 1 --2- 172.21.15.23:0/1595449254 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd440033f0 0x7efd44005840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:47:55.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.570+0000 7efd6939f700 1 --2- 172.21.15.23:0/1595449254 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd44005e80 0x7efd44008320 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7efd60006fd0 tx=0x7efd600091a0).ready entity=mon.1 client_cookie=10679f29c469a4d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:55.581 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.571+0000 7efd68b9e700 1 -- 172.21.15.23:0/1595449254 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd60003b40 con 0x7efd44005e80 2020-11-20T11:47:55.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.571+0000 7efd68b9e700 1 -- 172.21.15.23:0/1595449254 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd60003ca0 con 0x7efd44005e80 2020-11-20T11:47:55.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.571+0000 7efd68b9e700 1 -- 172.21.15.23:0/1595449254 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd60005320 con 0x7efd44005e80 2020-11-20T11:47:55.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.572+0000 7efd6aba2700 1 -- 172.21.15.23:0/1595449254 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd44005e80 msgr2=0x7efd44008320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:55.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.572+0000 7efd6aba2700 1 --2- 172.21.15.23:0/1595449254 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd44005e80 0x7efd44008320 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7efd60006fd0 tx=0x7efd600091a0).stop 2020-11-20T11:47:55.582 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.572+0000 7efd6a3a1700 1 -- 172.21.15.23:0/1595449254 reap_dead start 2020-11-20T11:47:55.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.572+0000 7efd6aba2700 1 -- 172.21.15.23:0/1595449254 shutdown_connections 2020-11-20T11:47:55.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.572+0000 7efd6aba2700 1 --2- 172.21.15.23:0/1595449254 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd44005e80 0x7efd44008320 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:55.583 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.572+0000 7efd6aba2700 1 -- 172.21.15.23:0/1595449254 >> 172.21.15.23:0/1595449254 conn(0x7efd64063980 msgr2=0x7efd64063d90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:47:55.585 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.573+0000 7efd6aba2700 1 -- 172.21.15.23:0/1595449254 shutdown_connections 2020-11-20T11:47:55.585 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.573+0000 7efd6aba2700 1 -- 172.21.15.23:0/1595449254 wait complete. 2020-11-20T11:47:55.585 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.574+0000 7efd6aba2700 1 Processor -- start 2020-11-20T11:47:55.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.575+0000 7efd6aba2700 1 -- start start 2020-11-20T11:47:55.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.576+0000 7efd6aba2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd44005e80 0x7efd640ae020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:55.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.576+0000 7efd6aba2700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd640a7860 0x7efd6413e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:55.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.576+0000 7efd6aba2700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd6413d0c0 0x7efd6413e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:55.586 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.576+0000 7efd6aba2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd6413ea90 con 0x7efd44005e80 2020-11-20T11:47:55.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.576+0000 7efd6aba2700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd6413d640 con 0x7efd640a7860 2020-11-20T11:47:55.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.576+0000 7efd6aba2700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd6413d8e0 con 0x7efd6413d0c0 2020-11-20T11:47:55.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.576+0000 7efd69ba0700 1 -- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd44005e80 msgr2=0x7efd640ae020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:47:55.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.576+0000 7efd69ba0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd44005e80 0x7efd640ae020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:47:55.587 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.576+0000 7efd6a3a1700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd6413d0c0 0x7efd6413e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:55.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.576+0000 7efd6a3a1700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd6413d0c0 0x7efd6413e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:57008/0 (socket says 172.21.15.23:57008) 2020-11-20T11:47:55.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.576+0000 7efd6a3a1700 1 -- 172.21.15.23:0/746635910 learned_addr learned my addr 172.21.15.23:0/746635910 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:47:55.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.576+0000 7efd6a3a1700 1 -- 172.21.15.23:0/746635910 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd640a7860 msgr2=0x7efd6413e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:55.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.576+0000 7efd6a3a1700 1 --2- 172.21.15.23:0/746635910 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd640a7860 0x7efd6413e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:47:55.588 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.576+0000 7efd6a3a1700 1 -- 172.21.15.23:0/746635910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd44005e80 msgr2=0x7efd640ae020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:47:55.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.576+0000 7efd6a3a1700 1 --2- 172.21.15.23:0/746635910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd44005e80 0x7efd640ae020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:47:55.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.576+0000 7efd6a3a1700 1 -- 172.21.15.23:0/746635910 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd6413db80 con 0x7efd6413d0c0 2020-11-20T11:47:55.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.577+0000 7efd6a3a1700 1 --2- 172.21.15.23:0/746635910 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd6413d0c0 0x7efd6413e5a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7efd640aff80 tx=0x7efd5c007e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:55.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.577+0000 7efd51ffb700 1 -- 172.21.15.23:0/746635910 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5c0020b0 con 0x7efd6413d0c0 2020-11-20T11:47:55.589 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.578+0000 7efd6aba2700 1 -- 172.21.15.23:0/746635910 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd640a92c0 con 0x7efd6413d0c0 2020-11-20T11:47:55.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.578+0000 7efd6aba2700 1 -- 172.21.15.23:0/746635910 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd64133510 con 0x7efd6413d0c0 2020-11-20T11:47:55.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.578+0000 7efd51ffb700 1 -- 172.21.15.23:0/746635910 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd5c005bc0 con 0x7efd6413d0c0 2020-11-20T11:47:55.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.579+0000 7efd51ffb700 1 -- 172.21.15.23:0/746635910 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5c0070d0 con 0x7efd6413d0c0 2020-11-20T11:47:55.590 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.580+0000 7efd51ffb700 1 -- 172.21.15.23:0/746635910 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7efd5c01a060 con 0x7efd6413d0c0 2020-11-20T11:47:55.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.580+0000 7efd6aba2700 1 -- 172.21.15.23:0/746635910 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd64057dc0 con 0x7efd6413d0c0 2020-11-20T11:47:55.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.581+0000 7efd51ffb700 1 --2- 172.21.15.23:0/746635910 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efd48056d90 0x7efd48059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:55.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.581+0000 7efd51ffb700 1 -- 172.21.15.23:0/746635910 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7efd5c06f7d0 con 0x7efd6413d0c0 2020-11-20T11:47:55.591 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.582+0000 7efd69ba0700 1 --2- 172.21.15.23:0/746635910 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efd48056d90 0x7efd48059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:55.592 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.582+0000 7efd69ba0700 1 --2- 172.21.15.23:0/746635910 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efd48056d90 0x7efd48059250 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7efd54003610 tx=0x7efd540090f0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:47:55.594 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.591+0000 7efd51ffb700 1 -- 172.21.15.23:0/746635910 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7efd5c044a30 con 0x7efd6413d0c0 2020-11-20T11:47:55.931 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.928+0000 7efd6aba2700 1 -- 172.21.15.23:0/746635910 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7efd64044710 con 0x7efd6413d0c0 2020-11-20T11:47:55.932 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.929+0000 7efd51ffb700 1 -- 172.21.15.23:0/746635910 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1479 (secure 0 0 0) 0x7efd5c018070 con 0x7efd6413d0c0 2020-11-20T11:47:55.932 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:47:55.934 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.931+0000 7efd6aba2700 1 -- 172.21.15.23:0/746635910 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efd48056d90 msgr2=0x7efd48059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:55.934 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.931+0000 7efd6aba2700 1 --2- 172.21.15.23:0/746635910 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efd48056d90 0x7efd48059250 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7efd54003610 tx=0x7efd540090f0).stop 2020-11-20T11:47:55.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.931+0000 7efd6aba2700 1 -- 172.21.15.23:0/746635910 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd6413d0c0 msgr2=0x7efd6413e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:47:55.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.931+0000 7efd6aba2700 1 --2- 172.21.15.23:0/746635910 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd6413d0c0 0x7efd6413e5a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7efd640aff80 tx=0x7efd5c007e60).stop 2020-11-20T11:47:55.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.931+0000 7efd6aba2700 1 -- 172.21.15.23:0/746635910 shutdown_connections 2020-11-20T11:47:55.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.931+0000 7efd6aba2700 1 --2- 172.21.15.23:0/746635910 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efd48056d90 0x7efd48059250 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:55.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.931+0000 7efd6aba2700 1 --2- 172.21.15.23:0/746635910 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efd6413d0c0 0x7efd6413e5a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:47:55.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.931+0000 7efd6aba2700 1 --2- 172.21.15.23:0/746635910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd44005e80 0x7efd640ae020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:47:55.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.931+0000 7efd6aba2700 1 --2- 172.21.15.23:0/746635910 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efd640a7860 0x7efd6413e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:47:55.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.931+0000 7efd6aba2700 1 -- 172.21.15.23:0/746635910 >> 172.21.15.23:0/746635910 conn(0x7efd64063980 msgr2=0x7efd64068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:47:55.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.931+0000 7efd6aba2700 1 -- 172.21.15.23:0/746635910 shutdown_connections 2020-11-20T11:47:55.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:55.931+0000 7efd6aba2700 1 -- 172.21.15.23:0/746635910 wait complete. 2020-11-20T11:47:56.577 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:56.556+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 247 slow ops, oldest is osd_op(client.4659.0:61014 4.2s0 4:5196b857:::10000000386.0000000c:head [write 3321856~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:57.622 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:57.618+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 246 slow ops, oldest is osd_op(client.4659.0:61014 4.2s0 4:5196b857:::10000000386.0000000c:head [write 3321856~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:58.677 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:58.610+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.4659.0:61611 4.2s0 4:4fa56316:::100000003a1.0000000f:head [write 4071424~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:58.947 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:47:59.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:59.098+0000 7efdd50fb700 1 Processor -- start 2020-11-20T11:47:59.103 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:59.099+0000 7efdd50fb700 1 -- start start 2020-11-20T11:47:59.104 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:59.100+0000 7efdd50fb700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7efdd00ad7c0 0x7efdd00a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:47:59.104 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:59.100+0000 7efdd50fb700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd00a6c60 con 0x7efdd00a67d0 2020-11-20T11:47:59.104 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:59.100+0000 7efdd50fb700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd00a3cc0 con 0x7efdd0063fe0 2020-11-20T11:47:59.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:59.100+0000 7efdd50fb700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdd0062670 con 0x7efdd00ad7c0 2020-11-20T11:47:59.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:59.100+0000 7efdcf59e700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7efdd00a67d0 legacy=0x7efdd0061f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2020-11-20T11:47:59.107 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:59.103+0000 7efdced9d700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7efdd00ad7c0 0x7efdd00a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:59.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:59.104+0000 7efdced9d700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7efdd00ad7c0 0x7efdd00a62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:39672/0 (socket says 172.21.15.23:39672) 2020-11-20T11:47:59.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:59.104+0000 7efdced9d700 1 -- 172.21.15.23:0/62686311 learned_addr learned my addr 172.21.15.23:0/62686311 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:47:59.108 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:59.104+0000 7efdced9d700 1 -- 172.21.15.23:0/62686311 >> v2:172.21.15.155:3300/0 conn(0x7efdd00ad7c0 msgr2=0x7efdd00a62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:47:59.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:59.104+0000 7efdced9d700 1 -- 172.21.15.23:0/62686311 >> v2:172.21.15.155:3300/0 conn(0x7efdd00ad7c0 msgr2=0x7efdd00a62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:47:59.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:59.104+0000 7efdced9d700 1 --2- 172.21.15.23:0/62686311 >> v2:172.21.15.155:3300/0 conn(0x7efdd00ad7c0 0x7efdd00a62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:47:59.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:59.104+0000 7efdced9d700 1 --2- 172.21.15.23:0/62686311 >> v2:172.21.15.155:3300/0 conn(0x7efdd00ad7c0 0x7efdd00a62e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:47:59.304 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:59.301+0000 7efdcf59e700 1 -- 172.21.15.23:0/62686311 >> v1:172.21.15.23:6789/0 conn(0x7efdd00a67d0 legacy=0x7efdd0061f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2020-11-20T11:47:59.311 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:59.305+0000 7efdced9d700 1 --2- 172.21.15.23:0/62686311 >> v2:172.21.15.155:3300/0 conn(0x7efdd00ad7c0 0x7efdd00a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:59.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:59.305+0000 7efdced9d700 1 -- 172.21.15.23:0/62686311 >> v2:172.21.15.155:3300/0 conn(0x7efdd00ad7c0 msgr2=0x7efdd00a62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:47:59.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:59.305+0000 7efdced9d700 1 -- 172.21.15.23:0/62686311 >> v2:172.21.15.155:3300/0 conn(0x7efdd00ad7c0 msgr2=0x7efdd00a62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:47:59.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:59.305+0000 7efdced9d700 1 --2- 172.21.15.23:0/62686311 >> v2:172.21.15.155:3300/0 conn(0x7efdd00ad7c0 0x7efdd00a62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:47:59.312 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:59.305+0000 7efdced9d700 1 --2- 172.21.15.23:0/62686311 >> v2:172.21.15.155:3300/0 conn(0x7efdd00ad7c0 0x7efdd00a62e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:47:59.694 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:47:59.671+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4659.0:61611 4.2s0 4:4fa56316:::100000003a1.0000000f:head [write 4071424~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:47:59.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:59.702+0000 7efdcf59e700 1 -- 172.21.15.23:0/62686311 >> v1:172.21.15.23:6789/0 conn(0x7efdd00a67d0 legacy=0x7efdd0061f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2020-11-20T11:47:59.709 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:59.706+0000 7efdced9d700 1 --2- 172.21.15.23:0/62686311 >> v2:172.21.15.155:3300/0 conn(0x7efdd00ad7c0 0x7efdd00a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:47:59.710 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:59.706+0000 7efdced9d700 1 -- 172.21.15.23:0/62686311 >> v2:172.21.15.155:3300/0 conn(0x7efdd00ad7c0 msgr2=0x7efdd00a62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:47:59.710 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:59.706+0000 7efdced9d700 1 -- 172.21.15.23:0/62686311 >> v2:172.21.15.155:3300/0 conn(0x7efdd00ad7c0 msgr2=0x7efdd00a62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:47:59.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:59.706+0000 7efdced9d700 1 --2- 172.21.15.23:0/62686311 >> v2:172.21.15.155:3300/0 conn(0x7efdd00ad7c0 0x7efdd00a62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:47:59.711 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:47:59.706+0000 7efdced9d700 1 --2- 172.21.15.23:0/62686311 >> v2:172.21.15.155:3300/0 conn(0x7efdd00ad7c0 0x7efdd00a62e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:48:00.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:00.503+0000 7efdcf59e700 1 -- 172.21.15.23:0/62686311 >> v1:172.21.15.23:6789/0 conn(0x7efdd00a67d0 legacy=0x7efdd0061f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2020-11-20T11:48:00.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:00.507+0000 7efdced9d700 1 --2- 172.21.15.23:0/62686311 >> v2:172.21.15.155:3300/0 conn(0x7efdd00ad7c0 0x7efdd00a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:00.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:00.507+0000 7efdced9d700 1 -- 172.21.15.23:0/62686311 >> v2:172.21.15.155:3300/0 conn(0x7efdd00ad7c0 msgr2=0x7efdd00a62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:48:00.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:00.507+0000 7efdced9d700 1 -- 172.21.15.23:0/62686311 >> v2:172.21.15.155:3300/0 conn(0x7efdd00ad7c0 msgr2=0x7efdd00a62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:48:00.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:00.507+0000 7efdced9d700 1 --2- 172.21.15.23:0/62686311 >> v2:172.21.15.155:3300/0 conn(0x7efdd00ad7c0 0x7efdd00a62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:48:00.511 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:00.507+0000 7efdced9d700 1 --2- 172.21.15.23:0/62686311 >> v2:172.21.15.155:3300/0 conn(0x7efdd00ad7c0 0x7efdd00a62e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:48:00.779 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:00.731+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 251 slow ops, oldest is osd_op(client.4659.0:61611 4.2s0 4:4fa56316:::100000003a1.0000000f:head [write 4071424~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:01.742 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:01.701+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 250 slow ops, oldest is osd_op(client.4659.0:61611 4.2s0 4:4fa56316:::100000003a1.0000000f:head [write 4071424~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:02.072 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.069+0000 7efdcdd9b700 1 -- 172.21.15.23:0/62686311 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3574461203 0 0) 0x7efdd00a3cc0 con 0x7efdd0063fe0 2020-11-20T11:48:02.073 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.069+0000 7efdcdd9b700 1 -- 172.21.15.23:0/62686311 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdbc003280 con 0x7efdd0063fe0 2020-11-20T11:48:02.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.076+0000 7efdcdd9b700 1 -- 172.21.15.23:0/62686311 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3368987336 0 0) 0x7efdbc003280 con 0x7efdd0063fe0 2020-11-20T11:48:02.089 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.076+0000 7efdcdd9b700 1 -- 172.21.15.23:0/62686311 >> v2:172.21.15.155:3300/0 conn(0x7efdd00ad7c0 msgr2=0x7efdd00a62e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:48:02.089 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.076+0000 7efdcdd9b700 1 --2- 172.21.15.23:0/62686311 >> v2:172.21.15.155:3300/0 conn(0x7efdd00ad7c0 0x7efdd00a62e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:02.089 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.076+0000 7efdcdd9b700 1 -- 172.21.15.23:0/62686311 >> v1:172.21.15.23:6789/0 conn(0x7efdd00a67d0 legacy=0x7efdd0061f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:48:02.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.076+0000 7efdcdd9b700 1 -- 172.21.15.23:0/62686311 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd00a4640 con 0x7efdd0063fe0 2020-11-20T11:48:02.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.076+0000 7efdcdd9b700 1 -- 172.21.15.23:0/62686311 <== mon.1 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7efdc0001500 con 0x7efdd0063fe0 2020-11-20T11:48:02.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.076+0000 7efdcdd9b700 1 -- 172.21.15.23:0/62686311 >> v1:172.21.15.155:6789/0 conn(0x7efdd0063fe0 legacy=0x7efdd0061850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:02.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.076+0000 7efdcdd9b700 1 --2- 172.21.15.23:0/62686311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efdbc003770 0x7efdbc003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:02.090 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.077+0000 7efdcdd9b700 1 --2- 172.21.15.23:0/62686311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdbc004080 0x7efdbc004560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:02.091 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.077+0000 7efdcdd9b700 1 --2- 172.21.15.23:0/62686311 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdbc004b60 0x7efdbc005010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:02.091 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.077+0000 7efdcdd9b700 1 -- 172.21.15.23:0/62686311 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd0062670 con 0x7efdbc004080 2020-11-20T11:48:02.091 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.077+0000 7efdcdd9b700 1 -- 172.21.15.23:0/62686311 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdbc00b8a0 con 0x7efdbc003770 2020-11-20T11:48:02.091 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.077+0000 7efdcdd9b700 1 -- 172.21.15.23:0/62686311 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdbc00ba60 con 0x7efdbc004b60 2020-11-20T11:48:02.091 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.077+0000 7efdce59c700 1 -- 172.21.15.23:0/62686311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdbc004080 msgr2=0x7efdbc004560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:48:02.092 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.077+0000 7efdce59c700 1 --2- 172.21.15.23:0/62686311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdbc004080 0x7efdbc004560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:48:02.092 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.077+0000 7efdced9d700 1 --2- 172.21.15.23:0/62686311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efdbc003770 0x7efdbc003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:02.092 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.077+0000 7efdcf59e700 1 --2- 172.21.15.23:0/62686311 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdbc004b60 0x7efdbc005010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:02.092 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.077+0000 7efdced9d700 1 -- 172.21.15.23:0/62686311 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdbc004b60 msgr2=0x7efdbc005010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:48:02.092 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.077+0000 7efdced9d700 1 --2- 172.21.15.23:0/62686311 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdbc004b60 0x7efdbc005010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:02.093 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.077+0000 7efdced9d700 1 -- 172.21.15.23:0/62686311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdbc004080 msgr2=0x7efdbc004560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:48:02.093 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.077+0000 7efdced9d700 1 --2- 172.21.15.23:0/62686311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdbc004080 0x7efdbc004560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:48:02.093 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.077+0000 7efdced9d700 1 -- 172.21.15.23:0/62686311 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdbc003280 con 0x7efdbc003770 2020-11-20T11:48:02.094 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.077+0000 7efdcf59e700 1 -- 172.21.15.23:0/62686311 reap_dead start 2020-11-20T11:48:02.094 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.077+0000 7efdced9d700 1 --2- 172.21.15.23:0/62686311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efdbc003770 0x7efdbc003b70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7efdc8002de0 tx=0x7efdc8008040).ready entity=mon.1 client_cookie=b381258e798f8c02 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:02.094 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.078+0000 7efdcdd9b700 1 -- 172.21.15.23:0/62686311 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc8013d70 con 0x7efdbc003770 2020-11-20T11:48:02.094 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.078+0000 7efdcdd9b700 1 -- 172.21.15.23:0/62686311 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdc8013ed0 con 0x7efdbc003770 2020-11-20T11:48:02.094 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.078+0000 7efdcdd9b700 1 -- 172.21.15.23:0/62686311 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc8016690 con 0x7efdbc003770 2020-11-20T11:48:02.095 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.079+0000 7efdd50fb700 1 -- 172.21.15.23:0/62686311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efdbc003770 msgr2=0x7efdbc003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:02.095 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.079+0000 7efdd50fb700 1 --2- 172.21.15.23:0/62686311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efdbc003770 0x7efdbc003b70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7efdc8002de0 tx=0x7efdc8008040).stop 2020-11-20T11:48:02.095 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.079+0000 7efdd50fb700 1 -- 172.21.15.23:0/62686311 shutdown_connections 2020-11-20T11:48:02.095 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.079+0000 7efdd50fb700 1 --2- 172.21.15.23:0/62686311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efdbc003770 0x7efdbc003b70 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:02.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.079+0000 7efdd50fb700 1 -- 172.21.15.23:0/62686311 >> 172.21.15.23:0/62686311 conn(0x7efdd0060c80 msgr2=0x7efdd00acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:48:02.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.079+0000 7efdd50fb700 1 -- 172.21.15.23:0/62686311 shutdown_connections 2020-11-20T11:48:02.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.079+0000 7efdd50fb700 1 -- 172.21.15.23:0/62686311 wait complete. 2020-11-20T11:48:02.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.080+0000 7efdd50fb700 1 Processor -- start 2020-11-20T11:48:02.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.080+0000 7efdd50fb700 1 -- start start 2020-11-20T11:48:02.097 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.081+0000 7efdd50fb700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdbc003770 0x7efdd0066480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:02.097 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.081+0000 7efdd50fb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdd0142c10 0x7efdd0151c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:02.097 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.081+0000 7efdd50fb700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efdd0152100 0x7efdd0152560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:02.097 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.081+0000 7efdd50fb700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd0152a50 con 0x7efdd0142c10 2020-11-20T11:48:02.097 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.081+0000 7efdd50fb700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd0065aa0 con 0x7efdd0152100 2020-11-20T11:48:02.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.081+0000 7efdd50fb700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdd0065d40 con 0x7efdbc003770 2020-11-20T11:48:02.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.081+0000 7efdce59c700 1 -- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdd0142c10 msgr2=0x7efdd0151c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:48:02.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.081+0000 7efdced9d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdbc003770 0x7efdd0066480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:02.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.081+0000 7efdce59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdd0142c10 0x7efdd0151c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:48:02.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.081+0000 7efdced9d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdbc003770 0x7efdd0066480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:57040/0 (socket says 172.21.15.23:57040) 2020-11-20T11:48:02.099 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.081+0000 7efdced9d700 1 -- 172.21.15.23:0/3145564758 learned_addr learned my addr 172.21.15.23:0/3145564758 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:48:02.099 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.082+0000 7efdcf59e700 1 --2- 172.21.15.23:0/3145564758 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efdd0152100 0x7efdd0152560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:02.099 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.082+0000 7efdced9d700 1 -- 172.21.15.23:0/3145564758 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efdd0152100 msgr2=0x7efdd0152560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:02.099 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.082+0000 7efdced9d700 1 --2- 172.21.15.23:0/3145564758 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efdd0152100 0x7efdd0152560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:02.099 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.082+0000 7efdced9d700 1 -- 172.21.15.23:0/3145564758 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdd0142c10 msgr2=0x7efdd0151c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:48:02.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.082+0000 7efdced9d700 1 --2- 172.21.15.23:0/3145564758 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdd0142c10 0x7efdd0151c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:48:02.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.082+0000 7efdced9d700 1 -- 172.21.15.23:0/3145564758 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd0065fe0 con 0x7efdbc003770 2020-11-20T11:48:02.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.082+0000 7efdced9d700 1 --2- 172.21.15.23:0/3145564758 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdbc003770 0x7efdd0066480 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7efdc80026c0 tx=0x7efdc8008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:02.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.082+0000 7efdb6ffd700 1 -- 172.21.15.23:0/3145564758 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc8025070 con 0x7efdbc003770 2020-11-20T11:48:02.100 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.082+0000 7efdb6ffd700 1 -- 172.21.15.23:0/3145564758 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdc8016e60 con 0x7efdbc003770 2020-11-20T11:48:02.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.082+0000 7efdb6ffd700 1 -- 172.21.15.23:0/3145564758 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc8013d70 con 0x7efdbc003770 2020-11-20T11:48:02.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.082+0000 7efdd50fb700 1 -- 172.21.15.23:0/3145564758 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdd01399a0 con 0x7efdbc003770 2020-11-20T11:48:02.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.082+0000 7efdd50fb700 1 -- 172.21.15.23:0/3145564758 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdd0066b90 con 0x7efdbc003770 2020-11-20T11:48:02.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.083+0000 7efdd50fb700 1 -- 172.21.15.23:0/3145564758 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdd0061650 con 0x7efdbc003770 2020-11-20T11:48:02.101 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.083+0000 7efdb6ffd700 1 -- 172.21.15.23:0/3145564758 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7efdc8030030 con 0x7efdbc003770 2020-11-20T11:48:02.102 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.084+0000 7efdb6ffd700 1 --2- 172.21.15.23:0/3145564758 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efdac056d10 0x7efdac0591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:02.102 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.084+0000 7efdb6ffd700 1 -- 172.21.15.23:0/3145564758 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7efdc8072400 con 0x7efdbc003770 2020-11-20T11:48:02.102 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.084+0000 7efdce59c700 1 --2- 172.21.15.23:0/3145564758 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efdac056d10 0x7efdac0591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:02.102 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.084+0000 7efdce59c700 1 --2- 172.21.15.23:0/3145564758 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efdac056d10 0x7efdac0591d0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7efdc0001050 tx=0x7efdd00a47d0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:02.103 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.088+0000 7efdb6ffd700 1 -- 172.21.15.23:0/3145564758 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7efdc8002e10 con 0x7efdbc003770 2020-11-20T11:48:02.381 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.378+0000 7efdd50fb700 1 -- 172.21.15.23:0/3145564758 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7efdd005c520 con 0x7efdbc003770 2020-11-20T11:48:02.382 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.378+0000 7efdb6ffd700 1 -- 172.21.15.23:0/3145564758 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1472 (secure 0 0 0) 0x7efdc801a070 con 0x7efdbc003770 2020-11-20T11:48:02.382 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":140,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:48:02.384 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.380+0000 7efdd50fb700 1 -- 172.21.15.23:0/3145564758 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efdac056d10 msgr2=0x7efdac0591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:02.384 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.380+0000 7efdd50fb700 1 --2- 172.21.15.23:0/3145564758 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efdac056d10 0x7efdac0591d0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7efdc0001050 tx=0x7efdd00a47d0).stop 2020-11-20T11:48:02.385 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.380+0000 7efdd50fb700 1 -- 172.21.15.23:0/3145564758 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdbc003770 msgr2=0x7efdd0066480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:02.385 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.380+0000 7efdd50fb700 1 --2- 172.21.15.23:0/3145564758 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdbc003770 0x7efdd0066480 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7efdc80026c0 tx=0x7efdc8008040).stop 2020-11-20T11:48:02.386 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.380+0000 7efdd50fb700 1 -- 172.21.15.23:0/3145564758 shutdown_connections 2020-11-20T11:48:02.386 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.380+0000 7efdd50fb700 1 --2- 172.21.15.23:0/3145564758 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efdd0152100 0x7efdd0152560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:02.386 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.380+0000 7efdd50fb700 1 --2- 172.21.15.23:0/3145564758 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efdac056d10 0x7efdac0591d0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:02.387 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.380+0000 7efdd50fb700 1 --2- 172.21.15.23:0/3145564758 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efdbc003770 0x7efdd0066480 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:02.387 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.380+0000 7efdd50fb700 1 --2- 172.21.15.23:0/3145564758 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdd0142c10 0x7efdd0151c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:48:02.387 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.380+0000 7efdd50fb700 1 -- 172.21.15.23:0/3145564758 >> 172.21.15.23:0/3145564758 conn(0x7efdd0060c80 msgr2=0x7efdd013f160 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:48:02.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.380+0000 7efdd50fb700 1 -- 172.21.15.23:0/3145564758 shutdown_connections 2020-11-20T11:48:02.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.380+0000 7efdd50fb700 1 -- 172.21.15.23:0/3145564758 wait complete. 2020-11-20T11:48:02.396 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2020-11-20T11:48:02.396 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:48:02.551 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.542+0000 7f5595567700 1 Processor -- start 2020-11-20T11:48:02.551 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.542+0000 7f5595567700 1 -- start start 2020-11-20T11:48:02.551 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.543+0000 7f5595567700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f559007fbd0 0x7f559014fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:02.552 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.543+0000 7f5595567700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5590063fe0 0x7f55901521c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:02.552 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.543+0000 7f5595567700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55900a4330 con 0x7f559007f7f0 2020-11-20T11:48:02.552 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.543+0000 7f5595567700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55900a4560 con 0x7f559007fbd0 2020-11-20T11:48:02.552 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.543+0000 7f5595567700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55900ad830 con 0x7f5590063fe0 2020-11-20T11:48:02.553 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.543+0000 7f558e7fc700 1 -- >> v2:172.21.15.23:3300/0 conn(0x7f559007fbd0 msgr2=0x7f559014fcc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:48:02.553 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.543+0000 7f558e7fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f559007fbd0 0x7f559014fcc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:48:02.553 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.543+0000 7f558effd700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f559007f7f0 0x7f55900a5a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:39032/0 (socket says 172.21.15.23:39032) 2020-11-20T11:48:02.553 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.543+0000 7f558effd700 1 -- 172.21.15.23:0/3553739976 learned_addr learned my addr 172.21.15.23:0/3553739976 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:48:02.553 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.543+0000 7f558f7fe700 1 --2- 172.21.15.23:0/3553739976 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5590063fe0 0x7f55901521c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:02.554 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.546+0000 7f558f7fe700 1 -- 172.21.15.23:0/3553739976 >> v2:172.21.15.23:3300/0 conn(0x7f559007fbd0 msgr2=0x7f559014fcc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:48:02.554 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.546+0000 7f558f7fe700 1 --2- 172.21.15.23:0/3553739976 >> v2:172.21.15.23:3300/0 conn(0x7f559007fbd0 0x7f559014fcc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:48:02.554 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.546+0000 7f558f7fe700 1 -- 172.21.15.23:0/3553739976 >> v1:172.21.15.155:6789/0 conn(0x7f559007f7f0 legacy=0x7f55900a5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:02.554 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.546+0000 7f558f7fe700 1 -- 172.21.15.23:0/3553739976 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55900ad600 con 0x7f5590063fe0 2020-11-20T11:48:02.554 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.544+0000 7f558dffb700 1 -- 172.21.15.23:0/3553739976 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3404280028 0 0) 0x7f55900a4330 con 0x7f559007f7f0 2020-11-20T11:48:02.555 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.546+0000 7f558f7fe700 1 --2- 172.21.15.23:0/3553739976 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5590063fe0 0x7f55901521c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f5588009fd0 tx=0x7f5588009e50).ready entity=mon.2 client_cookie=be565b42b5ac7acc server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:02.555 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.546+0000 7f558dffb700 1 -- 172.21.15.23:0/3553739976 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5588014070 con 0x7f5590063fe0 2020-11-20T11:48:02.555 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.546+0000 7f558dffb700 1 -- 172.21.15.23:0/3553739976 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f558800f590 con 0x7f5590063fe0 2020-11-20T11:48:02.555 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.547+0000 7f558dffb700 1 -- 172.21.15.23:0/3553739976 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f558800f910 con 0x7f5590063fe0 2020-11-20T11:48:02.555 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.547+0000 7f5595567700 1 -- 172.21.15.23:0/3553739976 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5590063fe0 msgr2=0x7f55901521c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:02.556 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.547+0000 7f5595567700 1 --2- 172.21.15.23:0/3553739976 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5590063fe0 0x7f55901521c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f5588009fd0 tx=0x7f5588009e50).stop 2020-11-20T11:48:02.556 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.547+0000 7f5595567700 1 -- 172.21.15.23:0/3553739976 shutdown_connections 2020-11-20T11:48:02.556 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.547+0000 7f5595567700 1 --2- 172.21.15.23:0/3553739976 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5590063fe0 0x7f55901521c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:02.556 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.547+0000 7f5595567700 1 --2- 172.21.15.23:0/3553739976 >> v2:172.21.15.23:3300/0 conn(0x7f559007fbd0 0x7f559014fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:48:02.557 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.547+0000 7f5595567700 1 -- 172.21.15.23:0/3553739976 >> 172.21.15.23:0/3553739976 conn(0x7f5590060c80 msgr2=0x7f55900acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:48:02.557 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.547+0000 7f5595567700 1 -- 172.21.15.23:0/3553739976 shutdown_connections 2020-11-20T11:48:02.557 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.547+0000 7f5595567700 1 -- 172.21.15.23:0/3553739976 wait complete. 2020-11-20T11:48:02.560 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.548+0000 7f5595567700 1 Processor -- start 2020-11-20T11:48:02.561 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.549+0000 7f5595567700 1 -- start start 2020-11-20T11:48:02.561 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.549+0000 7f5595567700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5590063fe0 0x7f559014d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:02.561 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.549+0000 7f5595567700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f559007f7f0 0x7f55900a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:02.561 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.549+0000 7f5595567700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55900a2320 0x7f55900a2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:02.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.549+0000 7f5595567700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55900703f0 con 0x7f55900a2320 2020-11-20T11:48:02.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.549+0000 7f5595567700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55900a1e20 con 0x7f559007f7f0 2020-11-20T11:48:02.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.549+0000 7f5595567700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55900a2dc0 con 0x7f5590063fe0 2020-11-20T11:48:02.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.549+0000 7f558f7fe700 1 -- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55900a2320 msgr2=0x7f55900a2760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:48:02.562 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.549+0000 7f558f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55900a2320 0x7f55900a2760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:48:02.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.549+0000 7f558effd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5590063fe0 0x7f559014d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:02.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.549+0000 7f558effd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5590063fe0 0x7f559014d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:57054/0 (socket says 172.21.15.23:57054) 2020-11-20T11:48:02.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.549+0000 7f558effd700 1 -- 172.21.15.23:0/2638763091 learned_addr learned my addr 172.21.15.23:0/2638763091 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:48:02.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.549+0000 7f558e7fc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f559007f7f0 0x7f55900a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:02.563 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.550+0000 7f558effd700 1 -- 172.21.15.23:0/2638763091 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f559007f7f0 msgr2=0x7f55900a1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:02.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.550+0000 7f558effd700 1 --2- 172.21.15.23:0/2638763091 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f559007f7f0 0x7f55900a1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:02.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.550+0000 7f558effd700 1 -- 172.21.15.23:0/2638763091 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55900a2320 msgr2=0x7f55900a2760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:48:02.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.550+0000 7f558effd700 1 --2- 172.21.15.23:0/2638763091 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55900a2320 0x7f55900a2760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:48:02.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.550+0000 7f558effd700 1 -- 172.21.15.23:0/2638763091 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55900a11f0 con 0x7f5590063fe0 2020-11-20T11:48:02.564 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.550+0000 7f558effd700 1 --2- 172.21.15.23:0/2638763091 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5590063fe0 0x7f559014d460 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f558000c760 tx=0x7f5580007b70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:02.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.550+0000 7f5576ffd700 1 -- 172.21.15.23:0/2638763091 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55800072e0 con 0x7f5590063fe0 2020-11-20T11:48:02.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.550+0000 7f5576ffd700 1 -- 172.21.15.23:0/2638763091 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5580007440 con 0x7f5590063fe0 2020-11-20T11:48:02.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.550+0000 7f5595567700 1 -- 172.21.15.23:0/2638763091 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55900a1490 con 0x7f5590063fe0 2020-11-20T11:48:02.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.550+0000 7f5576ffd700 1 -- 172.21.15.23:0/2638763091 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5580005840 con 0x7f5590063fe0 2020-11-20T11:48:02.565 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.550+0000 7f5595567700 1 -- 172.21.15.23:0/2638763091 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f559015a310 con 0x7f5590063fe0 2020-11-20T11:48:02.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.551+0000 7f5595567700 1 --2- 172.21.15.23:0/2638763091 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f559007fbd0 0x7f559015a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:02.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.551+0000 7f5595567700 1 -- 172.21.15.23:0/2638763091 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5590061630 con 0x7f559007fbd0 2020-11-20T11:48:02.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.551+0000 7f558e7fc700 1 --2- 172.21.15.23:0/2638763091 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f559007fbd0 0x7f559015a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:02.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.551+0000 7f5576ffd700 1 -- 172.21.15.23:0/2638763091 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f558001a060 con 0x7f5590063fe0 2020-11-20T11:48:02.566 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.551+0000 7f558e7fc700 1 -- 172.21.15.23:0/2638763091 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f55900a1620 con 0x7f559007fbd0 2020-11-20T11:48:02.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.551+0000 7f5576ffd700 1 --2- 172.21.15.23:0/2638763091 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f556c056d10 0x7f556c0591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:02.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.551+0000 7f558e7fc700 1 --2- 172.21.15.23:0/2638763091 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f559007fbd0 0x7f559015a5f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f55900a5590 tx=0x7f5584006de0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:02.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.551+0000 7f5576ffd700 1 -- 172.21.15.23:0/2638763091 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f558006f840 con 0x7f5590063fe0 2020-11-20T11:48:02.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.551+0000 7f558f7fe700 1 --2- 172.21.15.23:0/2638763091 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f556c056d10 0x7f556c0591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:02.567 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.551+0000 7f5576ffd700 1 -- 172.21.15.23:0/2638763091 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5584006510 con 0x7f559007fbd0 2020-11-20T11:48:02.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.552+0000 7f5576ffd700 1 -- 172.21.15.23:0/2638763091 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f558400b930 con 0x7f559007fbd0 2020-11-20T11:48:02.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.552+0000 7f5576ffd700 1 -- 172.21.15.23:0/2638763091 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f559007fbd0 msgr2=0x7f559015a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:02.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.552+0000 7f5576ffd700 1 --2- 172.21.15.23:0/2638763091 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f559007fbd0 0x7f559015a5f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f55900a5590 tx=0x7f5584006de0).stop 2020-11-20T11:48:02.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.552+0000 7f5576ffd700 1 -- 172.21.15.23:0/2638763091 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f559007fbd0 msgr2=0x7f559015a5f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:48:02.568 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.552+0000 7f5576ffd700 1 --2- 172.21.15.23:0/2638763091 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f559007fbd0 0x7f559015a5f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:02.569 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.552+0000 7f558f7fe700 1 --2- 172.21.15.23:0/2638763091 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f556c056d10 0x7f556c0591d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f5588006f90 tx=0x7f5588016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:02.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.558+0000 7f5595567700 1 --2- 172.21.15.23:0/2638763091 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f559015aae0 0x7f5590156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:02.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.558+0000 7f5595567700 1 -- 172.21.15.23:0/2638763091 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559005c520 con 0x7f559015aae0 2020-11-20T11:48:02.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.559+0000 7f558e7fc700 1 --2- 172.21.15.23:0/2638763091 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f559015aae0 0x7f5590156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:02.571 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.559+0000 7f558e7fc700 1 -- 172.21.15.23:0/2638763091 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5590044710 con 0x7f559015aae0 2020-11-20T11:48:02.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.559+0000 7f558e7fc700 1 --2- 172.21.15.23:0/2638763091 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f559015aae0 0x7f5590156b00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f55900a5590 tx=0x7f5584016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:02.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.559+0000 7f5576ffd700 1 -- 172.21.15.23:0/2638763091 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5584013d50 con 0x7f559015aae0 2020-11-20T11:48:02.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.559+0000 7f5576ffd700 1 -- 172.21.15.23:0/2638763091 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4435 (secure 0 0 0) 0x7f558400b720 con 0x7f559015aae0 2020-11-20T11:48:02.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.559+0000 7f5576ffd700 1 -- 172.21.15.23:0/2638763091 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f559015aae0 msgr2=0x7f5590156b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:02.572 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.559+0000 7f5576ffd700 1 --2- 172.21.15.23:0/2638763091 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f559015aae0 0x7f5590156b00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f55900a5590 tx=0x7f5584016000).stop 2020-11-20T11:48:02.573 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.560+0000 7f5576ffd700 1 -- 172.21.15.23:0/2638763091 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f559015aae0 msgr2=0x7f5590156b00 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:48:02.573 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.560+0000 7f5576ffd700 1 --2- 172.21.15.23:0/2638763091 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f559015aae0 0x7f5590156b00 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0x7f55900a5590 tx=0x7f5584016000).stop 2020-11-20T11:48:02.573 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.561+0000 7f5595567700 1 -- 172.21.15.23:0/2638763091 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f556c056d10 msgr2=0x7f556c0591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:02.573 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.561+0000 7f5595567700 1 --2- 172.21.15.23:0/2638763091 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f556c056d10 0x7f556c0591d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f5588006f90 tx=0x7f5588016000).stop 2020-11-20T11:48:02.574 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.561+0000 7f558f7fe700 1 -- 172.21.15.23:0/2638763091 reap_dead start 2020-11-20T11:48:02.574 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.561+0000 7f5595567700 1 -- 172.21.15.23:0/2638763091 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5590063fe0 msgr2=0x7f559014d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:02.574 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.562+0000 7f5595567700 1 --2- 172.21.15.23:0/2638763091 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5590063fe0 0x7f559014d460 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f558000c760 tx=0x7f5580007b70).stop 2020-11-20T11:48:02.574 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.562+0000 7f5595567700 1 -- 172.21.15.23:0/2638763091 shutdown_connections 2020-11-20T11:48:02.574 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.562+0000 7f5595567700 1 --2- 172.21.15.23:0/2638763091 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5590063fe0 0x7f559014d460 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:02.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.562+0000 7f5595567700 1 -- 172.21.15.23:0/2638763091 >> 172.21.15.23:0/2638763091 conn(0x7f5590060c80 msgr2=0x7f55900a5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:48:02.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.562+0000 7f5595567700 1 -- 172.21.15.23:0/2638763091 shutdown_connections 2020-11-20T11:48:02.575 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.562+0000 7f5595567700 1 -- 172.21.15.23:0/2638763091 wait complete. 2020-11-20T11:48:02.575 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:48:02.575 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:48:02.576 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:48:02.576 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:48:02.576 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 140, 2020-11-20T11:48:02.576 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:48:02.577 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:48:02.577 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:48:02.577 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:48:02.577 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 0, 2020-11-20T11:48:02.577 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:48:02.578 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:48:02.578 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:48:02.578 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:48:02.578 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:48:02.578 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:48:02.579 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:48:02.579 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:48:02.579 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:48:02.579 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:48:02.579 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:48:02.580 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:48:02.580 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:48:02.580 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:48:02.580 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:48:02.580 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:48:02.581 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:48:02.581 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:48:02.581 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:48:02.581 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:48:02.581 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:48:02.582 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:48:02.582 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:48:02.582 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:02.582 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:48:02.582 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:48:02.583 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:48:02.583 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:48:02.583 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:48:02.583 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:48:02.583 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:48:02.584 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:48:02.584 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:48:02.584 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:48:02.584 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:48:02.584 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:48:02.585 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:48:02.585 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:48:02.585 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:48:02.585 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:48:02.585 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:48:02.586 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:48:02.586 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:48:02.586 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:48:02.586 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:48:02.586 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:48:02.587 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:48:02.587 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:48:02.587 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:48:02.587 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:02.587 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:48:02.588 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:02.588 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:48:02.588 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:48:02.588 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:48:02.588 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:48:02.589 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:02.589 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:48:02.589 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:48:02.589 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:48:02.589 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:02.590 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:02.590 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:48:02.590 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:48:02.590 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:48:02.590 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:02.591 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:48:02.591 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:02.591 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:48:02.591 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:48:02.591 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:48:02.592 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:48:02.592 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:48:02.592 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:02.592 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:02.592 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:48:02.593 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:48:02.593 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:48:02.593 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:48:02.593 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:02.593 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:48:02.594 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:48:02.594 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:48:02.594 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:02.594 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:02.594 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:48:02.595 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:48:02.595 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:48:02.595 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:02.595 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:48:02.595 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:02.595 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:48:02.596 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:48:02.596 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:48:02.596 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:48:02.596 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:48:02.596 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:02.597 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:02.597 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:48:02.597 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:48:02.597 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:48:02.597 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:48:02.598 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:02.598 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:48:02.598 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:48:02.598 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:48:02.599 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:02.599 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:02.600 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:48:02.600 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:48:02.600 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:48:02.600 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:02.600 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:48:02.601 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:02.601 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:48:02.601 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:48:02.601 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:48:02.601 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:48:02.603 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:48:02.603 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:02.603 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:48:02.604 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:02.604 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:48:02.604 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:48:02.605 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:02.605 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:48:02.606 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:48:02.606 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:48:02.606 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:02.607 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:48:02.607 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:48:02.608 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:02.608 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:48:02.608 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:48:02.609 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:48:02.609 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:02.610 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:48:02.610 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:48:02.611 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:02.611 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:48:02.611 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:48:02.612 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:48:02.612 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:02.612 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:48:02.612 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:02.613 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:48:02.613 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:48:02.615 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:48:02.733 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:02.682+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 255 slow ops, oldest is osd_op(client.4659.0:62551 4.2s0 4:5ef0fbf2:::10000000378.00000001:head [write 2232320~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:02.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.750+0000 7f0a3572e700 1 Processor -- start 2020-11-20T11:48:02.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.750+0000 7f0a3572e700 1 -- start start 2020-11-20T11:48:02.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.751+0000 7f0a3572e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a30066210 0x7f0a3013adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:02.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.751+0000 7f0a3572e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0a300665f0 0x7f0a300ab2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:02.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.751+0000 7f0a3572e700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a3013bd10 con 0x7f0a300a8ff0 2020-11-20T11:48:02.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.751+0000 7f0a3572e700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a300aab60 con 0x7f0a300665f0 2020-11-20T11:48:02.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.751+0000 7f0a3572e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a300aae00 con 0x7f0a30066210 2020-11-20T11:48:02.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.751+0000 7f0a34f2d700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f0a300a8ff0 legacy=0x7f0a30137f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2020-11-20T11:48:02.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.751+0000 7f0a2ffff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a30066210 0x7f0a3013adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:02.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.751+0000 7f0a2ffff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a30066210 0x7f0a3013adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:57066/0 (socket says 172.21.15.23:57066) 2020-11-20T11:48:02.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.751+0000 7f0a2ffff700 1 -- 172.21.15.23:0/111914883 learned_addr learned my addr 172.21.15.23:0/111914883 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:48:02.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.752+0000 7f0a2ffff700 1 -- 172.21.15.23:0/111914883 >> v2:172.21.15.155:3300/0 conn(0x7f0a300665f0 msgr2=0x7f0a300ab2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:48:02.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.751+0000 7f0a2f7fe700 1 --2- 172.21.15.23:0/111914883 >> v2:172.21.15.155:3300/0 conn(0x7f0a300665f0 0x7f0a300ab2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:02.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.753+0000 7f0a2ffff700 1 --2- 172.21.15.23:0/111914883 >> v2:172.21.15.155:3300/0 conn(0x7f0a300665f0 0x7f0a300ab2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:02.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.753+0000 7f0a2ffff700 1 -- 172.21.15.23:0/111914883 >> v1:172.21.15.23:6789/0 conn(0x7f0a300a8ff0 legacy=0x7f0a30137f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:48:02.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.753+0000 7f0a2ffff700 1 -- 172.21.15.23:0/111914883 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a300aa930 con 0x7f0a30066210 2020-11-20T11:48:02.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.753+0000 7f0a2f7fe700 1 --2- 172.21.15.23:0/111914883 >> v2:172.21.15.155:3300/0 conn(0x7f0a300665f0 0x7f0a300ab2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:48:02.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.754+0000 7f0a2ffff700 1 --2- 172.21.15.23:0/111914883 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a30066210 0x7f0a3013adb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f0a20009a10 tx=0x7f0a20006ef0).ready entity=mon.2 client_cookie=a221f154bc50b36c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:02.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.754+0000 7f0a2effd700 1 -- 172.21.15.23:0/111914883 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2000b820 con 0x7f0a30066210 2020-11-20T11:48:02.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.754+0000 7f0a2effd700 1 -- 172.21.15.23:0/111914883 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a20000e90 con 0x7f0a30066210 2020-11-20T11:48:02.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.754+0000 7f0a2effd700 1 -- 172.21.15.23:0/111914883 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2000bb90 con 0x7f0a30066210 2020-11-20T11:48:02.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.755+0000 7f0a3572e700 1 -- 172.21.15.23:0/111914883 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a30066210 msgr2=0x7f0a3013adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:02.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.755+0000 7f0a3572e700 1 --2- 172.21.15.23:0/111914883 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a30066210 0x7f0a3013adb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f0a20009a10 tx=0x7f0a20006ef0).stop 2020-11-20T11:48:02.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.755+0000 7f0a3572e700 1 -- 172.21.15.23:0/111914883 shutdown_connections 2020-11-20T11:48:02.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.755+0000 7f0a3572e700 1 --2- 172.21.15.23:0/111914883 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a30066210 0x7f0a3013adb0 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0x7f0a20009a10 tx=0x7f0a20006ef0).stop 2020-11-20T11:48:02.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.755+0000 7f0a3572e700 1 --2- 172.21.15.23:0/111914883 >> v2:172.21.15.155:3300/0 conn(0x7f0a300665f0 0x7f0a300ab2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:02.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.755+0000 7f0a3572e700 1 -- 172.21.15.23:0/111914883 >> 172.21.15.23:0/111914883 conn(0x7f0a300612c0 msgr2=0x7f0a300616d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:48:02.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.759+0000 7f0a3572e700 1 -- 172.21.15.23:0/111914883 shutdown_connections 2020-11-20T11:48:02.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.761+0000 7f0a3572e700 1 -- 172.21.15.23:0/111914883 wait complete. 2020-11-20T11:48:02.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.762+0000 7f0a3572e700 1 Processor -- start 2020-11-20T11:48:02.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.762+0000 7f0a3572e700 1 -- start start 2020-11-20T11:48:02.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.763+0000 7f0a3572e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a300665f0 0x7f0a300a1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:02.769 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.763+0000 7f0a2ffff700 1 -- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a300665f0 msgr2=0x7f0a300a1e20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:48:02.769 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.763+0000 7f0a2ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a300665f0 0x7f0a300a1e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:48:02.770 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.763+0000 7f0a3572e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a300a8ff0 0x7f0a300a3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:02.771 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.764+0000 7f0a3572e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a3014a890 0x7f0a3014acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:02.771 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.764+0000 7f0a3572e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3014b1a0 con 0x7f0a300665f0 2020-11-20T11:48:02.771 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.764+0000 7f0a3572e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a300a23c0 con 0x7f0a300a8ff0 2020-11-20T11:48:02.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.764+0000 7f0a3572e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a300a2660 con 0x7f0a3014a890 2020-11-20T11:48:02.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.764+0000 7f0a2f7fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a300a8ff0 0x7f0a300a3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:02.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.764+0000 7f0a2f7fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a300a8ff0 0x7f0a300a3ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:39730/0 (socket says 172.21.15.23:39730) 2020-11-20T11:48:02.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.764+0000 7f0a2f7fe700 1 -- 172.21.15.23:0/3589693360 learned_addr learned my addr 172.21.15.23:0/3589693360 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:48:02.772 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.764+0000 7f0a34f2d700 1 --2- 172.21.15.23:0/3589693360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a3014a890 0x7f0a3014acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:02.773 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.764+0000 7f0a2f7fe700 1 -- 172.21.15.23:0/3589693360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a3014a890 msgr2=0x7f0a3014acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:02.773 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.764+0000 7f0a2f7fe700 1 --2- 172.21.15.23:0/3589693360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a3014a890 0x7f0a3014acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:02.773 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.764+0000 7f0a2f7fe700 1 -- 172.21.15.23:0/3589693360 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a300665f0 msgr2=0x7f0a300a1e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:48:02.773 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.764+0000 7f0a2f7fe700 1 --2- 172.21.15.23:0/3589693360 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a300665f0 0x7f0a300a1e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:48:02.774 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.764+0000 7f0a2f7fe700 1 -- 172.21.15.23:0/3589693360 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a3014a3f0 con 0x7f0a300a8ff0 2020-11-20T11:48:02.774 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.764+0000 7f0a2f7fe700 1 --2- 172.21.15.23:0/3589693360 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a300a8ff0 0x7f0a300a3ce0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f0a24012f70 tx=0x7f0a24008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:02.774 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.765+0000 7f0a17fff700 1 -- 172.21.15.23:0/3589693360 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a24007c10 con 0x7f0a300a8ff0 2020-11-20T11:48:02.774 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.765+0000 7f0a17fff700 1 -- 172.21.15.23:0/3589693360 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a24007d70 con 0x7f0a300a8ff0 2020-11-20T11:48:02.774 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.765+0000 7f0a17fff700 1 -- 172.21.15.23:0/3589693360 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a24012380 con 0x7f0a300a8ff0 2020-11-20T11:48:02.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.765+0000 7f0a3572e700 1 -- 172.21.15.23:0/3589693360 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a300a4510 con 0x7f0a300a8ff0 2020-11-20T11:48:02.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.765+0000 7f0a3572e700 1 -- 172.21.15.23:0/3589693360 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a301361a0 con 0x7f0a300a8ff0 2020-11-20T11:48:02.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.766+0000 7f0a17fff700 1 -- 172.21.15.23:0/3589693360 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0a2402c3a0 con 0x7f0a300a8ff0 2020-11-20T11:48:02.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.767+0000 7f0a17fff700 1 --2- 172.21.15.23:0/3589693360 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0a04056d10 0x7f0a040591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:02.775 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.767+0000 7f0a17fff700 1 -- 172.21.15.23:0/3589693360 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f0a24080770 con 0x7f0a300a8ff0 2020-11-20T11:48:02.776 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.768+0000 7f0a3572e700 1 --2- 172.21.15.23:0/3589693360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a300622d0 0x7f0a30136410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:02.776 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.768+0000 7f0a3572e700 1 -- 172.21.15.23:0/3589693360 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a30044d80 con 0x7f0a300622d0 2020-11-20T11:48:02.777 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.770+0000 7f0a34f2d700 1 --2- 172.21.15.23:0/3589693360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a300622d0 0x7f0a30136410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:02.777 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.770+0000 7f0a2ffff700 1 --2- 172.21.15.23:0/3589693360 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0a04056d10 0x7f0a040591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:02.778 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.774+0000 7f0a2ffff700 1 --2- 172.21.15.23:0/3589693360 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0a04056d10 0x7f0a040591d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f0a300ab8c0 tx=0x7f0a20016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:02.779 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.775+0000 7f0a34f2d700 1 -- 172.21.15.23:0/3589693360 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0a30062760 con 0x7f0a300622d0 2020-11-20T11:48:02.779 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.775+0000 7f0a34f2d700 1 --2- 172.21.15.23:0/3589693360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a300622d0 0x7f0a30136410 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f0a28003a60 tx=0x7f0a28006f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:02.779 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.776+0000 7f0a17fff700 1 -- 172.21.15.23:0/3589693360 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2800e960 con 0x7f0a300622d0 2020-11-20T11:48:02.780 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.776+0000 7f0a17fff700 1 -- 172.21.15.23:0/3589693360 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f0a2800eac0 con 0x7f0a300622d0 2020-11-20T11:48:02.780 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.776+0000 7f0a17fff700 1 -- 172.21.15.23:0/3589693360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a300622d0 msgr2=0x7f0a30136410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:02.780 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.776+0000 7f0a17fff700 1 --2- 172.21.15.23:0/3589693360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a300622d0 0x7f0a30136410 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f0a28003a60 tx=0x7f0a28006f90).stop 2020-11-20T11:48:02.781 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.776+0000 7f0a17fff700 1 -- 172.21.15.23:0/3589693360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a300622d0 msgr2=0x7f0a30136410 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:48:02.781 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.776+0000 7f0a17fff700 1 --2- 172.21.15.23:0/3589693360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a300622d0 0x7f0a30136410 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0x7f0a28003a60 tx=0x7f0a28006f90).stop 2020-11-20T11:48:02.791 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.788+0000 7f0a3572e700 1 --2- 172.21.15.23:0/3589693360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a30136900 0x7f0a3014d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:02.791 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.788+0000 7f0a3572e700 1 -- 172.21.15.23:0/3589693360 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3014a580 con 0x7f0a30136900 2020-11-20T11:48:02.791 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.788+0000 7f0a34f2d700 1 --2- 172.21.15.23:0/3589693360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a30136900 0x7f0a3014d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:02.793 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.788+0000 7f0a34f2d700 1 -- 172.21.15.23:0/3589693360 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0a30057dc0 con 0x7f0a30136900 2020-11-20T11:48:02.793 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.788+0000 7f0a34f2d700 1 --2- 172.21.15.23:0/3589693360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a30136900 0x7f0a3014d830 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f0a28003a60 tx=0x7f0a28017000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:02.793 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.789+0000 7f0a17fff700 1 -- 172.21.15.23:0/3589693360 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a280125c0 con 0x7f0a30136900 2020-11-20T11:48:02.793 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.789+0000 7f0a17fff700 1 -- 172.21.15.23:0/3589693360 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4773 (secure 0 0 0) 0x7f0a2800e750 con 0x7f0a30136900 2020-11-20T11:48:02.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.789+0000 7f0a17fff700 1 -- 172.21.15.23:0/3589693360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a30136900 msgr2=0x7f0a3014d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:02.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.789+0000 7f0a17fff700 1 --2- 172.21.15.23:0/3589693360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a30136900 0x7f0a3014d830 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f0a28003a60 tx=0x7f0a28017000).stop 2020-11-20T11:48:02.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.789+0000 7f0a17fff700 1 -- 172.21.15.23:0/3589693360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a30136900 msgr2=0x7f0a3014d830 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:48:02.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.789+0000 7f0a17fff700 1 --2- 172.21.15.23:0/3589693360 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0a30136900 0x7f0a3014d830 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:02.794 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.789+0000 7f0a3572e700 1 -- 172.21.15.23:0/3589693360 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0a04056d10 msgr2=0x7f0a040591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:02.795 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.789+0000 7f0a3572e700 1 --2- 172.21.15.23:0/3589693360 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0a04056d10 0x7f0a040591d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f0a300ab8c0 tx=0x7f0a20016000).stop 2020-11-20T11:48:02.795 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.789+0000 7f0a3572e700 1 -- 172.21.15.23:0/3589693360 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a300a8ff0 msgr2=0x7f0a300a3ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:02.795 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.789+0000 7f0a3572e700 1 --2- 172.21.15.23:0/3589693360 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a300a8ff0 0x7f0a300a3ce0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f0a24012f70 tx=0x7f0a24008040).stop 2020-11-20T11:48:02.795 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.789+0000 7f0a34f2d700 1 -- 172.21.15.23:0/3589693360 reap_dead start 2020-11-20T11:48:02.796 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.789+0000 7f0a3572e700 1 -- 172.21.15.23:0/3589693360 shutdown_connections 2020-11-20T11:48:02.796 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.789+0000 7f0a3572e700 1 --2- 172.21.15.23:0/3589693360 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0a300a8ff0 0x7f0a300a3ce0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:02.796 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.789+0000 7f0a3572e700 1 -- 172.21.15.23:0/3589693360 >> 172.21.15.23:0/3589693360 conn(0x7f0a300612c0 msgr2=0x7f0a300616a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:48:02.797 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.789+0000 7f0a3572e700 1 -- 172.21.15.23:0/3589693360 shutdown_connections 2020-11-20T11:48:02.797 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:02.789+0000 7f0a3572e700 1 -- 172.21.15.23:0/3589693360 wait complete. 2020-11-20T11:48:02.797 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:48:02.797 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:48:02.798 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:48:02.798 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:48:02.798 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 140, 2020-11-20T11:48:02.798 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:48:02.798 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:48:02.799 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:48:02.799 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:48:02.799 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 0, 2020-11-20T11:48:02.799 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:48:02.800 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:48:02.800 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:48:02.800 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:48:02.800 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:48:02.801 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:48:02.801 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:48:02.801 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:48:02.801 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:48:02.802 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:48:02.802 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:48:02.802 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:48:02.802 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:48:02.802 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:48:02.802 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:48:02.803 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:48:02.803 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:48:02.803 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:48:02.803 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:48:02.803 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:48:02.804 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:48:02.804 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:48:02.804 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:48:02.804 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:02.804 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:48:02.805 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:48:02.805 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:48:02.805 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:48:02.805 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:48:02.805 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:48:02.806 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:48:02.806 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:48:02.806 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:48:02.806 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:48:02.806 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:48:02.807 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:48:02.807 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:48:02.807 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:48:02.807 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:48:02.808 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:48:02.808 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:48:02.808 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:48:02.808 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:48:02.809 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:48:02.809 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:48:02.809 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:48:02.809 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:48:02.809 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:48:02.810 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:48:02.810 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:02.810 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:48:02.810 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:02.810 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:48:02.811 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:48:02.811 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:48:02.811 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:48:02.811 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:02.811 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:48:02.812 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:48:02.812 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:48:02.812 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:02.812 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:02.812 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:48:02.813 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:48:02.813 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:48:02.813 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:02.813 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:48:02.814 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:02.814 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:48:02.814 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:48:02.814 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:48:02.814 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:48:02.815 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:48:02.815 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:02.815 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:02.815 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:48:02.815 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:48:02.815 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:48:02.816 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:48:02.816 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:02.816 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:48:02.816 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:48:02.816 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:48:02.817 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:02.817 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:02.817 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:48:02.817 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:48:02.818 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:48:02.818 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:02.818 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:48:02.818 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:02.818 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:48:02.818 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:48:02.819 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:48:02.819 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:48:02.819 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:48:02.819 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:02.819 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:02.820 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:48:02.820 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:48:02.820 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:48:02.820 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:48:02.820 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:02.821 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:48:02.821 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:48:02.821 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:48:02.821 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:02.821 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:02.822 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:48:02.822 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:48:02.822 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:48:02.822 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:02.822 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:48:02.823 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:02.823 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:48:02.823 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:48:02.823 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:48:02.823 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:48:02.824 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:48:02.824 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:02.824 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:48:02.824 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:02.824 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:48:02.825 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:48:02.825 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:02.825 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:48:02.825 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:48:02.825 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:48:02.826 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:02.826 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:48:02.826 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:48:02.826 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:02.827 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:48:02.827 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:48:02.827 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:48:02.827 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:02.828 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:48:02.828 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:48:02.828 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:02.828 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:48:02.828 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:48:02.829 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 6 2020-11-20T11:48:02.829 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:02.829 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:48:02.829 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:48:02.830 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:02.830 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:48:02.830 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:48:02.830 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:48:02.830 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:02.831 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:48:02.831 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:48:02.831 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:02.831 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:48:02.831 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:48:02.831 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:48:02.832 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:02.832 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:48:02.832 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:02.832 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:48:02.832 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:48:02.834 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2020-11-20T11:48:03.805 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:03.784+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 255 slow ops, oldest is osd_op(client.4659.0:62551 4.2s0 4:5ef0fbf2:::10000000378.00000001:head [write 2232320~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:04.839 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:04.775+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 259 slow ops, oldest is osd_op(client.4659.0:62551 4.2s0 4:5ef0fbf2:::10000000378.00000001:head [write 2232320~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:05.827 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:05.795+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4659.0:62551 4.2s0 4:5ef0fbf2:::10000000378.00000001:head [write 2232320~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:06.824 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:06.802+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 268 slow ops, oldest is osd_op(client.4659.0:62551 4.2s0 4:5ef0fbf2:::10000000378.00000001:head [write 2232320~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:07.892 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:07.858+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 270 slow ops, oldest is osd_op(client.4659.0:62551 4.2s0 4:5ef0fbf2:::10000000378.00000001:head [write 2232320~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:08.964 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:08.915+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 270 slow ops, oldest is osd_op(client.4659.0:62647 4.2s0 4:509aaf86:::100000003b4.00000015:head [write 4063232~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:09.937 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:09.902+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 274 slow ops, oldest is osd_op(client.4659.0:63139 4.2s0 4:5275af7e:::100000003c5.00000012:head [write 2494464~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:10.987 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:10.929+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 273 slow ops, oldest is osd_op(client.4659.0:63139 4.2s0 4:5275af7e:::100000003c5.00000012:head [write 2494464~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:11.963 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:11.944+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 275 slow ops, oldest is osd_op(client.4659.0:63139 4.2s0 4:5275af7e:::100000003c5.00000012:head [write 2494464~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:13.055 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:13.033+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 278 slow ops, oldest is osd_op(client.4659.0:63713 4.2s0 4:5050cdd3:::10000000378.00000009:head [write 1712128~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:14.045 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:14.012+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 278 slow ops, oldest is osd_op(client.4659.0:63713 4.2s0 4:5050cdd3:::10000000378.00000009:head [write 1712128~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:15.069 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:15.049+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 280 slow ops, oldest is osd_op(client.4659.0:63713 4.2s0 4:5050cdd3:::10000000378.00000009:head [write 1712128~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:16.125 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:16.090+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 280 slow ops, oldest is osd_op(client.4659.0:63713 4.2s0 4:5050cdd3:::10000000378.00000009:head [write 1712128~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:17.153 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:17.102+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 281 slow ops, oldest is osd_op(client.4659.0:64155 4.2s0 4:5c9d31b4:::100000003a1.0000000a:head [write 1810432~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:18.149 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:18.116+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 287 slow ops, oldest is osd_op(client.4659.0:64155 4.2s0 4:5c9d31b4:::100000003a1.0000000a:head [write 1810432~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:19.220 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:19.179+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 290 slow ops, oldest is osd_op(client.4659.0:64155 4.2s0 4:5c9d31b4:::100000003a1.0000000a:head [write 1810432~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:20.244 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:20.190+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 289 slow ops, oldest is osd_op(client.4659.0:64498 4.2s0 4:5269465d:::1000000038d.00000018:head [write 3936256~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:21.248 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:21.215+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 292 slow ops, oldest is osd_op(client.4659.0:64498 4.2s0 4:5269465d:::1000000038d.00000018:head [write 3936256~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:22.282 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:22.226+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 296 slow ops, oldest is osd_op(client.4659.0:64863 4.2s0 4:548adc15:::10000000360.00000002:head [write 3584000~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:22.834 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2020-11-20T11:48:22.835 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2020-11-20T11:48:22.835 INFO:tasks.ceph.mon.a:Restarting daemon 2020-11-20T11:48:22.835 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2020-11-20T11:48:22.838 INFO:tasks.ceph.mon.a:Started 2020-11-20T11:48:22.838 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:48:22.839 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:48:22.980 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.976+0000 7f6b799be700 1 Processor -- start 2020-11-20T11:48:22.980 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.976+0000 7f6b799be700 1 -- start start 2020-11-20T11:48:22.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.976+0000 7f6b799be700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6b740aae90 0x7f6b741381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:22.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.976+0000 7f6b799be700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b7413e600 con 0x7f6b7413aeb0 2020-11-20T11:48:22.983 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.976+0000 7f6b799be700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b7413e8a0 con 0x7f6b740aaab0 2020-11-20T11:48:22.983 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.976+0000 7f6b799be700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b740ab6d0 con 0x7f6b740aae90 2020-11-20T11:48:22.983 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.976+0000 7f6b737fe700 1 -- >> v1:172.21.15.23:6789/0 conn(0x7f6b7413aeb0 legacy=0x7f6b740a8c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.23:6789/0 2020-11-20T11:48:22.983 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.977+0000 7f6b727fc700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6b740aae90 0x7f6b741381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:22.983 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.977+0000 7f6b727fc700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6b740aae90 0x7f6b741381a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:39740/0 (socket says 172.21.15.23:39740) 2020-11-20T11:48:22.984 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.977+0000 7f6b72ffd700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f6b740aaab0 0x7f6b7413ed40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:39072/0 (socket says 172.21.15.23:39072) 2020-11-20T11:48:22.984 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.977+0000 7f6b727fc700 1 -- 172.21.15.23:0/2101260702 learned_addr learned my addr 172.21.15.23:0/2101260702 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:48:22.984 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.977+0000 7f6b727fc700 1 -- 172.21.15.23:0/2101260702 >> v1:172.21.15.155:6789/0 conn(0x7f6b740aaab0 legacy=0x7f6b7413ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:22.984 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.977+0000 7f6b727fc700 1 -- 172.21.15.23:0/2101260702 >> v1:172.21.15.23:6789/0 conn(0x7f6b7413aeb0 legacy=0x7f6b740a8c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:48:22.985 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.977+0000 7f6b727fc700 1 -- 172.21.15.23:0/2101260702 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b740a9370 con 0x7f6b740aae90 2020-11-20T11:48:22.985 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.977+0000 7f6b727fc700 1 --2- 172.21.15.23:0/2101260702 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b740aae90 0x7f6b741381a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f6b68009a10 tx=0x7f6b68006ef0).ready entity=mon.1 client_cookie=787422d13f6508dc server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:22.985 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.977+0000 7f6b71ffb700 1 -- 172.21.15.23:0/2101260702 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6800b820 con 0x7f6b740aae90 2020-11-20T11:48:22.985 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.977+0000 7f6b71ffb700 1 -- 172.21.15.23:0/2101260702 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b68000e90 con 0x7f6b740aae90 2020-11-20T11:48:22.986 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.977+0000 7f6b71ffb700 1 -- 172.21.15.23:0/2101260702 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6800bb90 con 0x7f6b740aae90 2020-11-20T11:48:22.986 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.978+0000 7f6b799be700 1 -- 172.21.15.23:0/2101260702 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b740aae90 msgr2=0x7f6b741381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:22.986 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.978+0000 7f6b799be700 1 --2- 172.21.15.23:0/2101260702 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b740aae90 0x7f6b741381a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f6b68009a10 tx=0x7f6b68006ef0).stop 2020-11-20T11:48:22.986 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.978+0000 7f6b799be700 1 -- 172.21.15.23:0/2101260702 shutdown_connections 2020-11-20T11:48:22.986 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.978+0000 7f6b799be700 1 --2- 172.21.15.23:0/2101260702 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b740aae90 0x7f6b741381a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:22.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.978+0000 7f6b799be700 1 -- 172.21.15.23:0/2101260702 >> 172.21.15.23:0/2101260702 conn(0x7f6b740678e0 msgr2=0x7f6b74067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:48:22.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.978+0000 7f6b799be700 1 -- 172.21.15.23:0/2101260702 shutdown_connections 2020-11-20T11:48:22.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.978+0000 7f6b799be700 1 -- 172.21.15.23:0/2101260702 wait complete. 2020-11-20T11:48:22.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.979+0000 7f6b799be700 1 Processor -- start 2020-11-20T11:48:22.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.979+0000 7f6b799be700 1 -- start start 2020-11-20T11:48:22.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.980+0000 7f6b799be700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6b740aaab0 0x7f6b740a73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:22.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.980+0000 7f6b799be700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b740aae90 0x7f6b74134a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:22.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.980+0000 7f6b799be700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b740a80b0 0x7f6b74134f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:22.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.980+0000 7f6b799be700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7414a880 con 0x7f6b740a80b0 2020-11-20T11:48:22.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.980+0000 7f6b799be700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b740a5930 con 0x7f6b740aae90 2020-11-20T11:48:22.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.980+0000 7f6b799be700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b740a5b90 con 0x7f6b740aaab0 2020-11-20T11:48:22.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.980+0000 7f6b737fe700 1 -- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b740a80b0 msgr2=0x7f6b74134f40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:3300/0 2020-11-20T11:48:22.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.980+0000 7f6b737fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b740a80b0 0x7f6b74134f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:48:22.992 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.980+0000 7f6b727fc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b740aae90 0x7f6b74134a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:22.992 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.980+0000 7f6b727fc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b740aae90 0x7f6b74134a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:39748/0 (socket says 172.21.15.23:39748) 2020-11-20T11:48:22.992 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.980+0000 7f6b727fc700 1 -- 172.21.15.23:0/894194731 learned_addr learned my addr 172.21.15.23:0/894194731 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:48:22.992 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.980+0000 7f6b72ffd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6b740aaab0 0x7f6b740a73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:22.993 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.980+0000 7f6b727fc700 1 -- 172.21.15.23:0/894194731 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6b740aaab0 msgr2=0x7f6b740a73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:22.993 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.980+0000 7f6b727fc700 1 --2- 172.21.15.23:0/894194731 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6b740aaab0 0x7f6b740a73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:22.993 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.980+0000 7f6b727fc700 1 -- 172.21.15.23:0/894194731 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b740a80b0 msgr2=0x7f6b74134f40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:48:22.993 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.980+0000 7f6b727fc700 1 --2- 172.21.15.23:0/894194731 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b740a80b0 0x7f6b74134f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:48:22.994 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.980+0000 7f6b727fc700 1 -- 172.21.15.23:0/894194731 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b740a5e30 con 0x7f6b740aae90 2020-11-20T11:48:22.994 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.981+0000 7f6b727fc700 1 --2- 172.21.15.23:0/894194731 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b740aae90 0x7f6b74134a50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f6b6800b3c0 tx=0x7f6b68016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:22.994 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.981+0000 7f6b5affd700 1 -- 172.21.15.23:0/894194731 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6801c440 con 0x7f6b740aae90 2020-11-20T11:48:22.994 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.981+0000 7f6b5affd700 1 -- 172.21.15.23:0/894194731 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b6801c5a0 con 0x7f6b740aae90 2020-11-20T11:48:22.994 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.981+0000 7f6b799be700 1 -- 172.21.15.23:0/894194731 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b740a78a0 con 0x7f6b740aae90 2020-11-20T11:48:22.995 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.981+0000 7f6b5affd700 1 -- 172.21.15.23:0/894194731 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6801c940 con 0x7f6b740aae90 2020-11-20T11:48:22.995 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.981+0000 7f6b799be700 1 -- 172.21.15.23:0/894194731 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b740a7da0 con 0x7f6b740aae90 2020-11-20T11:48:22.995 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.982+0000 7f6b799be700 1 -- 172.21.15.23:0/894194731 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b740b0bd0 con 0x7f6b740aae90 2020-11-20T11:48:22.995 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.982+0000 7f6b5affd700 1 -- 172.21.15.23:0/894194731 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6b6802c070 con 0x7f6b740aae90 2020-11-20T11:48:22.995 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.982+0000 7f6b5affd700 1 --2- 172.21.15.23:0/894194731 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6b50056d40 0x7f6b50059200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:22.996 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.982+0000 7f6b5affd700 1 -- 172.21.15.23:0/894194731 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f6b68071f00 con 0x7f6b740aae90 2020-11-20T11:48:22.996 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.982+0000 7f6b72ffd700 1 --2- 172.21.15.23:0/894194731 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6b50056d40 0x7f6b50059200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:22.996 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.983+0000 7f6b72ffd700 1 --2- 172.21.15.23:0/894194731 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6b50056d40 0x7f6b50059200 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f6b64004610 tx=0x7f6b64007bc0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:22.996 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:22.986+0000 7f6b5affd700 1 -- 172.21.15.23:0/894194731 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6b680471f0 con 0x7f6b740aae90 2020-11-20T11:48:23.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:23.265+0000 7f6b799be700 1 -- 172.21.15.23:0/894194731 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6b7405c7d0 con 0x7f6b740aae90 2020-11-20T11:48:23.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:23.266+0000 7f6b5affd700 1 -- 172.21.15.23:0/894194731 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1473 (secure 0 0 0) 0x7f6b68020070 con 0x7f6b740aae90 2020-11-20T11:48:23.270 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":140,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:48:23.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:23.267+0000 7f6b799be700 1 -- 172.21.15.23:0/894194731 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6b50056d40 msgr2=0x7f6b50059200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:23.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:23.267+0000 7f6b799be700 1 --2- 172.21.15.23:0/894194731 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6b50056d40 0x7f6b50059200 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f6b64004610 tx=0x7f6b64007bc0).stop 2020-11-20T11:48:23.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:23.267+0000 7f6b799be700 1 -- 172.21.15.23:0/894194731 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b740aae90 msgr2=0x7f6b74134a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:23.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:23.267+0000 7f6b799be700 1 --2- 172.21.15.23:0/894194731 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b740aae90 0x7f6b74134a50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f6b6800b3c0 tx=0x7f6b68016000).stop 2020-11-20T11:48:23.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:23.267+0000 7f6b799be700 1 -- 172.21.15.23:0/894194731 shutdown_connections 2020-11-20T11:48:23.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:23.267+0000 7f6b799be700 1 --2- 172.21.15.23:0/894194731 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b740a80b0 0x7f6b74134f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:48:23.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:23.267+0000 7f6b799be700 1 --2- 172.21.15.23:0/894194731 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6b50056d40 0x7f6b50059200 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:23.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:23.267+0000 7f6b799be700 1 --2- 172.21.15.23:0/894194731 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6b740aaab0 0x7f6b740a73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:23.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:23.267+0000 7f6b799be700 1 --2- 172.21.15.23:0/894194731 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6b740aae90 0x7f6b74134a50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:23.274 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:23.267+0000 7f6b799be700 1 -- 172.21.15.23:0/894194731 >> 172.21.15.23:0/894194731 conn(0x7f6b740678e0 msgr2=0x7f6b74138e20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:48:23.274 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:23.267+0000 7f6b799be700 1 -- 172.21.15.23:0/894194731 shutdown_connections 2020-11-20T11:48:23.274 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:23.267+0000 7f6b799be700 1 -- 172.21.15.23:0/894194731 wait complete. 2020-11-20T11:48:23.297 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:23.250+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4659.0:64863 4.2s0 4:548adc15:::10000000360.00000002:head [write 3584000~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:24.313 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:24.294+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4659.0:65360 4.2s0 4:48389afd:::100000003b9.00000013:head [write 4014080~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:25.333 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:25.312+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 299 slow ops, oldest is osd_op(client.4659.0:65360 4.2s0 4:48389afd:::100000003b9.00000013:head [write 4014080~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:26.284 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:48:26.360 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:26.324+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 297 slow ops, oldest is osd_op(client.4659.0:65360 4.2s0 4:48389afd:::100000003b9.00000013:head [write 4014080~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:26.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:26.440+0000 7f5e13280700 1 Processor -- start 2020-11-20T11:48:26.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:26.440+0000 7f5e13280700 1 -- start start 2020-11-20T11:48:26.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:26.440+0000 7f5e13280700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e0c064f20 0x7f5e0c0a9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:26.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:26.440+0000 7f5e13280700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5e0c13b100 0x7f5e0c066770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:26.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:26.440+0000 7f5e13280700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0c13b810 con 0x7f5e0c066c60 2020-11-20T11:48:26.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:26.441+0000 7f5e13280700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e0c13bab0 con 0x7f5e0c13b100 2020-11-20T11:48:26.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:26.441+0000 7f5e13280700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c13bd50 con 0x7f5e0c064f20 2020-11-20T11:48:26.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:26.441+0000 7f5e1181d700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f5e0c066c60 0x7f5e0c13e650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:39102/0 (socket says 172.21.15.23:39102) 2020-11-20T11:48:26.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:26.441+0000 7f5e1101c700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e0c064f20 0x7f5e0c0a9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:26.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:26.441+0000 7f5e1181d700 1 -- 172.21.15.23:0/405838939 learned_addr learned my addr 172.21.15.23:0/405838939 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:48:26.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:26.441+0000 7f5e1081b700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5e0c13b100 0x7f5e0c066770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:26.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:26.441+0000 7f5e1101c700 1 -- 172.21.15.23:0/405838939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e0c064f20 msgr2=0x7f5e0c0a9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:48:26.448 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:26.441+0000 7f5e1101c700 1 -- 172.21.15.23:0/405838939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e0c064f20 msgr2=0x7f5e0c0a9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:48:26.448 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:26.441+0000 7f5e1101c700 1 --2- 172.21.15.23:0/405838939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e0c064f20 0x7f5e0c0a9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:48:26.448 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:26.441+0000 7f5e1101c700 1 --2- 172.21.15.23:0/405838939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e0c064f20 0x7f5e0c0a9d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:48:26.448 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:26.441+0000 7f5e1081b700 1 -- 172.21.15.23:0/405838939 >> v2:172.21.15.155:3300/0 conn(0x7f5e0c13b100 msgr2=0x7f5e0c066770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:48:26.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:26.441+0000 7f5e1081b700 1 -- 172.21.15.23:0/405838939 >> v2:172.21.15.155:3300/0 conn(0x7f5e0c13b100 msgr2=0x7f5e0c066770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:48:26.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:26.441+0000 7f5e1081b700 1 --2- 172.21.15.23:0/405838939 >> v2:172.21.15.155:3300/0 conn(0x7f5e0c13b100 0x7f5e0c066770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:48:26.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:26.441+0000 7f5e1081b700 1 --2- 172.21.15.23:0/405838939 >> v2:172.21.15.155:3300/0 conn(0x7f5e0c13b100 0x7f5e0c066770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:48:26.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:26.642+0000 7f5e1101c700 1 --2- 172.21.15.23:0/405838939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e0c064f20 0x7f5e0c0a9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:26.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:26.642+0000 7f5e1081b700 1 --2- 172.21.15.23:0/405838939 >> v2:172.21.15.155:3300/0 conn(0x7f5e0c13b100 0x7f5e0c066770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:26.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:26.642+0000 7f5e1081b700 1 -- 172.21.15.23:0/405838939 >> v2:172.21.15.155:3300/0 conn(0x7f5e0c13b100 msgr2=0x7f5e0c066770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:48:26.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:26.642+0000 7f5e1081b700 1 -- 172.21.15.23:0/405838939 >> v2:172.21.15.155:3300/0 conn(0x7f5e0c13b100 msgr2=0x7f5e0c066770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:48:26.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:26.642+0000 7f5e1081b700 1 --2- 172.21.15.23:0/405838939 >> v2:172.21.15.155:3300/0 conn(0x7f5e0c13b100 0x7f5e0c066770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:48:26.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:26.642+0000 7f5e1081b700 1 --2- 172.21.15.23:0/405838939 >> v2:172.21.15.155:3300/0 conn(0x7f5e0c13b100 0x7f5e0c066770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:48:26.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:26.643+0000 7f5e1101c700 1 -- 172.21.15.23:0/405838939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e0c064f20 msgr2=0x7f5e0c0a9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:48:26.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:26.643+0000 7f5e1101c700 1 -- 172.21.15.23:0/405838939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e0c064f20 msgr2=0x7f5e0c0a9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:48:26.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:26.643+0000 7f5e1101c700 1 --2- 172.21.15.23:0/405838939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e0c064f20 0x7f5e0c0a9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:48:26.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:26.643+0000 7f5e1101c700 1 --2- 172.21.15.23:0/405838939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e0c064f20 0x7f5e0c0a9d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:48:27.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:27.043+0000 7f5e1081b700 1 --2- 172.21.15.23:0/405838939 >> v2:172.21.15.155:3300/0 conn(0x7f5e0c13b100 0x7f5e0c066770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:27.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:27.043+0000 7f5e1081b700 1 -- 172.21.15.23:0/405838939 >> v2:172.21.15.155:3300/0 conn(0x7f5e0c13b100 msgr2=0x7f5e0c066770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:48:27.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:27.043+0000 7f5e1081b700 1 -- 172.21.15.23:0/405838939 >> v2:172.21.15.155:3300/0 conn(0x7f5e0c13b100 msgr2=0x7f5e0c066770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:48:27.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:27.043+0000 7f5e1081b700 1 --2- 172.21.15.23:0/405838939 >> v2:172.21.15.155:3300/0 conn(0x7f5e0c13b100 0x7f5e0c066770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:48:27.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:27.043+0000 7f5e1081b700 1 --2- 172.21.15.23:0/405838939 >> v2:172.21.15.155:3300/0 conn(0x7f5e0c13b100 0x7f5e0c066770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:48:27.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:27.044+0000 7f5e1101c700 1 --2- 172.21.15.23:0/405838939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e0c064f20 0x7f5e0c0a9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:27.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:27.044+0000 7f5e1101c700 1 -- 172.21.15.23:0/405838939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e0c064f20 msgr2=0x7f5e0c0a9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:48:27.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:27.044+0000 7f5e1101c700 1 -- 172.21.15.23:0/405838939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e0c064f20 msgr2=0x7f5e0c0a9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:48:27.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:27.044+0000 7f5e1101c700 1 --2- 172.21.15.23:0/405838939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e0c064f20 0x7f5e0c0a9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:48:27.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:27.044+0000 7f5e1101c700 1 --2- 172.21.15.23:0/405838939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e0c064f20 0x7f5e0c0a9d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:48:27.435 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:27.388+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 302 slow ops, oldest is osd_op(client.4659.0:65360 4.2s0 4:48389afd:::100000003b9.00000013:head [write 4014080~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:27.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:27.845+0000 7f5e1081b700 1 --2- 172.21.15.23:0/405838939 >> v2:172.21.15.155:3300/0 conn(0x7f5e0c13b100 0x7f5e0c066770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:27.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:27.845+0000 7f5e1101c700 1 --2- 172.21.15.23:0/405838939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e0c064f20 0x7f5e0c0a9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:27.850 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:27.845+0000 7f5e1101c700 1 -- 172.21.15.23:0/405838939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e0c064f20 msgr2=0x7f5e0c0a9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:48:27.850 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:27.845+0000 7f5e1101c700 1 -- 172.21.15.23:0/405838939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e0c064f20 msgr2=0x7f5e0c0a9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:48:27.850 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:27.845+0000 7f5e1101c700 1 --2- 172.21.15.23:0/405838939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e0c064f20 0x7f5e0c0a9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:48:27.850 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:27.845+0000 7f5e1101c700 1 --2- 172.21.15.23:0/405838939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e0c064f20 0x7f5e0c0a9d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:48:27.850 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:27.845+0000 7f5e1081b700 1 -- 172.21.15.23:0/405838939 >> v2:172.21.15.155:3300/0 conn(0x7f5e0c13b100 msgr2=0x7f5e0c066770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:48:27.851 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:27.845+0000 7f5e1081b700 1 -- 172.21.15.23:0/405838939 >> v2:172.21.15.155:3300/0 conn(0x7f5e0c13b100 msgr2=0x7f5e0c066770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:48:27.851 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:27.845+0000 7f5e1081b700 1 --2- 172.21.15.23:0/405838939 >> v2:172.21.15.155:3300/0 conn(0x7f5e0c13b100 0x7f5e0c066770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:48:27.851 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:27.845+0000 7f5e1081b700 1 --2- 172.21.15.23:0/405838939 >> v2:172.21.15.155:3300/0 conn(0x7f5e0c13b100 0x7f5e0c066770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:48:28.481 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:28.426+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 298 slow ops, oldest is osd_op(client.4659.0:65474 4.2s0 4:5b26cd0c:::100000003b9.00000017:head [write 356352~4096 in=4096b,write 1855488~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:29.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:29.441+0000 7f5dfaffd700 1 -- 172.21.15.23:0/405838939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e0c064f20 msgr2=0x7f5e0c0a9d50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:48:29.445 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:29.441+0000 7f5dfaffd700 1 --2- 172.21.15.23:0/405838939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e0c064f20 0x7f5e0c0a9d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:29.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:29.441+0000 7f5dfaffd700 1 -- 172.21.15.23:0/405838939 >> v2:172.21.15.155:3300/0 conn(0x7f5e0c13b100 msgr2=0x7f5e0c066770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:48:29.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:29.441+0000 7f5dfaffd700 1 --2- 172.21.15.23:0/405838939 >> v2:172.21.15.155:3300/0 conn(0x7f5e0c13b100 0x7f5e0c066770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:29.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:29.441+0000 7f5dfaffd700 1 -- 172.21.15.23:0/405838939 >> v1:172.21.15.155:6789/0 conn(0x7f5e0c066c60 legacy=0x7f5e0c13e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:29.446 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:29.441+0000 7f5dfaffd700 1 --2- 172.21.15.23:0/405838939 >> v2:172.21.15.23:3300/0 conn(0x7f5dec007080 0x7f5dec009540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:29.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:29.441+0000 7f5dfaffd700 1 -- 172.21.15.23:0/405838939 --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0c13b560 con 0x7f5dec003410 2020-11-20T11:48:29.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:29.441+0000 7f5dfaffd700 1 -- 172.21.15.23:0/405838939 --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e0c13bab0 con 0x7f5dec007080 2020-11-20T11:48:29.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:29.441+0000 7f5e1081b700 1 --2- 172.21.15.23:0/405838939 >> v2:172.21.15.23:3300/0 conn(0x7f5dec007080 0x7f5dec009540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:29.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:29.441+0000 7f5e1081b700 1 -- 172.21.15.23:0/405838939 >> v2:172.21.15.23:3300/0 conn(0x7f5dec007080 msgr2=0x7f5dec009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:48:29.447 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:29.441+0000 7f5e1081b700 1 -- 172.21.15.23:0/405838939 >> v2:172.21.15.23:3300/0 conn(0x7f5dec007080 msgr2=0x7f5dec009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:48:29.448 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:29.441+0000 7f5e1081b700 1 --2- 172.21.15.23:0/405838939 >> v2:172.21.15.23:3300/0 conn(0x7f5dec007080 0x7f5dec009540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:48:29.448 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:29.441+0000 7f5e1081b700 1 --2- 172.21.15.23:0/405838939 >> v2:172.21.15.23:3300/0 conn(0x7f5dec007080 0x7f5dec009540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:48:29.513 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:29.492+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 302 slow ops, oldest is osd_op(client.4659.0:66087 4.2s0 4:52da5114:::100000003c5.00000000:head [write 1110016~4096 in=4096b,write 2621440~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:29.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:29.642+0000 7f5e1081b700 1 --2- 172.21.15.23:0/405838939 >> v2:172.21.15.23:3300/0 conn(0x7f5dec007080 0x7f5dec009540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:29.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:29.642+0000 7f5e1081b700 1 -- 172.21.15.23:0/405838939 >> v2:172.21.15.23:3300/0 conn(0x7f5dec007080 msgr2=0x7f5dec009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:48:29.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:29.642+0000 7f5e1081b700 1 -- 172.21.15.23:0/405838939 >> v2:172.21.15.23:3300/0 conn(0x7f5dec007080 msgr2=0x7f5dec009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:48:29.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:29.642+0000 7f5e1081b700 1 --2- 172.21.15.23:0/405838939 >> v2:172.21.15.23:3300/0 conn(0x7f5dec007080 0x7f5dec009540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:48:29.647 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:29.642+0000 7f5e1081b700 1 --2- 172.21.15.23:0/405838939 >> v2:172.21.15.23:3300/0 conn(0x7f5dec007080 0x7f5dec009540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:48:30.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.043+0000 7f5e1081b700 1 --2- 172.21.15.23:0/405838939 >> v2:172.21.15.23:3300/0 conn(0x7f5dec007080 0x7f5dec009540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:30.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.043+0000 7f5e1081b700 1 -- 172.21.15.23:0/405838939 >> v2:172.21.15.23:3300/0 conn(0x7f5dec007080 msgr2=0x7f5dec009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:48:30.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.043+0000 7f5e1081b700 1 -- 172.21.15.23:0/405838939 >> v2:172.21.15.23:3300/0 conn(0x7f5dec007080 msgr2=0x7f5dec009540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:48:30.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.043+0000 7f5e1081b700 1 --2- 172.21.15.23:0/405838939 >> v2:172.21.15.23:3300/0 conn(0x7f5dec007080 0x7f5dec009540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:48:30.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.043+0000 7f5e1081b700 1 --2- 172.21.15.23:0/405838939 >> v2:172.21.15.23:3300/0 conn(0x7f5dec007080 0x7f5dec009540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:48:30.542 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:30.509+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4659.0:66087 4.2s0 4:52da5114:::100000003c5.00000000:head [write 1110016~4096 in=4096b,write 2621440~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:30.733 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.729+0000 7f5dfbfff700 1 -- 172.21.15.23:0/405838939 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3686934982 0 0) 0x7f5e0c13b560 con 0x7f5dec003410 2020-11-20T11:48:30.733 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.729+0000 7f5dfbfff700 1 -- 172.21.15.23:0/405838939 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5df4003280 con 0x7f5dec003410 2020-11-20T11:48:30.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.732+0000 7f5dfbfff700 1 -- 172.21.15.23:0/405838939 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3943832999 0 0) 0x7f5df4003280 con 0x7f5dec003410 2020-11-20T11:48:30.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.732+0000 7f5dfbfff700 1 -- 172.21.15.23:0/405838939 >> v2:172.21.15.23:3300/0 conn(0x7f5dec007080 msgr2=0x7f5dec009540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:48:30.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.732+0000 7f5dfbfff700 1 --2- 172.21.15.23:0/405838939 >> v2:172.21.15.23:3300/0 conn(0x7f5dec007080 0x7f5dec009540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:30.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.732+0000 7f5dfbfff700 1 -- 172.21.15.23:0/405838939 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dec00ac50 con 0x7f5dec003410 2020-11-20T11:48:30.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.732+0000 7f5dfbfff700 1 -- 172.21.15.23:0/405838939 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f5e0800d1c0 con 0x7f5dec003410 2020-11-20T11:48:30.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.732+0000 7f5dfbfff700 1 -- 172.21.15.23:0/405838939 >> v1:172.21.15.23:6789/0 conn(0x7f5dec003410 legacy=0x7f5dec005850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:30.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.732+0000 7f5e1181d700 1 -- 172.21.15.23:0/405838939 reap_dead start 2020-11-20T11:48:30.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.732+0000 7f5dfbfff700 1 --2- 172.21.15.23:0/405838939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e0c064f20 0x7f5df4003770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:30.744 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.732+0000 7f5dfbfff700 1 --2- 172.21.15.23:0/405838939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5df4003cb0 0x7f5df4004520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:30.744 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.732+0000 7f5dfbfff700 1 --2- 172.21.15.23:0/405838939 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5df4004a10 0x7f5df4004e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:30.744 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.732+0000 7f5dfbfff700 1 -- 172.21.15.23:0/405838939 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c13bab0 con 0x7f5df4004a10 2020-11-20T11:48:30.744 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.732+0000 7f5dfbfff700 1 -- 172.21.15.23:0/405838939 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df40042a0 con 0x7f5df4003cb0 2020-11-20T11:48:30.744 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.732+0000 7f5dfbfff700 1 -- 172.21.15.23:0/405838939 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5df400b770 con 0x7f5e0c064f20 2020-11-20T11:48:30.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.732+0000 7f5e1181d700 1 --2- 172.21.15.23:0/405838939 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5df4004a10 0x7f5df4004e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:30.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.732+0000 7f5e1081b700 1 --2- 172.21.15.23:0/405838939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5df4003cb0 0x7f5df4004520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:30.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.732+0000 7f5e1101c700 1 --2- 172.21.15.23:0/405838939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e0c064f20 0x7f5df4003770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:30.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.733+0000 7f5e1181d700 1 -- 172.21.15.23:0/405838939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e0c064f20 msgr2=0x7f5df4003770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:48:30.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.733+0000 7f5e1181d700 1 --2- 172.21.15.23:0/405838939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e0c064f20 0x7f5df4003770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:30.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.733+0000 7f5e1181d700 1 -- 172.21.15.23:0/405838939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5df4003cb0 msgr2=0x7f5df4004520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:48:30.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.733+0000 7f5e1181d700 1 --2- 172.21.15.23:0/405838939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5df4003cb0 0x7f5df4004520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:30.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.733+0000 7f5e1181d700 1 -- 172.21.15.23:0/405838939 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df4003280 con 0x7f5df4004a10 2020-11-20T11:48:30.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.733+0000 7f5e1181d700 1 --2- 172.21.15.23:0/405838939 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5df4004a10 0x7f5df4004e90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f5e0c068360 tx=0x7f5e04007f20).ready entity=mon.0 client_cookie=9da3b42d5687cae5 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:30.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.733+0000 7f5dfbfff700 1 -- 172.21.15.23:0/405838939 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e04005bc0 con 0x7f5df4004a10 2020-11-20T11:48:30.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.733+0000 7f5dfbfff700 1 -- 172.21.15.23:0/405838939 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e04005d20 con 0x7f5df4004a10 2020-11-20T11:48:30.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.733+0000 7f5dfbfff700 1 -- 172.21.15.23:0/405838939 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e040072e0 con 0x7f5df4004a10 2020-11-20T11:48:30.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.734+0000 7f5e13280700 1 -- 172.21.15.23:0/405838939 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5df4004a10 msgr2=0x7f5df4004e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:30.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.734+0000 7f5e13280700 1 --2- 172.21.15.23:0/405838939 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5df4004a10 0x7f5df4004e90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f5e0c068360 tx=0x7f5e04007f20).stop 2020-11-20T11:48:30.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.734+0000 7f5e13280700 1 -- 172.21.15.23:0/405838939 shutdown_connections 2020-11-20T11:48:30.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.734+0000 7f5e13280700 1 --2- 172.21.15.23:0/405838939 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5df4004a10 0x7f5df4004e90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:30.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.734+0000 7f5e13280700 1 --2- 172.21.15.23:0/405838939 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5df4003cb0 0x7f5df4004520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:30.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.734+0000 7f5e13280700 1 --2- 172.21.15.23:0/405838939 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e0c064f20 0x7f5df4003770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:30.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.734+0000 7f5e13280700 1 -- 172.21.15.23:0/405838939 >> 172.21.15.23:0/405838939 conn(0x7f5e0c0601b0 msgr2=0x7f5e0c0605c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:48:30.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.734+0000 7f5e13280700 1 -- 172.21.15.23:0/405838939 shutdown_connections 2020-11-20T11:48:30.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.734+0000 7f5e13280700 1 -- 172.21.15.23:0/405838939 wait complete. 2020-11-20T11:48:30.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.735+0000 7f5e13280700 1 Processor -- start 2020-11-20T11:48:30.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.735+0000 7f5e13280700 1 -- start start 2020-11-20T11:48:30.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.736+0000 7f5e13280700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5df4003cb0 0x7f5e0c0a2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:30.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.736+0000 7f5e13280700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5df4004a10 0x7f5e0c09e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:30.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.736+0000 7f5e13280700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e0c064f20 0x7f5e0c09ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:30.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.736+0000 7f5e13280700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c0a0310 con 0x7f5e0c064f20 2020-11-20T11:48:30.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.736+0000 7f5e13280700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c09fdd0 con 0x7f5df4003cb0 2020-11-20T11:48:30.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.736+0000 7f5e13280700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c0a0070 con 0x7f5df4004a10 2020-11-20T11:48:30.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.736+0000 7f5e1181d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e0c064f20 0x7f5e0c09ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:30.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.736+0000 7f5e1181d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e0c064f20 0x7f5e0c09ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46912/0 (socket says 172.21.15.23:46912) 2020-11-20T11:48:30.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.736+0000 7f5e1181d700 1 -- 172.21.15.23:0/1564986241 learned_addr learned my addr 172.21.15.23:0/1564986241 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:48:30.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.736+0000 7f5e1081b700 1 --2- 172.21.15.23:0/1564986241 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5df4004a10 0x7f5e0c09e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:30.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.736+0000 7f5e1101c700 1 --2- 172.21.15.23:0/1564986241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5df4003cb0 0x7f5e0c0a2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:30.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.736+0000 7f5e1081b700 1 -- 172.21.15.23:0/1564986241 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5df4004a10 msgr2=0x7f5e0c09e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2020-11-20T11:48:30.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.736+0000 7f5e1081b700 1 -- 172.21.15.23:0/1564986241 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5df4004a10 msgr2=0x7f5e0c09e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:48:30.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.736+0000 7f5e1081b700 1 --2- 172.21.15.23:0/1564986241 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5df4004a10 0x7f5e0c09e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:48:30.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.736+0000 7f5e1081b700 1 --2- 172.21.15.23:0/1564986241 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5df4004a10 0x7f5e0c09e990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:48:30.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.736+0000 7f5e1181d700 1 -- 172.21.15.23:0/1564986241 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5df4004a10 msgr2=0x7f5e0c09e990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:48:30.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.736+0000 7f5e1181d700 1 --2- 172.21.15.23:0/1564986241 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5df4004a10 0x7f5e0c09e990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:30.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.736+0000 7f5e1181d700 1 -- 172.21.15.23:0/1564986241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5df4003cb0 msgr2=0x7f5e0c0a2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:30.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.736+0000 7f5e1181d700 1 --2- 172.21.15.23:0/1564986241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5df4003cb0 0x7f5e0c0a2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:30.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.736+0000 7f5e1181d700 1 -- 172.21.15.23:0/1564986241 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0c09f9a0 con 0x7f5e0c064f20 2020-11-20T11:48:30.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.736+0000 7f5e1101c700 1 --2- 172.21.15.23:0/1564986241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5df4003cb0 0x7f5e0c0a2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:48:30.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.736+0000 7f5e1181d700 1 --2- 172.21.15.23:0/1564986241 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e0c064f20 0x7f5e0c09ee80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f5e0c063dc0 tx=0x7f5e04015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:30.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.737+0000 7f5df97fa700 1 -- 172.21.15.23:0/1564986241 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e040077e0 con 0x7f5e0c064f20 2020-11-20T11:48:30.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.737+0000 7f5df97fa700 1 -- 172.21.15.23:0/1564986241 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e04005e90 con 0x7f5e0c064f20 2020-11-20T11:48:30.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.737+0000 7f5e13280700 1 -- 172.21.15.23:0/1564986241 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e0c13c260 con 0x7f5e0c064f20 2020-11-20T11:48:30.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.737+0000 7f5df97fa700 1 -- 172.21.15.23:0/1564986241 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0400cc10 con 0x7f5e0c064f20 2020-11-20T11:48:30.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.737+0000 7f5e13280700 1 -- 172.21.15.23:0/1564986241 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e0c13d970 con 0x7f5e0c064f20 2020-11-20T11:48:30.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.738+0000 7f5e13280700 1 -- 172.21.15.23:0/1564986241 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e0c061650 con 0x7f5e0c064f20 2020-11-20T11:48:30.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.738+0000 7f5df97fa700 1 -- 172.21.15.23:0/1564986241 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5e04026070 con 0x7f5e0c064f20 2020-11-20T11:48:30.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.738+0000 7f5df97fa700 1 --2- 172.21.15.23:0/1564986241 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5de4056d60 0x7f5de4059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:30.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.738+0000 7f5df97fa700 1 -- 172.21.15.23:0/1564986241 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f5e04070930 con 0x7f5e0c064f20 2020-11-20T11:48:30.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.738+0000 7f5e1101c700 1 --2- 172.21.15.23:0/1564986241 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5de4056d60 0x7f5de4059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:30.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.739+0000 7f5e1101c700 1 --2- 172.21.15.23:0/1564986241 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5de4056d60 0x7f5de4059220 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f5e08006bc0 tx=0x7f5e08008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:30.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:30.742+0000 7f5df97fa700 1 -- 172.21.15.23:0/1564986241 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5e04045c20 con 0x7f5e0c064f20 2020-11-20T11:48:31.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:31.054+0000 7f5e13280700 1 -- 172.21.15.23:0/1564986241 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5e0c057dc0 con 0x7f5e0c064f20 2020-11-20T11:48:31.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:31.055+0000 7f5df97fa700 1 -- 172.21.15.23:0/1564986241 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1472 (secure 0 0 0) 0x7f5e04049240 con 0x7f5e0c064f20 2020-11-20T11:48:31.059 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":144,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:48:31.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:31.056+0000 7f5e13280700 1 -- 172.21.15.23:0/1564986241 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5de4056d60 msgr2=0x7f5de4059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:31.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:31.056+0000 7f5e13280700 1 --2- 172.21.15.23:0/1564986241 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5de4056d60 0x7f5de4059220 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f5e08006bc0 tx=0x7f5e08008040).stop 2020-11-20T11:48:31.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:31.056+0000 7f5e13280700 1 -- 172.21.15.23:0/1564986241 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e0c064f20 msgr2=0x7f5e0c09ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:31.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:31.056+0000 7f5e13280700 1 --2- 172.21.15.23:0/1564986241 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e0c064f20 0x7f5e0c09ee80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f5e0c063dc0 tx=0x7f5e04015000).stop 2020-11-20T11:48:31.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:31.056+0000 7f5e13280700 1 -- 172.21.15.23:0/1564986241 shutdown_connections 2020-11-20T11:48:31.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:31.056+0000 7f5e13280700 1 --2- 172.21.15.23:0/1564986241 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e0c064f20 0x7f5e0c09ee80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:31.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:31.056+0000 7f5e13280700 1 --2- 172.21.15.23:0/1564986241 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5df4003cb0 0x7f5e0c0a2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:31.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:31.056+0000 7f5e13280700 1 --2- 172.21.15.23:0/1564986241 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5de4056d60 0x7f5de4059220 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:31.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:31.056+0000 7f5e13280700 1 --2- 172.21.15.23:0/1564986241 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5df4004a10 0x7f5e0c09e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:31.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:31.056+0000 7f5e13280700 1 -- 172.21.15.23:0/1564986241 >> 172.21.15.23:0/1564986241 conn(0x7f5e0c0601b0 msgr2=0x7f5e0c1395b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:48:31.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:31.056+0000 7f5e13280700 1 -- 172.21.15.23:0/1564986241 shutdown_connections 2020-11-20T11:48:31.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:31.056+0000 7f5e13280700 1 -- 172.21.15.23:0/1564986241 wait complete. 2020-11-20T11:48:31.615 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:31.596+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 299 slow ops, oldest is osd_op(client.4659.0:66087 4.2s0 4:52da5114:::100000003c5.00000000:head [write 1110016~4096 in=4096b,write 2621440~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:32.654 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:32.591+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 296 slow ops, oldest is osd_op(client.4659.0:66087 4.2s0 4:52da5114:::100000003c5.00000000:head [write 1110016~4096 in=4096b,write 2621440~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:33.618 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:33.598+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 296 slow ops, oldest is osd_op(client.4659.0:66219 4.2s0 4:48e2ec13:::10000000356.00000002:head [write 1634304~4096 in=4096b,write 2953216~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:34.077 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:48:34.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:34.254+0000 7efeaffff700 1 Processor -- start 2020-11-20T11:48:34.258 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:34.254+0000 7efeaffff700 1 -- start start 2020-11-20T11:48:34.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:34.254+0000 7efeaffff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7efeb00a9040 0x7efeb00aaf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:34.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:34.254+0000 7efeaffff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb013ec20 con 0x7efeb00ace70 2020-11-20T11:48:34.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:34.254+0000 7efeaffff700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb00ad860 con 0x7efeb00ad2f0 2020-11-20T11:48:34.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:34.254+0000 7efeaffff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeb00aa8a0 con 0x7efeb00a9040 2020-11-20T11:48:34.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:34.255+0000 7efeaeffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7efeb00a9040 0x7efeb00aaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:34.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:34.255+0000 7efea7fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7efeb00ace70 0x7efeb013de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:32934/0 (socket says 172.21.15.23:32934) 2020-11-20T11:48:34.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:34.255+0000 7efea7fff700 1 -- 172.21.15.23:0/853368061 learned_addr learned my addr 172.21.15.23:0/853368061 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:48:34.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:34.255+0000 7efeaeffd700 1 -- 172.21.15.23:0/853368061 >> v2:172.21.15.23:3300/0 conn(0x7efeb00a9040 msgr2=0x7efeb00aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:48:34.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:34.255+0000 7efeaeffd700 1 -- 172.21.15.23:0/853368061 >> v2:172.21.15.23:3300/0 conn(0x7efeb00a9040 msgr2=0x7efeb00aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:48:34.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:34.255+0000 7efeaeffd700 1 --2- 172.21.15.23:0/853368061 >> v2:172.21.15.23:3300/0 conn(0x7efeb00a9040 0x7efeb00aaf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:48:34.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:34.255+0000 7efeaeffd700 1 --2- 172.21.15.23:0/853368061 >> v2:172.21.15.23:3300/0 conn(0x7efeb00a9040 0x7efeb00aaf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:48:34.460 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:34.456+0000 7efeaeffd700 1 --2- 172.21.15.23:0/853368061 >> v2:172.21.15.23:3300/0 conn(0x7efeb00a9040 0x7efeb00aaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:34.460 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:34.456+0000 7efeaeffd700 1 -- 172.21.15.23:0/853368061 >> v2:172.21.15.23:3300/0 conn(0x7efeb00a9040 msgr2=0x7efeb00aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:48:34.460 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:34.456+0000 7efeaeffd700 1 -- 172.21.15.23:0/853368061 >> v2:172.21.15.23:3300/0 conn(0x7efeb00a9040 msgr2=0x7efeb00aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:48:34.461 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:34.456+0000 7efeaeffd700 1 --2- 172.21.15.23:0/853368061 >> v2:172.21.15.23:3300/0 conn(0x7efeb00a9040 0x7efeb00aaf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:48:34.461 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:34.456+0000 7efeaeffd700 1 --2- 172.21.15.23:0/853368061 >> v2:172.21.15.23:3300/0 conn(0x7efeb00a9040 0x7efeb00aaf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:48:34.686 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:34.648+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 296 slow ops, oldest is osd_op(client.4659.0:66509 4.2s0 4:5bc0f152:::100000003c2.00000013:head [write 1486848~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:34.861 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:34.857+0000 7efeaeffd700 1 --2- 172.21.15.23:0/853368061 >> v2:172.21.15.23:3300/0 conn(0x7efeb00a9040 0x7efeb00aaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:34.861 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:34.857+0000 7efeaeffd700 1 -- 172.21.15.23:0/853368061 >> v2:172.21.15.23:3300/0 conn(0x7efeb00a9040 msgr2=0x7efeb00aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:48:34.862 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:34.857+0000 7efeaeffd700 1 -- 172.21.15.23:0/853368061 >> v2:172.21.15.23:3300/0 conn(0x7efeb00a9040 msgr2=0x7efeb00aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:48:34.862 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:34.857+0000 7efeaeffd700 1 --2- 172.21.15.23:0/853368061 >> v2:172.21.15.23:3300/0 conn(0x7efeb00a9040 0x7efeb00aaf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:48:34.862 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:34.857+0000 7efeaeffd700 1 --2- 172.21.15.23:0/853368061 >> v2:172.21.15.23:3300/0 conn(0x7efeb00a9040 0x7efeb00aaf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:48:35.662 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:35.658+0000 7efeaeffd700 1 --2- 172.21.15.23:0/853368061 >> v2:172.21.15.23:3300/0 conn(0x7efeb00a9040 0x7efeb00aaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:35.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:35.658+0000 7efeaeffd700 1 -- 172.21.15.23:0/853368061 >> v2:172.21.15.23:3300/0 conn(0x7efeb00a9040 msgr2=0x7efeb00aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:48:35.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:35.658+0000 7efeaeffd700 1 -- 172.21.15.23:0/853368061 >> v2:172.21.15.23:3300/0 conn(0x7efeb00a9040 msgr2=0x7efeb00aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:48:35.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:35.658+0000 7efeaeffd700 1 --2- 172.21.15.23:0/853368061 >> v2:172.21.15.23:3300/0 conn(0x7efeb00a9040 0x7efeb00aaf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:48:35.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:35.658+0000 7efeaeffd700 1 --2- 172.21.15.23:0/853368061 >> v2:172.21.15.23:3300/0 conn(0x7efeb00a9040 0x7efeb00aaf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:48:35.688 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:35.650+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 296 slow ops, oldest is osd_op(client.4659.0:66509 4.2s0 4:5bc0f152:::100000003c2.00000013:head [write 1486848~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:36.705 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:36.685+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 294 slow ops, oldest is osd_op(client.4659.0:66860 4.2s0 4:5196b857:::10000000386.0000000c:head [write 4100096~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:36.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.741+0000 7efeae7fc700 1 -- 172.21.15.23:0/853368061 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2645721084 0 0) 0x7efeb00ad860 con 0x7efeb00ad2f0 2020-11-20T11:48:36.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.741+0000 7efeae7fc700 1 -- 172.21.15.23:0/853368061 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe90003280 con 0x7efeb00ad2f0 2020-11-20T11:48:36.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.741+0000 7efeae7fc700 1 -- 172.21.15.23:0/853368061 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2716943461 0 0) 0x7efe90003280 con 0x7efeb00ad2f0 2020-11-20T11:48:36.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.741+0000 7efeae7fc700 1 -- 172.21.15.23:0/853368061 >> v2:172.21.15.23:3300/0 conn(0x7efeb00a9040 msgr2=0x7efeb00aaf30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:48:36.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.741+0000 7efeae7fc700 1 --2- 172.21.15.23:0/853368061 >> v2:172.21.15.23:3300/0 conn(0x7efeb00a9040 0x7efeb00aaf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:36.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.741+0000 7efeae7fc700 1 -- 172.21.15.23:0/853368061 >> v1:172.21.15.23:6789/0 conn(0x7efeb00ace70 legacy=0x7efeb013de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:36.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.741+0000 7efeae7fc700 1 -- 172.21.15.23:0/853368061 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb00aaa90 con 0x7efeb00ad2f0 2020-11-20T11:48:36.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.741+0000 7efeae7fc700 1 -- 172.21.15.23:0/853368061 <== mon.1 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7efea80018a0 con 0x7efeb00ad2f0 2020-11-20T11:48:36.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.741+0000 7efeae7fc700 1 -- 172.21.15.23:0/853368061 >> v1:172.21.15.155:6789/0 conn(0x7efeb00ad2f0 legacy=0x7efeb013e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:36.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.741+0000 7efeae7fc700 1 --2- 172.21.15.23:0/853368061 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe90003770 0x7efe90003b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:36.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.741+0000 7efeae7fc700 1 --2- 172.21.15.23:0/853368061 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe90004520 0x7efe900049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:36.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.741+0000 7efeae7fc700 1 --2- 172.21.15.23:0/853368061 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe90004e90 0x7efe9000b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:36.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.741+0000 7efeae7fc700 1 -- 172.21.15.23:0/853368061 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe90004250 con 0x7efe90003770 2020-11-20T11:48:36.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.741+0000 7efeae7fc700 1 -- 172.21.15.23:0/853368061 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe9000bc60 con 0x7efe90004520 2020-11-20T11:48:36.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.741+0000 7efeae7fc700 1 -- 172.21.15.23:0/853368061 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe9000bda0 con 0x7efe90004e90 2020-11-20T11:48:36.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.742+0000 7efea7fff700 1 --2- 172.21.15.23:0/853368061 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe90004520 0x7efe900049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:36.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.742+0000 7efeaf7fe700 1 --2- 172.21.15.23:0/853368061 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe90004e90 0x7efe9000b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:36.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.742+0000 7efeaf7fe700 1 -- 172.21.15.23:0/853368061 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe90004520 msgr2=0x7efe900049a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:48:36.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.742+0000 7efeaf7fe700 1 --2- 172.21.15.23:0/853368061 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe90004520 0x7efe900049a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:36.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.742+0000 7efeaf7fe700 1 -- 172.21.15.23:0/853368061 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe90003770 msgr2=0x7efe90003b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:48:36.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.742+0000 7efeaf7fe700 1 --2- 172.21.15.23:0/853368061 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe90003770 0x7efe90003b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:48:36.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.742+0000 7efeaf7fe700 1 -- 172.21.15.23:0/853368061 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb00aaa90 con 0x7efe90004e90 2020-11-20T11:48:36.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.742+0000 7efeaf7fe700 1 -- 172.21.15.23:0/853368061 reap_dead start 2020-11-20T11:48:36.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.743+0000 7efeaf7fe700 1 --2- 172.21.15.23:0/853368061 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe90004e90 0x7efe9000b770 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7efea800c670 tx=0x7efea8007b30).ready entity=mon.2 client_cookie=da18f101ce8da42 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:36.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.743+0000 7efeae7fc700 1 -- 172.21.15.23:0/853368061 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea800e4a0 con 0x7efe90004e90 2020-11-20T11:48:36.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.743+0000 7efeae7fc700 1 -- 172.21.15.23:0/853368061 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efea800e640 con 0x7efe90004e90 2020-11-20T11:48:36.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.743+0000 7efeae7fc700 1 -- 172.21.15.23:0/853368061 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea800ea00 con 0x7efe90004e90 2020-11-20T11:48:36.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.743+0000 7efeaffff700 1 -- 172.21.15.23:0/853368061 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe90004e90 msgr2=0x7efe9000b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:36.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.743+0000 7efeaffff700 1 --2- 172.21.15.23:0/853368061 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe90004e90 0x7efe9000b770 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7efea800c670 tx=0x7efea8007b30).stop 2020-11-20T11:48:36.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.743+0000 7efeaffff700 1 -- 172.21.15.23:0/853368061 shutdown_connections 2020-11-20T11:48:36.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.743+0000 7efeaffff700 1 --2- 172.21.15.23:0/853368061 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe90004e90 0x7efe9000b770 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:36.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.743+0000 7efeaffff700 1 -- 172.21.15.23:0/853368061 >> 172.21.15.23:0/853368061 conn(0x7efeb00683b0 msgr2=0x7efeb00b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:48:36.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.744+0000 7efeaffff700 1 -- 172.21.15.23:0/853368061 shutdown_connections 2020-11-20T11:48:36.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.744+0000 7efeaffff700 1 -- 172.21.15.23:0/853368061 wait complete. 2020-11-20T11:48:36.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.745+0000 7efeaffff700 1 Processor -- start 2020-11-20T11:48:36.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.745+0000 7efeaffff700 1 -- start start 2020-11-20T11:48:36.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.745+0000 7efeaffff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe90004e90 0x7efeb00a14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:36.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.745+0000 7efeaffff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efeb009f360 0x7efeb009e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:36.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.745+0000 7efeaffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efeb009e8f0 0x7efeb0149c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:36.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.745+0000 7efeaffff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb00a0820 con 0x7efeb009e8f0 2020-11-20T11:48:36.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.745+0000 7efeaffff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb009ed70 con 0x7efe90004e90 2020-11-20T11:48:36.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.745+0000 7efeaffff700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeb009f8e0 con 0x7efeb009f360 2020-11-20T11:48:36.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.745+0000 7efeaf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efeb009e8f0 0x7efeb0149c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:36.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.745+0000 7efeaeffd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe90004e90 0x7efeb00a14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:36.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.745+0000 7efeaf7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efeb009e8f0 0x7efeb0149c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46938/0 (socket says 172.21.15.23:46938) 2020-11-20T11:48:36.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.745+0000 7efeaeffd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe90004e90 0x7efeb00a14b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:39830/0 (socket says 172.21.15.23:39830) 2020-11-20T11:48:36.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.745+0000 7efeaf7fe700 1 -- 172.21.15.23:0/58147711 learned_addr learned my addr 172.21.15.23:0/58147711 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:48:36.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.746+0000 7efea7fff700 1 --2- 172.21.15.23:0/58147711 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efeb009f360 0x7efeb009e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:36.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.746+0000 7efea7fff700 1 -- 172.21.15.23:0/58147711 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efeb009f360 msgr2=0x7efeb009e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2020-11-20T11:48:36.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.746+0000 7efea7fff700 1 -- 172.21.15.23:0/58147711 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efeb009f360 msgr2=0x7efeb009e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:48:36.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.746+0000 7efea7fff700 1 --2- 172.21.15.23:0/58147711 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efeb009f360 0x7efeb009e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:48:36.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.746+0000 7efea7fff700 1 --2- 172.21.15.23:0/58147711 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efeb009f360 0x7efeb009e400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:48:36.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.746+0000 7efeaeffd700 1 -- 172.21.15.23:0/58147711 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efeb009f360 msgr2=0x7efeb009e400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:48:36.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.746+0000 7efeaeffd700 1 --2- 172.21.15.23:0/58147711 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efeb009f360 0x7efeb009e400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:36.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.746+0000 7efeaeffd700 1 -- 172.21.15.23:0/58147711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efeb009e8f0 msgr2=0x7efeb0149c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:36.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.746+0000 7efeaeffd700 1 --2- 172.21.15.23:0/58147711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efeb009e8f0 0x7efeb0149c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:36.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.746+0000 7efeaeffd700 1 -- 172.21.15.23:0/58147711 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb009fb80 con 0x7efe90004e90 2020-11-20T11:48:36.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.746+0000 7efeaeffd700 1 --2- 172.21.15.23:0/58147711 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe90004e90 0x7efeb00a14b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7efea0006be0 tx=0x7efea0008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:36.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.748+0000 7efea6ffd700 1 -- 172.21.15.23:0/58147711 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea00156a0 con 0x7efe90004e90 2020-11-20T11:48:36.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.749+0000 7efea6ffd700 1 -- 172.21.15.23:0/58147711 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efea0015820 con 0x7efe90004e90 2020-11-20T11:48:36.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.749+0000 7efeaffff700 1 -- 172.21.15.23:0/58147711 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeb009fe20 con 0x7efe90004e90 2020-11-20T11:48:36.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.749+0000 7efeaffff700 1 -- 172.21.15.23:0/58147711 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeb009d610 con 0x7efe90004e90 2020-11-20T11:48:36.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.750+0000 7efea6ffd700 1 -- 172.21.15.23:0/58147711 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea0015be0 con 0x7efe90004e90 2020-11-20T11:48:36.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.750+0000 7efea6ffd700 1 -- 172.21.15.23:0/58147711 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7efea0012040 con 0x7efe90004e90 2020-11-20T11:48:36.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.750+0000 7efeaffff700 1 -- 172.21.15.23:0/58147711 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeb0057dc0 con 0x7efe90004e90 2020-11-20T11:48:36.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.750+0000 7efea6ffd700 1 --2- 172.21.15.23:0/58147711 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efe8c056d60 0x7efe8c059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:36.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.750+0000 7efea6ffd700 1 -- 172.21.15.23:0/58147711 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7efea0070480 con 0x7efe90004e90 2020-11-20T11:48:36.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.750+0000 7efea7fff700 1 --2- 172.21.15.23:0/58147711 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efe8c056d60 0x7efe8c059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:36.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.751+0000 7efea7fff700 1 --2- 172.21.15.23:0/58147711 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efe8c056d60 0x7efe8c059220 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7efeb00a78d0 tx=0x7efe98009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:36.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:36.754+0000 7efea6ffd700 1 -- 172.21.15.23:0/58147711 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7efea0045700 con 0x7efe90004e90 2020-11-20T11:48:37.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:37.055+0000 7efeaffff700 1 -- 172.21.15.23:0/58147711 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7efeb00a9420 con 0x7efe90004e90 2020-11-20T11:48:37.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:37.056+0000 7efea6ffd700 1 -- 172.21.15.23:0/58147711 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1472 (secure 0 0 0) 0x7efea0048d20 con 0x7efe90004e90 2020-11-20T11:48:37.060 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":148,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:48:37.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:37.057+0000 7efeaffff700 1 -- 172.21.15.23:0/58147711 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efe8c056d60 msgr2=0x7efe8c059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:37.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:37.057+0000 7efeaffff700 1 --2- 172.21.15.23:0/58147711 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efe8c056d60 0x7efe8c059220 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7efeb00a78d0 tx=0x7efe98009040).stop 2020-11-20T11:48:37.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:37.057+0000 7efeaffff700 1 -- 172.21.15.23:0/58147711 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe90004e90 msgr2=0x7efeb00a14b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:37.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:37.057+0000 7efeaffff700 1 --2- 172.21.15.23:0/58147711 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe90004e90 0x7efeb00a14b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7efea0006be0 tx=0x7efea0008040).stop 2020-11-20T11:48:37.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:37.057+0000 7efeaffff700 1 -- 172.21.15.23:0/58147711 shutdown_connections 2020-11-20T11:48:37.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:37.057+0000 7efeaffff700 1 --2- 172.21.15.23:0/58147711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efeb009e8f0 0x7efeb0149c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:37.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:37.057+0000 7efeaffff700 1 --2- 172.21.15.23:0/58147711 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe90004e90 0x7efeb00a14b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:37.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:37.057+0000 7efeaffff700 1 --2- 172.21.15.23:0/58147711 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efe8c056d60 0x7efe8c059220 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:37.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:37.057+0000 7efeaffff700 1 --2- 172.21.15.23:0/58147711 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efeb009f360 0x7efeb009e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:37.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:37.057+0000 7efeaffff700 1 -- 172.21.15.23:0/58147711 >> 172.21.15.23:0/58147711 conn(0x7efeb00683b0 msgr2=0x7efeb00aef80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:48:37.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:37.058+0000 7efeaffff700 1 -- 172.21.15.23:0/58147711 shutdown_connections 2020-11-20T11:48:37.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:37.058+0000 7efeaffff700 1 -- 172.21.15.23:0/58147711 wait complete. 2020-11-20T11:48:37.936 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:37.899+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 293 slow ops, oldest is osd_op(client.4659.0:67009 4.2s0 4:5da044e0:::10000000364.00000018:head [write 1347584~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:38.979 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:38.949+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 296 slow ops, oldest is osd_op(client.4659.0:67202 4.2s0 4:5da044e0:::10000000364.00000018:head [write 1482752~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:40.003 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:39.942+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 294 slow ops, oldest is osd_op(client.4659.0:67554 4.2s0 4:4bcd2dac:::10000000388.00000008:head [write 1056768~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:40.079 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:48:40.316 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:40.313+0000 7fac3696e700 1 Processor -- start 2020-11-20T11:48:40.316 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:40.313+0000 7fac3696e700 1 -- start start 2020-11-20T11:48:40.317 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:40.313+0000 7fac3696e700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fac300a6e10 0x7fac30061850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:40.318 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:40.313+0000 7fac3696e700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac30062ad0 con 0x7fac30137870 2020-11-20T11:48:40.318 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:40.313+0000 7fac3696e700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac3013b580 con 0x7fac30137410 2020-11-20T11:48:40.318 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:40.313+0000 7fac3696e700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac300a61f0 con 0x7fac300a6e10 2020-11-20T11:48:40.318 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:40.314+0000 7fac34f0b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fac30137870 0x7fac3013ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:32996/0 (socket says 172.21.15.23:32996) 2020-11-20T11:48:40.319 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:40.314+0000 7fac34f0b700 1 -- 172.21.15.23:0/3218211288 learned_addr learned my addr 172.21.15.23:0/3218211288 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:48:40.319 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:40.314+0000 7fac2ffff700 1 --2- 172.21.15.23:0/3218211288 >> v2:172.21.15.155:3300/0 conn(0x7fac300a6e10 0x7fac30061850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:40.319 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:40.314+0000 7fac2ffff700 1 -- 172.21.15.23:0/3218211288 >> v2:172.21.15.155:3300/0 conn(0x7fac300a6e10 msgr2=0x7fac30061850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:48:40.319 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:40.314+0000 7fac2ffff700 1 -- 172.21.15.23:0/3218211288 >> v2:172.21.15.155:3300/0 conn(0x7fac300a6e10 msgr2=0x7fac30061850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:48:40.319 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:40.314+0000 7fac2ffff700 1 --2- 172.21.15.23:0/3218211288 >> v2:172.21.15.155:3300/0 conn(0x7fac300a6e10 0x7fac30061850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:48:40.320 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:40.314+0000 7fac2ffff700 1 --2- 172.21.15.23:0/3218211288 >> v2:172.21.15.155:3300/0 conn(0x7fac300a6e10 0x7fac30061850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:48:40.518 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:40.515+0000 7fac2ffff700 1 --2- 172.21.15.23:0/3218211288 >> v2:172.21.15.155:3300/0 conn(0x7fac300a6e10 0x7fac30061850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:40.518 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:40.515+0000 7fac2ffff700 1 -- 172.21.15.23:0/3218211288 >> v2:172.21.15.155:3300/0 conn(0x7fac300a6e10 msgr2=0x7fac30061850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:48:40.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:40.515+0000 7fac2ffff700 1 -- 172.21.15.23:0/3218211288 >> v2:172.21.15.155:3300/0 conn(0x7fac300a6e10 msgr2=0x7fac30061850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:48:40.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:40.515+0000 7fac2ffff700 1 --2- 172.21.15.23:0/3218211288 >> v2:172.21.15.155:3300/0 conn(0x7fac300a6e10 0x7fac30061850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:48:40.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:40.515+0000 7fac2ffff700 1 --2- 172.21.15.23:0/3218211288 >> v2:172.21.15.155:3300/0 conn(0x7fac300a6e10 0x7fac30061850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:48:40.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:40.916+0000 7fac2ffff700 1 --2- 172.21.15.23:0/3218211288 >> v2:172.21.15.155:3300/0 conn(0x7fac300a6e10 0x7fac30061850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:40.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:40.916+0000 7fac2ffff700 1 -- 172.21.15.23:0/3218211288 >> v2:172.21.15.155:3300/0 conn(0x7fac300a6e10 msgr2=0x7fac30061850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:48:40.920 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:40.916+0000 7fac2ffff700 1 -- 172.21.15.23:0/3218211288 >> v2:172.21.15.155:3300/0 conn(0x7fac300a6e10 msgr2=0x7fac30061850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:48:40.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:40.916+0000 7fac2ffff700 1 --2- 172.21.15.23:0/3218211288 >> v2:172.21.15.155:3300/0 conn(0x7fac300a6e10 0x7fac30061850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:48:40.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:40.916+0000 7fac2ffff700 1 --2- 172.21.15.23:0/3218211288 >> v2:172.21.15.155:3300/0 conn(0x7fac300a6e10 0x7fac30061850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:48:41.010 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:40.960+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 295 slow ops, oldest is osd_op(client.4659.0:67554 4.2s0 4:4bcd2dac:::10000000388.00000008:head [write 1056768~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:41.721 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:41.717+0000 7fac2ffff700 1 --2- 172.21.15.23:0/3218211288 >> v2:172.21.15.155:3300/0 conn(0x7fac300a6e10 0x7fac30061850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:41.721 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:41.717+0000 7fac2ffff700 1 -- 172.21.15.23:0/3218211288 >> v2:172.21.15.155:3300/0 conn(0x7fac300a6e10 msgr2=0x7fac30061850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-11-20T11:48:41.722 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:41.717+0000 7fac2ffff700 1 -- 172.21.15.23:0/3218211288 >> v2:172.21.15.155:3300/0 conn(0x7fac300a6e10 msgr2=0x7fac30061850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:48:41.722 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:41.717+0000 7fac2ffff700 1 --2- 172.21.15.23:0/3218211288 >> v2:172.21.15.155:3300/0 conn(0x7fac300a6e10 0x7fac30061850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:48:41.722 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:41.717+0000 7fac2ffff700 1 --2- 172.21.15.23:0/3218211288 >> v2:172.21.15.155:3300/0 conn(0x7fac300a6e10 0x7fac30061850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:48:42.058 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:42.024+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 295 slow ops, oldest is osd_op(client.4659.0:67780 4.2s0 4:542fa72b:::100000003c2.00000000:head [write 1961984~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:42.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.746+0000 7fac2effd700 1 -- 172.21.15.23:0/3218211288 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1101686428 0 0) 0x7fac3013b580 con 0x7fac30137410 2020-11-20T11:48:42.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.746+0000 7fac2effd700 1 -- 172.21.15.23:0/3218211288 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac1c003280 con 0x7fac30137410 2020-11-20T11:48:42.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.747+0000 7fac2effd700 1 -- 172.21.15.23:0/3218211288 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2463523709 0 0) 0x7fac1c003280 con 0x7fac30137410 2020-11-20T11:48:42.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.747+0000 7fac2effd700 1 -- 172.21.15.23:0/3218211288 >> v2:172.21.15.155:3300/0 conn(0x7fac300a6e10 msgr2=0x7fac30061850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:48:42.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.747+0000 7fac2effd700 1 --2- 172.21.15.23:0/3218211288 >> v2:172.21.15.155:3300/0 conn(0x7fac300a6e10 0x7fac30061850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:42.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.747+0000 7fac2effd700 1 -- 172.21.15.23:0/3218211288 >> v1:172.21.15.23:6789/0 conn(0x7fac30137870 legacy=0x7fac3013ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:42.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.747+0000 7fac2effd700 1 -- 172.21.15.23:0/3218211288 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac300a63e0 con 0x7fac30137410 2020-11-20T11:48:42.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.747+0000 7fac2effd700 1 -- 172.21.15.23:0/3218211288 <== mon.1 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7fac20001520 con 0x7fac30137410 2020-11-20T11:48:42.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.747+0000 7fac2effd700 1 -- 172.21.15.23:0/3218211288 >> v1:172.21.15.155:6789/0 conn(0x7fac30137410 legacy=0x7fac3013a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:42.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.747+0000 7fac2effd700 1 --2- 172.21.15.23:0/3218211288 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac1c003770 0x7fac1c003b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:42.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.747+0000 7fac2effd700 1 --2- 172.21.15.23:0/3218211288 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fac1c004080 0x7fac1c004520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:42.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.747+0000 7fac2effd700 1 --2- 172.21.15.23:0/3218211288 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac1c004ab0 0x7fac1c004f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:42.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.747+0000 7fac2effd700 1 -- 172.21.15.23:0/3218211288 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c00fdb0 con 0x7fac1c004ab0 2020-11-20T11:48:42.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.747+0000 7fac2effd700 1 -- 172.21.15.23:0/3218211288 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c00ff20 con 0x7fac1c003770 2020-11-20T11:48:42.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.747+0000 7fac2effd700 1 -- 172.21.15.23:0/3218211288 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c0100e0 con 0x7fac1c004080 2020-11-20T11:48:42.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.747+0000 7fac2ffff700 1 --2- 172.21.15.23:0/3218211288 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac1c003770 0x7fac1c003b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:42.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.747+0000 7fac2f7fe700 1 --2- 172.21.15.23:0/3218211288 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fac1c004080 0x7fac1c004520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:42.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.748+0000 7fac2ffff700 1 -- 172.21.15.23:0/3218211288 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fac1c004080 msgr2=0x7fac1c004520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:48:42.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.748+0000 7fac2ffff700 1 --2- 172.21.15.23:0/3218211288 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fac1c004080 0x7fac1c004520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:42.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.748+0000 7fac2ffff700 1 -- 172.21.15.23:0/3218211288 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac1c004ab0 msgr2=0x7fac1c004f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:48:42.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.748+0000 7fac2ffff700 1 --2- 172.21.15.23:0/3218211288 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac1c004ab0 0x7fac1c004f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:48:42.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.748+0000 7fac2ffff700 1 -- 172.21.15.23:0/3218211288 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac1c003280 con 0x7fac1c003770 2020-11-20T11:48:42.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.748+0000 7fac34f0b700 1 -- 172.21.15.23:0/3218211288 reap_dead start 2020-11-20T11:48:42.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.748+0000 7fac2ffff700 1 --2- 172.21.15.23:0/3218211288 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac1c003770 0x7fac1c003b90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fac24011a20 tx=0x7fac24008040).ready entity=mon.1 client_cookie=d820dabf497473a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:42.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.748+0000 7fac2effd700 1 -- 172.21.15.23:0/3218211288 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac240139e0 con 0x7fac1c003770 2020-11-20T11:48:42.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.748+0000 7fac2effd700 1 -- 172.21.15.23:0/3218211288 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac24013b40 con 0x7fac1c003770 2020-11-20T11:48:42.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.748+0000 7fac2effd700 1 -- 172.21.15.23:0/3218211288 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac24013ec0 con 0x7fac1c003770 2020-11-20T11:48:42.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.749+0000 7fac3696e700 1 -- 172.21.15.23:0/3218211288 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac1c003770 msgr2=0x7fac1c003b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:42.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.749+0000 7fac3696e700 1 --2- 172.21.15.23:0/3218211288 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac1c003770 0x7fac1c003b90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fac24011a20 tx=0x7fac24008040).stop 2020-11-20T11:48:42.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.749+0000 7fac3696e700 1 -- 172.21.15.23:0/3218211288 shutdown_connections 2020-11-20T11:48:42.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.749+0000 7fac3696e700 1 --2- 172.21.15.23:0/3218211288 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac1c003770 0x7fac1c003b90 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:42.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.749+0000 7fac3696e700 1 -- 172.21.15.23:0/3218211288 >> 172.21.15.23:0/3218211288 conn(0x7fac30060c80 msgr2=0x7fac300acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:48:42.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.749+0000 7fac3696e700 1 -- 172.21.15.23:0/3218211288 shutdown_connections 2020-11-20T11:48:42.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.749+0000 7fac3696e700 1 -- 172.21.15.23:0/3218211288 wait complete. 2020-11-20T11:48:42.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.751+0000 7fac3696e700 1 Processor -- start 2020-11-20T11:48:42.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.751+0000 7fac3696e700 1 -- start start 2020-11-20T11:48:42.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.752+0000 7fac3696e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fac1c003770 0x7fac30067ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:42.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.752+0000 7fac3696e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac300ac040 0x7fac3015a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:42.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.752+0000 7fac3696e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac3015acd0 0x7fac30065ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:42.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.752+0000 7fac3696e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac300a9e20 con 0x7fac300ac040 2020-11-20T11:48:42.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.752+0000 7fac3696e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac300ac4c0 con 0x7fac3015acd0 2020-11-20T11:48:42.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.752+0000 7fac3696e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac300aa010 con 0x7fac1c003770 2020-11-20T11:48:42.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.753+0000 7fac34f0b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac3015acd0 0x7fac30065ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:42.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.753+0000 7fac34f0b700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac3015acd0 0x7fac30065ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:39908/0 (socket says 172.21.15.23:39908) 2020-11-20T11:48:42.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.753+0000 7fac34f0b700 1 -- 172.21.15.23:0/2444935013 learned_addr learned my addr 172.21.15.23:0/2444935013 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:48:42.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.753+0000 7fac2ffff700 1 --2- 172.21.15.23:0/2444935013 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fac1c003770 0x7fac30067ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:42.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.753+0000 7fac2f7fe700 1 --2- 172.21.15.23:0/2444935013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac300ac040 0x7fac3015a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:42.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.753+0000 7fac34f0b700 1 -- 172.21.15.23:0/2444935013 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fac1c003770 msgr2=0x7fac30067ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:42.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.753+0000 7fac34f0b700 1 --2- 172.21.15.23:0/2444935013 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fac1c003770 0x7fac30067ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:42.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.753+0000 7fac34f0b700 1 -- 172.21.15.23:0/2444935013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac300ac040 msgr2=0x7fac3015a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:42.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.753+0000 7fac34f0b700 1 --2- 172.21.15.23:0/2444935013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac300ac040 0x7fac3015a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:42.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.753+0000 7fac34f0b700 1 -- 172.21.15.23:0/2444935013 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac300aa2b0 con 0x7fac3015acd0 2020-11-20T11:48:42.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.753+0000 7fac34f0b700 1 --2- 172.21.15.23:0/2444935013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac3015acd0 0x7fac30065ec0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fac30137c50 tx=0x7fac28007ce0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:42.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.754+0000 7fac17fff700 1 -- 172.21.15.23:0/2444935013 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac28005bc0 con 0x7fac3015acd0 2020-11-20T11:48:42.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.754+0000 7fac3696e700 1 -- 172.21.15.23:0/2444935013 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac300aa550 con 0x7fac3015acd0 2020-11-20T11:48:42.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.754+0000 7fac3696e700 1 -- 172.21.15.23:0/2444935013 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac30138470 con 0x7fac3015acd0 2020-11-20T11:48:42.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.754+0000 7fac17fff700 1 -- 172.21.15.23:0/2444935013 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac28005d20 con 0x7fac3015acd0 2020-11-20T11:48:42.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.754+0000 7fac17fff700 1 -- 172.21.15.23:0/2444935013 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac28007280 con 0x7fac3015acd0 2020-11-20T11:48:42.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.756+0000 7fac3696e700 1 -- 172.21.15.23:0/2444935013 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac300adca0 con 0x7fac3015acd0 2020-11-20T11:48:42.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.756+0000 7fac17fff700 1 -- 172.21.15.23:0/2444935013 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fac2801b060 con 0x7fac3015acd0 2020-11-20T11:48:42.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.757+0000 7fac17fff700 1 --2- 172.21.15.23:0/2444935013 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fac04056db0 0x7fac04059270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:42.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.757+0000 7fac17fff700 1 -- 172.21.15.23:0/2444935013 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fac2806f810 con 0x7fac3015acd0 2020-11-20T11:48:42.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.757+0000 7fac2ffff700 1 --2- 172.21.15.23:0/2444935013 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fac04056db0 0x7fac04059270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:42.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.759+0000 7fac2ffff700 1 --2- 172.21.15.23:0/2444935013 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fac04056db0 0x7fac04059270 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fac24012010 tx=0x7fac24008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:42.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:42.761+0000 7fac17fff700 1 -- 172.21.15.23:0/2444935013 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fac28044a70 con 0x7fac3015acd0 2020-11-20T11:48:43.027 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:43.024+0000 7fac3696e700 1 -- 172.21.15.23:0/2444935013 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fac3005c520 con 0x7fac3015acd0 2020-11-20T11:48:43.028 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:43.025+0000 7fac17fff700 1 -- 172.21.15.23:0/2444935013 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1472 (secure 0 0 0) 0x7fac28019070 con 0x7fac3015acd0 2020-11-20T11:48:43.028 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":152,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:48:43.030 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:43.004+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 296 slow ops, oldest is osd_op(client.4659.0:67780 4.2s0 4:542fa72b:::100000003c2.00000000:head [write 1961984~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:43.032 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:43.029+0000 7fac3696e700 1 -- 172.21.15.23:0/2444935013 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fac04056db0 msgr2=0x7fac04059270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:43.032 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:43.029+0000 7fac3696e700 1 --2- 172.21.15.23:0/2444935013 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fac04056db0 0x7fac04059270 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fac24012010 tx=0x7fac24008040).stop 2020-11-20T11:48:43.032 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:43.029+0000 7fac3696e700 1 -- 172.21.15.23:0/2444935013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac3015acd0 msgr2=0x7fac30065ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:43.033 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:43.029+0000 7fac3696e700 1 --2- 172.21.15.23:0/2444935013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac3015acd0 0x7fac30065ec0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fac30137c50 tx=0x7fac28007ce0).stop 2020-11-20T11:48:43.033 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:43.029+0000 7fac3696e700 1 -- 172.21.15.23:0/2444935013 shutdown_connections 2020-11-20T11:48:43.033 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:43.029+0000 7fac3696e700 1 --2- 172.21.15.23:0/2444935013 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac3015acd0 0x7fac30065ec0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:43.034 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:43.029+0000 7fac3696e700 1 --2- 172.21.15.23:0/2444935013 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fac04056db0 0x7fac04059270 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:43.034 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:43.029+0000 7fac3696e700 1 --2- 172.21.15.23:0/2444935013 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fac1c003770 0x7fac30067ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:43.034 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:43.029+0000 7fac3696e700 1 --2- 172.21.15.23:0/2444935013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac300ac040 0x7fac3015a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:43.034 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:43.029+0000 7fac3696e700 1 -- 172.21.15.23:0/2444935013 >> 172.21.15.23:0/2444935013 conn(0x7fac30060c80 msgr2=0x7fac30061850 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:48:43.035 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:43.032+0000 7fac3696e700 1 -- 172.21.15.23:0/2444935013 shutdown_connections 2020-11-20T11:48:43.036 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:43.033+0000 7fac3696e700 1 -- 172.21.15.23:0/2444935013 wait complete. 2020-11-20T11:48:44.123 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:44.092+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 294 slow ops, oldest is osd_op(client.4659.0:67973 4.2s0 4:5a9d432b:::1000000036b.0000000c:head [write 3174400~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:45.190 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:45.158+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 294 slow ops, oldest is osd_op(client.4659.0:68096 4.2s0 4:543d101f:::10000000388.00000009:head [write 2560000~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:46.058 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:48:46.195 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:46.161+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 293 slow ops, oldest is osd_op(client.4659.0:68285 4.2s0 4:5f2195fc:::10000000356.00000012:head [write 3727360~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:46.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.345+0000 7f3af1fec700 1 Processor -- start 2020-11-20T11:48:46.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.345+0000 7f3af1fec700 1 -- start start 2020-11-20T11:48:46.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.346+0000 7f3af1fec700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3aec0a9040 0x7f3aec0aaf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:46.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.346+0000 7f3af1fec700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec0ad2f0 0x7f3aec13e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:46.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.346+0000 7f3af1fec700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aec0ad860 con 0x7f3aec0ace70 2020-11-20T11:48:46.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.346+0000 7f3af1fec700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3aec0aa8a0 con 0x7f3aec0a9040 2020-11-20T11:48:46.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.346+0000 7f3af1fec700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec0aaa90 con 0x7f3aec0ad2f0 2020-11-20T11:48:46.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.346+0000 7f3af17eb700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec0ad2f0 0x7f3aec13e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:46.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.346+0000 7f3af0fea700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3aec0a9040 0x7f3aec0aaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:46.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.346+0000 7f3af0fea700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3aec0a9040 0x7f3aec0aaf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:39914/0 (socket says 172.21.15.23:39914) 2020-11-20T11:48:46.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.346+0000 7f3aebfff700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f3aec0ace70 0x7f3aec13de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:39242/0 (socket says 172.21.15.23:39242) 2020-11-20T11:48:46.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.346+0000 7f3af17eb700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec0ad2f0 0x7f3aec13e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:57262/0 (socket says 172.21.15.23:57262) 2020-11-20T11:48:46.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.346+0000 7f3af0fea700 1 -- 172.21.15.23:0/2190365964 learned_addr learned my addr 172.21.15.23:0/2190365964 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:48:46.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.346+0000 7f3af0fea700 1 -- 172.21.15.23:0/2190365964 >> v2:172.21.15.155:3300/0 conn(0x7f3aec0a9040 msgr2=0x7f3aec0aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:48:46.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.346+0000 7f3af0fea700 1 -- 172.21.15.23:0/2190365964 >> v2:172.21.15.155:3300/0 conn(0x7f3aec0a9040 msgr2=0x7f3aec0aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:48:46.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.346+0000 7f3af17eb700 1 -- 172.21.15.23:0/2190365964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec0ad2f0 msgr2=0x7f3aec13e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:48:46.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.346+0000 7f3af17eb700 1 -- 172.21.15.23:0/2190365964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec0ad2f0 msgr2=0x7f3aec13e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:48:46.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.346+0000 7f3af0fea700 1 --2- 172.21.15.23:0/2190365964 >> v2:172.21.15.155:3300/0 conn(0x7f3aec0a9040 0x7f3aec0aaf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:48:46.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.346+0000 7f3af17eb700 1 --2- 172.21.15.23:0/2190365964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec0ad2f0 0x7f3aec13e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:48:46.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.346+0000 7f3af0fea700 1 --2- 172.21.15.23:0/2190365964 >> v2:172.21.15.155:3300/0 conn(0x7f3aec0a9040 0x7f3aec0aaf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:48:46.354 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.346+0000 7f3af17eb700 1 --2- 172.21.15.23:0/2190365964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec0ad2f0 0x7f3aec13e510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:48:46.550 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.547+0000 7f3af17eb700 1 --2- 172.21.15.23:0/2190365964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec0ad2f0 0x7f3aec13e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:46.551 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.547+0000 7f3af0fea700 1 --2- 172.21.15.23:0/2190365964 >> v2:172.21.15.155:3300/0 conn(0x7f3aec0a9040 0x7f3aec0aaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:46.551 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.547+0000 7f3af17eb700 1 -- 172.21.15.23:0/2190365964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec0ad2f0 msgr2=0x7f3aec13e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:48:46.551 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.547+0000 7f3af17eb700 1 -- 172.21.15.23:0/2190365964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec0ad2f0 msgr2=0x7f3aec13e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:48:46.551 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.547+0000 7f3af17eb700 1 --2- 172.21.15.23:0/2190365964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec0ad2f0 0x7f3aec13e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:48:46.552 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.547+0000 7f3af0fea700 1 -- 172.21.15.23:0/2190365964 >> v2:172.21.15.155:3300/0 conn(0x7f3aec0a9040 msgr2=0x7f3aec0aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:48:46.552 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.547+0000 7f3af0fea700 1 -- 172.21.15.23:0/2190365964 >> v2:172.21.15.155:3300/0 conn(0x7f3aec0a9040 msgr2=0x7f3aec0aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:48:46.552 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.547+0000 7f3af17eb700 1 --2- 172.21.15.23:0/2190365964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec0ad2f0 0x7f3aec13e510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:48:46.553 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.547+0000 7f3af0fea700 1 --2- 172.21.15.23:0/2190365964 >> v2:172.21.15.155:3300/0 conn(0x7f3aec0a9040 0x7f3aec0aaf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:48:46.553 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.547+0000 7f3af0fea700 1 --2- 172.21.15.23:0/2190365964 >> v2:172.21.15.155:3300/0 conn(0x7f3aec0a9040 0x7f3aec0aaf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2020-11-20T11:48:46.951 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.948+0000 7f3af17eb700 1 --2- 172.21.15.23:0/2190365964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec0ad2f0 0x7f3aec13e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:46.952 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.948+0000 7f3af17eb700 1 -- 172.21.15.23:0/2190365964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec0ad2f0 msgr2=0x7f3aec13e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:48:46.952 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.948+0000 7f3af17eb700 1 -- 172.21.15.23:0/2190365964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec0ad2f0 msgr2=0x7f3aec13e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:48:46.952 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.948+0000 7f3af17eb700 1 --2- 172.21.15.23:0/2190365964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec0ad2f0 0x7f3aec13e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:48:46.952 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.948+0000 7f3af17eb700 1 --2- 172.21.15.23:0/2190365964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec0ad2f0 0x7f3aec13e510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:48:46.953 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.948+0000 7f3af0fea700 1 --2- 172.21.15.23:0/2190365964 >> v2:172.21.15.155:3300/0 conn(0x7f3aec0a9040 0x7f3aec0aaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:46.953 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.948+0000 7f3af0fea700 1 -- 172.21.15.23:0/2190365964 >> v2:172.21.15.155:3300/0 conn(0x7f3aec0a9040 msgr2=0x7f3aec0aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:48:46.953 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.948+0000 7f3af0fea700 1 -- 172.21.15.23:0/2190365964 >> v2:172.21.15.155:3300/0 conn(0x7f3aec0a9040 msgr2=0x7f3aec0aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:48:46.953 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.948+0000 7f3af0fea700 1 --2- 172.21.15.23:0/2190365964 >> v2:172.21.15.155:3300/0 conn(0x7f3aec0a9040 0x7f3aec0aaf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:48:46.953 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:46.948+0000 7f3af0fea700 1 --2- 172.21.15.23:0/2190365964 >> v2:172.21.15.155:3300/0 conn(0x7f3aec0a9040 0x7f3aec0aaf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2020-11-20T11:48:47.253 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:47.219+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 294 slow ops, oldest is osd_op(client.4659.0:69551 4.2s0 4:46c0f236:::1000000036d.00000017:head [write 1572864~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:47.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:47.749+0000 7f3af17eb700 1 --2- 172.21.15.23:0/2190365964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec0ad2f0 0x7f3aec13e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:47.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:47.749+0000 7f3af0fea700 1 --2- 172.21.15.23:0/2190365964 >> v2:172.21.15.155:3300/0 conn(0x7f3aec0a9040 0x7f3aec0aaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:47.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:47.749+0000 7f3af0fea700 1 -- 172.21.15.23:0/2190365964 >> v2:172.21.15.155:3300/0 conn(0x7f3aec0a9040 msgr2=0x7f3aec0aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-11-20T11:48:47.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:47.749+0000 7f3af0fea700 1 -- 172.21.15.23:0/2190365964 >> v2:172.21.15.155:3300/0 conn(0x7f3aec0a9040 msgr2=0x7f3aec0aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:48:47.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:47.749+0000 7f3af17eb700 1 -- 172.21.15.23:0/2190365964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec0ad2f0 msgr2=0x7f3aec13e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2020-11-20T11:48:47.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:47.749+0000 7f3af0fea700 1 --2- 172.21.15.23:0/2190365964 >> v2:172.21.15.155:3300/0 conn(0x7f3aec0a9040 0x7f3aec0aaf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:48:47.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:47.749+0000 7f3af17eb700 1 -- 172.21.15.23:0/2190365964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec0ad2f0 msgr2=0x7f3aec13e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-11-20T11:48:47.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:47.749+0000 7f3af17eb700 1 --2- 172.21.15.23:0/2190365964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec0ad2f0 0x7f3aec13e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:48:47.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:47.749+0000 7f3af0fea700 1 --2- 172.21.15.23:0/2190365964 >> v2:172.21.15.155:3300/0 conn(0x7f3aec0a9040 0x7f3aec0aaf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:48:47.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:47.749+0000 7f3af17eb700 1 --2- 172.21.15.23:0/2190365964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec0ad2f0 0x7f3aec13e510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2020-11-20T11:48:48.251 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:48.210+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 291 slow ops, oldest is osd_op(client.4659.0:69551 4.2s0 4:46c0f236:::1000000036d.00000017:head [write 1572864~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:48.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.747+0000 7f3aeb7fe700 1 -- 172.21.15.23:0/2190365964 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1423377092 0 0) 0x7f3aec0ad860 con 0x7f3aec0ace70 2020-11-20T11:48:48.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.747+0000 7f3aeb7fe700 1 -- 172.21.15.23:0/2190365964 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ad0003280 con 0x7f3aec0ace70 2020-11-20T11:48:48.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.747+0000 7f3aeb7fe700 1 -- 172.21.15.23:0/2190365964 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3559696998 0 0) 0x7f3ad0003280 con 0x7f3aec0ace70 2020-11-20T11:48:48.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.747+0000 7f3aeb7fe700 1 -- 172.21.15.23:0/2190365964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec0ad2f0 msgr2=0x7f3aec13e510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:48:48.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.747+0000 7f3aeb7fe700 1 --2- 172.21.15.23:0/2190365964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec0ad2f0 0x7f3aec13e510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:48.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.747+0000 7f3aeb7fe700 1 -- 172.21.15.23:0/2190365964 >> v2:172.21.15.155:3300/0 conn(0x7f3aec0a9040 msgr2=0x7f3aec0aaf30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:48:48.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.747+0000 7f3aeb7fe700 1 --2- 172.21.15.23:0/2190365964 >> v2:172.21.15.155:3300/0 conn(0x7f3aec0a9040 0x7f3aec0aaf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:48.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.747+0000 7f3aeb7fe700 1 -- 172.21.15.23:0/2190365964 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aec0a9420 con 0x7f3aec0ace70 2020-11-20T11:48:48.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.747+0000 7f3aeb7fe700 1 -- 172.21.15.23:0/2190365964 <== mon.1 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f3ad8001520 con 0x7f3aec0ace70 2020-11-20T11:48:48.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.747+0000 7f3aeb7fe700 1 -- 172.21.15.23:0/2190365964 >> v1:172.21.15.155:6789/0 conn(0x7f3aec0ace70 legacy=0x7f3aec13de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:48.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.747+0000 7f3aeb7fe700 1 --2- 172.21.15.23:0/2190365964 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ad0003770 0x7f3ad0003b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:48.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.747+0000 7f3aeb7fe700 1 --2- 172.21.15.23:0/2190365964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ad0004520 0x7f3ad00049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:48.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.747+0000 7f3aeb7fe700 1 --2- 172.21.15.23:0/2190365964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec0ad2f0 0x7f3ad0004e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:48.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.747+0000 7f3aeb7fe700 1 -- 172.21.15.23:0/2190365964 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec0aaa90 con 0x7f3ad0004520 2020-11-20T11:48:48.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.747+0000 7f3aeb7fe700 1 -- 172.21.15.23:0/2190365964 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad00042b0 con 0x7f3ad0003770 2020-11-20T11:48:48.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.747+0000 7f3aeb7fe700 1 -- 172.21.15.23:0/2190365964 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad000b770 con 0x7f3aec0ad2f0 2020-11-20T11:48:48.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.748+0000 7f3af0fea700 1 --2- 172.21.15.23:0/2190365964 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ad0003770 0x7f3ad0003b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:48.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.748+0000 7f3af17eb700 1 --2- 172.21.15.23:0/2190365964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec0ad2f0 0x7f3ad0004e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:48.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.748+0000 7f3af0fea700 1 -- 172.21.15.23:0/2190365964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec0ad2f0 msgr2=0x7f3ad0004e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:48:48.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.748+0000 7f3af0fea700 1 --2- 172.21.15.23:0/2190365964 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec0ad2f0 0x7f3ad0004e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:48.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.748+0000 7f3af0fea700 1 -- 172.21.15.23:0/2190365964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ad0004520 msgr2=0x7f3ad00049a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:48:48.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.748+0000 7f3af0fea700 1 --2- 172.21.15.23:0/2190365964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3ad0004520 0x7f3ad00049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:48:48.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.748+0000 7f3af0fea700 1 -- 172.21.15.23:0/2190365964 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad0003280 con 0x7f3ad0003770 2020-11-20T11:48:48.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.748+0000 7f3af0fea700 1 --2- 172.21.15.23:0/2190365964 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ad0003770 0x7f3ad0003b90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f3ae0011ad0 tx=0x7f3ae0008040).ready entity=mon.1 client_cookie=e422f62e56186f71 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:48.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.748+0000 7f3aeb7fe700 1 -- 172.21.15.23:0/2190365964 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae0013980 con 0x7f3ad0003770 2020-11-20T11:48:48.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.748+0000 7f3aeb7fe700 1 -- 172.21.15.23:0/2190365964 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae0013ae0 con 0x7f3ad0003770 2020-11-20T11:48:48.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.748+0000 7f3aeb7fe700 1 -- 172.21.15.23:0/2190365964 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae0013e60 con 0x7f3ad0003770 2020-11-20T11:48:48.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.749+0000 7f3af1fec700 1 -- 172.21.15.23:0/2190365964 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ad0003770 msgr2=0x7f3ad0003b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:48.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.749+0000 7f3af1fec700 1 --2- 172.21.15.23:0/2190365964 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ad0003770 0x7f3ad0003b90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f3ae0011ad0 tx=0x7f3ae0008040).stop 2020-11-20T11:48:48.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.749+0000 7f3af17eb700 1 -- 172.21.15.23:0/2190365964 reap_dead start 2020-11-20T11:48:48.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.749+0000 7f3af1fec700 1 -- 172.21.15.23:0/2190365964 shutdown_connections 2020-11-20T11:48:48.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.749+0000 7f3af1fec700 1 -- 172.21.15.23:0/2190365964 >> 172.21.15.23:0/2190365964 conn(0x7f3aec0683b0 msgr2=0x7f3aec0b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:48:48.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.749+0000 7f3af1fec700 1 -- 172.21.15.23:0/2190365964 shutdown_connections 2020-11-20T11:48:48.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.749+0000 7f3af1fec700 1 -- 172.21.15.23:0/2190365964 wait complete. 2020-11-20T11:48:48.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.751+0000 7f3af1fec700 1 Processor -- start 2020-11-20T11:48:48.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.751+0000 7f3af1fec700 1 -- start start 2020-11-20T11:48:48.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.752+0000 7f3af1fec700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ad0003770 0x7f3aec140bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:48.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.752+0000 7f3af1fec700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec09eaf0 0x7f3aec09d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:48.761 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.752+0000 7f3af1fec700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3aec09ef70 0x7f3aec09d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:48.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.752+0000 7f3af1fec700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec0a20b0 con 0x7f3aec09ef70 2020-11-20T11:48:48.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.752+0000 7f3af1fec700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec09ffa0 con 0x7f3ad0003770 2020-11-20T11:48:48.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.752+0000 7f3af1fec700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec0a0240 con 0x7f3aec09eaf0 2020-11-20T11:48:48.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.752+0000 7f3af17eb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3aec09ef70 0x7f3aec09d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:48.762 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.752+0000 7f3aebfff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec09eaf0 0x7f3aec09d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:48.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.752+0000 7f3af0fea700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ad0003770 0x7f3aec140bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:48.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.752+0000 7f3af17eb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3aec09ef70 0x7f3aec09d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47044/0 (socket says 172.21.15.23:47044) 2020-11-20T11:48:48.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.752+0000 7f3af17eb700 1 -- 172.21.15.23:0/3477151231 learned_addr learned my addr 172.21.15.23:0/3477151231 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:48:48.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.752+0000 7f3aebfff700 1 -- 172.21.15.23:0/3477151231 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec09eaf0 msgr2=0x7f3aec09d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2020-11-20T11:48:48.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.752+0000 7f3aebfff700 1 -- 172.21.15.23:0/3477151231 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec09eaf0 msgr2=0x7f3aec09d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2020-11-20T11:48:48.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.752+0000 7f3aebfff700 1 --2- 172.21.15.23:0/3477151231 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec09eaf0 0x7f3aec09d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2020-11-20T11:48:48.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.752+0000 7f3aebfff700 1 --2- 172.21.15.23:0/3477151231 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec09eaf0 0x7f3aec09d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2020-11-20T11:48:48.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.753+0000 7f3af17eb700 1 -- 172.21.15.23:0/3477151231 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec09eaf0 msgr2=0x7f3aec09d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:48:48.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.753+0000 7f3af17eb700 1 --2- 172.21.15.23:0/3477151231 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec09eaf0 0x7f3aec09d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:48.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.753+0000 7f3af17eb700 1 -- 172.21.15.23:0/3477151231 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ad0003770 msgr2=0x7f3aec140bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:48.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.753+0000 7f3af17eb700 1 --2- 172.21.15.23:0/3477151231 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ad0003770 0x7f3aec140bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:48.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.753+0000 7f3af17eb700 1 -- 172.21.15.23:0/3477151231 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aec0a06e0 con 0x7f3aec09ef70 2020-11-20T11:48:48.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.753+0000 7f3af17eb700 1 --2- 172.21.15.23:0/3477151231 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3aec09ef70 0x7f3aec09d990 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f3aec05c250 tx=0x7f3ae4008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:48.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.755+0000 7f3ae8ff9700 1 -- 172.21.15.23:0/3477151231 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae4010670 con 0x7f3aec09ef70 2020-11-20T11:48:48.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.755+0000 7f3af1fec700 1 -- 172.21.15.23:0/3477151231 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aec0a22e0 con 0x7f3aec09ef70 2020-11-20T11:48:48.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.755+0000 7f3af1fec700 1 -- 172.21.15.23:0/3477151231 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aec0a7af0 con 0x7f3aec09ef70 2020-11-20T11:48:48.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.756+0000 7f3ae8ff9700 1 -- 172.21.15.23:0/3477151231 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae400c070 con 0x7f3aec09ef70 2020-11-20T11:48:48.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.756+0000 7f3ae8ff9700 1 -- 172.21.15.23:0/3477151231 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae4015bb0 con 0x7f3aec09ef70 2020-11-20T11:48:48.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.757+0000 7f3af1fec700 1 -- 172.21.15.23:0/3477151231 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aec0a8b70 con 0x7f3aec09ef70 2020-11-20T11:48:48.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.757+0000 7f3ae8ff9700 1 -- 172.21.15.23:0/3477151231 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3ae4029070 con 0x7f3aec09ef70 2020-11-20T11:48:48.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.757+0000 7f3ae8ff9700 1 --2- 172.21.15.23:0/3477151231 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3ac4056d90 0x7f3ac4059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:48.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.758+0000 7f3ae8ff9700 1 -- 172.21.15.23:0/3477151231 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f3ae4049980 con 0x7f3aec09ef70 2020-11-20T11:48:48.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.758+0000 7f3af0fea700 1 --2- 172.21.15.23:0/3477151231 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3ac4056d90 0x7f3ac4059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:48.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.758+0000 7f3af0fea700 1 --2- 172.21.15.23:0/3477151231 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3ac4056d90 0x7f3ac4059250 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f3ae0006980 tx=0x7f3ae0008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:48.768 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:48.762+0000 7f3ae8ff9700 1 -- 172.21.15.23:0/3477151231 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f3ae4045d20 con 0x7f3aec09ef70 2020-11-20T11:48:49.046 INFO:tasks.workunit.client.0.smithi023.stdout:Syncing()...42 sec 2020-11-20T11:48:49.046 INFO:tasks.workunit.client.0.smithi023.stdout:FFSB benchmark finished at: Fri Nov 20 11:48:49 2020 2020-11-20T11:48:49.046 INFO:tasks.workunit.client.0.smithi023.stdout: 2020-11-20T11:48:49.046 INFO:tasks.workunit.client.0.smithi023.stdout:Results: 2020-11-20T11:48:49.047 INFO:tasks.workunit.client.0.smithi023.stdout:Benchmark took 474.33 sec 2020-11-20T11:48:49.047 INFO:tasks.workunit.client.0.smithi023.stdout: 2020-11-20T11:48:49.047 INFO:tasks.workunit.client.0.smithi023.stdout:Total Results 2020-11-20T11:48:49.047 INFO:tasks.workunit.client.0.smithi023.stdout:=============== 2020-11-20T11:48:49.047 INFO:tasks.workunit.client.0.smithi023.stdout: Op Name Transactions Trans/sec % Trans % Op Weight Throughput 2020-11-20T11:48:49.048 INFO:tasks.workunit.client.0.smithi023.stdout: ======= ============ ========= ======= =========== ========== 2020-11-20T11:48:49.048 INFO:tasks.workunit.client.0.smithi023.stdout: write : 249600 526.21 100.000% 100.000% 41.1MB/sec 2020-11-20T11:48:49.048 INFO:tasks.workunit.client.0.smithi023.stdout:- 2020-11-20T11:48:49.048 INFO:tasks.workunit.client.0.smithi023.stdout:526.21 Transactions per Second 2020-11-20T11:48:49.048 INFO:tasks.workunit.client.0.smithi023.stdout: 2020-11-20T11:48:49.049 INFO:tasks.workunit.client.0.smithi023.stdout:Throughput Results 2020-11-20T11:48:49.049 INFO:tasks.workunit.client.0.smithi023.stdout:=================== 2020-11-20T11:48:49.049 INFO:tasks.workunit.client.0.smithi023.stdout:Write Throughput: 41.1MB/sec 2020-11-20T11:48:49.049 INFO:tasks.workunit.client.0.smithi023.stdout: 2020-11-20T11:48:49.050 INFO:tasks.workunit.client.0.smithi023.stdout:System Call Latency statistics in millisecs 2020-11-20T11:48:49.050 INFO:tasks.workunit.client.0.smithi023.stdout:===== 2020-11-20T11:48:49.050 INFO:tasks.workunit.client.0.smithi023.stdout: Min Avg Max Total Calls 2020-11-20T11:48:49.050 INFO:tasks.workunit.client.0.smithi023.stdout: ======== ======== ======== ============ 2020-11-20T11:48:49.050 INFO:tasks.workunit.client.0.smithi023.stdout:[ open] 0.728000 24.367913 2348.156006 195 2020-11-20T11:48:49.051 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2020-11-20T11:48:49.051 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[1] 2000.000000 - 5000.000000 : 1 2020-11-20T11:48:49.051 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2020-11-20T11:48:49.051 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2020-11-20T11:48:49.051 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2020-11-20T11:48:49.052 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2020-11-20T11:48:49.052 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[6] 50.000000 - 100.000000 : 10 2020-11-20T11:48:49.052 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[7] 20.000000 - 50.000000 : 27 2020-11-20T11:48:49.052 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[8] 10.000000 - 20.000000 : 21 2020-11-20T11:48:49.052 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[9] 5.000000 - 10.000000 : 46 2020-11-20T11:48:49.052 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[10] 2.000000 - 5.000000 : 70 2020-11-20T11:48:49.053 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[11] 1.000000 - 2.000000 : 14 2020-11-20T11:48:49.053 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[12] 0.500000 - 1.000000 : 6 2020-11-20T11:48:49.053 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[13] 0.200000 - 0.500000 : 0 2020-11-20T11:48:49.053 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[14] 0.100000 - 0.200000 : 0 2020-11-20T11:48:49.054 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[15] 0.050000 - 0.100000 : 0 2020-11-20T11:48:49.054 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2020-11-20T11:48:49.054 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2020-11-20T11:48:49.054 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2020-11-20T11:48:49.054 INFO:tasks.workunit.client.0.smithi023.stdout: 2020-11-20T11:48:49.054 INFO:tasks.workunit.client.0.smithi023.stdout:[ write] 0.092000 49.927689 3862.764893 249600 2020-11-20T11:48:49.055 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2020-11-20T11:48:49.055 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[1] 2000.000000 - 5000.000000 : 34 2020-11-20T11:48:49.055 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[2] 1000.000000 - 2000.000000 : 391 2020-11-20T11:48:49.055 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[3] 500.000000 - 1000.000000 : 1828 2020-11-20T11:48:49.055 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[4] 200.000000 - 500.000000 : 10412 2020-11-20T11:48:49.056 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[5] 100.000000 - 200.000000 : 21202 2020-11-20T11:48:49.056 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[6] 50.000000 - 100.000000 : 31933 2020-11-20T11:48:49.056 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[7] 20.000000 - 50.000000 : 48312 2020-11-20T11:48:49.056 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[8] 10.000000 - 20.000000 : 33689 2020-11-20T11:48:49.056 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[9] 5.000000 - 10.000000 : 29265 2020-11-20T11:48:49.057 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[10] 2.000000 - 5.000000 : 25102 2020-11-20T11:48:49.057 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[11] 1.000000 - 2.000000 : 10147 2020-11-20T11:48:49.057 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[12] 0.500000 - 1.000000 : 6647 2020-11-20T11:48:49.057 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[13] 0.200000 - 0.500000 : 21086 2020-11-20T11:48:49.057 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[14] 0.100000 - 0.200000 : 9521 2020-11-20T11:48:49.058 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[15] 0.050000 - 0.100000 : 31 2020-11-20T11:48:49.058 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2020-11-20T11:48:49.058 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2020-11-20T11:48:49.058 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2020-11-20T11:48:49.058 INFO:tasks.workunit.client.0.smithi023.stdout: 2020-11-20T11:48:49.059 INFO:tasks.workunit.client.0.smithi023.stdout:[ lseek] 0.000000 0.001363 2.439000 249600 2020-11-20T11:48:49.059 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2020-11-20T11:48:49.059 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2020-11-20T11:48:49.059 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2020-11-20T11:48:49.059 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2020-11-20T11:48:49.060 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2020-11-20T11:48:49.060 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2020-11-20T11:48:49.060 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2020-11-20T11:48:49.060 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[7] 20.000000 - 50.000000 : 0 2020-11-20T11:48:49.060 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[8] 10.000000 - 20.000000 : 0 2020-11-20T11:48:49.061 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[9] 5.000000 - 10.000000 : 0 2020-11-20T11:48:49.061 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[10] 2.000000 - 5.000000 : 1 2020-11-20T11:48:49.061 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[11] 1.000000 - 2.000000 : 0 2020-11-20T11:48:49.061 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[12] 0.500000 - 1.000000 : 0 2020-11-20T11:48:49.061 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[13] 0.200000 - 0.500000 : 1 2020-11-20T11:48:49.062 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[14] 0.100000 - 0.200000 : 3 2020-11-20T11:48:49.062 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[15] 0.050000 - 0.100000 : 1 2020-11-20T11:48:49.062 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[16] 0.020000 - 0.050000 : 7 2020-11-20T11:48:49.062 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[17] 0.010000 - 0.020000 : 103 2020-11-20T11:48:49.062 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[18] 0.000000 - 0.010000 : 249484 2020-11-20T11:48:49.063 INFO:tasks.workunit.client.0.smithi023.stdout: 2020-11-20T11:48:49.063 INFO:tasks.workunit.client.0.smithi023.stdout:[ close] 0.031000 9.986318 379.705994 195 2020-11-20T11:48:49.063 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2020-11-20T11:48:49.063 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2020-11-20T11:48:49.063 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2020-11-20T11:48:49.064 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2020-11-20T11:48:49.064 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[4] 200.000000 - 500.000000 : 3 2020-11-20T11:48:49.064 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[5] 100.000000 - 200.000000 : 3 2020-11-20T11:48:49.064 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[6] 50.000000 - 100.000000 : 3 2020-11-20T11:48:49.064 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[7] 20.000000 - 50.000000 : 9 2020-11-20T11:48:49.064 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[8] 10.000000 - 20.000000 : 2 2020-11-20T11:48:49.065 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[9] 5.000000 - 10.000000 : 7 2020-11-20T11:48:49.065 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[10] 2.000000 - 5.000000 : 13 2020-11-20T11:48:49.065 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[11] 1.000000 - 2.000000 : 5 2020-11-20T11:48:49.065 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[12] 0.500000 - 1.000000 : 10 2020-11-20T11:48:49.065 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[13] 0.200000 - 0.500000 : 26 2020-11-20T11:48:49.066 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[14] 0.100000 - 0.200000 : 25 2020-11-20T11:48:49.066 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[15] 0.050000 - 0.100000 : 50 2020-11-20T11:48:49.066 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[16] 0.020000 - 0.050000 : 39 2020-11-20T11:48:49.066 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2020-11-20T11:48:49.066 INFO:tasks.workunit.client.0.smithi023.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2020-11-20T11:48:49.067 INFO:tasks.workunit.client.0.smithi023.stdout: 2020-11-20T11:48:49.067 INFO:tasks.workunit.client.0.smithi023.stdout: 2020-11-20T11:48:49.067 INFO:tasks.workunit.client.0.smithi023.stdout: 2020-11-20T11:48:49.067 INFO:tasks.workunit.client.0.smithi023.stdout: 2020-11-20T11:48:49.067 INFO:tasks.workunit.client.0.smithi023.stdout:0.1% User Time 2020-11-20T11:48:49.068 INFO:tasks.workunit.client.0.smithi023.stdout:11.4% System Time 2020-11-20T11:48:49.068 INFO:tasks.workunit.client.0.smithi023.stdout:11.4% CPU Utilization 2020-11-20T11:48:49.164 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:49.161+0000 7f3af1fec700 1 -- 172.21.15.23:0/3477151231 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3aec0a0400 con 0x7f3aec09ef70 2020-11-20T11:48:49.165 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:49.161+0000 7f3ae8ff9700 1 -- 172.21.15.23:0/3477151231 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1472 (secure 0 0 0) 0x7f3ae4049340 con 0x7f3aec09ef70 2020-11-20T11:48:49.167 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":156,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:48:49.170 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:49.167+0000 7f3af1fec700 1 -- 172.21.15.23:0/3477151231 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3ac4056d90 msgr2=0x7f3ac4059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:49.170 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:49.167+0000 7f3af1fec700 1 --2- 172.21.15.23:0/3477151231 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3ac4056d90 0x7f3ac4059250 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f3ae0006980 tx=0x7f3ae0008040).stop 2020-11-20T11:48:49.171 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:49.167+0000 7f3af1fec700 1 -- 172.21.15.23:0/3477151231 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3aec09ef70 msgr2=0x7f3aec09d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:49.171 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:49.167+0000 7f3af1fec700 1 --2- 172.21.15.23:0/3477151231 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3aec09ef70 0x7f3aec09d990 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f3aec05c250 tx=0x7f3ae4008040).stop 2020-11-20T11:48:49.171 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:49.167+0000 7f3af1fec700 1 -- 172.21.15.23:0/3477151231 shutdown_connections 2020-11-20T11:48:49.171 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:49.167+0000 7f3af1fec700 1 --2- 172.21.15.23:0/3477151231 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3aec09ef70 0x7f3aec09d990 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:49.171 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:49.167+0000 7f3af1fec700 1 --2- 172.21.15.23:0/3477151231 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3ad0003770 0x7f3aec140bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:49.172 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:49.167+0000 7f3af1fec700 1 --2- 172.21.15.23:0/3477151231 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3ac4056d90 0x7f3ac4059250 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:49.172 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:49.167+0000 7f3af1fec700 1 --2- 172.21.15.23:0/3477151231 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3aec09eaf0 0x7f3aec09d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:49.172 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:49.167+0000 7f3af1fec700 1 -- 172.21.15.23:0/3477151231 >> 172.21.15.23:0/3477151231 conn(0x7f3aec0683b0 msgr2=0x7f3aec0aba40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:48:49.172 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:49.167+0000 7f3af1fec700 1 -- 172.21.15.23:0/3477151231 shutdown_connections 2020-11-20T11:48:49.173 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:49.167+0000 7f3af1fec700 1 -- 172.21.15.23:0/3477151231 wait complete. 2020-11-20T11:48:49.291 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:49.243+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 303 slow ops, oldest is osd_op(client.4659.0:69551 4.2s0 4:46c0f236:::1000000036d.00000017:head [write 1572864~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:50.199 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-11-20T11:48:50.199 INFO:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-11-20T11:48:50.250 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2020-11-20T11:48:50.251 INFO:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2020-11-20T11:48:50.298 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:50.262+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 307 slow ops, oldest is osd_op(client.4659.0:69551 4.2s0 4:46c0f236:::1000000036d.00000017:head [write 1572864~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:51.315 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:51.260+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 307 slow ops, oldest is osd_op(client.4659.0:69551 4.2s0 4:46c0f236:::1000000036d.00000017:head [write 1572864~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:52.185 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:48:52.321 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:52.287+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 306 slow ops, oldest is osd_op(client.4659.0:69551 4.2s0 4:46c0f236:::1000000036d.00000017:head [write 1572864~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:52.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.334+0000 7f18db68f700 1 Processor -- start 2020-11-20T11:48:52.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.335+0000 7f18db68f700 1 -- start start 2020-11-20T11:48:52.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.335+0000 7f18db68f700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f18d40a9d40 0x7f18d40aa1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:52.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.335+0000 7f18db68f700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f18d40abb80 0x7f18d40abff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:52.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.335+0000 7f18db68f700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d4067bc0 con 0x7f18d4060a80 2020-11-20T11:48:52.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.335+0000 7f18db68f700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18d4062060 con 0x7f18d40a9d40 2020-11-20T11:48:52.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.335+0000 7f18db68f700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18d4067e60 con 0x7f18d40abb80 2020-11-20T11:48:52.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.335+0000 7f18d8c2a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f18d40a9d40 0x7f18d40aa1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:52.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.336+0000 7f18d942b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f18d4060a80 0x7f18d4061950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:33068/0 (socket says 172.21.15.23:33068) 2020-11-20T11:48:52.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.336+0000 7f18d8c2a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f18d40a9d40 0x7f18d40aa1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47054/0 (socket says 172.21.15.23:47054) 2020-11-20T11:48:52.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.336+0000 7f18d942b700 1 -- 172.21.15.23:0/4046253286 learned_addr learned my addr 172.21.15.23:0/4046253286 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:48:52.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.336+0000 7f18d9c2c700 1 --2- 172.21.15.23:0/4046253286 >> v2:172.21.15.155:3300/0 conn(0x7f18d40abb80 0x7f18d40abff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:52.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.336+0000 7f18d8c2a700 1 -- 172.21.15.23:0/4046253286 >> v2:172.21.15.155:3300/0 conn(0x7f18d40abb80 msgr2=0x7f18d40abff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:48:52.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.336+0000 7f18d8c2a700 1 --2- 172.21.15.23:0/4046253286 >> v2:172.21.15.155:3300/0 conn(0x7f18d40abb80 0x7f18d40abff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:52.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.336+0000 7f18d8c2a700 1 -- 172.21.15.23:0/4046253286 >> v1:172.21.15.23:6789/0 conn(0x7f18d4060a80 legacy=0x7f18d4061950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:52.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.336+0000 7f18c3fff700 1 -- 172.21.15.23:0/4046253286 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3868416988 0 0) 0x7f18d4067bc0 con 0x7f18d4060a80 2020-11-20T11:48:52.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.336+0000 7f18d8c2a700 1 -- 172.21.15.23:0/4046253286 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d4068100 con 0x7f18d40a9d40 2020-11-20T11:48:52.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.336+0000 7f18d8c2a700 1 --2- 172.21.15.23:0/4046253286 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18d40a9d40 0x7f18d40aa1c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f18d40ab950 tx=0x7f18c4006e30).ready entity=mon.0 client_cookie=c39317344d67aa21 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:52.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.336+0000 7f18c3fff700 1 -- 172.21.15.23:0/4046253286 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c40097b0 con 0x7f18d40a9d40 2020-11-20T11:48:52.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.336+0000 7f18c3fff700 1 -- 172.21.15.23:0/4046253286 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18c4000e90 con 0x7f18d40a9d40 2020-11-20T11:48:52.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.336+0000 7f18c3fff700 1 -- 172.21.15.23:0/4046253286 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c4009b20 con 0x7f18d40a9d40 2020-11-20T11:48:52.345 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.337+0000 7f18db68f700 1 -- 172.21.15.23:0/4046253286 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18d40a9d40 msgr2=0x7f18d40aa1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:52.346 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.337+0000 7f18db68f700 1 --2- 172.21.15.23:0/4046253286 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18d40a9d40 0x7f18d40aa1c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f18d40ab950 tx=0x7f18c4006e30).stop 2020-11-20T11:48:52.346 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.337+0000 7f18db68f700 1 -- 172.21.15.23:0/4046253286 shutdown_connections 2020-11-20T11:48:52.346 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.337+0000 7f18db68f700 1 --2- 172.21.15.23:0/4046253286 >> v2:172.21.15.155:3300/0 conn(0x7f18d40abb80 0x7f18d40abff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:52.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.337+0000 7f18db68f700 1 --2- 172.21.15.23:0/4046253286 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18d40a9d40 0x7f18d40aa1c0 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0x7f18d40ab950 tx=0x7f18c4006e30).stop 2020-11-20T11:48:52.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.337+0000 7f18db68f700 1 -- 172.21.15.23:0/4046253286 >> 172.21.15.23:0/4046253286 conn(0x7f18d4066730 msgr2=0x7f18d40601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:48:52.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.338+0000 7f18db68f700 1 -- 172.21.15.23:0/4046253286 shutdown_connections 2020-11-20T11:48:52.347 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.338+0000 7f18db68f700 1 -- 172.21.15.23:0/4046253286 wait complete. 2020-11-20T11:48:52.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.339+0000 7f18db68f700 1 Processor -- start 2020-11-20T11:48:52.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.339+0000 7f18db68f700 1 -- start start 2020-11-20T11:48:52.348 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.340+0000 7f18db68f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18d4060a80 0x7f18d4136900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:52.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.340+0000 7f18db68f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18d40abb80 0x7f18d414d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:52.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.340+0000 7f18db68f700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18d40a8ba0 0x7f18d40a9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:52.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.340+0000 7f18db68f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d414dcd0 con 0x7f18d4060a80 2020-11-20T11:48:52.349 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.340+0000 7f18db68f700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d40aeef0 con 0x7f18d40abb80 2020-11-20T11:48:52.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.340+0000 7f18db68f700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18d4137660 con 0x7f18d40a8ba0 2020-11-20T11:48:52.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.340+0000 7f18d942b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18d4060a80 0x7f18d4136900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:52.350 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.340+0000 7f18d942b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18d4060a80 0x7f18d4136900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47056/0 (socket says 172.21.15.23:47056) 2020-11-20T11:48:52.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.340+0000 7f18d942b700 1 -- 172.21.15.23:0/3829989981 learned_addr learned my addr 172.21.15.23:0/3829989981 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:48:52.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.340+0000 7f18d8c2a700 1 --2- 172.21.15.23:0/3829989981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18d40abb80 0x7f18d414d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:52.351 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.340+0000 7f18d9c2c700 1 --2- 172.21.15.23:0/3829989981 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18d40a8ba0 0x7f18d40a9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:52.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.341+0000 7f18d8c2a700 1 -- 172.21.15.23:0/3829989981 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18d40a8ba0 msgr2=0x7f18d40a9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:52.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.341+0000 7f18d8c2a700 1 --2- 172.21.15.23:0/3829989981 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18d40a8ba0 0x7f18d40a9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:52.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.341+0000 7f18d8c2a700 1 -- 172.21.15.23:0/3829989981 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18d4060a80 msgr2=0x7f18d4136900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:52.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.341+0000 7f18d8c2a700 1 --2- 172.21.15.23:0/3829989981 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18d4060a80 0x7f18d4136900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:52.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.341+0000 7f18d8c2a700 1 -- 172.21.15.23:0/3829989981 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d414deb0 con 0x7f18d40abb80 2020-11-20T11:48:52.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.341+0000 7f18d942b700 1 --2- 172.21.15.23:0/3829989981 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18d4060a80 0x7f18d4136900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:48:52.354 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.341+0000 7f18d9c2c700 1 --2- 172.21.15.23:0/3829989981 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18d40a8ba0 0x7f18d40a9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:48:52.354 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.341+0000 7f18d8c2a700 1 --2- 172.21.15.23:0/3829989981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18d40abb80 0x7f18d414d7e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f18c4012000 tx=0x7f18c4015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:52.354 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.341+0000 7f18c17fa700 1 -- 172.21.15.23:0/3829989981 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c4011b40 con 0x7f18d40abb80 2020-11-20T11:48:52.355 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.341+0000 7f18c17fa700 1 -- 172.21.15.23:0/3829989981 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18c4011ca0 con 0x7f18d40abb80 2020-11-20T11:48:52.355 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.341+0000 7f18db68f700 1 -- 172.21.15.23:0/3829989981 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18d414e150 con 0x7f18d40abb80 2020-11-20T11:48:52.356 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.341+0000 7f18c17fa700 1 -- 172.21.15.23:0/3829989981 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c401d5f0 con 0x7f18d40abb80 2020-11-20T11:48:52.356 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.341+0000 7f18db68f700 1 -- 172.21.15.23:0/3829989981 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18d4138990 con 0x7f18d40abb80 2020-11-20T11:48:52.357 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.343+0000 7f18c17fa700 1 -- 172.21.15.23:0/3829989981 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f18c401a040 con 0x7f18d40abb80 2020-11-20T11:48:52.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.347+0000 7f18db68f700 1 -- 172.21.15.23:0/3829989981 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18d414e5b0 con 0x7f18d40abb80 2020-11-20T11:48:52.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.348+0000 7f18c17fa700 1 --2- 172.21.15.23:0/3829989981 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f18ac056d60 0x7f18ac059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:52.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.348+0000 7f18c17fa700 1 -- 172.21.15.23:0/3829989981 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f18c4071240 con 0x7f18d40abb80 2020-11-20T11:48:52.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.348+0000 7f18d942b700 1 --2- 172.21.15.23:0/3829989981 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f18ac056d60 0x7f18ac059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:52.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.348+0000 7f18d942b700 1 --2- 172.21.15.23:0/3829989981 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f18ac056d60 0x7f18ac059220 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f18d0001710 tx=0x7f18d0007b00).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:52.360 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.355+0000 7f18c17fa700 1 -- 172.21.15.23:0/3829989981 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f18c40464d0 con 0x7f18d40abb80 2020-11-20T11:48:52.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.645+0000 7f18db68f700 1 -- 172.21.15.23:0/3829989981 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f18d4044710 con 0x7f18d40abb80 2020-11-20T11:48:52.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.646+0000 7f18c17fa700 1 -- 172.21.15.23:0/3829989981 <== mon.1 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1478 (secure 0 0 0) 0x7f18c4049af0 con 0x7f18d40abb80 2020-11-20T11:48:52.649 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":158,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:48:52.652 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.648+0000 7f18db68f700 1 -- 172.21.15.23:0/3829989981 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f18ac056d60 msgr2=0x7f18ac059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:52.652 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.648+0000 7f18db68f700 1 --2- 172.21.15.23:0/3829989981 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f18ac056d60 0x7f18ac059220 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f18d0001710 tx=0x7f18d0007b00).stop 2020-11-20T11:48:52.653 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.648+0000 7f18db68f700 1 -- 172.21.15.23:0/3829989981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18d40abb80 msgr2=0x7f18d414d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:52.653 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.648+0000 7f18db68f700 1 --2- 172.21.15.23:0/3829989981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18d40abb80 0x7f18d414d7e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f18c4012000 tx=0x7f18c4015000).stop 2020-11-20T11:48:52.653 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.648+0000 7f18db68f700 1 -- 172.21.15.23:0/3829989981 shutdown_connections 2020-11-20T11:48:52.653 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.648+0000 7f18db68f700 1 --2- 172.21.15.23:0/3829989981 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f18ac056d60 0x7f18ac059220 secure :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0x7f18d0001710 tx=0x7f18d0007b00).stop 2020-11-20T11:48:52.654 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.648+0000 7f18db68f700 1 --2- 172.21.15.23:0/3829989981 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f18d40a8ba0 0x7f18d40a9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:52.654 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.648+0000 7f18db68f700 1 --2- 172.21.15.23:0/3829989981 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18d4060a80 0x7f18d4136900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:52.654 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.648+0000 7f18db68f700 1 --2- 172.21.15.23:0/3829989981 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f18d40abb80 0x7f18d414d7e0 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0x7f18c4012000 tx=0x7f18c4015000).stop 2020-11-20T11:48:52.654 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.648+0000 7f18db68f700 1 -- 172.21.15.23:0/3829989981 >> 172.21.15.23:0/3829989981 conn(0x7f18d4066730 msgr2=0x7f18d4061930 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:48:52.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.648+0000 7f18db68f700 1 -- 172.21.15.23:0/3829989981 shutdown_connections 2020-11-20T11:48:52.656 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.653+0000 7f18db68f700 1 -- 172.21.15.23:0/3829989981 wait complete. 2020-11-20T11:48:52.671 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:48:52.672 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2020-11-20T11:48:52.823 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.820+0000 7f996dbca700 1 Processor -- start 2020-11-20T11:48:52.824 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.820+0000 7f996dbca700 1 -- start start 2020-11-20T11:48:52.825 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.821+0000 7f996dbca700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f99680aaab0 0x7f996813ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:52.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.821+0000 7f996dbca700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f996813aeb0 0x7f99680a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:52.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.821+0000 7f996dbca700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99681388b0 con 0x7f99680aae90 2020-11-20T11:48:52.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.821+0000 7f996dbca700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f996813e600 con 0x7f996813aeb0 2020-11-20T11:48:52.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.821+0000 7f996dbca700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f996813e8a0 con 0x7f99680aaab0 2020-11-20T11:48:52.826 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.821+0000 7f9967fff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f996813aeb0 0x7f99680a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:52.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.821+0000 7f9966ffd700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f99680aae90 0x7f99681381a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:33082/0 (socket says 172.21.15.23:33082) 2020-11-20T11:48:52.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.821+0000 7f9966ffd700 1 -- 172.21.15.23:0/3279462260 learned_addr learned my addr 172.21.15.23:0/3279462260 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:48:52.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.821+0000 7f99677fe700 1 --2- 172.21.15.23:0/3279462260 >> v2:172.21.15.155:3300/0 conn(0x7f99680aaab0 0x7f996813ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:52.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.822+0000 7f9967fff700 1 -- 172.21.15.23:0/3279462260 >> v2:172.21.15.155:3300/0 conn(0x7f99680aaab0 msgr2=0x7f996813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:48:52.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.822+0000 7f9967fff700 1 --2- 172.21.15.23:0/3279462260 >> v2:172.21.15.155:3300/0 conn(0x7f99680aaab0 0x7f996813ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:52.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.822+0000 7f9967fff700 1 -- 172.21.15.23:0/3279462260 >> v1:172.21.15.23:6789/0 conn(0x7f99680aae90 legacy=0x7f99681381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:52.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.822+0000 7f9967fff700 1 -- 172.21.15.23:0/3279462260 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99680ab6d0 con 0x7f996813aeb0 2020-11-20T11:48:52.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.822+0000 7f99667fc700 1 -- 172.21.15.23:0/3279462260 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 122660529 0 0) 0x7f99681388b0 con 0x7f99680aae90 2020-11-20T11:48:52.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.822+0000 7f9967fff700 1 --2- 172.21.15.23:0/3279462260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f996813aeb0 0x7f99680a8c60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f99680b23d0 tx=0x7f9960009220).ready entity=mon.0 client_cookie=d4a927de884fcc71 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:52.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.822+0000 7f99667fc700 1 -- 172.21.15.23:0/3279462260 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996000ba00 con 0x7f996813aeb0 2020-11-20T11:48:52.832 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.822+0000 7f99667fc700 1 -- 172.21.15.23:0/3279462260 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f996000bb60 con 0x7f996813aeb0 2020-11-20T11:48:52.832 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.822+0000 7f99667fc700 1 -- 172.21.15.23:0/3279462260 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9960011400 con 0x7f996813aeb0 2020-11-20T11:48:52.833 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.822+0000 7f996dbca700 1 -- 172.21.15.23:0/3279462260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f996813aeb0 msgr2=0x7f99680a8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:52.833 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.822+0000 7f996dbca700 1 --2- 172.21.15.23:0/3279462260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f996813aeb0 0x7f99680a8c60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f99680b23d0 tx=0x7f9960009220).stop 2020-11-20T11:48:52.833 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.822+0000 7f996dbca700 1 -- 172.21.15.23:0/3279462260 shutdown_connections 2020-11-20T11:48:52.833 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.822+0000 7f996dbca700 1 --2- 172.21.15.23:0/3279462260 >> v2:172.21.15.155:3300/0 conn(0x7f99680aaab0 0x7f996813ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:52.834 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.822+0000 7f996dbca700 1 --2- 172.21.15.23:0/3279462260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f996813aeb0 0x7f99680a8c60 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:52.834 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.822+0000 7f996dbca700 1 -- 172.21.15.23:0/3279462260 >> 172.21.15.23:0/3279462260 conn(0x7f99680678e0 msgr2=0x7f9968067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:48:52.834 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.822+0000 7f996dbca700 1 -- 172.21.15.23:0/3279462260 shutdown_connections 2020-11-20T11:48:52.834 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.822+0000 7f996dbca700 1 -- 172.21.15.23:0/3279462260 wait complete. 2020-11-20T11:48:52.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.824+0000 7f996dbca700 1 Processor -- start 2020-11-20T11:48:52.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.824+0000 7f996dbca700 1 -- start start 2020-11-20T11:48:52.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.824+0000 7f996dbca700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f99680aaab0 0x7f9968135610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:52.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.824+0000 7f996dbca700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99680aae90 0x7f99681349b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:52.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.824+0000 7f996dbca700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9968149fb0 0x7f996813d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:52.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.824+0000 7f996dbca700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9968134040 con 0x7f9968149fb0 2020-11-20T11:48:52.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.824+0000 7f996dbca700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9968134270 con 0x7f99680aae90 2020-11-20T11:48:52.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.824+0000 7f996dbca700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9968134510 con 0x7f99680aaab0 2020-11-20T11:48:52.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.824+0000 7f9967fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9968149fb0 0x7f996813d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:52.837 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.824+0000 7f9967fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9968149fb0 0x7f996813d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47072/0 (socket says 172.21.15.23:47072) 2020-11-20T11:48:52.837 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.824+0000 7f9967fff700 1 -- 172.21.15.23:0/2324739492 learned_addr learned my addr 172.21.15.23:0/2324739492 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:48:52.837 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.825+0000 7f9966ffd700 1 --2- 172.21.15.23:0/2324739492 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99680aae90 0x7f99681349b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:52.837 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.825+0000 7f99677fe700 1 --2- 172.21.15.23:0/2324739492 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f99680aaab0 0x7f9968135610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:52.837 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.825+0000 7f9967fff700 1 -- 172.21.15.23:0/2324739492 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f99680aaab0 msgr2=0x7f9968135610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:52.838 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.825+0000 7f9967fff700 1 --2- 172.21.15.23:0/2324739492 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f99680aaab0 0x7f9968135610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:52.838 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.825+0000 7f9967fff700 1 -- 172.21.15.23:0/2324739492 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99680aae90 msgr2=0x7f99681349b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:52.838 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.825+0000 7f9967fff700 1 --2- 172.21.15.23:0/2324739492 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99680aae90 0x7f99681349b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:52.838 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.825+0000 7f9967fff700 1 -- 172.21.15.23:0/2324739492 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f996814a430 con 0x7f9968149fb0 2020-11-20T11:48:52.839 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.825+0000 7f9966ffd700 1 --2- 172.21.15.23:0/2324739492 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99680aae90 0x7f99681349b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:48:52.839 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.825+0000 7f9967fff700 1 --2- 172.21.15.23:0/2324739492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9968149fb0 0x7f996813d7a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f99600090b0 tx=0x7f9960016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:52.839 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.825+0000 7f99477fe700 1 -- 172.21.15.23:0/2324739492 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996001b6b0 con 0x7f9968149fb0 2020-11-20T11:48:52.839 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.825+0000 7f99477fe700 1 -- 172.21.15.23:0/2324739492 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f996001b810 con 0x7f9968149fb0 2020-11-20T11:48:52.839 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.825+0000 7f996dbca700 1 -- 172.21.15.23:0/2324739492 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f996813c2b0 con 0x7f9968149fb0 2020-11-20T11:48:52.840 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.825+0000 7f99477fe700 1 -- 172.21.15.23:0/2324739492 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996001bbb0 con 0x7f9968149fb0 2020-11-20T11:48:52.840 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.825+0000 7f996dbca700 1 -- 172.21.15.23:0/2324739492 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f996813dc90 con 0x7f9968149fb0 2020-11-20T11:48:52.840 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.826+0000 7f996dbca700 1 --2- 172.21.15.23:0/2324739492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f996813aeb0 0x7f996814a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:52.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.826+0000 7f996dbca700 1 -- 172.21.15.23:0/2324739492 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99680b0d10 con 0x7f996813aeb0 2020-11-20T11:48:52.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.826+0000 7f99477fe700 1 -- 172.21.15.23:0/2324739492 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f9960017070 con 0x7f9968149fb0 2020-11-20T11:48:52.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.826+0000 7f99477fe700 1 --2- 172.21.15.23:0/2324739492 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9940056d60 0x7f9940059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:52.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.827+0000 7f99477fe700 1 -- 172.21.15.23:0/2324739492 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f99600723b0 con 0x7f9968149fb0 2020-11-20T11:48:52.841 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.827+0000 7f9966ffd700 1 --2- 172.21.15.23:0/2324739492 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9940056d60 0x7f9940059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:52.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.827+0000 7f9966ffd700 1 --2- 172.21.15.23:0/2324739492 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9940056d60 0x7f9940059220 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f995c0015d0 tx=0x7f995c009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:52.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.828+0000 7f99677fe700 1 --2- 172.21.15.23:0/2324739492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f996813aeb0 0x7f996814a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:52.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.829+0000 7f99677fe700 1 -- 172.21.15.23:0/2324739492 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f996805c7b0 con 0x7f996813aeb0 2020-11-20T11:48:52.842 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.829+0000 7f99677fe700 1 --2- 172.21.15.23:0/2324739492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f996813aeb0 0x7f996814a780 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f9958006fd0 tx=0x7f9958008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:52.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.829+0000 7f99477fe700 1 -- 172.21.15.23:0/2324739492 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9958013610 con 0x7f996813aeb0 2020-11-20T11:48:52.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.829+0000 7f99477fe700 1 -- 172.21.15.23:0/2324739492 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f9958000e90 con 0x7f996813aeb0 2020-11-20T11:48:52.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.829+0000 7f99477fe700 1 -- 172.21.15.23:0/2324739492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f996813aeb0 msgr2=0x7f996814a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:52.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.829+0000 7f99477fe700 1 --2- 172.21.15.23:0/2324739492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f996813aeb0 0x7f996814a780 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f9958006fd0 tx=0x7f9958008040).stop 2020-11-20T11:48:52.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.829+0000 7f99477fe700 1 -- 172.21.15.23:0/2324739492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f996813aeb0 msgr2=0x7f996814a780 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:48:52.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.829+0000 7f99477fe700 1 --2- 172.21.15.23:0/2324739492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f996813aeb0 0x7f996814a780 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:52.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.844+0000 7f996dbca700 1 --2- 172.21.15.23:0/2324739492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99680601b0 0x7f996814d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:52.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.844+0000 7f996dbca700 1 -- 172.21.15.23:0/2324739492 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9968044710 con 0x7f99680601b0 2020-11-20T11:48:52.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.845+0000 7f99677fe700 1 --2- 172.21.15.23:0/2324739492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99680601b0 0x7f996814d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:52.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.845+0000 7f99677fe700 1 -- 172.21.15.23:0/2324739492 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f996805c280 con 0x7f99680601b0 2020-11-20T11:48:52.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.845+0000 7f99677fe700 1 --2- 172.21.15.23:0/2324739492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99680601b0 0x7f996814d580 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f9958017000 tx=0x7f995801a000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:52.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.845+0000 7f99477fe700 1 -- 172.21.15.23:0/2324739492 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9958020400 con 0x7f99680601b0 2020-11-20T11:48:52.850 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.846+0000 7f99477fe700 1 -- 172.21.15.23:0/2324739492 <== mon.0 v2:172.21.15.23:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4786 (secure 0 0 0) 0x7f9958013400 con 0x7f99680601b0 2020-11-20T11:48:52.850 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.846+0000 7f99477fe700 1 -- 172.21.15.23:0/2324739492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99680601b0 msgr2=0x7f996814d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:52.851 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.846+0000 7f99477fe700 1 --2- 172.21.15.23:0/2324739492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99680601b0 0x7f996814d580 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f9958017000 tx=0x7f995801a000).stop 2020-11-20T11:48:52.851 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.846+0000 7f99477fe700 1 -- 172.21.15.23:0/2324739492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99680601b0 msgr2=0x7f996814d580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:48:52.851 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.846+0000 7f99477fe700 1 --2- 172.21.15.23:0/2324739492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f99680601b0 0x7f996814d580 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:52.851 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.846+0000 7f996dbca700 1 -- 172.21.15.23:0/2324739492 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9940056d60 msgr2=0x7f9940059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:52.851 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.846+0000 7f996dbca700 1 --2- 172.21.15.23:0/2324739492 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f9940056d60 0x7f9940059220 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f995c0015d0 tx=0x7f995c009040).stop 2020-11-20T11:48:52.852 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.846+0000 7f996dbca700 1 -- 172.21.15.23:0/2324739492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9968149fb0 msgr2=0x7f996813d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:52.852 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.846+0000 7f996dbca700 1 --2- 172.21.15.23:0/2324739492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9968149fb0 0x7f996813d7a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f99600090b0 tx=0x7f9960016000).stop 2020-11-20T11:48:52.852 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.846+0000 7f9967fff700 1 -- 172.21.15.23:0/2324739492 reap_dead start 2020-11-20T11:48:52.852 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.846+0000 7f9967fff700 1 -- 172.21.15.23:0/2324739492 reap_dead start 2020-11-20T11:48:52.852 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.846+0000 7f996dbca700 1 -- 172.21.15.23:0/2324739492 shutdown_connections 2020-11-20T11:48:52.853 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.846+0000 7f996dbca700 1 -- 172.21.15.23:0/2324739492 >> 172.21.15.23:0/2324739492 conn(0x7f99680678e0 msgr2=0x7f996813a2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:48:52.853 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.846+0000 7f996dbca700 1 -- 172.21.15.23:0/2324739492 shutdown_connections 2020-11-20T11:48:52.853 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:52.846+0000 7f996dbca700 1 -- 172.21.15.23:0/2324739492 wait complete. 2020-11-20T11:48:52.853 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:48:52.854 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:48:52.854 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:48:52.854 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "leader", 2020-11-20T11:48:52.854 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 158, 2020-11-20T11:48:52.854 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:48:52.855 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:48:52.855 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:48:52.855 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:48:52.855 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:48:52.855 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 3, 2020-11-20T11:48:52.856 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:48:52.856 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:48:52.856 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:48:52.856 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:48:52.856 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:48:52.857 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:48:52.857 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:48:52.857 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:48:52.857 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:48:52.857 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:48:52.858 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:48:52.858 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:48:52.858 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:48:52.858 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:48:52.858 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:48:52.859 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:48:52.859 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:48:52.859 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:48:52.859 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:48:52.859 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:48:52.860 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:48:52.860 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:48:52.860 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:48:52.860 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:52.860 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:48:52.861 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:48:52.861 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:48:52.861 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:48:52.861 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:48:52.861 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:48:52.862 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:48:52.862 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:48:52.862 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:48:52.862 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:48:52.862 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:48:52.863 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:48:52.863 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:48:52.863 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:48:52.863 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:48:52.863 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:48:52.864 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:48:52.864 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:48:52.864 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:48:52.864 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:48:52.864 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:48:52.865 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:48:52.865 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:48:52.865 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:48:52.865 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:48:52.865 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:52.866 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:48:52.866 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:52.866 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:48:52.866 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:48:52.866 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:48:52.867 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:48:52.867 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:52.867 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:48:52.867 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:48:52.867 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:48:52.868 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:52.868 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:52.868 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:48:52.868 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:48:52.868 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:48:52.869 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:52.869 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:48:52.869 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:52.869 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:48:52.869 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:48:52.870 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:48:52.870 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:48:52.870 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:48:52.870 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:52.870 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:52.871 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:48:52.871 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:48:52.871 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:48:52.871 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:48:52.871 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:52.872 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:48:52.872 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:48:52.872 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:48:52.872 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:52.872 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:52.873 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:48:52.873 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:48:52.873 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:48:52.873 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:52.874 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:48:52.874 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:52.874 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:48:52.874 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:48:52.874 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:48:52.875 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:48:52.875 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:48:52.875 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:52.875 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:52.875 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:48:52.876 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:48:52.876 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:48:52.876 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:48:52.876 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:52.876 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:48:52.877 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:48:52.877 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:48:52.877 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:52.877 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:52.877 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:48:52.878 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:48:52.878 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:48:52.878 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:52.878 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:48:52.878 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:52.879 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:48:52.879 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:48:52.879 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:48:52.879 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:48:52.879 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:48:52.880 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:52.880 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:48:52.880 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:52.880 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:48:52.880 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:48:52.881 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:52.881 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:48:52.881 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:48:52.881 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:48:52.881 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:52.882 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:48:52.882 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:48:52.882 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:52.882 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:48:52.882 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:48:52.883 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:48:52.883 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:52.883 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:48:52.883 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:48:52.883 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:52.883 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:48:52.884 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:48:52.884 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:48:52.884 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:52.884 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:48:52.884 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:48:52.885 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:52.885 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:48:52.885 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:48:52.885 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:48:52.886 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:52.886 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:48:52.886 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr": [ 2020-11-20T11:48:52.886 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:52.886 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:48:52.887 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:48:52.887 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 2 2020-11-20T11:48:52.887 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:52.887 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:48:52.887 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:52.888 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:48:52.888 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:48:52.889 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2020-11-20T11:48:53.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.043+0000 7f3fbbfff700 1 Processor -- start 2020-11-20T11:48:53.046 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.043+0000 7f3fbbfff700 1 -- start start 2020-11-20T11:48:53.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.043+0000 7f3fbbfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3fbc0a9040 0x7f3fbc0aaf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:53.048 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.043+0000 7f3fbbfff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fbc0ad2f0 0x7f3fbc13e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:53.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.043+0000 7f3fbbfff700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fbc0ad860 con 0x7f3fbc0ace70 2020-11-20T11:48:53.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.043+0000 7f3fbbfff700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fbc0aa8a0 con 0x7f3fbc0a9040 2020-11-20T11:48:53.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.043+0000 7f3fbbfff700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc0aaa90 con 0x7f3fbc0ad2f0 2020-11-20T11:48:53.049 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.043+0000 7f3fbaffd700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f3fbc0a9040 0x7f3fbc0aaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:53.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.043+0000 7f3fb3fff700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f3fbc0ace70 0x7f3fbc13de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:33100/0 (socket says 172.21.15.23:33100) 2020-11-20T11:48:53.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.043+0000 7f3fb3fff700 1 -- 172.21.15.23:0/2256228034 learned_addr learned my addr 172.21.15.23:0/2256228034 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:48:53.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.044+0000 7f3fbb7fe700 1 --2- 172.21.15.23:0/2256228034 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fbc0ad2f0 0x7f3fbc13e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:53.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.044+0000 7f3fbaffd700 1 -- 172.21.15.23:0/2256228034 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fbc0ad2f0 msgr2=0x7f3fbc13e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:48:53.050 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.044+0000 7f3fbaffd700 1 --2- 172.21.15.23:0/2256228034 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fbc0ad2f0 0x7f3fbc13e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:53.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.044+0000 7f3fbaffd700 1 -- 172.21.15.23:0/2256228034 >> v1:172.21.15.23:6789/0 conn(0x7f3fbc0ace70 legacy=0x7f3fbc13de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:53.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.044+0000 7f3fbaffd700 1 -- 172.21.15.23:0/2256228034 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fbc0a9420 con 0x7f3fbc0a9040 2020-11-20T11:48:53.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.044+0000 7f3fba7fc700 1 -- 172.21.15.23:0/2256228034 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1263989058 0 0) 0x7f3fbc0ad860 con 0x7f3fbc0ace70 2020-11-20T11:48:53.051 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.044+0000 7f3fbaffd700 1 --2- 172.21.15.23:0/2256228034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3fbc0a9040 0x7f3fbc0aaf30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f3fac00b9d0 tx=0x7f3fac006dd0).ready entity=mon.0 client_cookie=2a80d785b7884bfc server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:53.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.044+0000 7f3fba7fc700 1 -- 172.21.15.23:0/2256228034 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fac006030 con 0x7f3fbc0a9040 2020-11-20T11:48:53.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.044+0000 7f3fba7fc700 1 -- 172.21.15.23:0/2256228034 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fac006190 con 0x7f3fbc0a9040 2020-11-20T11:48:53.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.044+0000 7f3fba7fc700 1 -- 172.21.15.23:0/2256228034 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fac006510 con 0x7f3fbc0a9040 2020-11-20T11:48:53.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.045+0000 7f3fbbfff700 1 -- 172.21.15.23:0/2256228034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3fbc0a9040 msgr2=0x7f3fbc0aaf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:53.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.045+0000 7f3fbbfff700 1 --2- 172.21.15.23:0/2256228034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3fbc0a9040 0x7f3fbc0aaf30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f3fac00b9d0 tx=0x7f3fac006dd0).stop 2020-11-20T11:48:53.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.045+0000 7f3fbbfff700 1 -- 172.21.15.23:0/2256228034 shutdown_connections 2020-11-20T11:48:53.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.045+0000 7f3fbbfff700 1 --2- 172.21.15.23:0/2256228034 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fbc0ad2f0 0x7f3fbc13e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:53.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.045+0000 7f3fbbfff700 1 --2- 172.21.15.23:0/2256228034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3fbc0a9040 0x7f3fbc0aaf30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:53.053 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.045+0000 7f3fbbfff700 1 -- 172.21.15.23:0/2256228034 >> 172.21.15.23:0/2256228034 conn(0x7f3fbc0683b0 msgr2=0x7f3fbc0b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:48:53.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.045+0000 7f3fbbfff700 1 -- 172.21.15.23:0/2256228034 shutdown_connections 2020-11-20T11:48:53.054 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.045+0000 7f3fbbfff700 1 -- 172.21.15.23:0/2256228034 wait complete. 2020-11-20T11:48:53.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.046+0000 7f3fbbfff700 1 Processor -- start 2020-11-20T11:48:53.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.048+0000 7f3fbbfff700 1 -- start start 2020-11-20T11:48:53.057 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.049+0000 7f3fbbfff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fbc0a9040 0x7f3fbc09f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:53.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.049+0000 7f3fbbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3fbc0ace70 0x7f3fbc149cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:53.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.049+0000 7f3fbbfff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fbc0ad2f0 0x7f3fbc0a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:53.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.049+0000 7f3fbbfff700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc14a1b0 con 0x7f3fbc0ace70 2020-11-20T11:48:53.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.049+0000 7f3fbbfff700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc09ea60 con 0x7f3fbc0ad2f0 2020-11-20T11:48:53.058 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.049+0000 7f3fbbfff700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc09ed00 con 0x7f3fbc0a9040 2020-11-20T11:48:53.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.049+0000 7f3fbaffd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fbc0a9040 0x7f3fbc09f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:53.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.049+0000 7f3fbb7fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fbc0ad2f0 0x7f3fbc0a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:53.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.049+0000 7f3fbaffd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fbc0a9040 0x7f3fbc09f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:57326/0 (socket says 172.21.15.23:57326) 2020-11-20T11:48:53.059 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.049+0000 7f3fbb7fe700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fbc0ad2f0 0x7f3fbc0a0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:39984/0 (socket says 172.21.15.23:39984) 2020-11-20T11:48:53.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.049+0000 7f3fbaffd700 1 -- 172.21.15.23:0/519597740 learned_addr learned my addr 172.21.15.23:0/519597740 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:48:53.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.050+0000 7f3fbaffd700 1 -- 172.21.15.23:0/519597740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fbc0ad2f0 msgr2=0x7f3fbc0a0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:53.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.050+0000 7f3fbaffd700 1 --2- 172.21.15.23:0/519597740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fbc0ad2f0 0x7f3fbc0a0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:53.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.050+0000 7f3fbaffd700 1 -- 172.21.15.23:0/519597740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3fbc0ace70 msgr2=0x7f3fbc149cc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:48:53.060 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.050+0000 7f3fbaffd700 1 --2- 172.21.15.23:0/519597740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3fbc0ace70 0x7f3fbc149cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:48:53.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.050+0000 7f3fbaffd700 1 -- 172.21.15.23:0/519597740 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fbc09efa0 con 0x7f3fbc0a9040 2020-11-20T11:48:53.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.050+0000 7f3fbb7fe700 1 --2- 172.21.15.23:0/519597740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fbc0ad2f0 0x7f3fbc0a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:48:53.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.050+0000 7f3fbaffd700 1 --2- 172.21.15.23:0/519597740 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fbc0a9040 0x7f3fbc09f440 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f3fac006fd0 tx=0x7f3fac015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:53.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.050+0000 7f3fb2ffd700 1 -- 172.21.15.23:0/519597740 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fac005920 con 0x7f3fbc0a9040 2020-11-20T11:48:53.061 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.050+0000 7f3fb2ffd700 1 -- 172.21.15.23:0/519597740 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fac005a80 con 0x7f3fbc0a9040 2020-11-20T11:48:53.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.050+0000 7f3fb2ffd700 1 -- 172.21.15.23:0/519597740 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fac005e00 con 0x7f3fbc0a9040 2020-11-20T11:48:53.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.050+0000 7f3fbbfff700 1 -- 172.21.15.23:0/519597740 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fbc140860 con 0x7f3fbc0a9040 2020-11-20T11:48:53.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.050+0000 7f3fbbfff700 1 -- 172.21.15.23:0/519597740 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fbc14a580 con 0x7f3fbc0a9040 2020-11-20T11:48:53.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.051+0000 7f3fbbfff700 1 --2- 172.21.15.23:0/519597740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fbc14a830 0x7f3fbc09e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:53.062 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.051+0000 7f3fbbfff700 1 -- 172.21.15.23:0/519597740 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc05cf20 con 0x7f3fbc14a830 2020-11-20T11:48:53.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.052+0000 7f3fb3fff700 1 --2- 172.21.15.23:0/519597740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fbc14a830 0x7f3fbc09e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:53.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.052+0000 7f3fb2ffd700 1 -- 172.21.15.23:0/519597740 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3fac025030 con 0x7f3fbc0a9040 2020-11-20T11:48:53.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.053+0000 7f3fb3fff700 1 -- 172.21.15.23:0/519597740 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3fbc14ac10 con 0x7f3fbc14a830 2020-11-20T11:48:53.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.053+0000 7f3fb3fff700 1 --2- 172.21.15.23:0/519597740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fbc14a830 0x7f3fbc09e010 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f3fbc0b10c0 tx=0x7f3fa40079c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:53.063 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.053+0000 7f3fb2ffd700 1 --2- 172.21.15.23:0/519597740 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3f98056d60 0x7f3f98059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:53.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.053+0000 7f3fb2ffd700 1 -- 172.21.15.23:0/519597740 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f3fac0715f0 con 0x7f3fbc0a9040 2020-11-20T11:48:53.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.053+0000 7f3fbb7fe700 1 --2- 172.21.15.23:0/519597740 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3f98056d60 0x7f3f98059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:53.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.053+0000 7f3fb2ffd700 1 -- 172.21.15.23:0/519597740 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa40072e0 con 0x7f3fbc14a830 2020-11-20T11:48:53.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.053+0000 7f3fb2ffd700 1 -- 172.21.15.23:0/519597740 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7f3fa4007440 con 0x7f3fbc14a830 2020-11-20T11:48:53.064 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.053+0000 7f3fb2ffd700 1 -- 172.21.15.23:0/519597740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fbc14a830 msgr2=0x7f3fbc09e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:53.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.053+0000 7f3fb2ffd700 1 --2- 172.21.15.23:0/519597740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fbc14a830 0x7f3fbc09e010 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f3fbc0b10c0 tx=0x7f3fa40079c0).stop 2020-11-20T11:48:53.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.053+0000 7f3fb2ffd700 1 -- 172.21.15.23:0/519597740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fbc14a830 msgr2=0x7f3fbc09e010 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:48:53.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.053+0000 7f3fb2ffd700 1 --2- 172.21.15.23:0/519597740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fbc14a830 0x7f3fbc09e010 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0x7f3fbc0b10c0 tx=0x7f3fa40079c0).stop 2020-11-20T11:48:53.065 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.054+0000 7f3fbb7fe700 1 --2- 172.21.15.23:0/519597740 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3f98056d60 0x7f3f98059220 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f3fbc0a0a90 tx=0x7f3fb4009220).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:53.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.062+0000 7f3fbbfff700 1 --2- 172.21.15.23:0/519597740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fbc09e500 0x7f3fbc14d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:53.066 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.062+0000 7f3fbbfff700 1 -- 172.21.15.23:0/519597740 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc09eba0 con 0x7f3fbc09e500 2020-11-20T11:48:53.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.062+0000 7f3fb3fff700 1 --2- 172.21.15.23:0/519597740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fbc09e500 0x7f3fbc14d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:53.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.062+0000 7f3fb3fff700 1 -- 172.21.15.23:0/519597740 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3fbc057dc0 con 0x7f3fbc09e500 2020-11-20T11:48:53.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.062+0000 7f3fb3fff700 1 --2- 172.21.15.23:0/519597740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fbc09e500 0x7f3fbc14d290 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f3fa4007c70 tx=0x7f3fa4015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:53.067 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.063+0000 7f3fb2ffd700 1 -- 172.21.15.23:0/519597740 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa4013bb0 con 0x7f3fbc09e500 2020-11-20T11:48:53.068 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.063+0000 7f3fb2ffd700 1 -- 172.21.15.23:0/519597740 <== mon.1 v2:172.21.15.155:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4444 (secure 0 0 0) 0x7f3fa40070d0 con 0x7f3fbc09e500 2020-11-20T11:48:53.068 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.063+0000 7f3fb2ffd700 1 -- 172.21.15.23:0/519597740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fbc09e500 msgr2=0x7f3fbc14d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:53.068 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.063+0000 7f3fb2ffd700 1 --2- 172.21.15.23:0/519597740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fbc09e500 0x7f3fbc14d290 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f3fa4007c70 tx=0x7f3fa4015000).stop 2020-11-20T11:48:53.068 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.063+0000 7f3fb2ffd700 1 -- 172.21.15.23:0/519597740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fbc09e500 msgr2=0x7f3fbc14d290 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:48:53.068 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.063+0000 7f3fb2ffd700 1 --2- 172.21.15.23:0/519597740 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fbc09e500 0x7f3fbc14d290 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0x7f3fa4007c70 tx=0x7f3fa4015000).stop 2020-11-20T11:48:53.070 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:48:53.070 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:48:53.070 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:48:53.071 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:48:53.071 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 158, 2020-11-20T11:48:53.071 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:48:53.071 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:48:53.071 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:48:53.072 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:48:53.072 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:48:53.072 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 3, 2020-11-20T11:48:53.072 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:48:53.072 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:48:53.073 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:48:53.073 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:48:53.073 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:48:53.073 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:48:53.073 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:48:53.074 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:48:53.074 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:48:53.074 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:48:53.074 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:48:53.074 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:48:53.075 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:48:53.075 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:48:53.075 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:48:53.075 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:48:53.075 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:48:53.076 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:48:53.076 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:48:53.076 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:48:53.076 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:48:53.076 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:48:53.077 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:48:53.077 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:53.077 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:48:53.077 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:48:53.077 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:48:53.078 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:48:53.078 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:48:53.078 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:48:53.078 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:48:53.078 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:48:53.078 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:48:53.079 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:48:53.079 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:48:53.079 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:48:53.079 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:48:53.079 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:48:53.080 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:48:53.080 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:48:53.080 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:48:53.080 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:48:53.081 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:48:53.081 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:48:53.081 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:48:53.081 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:48:53.081 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:48:53.082 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:48:53.082 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:48:53.082 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:53.082 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:48:53.082 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:53.083 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:48:53.083 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:48:53.083 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:48:53.083 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:48:53.084 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:53.084 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:48:53.084 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:48:53.084 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:48:53.085 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:53.085 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:53.085 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:48:53.085 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:48:53.086 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:48:53.086 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:53.086 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:48:53.086 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:53.086 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:48:53.087 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:48:53.087 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:48:53.087 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:48:53.087 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:48:53.087 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:53.088 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:53.088 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:48:53.088 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:48:53.088 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:48:53.088 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:48:53.089 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:53.089 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:48:53.089 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:48:53.089 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:48:53.090 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:53.090 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:53.090 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:48:53.090 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:48:53.090 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:48:53.090 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:53.091 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:48:53.091 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:53.091 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:48:53.091 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:48:53.091 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:48:53.092 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:48:53.092 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:48:53.092 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:53.092 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:53.092 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:48:53.093 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:48:53.093 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:48:53.093 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:48:53.093 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:53.093 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:48:53.094 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:48:53.094 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:48:53.094 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:53.094 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:53.094 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:48:53.095 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:48:53.095 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:48:53.095 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:53.095 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:48:53.095 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:53.096 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:48:53.096 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:48:53.096 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:48:53.096 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:48:53.096 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:48:53.097 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:53.097 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:48:53.097 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:53.097 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:48:53.098 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:48:53.098 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:53.098 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:48:53.098 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:48:53.098 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:48:53.099 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:53.099 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:48:53.099 INFO:teuthology.orchestra.run.smithi023.stdout: "osd": [ 2020-11-20T11:48:53.099 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:53.099 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:48:53.100 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:48:53.100 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 6 2020-11-20T11:48:53.100 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:53.100 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:48:53.100 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:48:53.101 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:53.101 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:48:53.101 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:48:53.101 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 5 2020-11-20T11:48:53.101 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:53.102 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:48:53.102 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:53.102 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:48:53.102 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:48:53.102 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.067+0000 7f3fbbfff700 1 -- 172.21.15.23:0/519597740 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3f98056d60 msgr2=0x7f3f98059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:53.103 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.067+0000 7f3fbbfff700 1 --2- 172.21.15.23:0/519597740 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f3f98056d60 0x7f3f98059220 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f3fbc0a0a90 tx=0x7f3fb4009220).stop 2020-11-20T11:48:53.103 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.067+0000 7f3fbbfff700 1 -- 172.21.15.23:0/519597740 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fbc0a9040 msgr2=0x7f3fbc09f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:53.103 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.067+0000 7f3fbbfff700 1 --2- 172.21.15.23:0/519597740 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fbc0a9040 0x7f3fbc09f440 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f3fac006fd0 tx=0x7f3fac015000).stop 2020-11-20T11:48:53.103 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.067+0000 7f3fbb7fe700 1 -- 172.21.15.23:0/519597740 reap_dead start 2020-11-20T11:48:53.103 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.067+0000 7f3fbb7fe700 1 -- 172.21.15.23:0/519597740 reap_dead start 2020-11-20T11:48:53.104 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.067+0000 7f3fbbfff700 1 -- 172.21.15.23:0/519597740 shutdown_connections 2020-11-20T11:48:53.104 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.067+0000 7f3fbbfff700 1 -- 172.21.15.23:0/519597740 >> 172.21.15.23:0/519597740 conn(0x7f3fbc0683b0 msgr2=0x7f3fbc0ac330 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:48:53.104 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.067+0000 7f3fbbfff700 1 -- 172.21.15.23:0/519597740 shutdown_connections 2020-11-20T11:48:53.104 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.067+0000 7f3fbbfff700 1 -- 172.21.15.23:0/519597740 wait complete. 2020-11-20T11:48:53.106 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2020-11-20T11:48:53.256 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.253+0000 7fb10b4c4700 1 Processor -- start 2020-11-20T11:48:53.257 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.253+0000 7fb10b4c4700 1 -- start start 2020-11-20T11:48:53.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.254+0000 7fb10b4c4700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb1040aaab0 0x7fb10413ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:53.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.254+0000 7fb10b4c4700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb1040aae90 0x7fb1041381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:53.259 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.254+0000 7fb10b4c4700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb10413e600 con 0x7fb10413aeb0 2020-11-20T11:48:53.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.254+0000 7fb10b4c4700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb10413e8a0 con 0x7fb1040aaab0 2020-11-20T11:48:53.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.254+0000 7fb10b4c4700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1040ab6d0 con 0x7fb1040aae90 2020-11-20T11:48:53.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.254+0000 7fb109260700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fb1040aaab0 0x7fb10413ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:53.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.254+0000 7fb109a61700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fb10413aeb0 0x7fb1040a8c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:33118/0 (socket says 172.21.15.23:33118) 2020-11-20T11:48:53.260 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.254+0000 7fb109a61700 1 -- 172.21.15.23:0/2275925893 learned_addr learned my addr 172.21.15.23:0/2275925893 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:48:53.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.254+0000 7fb108a5f700 1 --2- 172.21.15.23:0/2275925893 >> v2:172.21.15.155:3300/0 conn(0x7fb1040aae90 0x7fb1041381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:53.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.255+0000 7fb109260700 1 -- 172.21.15.23:0/2275925893 >> v2:172.21.15.155:3300/0 conn(0x7fb1040aae90 msgr2=0x7fb1041381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:48:53.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.255+0000 7fb109260700 1 --2- 172.21.15.23:0/2275925893 >> v2:172.21.15.155:3300/0 conn(0x7fb1040aae90 0x7fb1041381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:53.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.255+0000 7fb109260700 1 -- 172.21.15.23:0/2275925893 >> v1:172.21.15.23:6789/0 conn(0x7fb10413aeb0 legacy=0x7fb1040a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:53.261 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.255+0000 7fb0f3fff700 1 -- 172.21.15.23:0/2275925893 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4193104666 0 0) 0x7fb10413e600 con 0x7fb10413aeb0 2020-11-20T11:48:53.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.255+0000 7fb109260700 1 -- 172.21.15.23:0/2275925893 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1040a9370 con 0x7fb1040aaab0 2020-11-20T11:48:53.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.255+0000 7fb109260700 1 --2- 172.21.15.23:0/2275925893 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1040aaab0 0x7fb10413ed40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fb1040ab930 tx=0x7fb0f4006e30).ready entity=mon.0 client_cookie=c14522603e96667c server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:53.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.255+0000 7fb0f3fff700 1 -- 172.21.15.23:0/2275925893 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f40097b0 con 0x7fb1040aaab0 2020-11-20T11:48:53.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.255+0000 7fb0f3fff700 1 -- 172.21.15.23:0/2275925893 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0f4000e90 con 0x7fb1040aaab0 2020-11-20T11:48:53.262 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.255+0000 7fb0f3fff700 1 -- 172.21.15.23:0/2275925893 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f4009b20 con 0x7fb1040aaab0 2020-11-20T11:48:53.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.255+0000 7fb10b4c4700 1 -- 172.21.15.23:0/2275925893 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1040aaab0 msgr2=0x7fb10413ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:53.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.255+0000 7fb10b4c4700 1 --2- 172.21.15.23:0/2275925893 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1040aaab0 0x7fb10413ed40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fb1040ab930 tx=0x7fb0f4006e30).stop 2020-11-20T11:48:53.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.255+0000 7fb10b4c4700 1 -- 172.21.15.23:0/2275925893 shutdown_connections 2020-11-20T11:48:53.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.255+0000 7fb10b4c4700 1 --2- 172.21.15.23:0/2275925893 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1040aaab0 0x7fb10413ed40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:53.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.255+0000 7fb10b4c4700 1 --2- 172.21.15.23:0/2275925893 >> v2:172.21.15.155:3300/0 conn(0x7fb1040aae90 0x7fb1041381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:53.263 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.255+0000 7fb10b4c4700 1 -- 172.21.15.23:0/2275925893 >> 172.21.15.23:0/2275925893 conn(0x7fb1040678e0 msgr2=0x7fb104067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:48:53.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.255+0000 7fb10b4c4700 1 -- 172.21.15.23:0/2275925893 shutdown_connections 2020-11-20T11:48:53.264 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.255+0000 7fb10b4c4700 1 -- 172.21.15.23:0/2275925893 wait complete. 2020-11-20T11:48:53.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.256+0000 7fb10b4c4700 1 Processor -- start 2020-11-20T11:48:53.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.256+0000 7fb10b4c4700 1 -- start start 2020-11-20T11:48:53.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.257+0000 7fb10b4c4700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1040aaab0 0x7fb1040a7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:53.267 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.257+0000 7fb10b4c4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1040aae90 0x7fb10413d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:53.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.257+0000 7fb10b4c4700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb10413dc10 0x7fb1040a72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:53.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.257+0000 7fb10b4c4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1040a5490 con 0x7fb1040aae90 2020-11-20T11:48:53.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.257+0000 7fb10b4c4700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1040a56f0 con 0x7fb10413dc10 2020-11-20T11:48:53.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.257+0000 7fb10b4c4700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1040a5990 con 0x7fb1040aaab0 2020-11-20T11:48:53.268 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.257+0000 7fb108a5f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1040aae90 0x7fb10413d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:53.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.257+0000 7fb108a5f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1040aae90 0x7fb10413d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47102/0 (socket says 172.21.15.23:47102) 2020-11-20T11:48:53.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.257+0000 7fb108a5f700 1 -- 172.21.15.23:0/774436672 learned_addr learned my addr 172.21.15.23:0/774436672 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:48:53.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.258+0000 7fb109a61700 1 --2- 172.21.15.23:0/774436672 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb10413dc10 0x7fb1040a72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:53.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.258+0000 7fb109260700 1 --2- 172.21.15.23:0/774436672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1040aaab0 0x7fb1040a7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:53.269 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.258+0000 7fb108a5f700 1 -- 172.21.15.23:0/774436672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1040aaab0 msgr2=0x7fb1040a7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:53.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.258+0000 7fb108a5f700 1 --2- 172.21.15.23:0/774436672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1040aaab0 0x7fb1040a7ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:53.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.258+0000 7fb108a5f700 1 -- 172.21.15.23:0/774436672 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb10413dc10 msgr2=0x7fb1040a72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:53.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.258+0000 7fb108a5f700 1 --2- 172.21.15.23:0/774436672 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb10413dc10 0x7fb1040a72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:53.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.258+0000 7fb108a5f700 1 -- 172.21.15.23:0/774436672 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1040a5e30 con 0x7fb1040aae90 2020-11-20T11:48:53.270 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.258+0000 7fb108a5f700 1 --2- 172.21.15.23:0/774436672 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1040aae90 0x7fb10413d720 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fb10000eba0 tx=0x7fb100006e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:53.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.258+0000 7fb0f17fa700 1 -- 172.21.15.23:0/774436672 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10000c9d0 con 0x7fb1040aae90 2020-11-20T11:48:53.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.258+0000 7fb0f17fa700 1 -- 172.21.15.23:0/774436672 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb10000cb30 con 0x7fb1040aae90 2020-11-20T11:48:53.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.258+0000 7fb0f17fa700 1 -- 172.21.15.23:0/774436672 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1000123f0 con 0x7fb1040aae90 2020-11-20T11:48:53.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.258+0000 7fb10b4c4700 1 -- 172.21.15.23:0/774436672 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1040b02d0 con 0x7fb1040aae90 2020-11-20T11:48:53.271 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.258+0000 7fb10b4c4700 1 -- 172.21.15.23:0/774436672 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1040a79e0 con 0x7fb1040aae90 2020-11-20T11:48:53.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.259+0000 7fb10b4c4700 1 --2- 172.21.15.23:0/774436672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb10413aeb0 0x7fb1041355d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:53.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.259+0000 7fb10b4c4700 1 -- 172.21.15.23:0/774436672 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb10405c7d0 con 0x7fb10413aeb0 2020-11-20T11:48:53.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.259+0000 7fb0f17fa700 1 -- 172.21.15.23:0/774436672 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fb1000127e0 con 0x7fb1040aae90 2020-11-20T11:48:53.272 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.259+0000 7fb109260700 1 --2- 172.21.15.23:0/774436672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb10413aeb0 0x7fb1041355d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:53.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.259+0000 7fb0f17fa700 1 --2- 172.21.15.23:0/774436672 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb0dc056d60 0x7fb0dc059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:53.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.259+0000 7fb109260700 1 -- 172.21.15.23:0/774436672 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb1040a5b50 con 0x7fb10413aeb0 2020-11-20T11:48:53.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.259+0000 7fb0f17fa700 1 -- 172.21.15.23:0/774436672 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fb1000704c0 con 0x7fb1040aae90 2020-11-20T11:48:53.273 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.259+0000 7fb109260700 1 --2- 172.21.15.23:0/774436672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb10413aeb0 0x7fb1041355d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fb1040ab930 tx=0x7fb0f4015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:53.274 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.260+0000 7fb109a61700 1 --2- 172.21.15.23:0/774436672 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb0dc056d60 0x7fb0dc059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:53.274 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.260+0000 7fb0f17fa700 1 -- 172.21.15.23:0/774436672 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f40119c0 con 0x7fb10413aeb0 2020-11-20T11:48:53.274 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.260+0000 7fb109a61700 1 --2- 172.21.15.23:0/774436672 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb0dc056d60 0x7fb0dc059220 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7fb0fc007fd0 tx=0x7fb0fc009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:53.274 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.260+0000 7fb0f17fa700 1 -- 172.21.15.23:0/774436672 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8042 (secure 0 0 0) 0x7fb0f4011b20 con 0x7fb10413aeb0 2020-11-20T11:48:53.274 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.260+0000 7fb0f17fa700 1 -- 172.21.15.23:0/774436672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb10413aeb0 msgr2=0x7fb1041355d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:53.275 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.260+0000 7fb0f17fa700 1 --2- 172.21.15.23:0/774436672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb10413aeb0 0x7fb1041355d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fb1040ab930 tx=0x7fb0f4015000).stop 2020-11-20T11:48:53.275 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.260+0000 7fb0f17fa700 1 -- 172.21.15.23:0/774436672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb10413aeb0 msgr2=0x7fb1041355d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:48:53.275 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.260+0000 7fb0f17fa700 1 --2- 172.21.15.23:0/774436672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb10413aeb0 0x7fb1041355d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:53.276 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.267+0000 7fb10b4c4700 1 --2- 172.21.15.23:0/774436672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1040601b0 0x7fb104135ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:53.276 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.267+0000 7fb10b4c4700 1 -- 172.21.15.23:0/774436672 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb104044710 con 0x7fb1040601b0 2020-11-20T11:48:53.276 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.267+0000 7fb109260700 1 --2- 172.21.15.23:0/774436672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1040601b0 0x7fb104135ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:53.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.268+0000 7fb109260700 1 -- 172.21.15.23:0/774436672 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb10405c280 con 0x7fb1040601b0 2020-11-20T11:48:53.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.268+0000 7fb109260700 1 --2- 172.21.15.23:0/774436672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1040601b0 0x7fb104135ac0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fb0f4006e70 tx=0x7fb0f401f000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:53.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.268+0000 7fb0f17fa700 1 -- 172.21.15.23:0/774436672 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f402d770 con 0x7fb1040601b0 2020-11-20T11:48:53.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.268+0000 7fb0f17fa700 1 -- 172.21.15.23:0/774436672 <== mon.2 v2:172.21.15.155:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4444 (secure 0 0 0) 0x7fb0f40117b0 con 0x7fb1040601b0 2020-11-20T11:48:53.277 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.268+0000 7fb0f17fa700 1 -- 172.21.15.23:0/774436672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1040601b0 msgr2=0x7fb104135ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:53.278 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.268+0000 7fb0f17fa700 1 --2- 172.21.15.23:0/774436672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1040601b0 0x7fb104135ac0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fb0f4006e70 tx=0x7fb0f401f000).stop 2020-11-20T11:48:53.278 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.268+0000 7fb0f17fa700 1 -- 172.21.15.23:0/774436672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1040601b0 msgr2=0x7fb104135ac0 secure :-1 s=STATE_CLOSED l=1).mark_down 2020-11-20T11:48:53.278 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.268+0000 7fb0f17fa700 1 --2- 172.21.15.23:0/774436672 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1040601b0 0x7fb104135ac0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:53.280 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:48:53.280 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:48:53.281 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:48:53.281 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "peon", 2020-11-20T11:48:53.281 INFO:teuthology.orchestra.run.smithi023.stdout: "election_epoch": 158, 2020-11-20T11:48:53.281 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum": [ 2020-11-20T11:48:53.282 INFO:teuthology.orchestra.run.smithi023.stdout: 0, 2020-11-20T11:48:53.282 INFO:teuthology.orchestra.run.smithi023.stdout: 1, 2020-11-20T11:48:53.282 INFO:teuthology.orchestra.run.smithi023.stdout: 2 2020-11-20T11:48:53.282 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:48:53.283 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_age": 3, 2020-11-20T11:48:53.283 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:48:53.283 INFO:teuthology.orchestra.run.smithi023.stdout: "required_con": "2449958747317026820", 2020-11-20T11:48:53.283 INFO:teuthology.orchestra.run.smithi023.stdout: "required_mon": [ 2020-11-20T11:48:53.284 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:48:53.284 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:48:53.284 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:48:53.284 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:48:53.284 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:48:53.285 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:48:53.285 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:48:53.285 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:48:53.285 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:48:53.286 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_con": "4540138297136906239", 2020-11-20T11:48:53.286 INFO:teuthology.orchestra.run.smithi023.stdout: "quorum_mon": [ 2020-11-20T11:48:53.286 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:48:53.286 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:48:53.286 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:48:53.287 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:48:53.287 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:48:53.287 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:48:53.287 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:48:53.287 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:48:53.288 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:48:53.288 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:53.288 INFO:teuthology.orchestra.run.smithi023.stdout: "outside_quorum": [], 2020-11-20T11:48:53.288 INFO:teuthology.orchestra.run.smithi023.stdout: "extra_probe_peers": [], 2020-11-20T11:48:53.288 INFO:teuthology.orchestra.run.smithi023.stdout: "sync_provider": [], 2020-11-20T11:48:53.289 INFO:teuthology.orchestra.run.smithi023.stdout: "monmap": { 2020-11-20T11:48:53.289 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 1, 2020-11-20T11:48:53.289 INFO:teuthology.orchestra.run.smithi023.stdout: "fsid": "4b071d14-8a39-4afe-8acc-4218fa4790c7", 2020-11-20T11:48:53.289 INFO:teuthology.orchestra.run.smithi023.stdout: "modified": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:48:53.290 INFO:teuthology.orchestra.run.smithi023.stdout: "created": "2020-11-20T11:31:30.112250Z", 2020-11-20T11:48:53.290 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release": 16, 2020-11-20T11:48:53.290 INFO:teuthology.orchestra.run.smithi023.stdout: "min_mon_release_name": "pacific", 2020-11-20T11:48:53.290 INFO:teuthology.orchestra.run.smithi023.stdout: "election_strategy": 1, 2020-11-20T11:48:53.290 INFO:teuthology.orchestra.run.smithi023.stdout: "disallowed_leaders: ": "", 2020-11-20T11:48:53.291 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false, 2020-11-20T11:48:53.291 INFO:teuthology.orchestra.run.smithi023.stdout: "features": { 2020-11-20T11:48:53.291 INFO:teuthology.orchestra.run.smithi023.stdout: "persistent": [ 2020-11-20T11:48:53.291 INFO:teuthology.orchestra.run.smithi023.stdout: "kraken", 2020-11-20T11:48:53.291 INFO:teuthology.orchestra.run.smithi023.stdout: "luminous", 2020-11-20T11:48:53.292 INFO:teuthology.orchestra.run.smithi023.stdout: "mimic", 2020-11-20T11:48:53.292 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap-prune", 2020-11-20T11:48:53.292 INFO:teuthology.orchestra.run.smithi023.stdout: "nautilus", 2020-11-20T11:48:53.292 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus", 2020-11-20T11:48:53.292 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific", 2020-11-20T11:48:53.293 INFO:teuthology.orchestra.run.smithi023.stdout: "elector-pinging" 2020-11-20T11:48:53.293 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:48:53.293 INFO:teuthology.orchestra.run.smithi023.stdout: "optional": [] 2020-11-20T11:48:53.293 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:53.293 INFO:teuthology.orchestra.run.smithi023.stdout: "mons": [ 2020-11-20T11:48:53.294 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:53.294 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 0, 2020-11-20T11:48:53.294 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "a", 2020-11-20T11:48:53.294 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:48:53.294 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:48:53.295 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:53.295 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:48:53.295 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:3300", 2020-11-20T11:48:53.295 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:48:53.295 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:53.296 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:53.296 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:48:53.296 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789", 2020-11-20T11:48:53.296 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:48:53.296 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:53.297 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:48:53.297 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:53.297 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6789/0", 2020-11-20T11:48:53.297 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.23:6789/0", 2020-11-20T11:48:53.297 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:48:53.297 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:48:53.298 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:48:53.298 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:53.298 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:53.298 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 1, 2020-11-20T11:48:53.298 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "b", 2020-11-20T11:48:53.299 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:48:53.299 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:48:53.299 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:53.299 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:48:53.299 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3300", 2020-11-20T11:48:53.300 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:48:53.300 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:53.300 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:53.300 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:48:53.300 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789", 2020-11-20T11:48:53.301 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:48:53.305 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:53.305 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:48:53.305 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:53.306 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6789/0", 2020-11-20T11:48:53.306 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6789/0", 2020-11-20T11:48:53.306 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:48:53.306 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:48:53.306 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:48:53.307 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:53.307 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:53.307 INFO:teuthology.orchestra.run.smithi023.stdout: "rank": 2, 2020-11-20T11:48:53.307 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "c", 2020-11-20T11:48:53.307 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addrs": { 2020-11-20T11:48:53.308 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2020-11-20T11:48:53.308 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:53.308 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2020-11-20T11:48:53.308 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:3301", 2020-11-20T11:48:53.308 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:48:53.309 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:53.309 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:53.309 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2020-11-20T11:48:53.309 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790", 2020-11-20T11:48:53.309 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 0 2020-11-20T11:48:53.310 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:53.310 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:48:53.310 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:53.310 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.155:6790/0", 2020-11-20T11:48:53.310 INFO:teuthology.orchestra.run.smithi023.stdout: "public_addr": "172.21.15.155:6790/0", 2020-11-20T11:48:53.311 INFO:teuthology.orchestra.run.smithi023.stdout: "priority": 0, 2020-11-20T11:48:53.311 INFO:teuthology.orchestra.run.smithi023.stdout: "weight": 0, 2020-11-20T11:48:53.311 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_location": "{}" 2020-11-20T11:48:53.311 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:53.311 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:48:53.311 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:53.312 INFO:teuthology.orchestra.run.smithi023.stdout: "feature_map": { 2020-11-20T11:48:53.312 INFO:teuthology.orchestra.run.smithi023.stdout: "mon": [ 2020-11-20T11:48:53.312 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:53.312 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:48:53.312 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:48:53.313 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:48:53.313 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:53.313 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:48:53.313 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": [ 2020-11-20T11:48:53.314 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:53.314 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:48:53.314 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:48:53.314 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:48:53.314 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:53.314 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2020-11-20T11:48:53.315 INFO:teuthology.orchestra.run.smithi023.stdout: "client": [ 2020-11-20T11:48:53.315 INFO:teuthology.orchestra.run.smithi023.stdout: { 2020-11-20T11:48:53.315 INFO:teuthology.orchestra.run.smithi023.stdout: "features": "0x3f01cfb9fffdffff", 2020-11-20T11:48:53.315 INFO:teuthology.orchestra.run.smithi023.stdout: "release": "luminous", 2020-11-20T11:48:53.315 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 1 2020-11-20T11:48:53.316 INFO:teuthology.orchestra.run.smithi023.stdout: } 2020-11-20T11:48:53.316 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2020-11-20T11:48:53.316 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2020-11-20T11:48:53.316 INFO:teuthology.orchestra.run.smithi023.stdout: "stretch_mode": false 2020-11-20T11:48:53.316 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:48:53.317 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.277+0000 7fb10b4c4700 1 -- 172.21.15.23:0/774436672 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb0dc056d60 msgr2=0x7fb0dc059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:53.317 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.277+0000 7fb10b4c4700 1 --2- 172.21.15.23:0/774436672 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fb0dc056d60 0x7fb0dc059220 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7fb0fc007fd0 tx=0x7fb0fc009040).stop 2020-11-20T11:48:53.317 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.277+0000 7fb10b4c4700 1 -- 172.21.15.23:0/774436672 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1040aae90 msgr2=0x7fb10413d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:53.317 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.277+0000 7fb10b4c4700 1 --2- 172.21.15.23:0/774436672 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1040aae90 0x7fb10413d720 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fb10000eba0 tx=0x7fb100006e30).stop 2020-11-20T11:48:53.317 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.277+0000 7fb109a61700 1 -- 172.21.15.23:0/774436672 reap_dead start 2020-11-20T11:48:53.318 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.277+0000 7fb109a61700 1 -- 172.21.15.23:0/774436672 reap_dead start 2020-11-20T11:48:53.319 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.279+0000 7fb10b4c4700 1 -- 172.21.15.23:0/774436672 shutdown_connections 2020-11-20T11:48:53.319 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.279+0000 7fb10b4c4700 1 -- 172.21.15.23:0/774436672 >> 172.21.15.23:0/774436672 conn(0x7fb1040678e0 msgr2=0x7fb104138e20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:48:53.319 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.279+0000 7fb10b4c4700 1 -- 172.21.15.23:0/774436672 shutdown_connections 2020-11-20T11:48:53.320 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.279+0000 7fb10b4c4700 1 -- 172.21.15.23:0/774436672 wait complete. 2020-11-20T11:48:53.320 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2020-11-20T11:48:53.320 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2020-11-20T11:48:53.459 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.456+0000 7f5118b4a700 1 Processor -- start 2020-11-20T11:48:53.459 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.456+0000 7f5118b4a700 1 -- start start 2020-11-20T11:48:53.461 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.456+0000 7f5118b4a700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f51140a2080 0x7f51140aef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:53.462 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.456+0000 7f5118b4a700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5114067a10 0x7f5114067e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:53.462 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.456+0000 7f5118b4a700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51140aaab0 con 0x7f51140a2460 2020-11-20T11:48:53.462 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.456+0000 7f5118b4a700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f511413aa50 con 0x7f51140a2080 2020-11-20T11:48:53.462 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.456+0000 7f5118b4a700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51140aad50 con 0x7f5114067a10 2020-11-20T11:48:53.462 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.457+0000 7f51137fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f51140a2080 0x7f51140aef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:53.463 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.457+0000 7f5112ffd700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f51140a2460 0x7f5114137df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:33132/0 (socket says 172.21.15.23:33132) 2020-11-20T11:48:53.463 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.457+0000 7f5112ffd700 1 -- 172.21.15.23:0/3187363537 learned_addr learned my addr 172.21.15.23:0/3187363537 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:48:53.463 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.457+0000 7f51137fe700 1 -- 172.21.15.23:0/3187363537 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5114067a10 msgr2=0x7f5114067e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:48:53.463 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.457+0000 7f5113fff700 1 --2- 172.21.15.23:0/3187363537 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5114067a10 0x7f5114067e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:53.463 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.457+0000 7f51137fe700 1 --2- 172.21.15.23:0/3187363537 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5114067a10 0x7f5114067e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:53.464 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.457+0000 7f51137fe700 1 -- 172.21.15.23:0/3187363537 >> v1:172.21.15.23:6789/0 conn(0x7f51140a2460 legacy=0x7f5114137df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:53.464 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.457+0000 7f51127fc700 1 -- 172.21.15.23:0/3187363537 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981698076 0 0) 0x7f51140aaab0 con 0x7f51140a2460 2020-11-20T11:48:53.464 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.457+0000 7f51137fe700 1 -- 172.21.15.23:0/3187363537 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f511413b920 con 0x7f51140a2080 2020-11-20T11:48:53.464 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.457+0000 7f51137fe700 1 --2- 172.21.15.23:0/3187363537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51140a2080 0x7f51140aef10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f510400b9d0 tx=0x7f5104006dd0).ready entity=mon.0 client_cookie=c22fbd338fa16310 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:53.464 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.457+0000 7f51127fc700 1 -- 172.21.15.23:0/3187363537 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5104006030 con 0x7f51140a2080 2020-11-20T11:48:53.465 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.457+0000 7f51127fc700 1 -- 172.21.15.23:0/3187363537 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5104006190 con 0x7f51140a2080 2020-11-20T11:48:53.465 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.457+0000 7f51127fc700 1 -- 172.21.15.23:0/3187363537 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5104006510 con 0x7f51140a2080 2020-11-20T11:48:53.465 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.458+0000 7f5118b4a700 1 -- 172.21.15.23:0/3187363537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51140a2080 msgr2=0x7f51140aef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:53.465 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.458+0000 7f5118b4a700 1 --2- 172.21.15.23:0/3187363537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51140a2080 0x7f51140aef10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f510400b9d0 tx=0x7f5104006dd0).stop 2020-11-20T11:48:53.465 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.458+0000 7f5118b4a700 1 -- 172.21.15.23:0/3187363537 shutdown_connections 2020-11-20T11:48:53.466 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.458+0000 7f5118b4a700 1 --2- 172.21.15.23:0/3187363537 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5114067a10 0x7f5114067e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:53.466 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.458+0000 7f5118b4a700 1 --2- 172.21.15.23:0/3187363537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51140a2080 0x7f51140aef10 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:53.466 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.458+0000 7f5118b4a700 1 -- 172.21.15.23:0/3187363537 >> 172.21.15.23:0/3187363537 conn(0x7f5114063980 msgr2=0x7f5114063d90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:48:53.466 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.458+0000 7f5118b4a700 1 -- 172.21.15.23:0/3187363537 shutdown_connections 2020-11-20T11:48:53.466 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.458+0000 7f5118b4a700 1 -- 172.21.15.23:0/3187363537 wait complete. 2020-11-20T11:48:53.469 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.460+0000 7f5118b4a700 1 Processor -- start 2020-11-20T11:48:53.469 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.460+0000 7f5118b4a700 1 -- start start 2020-11-20T11:48:53.469 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.460+0000 7f5118b4a700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5114067a10 0x7f511413cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:53.470 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.461+0000 7f5118b4a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51140a2080 0x7f51140a85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:53.470 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.461+0000 7f5118b4a700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f51140a2460 0x7f511414d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:53.470 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.461+0000 7f5118b4a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51140a7c80 con 0x7f51140a2080 2020-11-20T11:48:53.470 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.461+0000 7f5118b4a700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51140a72a0 con 0x7f5114067a10 2020-11-20T11:48:53.470 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.461+0000 7f5118b4a700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51140a7540 con 0x7f51140a2460 2020-11-20T11:48:53.471 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.461+0000 7f5112ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51140a2080 0x7f51140a85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:53.471 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.461+0000 7f5112ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51140a2080 0x7f51140a85c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47122/0 (socket says 172.21.15.23:47122) 2020-11-20T11:48:53.471 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.461+0000 7f5112ffd700 1 -- 172.21.15.23:0/2277985686 learned_addr learned my addr 172.21.15.23:0/2277985686 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:48:53.471 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.461+0000 7f5113fff700 1 --2- 172.21.15.23:0/2277985686 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f51140a2460 0x7f511414d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:53.471 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.461+0000 7f51137fe700 1 --2- 172.21.15.23:0/2277985686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5114067a10 0x7f511413cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:53.472 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.461+0000 7f5112ffd700 1 -- 172.21.15.23:0/2277985686 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f51140a2460 msgr2=0x7f511414d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:53.472 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.461+0000 7f5112ffd700 1 --2- 172.21.15.23:0/2277985686 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f51140a2460 0x7f511414d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:53.472 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.461+0000 7f5112ffd700 1 -- 172.21.15.23:0/2277985686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5114067a10 msgr2=0x7f511413cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:53.472 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.461+0000 7f5112ffd700 1 --2- 172.21.15.23:0/2277985686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5114067a10 0x7f511413cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:53.472 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.461+0000 7f5112ffd700 1 -- 172.21.15.23:0/2277985686 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51140a77e0 con 0x7f51140a2080 2020-11-20T11:48:53.473 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.461+0000 7f51137fe700 1 --2- 172.21.15.23:0/2277985686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5114067a10 0x7f511413cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:48:53.473 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.461+0000 7f5112ffd700 1 --2- 172.21.15.23:0/2277985686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51140a2080 0x7f51140a85c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f510800c760 tx=0x7f5108007d20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:53.473 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.461+0000 7f50f37fe700 1 -- 172.21.15.23:0/2277985686 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51080072e0 con 0x7f51140a2080 2020-11-20T11:48:53.473 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.461+0000 7f50f37fe700 1 -- 172.21.15.23:0/2277985686 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5108007440 con 0x7f51140a2080 2020-11-20T11:48:53.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.461+0000 7f5118b4a700 1 -- 172.21.15.23:0/2277985686 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51140a6f90 con 0x7f51140a2080 2020-11-20T11:48:53.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.461+0000 7f50f37fe700 1 -- 172.21.15.23:0/2277985686 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5108005840 con 0x7f51140a2080 2020-11-20T11:48:53.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.461+0000 7f5118b4a700 1 -- 172.21.15.23:0/2277985686 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f511414d7a0 con 0x7f51140a2080 2020-11-20T11:48:53.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.463+0000 7f5118b4a700 1 -- 172.21.15.23:0/2277985686 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f511414da50 con 0x7f51140a2080 2020-11-20T11:48:53.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.463+0000 7f50f37fe700 1 -- 172.21.15.23:0/2277985686 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5108010070 con 0x7f51140a2080 2020-11-20T11:48:53.474 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.463+0000 7f50f37fe700 1 --2- 172.21.15.23:0/2277985686 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f50ec056d60 0x7f50ec059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:48:53.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.463+0000 7f50f37fe700 1 -- 172.21.15.23:0/2277985686 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f510806f730 con 0x7f51140a2080 2020-11-20T11:48:53.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.463+0000 7f51137fe700 1 --2- 172.21.15.23:0/2277985686 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f50ec056d60 0x7f50ec059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:48:53.475 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.463+0000 7f51137fe700 1 --2- 172.21.15.23:0/2277985686 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f50ec056d60 0x7f50ec059220 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f510400b9d0 tx=0x7f5104015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:48:53.476 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.467+0000 7f50f37fe700 1 -- 172.21.15.23:0/2277985686 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f5108044a20 con 0x7f51140a2080 2020-11-20T11:48:53.478 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:53.433+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 308 slow ops, oldest is osd_op(client.4659.0:69551 4.2s0 4:46c0f236:::1000000036d.00000017:head [write 1572864~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:53.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.753+0000 7f5118b4a700 1 -- 172.21.15.23:0/2277985686 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f5114057dc0 con 0x7f51140a2080 2020-11-20T11:48:53.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.755+0000 7f50f37fe700 1 -- 172.21.15.23:0/2277985686 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f5108048040 con 0x7f51140a2080 2020-11-20T11:48:53.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.761+0000 7f5118b4a700 1 -- 172.21.15.23:0/2277985686 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f50ec056d60 msgr2=0x7f50ec059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:53.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.761+0000 7f5118b4a700 1 --2- 172.21.15.23:0/2277985686 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f50ec056d60 0x7f50ec059220 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f510400b9d0 tx=0x7f5104015000).stop 2020-11-20T11:48:53.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.761+0000 7f5118b4a700 1 -- 172.21.15.23:0/2277985686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51140a2080 msgr2=0x7f51140a85c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:48:53.765 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.761+0000 7f5118b4a700 1 --2- 172.21.15.23:0/2277985686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51140a2080 0x7f51140a85c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f510800c760 tx=0x7f5108007d20).stop 2020-11-20T11:48:53.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.762+0000 7f5118b4a700 1 -- 172.21.15.23:0/2277985686 shutdown_connections 2020-11-20T11:48:53.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.762+0000 7f5118b4a700 1 --2- 172.21.15.23:0/2277985686 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f50ec056d60 0x7f50ec059220 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:53.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.762+0000 7f5118b4a700 1 --2- 172.21.15.23:0/2277985686 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f51140a2460 0x7f511414d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:53.766 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.762+0000 7f5118b4a700 1 --2- 172.21.15.23:0/2277985686 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5114067a10 0x7f511413cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:53.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.762+0000 7f5118b4a700 1 --2- 172.21.15.23:0/2277985686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51140a2080 0x7f51140a85c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:48:53.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.762+0000 7f5118b4a700 1 -- 172.21.15.23:0/2277985686 >> 172.21.15.23:0/2277985686 conn(0x7f5114063980 msgr2=0x7f5114068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:48:53.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.762+0000 7f5118b4a700 1 -- 172.21.15.23:0/2277985686 shutdown_connections 2020-11-20T11:48:53.767 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:48:53.762+0000 7f5118b4a700 1 -- 172.21.15.23:0/2277985686 wait complete. 2020-11-20T11:48:53.780 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2020-11-20T11:48:54.531 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:54.509+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 307 slow ops, oldest is osd_op(client.4659.0:69736 4.2s0 4:46210aa0:::10000000353.00000002:head [write 1253376~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:55.549 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:55.516+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 304 slow ops, oldest is osd_op(client.4659.0:70096 4.2s0 4:5a9d432b:::1000000036b.0000000c:head [write 2203648~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:56.164 DEBUG:teuthology.parallel:result is None 2020-11-20T11:48:56.166 INFO:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2020-11-20T11:48:56.201 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2020-11-20T11:48:56.202 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2020-11-20T11:48:56.209 INFO:tasks.mon_thrash:joining mon_thrasher 2020-11-20T11:48:56.534 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:56.496+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 303 slow ops, oldest is osd_op(client.4659.0:70096 4.2s0 4:5a9d432b:::1000000036b.0000000c:head [write 2203648~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:57.536 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:57.519+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4659.0:70096 4.2s0 4:5a9d432b:::1000000036b.0000000c:head [write 2203648~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:58.601 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:58.557+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 298 slow ops, oldest is osd_op(client.4659.0:70096 4.2s0 4:5a9d432b:::1000000036b.0000000c:head [write 2203648~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:48:59.643 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:48:59.623+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 296 slow ops, oldest is osd_op(client.4659.0:70487 4.2s0 4:5da044e0:::10000000364.00000018:head [write 491520~4096 in=4096b,write 675840~4096 in=4096b,write 1835008~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:00.752 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:00.718+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 293 slow ops, oldest is osd_op(client.4659.0:70856 4.2s0 4:5b92f7a6:::10000000364.00000017:head [write 2580480~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:01.822 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:01.791+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 298 slow ops, oldest is osd_op(client.4659.0:70856 4.2s0 4:5b92f7a6:::10000000364.00000017:head [write 2580480~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:02.860 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:02.826+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 298 slow ops, oldest is osd_op(client.4659.0:70856 4.2s0 4:5b92f7a6:::10000000364.00000017:head [write 2580480~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:03.781 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-11-20T11:49:03.886 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:03.847+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 299 slow ops, oldest is osd_op(client.4659.0:71057 4.2s0 4:514d7452:::1000000034b.00000000:head [write 2154496~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:03.930 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.927+0000 7f08562cf700 1 Processor -- start 2020-11-20T11:49:03.930 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.927+0000 7f08562cf700 1 -- start start 2020-11-20T11:49:03.932 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.928+0000 7f08562cf700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f08500a6e10 0x7f0850061850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:49:03.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.928+0000 7f08562cf700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0850137870 0x7f085013ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:49:03.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.928+0000 7f08562cf700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0850062ad0 con 0x7f0850137410 2020-11-20T11:49:03.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.928+0000 7f08562cf700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08500a61f0 con 0x7f0850137870 2020-11-20T11:49:03.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.928+0000 7f08562cf700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08500a63e0 con 0x7f08500a6e10 2020-11-20T11:49:03.934 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.928+0000 7f085486c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0850137870 0x7f085013ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:49:03.934 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.928+0000 7f085486c700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f0850137870 0x7f085013ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47130/0 (socket says 172.21.15.23:47130) 2020-11-20T11:49:03.934 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.928+0000 7f085486c700 1 -- 172.21.15.23:0/357537647 learned_addr learned my addr 172.21.15.23:0/357537647 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:49:03.934 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.928+0000 7f084ffff700 1 --2- 172.21.15.23:0/357537647 >> v2:172.21.15.155:3300/0 conn(0x7f08500a6e10 0x7f0850061850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:49:03.934 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.928+0000 7f085486c700 1 -- 172.21.15.23:0/357537647 >> v2:172.21.15.155:3300/0 conn(0x7f08500a6e10 msgr2=0x7f0850061850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:49:03.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.928+0000 7f085486c700 1 --2- 172.21.15.23:0/357537647 >> v2:172.21.15.155:3300/0 conn(0x7f08500a6e10 0x7f0850061850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:49:03.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.928+0000 7f085486c700 1 -- 172.21.15.23:0/357537647 >> v1:172.21.15.155:6789/0 conn(0x7f0850137410 legacy=0x7f085013a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:49:03.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.928+0000 7f085486c700 1 -- 172.21.15.23:0/357537647 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08500a6680 con 0x7f0850137870 2020-11-20T11:49:03.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.928+0000 7f085486c700 1 --2- 172.21.15.23:0/357537647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0850137870 0x7f085013ae70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f08500aded0 tx=0x7f08480092d0).ready entity=mon.0 client_cookie=b99921c1828c9200 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:49:03.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.928+0000 7f084effd700 1 -- 172.21.15.23:0/357537647 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f084800bbf0 con 0x7f0850137870 2020-11-20T11:49:03.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.929+0000 7f084effd700 1 -- 172.21.15.23:0/357537647 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f084800bd50 con 0x7f0850137870 2020-11-20T11:49:03.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.929+0000 7f084effd700 1 -- 172.21.15.23:0/357537647 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0848010680 con 0x7f0850137870 2020-11-20T11:49:03.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.929+0000 7f08562cf700 1 -- 172.21.15.23:0/357537647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0850137870 msgr2=0x7f085013ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:49:03.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.929+0000 7f08562cf700 1 --2- 172.21.15.23:0/357537647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0850137870 0x7f085013ae70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f08500aded0 tx=0x7f08480092d0).stop 2020-11-20T11:49:03.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.929+0000 7f08562cf700 1 -- 172.21.15.23:0/357537647 shutdown_connections 2020-11-20T11:49:03.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.929+0000 7f08562cf700 1 --2- 172.21.15.23:0/357537647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0850137870 0x7f085013ae70 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:49:03.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.929+0000 7f08562cf700 1 --2- 172.21.15.23:0/357537647 >> v2:172.21.15.155:3300/0 conn(0x7f08500a6e10 0x7f0850061850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:49:03.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.929+0000 7f08562cf700 1 -- 172.21.15.23:0/357537647 >> 172.21.15.23:0/357537647 conn(0x7f0850060c80 msgr2=0x7f08500acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:49:03.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.929+0000 7f08562cf700 1 -- 172.21.15.23:0/357537647 shutdown_connections 2020-11-20T11:49:03.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.929+0000 7f08562cf700 1 -- 172.21.15.23:0/357537647 wait complete. 2020-11-20T11:49:03.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.930+0000 7f08562cf700 1 Processor -- start 2020-11-20T11:49:03.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.930+0000 7f08562cf700 1 -- start start 2020-11-20T11:49:03.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.931+0000 7f08562cf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08500a6e10 0x7f0850067870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:49:03.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.931+0000 7f08562cf700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0850137410 0x7f085009e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:49:03.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.931+0000 7f08562cf700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0850137870 0x7f08501339b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:49:03.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.931+0000 7f08562cf700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f085009dbb0 con 0x7f08500a6e10 2020-11-20T11:49:03.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.931+0000 7f08562cf700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0850134f80 con 0x7f0850137410 2020-11-20T11:49:03.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.931+0000 7f08562cf700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08500acd20 con 0x7f0850137870 2020-11-20T11:49:03.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.931+0000 7f084ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08500a6e10 0x7f0850067870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:49:03.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.931+0000 7f084ffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08500a6e10 0x7f0850067870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47134/0 (socket says 172.21.15.23:47134) 2020-11-20T11:49:03.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.931+0000 7f084ffff700 1 -- 172.21.15.23:0/2719646165 learned_addr learned my addr 172.21.15.23:0/2719646165 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:49:03.941 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.931+0000 7f084f7fe700 1 --2- 172.21.15.23:0/2719646165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0850137410 0x7f085009e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:49:03.942 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.931+0000 7f085486c700 1 --2- 172.21.15.23:0/2719646165 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0850137870 0x7f08501339b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:49:03.942 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.931+0000 7f084ffff700 1 -- 172.21.15.23:0/2719646165 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0850137870 msgr2=0x7f08501339b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:49:03.942 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.931+0000 7f084ffff700 1 --2- 172.21.15.23:0/2719646165 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0850137870 0x7f08501339b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:49:03.942 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.931+0000 7f084ffff700 1 -- 172.21.15.23:0/2719646165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0850137410 msgr2=0x7f085009e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:49:03.942 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.931+0000 7f084ffff700 1 --2- 172.21.15.23:0/2719646165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0850137410 0x7f085009e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:49:03.943 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.931+0000 7f084ffff700 1 -- 172.21.15.23:0/2719646165 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08500acfc0 con 0x7f08500a6e10 2020-11-20T11:49:03.943 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.931+0000 7f084ffff700 1 --2- 172.21.15.23:0/2719646165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08500a6e10 0x7f0850067870 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f084000db30 tx=0x7f0840008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:49:03.943 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.933+0000 7f082ffff700 1 -- 172.21.15.23:0/2719646165 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f084000de40 con 0x7f08500a6e10 2020-11-20T11:49:03.943 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.933+0000 7f082ffff700 1 -- 172.21.15.23:0/2719646165 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08400097c0 con 0x7f08500a6e10 2020-11-20T11:49:03.943 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.933+0000 7f082ffff700 1 -- 172.21.15.23:0/2719646165 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0840009b40 con 0x7f08500a6e10 2020-11-20T11:49:03.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.935+0000 7f08562cf700 1 -- 172.21.15.23:0/2719646165 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08500ad260 con 0x7f08500a6e10 2020-11-20T11:49:03.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.935+0000 7f08562cf700 1 -- 172.21.15.23:0/2719646165 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0850133ea0 con 0x7f08500a6e10 2020-11-20T11:49:03.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.936+0000 7f08562cf700 1 -- 172.21.15.23:0/2719646165 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f085005c520 con 0x7f08500a6e10 2020-11-20T11:49:03.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.945+0000 7f082ffff700 1 -- 172.21.15.23:0/2719646165 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f0840013480 con 0x7f08500a6e10 2020-11-20T11:49:03.949 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.946+0000 7f082ffff700 1 --2- 172.21.15.23:0/2719646165 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0828056d90 0x7f0828059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:49:03.949 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.946+0000 7f082ffff700 1 -- 172.21.15.23:0/2719646165 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f0840070d60 con 0x7f08500a6e10 2020-11-20T11:49:03.949 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.946+0000 7f082ffff700 1 -- 172.21.15.23:0/2719646165 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f0840071150 con 0x7f08500a6e10 2020-11-20T11:49:03.950 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.946+0000 7f084f7fe700 1 --2- 172.21.15.23:0/2719646165 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0828056d90 0x7f0828059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:49:03.950 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:03.947+0000 7f084f7fe700 1 --2- 172.21.15.23:0/2719646165 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0828056d90 0x7f0828059250 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f0844007fd0 tx=0x7f0844007bd0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:49:04.232 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.229+0000 7f08562cf700 1 -- 172.21.15.23:0/2719646165 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0850044710 con 0x7f08500a6e10 2020-11-20T11:49:04.233 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.230+0000 7f082ffff700 1 -- 172.21.15.23:0/2719646165 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+2128 (secure 0 0 0) 0x7f0840045f60 con 0x7f08500a6e10 2020-11-20T11:49:04.233 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:49:04.234 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4223,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.155:6835/564012815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6834","nonce":564012815},{"type":"v1","addr":"172.21.15.155:6835","nonce":564012815}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0,"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-11-20T11:32:25.469149+0000","modified":"2020-11-20T11:32:28.118344+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.23:6833/919963501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":919963501},{"type":"v1","addr":"172.21.15.23:6833","nonce":919963501}]},"join_fscid":-1,"export_targets":[],"features":4540138297136906239,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-11-20T11:49:04.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.231+0000 7f08562cf700 1 -- 172.21.15.23:0/2719646165 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0828056d90 msgr2=0x7f0828059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:49:04.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.231+0000 7f08562cf700 1 --2- 172.21.15.23:0/2719646165 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0828056d90 0x7f0828059250 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f0844007fd0 tx=0x7f0844007bd0).stop 2020-11-20T11:49:04.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.231+0000 7f08562cf700 1 -- 172.21.15.23:0/2719646165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08500a6e10 msgr2=0x7f0850067870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:49:04.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.231+0000 7f08562cf700 1 --2- 172.21.15.23:0/2719646165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08500a6e10 0x7f0850067870 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f084000db30 tx=0x7f0840008040).stop 2020-11-20T11:49:04.235 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.231+0000 7f08562cf700 1 -- 172.21.15.23:0/2719646165 shutdown_connections 2020-11-20T11:49:04.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.231+0000 7f08562cf700 1 --2- 172.21.15.23:0/2719646165 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0828056d90 0x7f0828059250 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:49:04.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.231+0000 7f08562cf700 1 --2- 172.21.15.23:0/2719646165 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0850137870 0x7f08501339b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:49:04.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.231+0000 7f08562cf700 1 --2- 172.21.15.23:0/2719646165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f08500a6e10 0x7f0850067870 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:49:04.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.231+0000 7f08562cf700 1 --2- 172.21.15.23:0/2719646165 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0850137410 0x7f085009e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:49:04.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.231+0000 7f08562cf700 1 -- 172.21.15.23:0/2719646165 >> 172.21.15.23:0/2719646165 conn(0x7f0850060c80 msgr2=0x7f08500a97d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:49:04.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.231+0000 7f08562cf700 1 -- 172.21.15.23:0/2719646165 shutdown_connections 2020-11-20T11:49:04.237 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.231+0000 7f08562cf700 1 -- 172.21.15.23:0/2719646165 wait complete. 2020-11-20T11:49:04.237 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 7 2020-11-20T11:49:04.246 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2020-11-20T11:49:04.246 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2020-11-20T11:49:04.451 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.395+0000 7f2021920700 1 Processor -- start 2020-11-20T11:49:04.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.395+0000 7f2021920700 1 -- start start 2020-11-20T11:49:04.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.396+0000 7f2021920700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f201c064d60 0x7f201c0a7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:49:04.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.396+0000 7f2021920700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f201c065140 0x7f201c0a8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:49:04.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.396+0000 7f2021920700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f201c0a92e0 con 0x7f201c0a8e50 2020-11-20T11:49:04.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.396+0000 7f2021920700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f201c0a7f80 con 0x7f201c065140 2020-11-20T11:49:04.453 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.396+0000 7f2021920700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f201c0a8220 con 0x7f201c064d60 2020-11-20T11:49:04.453 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.396+0000 7f201a7fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f201c065140 0x7f201c0a8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:49:04.453 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.396+0000 7f201a7fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f201c065140 0x7f201c0a8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47142/0 (socket says 172.21.15.23:47142) 2020-11-20T11:49:04.453 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.396+0000 7f201a7fc700 1 -- 172.21.15.23:0/2398929083 learned_addr learned my addr 172.21.15.23:0/2398929083 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:49:04.454 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.396+0000 7f201affd700 1 --2- 172.21.15.23:0/2398929083 >> v2:172.21.15.155:3300/0 conn(0x7f201c064d60 0x7f201c0a7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:49:04.454 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.396+0000 7f2019ffb700 1 -- 172.21.15.23:0/2398929083 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3200255397 0 0) 0x7f201c0a92e0 con 0x7f201c0a8e50 2020-11-20T11:49:04.454 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.447+0000 7f2019ffb700 1 -- 172.21.15.23:0/2398929083 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2008003280 con 0x7f201c0a8e50 2020-11-20T11:49:04.454 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.447+0000 7f201a7fc700 1 -- 172.21.15.23:0/2398929083 >> v2:172.21.15.155:3300/0 conn(0x7f201c064d60 msgr2=0x7f201c0a7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:49:04.454 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.447+0000 7f201a7fc700 1 --2- 172.21.15.23:0/2398929083 >> v2:172.21.15.155:3300/0 conn(0x7f201c064d60 0x7f201c0a7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:49:04.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.447+0000 7f201a7fc700 1 -- 172.21.15.23:0/2398929083 >> v1:172.21.15.155:6789/0 conn(0x7f201c0a8e50 legacy=0x7f201c066770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:49:04.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.447+0000 7f201a7fc700 1 -- 172.21.15.23:0/2398929083 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f201c0a84c0 con 0x7f201c065140 2020-11-20T11:49:04.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.447+0000 7f201a7fc700 1 --2- 172.21.15.23:0/2398929083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f201c065140 0x7f201c0a8960 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f201c068270 tx=0x7f200c006ef0).ready entity=mon.0 client_cookie=581894778b157201 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:49:04.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.447+0000 7f2019ffb700 1 -- 172.21.15.23:0/2398929083 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200c00b680 con 0x7f201c065140 2020-11-20T11:49:04.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.447+0000 7f2019ffb700 1 -- 172.21.15.23:0/2398929083 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f200c000e90 con 0x7f201c065140 2020-11-20T11:49:04.456 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.447+0000 7f2019ffb700 1 -- 172.21.15.23:0/2398929083 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200c00b9f0 con 0x7f201c065140 2020-11-20T11:49:04.456 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.448+0000 7f2021920700 1 -- 172.21.15.23:0/2398929083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f201c065140 msgr2=0x7f201c0a8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:49:04.456 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.448+0000 7f2021920700 1 --2- 172.21.15.23:0/2398929083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f201c065140 0x7f201c0a8960 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f201c068270 tx=0x7f200c006ef0).stop 2020-11-20T11:49:04.456 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.448+0000 7f2021920700 1 -- 172.21.15.23:0/2398929083 shutdown_connections 2020-11-20T11:49:04.456 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.448+0000 7f2021920700 1 --2- 172.21.15.23:0/2398929083 >> v2:172.21.15.155:3300/0 conn(0x7f201c064d60 0x7f201c0a7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:49:04.457 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.448+0000 7f2021920700 1 --2- 172.21.15.23:0/2398929083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f201c065140 0x7f201c0a8960 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:49:04.457 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.448+0000 7f2021920700 1 -- 172.21.15.23:0/2398929083 >> 172.21.15.23:0/2398929083 conn(0x7f201c0601b0 msgr2=0x7f201c0605c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:49:04.457 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.448+0000 7f2021920700 1 -- 172.21.15.23:0/2398929083 shutdown_connections 2020-11-20T11:49:04.457 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.448+0000 7f2021920700 1 -- 172.21.15.23:0/2398929083 wait complete. 2020-11-20T11:49:04.460 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.450+0000 7f2021920700 1 Processor -- start 2020-11-20T11:49:04.460 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.450+0000 7f2021920700 1 -- start start 2020-11-20T11:49:04.460 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.451+0000 7f2021920700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f201c064d60 0x7f201c0a2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:49:04.461 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.451+0000 7f2021920700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f201c065140 0x7f201c09dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:49:04.461 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.451+0000 7f2021920700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f201c0a8e50 0x7f201c0a5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:49:04.461 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.451+0000 7f2021920700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f201c09d4a0 con 0x7f201c0a8e50 2020-11-20T11:49:04.461 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.451+0000 7f2021920700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f201c0a0630 con 0x7f201c064d60 2020-11-20T11:49:04.461 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.451+0000 7f2021920700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f201c09d700 con 0x7f201c065140 2020-11-20T11:49:04.462 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.451+0000 7f201b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f201c0a8e50 0x7f201c0a5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:49:04.462 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.451+0000 7f201b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f201c0a8e50 0x7f201c0a5d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47148/0 (socket says 172.21.15.23:47148) 2020-11-20T11:49:04.462 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.451+0000 7f201b7fe700 1 -- 172.21.15.23:0/2516936668 learned_addr learned my addr 172.21.15.23:0/2516936668 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:49:04.462 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.451+0000 7f201affd700 1 --2- 172.21.15.23:0/2516936668 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f201c064d60 0x7f201c0a2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:49:04.462 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.451+0000 7f201a7fc700 1 --2- 172.21.15.23:0/2516936668 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f201c065140 0x7f201c09dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:49:04.463 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.451+0000 7f201b7fe700 1 -- 172.21.15.23:0/2516936668 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f201c065140 msgr2=0x7f201c09dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:49:04.463 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.451+0000 7f201b7fe700 1 --2- 172.21.15.23:0/2516936668 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f201c065140 0x7f201c09dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:49:04.463 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.451+0000 7f201b7fe700 1 -- 172.21.15.23:0/2516936668 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f201c064d60 msgr2=0x7f201c0a2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:49:04.463 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.451+0000 7f201b7fe700 1 --2- 172.21.15.23:0/2516936668 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f201c064d60 0x7f201c0a2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:49:04.463 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.451+0000 7f201b7fe700 1 -- 172.21.15.23:0/2516936668 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f201c09f940 con 0x7f201c0a8e50 2020-11-20T11:49:04.464 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.451+0000 7f201affd700 1 --2- 172.21.15.23:0/2516936668 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f201c064d60 0x7f201c0a2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:49:04.464 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.452+0000 7f201b7fe700 1 --2- 172.21.15.23:0/2516936668 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f201c0a8e50 0x7f201c0a5d80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f201400ca60 tx=0x7f2014007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:49:04.464 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.452+0000 7f2012ffd700 1 -- 172.21.15.23:0/2516936668 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201400ce40 con 0x7f201c0a8e50 2020-11-20T11:49:04.464 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.452+0000 7f2012ffd700 1 -- 172.21.15.23:0/2516936668 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2014002240 con 0x7f201c0a8e50 2020-11-20T11:49:04.464 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.452+0000 7f2021920700 1 -- 172.21.15.23:0/2516936668 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f201c09fbf0 con 0x7f201c0a8e50 2020-11-20T11:49:04.465 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.452+0000 7f2012ffd700 1 -- 172.21.15.23:0/2516936668 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2014001590 con 0x7f201c0a8e50 2020-11-20T11:49:04.465 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.452+0000 7f2021920700 1 -- 172.21.15.23:0/2516936668 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f201c0a0020 con 0x7f201c0a8e50 2020-11-20T11:49:04.465 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.453+0000 7f2021920700 1 -- 172.21.15.23:0/2516936668 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f201c061650 con 0x7f201c0a8e50 2020-11-20T11:49:04.465 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.453+0000 7f2012ffd700 1 -- 172.21.15.23:0/2516936668 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f2014019030 con 0x7f201c0a8e50 2020-11-20T11:49:04.466 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.454+0000 7f2012ffd700 1 --2- 172.21.15.23:0/2516936668 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1ff8056d60 0x7f1ff8059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:49:04.466 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.454+0000 7f2012ffd700 1 -- 172.21.15.23:0/2516936668 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f201406f4f0 con 0x7f201c0a8e50 2020-11-20T11:49:04.466 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.459+0000 7f201affd700 1 --2- 172.21.15.23:0/2516936668 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1ff8056d60 0x7f1ff8059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:49:04.467 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.459+0000 7f201affd700 1 --2- 172.21.15.23:0/2516936668 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1ff8056d60 0x7f1ff8059220 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f200400c560 tx=0x7f2004010040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:49:04.468 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.465+0000 7f2012ffd700 1 -- 172.21.15.23:0/2516936668 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f2014043fd0 con 0x7f201c0a8e50 2020-11-20T11:49:04.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.752+0000 7f2021920700 1 -- 172.21.15.23:0/2516936668 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f201c09d8c0 con 0x7f201c0a8e50 2020-11-20T11:49:04.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.752+0000 7f2012ffd700 1 -- 172.21.15.23:0/2516936668 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1479 (secure 0 0 0) 0x7f2014044890 con 0x7f201c0a8e50 2020-11-20T11:49:04.756 INFO:teuthology.orchestra.run.smithi023.stdout:{"election_epoch":158,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","modified":"2020-11-20T11:31:30.112250Z","created":"2020-11-20T11:31:30.112250Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3300","nonce":0},{"type":"v1","addr":"172.21.15.155:6789","nonce":0}]},"addr":"172.21.15.155:6789/0","public_addr":"172.21.15.155:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:3301","nonce":0},{"type":"v1","addr":"172.21.15.155:6790","nonce":0}]},"addr":"172.21.15.155:6790/0","public_addr":"172.21.15.155:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2020-11-20T11:49:04.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.755+0000 7f2021920700 1 -- 172.21.15.23:0/2516936668 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1ff8056d60 msgr2=0x7f1ff8059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:49:04.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.755+0000 7f2021920700 1 --2- 172.21.15.23:0/2516936668 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1ff8056d60 0x7f1ff8059220 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f200400c560 tx=0x7f2004010040).stop 2020-11-20T11:49:04.758 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.755+0000 7f2021920700 1 -- 172.21.15.23:0/2516936668 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f201c0a8e50 msgr2=0x7f201c0a5d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:49:04.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.755+0000 7f2021920700 1 --2- 172.21.15.23:0/2516936668 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f201c0a8e50 0x7f201c0a5d80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f201400ca60 tx=0x7f2014007e10).stop 2020-11-20T11:49:04.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.755+0000 7f2021920700 1 -- 172.21.15.23:0/2516936668 shutdown_connections 2020-11-20T11:49:04.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.755+0000 7f2021920700 1 --2- 172.21.15.23:0/2516936668 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f201c0a8e50 0x7f201c0a5d80 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:49:04.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.755+0000 7f2021920700 1 --2- 172.21.15.23:0/2516936668 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f201c064d60 0x7f201c0a2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:49:04.759 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.755+0000 7f2021920700 1 --2- 172.21.15.23:0/2516936668 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f1ff8056d60 0x7f1ff8059220 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:49:04.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.755+0000 7f2021920700 1 --2- 172.21.15.23:0/2516936668 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f201c065140 0x7f201c09dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:49:04.760 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.755+0000 7f2021920700 1 -- 172.21.15.23:0/2516936668 >> 172.21.15.23:0/2516936668 conn(0x7f201c0601b0 msgr2=0x7f201c14d840 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:49:04.763 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.761+0000 7f2021920700 1 -- 172.21.15.23:0/2516936668 shutdown_connections 2020-11-20T11:49:04.764 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:49:04.761+0000 7f2021920700 1 -- 172.21.15.23:0/2516936668 wait complete. 2020-11-20T11:49:04.786 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2020-11-20T11:49:04.787 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2020-11-20T11:49:04.859 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2020-11-20T11:49:04.860 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi023.front.sepia.ceph.com... 2020-11-20T11:49:04.860 INFO:teuthology.orchestra.run:Running command with timeout 1800 2020-11-20T11:49:04.860 INFO:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2020-11-20T11:49:04.913 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:04.857+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 300 slow ops, oldest is osd_op(client.4659.0:71246 4.2s0 4:45b4350a:::10000000380.00000003:head [write 3457024~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:05.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi023.stderr:ceph-fuse[28386]: fuse finished with error 0 and tester_r 0 2020-11-20T11:49:05.180 INFO:teuthology.orchestra.run:waiting for 900 2020-11-20T11:49:05.923 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:05.871+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 298 slow ops, oldest is osd_op(client.4659.0:71366 4.2s0 4:5b92f7a6:::10000000364.00000017:head [write 3960832~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:06.991 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:06.955+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 295 slow ops, oldest is osd_op(client.4659.0:72018 4.2s0 4:48e3a637:::10000000378.00000008:head [write 2023424~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:08.060 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:08.011+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 295 slow ops, oldest is osd_op(client.4659.0:72018 4.2s0 4:48e3a637:::10000000378.00000008:head [write 2023424~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:09.026 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:08.975+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 299 slow ops, oldest is osd_op(client.4659.0:72018 4.2s0 4:48e3a637:::10000000378.00000008:head [write 2023424~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:10.094 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:10.055+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 298 slow ops, oldest is osd_op(client.4659.0:72201 4.2s0 4:5aaae2bc:::10000000358.00000006:head [write 843776~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:11.169 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:11.147+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 297 slow ops, oldest is osd_op(client.4659.0:72201 4.2s0 4:5aaae2bc:::10000000358.00000006:head [write 843776~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:12.275 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:12.238+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 294 slow ops, oldest is osd_op(client.4659.0:72476 4.2s0 4:5bc0f152:::100000003c2.00000013:head [write 1703936~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:13.365 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:13.318+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 293 slow ops, oldest is osd_op(client.4659.0:72476 4.2s0 4:5bc0f152:::100000003c2.00000013:head [write 1703936~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:14.418 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:14.376+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 290 slow ops, oldest is osd_op(client.4659.0:73926 4.2s0 4:5930cd20:::10000000378.00000005:head [write 2396160~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:15.446 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:15.392+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 295 slow ops, oldest is osd_op(client.4659.0:73926 4.2s0 4:5930cd20:::10000000378.00000005:head [write 2396160~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:16.471 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:16.452+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 294 slow ops, oldest is osd_op(client.4659.0:73926 4.2s0 4:5930cd20:::10000000378.00000005:head [write 2396160~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:17.547 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:17.514+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 293 slow ops, oldest is osd_op(client.4659.0:73926 4.2s0 4:5930cd20:::10000000378.00000005:head [write 2396160~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:18.601 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:18.564+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 292 slow ops, oldest is osd_op(client.4659.0:73926 4.2s0 4:5930cd20:::10000000378.00000005:head [write 2396160~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:19.629 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:19.599+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 294 slow ops, oldest is osd_op(client.4659.0:73926 4.2s0 4:5930cd20:::10000000378.00000005:head [write 2396160~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:20.656 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:20.626+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 292 slow ops, oldest is osd_op(client.4659.0:73926 4.2s0 4:5930cd20:::10000000378.00000005:head [write 2396160~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:21.686 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:21.666+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 291 slow ops, oldest is osd_op(client.4659.0:73926 4.2s0 4:5930cd20:::10000000378.00000005:head [write 2396160~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:22.742 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:22.722+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 287 slow ops, oldest is osd_op(client.4659.0:74033 4.2s0 4:46210aa0:::10000000353.00000002:head [write 2457600~4096 in=4096b,write 4132864~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:23.793 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:23.758+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 284 slow ops, oldest is osd_op(client.4659.0:74413 4.2s0 4:4c361bf0:::1000000034d.00000009:head [write 2224128~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:24.893 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:24.861+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 281 slow ops, oldest is osd_op(client.4659.0:74413 4.2s0 4:4c361bf0:::1000000034d.00000009:head [write 2224128~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:25.937 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:25.903+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 285 slow ops, oldest is osd_op(client.4659.0:74579 4.2s0 4:4ad665d9:::1000000039c.00000001:head [write 2392064~4096 in=4096b,write 3366912~4096 in=4096b,write 4100096~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:26.980 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:26.958+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 279 slow ops, oldest is osd_op(client.4659.0:74681 4.2s0 4:4ad665d9:::1000000039c.00000001:head [write 643072~4096 in=4096b,write 1269760~4096 in=4096b,write 1642496~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:28.040 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:28.019+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 276 slow ops, oldest is osd_op(client.4659.0:74966 4.2s0 4:520b1123:::1000000036d.00000014:head [write 557056~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:29.095 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:29.074+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 275 slow ops, oldest is osd_op(client.4659.0:74966 4.2s0 4:520b1123:::1000000036d.00000014:head [write 557056~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:30.123 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:30.060+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 272 slow ops, oldest is osd_op(client.4659.0:75123 4.2s0 4:4fac3541:::1000000038d.00000003:head [write 1740800~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:31.159 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:31.127+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 272 slow ops, oldest is osd_op(client.4659.0:75366 4.2s0 4:5aaae2bc:::10000000358.00000006:head [write 3678208~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:32.142 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:32.104+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 270 slow ops, oldest is osd_op(client.4659.0:75616 4.2s0 4:404eafb2:::10000000352.00000007:head [write 1216512~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:33.183 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:33.161+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 268 slow ops, oldest is osd_op(client.4659.0:75616 4.2s0 4:404eafb2:::10000000352.00000007:head [write 1216512~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:34.281 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:34.259+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 269 slow ops, oldest is osd_op(client.4659.0:75884 4.2s0 4:543d101f:::10000000388.00000009:head [write 196608~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:35.381 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:35.352+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 266 slow ops, oldest is osd_op(client.4659.0:76035 4.2s0 4:5c05d725:::1000000036d.00000011:head [write 3694592~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:36.459 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:36.423+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4659.0:76127 4.2s0 4:4258020c:::10000000356.0000000d:head [write 1339392~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:37.548 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:37.526+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 268 slow ops, oldest is osd_op(client.4659.0:76356 4.2s0 4:4fa3dc39:::100000003ab.00000003:head [write 3018752~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:38.546 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:38.502+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 267 slow ops, oldest is osd_op(client.4659.0:76501 4.2s0 4:5a5d3f90:::10000000378.0000000a:head [write 0~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:39.575 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:39.541+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 265 slow ops, oldest is osd_op(client.4659.0:76917 4.2s0 4:450d509f:::100000003ab.0000000a:head [write 1318912~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:40.663 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:40.597+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 264 slow ops, oldest is osd_op(client.4659.0:76917 4.2s0 4:450d509f:::100000003ab.0000000a:head [write 1318912~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:41.636 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:41.613+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 263 slow ops, oldest is osd_op(client.4659.0:76917 4.2s0 4:450d509f:::100000003ab.0000000a:head [write 1318912~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:42.729 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:42.694+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 257 slow ops, oldest is osd_op(client.4659.0:77080 4.2s0 4:4535290c:::100000003ab.00000010:head [write 2674688~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:43.746 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:43.706+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 255 slow ops, oldest is osd_op(client.4659.0:77334 4.2s0 4:4535290c:::100000003ab.00000010:head [write 495616~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:44.762 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:44.740+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 262 slow ops, oldest is osd_op(client.4659.0:77334 4.2s0 4:4535290c:::100000003ab.00000010:head [write 495616~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:45.845 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:45.823+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 260 slow ops, oldest is osd_op(client.4659.0:77673 4.2s0 4:515c1c8d:::10000000356.00000008:head [write 3145728~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:46.901 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:46.852+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 257 slow ops, oldest is osd_op(client.4659.0:77673 4.2s0 4:515c1c8d:::10000000356.00000008:head [write 3145728~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:47.891 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:47.869+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 255 slow ops, oldest is osd_op(client.4659.0:77970 4.2s0 4:5a9d432b:::1000000036b.0000000c:head [write 618496~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:48.926 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:48.903+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 252 slow ops, oldest is osd_op(client.4659.0:77970 4.2s0 4:5a9d432b:::1000000036b.0000000c:head [write 618496~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:50.017 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:49.994+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 251 slow ops, oldest is osd_op(client.4659.0:78487 4.2s0 4:41f43689:::1000000038d.00000000:head [write 1941504~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:51.027 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:51.003+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 250 slow ops, oldest is osd_op(client.4659.0:78487 4.2s0 4:41f43689:::1000000038d.00000000:head [write 1941504~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:52.110 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:52.076+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.4659.0:78487 4.2s0 4:41f43689:::1000000038d.00000000:head [write 1941504~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:53.149 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:53.112+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 246 slow ops, oldest is osd_op(client.4659.0:78487 4.2s0 4:41f43689:::1000000038d.00000000:head [write 1941504~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:54.163 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:54.145+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 241 slow ops, oldest is osd_op(client.4659.0:78806 4.2s0 4:4c96ca10:::1000000034b.0000000d:head [write 696320~4096 in=4096b,write 1622016~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:55.276 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:55.252+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 239 slow ops, oldest is osd_op(client.4659.0:78806 4.2s0 4:4c96ca10:::1000000034b.0000000d:head [write 696320~4096 in=4096b,write 1622016~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:56.377 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:56.335+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 240 slow ops, oldest is osd_op(client.4659.0:78998 4.2s0 4:5930cd20:::10000000378.00000005:head [write 2240512~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:57.377 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:57.318+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 236 slow ops, oldest is osd_op(client.4659.0:79121 4.2s0 4:5b719265:::1000000039e.0000000c:head [write 2863104~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:58.393 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:58.367+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 234 slow ops, oldest is osd_op(client.4659.0:79412 4.2s0 4:488e1ee8:::1000000039f.00000013:head [write 2568192~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:49:59.417 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:49:59.396+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.4659.0:79412 4.2s0 4:488e1ee8:::1000000039f.00000013:head [write 2568192~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:00.510 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:00.486+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 235 slow ops, oldest is osd_op(client.4659.0:79710 4.2s0 4:5bc0f152:::100000003c2.00000013:head [write 1404928~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:01.532 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:01.479+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 235 slow ops, oldest is osd_op(client.4659.0:79710 4.2s0 4:5bc0f152:::100000003c2.00000013:head [write 1404928~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:02.520 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:02.466+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 234 slow ops, oldest is osd_op(client.4659.0:80064 4.2s0 4:5e463c85:::100000003c3.00000004:head [write 1748992~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:03.599 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:03.529+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 228 slow ops, oldest is osd_op(client.4659.0:80064 4.2s0 4:5e463c85:::100000003c3.00000004:head [write 1748992~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:04.688 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:04.631+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 227 slow ops, oldest is osd_op(client.4659.0:80185 4.2s0 4:494c3c3b:::1000000034b.00000004:head [write 2686976~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:05.700 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:05.655+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 234 slow ops, oldest is osd_op(client.4659.0:80597 4.2s0 4:488e1ee8:::1000000039f.00000013:head [write 3522560~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:06.748 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:06.705+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 233 slow ops, oldest is osd_op(client.4659.0:80597 4.2s0 4:488e1ee8:::1000000039f.00000013:head [write 3522560~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:07.733 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:07.689+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 232 slow ops, oldest is osd_op(client.4659.0:80597 4.2s0 4:488e1ee8:::1000000039f.00000013:head [write 3522560~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:08.757 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:08.689+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 227 slow ops, oldest is osd_op(client.4659.0:81199 4.2s0 4:5c05d725:::1000000036d.00000011:head [write 4014080~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:09.744 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:09.706+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 226 slow ops, oldest is osd_op(client.4659.0:81199 4.2s0 4:5c05d725:::1000000036d.00000011:head [write 4014080~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:10.807 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:10.764+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 227 slow ops, oldest is osd_op(client.4659.0:81199 4.2s0 4:5c05d725:::1000000036d.00000011:head [write 4014080~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:11.846 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:11.803+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 223 slow ops, oldest is osd_op(client.4659.0:81199 4.2s0 4:5c05d725:::1000000036d.00000011:head [write 4014080~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:12.877 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:12.842+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 221 slow ops, oldest is osd_op(client.4659.0:81285 4.2s0 4:58da19fb:::10000000388.00000002:head [write 3768320~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:13.953 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:13.916+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 220 slow ops, oldest is osd_op(client.4659.0:81478 4.2s0 4:542fa72b:::100000003c2.00000000:head [write 3477504~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:14.979 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:14.956+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 224 slow ops, oldest is osd_op(client.4659.0:81965 4.2s0 4:436fdcda:::1000000036f.00000015:head [write 2060288~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:15.989 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:15.923+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 223 slow ops, oldest is osd_op(client.4659.0:81965 4.2s0 4:436fdcda:::1000000036f.00000015:head [write 2060288~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:17.029 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:16.989+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 222 slow ops, oldest is osd_op(client.4659.0:81965 4.2s0 4:436fdcda:::1000000036f.00000015:head [write 2060288~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:18.095 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:18.040+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.4659.0:82145 4.2s0 4:4258020c:::10000000356.0000000d:head [write 2433024~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:19.094 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:19.073+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 220 slow ops, oldest is osd_op(client.4659.0:82303 4.2s0 4:4535290c:::100000003ab.00000010:head [write 3182592~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:20.121 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:20.096+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 215 slow ops, oldest is osd_op(client.4659.0:82734 4.2s0 4:5050cdd3:::10000000378.00000009:head [write 901120~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:21.171 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:21.108+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 221 slow ops, oldest is osd_op(client.4659.0:82734 4.2s0 4:5050cdd3:::10000000378.00000009:head [write 901120~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:22.166 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:22.140+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 221 slow ops, oldest is osd_op(client.4659.0:82734 4.2s0 4:5050cdd3:::10000000378.00000009:head [write 901120~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:23.182 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:23.175+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.4659.0:82882 4.2s0 4:5fa1e09a:::1000000036b.00000002:head [write 593920~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:24.266 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:24.193+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 218 slow ops, oldest is osd_op(client.4659.0:82906 4.2s0 4:506fa6e5:::100000003ab.00000011:head [write 1454080~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:25.259 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:25.213+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 215 slow ops, oldest is osd_op(client.4659.0:83084 4.2s0 4:4ea2e919:::100000003a9.0000000b:head [write 380928~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:26.313 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:26.264+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 220 slow ops, oldest is osd_op(client.4659.0:83254 4.2s0 4:52feed57:::10000000356.00000011:head [write 2785280~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:27.364 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:27.340+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 220 slow ops, oldest is osd_op(client.4659.0:83860 4.2s0 4:4fa12522:::1000000036f.00000012:head [write 1585152~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:28.404 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:28.380+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 220 slow ops, oldest is osd_op(client.4659.0:83860 4.2s0 4:4fa12522:::1000000036f.00000012:head [write 1585152~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:29.457 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:29.418+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 219 slow ops, oldest is osd_op(client.4659.0:83860 4.2s0 4:4fa12522:::1000000036f.00000012:head [write 1585152~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:30.492 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:30.467+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 217 slow ops, oldest is osd_op(client.4659.0:84190 4.2s0 4:41f43689:::1000000038d.00000000:head [write 815104~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:31.513 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:31.487+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 215 slow ops, oldest is osd_op(client.4659.0:84190 4.2s0 4:41f43689:::1000000038d.00000000:head [write 815104~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:32.541 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:32.492+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 217 slow ops, oldest is osd_op(client.4659.0:84190 4.2s0 4:41f43689:::1000000038d.00000000:head [write 815104~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:33.613 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:33.548+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 215 slow ops, oldest is osd_op(client.4659.0:84355 4.2s0 4:4c5b03aa:::10000000369.00000003:head [write 475136~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:34.626 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:34.581+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.4659.0:84442 4.2s0 4:48e3a637:::10000000378.00000008:head [write 1609728~4096 in=4096b,write 3051520~4096 in=4096b,write 4169728~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:35.677 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:35.643+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 213 slow ops, oldest is osd_op(client.4659.0:84442 4.2s0 4:48e3a637:::10000000378.00000008:head [write 1609728~4096 in=4096b,write 3051520~4096 in=4096b,write 4169728~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:36.679 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:36.628+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 210 slow ops, oldest is osd_op(client.4659.0:85473 4.2s0 4:4c5b03aa:::10000000369.00000003:head [write 2134016~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:37.713 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:37.690+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 209 slow ops, oldest is osd_op(client.4659.0:85473 4.2s0 4:4c5b03aa:::10000000369.00000003:head [write 2134016~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:38.784 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:38.742+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 209 slow ops, oldest is osd_op(client.4659.0:85473 4.2s0 4:4c5b03aa:::10000000369.00000003:head [write 2134016~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:39.824 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:39.798+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 205 slow ops, oldest is osd_op(client.4659.0:85473 4.2s0 4:4c5b03aa:::10000000369.00000003:head [write 2134016~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:40.881 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:40.853+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 204 slow ops, oldest is osd_op(client.4659.0:85473 4.2s0 4:4c5b03aa:::10000000369.00000003:head [write 2134016~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:41.921 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:41.894+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 202 slow ops, oldest is osd_op(client.4659.0:85473 4.2s0 4:4c5b03aa:::10000000369.00000003:head [write 2134016~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:42.980 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:42.968+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 203 slow ops, oldest is osd_op(client.4659.0:85789 4.2s0 4:4385b3f0:::100000003ab.00000014:head [write 106496~4096 in=4096b,write 2646016~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:44.047 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:44.042+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 201 slow ops, oldest is osd_op(client.4659.0:85789 4.2s0 4:4385b3f0:::100000003ab.00000014:head [write 106496~4096 in=4096b,write 2646016~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:45.126 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:45.103+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 198 slow ops, oldest is osd_op(client.4659.0:85857 4.2s0 4:4dd551b2:::100000003b0.0000000a:head [write 3198976~4096 in=4096b,write 4055040~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:46.178 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:46.151+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4659.0:86623 4.2s0 4:4c6c3657:::10000000356.0000000e:head [write 2633728~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:47.223 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:47.195+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 195 slow ops, oldest is osd_op(client.4659.0:86623 4.2s0 4:4c6c3657:::10000000356.0000000e:head [write 2633728~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:48.310 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:48.282+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 193 slow ops, oldest is osd_op(client.4659.0:86623 4.2s0 4:4c6c3657:::10000000356.0000000e:head [write 2633728~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:49.333 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:49.278+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 194 slow ops, oldest is osd_op(client.4659.0:86623 4.2s0 4:4c6c3657:::10000000356.0000000e:head [write 2633728~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:50.433 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:50.406+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 193 slow ops, oldest is osd_op(client.4659.0:86829 4.2s0 4:4e36b6f3:::100000003b0.00000000:head [write 2371584~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:51.497 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:51.452+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 193 slow ops, oldest is osd_op(client.4659.0:87325 4.2s0 4:4c1fd321:::10000000352.00000013:head [write 2793472~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:52.484 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:52.473+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 193 slow ops, oldest is osd_op(client.4659.0:87325 4.2s0 4:4c1fd321:::10000000352.00000013:head [write 2793472~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:53.582 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:53.555+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 190 slow ops, oldest is osd_op(client.4659.0:87325 4.2s0 4:4c1fd321:::10000000352.00000013:head [write 2793472~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:54.644 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:54.599+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 185 slow ops, oldest is osd_op(client.4659.0:87617 4.2s0 4:4c5b03aa:::10000000369.00000003:head [write 3264512~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:55.694 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:55.653+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 182 slow ops, oldest is osd_op(client.4659.0:87658 4.2s0 4:543d101f:::10000000388.00000009:head [write 3137536~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:56.767 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:56.723+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 178 slow ops, oldest is osd_op(client.4659.0:87904 4.2s0 4:5b26cd0c:::100000003b9.00000017:head [write 3579904~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:57.754 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:57.716+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 173 slow ops, oldest is osd_op(client.4659.0:88015 4.2s0 4:5050cdd3:::10000000378.00000009:head [write 495616~4096 in=4096b,write 3284992~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:58.741 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:58.706+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 169 slow ops, oldest is osd_op(client.4659.0:88192 4.2s0 4:53cf36d9:::100000003a9.00000013:head [write 2752512~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:50:59.816 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:50:59.780+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 167 slow ops, oldest is osd_op(client.4659.0:88703 4.2s0 4:5fc78031:::100000003ab.00000016:head [write 3346432~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:00.828 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:00.790+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 169 slow ops, oldest is osd_op(client.4659.0:88703 4.2s0 4:5fc78031:::100000003ab.00000016:head [write 3346432~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:01.815 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:01.791+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 168 slow ops, oldest is osd_op(client.4659.0:88703 4.2s0 4:5fc78031:::100000003ab.00000016:head [write 3346432~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:02.839 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:02.811+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 165 slow ops, oldest is osd_op(client.4659.0:88761 4.2s0 4:5930cd20:::10000000378.00000005:head [write 3575808~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:03.837 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:03.787+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 159 slow ops, oldest is osd_op(client.4659.0:89117 4.2s0 4:5161228a:::100000003a9.00000001:head [write 1703936~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:04.848 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:04.842+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 157 slow ops, oldest is osd_op(client.4659.0:89117 4.2s0 4:5161228a:::100000003a9.00000001:head [write 1703936~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:05.840 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:05.812+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 153 slow ops, oldest is osd_op(client.4659.0:89211 4.2s0 4:5159874e:::1000000039c.00000000:head [write 212992~4096 in=4096b,write 2891776~4096 in=4096b,write 3981312~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:06.882 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:06.849+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 150 slow ops, oldest is osd_op(client.4659.0:89436 4.2s0 4:4c96ca10:::1000000034b.0000000d:head [write 3149824~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:07.915 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:07.910+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 146 slow ops, oldest is osd_op(client.4659.0:89829 4.2s0 4:5aa439e2:::1000000039a.00000018:head [write 495616~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:09.020 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:08.989+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 144 slow ops, oldest is osd_op(client.4659.0:89829 4.2s0 4:5aa439e2:::1000000039a.00000018:head [write 495616~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:10.086 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:10.036+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 142 slow ops, oldest is osd_op(client.4659.0:89953 4.2s0 4:40dde1ee:::1000000036b.0000000d:head [write 86016~4096 in=4096b,write 1314816~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:11.140 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:11.087+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4659.0:90155 4.2s0 4:4bcd2dac:::10000000388.00000008:head [write 1835008~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:12.081 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:12.055+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 140 slow ops, oldest is osd_op(client.4659.0:90155 4.2s0 4:4bcd2dac:::10000000388.00000008:head [write 1835008~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:13.131 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:13.084+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 134 slow ops, oldest is osd_op(client.4659.0:90528 4.2s0 4:4ea2e919:::100000003a9.0000000b:head [write 2678784~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:14.160 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:14.131+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 131 slow ops, oldest is osd_op(client.4659.0:90528 4.2s0 4:4ea2e919:::100000003a9.0000000b:head [write 2678784~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:15.194 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:15.164+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 125 slow ops, oldest is osd_op(client.4659.0:90992 4.2s0 4:48389afd:::100000003b9.00000013:head [write 3866624~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:16.218 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:16.191+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 122 slow ops, oldest is osd_op(client.4659.0:90992 4.2s0 4:48389afd:::100000003b9.00000013:head [write 3866624~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:17.214 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:17.172+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 119 slow ops, oldest is osd_op(client.4659.0:90992 4.2s0 4:48389afd:::100000003b9.00000013:head [write 3866624~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:18.287 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:18.233+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 113 slow ops, oldest is osd_op(client.4659.0:91016 4.2s0 4:48e3a637:::10000000378.00000008:head [write 1417216~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:19.265 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:19.260+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 109 slow ops, oldest is osd_op(client.4659.0:91182 4.2s0 4:4ea2e919:::100000003a9.0000000b:head [write 819200~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:20.289 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:20.255+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 107 slow ops, oldest is osd_op(client.4659.0:91619 4.2s0 4:45e93c86:::100000003c1.00000005:head [write 290816~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:21.281 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:21.262+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 102 slow ops, oldest is osd_op(client.4659.0:91619 4.2s0 4:45e93c86:::100000003c1.00000005:head [write 290816~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:22.287 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:22.283+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 97 slow ops, oldest is osd_op(client.4659.0:91733 4.2s0 4:5930cd20:::10000000378.00000005:head [write 2191360~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:23.340 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:23.308+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 95 slow ops, oldest is osd_op(client.4659.0:91853 4.2s0 4:48e2ec13:::10000000356.00000002:head [write 1527808~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:24.371 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:24.365+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 93 slow ops, oldest is osd_op(client.4659.0:91988 4.2s0 4:4be7d3c6:::1000000039f.00000004:head [write 81920~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:25.416 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:25.412+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 91 slow ops, oldest is osd_op(client.4659.0:92119 4.2s0 4:5aaae2bc:::10000000358.00000006:head [write 1368064~4096 in=4096b,write 2838528~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:27.577 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:27.533+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 87 slow ops, oldest is osd_op(client.4659.0:92616 4.2s0 4:4fa3dc39:::100000003ab.00000003:head [write 692224~4096 in=4096b,write 3551232~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:28.649 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:28.645+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 86 slow ops, oldest is osd_op(client.4659.0:92836 4.2s0 4:59b11a87:::1000000035f.00000011:head [write 1843200~4096 in=4096b,write 2899968~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:29.723 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:29.640+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4659.0:93545 4.2s0 4:5ef0fbf2:::10000000378.00000001:head [write 135168~4096 in=4096b,write 2686976~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:30.765 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:30.704+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 84 slow ops, oldest is osd_op(client.4659.0:93545 4.2s0 4:5ef0fbf2:::10000000378.00000001:head [write 135168~4096 in=4096b,write 2686976~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:31.760 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:31.711+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 82 slow ops, oldest is osd_op(client.4659.0:93840 4.2s0 4:4258020c:::10000000356.0000000d:head [write 573440~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:32.748 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:32.704+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 80 slow ops, oldest is osd_op(client.4659.0:94303 4.2s0 4:5da1581e:::10000000397.00000012:head [write 1777664~4096 in=4096b,write 2187264~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:33.707 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:33.672+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 77 slow ops, oldest is osd_op(client.4659.0:94777 4.2s0 4:5c86b786:::10000000374.00000008:head [write 368640~4096 in=4096b,write 3145728~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:34.726 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:34.686+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 76 slow ops, oldest is osd_op(client.4659.0:95064 4.2s0 4:5ee13e11:::100000003ab.0000000e:head [write 1130496~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:35.704 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:35.676+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 74 slow ops, oldest is osd_op(client.4659.0:95547 4.2s0 4:4ad665d9:::1000000039c.00000001:head [write 573440~4096 in=4096b,write 1277952~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:36.717 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:36.691+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 72 slow ops, oldest is osd_op(client.4659.0:96188 4.2s0 4:5ee13e11:::100000003ab.0000000e:head [write 3452928~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:37.773 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:37.739+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 71 slow ops, oldest is osd_op(client.4659.0:96458 4.2s0 4:55243c39:::1000000036f.00000007:head [write 315392~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:38.772 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:38.733+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 69 slow ops, oldest is osd_op(client.4659.0:96750 4.2s0 4:585818ed:::100000003ab.0000000c:head [write 3895296~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:39.771 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:39.722+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4659.0:97492 4.2s0 4:410d2b24:::100000003a5.00000016:head [write 1011712~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:40.768 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:40.730+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 65 slow ops, oldest is osd_op(client.4659.0:97492 4.2s0 4:410d2b24:::100000003a5.00000016:head [write 1011712~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:41.770 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:41.740+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 61 slow ops, oldest is osd_op(client.4659.0:97961 4.2s0 4:506fa6e5:::100000003ab.00000011:head [write 1998848~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:42.745 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:42.710+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 60 slow ops, oldest is osd_op(client.4659.0:98474 4.2s0 4:4e9e94e3:::10000000382.00000012:head [write 1757184~4096 in=4096b,write 2162688~4096 in=4096b,write 2228224~4096 in=4096b,write 2465792~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:43.797 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:43.765+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 56 slow ops, oldest is osd_op(client.4659.0:99011 4.2s0 4:587a169a:::100000003b0.00000003:head [write 3080192~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:44.817 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:44.772+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 53 slow ops, oldest is osd_op(client.4659.0:99405 4.2s0 4:494c3c3b:::1000000034b.00000004:head [write 65536~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:45.851 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:45.804+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 51 slow ops, oldest is osd_op(client.4659.0:99915 4.2s0 4:4ea04888:::10000000397.00000009:head [write 3149824~4096 in=4096b,write 3600384~4096 in=4096b,write 3727360~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:46.800 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:46.766+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 46 slow ops, oldest is osd_op(client.4659.0:100837 4.2s0 4:5a6e0114:::10000000350.00000000:head [write 819200~4096 in=4096b,write 1368064~4096 in=4096b,write 3489792~4096 in=4096b,write 3788800~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:47.825 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:47.768+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 46 slow ops, oldest is osd_op(client.4659.0:100837 4.2s0 4:5a6e0114:::10000000350.00000000:head [write 819200~4096 in=4096b,write 1368064~4096 in=4096b,write 3489792~4096 in=4096b,write 3788800~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:48.827 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:48.793+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 44 slow ops, oldest is osd_op(client.4659.0:101186 4.2s0 4:4aa3c9fb:::10000000374.00000013:head [write 126976~4096 in=4096b,write 606208~4096 in=4096b,write 3219456~4096 in=4096b,write 3657728~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:49.868 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:49.817+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4659.0:102170 4.2s0 4:514d7452:::1000000034b.00000000:head [write 344064~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:50.849 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:50.812+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 40 slow ops, oldest is osd_op(client.4659.0:102170 4.2s0 4:514d7452:::1000000034b.00000000:head [write 344064~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:51.872 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:51.830+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 34 slow ops, oldest is osd_op(client.4659.0:102928 4.2s0 4:485235cd:::1000000037b.00000002:head [write 1069056~4096 in=4096b,write 1974272~4096 in=4096b,write 2404352~4096 in=4096b,write 3956736~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:52.792 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:52.783+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 32 slow ops, oldest is osd_op(client.4659.0:103575 4.2s0 4:59881595:::10000000350.0000000b:head [write 2015232~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:53.832 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:53.789+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 32 slow ops, oldest is osd_op(client.4659.0:103575 4.2s0 4:59881595:::10000000350.0000000b:head [write 2015232~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:54.802 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:54.767+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 29 slow ops, oldest is osd_op(client.4659.0:104224 4.2s0 4:521d4d5d:::100000003b0.00000008:head [write 503808~4096 in=4096b,write 1748992~4096 in=4096b,write 1957888~4096 in=4096b,write 3428352~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:55.863 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:55.809+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 27 slow ops, oldest is osd_op(client.4659.0:104518 4.2s0 4:5c05d725:::1000000036d.00000011:head [write 1675264~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:56.855 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:56.811+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 22 slow ops, oldest is osd_op(client.4659.0:105034 4.2s0 4:59881595:::10000000350.0000000b:head [write 2535424~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:57.841 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:57.829+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 20 slow ops, oldest is osd_op(client.4659.0:105646 4.2s0 4:54b73ea8:::1000000037f.00000007:head [write 589824~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:58.828 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:58.807+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4659.0:106089 4.2s0 4:5c660378:::100000003b5.00000009:head [write 1175552~4096 in=4096b,write 1855488~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:51:59.829 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:51:59.782+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4659.0:106385 4.2s0 4:531f3a23:::10000000378.00000014:head [write 1597440~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:52:00.847 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:52:00.825+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4659.0:107294 4.2s0 4:4dd551b2:::100000003b0.0000000a:head [write 3457024~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:52:01.833 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:52:01.808+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4659.0:107414 4.2s0 4:494c3c3b:::1000000034b.00000004:head [write 290816~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:52:02.836 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:52:02.816+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 8 slow ops, oldest is osd_op(client.4659.0:107953 4.2s0 4:485235cd:::1000000037b.00000002:head [write 1458176~4096 in=4096b,write 3600384~4096 in=4096b,write 3731456~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:52:03.840 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:52:03.804+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4659.0:108456 4.2s0 4:411d4d8e:::1000000036d.0000000a:head [write 708608~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:52:04.848 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:52:04.823+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4659.0:109717 4.2s0 4:50954393:::10000000353.00000013:head [write 2580480~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:52:05.873 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:52:05.842+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4659.0:109717 4.2s0 4:50954393:::10000000353.00000013:head [write 2580480~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:52:06.892 INFO:tasks.ceph.osd.1.smithi023.stderr:2020-11-20T11:52:06.872+0000 7f595c3e9700 -1 osd.1 25 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4659.0:109986 4.2s0 4:4effe0b7:::100000003b0.0000000e:head [write 3084288~4096 in=4096b,write 3866624~4096 in=4096b,write 4083712~4096 in=4096b] snapc 1=[] ondisk+write+known_if_redirected e25) 2020-11-20T11:52:14.327 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi023.front.sepia.ceph.com 2020-11-20T11:52:14.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T11:52:14.328 INFO:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2020-11-20T11:52:14.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T11:52:14.745 INFO:teuthology.orchestra.run.smithi023:> ip netns list 2020-11-20T11:52:15.232 INFO:teuthology.orchestra.run.smithi023.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2020-11-20T11:52:15.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T11:52:15.233 INFO:teuthology.orchestra.run.smithi023:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2020-11-20T11:52:16.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2020-11-20T11:52:16.413 INFO:teuthology.orchestra.run.smithi023:> sudo ip link delete ceph-brx 2020-11-20T11:52:16.582 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2020-11-20T11:52:16.592 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2020-11-20T11:52:16.592 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T11:52:18.444 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.441+0000 7f8bcc393700 1 Processor -- start 2020-11-20T11:52:19.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.441+0000 7f8bcc393700 1 -- start start 2020-11-20T11:52:19.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.442+0000 7f8bcc393700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8bc40608c0 0x7f8bc40aa0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:19.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.442+0000 7f8bcc393700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f8bc4060ca0 0x7f8bc40a6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:19.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.442+0000 7f8bcc393700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc40aa590 con 0x7f8bc40abbb0 2020-11-20T11:52:19.215 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.442+0000 7f8bcc393700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bc40ab300 con 0x7f8bc4060ca0 2020-11-20T11:52:19.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.442+0000 7f8bcc393700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc40aabe0 con 0x7f8bc40608c0 2020-11-20T11:52:19.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.442+0000 7f8bca12f700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8bc40608c0 0x7f8bc40aa0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:19.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.442+0000 7f8bca12f700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8bc40608c0 0x7f8bc40aa0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:57392/0 (socket says 172.21.15.23:57392) 2020-11-20T11:52:19.216 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.442+0000 7f8bca12f700 1 -- 172.21.15.23:0/3631772456 learned_addr learned my addr 172.21.15.23:0/3631772456 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:52:19.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.443+0000 7f8bc992e700 1 --2- 172.21.15.23:0/3631772456 >> v2:172.21.15.23:3300/0 conn(0x7f8bc4060ca0 0x7f8bc40a6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:19.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.443+0000 7f8bc992e700 1 -- 172.21.15.23:0/3631772456 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8bc40608c0 msgr2=0x7f8bc40aa0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:52:19.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.443+0000 7f8bc992e700 1 --2- 172.21.15.23:0/3631772456 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8bc40608c0 0x7f8bc40aa0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:19.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.443+0000 7f8bc992e700 1 -- 172.21.15.23:0/3631772456 >> v1:172.21.15.23:6789/0 conn(0x7f8bc40abbb0 legacy=0x7f8bc40ac010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:19.217 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.443+0000 7f8bc992e700 1 -- 172.21.15.23:0/3631772456 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc40aae80 con 0x7f8bc4060ca0 2020-11-20T11:52:19.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.443+0000 7f8bc992e700 1 --2- 172.21.15.23:0/3631772456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bc4060ca0 0x7f8bc40a6d30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f8bb400dd70 tx=0x7f8bb400d2b0).ready entity=mon.0 client_cookie=f167090b016d07ef server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:19.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.444+0000 7f8bc912d700 1 -- 172.21.15.23:0/3631772456 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb400fbf0 con 0x7f8bc4060ca0 2020-11-20T11:52:19.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.444+0000 7f8bc912d700 1 -- 172.21.15.23:0/3631772456 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bb4000e90 con 0x7f8bc4060ca0 2020-11-20T11:52:19.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.444+0000 7f8bc912d700 1 -- 172.21.15.23:0/3631772456 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb4018960 con 0x7f8bc4060ca0 2020-11-20T11:52:19.218 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.445+0000 7f8bcc393700 1 -- 172.21.15.23:0/3631772456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bc4060ca0 msgr2=0x7f8bc40a6d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:19.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.445+0000 7f8bcc393700 1 --2- 172.21.15.23:0/3631772456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bc4060ca0 0x7f8bc40a6d30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f8bb400dd70 tx=0x7f8bb400d2b0).stop 2020-11-20T11:52:19.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.445+0000 7f8bcc393700 1 -- 172.21.15.23:0/3631772456 shutdown_connections 2020-11-20T11:52:19.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.445+0000 7f8bcc393700 1 --2- 172.21.15.23:0/3631772456 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8bc40608c0 0x7f8bc40aa0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:19.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.445+0000 7f8bcc393700 1 --2- 172.21.15.23:0/3631772456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bc4060ca0 0x7f8bc40a6d30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:19.219 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.445+0000 7f8bcc393700 1 -- 172.21.15.23:0/3631772456 >> 172.21.15.23:0/3631772456 conn(0x7f8bc4066750 msgr2=0x7f8bc40601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:19.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.445+0000 7f8bcc393700 1 -- 172.21.15.23:0/3631772456 shutdown_connections 2020-11-20T11:52:19.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.445+0000 7f8bcc393700 1 -- 172.21.15.23:0/3631772456 wait complete. 2020-11-20T11:52:19.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.447+0000 7f8bcc393700 1 Processor -- start 2020-11-20T11:52:19.220 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.447+0000 7f8bcc393700 1 -- start start 2020-11-20T11:52:19.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.448+0000 7f8bcc393700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bc40608c0 0x7f8bc413a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:19.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.448+0000 7f8bcc393700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bc4060ca0 0x7f8bc4134c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:19.221 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.448+0000 7f8bcc393700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8bc40abbb0 0x7f8bc4149940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:19.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.448+0000 7f8bc992e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bc4060ca0 0x7f8bc4134c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:19.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.448+0000 7f8bc992e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bc4060ca0 0x7f8bc4134c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47160/0 (socket says 172.21.15.23:47160) 2020-11-20T11:52:19.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.448+0000 7f8bc992e700 1 -- 172.21.15.23:0/3099395699 learned_addr learned my addr 172.21.15.23:0/3099395699 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:52:19.222 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.448+0000 7f8bcc393700 1 -- 172.21.15.23:0/3099395699 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc40a4950 con 0x7f8bc4060ca0 2020-11-20T11:52:19.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.448+0000 7f8bcc393700 1 -- 172.21.15.23:0/3099395699 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc4149200 con 0x7f8bc40608c0 2020-11-20T11:52:19.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.448+0000 7f8bcc393700 1 -- 172.21.15.23:0/3099395699 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc41494a0 con 0x7f8bc40abbb0 2020-11-20T11:52:19.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.448+0000 7f8bca12f700 1 --2- 172.21.15.23:0/3099395699 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bc40608c0 0x7f8bc413a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:19.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.449+0000 7f8bc992e700 1 -- 172.21.15.23:0/3099395699 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8bc40abbb0 msgr2=0x7f8bc4149940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:19.223 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.449+0000 7f8bc992e700 1 --2- 172.21.15.23:0/3099395699 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8bc40abbb0 0x7f8bc4149940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:52:19.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.449+0000 7f8bc992e700 1 -- 172.21.15.23:0/3099395699 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bc40608c0 msgr2=0x7f8bc413a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:19.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.449+0000 7f8bc992e700 1 --2- 172.21.15.23:0/3099395699 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bc40608c0 0x7f8bc413a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:19.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.449+0000 7f8bc992e700 1 -- 172.21.15.23:0/3099395699 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc4062720 con 0x7f8bc4060ca0 2020-11-20T11:52:19.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.449+0000 7f8bc992e700 1 --2- 172.21.15.23:0/3099395699 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bc4060ca0 0x7f8bc4134c30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f8bb400afd0 tx=0x7f8bb401b000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:19.224 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.449+0000 7f8bb27fc700 1 -- 172.21.15.23:0/3099395699 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb400a8f0 con 0x7f8bc4060ca0 2020-11-20T11:52:19.225 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.449+0000 7f8bb27fc700 1 -- 172.21.15.23:0/3099395699 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bb4005710 con 0x7f8bc4060ca0 2020-11-20T11:52:19.225 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.449+0000 7f8bcc393700 1 -- 172.21.15.23:0/3099395699 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bc4149e30 con 0x7f8bc4060ca0 2020-11-20T11:52:19.225 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.450+0000 7f8bb27fc700 1 -- 172.21.15.23:0/3099395699 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb4005a90 con 0x7f8bc4060ca0 2020-11-20T11:52:19.225 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.450+0000 7f8bcc393700 1 -- 172.21.15.23:0/3099395699 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bc4145de0 con 0x7f8bc4060ca0 2020-11-20T11:52:19.225 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.451+0000 7f8bcc393700 1 -- 172.21.15.23:0/3099395699 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bc4149670 con 0x7f8bc4060ca0 2020-11-20T11:52:19.226 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.451+0000 7f8bb27fc700 1 -- 172.21.15.23:0/3099395699 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f8bb402a030 con 0x7f8bc4060ca0 2020-11-20T11:52:19.226 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.451+0000 7f8bb27fc700 1 --2- 172.21.15.23:0/3099395699 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8b9c056d60 0x7f8b9c059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:19.226 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.451+0000 7f8bb27fc700 1 -- 172.21.15.23:0/3099395699 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f8bb4079060 con 0x7f8bc4060ca0 2020-11-20T11:52:19.226 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.452+0000 7f8bca12f700 1 --2- 172.21.15.23:0/3099395699 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8b9c056d60 0x7f8b9c059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:19.226 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.452+0000 7f8bca12f700 1 --2- 172.21.15.23:0/3099395699 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8b9c056d60 0x7f8b9c059220 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f8bc4062ef0 tx=0x7f8bc0006ef0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:19.227 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.456+0000 7f8bb27fc700 1 -- 172.21.15.23:0/3099395699 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f8bb404e400 con 0x7f8bc4060ca0 2020-11-20T11:52:19.227 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.655+0000 7f8bcc393700 1 -- 172.21.15.23:0/3099395699 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8bc4044710 con 0x7f8bc4060ca0 2020-11-20T11:52:19.227 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.658+0000 7f8bb27fc700 1 -- 172.21.15.23:0/3099395699 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17087 (secure 0 0 0) 0x7f8bb40362d0 con 0x7f8bc4060ca0 2020-11-20T11:52:19.227 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.660+0000 7f8bcc393700 1 -- 172.21.15.23:0/3099395699 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8b9c056d60 msgr2=0x7f8b9c059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:19.228 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.660+0000 7f8bcc393700 1 --2- 172.21.15.23:0/3099395699 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8b9c056d60 0x7f8b9c059220 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f8bc4062ef0 tx=0x7f8bc0006ef0).stop 2020-11-20T11:52:19.228 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.660+0000 7f8bcc393700 1 -- 172.21.15.23:0/3099395699 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bc4060ca0 msgr2=0x7f8bc4134c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:19.228 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.660+0000 7f8bcc393700 1 --2- 172.21.15.23:0/3099395699 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bc4060ca0 0x7f8bc4134c30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f8bb400afd0 tx=0x7f8bb401b000).stop 2020-11-20T11:52:19.228 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.660+0000 7f8bcc393700 1 -- 172.21.15.23:0/3099395699 shutdown_connections 2020-11-20T11:52:19.228 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.660+0000 7f8bcc393700 1 --2- 172.21.15.23:0/3099395699 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f8b9c056d60 0x7f8b9c059220 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:19.229 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.660+0000 7f8bcc393700 1 --2- 172.21.15.23:0/3099395699 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8bc40abbb0 0x7f8bc4149940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:52:19.229 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.660+0000 7f8bcc393700 1 --2- 172.21.15.23:0/3099395699 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8bc40608c0 0x7f8bc413a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:19.229 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.660+0000 7f8bcc393700 1 --2- 172.21.15.23:0/3099395699 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8bc4060ca0 0x7f8bc4134c30 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:19.229 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.660+0000 7f8bcc393700 1 -- 172.21.15.23:0/3099395699 >> 172.21.15.23:0/3099395699 conn(0x7f8bc4066750 msgr2=0x7f8bc40ac010 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:19.229 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.660+0000 7f8bcc393700 1 -- 172.21.15.23:0/3099395699 shutdown_connections 2020-11-20T11:52:19.230 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:18.660+0000 7f8bcc393700 1 -- 172.21.15.23:0/3099395699 wait complete. 2020-11-20T11:52:19.230 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:52:19.230 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":25,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","created":"2020-11-20T11:32:06.136679+0000","modified":"2020-11-20T11:32:26.470927+0000","last_up_change":"2020-11-20T11:32:13.536194+0000","last_in_change":"2020-11-20T11:32:13.536194+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T11:32:09.559868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T11:32:17.000782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T11:32:20.423341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T11:32:23.126273+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e9d219f5-e40e-4b2b-9468-deb9c4074a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":23083},{"type":"v1","addr":"172.21.15.23:6801","nonce":23083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":23083},{"type":"v1","addr":"172.21.15.23:6803","nonce":23083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":23083},{"type":"v1","addr":"172.21.15.23:6807","nonce":23083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":23083},{"type":"v1","addr":"172.21.15.23:6805","nonce":23083}]},"public_addr":"172.21.15.23:6801/23083","cluster_addr":"172.21.15.23:6803/23083","heartbeat_back_addr":"172.21.15.23:6807/23083","heartbeat_front_addr":"172.21.15.23:6805/23083","state":["exists","up"]},{"osd":1,"uuid":"2328adde-2108-4fe0-8138-f0d096a496a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":23099},{"type":"v1","addr":"172.21.15.23:6817","nonce":23099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":23099},{"type":"v1","addr":"172.21.15.23:6819","nonce":23099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":23099},{"type":"v1","addr":"172.21.15.23:6824","nonce":23099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":23099},{"type":"v1","addr":"172.21.15.23:6821","nonce":23099}]},"public_addr":"172.21.15.23:6817/23099","cluster_addr":"172.21.15.23:6819/23099","heartbeat_back_addr":"172.21.15.23:6824/23099","heartbeat_front_addr":"172.21.15.23:6821/23099","state":["exists","up"]},{"osd":2,"uuid":"13367708-59a2-4363-8a3a-ab0cb97145ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":23084},{"type":"v1","addr":"172.21.15.23:6809","nonce":23084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":23084},{"type":"v1","addr":"172.21.15.23:6811","nonce":23084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":23084},{"type":"v1","addr":"172.21.15.23:6815","nonce":23084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":23084},{"type":"v1","addr":"172.21.15.23:6813","nonce":23084}]},"public_addr":"172.21.15.23:6809/23084","cluster_addr":"172.21.15.23:6811/23084","heartbeat_back_addr":"172.21.15.23:6815/23084","heartbeat_front_addr":"172.21.15.23:6813/23084","state":["exists","up"]},{"osd":3,"uuid":"1f147dd7-9289-47b0-ad1c-4088618d6faa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6823","nonce":23114},{"type":"v1","addr":"172.21.15.23:6825","nonce":23114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":23114},{"type":"v1","addr":"172.21.15.23:6827","nonce":23114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":23114},{"type":"v1","addr":"172.21.15.23:6831","nonce":23114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":23114},{"type":"v1","addr":"172.21.15.23:6829","nonce":23114}]},"public_addr":"172.21.15.23:6825/23114","cluster_addr":"172.21.15.23:6827/23114","heartbeat_back_addr":"172.21.15.23:6831/23114","heartbeat_front_addr":"172.21.15.23:6829/23114","state":["exists","up"]},{"osd":4,"uuid":"d008b039-4bc5-4135-9678-1511e58efc29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":23280},{"type":"v1","addr":"172.21.15.155:6801","nonce":23280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":23280},{"type":"v1","addr":"172.21.15.155:6803","nonce":23280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":23280},{"type":"v1","addr":"172.21.15.155:6807","nonce":23280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":23280},{"type":"v1","addr":"172.21.15.155:6805","nonce":23280}]},"public_addr":"172.21.15.155:6801/23280","cluster_addr":"172.21.15.155:6803/23280","heartbeat_back_addr":"172.21.15.155:6807/23280","heartbeat_front_addr":"172.21.15.155:6805/23280","state":["exists","up"]},{"osd":5,"uuid":"56715e6f-cf6e-44a6-9e39-93ab1224095f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":23298},{"type":"v1","addr":"172.21.15.155:6817","nonce":23298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":23298},{"type":"v1","addr":"172.21.15.155:6819","nonce":23298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":23298},{"type":"v1","addr":"172.21.15.155:6823","nonce":23298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":23298},{"type":"v1","addr":"172.21.15.155:6821","nonce":23298}]},"public_addr":"172.21.15.155:6817/23298","cluster_addr":"172.21.15.155:6819/23298","heartbeat_back_addr":"172.21.15.155:6823/23298","heartbeat_front_addr":"172.21.15.155:6821/23298","state":["exists","up"]},{"osd":6,"uuid":"db1e61ca-8e14-4a1f-8607-6e01b148d61c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":23307},{"type":"v1","addr":"172.21.15.155:6825","nonce":23307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":23307},{"type":"v1","addr":"172.21.15.155:6827","nonce":23307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":23307},{"type":"v1","addr":"172.21.15.155:6831","nonce":23307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":23307},{"type":"v1","addr":"172.21.15.155:6829","nonce":23307}]},"public_addr":"172.21.15.155:6825/23307","cluster_addr":"172.21.15.155:6827/23307","heartbeat_back_addr":"172.21.15.155:6831/23307","heartbeat_front_addr":"172.21.15.155:6829/23307","state":["exists","up"]},{"osd":7,"uuid":"d44419be-ef89-462c-8cf1-3c6db2a6944b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":23281},{"type":"v1","addr":"172.21.15.155:6809","nonce":23281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":23281},{"type":"v1","addr":"172.21.15.155:6811","nonce":23281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":23281},{"type":"v1","addr":"172.21.15.155:6815","nonce":23281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":23281},{"type":"v1","addr":"172.21.15.155:6813","nonce":23281}]},"public_addr":"172.21.15.155:6809/23281","cluster_addr":"172.21.15.155:6811/23281","heartbeat_back_addr":"172.21.15.155:6815/23281","heartbeat_front_addr":"172.21.15.155:6813/23281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.486702+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.509396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.533414+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.509489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.467600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.513068+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.536636+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.531772+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T11:52:19.232 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-20T11:52:19.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.635+0000 7ffbdb9b0700 1 Processor -- start 2020-11-20T11:52:19.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.635+0000 7ffbdb9b0700 1 -- start start 2020-11-20T11:52:19.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.636+0000 7ffbdb9b0700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7ffbd40aaab0 0x7ffbd413ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:19.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.636+0000 7ffbdb9b0700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbd41388b0 con 0x7ffbd40aae90 2020-11-20T11:52:19.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.636+0000 7ffbdb9b0700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbd413e600 con 0x7ffbd413aeb0 2020-11-20T11:52:19.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.636+0000 7ffbdb9b0700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbd413e8a0 con 0x7ffbd40aaab0 2020-11-20T11:52:19.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.636+0000 7ffbd8f4b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7ffbd40aae90 0x7ffbd41381a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:33184/0 (socket says 172.21.15.23:33184) 2020-11-20T11:52:19.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.636+0000 7ffbd8f4b700 1 -- 172.21.15.23:0/1834659607 learned_addr learned my addr 172.21.15.23:0/1834659607 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:52:19.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.636+0000 7ffbd974c700 1 --2- 172.21.15.23:0/1834659607 >> v2:172.21.15.155:3300/0 conn(0x7ffbd40aaab0 0x7ffbd413ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:19.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.636+0000 7ffbc3fff700 1 -- 172.21.15.23:0/1834659607 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3743158165 0 0) 0x7ffbd41388b0 con 0x7ffbd40aae90 2020-11-20T11:52:19.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.636+0000 7ffbc3fff700 1 -- 172.21.15.23:0/1834659607 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbbc003280 con 0x7ffbd40aae90 2020-11-20T11:52:19.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.636+0000 7ffbc3fff700 1 -- 172.21.15.23:0/1834659607 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 354093738 0 0) 0x7ffbd413e600 con 0x7ffbd413aeb0 2020-11-20T11:52:19.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.636+0000 7ffbc3fff700 1 -- 172.21.15.23:0/1834659607 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbd41388b0 con 0x7ffbd413aeb0 2020-11-20T11:52:19.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.636+0000 7ffbd974c700 1 -- 172.21.15.23:0/1834659607 >> v1:172.21.15.155:6789/0 conn(0x7ffbd413aeb0 legacy=0x7ffbd40a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:19.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.636+0000 7ffbd974c700 1 -- 172.21.15.23:0/1834659607 >> v1:172.21.15.23:6789/0 conn(0x7ffbd40aae90 legacy=0x7ffbd41381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:19.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.636+0000 7ffbd974c700 1 -- 172.21.15.23:0/1834659607 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd40ab6d0 con 0x7ffbd40aaab0 2020-11-20T11:52:19.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.637+0000 7ffbd974c700 1 --2- 172.21.15.23:0/1834659607 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbd40aaab0 0x7ffbd413ed40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7ffbd0006fd0 tx=0x7ffbd0006e30).ready entity=mon.1 client_cookie=8d645a5eb942d764 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:19.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.637+0000 7ffbc3fff700 1 -- 172.21.15.23:0/1834659607 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd0009630 con 0x7ffbd40aaab0 2020-11-20T11:52:19.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.637+0000 7ffbc3fff700 1 -- 172.21.15.23:0/1834659607 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd0000e90 con 0x7ffbd40aaab0 2020-11-20T11:52:19.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.637+0000 7ffbc3fff700 1 -- 172.21.15.23:0/1834659607 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd00099a0 con 0x7ffbd40aaab0 2020-11-20T11:52:19.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.637+0000 7ffbdb9b0700 1 -- 172.21.15.23:0/1834659607 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbd40aaab0 msgr2=0x7ffbd413ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:19.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.637+0000 7ffbdb9b0700 1 --2- 172.21.15.23:0/1834659607 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbd40aaab0 0x7ffbd413ed40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7ffbd0006fd0 tx=0x7ffbd0006e30).stop 2020-11-20T11:52:19.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.637+0000 7ffbdb9b0700 1 -- 172.21.15.23:0/1834659607 shutdown_connections 2020-11-20T11:52:19.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.637+0000 7ffbdb9b0700 1 --2- 172.21.15.23:0/1834659607 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbd40aaab0 0x7ffbd413ed40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:19.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.637+0000 7ffbdb9b0700 1 -- 172.21.15.23:0/1834659607 >> 172.21.15.23:0/1834659607 conn(0x7ffbd40678e0 msgr2=0x7ffbd4067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:19.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.637+0000 7ffbdb9b0700 1 -- 172.21.15.23:0/1834659607 shutdown_connections 2020-11-20T11:52:19.646 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.637+0000 7ffbdb9b0700 1 -- 172.21.15.23:0/1834659607 wait complete. 2020-11-20T11:52:19.648 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.638+0000 7ffbdb9b0700 1 Processor -- start 2020-11-20T11:52:19.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.638+0000 7ffbdb9b0700 1 -- start start 2020-11-20T11:52:19.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.639+0000 7ffbdb9b0700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbd40aaab0 0x7ffbd4139f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:19.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.639+0000 7ffbdb9b0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbd40aae90 0x7ffbd4131930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:19.649 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.639+0000 7ffbdb9b0700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffbd413dc80 0x7ffbd4136910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:19.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.639+0000 7ffbdb9b0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd41355d0 con 0x7ffbd40aae90 2020-11-20T11:52:19.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.639+0000 7ffbdb9b0700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd413e190 con 0x7ffbd40aaab0 2020-11-20T11:52:19.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.639+0000 7ffbdb9b0700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd4134e90 con 0x7ffbd413dc80 2020-11-20T11:52:19.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.639+0000 7ffbd8f4b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbd40aae90 0x7ffbd4131930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:19.650 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.639+0000 7ffbd8f4b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbd40aae90 0x7ffbd4131930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47174/0 (socket says 172.21.15.23:47174) 2020-11-20T11:52:19.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.639+0000 7ffbd8f4b700 1 -- 172.21.15.23:0/426559127 learned_addr learned my addr 172.21.15.23:0/426559127 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:52:19.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.639+0000 7ffbd974c700 1 --2- 172.21.15.23:0/426559127 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbd40aaab0 0x7ffbd4139f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:19.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.639+0000 7ffbd9f4d700 1 --2- 172.21.15.23:0/426559127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffbd413dc80 0x7ffbd4136910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:19.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.640+0000 7ffbd8f4b700 1 -- 172.21.15.23:0/426559127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffbd413dc80 msgr2=0x7ffbd4136910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:19.651 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.640+0000 7ffbd8f4b700 1 --2- 172.21.15.23:0/426559127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffbd413dc80 0x7ffbd4136910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:19.652 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.640+0000 7ffbd8f4b700 1 -- 172.21.15.23:0/426559127 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbd40aaab0 msgr2=0x7ffbd4139f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:19.652 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.640+0000 7ffbd8f4b700 1 --2- 172.21.15.23:0/426559127 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbd40aaab0 0x7ffbd4139f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:19.652 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.640+0000 7ffbd8f4b700 1 -- 172.21.15.23:0/426559127 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd4135130 con 0x7ffbd40aae90 2020-11-20T11:52:19.652 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.640+0000 7ffbd8f4b700 1 --2- 172.21.15.23:0/426559127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbd40aae90 0x7ffbd4131930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7ffbc400c740 tx=0x7ffbc4007bb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:19.652 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.640+0000 7ffbc17fa700 1 -- 172.21.15.23:0/426559127 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc40072e0 con 0x7ffbd40aae90 2020-11-20T11:52:19.653 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.640+0000 7ffbc17fa700 1 -- 172.21.15.23:0/426559127 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbc4007440 con 0x7ffbd40aae90 2020-11-20T11:52:19.653 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.640+0000 7ffbc17fa700 1 -- 172.21.15.23:0/426559127 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc4005840 con 0x7ffbd40aae90 2020-11-20T11:52:19.653 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.640+0000 7ffbdb9b0700 1 -- 172.21.15.23:0/426559127 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbd413a850 con 0x7ffbd40aae90 2020-11-20T11:52:19.653 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.640+0000 7ffbdb9b0700 1 -- 172.21.15.23:0/426559127 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbd413d920 con 0x7ffbd40aae90 2020-11-20T11:52:19.654 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.641+0000 7ffbdb9b0700 1 -- 172.21.15.23:0/426559127 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbd405c7d0 con 0x7ffbd40aae90 2020-11-20T11:52:19.654 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.641+0000 7ffbc17fa700 1 -- 172.21.15.23:0/426559127 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7ffbc4010070 con 0x7ffbd40aae90 2020-11-20T11:52:19.654 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.641+0000 7ffbc17fa700 1 --2- 172.21.15.23:0/426559127 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ffbac056d60 0x7ffbac059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:19.654 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.642+0000 7ffbc17fa700 1 -- 172.21.15.23:0/426559127 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7ffbc406f810 con 0x7ffbd40aae90 2020-11-20T11:52:19.654 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.642+0000 7ffbd974c700 1 --2- 172.21.15.23:0/426559127 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ffbac056d60 0x7ffbac059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:19.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.642+0000 7ffbd974c700 1 --2- 172.21.15.23:0/426559127 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ffbac056d60 0x7ffbac059220 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7ffbd0006e70 tx=0x7ffbd0015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:19.655 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.645+0000 7ffbc17fa700 1 -- 172.21.15.23:0/426559127 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7ffbc4044b00 con 0x7ffbd40aae90 2020-11-20T11:52:19.823 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.820+0000 7ffbdb9b0700 1 -- 172.21.15.23:0/426559127 --> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffbd41352c0 con 0x7ffbac056d60 2020-11-20T11:52:19.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.825+0000 7ffbc17fa700 1 -- 172.21.15.23:0/426559127 <== mgr.4105 v2:172.21.15.155:6832/22693 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+106835 (secure 0 0 0) 0x7ffbd41352c0 con 0x7ffbac056d60 2020-11-20T11:52:19.832 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:52:19.833 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.830+0000 7ffbdb9b0700 1 -- 172.21.15.23:0/426559127 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ffbac056d60 msgr2=0x7ffbac059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:19.834 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.830+0000 7ffbdb9b0700 1 --2- 172.21.15.23:0/426559127 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ffbac056d60 0x7ffbac059220 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7ffbd0006e70 tx=0x7ffbd0015000).stop 2020-11-20T11:52:19.834 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.830+0000 7ffbdb9b0700 1 -- 172.21.15.23:0/426559127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbd40aae90 msgr2=0x7ffbd4131930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:19.834 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.830+0000 7ffbdb9b0700 1 --2- 172.21.15.23:0/426559127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbd40aae90 0x7ffbd4131930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7ffbc400c740 tx=0x7ffbc4007bb0).stop 2020-11-20T11:52:19.834 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.830+0000 7ffbdb9b0700 1 -- 172.21.15.23:0/426559127 shutdown_connections 2020-11-20T11:52:19.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.830+0000 7ffbdb9b0700 1 --2- 172.21.15.23:0/426559127 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7ffbac056d60 0x7ffbac059220 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:19.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.830+0000 7ffbdb9b0700 1 --2- 172.21.15.23:0/426559127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7ffbd413dc80 0x7ffbd4136910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:19.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.830+0000 7ffbdb9b0700 1 --2- 172.21.15.23:0/426559127 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7ffbd40aaab0 0x7ffbd4139f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:19.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.830+0000 7ffbdb9b0700 1 --2- 172.21.15.23:0/426559127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbd40aae90 0x7ffbd4131930 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:19.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.830+0000 7ffbdb9b0700 1 -- 172.21.15.23:0/426559127 >> 172.21.15.23:0/426559127 conn(0x7ffbd40678e0 msgr2=0x7ffbd40add70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:19.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.830+0000 7ffbdb9b0700 1 -- 172.21.15.23:0/426559127 shutdown_connections 2020-11-20T11:52:19.836 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.830+0000 7ffbdb9b0700 1 -- 172.21.15.23:0/426559127 wait complete. 2020-11-20T11:52:19.836 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2020-11-20T11:52:19.845 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":622,"stamp":"2020-11-20T11:52:19.792359+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":9576486111,"num_objects":2314,"num_object_clones":0,"num_object_copies":9202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2314,"num_whiteouts":0,"num_read":23,"num_read_kb":61,"num_write":254211,"num_write_kb":14114655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82409,"ondisk_log_size":82409,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":66,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":21481128,"kb_used_data":18849576,"kb_used_omap":0,"kb_used_meta":2611008,"kb_avail":733493592,"statfs":{"total":773094113280,"available":751097438208,"internally_reserved":0,"allocated":19301965824,"data_stored":19300668190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2673672192},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,4,2,12,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":150,"apply_latency_ms":150,"commit_latency_ns":150000000,"apply_latency_ns":150000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3751422935,"num_objects":-997,"num_object_clones":0,"num_object_copies":-3970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-997,"num_whiteouts":0,"num_read":3,"num_read_kb":6,"num_write":1677,"num_write_kb":2757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002822"},"pg_stats":[{"pgid":"4.2","version":"25'15278","reported_seq":"15290","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:16.537332+0000","last_change":"2020-11-20T11:32:25.125195+0000","last_active":"2020-11-20T11:52:16.537332+0000","last_peered":"2020-11-20T11:52:16.537332+0000","last_clean":"2020-11-20T11:52:16.537332+0000","last_became_active":"2020-11-20T11:32:25.124627+0000","last_became_peered":"2020-11-20T11:32:25.124627+0000","last_unstale":"2020-11-20T11:52:16.537332+0000","last_undegraded":"2020-11-20T11:52:16.537332+0000","last_fullsized":"2020-11-20T11:52:16.537332+0000","mapping_epoch":22,"log_start":"25'5209","ondisk_log_start":"25'5209","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":10069,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1308660080,"num_objects":315,"num_object_clones":0,"num_object_copies":1260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33239,"num_write_kb":1823424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"16","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.480586+0000","last_change":"2020-11-20T11:32:19.482604+0000","last_active":"2020-11-20T11:32:25.480586+0000","last_peered":"2020-11-20T11:32:25.480586+0000","last_clean":"2020-11-20T11:32:25.480586+0000","last_became_active":"2020-11-20T11:32:19.482197+0000","last_became_peered":"2020-11-20T11:32:19.482197+0000","last_unstale":"2020-11-20T11:32:25.480586+0000","last_undegraded":"2020-11-20T11:32:25.480586+0000","last_fullsized":"2020-11-20T11:32:25.480586+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"25'29","reported_seq":"42","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:12.129105+0000","last_change":"2020-11-20T11:32:22.098576+0000","last_active":"2020-11-20T11:52:12.129105+0000","last_peered":"2020-11-20T11:52:12.129105+0000","last_clean":"2020-11-20T11:52:12.129105+0000","last_became_active":"2020-11-20T11:32:22.098264+0000","last_became_peered":"2020-11-20T11:32:22.098264+0000","last_unstale":"2020-11-20T11:52:12.129105+0000","last_undegraded":"2020-11-20T11:52:12.129105+0000","last_fullsized":"2020-11-20T11:52:12.129105+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"25'13990","reported_seq":"14002","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:16.500458+0000","last_change":"2020-11-20T11:32:25.479776+0000","last_active":"2020-11-20T11:52:16.500458+0000","last_peered":"2020-11-20T11:52:16.500458+0000","last_clean":"2020-11-20T11:52:16.500458+0000","last_became_active":"2020-11-20T11:32:25.479350+0000","last_became_peered":"2020-11-20T11:32:25.479350+0000","last_unstale":"2020-11-20T11:52:16.500458+0000","last_undegraded":"2020-11-20T11:52:16.500458+0000","last_fullsized":"2020-11-20T11:52:16.500458+0000","mapping_epoch":22,"log_start":"25'3891","ondisk_log_start":"25'3891","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":10099,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1140857072,"num_objects":273,"num_object_clones":0,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31413,"num_write_kb":1652120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"16","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.479488+0000","last_change":"2020-11-20T11:32:18.924328+0000","last_active":"2020-11-20T11:32:25.479488+0000","last_peered":"2020-11-20T11:32:25.479488+0000","last_clean":"2020-11-20T11:32:25.479488+0000","last_became_active":"2020-11-20T11:32:18.923879+0000","last_became_peered":"2020-11-20T11:32:18.923879+0000","last_unstale":"2020-11-20T11:32:25.479488+0000","last_undegraded":"2020-11-20T11:32:25.479488+0000","last_fullsized":"2020-11-20T11:32:25.479488+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"25'1083","reported_seq":"1096","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:12.255599+0000","last_change":"2020-11-20T11:32:22.093559+0000","last_active":"2020-11-20T11:52:12.255599+0000","last_peered":"2020-11-20T11:52:12.255599+0000","last_clean":"2020-11-20T11:52:12.255599+0000","last_became_active":"2020-11-20T11:32:22.093123+0000","last_became_peered":"2020-11-20T11:32:22.093123+0000","last_unstale":"2020-11-20T11:52:12.255599+0000","last_undegraded":"2020-11-20T11:52:12.255599+0000","last_fullsized":"2020-11-20T11:52:12.255599+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":1083,"ondisk_log_size":1083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4554460,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1083,"num_write_kb":4776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"25'14326","reported_seq":"14338","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:16.500463+0000","last_change":"2020-11-20T11:32:25.125052+0000","last_active":"2020-11-20T11:52:16.500463+0000","last_peered":"2020-11-20T11:52:16.500463+0000","last_clean":"2020-11-20T11:52:16.500463+0000","last_became_active":"2020-11-20T11:32:25.124401+0000","last_became_peered":"2020-11-20T11:32:25.124401+0000","last_unstale":"2020-11-20T11:52:16.500463+0000","last_undegraded":"2020-11-20T11:52:16.500463+0000","last_fullsized":"2020-11-20T11:52:16.500463+0000","mapping_epoch":22,"log_start":"25'4285","ondisk_log_start":"25'4285","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":10041,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1279676720,"num_objects":306,"num_object_clones":0,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31036,"num_write_kb":1732953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"15","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.474417+0000","last_change":"2020-11-20T11:32:18.927388+0000","last_active":"2020-11-20T11:32:25.474417+0000","last_peered":"2020-11-20T11:32:25.474417+0000","last_clean":"2020-11-20T11:32:25.474417+0000","last_became_active":"2020-11-20T11:32:18.926981+0000","last_became_peered":"2020-11-20T11:32:18.926981+0000","last_unstale":"2020-11-20T11:32:25.474417+0000","last_undegraded":"2020-11-20T11:32:25.474417+0000","last_fullsized":"2020-11-20T11:32:25.474417+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"25'13","reported_seq":"26","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:12.252954+0000","last_change":"2020-11-20T11:32:22.088128+0000","last_active":"2020-11-20T11:52:12.252954+0000","last_peered":"2020-11-20T11:52:12.252954+0000","last_clean":"2020-11-20T11:52:12.252954+0000","last_became_active":"2020-11-20T11:32:22.087723+0000","last_became_peered":"2020-11-20T11:32:22.087723+0000","last_unstale":"2020-11-20T11:52:12.252954+0000","last_undegraded":"2020-11-20T11:52:12.252954+0000","last_fullsized":"2020-11-20T11:52:12.252954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"25'13482","reported_seq":"13494","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:19.368716+0000","last_change":"2020-11-20T11:32:25.492514+0000","last_active":"2020-11-20T11:52:19.368716+0000","last_peered":"2020-11-20T11:52:19.368716+0000","last_clean":"2020-11-20T11:52:19.368716+0000","last_became_active":"2020-11-20T11:32:25.491657+0000","last_became_peered":"2020-11-20T11:32:25.491657+0000","last_unstale":"2020-11-20T11:52:19.368716+0000","last_undegraded":"2020-11-20T11:52:19.368716+0000","last_fullsized":"2020-11-20T11:52:19.368716+0000","mapping_epoch":22,"log_start":"25'3448","ondisk_log_start":"25'3448","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":10034,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1170210816,"num_objects":279,"num_object_clones":0,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30651,"num_write_kb":1726809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"16","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.479536+0000","last_change":"2020-11-20T11:32:18.927044+0000","last_active":"2020-11-20T11:32:25.479536+0000","last_peered":"2020-11-20T11:32:25.479536+0000","last_clean":"2020-11-20T11:32:25.479536+0000","last_became_active":"2020-11-20T11:32:18.926711+0000","last_became_peered":"2020-11-20T11:32:18.926711+0000","last_unstale":"2020-11-20T11:32:25.479536+0000","last_undegraded":"2020-11-20T11:32:25.479536+0000","last_fullsized":"2020-11-20T11:32:25.479536+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'2","reported_seq":"20","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.488044+0000","last_change":"2020-11-20T11:32:14.446049+0000","last_active":"2020-11-20T11:32:25.488044+0000","last_peered":"2020-11-20T11:32:25.488044+0000","last_clean":"2020-11-20T11:32:25.488044+0000","last_became_active":"2020-11-20T11:32:14.445477+0000","last_became_peered":"2020-11-20T11:32:14.445477+0000","last_unstale":"2020-11-20T11:32:25.488044+0000","last_undegraded":"2020-11-20T11:32:25.488044+0000","last_fullsized":"2020-11-20T11:32:25.488044+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:10.515989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:10.515989+0000","last_clean_scrub_stamp":"2020-11-20T11:32:10.515989+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"25'4","reported_seq":"17","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:34:00.505799+0000","last_change":"2020-11-20T11:32:22.088160+0000","last_active":"2020-11-20T11:34:00.505799+0000","last_peered":"2020-11-20T11:34:00.505799+0000","last_clean":"2020-11-20T11:34:00.505799+0000","last_became_active":"2020-11-20T11:32:22.087777+0000","last_became_peered":"2020-11-20T11:32:22.087777+0000","last_unstale":"2020-11-20T11:34:00.505799+0000","last_undegraded":"2020-11-20T11:34:00.505799+0000","last_fullsized":"2020-11-20T11:34:00.505799+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"25'12243","reported_seq":"12255","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:17.629218+0000","last_change":"2020-11-20T11:32:25.491667+0000","last_active":"2020-11-20T11:52:17.629218+0000","last_peered":"2020-11-20T11:52:17.629218+0000","last_clean":"2020-11-20T11:52:17.629218+0000","last_became_active":"2020-11-20T11:32:25.491100+0000","last_became_peered":"2020-11-20T11:32:25.491100+0000","last_unstale":"2020-11-20T11:52:17.629218+0000","last_undegraded":"2020-11-20T11:52:17.629218+0000","last_fullsized":"2020-11-20T11:52:17.629218+0000","mapping_epoch":22,"log_start":"25'2152","ondisk_log_start":"25'2152","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":10091,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1069571072,"num_objects":256,"num_object_clones":0,"num_object_copies":1024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28072,"num_write_kb":1700287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"15","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.488045+0000","last_change":"2020-11-20T11:32:18.924600+0000","last_active":"2020-11-20T11:32:25.488045+0000","last_peered":"2020-11-20T11:32:25.488045+0000","last_clean":"2020-11-20T11:32:25.488045+0000","last_became_active":"2020-11-20T11:32:18.924324+0000","last_became_peered":"2020-11-20T11:32:18.924324+0000","last_unstale":"2020-11-20T11:32:25.488045+0000","last_undegraded":"2020-11-20T11:32:25.488045+0000","last_fullsized":"2020-11-20T11:32:25.488045+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"25'56","reported_seq":"92","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:12.255012+0000","last_change":"2020-11-20T11:32:22.088716+0000","last_active":"2020-11-20T11:52:12.255012+0000","last_peered":"2020-11-20T11:52:12.255012+0000","last_clean":"2020-11-20T11:52:12.255012+0000","last_became_active":"2020-11-20T11:32:22.088414+0000","last_became_peered":"2020-11-20T11:32:22.088414+0000","last_unstale":"2020-11-20T11:52:12.255012+0000","last_undegraded":"2020-11-20T11:52:12.255012+0000","last_fullsized":"2020-11-20T11:52:12.255012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":47377,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":61,"num_write":58,"num_write_kb":143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"25'14967","reported_seq":"14979","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:17.506647+0000","last_change":"2020-11-20T11:32:25.491507+0000","last_active":"2020-11-20T11:52:17.506647+0000","last_peered":"2020-11-20T11:52:17.506647+0000","last_clean":"2020-11-20T11:52:17.506647+0000","last_became_active":"2020-11-20T11:32:25.490917+0000","last_became_peered":"2020-11-20T11:32:25.490917+0000","last_unstale":"2020-11-20T11:52:17.506647+0000","last_undegraded":"2020-11-20T11:52:17.506647+0000","last_fullsized":"2020-11-20T11:52:17.506647+0000","mapping_epoch":22,"log_start":"25'4951","ondisk_log_start":"25'4951","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":10016,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1153433600,"num_objects":275,"num_object_clones":0,"num_object_copies":1100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33563,"num_write_kb":1816287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"15","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.488121+0000","last_change":"2020-11-20T11:32:19.482719+0000","last_active":"2020-11-20T11:32:25.488121+0000","last_peered":"2020-11-20T11:32:25.488121+0000","last_clean":"2020-11-20T11:32:25.488121+0000","last_became_active":"2020-11-20T11:32:19.482359+0000","last_became_peered":"2020-11-20T11:32:19.482359+0000","last_unstale":"2020-11-20T11:32:25.488121+0000","last_undegraded":"2020-11-20T11:32:25.488121+0000","last_fullsized":"2020-11-20T11:32:25.488121+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"25'558","reported_seq":"571","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:10.276646+0000","last_change":"2020-11-20T11:32:22.093099+0000","last_active":"2020-11-20T11:52:10.276646+0000","last_peered":"2020-11-20T11:52:10.276646+0000","last_clean":"2020-11-20T11:52:10.276646+0000","last_became_active":"2020-11-20T11:32:22.092619+0000","last_became_peered":"2020-11-20T11:32:22.092619+0000","last_unstale":"2020-11-20T11:52:10.276646+0000","last_undegraded":"2020-11-20T11:52:10.276646+0000","last_fullsized":"2020-11-20T11:52:10.276646+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":558,"ondisk_log_size":558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4194947,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":555,"num_write_kb":4297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"25'13906","reported_seq":"13917","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:15.164534+0000","last_change":"2020-11-20T11:32:25.492150+0000","last_active":"2020-11-20T11:52:15.164534+0000","last_peered":"2020-11-20T11:52:15.164534+0000","last_clean":"2020-11-20T11:52:15.164534+0000","last_became_active":"2020-11-20T11:32:25.491099+0000","last_became_peered":"2020-11-20T11:32:25.491099+0000","last_unstale":"2020-11-20T11:52:15.164534+0000","last_undegraded":"2020-11-20T11:52:15.164534+0000","last_fullsized":"2020-11-20T11:52:15.164534+0000","mapping_epoch":22,"log_start":"25'3829","ondisk_log_start":"25'3829","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":10077,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1275068416,"num_objects":304,"num_object_clones":0,"num_object_copies":1216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31586,"num_write_kb":1743046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"25'10","reported_seq":"23","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:09.920224+0000","last_change":"2020-11-20T11:32:22.461697+0000","last_active":"2020-11-20T11:52:09.920224+0000","last_peered":"2020-11-20T11:52:09.920224+0000","last_clean":"2020-11-20T11:52:09.920224+0000","last_became_active":"2020-11-20T11:32:22.461331+0000","last_became_peered":"2020-11-20T11:32:22.461331+0000","last_unstale":"2020-11-20T11:52:09.920224+0000","last_undegraded":"2020-11-20T11:52:09.920224+0000","last_fullsized":"2020-11-20T11:52:09.920224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"16","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.480381+0000","last_change":"2020-11-20T11:32:18.923886+0000","last_active":"2020-11-20T11:32:25.480381+0000","last_peered":"2020-11-20T11:32:25.480381+0000","last_clean":"2020-11-20T11:32:25.480381+0000","last_became_active":"2020-11-20T11:32:18.923448+0000","last_became_peered":"2020-11-20T11:32:18.923448+0000","last_unstale":"2020-11-20T11:32:25.480381+0000","last_undegraded":"2020-11-20T11:32:25.480381+0000","last_fullsized":"2020-11-20T11:32:25.480381+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"25'13864","reported_seq":"13876","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:18.213860+0000","last_change":"2020-11-20T11:32:25.491585+0000","last_active":"2020-11-20T11:52:18.213860+0000","last_peered":"2020-11-20T11:52:18.213860+0000","last_clean":"2020-11-20T11:52:18.213860+0000","last_became_active":"2020-11-20T11:32:25.491038+0000","last_became_peered":"2020-11-20T11:32:25.491038+0000","last_unstale":"2020-11-20T11:52:18.213860+0000","last_undegraded":"2020-11-20T11:52:18.213860+0000","last_fullsized":"2020-11-20T11:52:18.213860+0000","mapping_epoch":22,"log_start":"25'3777","ondisk_log_start":"25'3777","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":10087,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1170210816,"num_objects":279,"num_object_clones":0,"num_object_copies":1116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32545,"num_write_kb":1910315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"15","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.473625+0000","last_change":"2020-11-20T11:32:18.924052+0000","last_active":"2020-11-20T11:32:25.473625+0000","last_peered":"2020-11-20T11:32:25.473625+0000","last_clean":"2020-11-20T11:32:25.473625+0000","last_became_active":"2020-11-20T11:32:18.923668+0000","last_became_peered":"2020-11-20T11:32:18.923668+0000","last_unstale":"2020-11-20T11:32:25.473625+0000","last_undegraded":"2020-11-20T11:32:25.473625+0000","last_fullsized":"2020-11-20T11:32:25.473625+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"25'140","reported_seq":"153","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:09.083553+0000","last_change":"2020-11-20T11:32:22.093019+0000","last_active":"2020-11-20T11:52:09.083553+0000","last_peered":"2020-11-20T11:52:09.083553+0000","last_clean":"2020-11-20T11:52:09.083553+0000","last_became_active":"2020-11-20T11:32:22.092604+0000","last_became_peered":"2020-11-20T11:32:22.092604+0000","last_unstale":"2020-11-20T11:52:09.083553+0000","last_undegraded":"2020-11-20T11:52:09.083553+0000","last_fullsized":"2020-11-20T11:52:09.083553+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":140,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":353,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":9567688592,"num_objects":2287,"num_object_clones":0,"num_object_copies":9148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":252105,"num_write_kb":14105241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19022168064,"data_stored":19022168064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80514,"ondisk_log_size":80514,"up":32,"acting":32,"num_store_stats":8},{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":8797519,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":23,"num_read_kb":61,"num_write":2104,"num_write_kb":9404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17653760,"data_stored":17595038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1893,"ondisk_log_size":1893,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":6,"up_from":14,"seq":60129542388,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1897548,"kb_used_data":1650948,"kb_used_omap":0,"kb_used_meta":246592,"kb_avail":92474292,"statfs":{"total":96636764160,"available":94693675008,"internally_reserved":0,"allocated":1690570752,"data_stored":1690415888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":252510208},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 11:51:42 2020","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.0109999999999999,"15min":1.4450000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.45700000000000002},"max":{"1min":3.677,"5min":9.6129999999999995,"15min":22.257999999999999},"last":0.747},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.96999999999999997,"15min":1.4079999999999999},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":2.1379999999999999,"5min":6.3099999999999996,"15min":19.855},"last":0.77600000000000002}]},{"osd":1,"last update":"Fri Nov 20 11:51:42 2020","interfaces":[{"interface":"back","average":{"1min":5.4039999999999999,"5min":8.8640000000000008,"15min":6.9290000000000003},"min":{"1min":0.70999999999999996,"5min":0.504,"15min":0.504},"max":{"1min":23.738,"5min":39.268999999999998,"15min":42.552999999999997},"last":1.141},{"interface":"front","average":{"1min":6.1950000000000003,"5min":9.2330000000000005,"15min":6.8099999999999996},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":39.792999999999999,"5min":39.792999999999999,"15min":58.981000000000002},"last":4.4299999999999997}]},{"osd":2,"last update":"Fri Nov 20 11:51:42 2020","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.194,"15min":1.2050000000000001},"min":{"1min":0.65700000000000003,"5min":0.57999999999999996,"15min":0.57799999999999996},"max":{"1min":3.7090000000000001,"5min":6.0880000000000001,"15min":7.4790000000000001},"last":1.323},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0640000000000001,"15min":1.135},"min":{"1min":0.63200000000000001,"5min":0.61399999999999999,"15min":0.54200000000000004},"max":{"1min":2.2879999999999998,"5min":8.1519999999999992,"15min":8.1519999999999992},"last":3.9900000000000002}]},{"osd":3,"last update":"Fri Nov 20 11:51:42 2020","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.2470000000000001,"15min":1.4870000000000001},"min":{"1min":0.628,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":3.2400000000000002,"5min":9.7330000000000005,"15min":11.287000000000001},"last":1.675},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.1870000000000001,"15min":1.4770000000000001},"min":{"1min":0.65800000000000003,"5min":0.65300000000000002,"15min":0.61099999999999999},"max":{"1min":5.3170000000000002,"5min":5.3170000000000002,"15min":11.256},"last":1.4390000000000001}]},{"osd":4,"last update":"Fri Nov 20 11:51:42 2020","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":1.109,"15min":1.325},"min":{"1min":0.57999999999999996,"5min":0.57699999999999996,"15min":0.53500000000000003},"max":{"1min":1.319,"5min":9.1150000000000002,"15min":10.087999999999999},"last":1.8140000000000001},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.2649999999999999,"15min":1.3540000000000001},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.57399999999999995},"max":{"1min":1.655,"5min":8.9309999999999992,"15min":9.468},"last":1.542}]},{"osd":5,"last update":"Fri Nov 20 11:51:42 2020","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.135,"15min":1.274},"min":{"1min":0.68000000000000005,"5min":0.621,"15min":0.59499999999999997},"max":{"1min":1.52,"5min":7.8780000000000001,"15min":7.8780000000000001},"last":1.5620000000000001},{"interface":"front","average":{"1min":1.097,"5min":1.2609999999999999,"15min":1.494},"min":{"1min":0.69999999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.532,"5min":9.4779999999999998,"15min":9.4779999999999998},"last":1.9330000000000001}]},{"osd":7,"last update":"Fri Nov 20 11:51:42 2020","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.214,"15min":1.3680000000000001},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.70799999999999996},"max":{"1min":1.7569999999999999,"5min":4.3289999999999997,"15min":6.8760000000000003},"last":3.4159999999999999},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.258,"15min":1.423},"min":{"1min":0.79000000000000004,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.5549999999999999,"5min":8.0239999999999991,"15min":8.0239999999999991},"last":1.9850000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542388,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2836072,"kb_used_data":2505184,"kb_used_omap":0,"kb_used_meta":322688,"kb_avail":91535768,"statfs":{"total":96636764160,"available":93732626432,"internally_reserved":0,"allocated":2565308416,"data_stored":2565137903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":330432512},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 11:52:00 2020","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.98399999999999999,"15min":1.6080000000000001},"min":{"1min":0.48199999999999998,"5min":0.47099999999999997,"15min":0.45400000000000001},"max":{"1min":1.5760000000000001,"5min":6.2949999999999999,"15min":42.424999999999997},"last":3.3149999999999999},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.93999999999999995,"15min":1.728},"min":{"1min":0.498,"5min":0.49299999999999999,"15min":0.46700000000000003},"max":{"1min":2.1859999999999999,"5min":3.6600000000000001,"15min":41.997999999999998},"last":0.504}]},{"osd":1,"last update":"Fri Nov 20 11:52:05 2020","interfaces":[{"interface":"back","average":{"1min":5.0510000000000002,"5min":10.712999999999999,"15min":7.5990000000000002},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.47799999999999998},"max":{"1min":14.266999999999999,"5min":65.120000000000005,"15min":65.120000000000005},"last":4.0970000000000004},{"interface":"front","average":{"1min":5.3579999999999997,"5min":11.06,"15min":7.9379999999999997},"min":{"1min":0.79200000000000004,"5min":0.78400000000000003,"15min":0.69599999999999995},"max":{"1min":14.33,"5min":65.222999999999999,"15min":65.222999999999999},"last":4.2220000000000004}]},{"osd":2,"last update":"Fri Nov 20 11:52:00 2020","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.0609999999999999,"15min":1.1759999999999999},"min":{"1min":0.57499999999999996,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":2.1699999999999999,"5min":5.2380000000000004,"15min":22.370999999999999},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.2929999999999999,"15min":1.4399999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.52000000000000002},"max":{"1min":1.954,"5min":5.2130000000000001,"15min":8.8320000000000007},"last":3.0289999999999999}]},{"osd":4,"last update":"Fri Nov 20 11:52:00 2020","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.786,"15min":1.907},"min":{"1min":0.70799999999999996,"5min":0.70399999999999996,"15min":0.64100000000000001},"max":{"1min":1.8120000000000001,"5min":46.268000000000001,"15min":46.268000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3009999999999999,"15min":1.6399999999999999},"min":{"1min":0.91200000000000003,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.306,"5min":5.3659999999999997,"15min":40.478000000000002},"last":1.0940000000000001}]},{"osd":5,"last update":"Fri Nov 20 11:52:00 2020","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.9279999999999999,"15min":1.962},"min":{"1min":0.76400000000000001,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.8939999999999999,"5min":46.314999999999998,"15min":46.314999999999998},"last":0.92500000000000004},{"interface":"front","average":{"1min":1.21,"5min":1.369,"15min":1.9099999999999999},"min":{"1min":0.83499999999999996,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.9430000000000001,"5min":4.141,"15min":34.972999999999999},"last":0.84099999999999997}]},{"osd":6,"last update":"Fri Nov 20 11:52:00 2020","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.9890000000000001,"15min":1.782},"min":{"1min":1.0149999999999999,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":3.0259999999999998,"5min":46.219000000000001,"15min":46.219000000000001},"last":1.323},{"interface":"front","average":{"1min":1.4339999999999999,"5min":2.0680000000000001,"15min":1.9279999999999999},"min":{"1min":0.996,"5min":0.871,"15min":0.82699999999999996},"max":{"1min":2.085,"5min":46.469000000000001,"15min":46.469000000000001},"last":1.0509999999999999}]},{"osd":7,"last update":"Fri Nov 20 11:52:00 2020","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":2.0619999999999998,"15min":2.1070000000000002},"min":{"1min":1.016,"5min":0.91600000000000004,"15min":0.86799999999999999},"max":{"1min":2.069,"5min":46.432000000000002,"15min":46.432000000000002},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.587,"5min":2.129,"15min":1.9690000000000001},"min":{"1min":1.097,"5min":0.92200000000000004,"15min":0.91500000000000004},"max":{"1min":2.9169999999999998,"5min":46.348999999999997,"15min":46.348999999999997},"last":1.252}]}]},{"osd":4,"up_from":13,"seq":55834575091,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2035892,"kb_used_data":1778348,"kb_used_omap":0,"kb_used_meta":257536,"kb_avail":92335948,"statfs":{"total":96636764160,"available":94552010752,"internally_reserved":0,"allocated":1821028352,"data_stored":1820867325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":263716864},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.99199999999999999,"15min":1.6319999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":4.3280000000000003,"5min":9.1920000000000002,"15min":41.939999999999998},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.048,"15min":1.7430000000000001},"min":{"1min":0.57499999999999996,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":4.9480000000000004,"5min":5.5869999999999997,"15min":42.704999999999998},"last":1.335}]},{"osd":1,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":9.5839999999999996,"15min":7.2359999999999998},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.52500000000000002},"max":{"1min":21.244,"5min":39.244999999999997,"15min":76.847999999999999},"last":3.8029999999999999},{"interface":"front","average":{"1min":7.0979999999999999,"5min":9.9730000000000008,"15min":7.2450000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53200000000000003},"max":{"1min":20.898,"5min":39.472999999999999,"15min":164.745},"last":1.506}]},{"osd":2,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":1.0600000000000001,"15min":2.2549999999999999},"min":{"1min":0.621,"5min":0.53200000000000003,"15min":0.52800000000000002},"max":{"1min":2.02,"5min":4.681,"15min":269.16800000000001},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.92800000000000005,"5min":1.147,"15min":2.2269999999999999},"min":{"1min":0.69699999999999995,"5min":0.60599999999999998,"15min":0.56299999999999994},"max":{"1min":1.1839999999999999,"5min":5.6289999999999996,"15min":262.65300000000002},"last":1.202}]},{"osd":3,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.2010000000000001,"15min":1.742},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":3.427,"5min":10.930999999999999,"15min":40.585000000000001},"last":0.85699999999999998},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1890000000000001,"15min":1.6850000000000001},"min":{"1min":0.75,"5min":0.68200000000000005,"15min":0.59799999999999998},"max":{"1min":1.79,"5min":6.016,"15min":40.259999999999998},"last":1.619}]},{"osd":5,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.95299999999999996,"15min":1.3180000000000001},"min":{"1min":0.60399999999999998,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.4079999999999999,"5min":3.7189999999999999,"15min":18.222999999999999},"last":1.3600000000000001},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.2,"15min":1.3560000000000001},"min":{"1min":0.67800000000000005,"5min":0.61299999999999999,"15min":0.60299999999999998},"max":{"1min":1.95,"5min":6.1840000000000002,"15min":8.8239999999999998},"last":0.88100000000000001}]},{"osd":6,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.048,"15min":1.3640000000000001},"min":{"1min":0.78500000000000003,"5min":0.67800000000000005,"15min":0.63300000000000001},"max":{"1min":1.746,"5min":2.298,"15min":7.0819999999999999},"last":3.8439999999999999},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0609999999999999,"15min":1.359},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.67000000000000004},"max":{"1min":1.5369999999999999,"5min":1.7050000000000001,"15min":9.3200000000000003},"last":1.024}]},{"osd":7,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.1970000000000001,"15min":1.417},"min":{"1min":0.85999999999999999,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.8600000000000001,"5min":2.4390000000000001,"15min":6.9089999999999998},"last":1.085},{"interface":"front","average":{"1min":1.198,"5min":1.2070000000000001,"15min":1.573},"min":{"1min":0.98999999999999999,"5min":0.82499999999999996,"15min":0.77300000000000002},"max":{"1min":1.712,"5min":2.8700000000000001,"15min":9.4380000000000006},"last":3.4100000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575091,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3370636,"kb_used_data":2966404,"kb_used_omap":0,"kb_used_meta":404224,"kb_avail":91001204,"statfs":{"total":96636764160,"available":93185232896,"internally_reserved":0,"allocated":3037597696,"data_stored":3037436669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":413925376},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":6.1040000000000001,"5min":11.039999999999999,"15min":8.218},"min":{"1min":0.57499999999999996,"5min":0.497,"15min":0.47099999999999997},"max":{"1min":28.193000000000001,"5min":47.369,"15min":83.134},"last":37.985999999999997},{"interface":"front","average":{"1min":8.7210000000000001,"5min":11.254,"15min":8.5129999999999999},"min":{"1min":0.64500000000000002,"5min":0.51400000000000001,"15min":0.50700000000000001},"max":{"1min":39.725999999999999,"5min":47.856999999999999,"15min":82.665000000000006},"last":38.127000000000002}]},{"osd":2,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.95599999999999996,"15min":1.393},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.46700000000000003},"max":{"1min":1.901,"5min":4.7800000000000002,"15min":19.231000000000002},"last":0.79700000000000004},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.2310000000000001,"15min":1.6659999999999999},"min":{"1min":0.52300000000000002,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":6.5810000000000004,"5min":11.981,"15min":19.974},"last":0.77300000000000002}]},{"osd":3,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.139,"15min":1.6910000000000001},"min":{"1min":0.61599999999999999,"5min":0.51300000000000001,"15min":0.51200000000000001},"max":{"1min":2.7229999999999999,"5min":6.3440000000000003,"15min":41.634},"last":5.5890000000000004},{"interface":"front","average":{"1min":1.157,"5min":1.3240000000000001,"15min":2.0289999999999999},"min":{"1min":0.63400000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.657,"5min":6.3380000000000001,"15min":44.423000000000002},"last":5.5549999999999997}]},{"osd":4,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.258,"15min":1.8240000000000001},"min":{"1min":0.80100000000000005,"5min":0.73499999999999999,"15min":0.68899999999999995},"max":{"1min":1.782,"5min":4.3300000000000001,"15min":19.471},"last":1.857},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.319,"15min":1.665},"min":{"1min":0.91100000000000003,"5min":0.68200000000000005,"15min":0.68100000000000005},"max":{"1min":2.1619999999999999,"5min":6.7450000000000001,"15min":19.809000000000001},"last":2.0899999999999999}]},{"osd":5,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.3460000000000001,"15min":1.889},"min":{"1min":0.91300000000000003,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.7809999999999999,"5min":4.1879999999999997,"15min":20.457000000000001},"last":2.0499999999999998},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.425,"15min":1.853},"min":{"1min":0.82799999999999996,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":2.3839999999999999,"5min":6.633,"15min":20.565999999999999},"last":1.8879999999999999}]},{"osd":6,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.4630000000000001,"15min":1.8979999999999999},"min":{"1min":1.0900000000000001,"5min":0.77200000000000002,"15min":0.71599999999999997},"max":{"1min":2.254,"5min":6.5110000000000001,"15min":20.699000000000002},"last":2.1480000000000001},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.4890000000000001,"15min":1.895},"min":{"1min":1.022,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":2.0670000000000002,"5min":6.6219999999999999,"15min":20.609999999999999},"last":1.8280000000000001}]},{"osd":7,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.587,"15min":2.0489999999999999},"min":{"1min":1.119,"5min":0.90400000000000003,"15min":0.88400000000000001},"max":{"1min":2.1970000000000001,"5min":6.4950000000000001,"15min":20.495000000000001},"last":1.96},{"interface":"front","average":{"1min":1.518,"5min":1.5149999999999999,"15min":2.1829999999999998},"min":{"1min":1.0669999999999999,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":2.0190000000000001,"5min":5.0179999999999998,"15min":20.559000000000001},"last":2.1989999999999998}]}]},{"osd":7,"up_from":13,"seq":55834575091,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3314460,"kb_used_data":2894100,"kb_used_omap":0,"kb_used_meta":420352,"kb_avail":91057380,"statfs":{"total":96636764160,"available":93242757120,"internally_reserved":0,"allocated":2963558400,"data_stored":2963395919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":430440448},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 11:51:56 2020","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.92600000000000005,"15min":1.647},"min":{"1min":0.47499999999999998,"5min":0.44900000000000001,"15min":0.41799999999999998},"max":{"1min":1.2809999999999999,"5min":5.8170000000000002,"15min":43.341999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.872,"5min":0.96199999999999997,"15min":1.845},"min":{"1min":0.52100000000000002,"5min":0.49099999999999999,"15min":0.47299999999999998},"max":{"1min":1.7929999999999999,"5min":4.2809999999999997,"15min":42.759},"last":0.76400000000000001}]},{"osd":1,"last update":"Fri Nov 20 11:51:56 2020","interfaces":[{"interface":"back","average":{"1min":7.3440000000000003,"5min":10.259,"15min":7.4269999999999996},"min":{"1min":0.52300000000000002,"5min":0.52100000000000002,"15min":0.51800000000000002},"max":{"1min":36.731000000000002,"5min":45.369999999999997,"15min":47.488999999999997},"last":5.0640000000000001},{"interface":"front","average":{"1min":7.0300000000000002,"5min":10.272,"15min":7.375},"min":{"1min":0.71799999999999997,"5min":0.69099999999999995,"15min":0.55800000000000005},"max":{"1min":36.750999999999998,"5min":45.343000000000004,"15min":47.521999999999998},"last":5.399}]},{"osd":2,"last update":"Fri Nov 20 11:51:56 2020","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":1.0620000000000001,"15min":1.2170000000000001},"min":{"1min":0.58199999999999996,"5min":0.55900000000000005,"15min":0.53400000000000003},"max":{"1min":1.6799999999999999,"5min":3.9199999999999999,"15min":9.2040000000000006},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.96599999999999997,"15min":1.3280000000000001},"min":{"1min":0.59499999999999997,"5min":0.58999999999999997,"15min":0.56200000000000006},"max":{"1min":1.2669999999999999,"5min":4.1859999999999999,"15min":43.755000000000003},"last":1.052}]},{"osd":3,"last update":"Fri Nov 20 11:51:56 2020","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.206,"15min":1.837},"min":{"1min":0.79300000000000004,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.7549999999999999,"5min":4.3280000000000003,"15min":43.146999999999998},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.175,"5min":1.224,"15min":1.7849999999999999},"min":{"1min":0.79300000000000004,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":2.7789999999999999,"5min":4.3419999999999996,"15min":44.014000000000003},"last":1.137}]},{"osd":4,"last update":"Fri Nov 20 11:51:56 2020","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":0.99299999999999999,"15min":1.2909999999999999},"min":{"1min":0.67700000000000005,"5min":0.60699999999999998,"15min":0.59899999999999998},"max":{"1min":1.643,"5min":3.3460000000000001,"15min":9.9469999999999992},"last":0.64500000000000002},{"interface":"front","average":{"1min":1.04,"5min":0.96399999999999997,"15min":1.417},"min":{"1min":0.746,"5min":0.57499999999999996,"15min":0.57199999999999995},"max":{"1min":1.4379999999999999,"5min":1.8520000000000001,"15min":43.890999999999998},"last":0.69699999999999995}]},{"osd":5,"last update":"Fri Nov 20 11:51:56 2020","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.087,"15min":1.585},"min":{"1min":0.66900000000000004,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":2.327,"5min":4.2370000000000001,"15min":47.914999999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":1.298,"5min":1.2030000000000001,"15min":1.5269999999999999},"min":{"1min":0.88100000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":2.3530000000000002,"5min":7.1630000000000003,"15min":43.228000000000002},"last":1.208}]},{"osd":6,"last update":"Fri Nov 20 11:51:56 2020","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.1459999999999999,"15min":1.583},"min":{"1min":0.72699999999999998,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.895,"5min":7.0659999999999998,"15min":47.780000000000001},"last":1.071},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.2190000000000001,"15min":1.728},"min":{"1min":0.84799999999999998,"5min":0.76800000000000002,"15min":0.73199999999999998},"max":{"1min":1.958,"5min":3.4620000000000002,"15min":49.289999999999999},"last":1.1180000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575092,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2739892,"kb_used_data":2402796,"kb_used_omap":0,"kb_used_meta":335040,"kb_avail":91631948,"statfs":{"total":96636764160,"available":93831114752,"internally_reserved":0,"allocated":2460463104,"data_stored":2460300208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":343080960},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 11:51:41 2020","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.96099999999999997,"15min":1.657},"min":{"1min":0.47999999999999998,"5min":0.46700000000000003,"15min":0.40300000000000002},"max":{"1min":1.0429999999999999,"5min":5.1299999999999999,"15min":66.593999999999994},"last":11.228999999999999},{"interface":"front","average":{"1min":1.1279999999999999,"5min":0.96999999999999997,"15min":1.6499999999999999},"min":{"1min":0.49199999999999999,"5min":0.47899999999999998,"15min":0.45600000000000002},"max":{"1min":5.6239999999999997,"5min":5.6239999999999997,"15min":64.793000000000006},"last":11.215}]},{"osd":1,"last update":"Fri Nov 20 11:51:49 2020","interfaces":[{"interface":"back","average":{"1min":4.3719999999999999,"5min":12.068,"15min":7.2839999999999998},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.53300000000000003},"max":{"1min":16.989000000000001,"5min":71.489000000000004,"15min":71.489000000000004},"last":14.621},{"interface":"front","average":{"1min":4.3540000000000001,"5min":10.722,"15min":6.4649999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":17.050999999999998,"5min":72.109999999999999,"15min":72.109999999999999},"last":14.324}]},{"osd":2,"last update":"Fri Nov 20 11:51:49 2020","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":1.056,"15min":1.1699999999999999},"min":{"1min":0.63900000000000001,"5min":0.54900000000000004,"15min":0.53900000000000003},"max":{"1min":1.4410000000000001,"5min":4.9690000000000003,"15min":5.0490000000000004},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.1160000000000001,"15min":1.302},"min":{"1min":0.61299999999999999,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.496,"5min":4.9909999999999997,"15min":10.227},"last":0.84799999999999998}]},{"osd":3,"last update":"Fri Nov 20 11:51:49 2020","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":1.0820000000000001,"15min":1.5289999999999999},"min":{"1min":0.63300000000000001,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.431,"5min":4.0579999999999998,"15min":64.049999999999997},"last":0.82399999999999995},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.2110000000000001,"15min":1.4810000000000001},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.59299999999999997},"max":{"1min":3.5059999999999998,"5min":7.7110000000000003,"15min":63.718000000000004},"last":0.71599999999999997}]},{"osd":4,"last update":"Fri Nov 20 11:51:41 2020","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.99299999999999999,"15min":1.2010000000000001},"min":{"1min":0.65200000000000002,"5min":0.59299999999999997,"15min":0.56299999999999994},"max":{"1min":1.359,"5min":6.907,"15min":10.092000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":1.073,"5min":1.103,"15min":1.25},"min":{"1min":0.73499999999999999,"5min":0.66000000000000003,"15min":0.58499999999999996},"max":{"1min":1.5329999999999999,"5min":4.3280000000000003,"15min":6.1070000000000002},"last":0.91100000000000003}]},{"osd":6,"last update":"Fri Nov 20 11:51:49 2020","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.2090000000000001,"15min":1.456},"min":{"1min":0.76100000000000001,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.8120000000000001,"5min":6.3579999999999997,"15min":10.818},"last":0.75900000000000001},{"interface":"front","average":{"1min":1.216,"5min":1.218,"15min":1.347},"min":{"1min":0.81499999999999995,"5min":0.76700000000000002,"15min":0.68700000000000006},"max":{"1min":1.7929999999999999,"5min":4.1879999999999997,"15min":6.3280000000000003},"last":0.98399999999999999}]},{"osd":7,"last update":"Fri Nov 20 11:51:41 2020","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.1599999999999999,"15min":1.3},"min":{"1min":0.90700000000000003,"5min":0.73199999999999998,"15min":0.70599999999999996},"max":{"1min":1.5429999999999999,"5min":3.5539999999999998,"15min":5.2359999999999998},"last":0.94799999999999995},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.337,"15min":1.536},"min":{"1min":0.94899999999999995,"5min":0.876,"15min":0.78400000000000003},"max":{"1min":1.9490000000000001,"5min":4.5739999999999998,"15min":7.1929999999999996},"last":1.238}]}]},{"osd":1,"up_from":14,"seq":60129542384,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4090116,"kb_used_data":3607036,"kb_used_omap":0,"kb_used_meta":472832,"kb_avail":90281724,"statfs":{"total":96636764160,"available":92448485376,"internally_reserved":0,"allocated":3693604864,"data_stored":3693442451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":484179968},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 11:51:46 2020","interfaces":[{"interface":"back","average":{"1min":4.8540000000000001,"5min":5.6390000000000002,"15min":5.194},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.45900000000000002},"max":{"1min":24.831,"5min":70.593000000000004,"15min":70.593000000000004},"last":7.3979999999999997},{"interface":"front","average":{"1min":2.5369999999999999,"5min":4.7039999999999997,"15min":5.4210000000000003},"min":{"1min":0.70399999999999996,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":15.93,"5min":55.527000000000001,"15min":55.527000000000001},"last":0.85899999999999999}]},{"osd":2,"last update":"Fri Nov 20 11:51:46 2020","interfaces":[{"interface":"back","average":{"1min":2.8199999999999998,"5min":3.9830000000000001,"15min":3.782},"min":{"1min":0.66700000000000004,"5min":0.52600000000000002,"15min":0.47699999999999998},"max":{"1min":15.91,"5min":51.380000000000003,"15min":51.380000000000003},"last":0.89100000000000001},{"interface":"front","average":{"1min":5.569,"5min":8.7390000000000008,"15min":6.1740000000000004},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.56499999999999995},"max":{"1min":25.695,"5min":70.781999999999996,"15min":70.781999999999996},"last":3.976}]},{"osd":3,"last update":"Fri Nov 20 11:51:46 2020","interfaces":[{"interface":"back","average":{"1min":6.0220000000000002,"5min":9.3510000000000009,"15min":6.8819999999999997},"min":{"1min":0.748,"5min":0.748,"15min":0.61799999999999999},"max":{"1min":25.109999999999999,"5min":70.700999999999993,"15min":70.700999999999993},"last":3.1429999999999998},{"interface":"front","average":{"1min":6.0430000000000001,"5min":6.5910000000000002,"15min":5.6349999999999998},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.629},"max":{"1min":23.783000000000001,"5min":56.049999999999997,"15min":56.049999999999997},"last":1.2330000000000001}]},{"osd":4,"last update":"Fri Nov 20 11:51:46 2020","interfaces":[{"interface":"back","average":{"1min":5.7960000000000003,"5min":6.4189999999999996,"15min":5.165},"min":{"1min":0.90200000000000002,"5min":0.74299999999999999,"15min":0.73799999999999999},"max":{"1min":21.149999999999999,"5min":55.652000000000001,"15min":55.652000000000001},"last":1.101},{"interface":"front","average":{"1min":4.5759999999999996,"5min":5.2859999999999996,"15min":4.6829999999999998},"min":{"1min":0.91300000000000003,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":23.812000000000001,"5min":51.926000000000002,"15min":51.926000000000002},"last":1.1259999999999999}]},{"osd":5,"last update":"Fri Nov 20 11:51:46 2020","interfaces":[{"interface":"back","average":{"1min":6.1139999999999999,"5min":8.0169999999999995,"15min":5.8099999999999996},"min":{"1min":0.996,"5min":0.72199999999999998,"15min":0.69999999999999996},"max":{"1min":23.952999999999999,"5min":52.07,"15min":52.07},"last":1.2889999999999999},{"interface":"front","average":{"1min":7.0149999999999997,"5min":9.0860000000000003,"15min":6.7770000000000001},"min":{"1min":0.875,"5min":0.875,"15min":0.80800000000000005},"max":{"1min":24.981000000000002,"5min":52.262999999999998,"15min":52.262999999999998},"last":2.9119999999999999}]},{"osd":6,"last update":"Fri Nov 20 11:51:46 2020","interfaces":[{"interface":"back","average":{"1min":6.1840000000000002,"5min":7.4880000000000004,"15min":6.3049999999999997},"min":{"1min":1.167,"5min":1.087,"15min":0.88500000000000001},"max":{"1min":21.015999999999998,"5min":55.228000000000002,"15min":55.228000000000002},"last":1.389},{"interface":"front","average":{"1min":7.8600000000000003,"5min":9.5489999999999995,"15min":6.8339999999999996},"min":{"1min":1.1639999999999999,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":30.184999999999999,"5min":52.390000000000001,"15min":52.759},"last":1.4179999999999999}]},{"osd":7,"last update":"Fri Nov 20 11:51:46 2020","interfaces":[{"interface":"back","average":{"1min":7.3719999999999999,"5min":11.303000000000001,"15min":8.0090000000000003},"min":{"1min":1.018,"5min":1.018,"15min":0.96299999999999997},"max":{"1min":25.225999999999999,"5min":52.137999999999998,"15min":52.137999999999998},"last":3.032},{"interface":"front","average":{"1min":8.4309999999999992,"5min":10.443,"15min":7.6100000000000003},"min":{"1min":1.2849999999999999,"5min":1.214,"15min":0.97999999999999998},"max":{"1min":26.350000000000001,"5min":55.387999999999998,"15min":55.387999999999998},"last":1.5149999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542388,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1196512,"kb_used_data":1044760,"kb_used_omap":0,"kb_used_meta":151744,"kb_avail":93175328,"statfs":{"total":96636764160,"available":95411535872,"internally_reserved":0,"allocated":1069834240,"data_stored":1069671827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":155385856},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 11:51:55 2020","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.1639999999999999,"15min":1.7769999999999999},"min":{"1min":0.55900000000000005,"5min":0.503,"15min":0.436},"max":{"1min":4.5540000000000003,"5min":8.4770000000000003,"15min":68.239999999999995},"last":49.945999999999998},{"interface":"front","average":{"1min":1.018,"5min":1.2370000000000001,"15min":2.077},"min":{"1min":0.61099999999999999,"5min":0.52300000000000002,"15min":0.48999999999999999},"max":{"1min":2.3130000000000002,"5min":9.1310000000000002,"15min":68.804000000000002},"last":50.831000000000003}]},{"osd":1,"last update":"Fri Nov 20 11:51:55 2020","interfaces":[{"interface":"back","average":{"1min":6.2859999999999996,"5min":8.3379999999999992,"15min":6.9489999999999998},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.56299999999999994},"max":{"1min":31.527999999999999,"5min":53.654000000000003,"15min":72.744},"last":15.388},{"interface":"front","average":{"1min":6.5579999999999998,"5min":8.5719999999999992,"15min":7.2720000000000002},"min":{"1min":0.94699999999999995,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":31.486000000000001,"5min":53.222999999999999,"15min":73.438000000000002},"last":15.414999999999999}]},{"osd":3,"last update":"Fri Nov 20 11:51:55 2020","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":1.014,"15min":1.3180000000000001},"min":{"1min":0.58099999999999996,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":3.605,"5min":4.0339999999999998,"15min":21.465},"last":1.456},{"interface":"front","average":{"1min":1.413,"5min":1.3660000000000001,"15min":1.6439999999999999},"min":{"1min":0.77800000000000002,"5min":0.67800000000000005,"15min":0.48199999999999998},"max":{"1min":4.7050000000000001,"5min":5.0270000000000001,"15min":21.693000000000001},"last":1.6879999999999999}]},{"osd":4,"last update":"Fri Nov 20 11:51:55 2020","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.4239999999999999,"15min":1.4950000000000001},"min":{"1min":0.80900000000000005,"5min":0.67300000000000004,"15min":0.626},"max":{"1min":1.8759999999999999,"5min":4.6689999999999996,"15min":10.731999999999999},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.419,"5min":1.4179999999999999,"15min":1.5880000000000001},"min":{"1min":0.76200000000000001,"5min":0.72599999999999998,"15min":0.56499999999999995},"max":{"1min":4.5890000000000004,"5min":7.2699999999999996,"15min":9.6519999999999992},"last":1.121}]},{"osd":5,"last update":"Fri Nov 20 11:51:55 2020","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.369,"15min":1.4339999999999999},"min":{"1min":0.71699999999999997,"5min":0.70099999999999996,"15min":0.64100000000000001},"max":{"1min":4.7329999999999997,"5min":6.1859999999999999,"15min":12.045999999999999},"last":0.63},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.5629999999999999,"15min":1.7210000000000001},"min":{"1min":0.90200000000000002,"5min":0.82399999999999995,"15min":0.70299999999999996},"max":{"1min":2.0009999999999999,"5min":5.0620000000000003,"15min":11.856},"last":0.74199999999999999}]},{"osd":6,"last update":"Fri Nov 20 11:51:55 2020","interfaces":[{"interface":"back","average":{"1min":1.629,"5min":1.595,"15min":1.671},"min":{"1min":0.97899999999999998,"5min":0.81999999999999995,"15min":0.65100000000000002},"max":{"1min":5.0759999999999996,"5min":7.4130000000000003,"15min":7.4130000000000003},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.5880000000000001,"5min":1.5449999999999999,"15min":1.5760000000000001},"min":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.83899999999999997},"max":{"1min":4.8929999999999998,"5min":6.3209999999999997,"15min":7.5369999999999999},"last":0.93600000000000005}]},{"osd":7,"last update":"Fri Nov 20 11:51:55 2020","interfaces":[{"interface":"back","average":{"1min":1.675,"5min":1.734,"15min":1.7809999999999999},"min":{"1min":1.079,"5min":0.98999999999999999,"15min":0.85099999999999998},"max":{"1min":5.2240000000000002,"5min":7.54,"15min":7.54},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.6719999999999999,"5min":1.613,"15min":1.708},"min":{"1min":0.97599999999999998,"5min":0.89800000000000002,"15min":0.83799999999999997},"max":{"1min":5.04,"5min":6.4420000000000002,"15min":7.6909999999999998},"last":0.80900000000000005}]}]}]}} 2020-11-20T11:52:19.849 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-11-20T11:52:19.978 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.975+0000 7fa7ea0bb700 1 Processor -- start 2020-11-20T11:52:19.979 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.975+0000 7fa7ea0bb700 1 -- start start 2020-11-20T11:52:19.980 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.976+0000 7fa7ea0bb700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa7e40aaab0 0x7fa7e413ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:19.980 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.976+0000 7fa7ea0bb700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa7e40aae90 0x7fa7e41381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:19.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.976+0000 7fa7ea0bb700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa7e413aeb0 0x7fa7e40a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:19.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.976+0000 7fa7ea0bb700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7e413e600 con 0x7fa7e40aaab0 2020-11-20T11:52:19.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.976+0000 7fa7ea0bb700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7e413e8a0 con 0x7fa7e40aae90 2020-11-20T11:52:19.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.976+0000 7fa7ea0bb700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e40ab6d0 con 0x7fa7e413aeb0 2020-11-20T11:52:19.981 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.977+0000 7fa7e37fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa7e40aaab0 0x7fa7e413ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:19.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.977+0000 7fa7e37fe700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fa7e40aaab0 0x7fa7e413ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47180/0 (socket says 172.21.15.23:47180) 2020-11-20T11:52:19.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.977+0000 7fa7e37fe700 1 -- 172.21.15.23:0/2767759045 learned_addr learned my addr 172.21.15.23:0/2767759045 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:52:19.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.977+0000 7fa7e3fff700 1 --2- 172.21.15.23:0/2767759045 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa7e413aeb0 0x7fa7e40a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:19.982 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.977+0000 7fa7e2ffd700 1 --2- 172.21.15.23:0/2767759045 >> v2:172.21.15.155:3300/0 conn(0x7fa7e40aae90 0x7fa7e41381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:19.983 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.977+0000 7fa7e37fe700 1 -- 172.21.15.23:0/2767759045 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa7e413aeb0 msgr2=0x7fa7e40a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:52:19.983 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.977+0000 7fa7e37fe700 1 --2- 172.21.15.23:0/2767759045 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa7e413aeb0 0x7fa7e40a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:19.983 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.977+0000 7fa7e37fe700 1 -- 172.21.15.23:0/2767759045 >> v2:172.21.15.155:3300/0 conn(0x7fa7e40aae90 msgr2=0x7fa7e41381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:52:19.983 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.977+0000 7fa7e37fe700 1 --2- 172.21.15.23:0/2767759045 >> v2:172.21.15.155:3300/0 conn(0x7fa7e40aae90 0x7fa7e41381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:19.983 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.977+0000 7fa7e37fe700 1 -- 172.21.15.23:0/2767759045 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e413f230 con 0x7fa7e40aaab0 2020-11-20T11:52:19.984 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.977+0000 7fa7e3fff700 1 --2- 172.21.15.23:0/2767759045 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa7e413aeb0 0x7fa7e40a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:52:19.984 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.977+0000 7fa7e37fe700 1 --2- 172.21.15.23:0/2767759045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7e40aaab0 0x7fa7e413ed40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fa7e40ab930 tx=0x7fa7cc006ef0).ready entity=mon.0 client_cookie=aa545936660b6929 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:19.984 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.977+0000 7fa7e27fc700 1 -- 172.21.15.23:0/2767759045 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7cc00b860 con 0x7fa7e40aaab0 2020-11-20T11:52:19.984 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.977+0000 7fa7e27fc700 1 -- 172.21.15.23:0/2767759045 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7cc000e90 con 0x7fa7e40aaab0 2020-11-20T11:52:19.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.978+0000 7fa7ea0bb700 1 -- 172.21.15.23:0/2767759045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7e40aaab0 msgr2=0x7fa7e413ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:19.987 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.978+0000 7fa7ea0bb700 1 --2- 172.21.15.23:0/2767759045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7e40aaab0 0x7fa7e413ed40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fa7e40ab930 tx=0x7fa7cc006ef0).stop 2020-11-20T11:52:19.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.978+0000 7fa7ea0bb700 1 -- 172.21.15.23:0/2767759045 shutdown_connections 2020-11-20T11:52:19.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.978+0000 7fa7ea0bb700 1 --2- 172.21.15.23:0/2767759045 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa7e413aeb0 0x7fa7e40a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:19.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.978+0000 7fa7ea0bb700 1 --2- 172.21.15.23:0/2767759045 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7e40aaab0 0x7fa7e413ed40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:19.988 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.978+0000 7fa7ea0bb700 1 --2- 172.21.15.23:0/2767759045 >> v2:172.21.15.155:3300/0 conn(0x7fa7e40aae90 0x7fa7e41381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:19.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.978+0000 7fa7ea0bb700 1 -- 172.21.15.23:0/2767759045 >> 172.21.15.23:0/2767759045 conn(0x7fa7e40678e0 msgr2=0x7fa7e4067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:19.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.979+0000 7fa7ea0bb700 1 -- 172.21.15.23:0/2767759045 shutdown_connections 2020-11-20T11:52:19.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.979+0000 7fa7ea0bb700 1 -- 172.21.15.23:0/2767759045 wait complete. 2020-11-20T11:52:19.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.980+0000 7fa7ea0bb700 1 Processor -- start 2020-11-20T11:52:19.989 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.980+0000 7fa7ea0bb700 1 -- start start 2020-11-20T11:52:19.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.981+0000 7fa7ea0bb700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa7e40aaab0 0x7fa7e41412f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:19.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.981+0000 7fa7ea0bb700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa7e40aae90 0x7fa7e40a7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:19.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.981+0000 7fa7ea0bb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7e409f320 0x7fa7e40a6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:19.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.981+0000 7fa7ea0bb700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e40a7cf0 con 0x7fa7e409f320 2020-11-20T11:52:19.990 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.981+0000 7fa7ea0bb700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e40a66a0 con 0x7fa7e40aae90 2020-11-20T11:52:19.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.981+0000 7fa7ea0bb700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e40a7f10 con 0x7fa7e40aaab0 2020-11-20T11:52:19.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.981+0000 7fa7e3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7e409f320 0x7fa7e40a6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:19.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.981+0000 7fa7e3fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7e409f320 0x7fa7e40a6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47186/0 (socket says 172.21.15.23:47186) 2020-11-20T11:52:19.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.981+0000 7fa7e3fff700 1 -- 172.21.15.23:0/787598524 learned_addr learned my addr 172.21.15.23:0/787598524 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:52:19.991 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.981+0000 7fa7e2ffd700 1 --2- 172.21.15.23:0/787598524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa7e40aae90 0x7fa7e40a7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:19.992 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.981+0000 7fa7e37fe700 1 --2- 172.21.15.23:0/787598524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa7e40aaab0 0x7fa7e41412f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:19.992 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.982+0000 7fa7e3fff700 1 -- 172.21.15.23:0/787598524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa7e40aaab0 msgr2=0x7fa7e41412f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:19.992 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.982+0000 7fa7e3fff700 1 --2- 172.21.15.23:0/787598524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa7e40aaab0 0x7fa7e41412f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:19.992 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.982+0000 7fa7e3fff700 1 -- 172.21.15.23:0/787598524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa7e40aae90 msgr2=0x7fa7e40a7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:19.992 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.982+0000 7fa7e3fff700 1 --2- 172.21.15.23:0/787598524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa7e40aae90 0x7fa7e40a7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:19.993 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.982+0000 7fa7e3fff700 1 -- 172.21.15.23:0/787598524 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e40a81b0 con 0x7fa7e409f320 2020-11-20T11:52:19.993 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.982+0000 7fa7e3fff700 1 --2- 172.21.15.23:0/787598524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7e409f320 0x7fa7e40a6ad0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fa7dc00ef50 tx=0x7fa7dc009220).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:19.993 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.982+0000 7fa7db7fe700 1 -- 172.21.15.23:0/787598524 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7dc00cb80 con 0x7fa7e409f320 2020-11-20T11:52:19.993 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.982+0000 7fa7db7fe700 1 -- 172.21.15.23:0/787598524 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7dc00cce0 con 0x7fa7e409f320 2020-11-20T11:52:19.994 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.982+0000 7fa7ea0bb700 1 -- 172.21.15.23:0/787598524 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e40a6fc0 con 0x7fa7e409f320 2020-11-20T11:52:19.994 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.982+0000 7fa7db7fe700 1 -- 172.21.15.23:0/787598524 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7dc012620 con 0x7fa7e409f320 2020-11-20T11:52:19.994 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.983+0000 7fa7ea0bb700 1 -- 172.21.15.23:0/787598524 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e40a5480 con 0x7fa7e409f320 2020-11-20T11:52:19.994 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.984+0000 7fa7db7fe700 1 -- 172.21.15.23:0/787598524 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fa7dc012990 con 0x7fa7e409f320 2020-11-20T11:52:19.994 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.984+0000 7fa7ea0bb700 1 -- 172.21.15.23:0/787598524 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7e405c7d0 con 0x7fa7e409f320 2020-11-20T11:52:19.995 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.984+0000 7fa7db7fe700 1 --2- 172.21.15.23:0/787598524 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa7c0056d60 0x7fa7c0059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:19.995 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.984+0000 7fa7db7fe700 1 -- 172.21.15.23:0/787598524 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fa7dc06fe40 con 0x7fa7e409f320 2020-11-20T11:52:19.995 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.985+0000 7fa7e37fe700 1 --2- 172.21.15.23:0/787598524 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa7c0056d60 0x7fa7c0059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:19.996 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.985+0000 7fa7e37fe700 1 --2- 172.21.15.23:0/787598524 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa7c0056d60 0x7fa7c0059220 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7fa7cc006f30 tx=0x7fa7cc016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:19.996 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:19.992+0000 7fa7db7fe700 1 -- 172.21.15.23:0/787598524 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fa7dc075060 con 0x7fa7e409f320 2020-11-20T11:52:20.202 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:20.199+0000 7fa7ea0bb700 1 -- 172.21.15.23:0/787598524 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7e4044710 con 0x7fa7e409f320 2020-11-20T11:52:20.204 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:20.200+0000 7fa7db7fe700 1 -- 172.21.15.23:0/787598524 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+17087 (secure 0 0 0) 0x7fa7dc0450e0 con 0x7fa7e409f320 2020-11-20T11:52:20.204 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:52:20.204 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":25,"fsid":"4b071d14-8a39-4afe-8acc-4218fa4790c7","created":"2020-11-20T11:32:06.136679+0000","modified":"2020-11-20T11:32:26.470927+0000","last_up_change":"2020-11-20T11:32:13.536194+0000","last_in_change":"2020-11-20T11:32:13.536194+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-11-20T11:32:09.559868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-11-20T11:32:17.000782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2020-11-20T11:32:20.423341+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2020-11-20T11:32:23.126273+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e9d219f5-e40e-4b2b-9468-deb9c4074a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":23083},{"type":"v1","addr":"172.21.15.23:6801","nonce":23083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":23083},{"type":"v1","addr":"172.21.15.23:6803","nonce":23083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":23083},{"type":"v1","addr":"172.21.15.23:6807","nonce":23083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":23083},{"type":"v1","addr":"172.21.15.23:6805","nonce":23083}]},"public_addr":"172.21.15.23:6801/23083","cluster_addr":"172.21.15.23:6803/23083","heartbeat_back_addr":"172.21.15.23:6807/23083","heartbeat_front_addr":"172.21.15.23:6805/23083","state":["exists","up"]},{"osd":1,"uuid":"2328adde-2108-4fe0-8138-f0d096a496a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":23099},{"type":"v1","addr":"172.21.15.23:6817","nonce":23099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":23099},{"type":"v1","addr":"172.21.15.23:6819","nonce":23099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":23099},{"type":"v1","addr":"172.21.15.23:6824","nonce":23099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":23099},{"type":"v1","addr":"172.21.15.23:6821","nonce":23099}]},"public_addr":"172.21.15.23:6817/23099","cluster_addr":"172.21.15.23:6819/23099","heartbeat_back_addr":"172.21.15.23:6824/23099","heartbeat_front_addr":"172.21.15.23:6821/23099","state":["exists","up"]},{"osd":2,"uuid":"13367708-59a2-4363-8a3a-ab0cb97145ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":23084},{"type":"v1","addr":"172.21.15.23:6809","nonce":23084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":23084},{"type":"v1","addr":"172.21.15.23:6811","nonce":23084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":23084},{"type":"v1","addr":"172.21.15.23:6815","nonce":23084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":23084},{"type":"v1","addr":"172.21.15.23:6813","nonce":23084}]},"public_addr":"172.21.15.23:6809/23084","cluster_addr":"172.21.15.23:6811/23084","heartbeat_back_addr":"172.21.15.23:6815/23084","heartbeat_front_addr":"172.21.15.23:6813/23084","state":["exists","up"]},{"osd":3,"uuid":"1f147dd7-9289-47b0-ad1c-4088618d6faa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6823","nonce":23114},{"type":"v1","addr":"172.21.15.23:6825","nonce":23114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":23114},{"type":"v1","addr":"172.21.15.23:6827","nonce":23114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":23114},{"type":"v1","addr":"172.21.15.23:6831","nonce":23114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":23114},{"type":"v1","addr":"172.21.15.23:6829","nonce":23114}]},"public_addr":"172.21.15.23:6825/23114","cluster_addr":"172.21.15.23:6827/23114","heartbeat_back_addr":"172.21.15.23:6831/23114","heartbeat_front_addr":"172.21.15.23:6829/23114","state":["exists","up"]},{"osd":4,"uuid":"d008b039-4bc5-4135-9678-1511e58efc29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":23280},{"type":"v1","addr":"172.21.15.155:6801","nonce":23280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":23280},{"type":"v1","addr":"172.21.15.155:6803","nonce":23280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":23280},{"type":"v1","addr":"172.21.15.155:6807","nonce":23280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":23280},{"type":"v1","addr":"172.21.15.155:6805","nonce":23280}]},"public_addr":"172.21.15.155:6801/23280","cluster_addr":"172.21.15.155:6803/23280","heartbeat_back_addr":"172.21.15.155:6807/23280","heartbeat_front_addr":"172.21.15.155:6805/23280","state":["exists","up"]},{"osd":5,"uuid":"56715e6f-cf6e-44a6-9e39-93ab1224095f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":23298},{"type":"v1","addr":"172.21.15.155:6817","nonce":23298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":23298},{"type":"v1","addr":"172.21.15.155:6819","nonce":23298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6822","nonce":23298},{"type":"v1","addr":"172.21.15.155:6823","nonce":23298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":23298},{"type":"v1","addr":"172.21.15.155:6821","nonce":23298}]},"public_addr":"172.21.15.155:6817/23298","cluster_addr":"172.21.15.155:6819/23298","heartbeat_back_addr":"172.21.15.155:6823/23298","heartbeat_front_addr":"172.21.15.155:6821/23298","state":["exists","up"]},{"osd":6,"uuid":"db1e61ca-8e14-4a1f-8607-6e01b148d61c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":23307},{"type":"v1","addr":"172.21.15.155:6825","nonce":23307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":23307},{"type":"v1","addr":"172.21.15.155:6827","nonce":23307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":23307},{"type":"v1","addr":"172.21.15.155:6831","nonce":23307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6828","nonce":23307},{"type":"v1","addr":"172.21.15.155:6829","nonce":23307}]},"public_addr":"172.21.15.155:6825/23307","cluster_addr":"172.21.15.155:6827/23307","heartbeat_back_addr":"172.21.15.155:6831/23307","heartbeat_front_addr":"172.21.15.155:6829/23307","state":["exists","up"]},{"osd":7,"uuid":"d44419be-ef89-462c-8cf1-3c6db2a6944b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":23281},{"type":"v1","addr":"172.21.15.155:6809","nonce":23281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":23281},{"type":"v1","addr":"172.21.15.155:6811","nonce":23281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":23281},{"type":"v1","addr":"172.21.15.155:6815","nonce":23281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":23281},{"type":"v1","addr":"172.21.15.155:6813","nonce":23281}]},"public_addr":"172.21.15.155:6809/23281","cluster_addr":"172.21.15.155:6811/23281","heartbeat_back_addr":"172.21.15.155:6815/23281","heartbeat_front_addr":"172.21.15.155:6813/23281","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.486702+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.509396+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.533414+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.509489+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.467600+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.513068+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.536636+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-11-20T11:32:11.531772+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-11-20T11:52:20.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:20.202+0000 7fa7ea0bb700 1 -- 172.21.15.23:0/787598524 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa7c0056d60 msgr2=0x7fa7c0059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:20.206 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:20.202+0000 7fa7ea0bb700 1 --2- 172.21.15.23:0/787598524 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa7c0056d60 0x7fa7c0059220 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7fa7cc006f30 tx=0x7fa7cc016000).stop 2020-11-20T11:52:20.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:20.202+0000 7fa7ea0bb700 1 -- 172.21.15.23:0/787598524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7e409f320 msgr2=0x7fa7e40a6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:20.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:20.202+0000 7fa7ea0bb700 1 --2- 172.21.15.23:0/787598524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7e409f320 0x7fa7e40a6ad0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fa7dc00ef50 tx=0x7fa7dc009220).stop 2020-11-20T11:52:20.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:20.202+0000 7fa7ea0bb700 1 -- 172.21.15.23:0/787598524 shutdown_connections 2020-11-20T11:52:20.207 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:20.202+0000 7fa7ea0bb700 1 --2- 172.21.15.23:0/787598524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7e409f320 0x7fa7e40a6ad0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:20.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:20.202+0000 7fa7ea0bb700 1 --2- 172.21.15.23:0/787598524 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fa7c0056d60 0x7fa7c0059220 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:20.208 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:20.202+0000 7fa7ea0bb700 1 --2- 172.21.15.23:0/787598524 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa7e40aaab0 0x7fa7e41412f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:20.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:20.202+0000 7fa7ea0bb700 1 --2- 172.21.15.23:0/787598524 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa7e40aae90 0x7fa7e40a7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:20.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:20.202+0000 7fa7ea0bb700 1 -- 172.21.15.23:0/787598524 >> 172.21.15.23:0/787598524 conn(0x7fa7e40678e0 msgr2=0x7fa7e4139670 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:20.209 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:20.202+0000 7fa7ea0bb700 1 -- 172.21.15.23:0/787598524 shutdown_connections 2020-11-20T11:52:20.210 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:20.202+0000 7fa7ea0bb700 1 -- 172.21.15.23:0/787598524 wait complete. 2020-11-20T11:52:21.222 INFO:tasks.ceph:Scrubbing osd.0 2020-11-20T11:52:21.222 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2020-11-20T11:52:21.376 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.373+0000 7fab74027700 1 Processor -- start 2020-11-20T11:52:21.377 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.373+0000 7fab74027700 1 -- start start 2020-11-20T11:52:21.378 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.374+0000 7fab74027700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fab6c0ad5b0 0x7fab6c13b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:21.379 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.374+0000 7fab74027700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fab6c13b600 0x7fab6c13ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:21.379 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.374+0000 7fab74027700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab6c0a9570 con 0x7fab6c0b1b70 2020-11-20T11:52:21.379 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.374+0000 7fab74027700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab6c13bf40 con 0x7fab6c13b600 2020-11-20T11:52:21.379 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.374+0000 7fab74027700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab6c0aaad0 con 0x7fab6c0ad5b0 2020-11-20T11:52:21.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.374+0000 7fab725c4700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fab6c13b600 0x7fab6c13ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:21.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.374+0000 7fab725c4700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fab6c13b600 0x7fab6c13ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47198/0 (socket says 172.21.15.23:47198) 2020-11-20T11:52:21.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.374+0000 7fab725c4700 1 -- 172.21.15.23:0/1853051790 learned_addr learned my addr 172.21.15.23:0/1853051790 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:52:21.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.374+0000 7fab715c2700 1 --2- 172.21.15.23:0/1853051790 >> v2:172.21.15.155:3300/0 conn(0x7fab6c0ad5b0 0x7fab6c13b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:21.380 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.375+0000 7fab725c4700 1 -- 172.21.15.23:0/1853051790 >> v2:172.21.15.155:3300/0 conn(0x7fab6c0ad5b0 msgr2=0x7fab6c13b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:52:21.381 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.375+0000 7fab725c4700 1 --2- 172.21.15.23:0/1853051790 >> v2:172.21.15.155:3300/0 conn(0x7fab6c0ad5b0 0x7fab6c13b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:21.381 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.375+0000 7fab725c4700 1 -- 172.21.15.23:0/1853051790 >> v1:172.21.15.155:6789/0 conn(0x7fab6c0b1b70 legacy=0x7fab6c0ab170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:21.381 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.375+0000 7fab725c4700 1 -- 172.21.15.23:0/1853051790 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab6c0a97d0 con 0x7fab6c13b600 2020-11-20T11:52:21.381 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.375+0000 7fab725c4700 1 --2- 172.21.15.23:0/1853051790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab6c13b600 0x7fab6c13ba50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fab6c063310 tx=0x7fab640092d0).ready entity=mon.0 client_cookie=655a35c27597c1ee server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:21.381 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.375+0000 7fab70dc1700 1 -- 172.21.15.23:0/1853051790 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6400bbf0 con 0x7fab6c13b600 2020-11-20T11:52:21.382 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.375+0000 7fab70dc1700 1 -- 172.21.15.23:0/1853051790 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab6400bd50 con 0x7fab6c13b600 2020-11-20T11:52:21.382 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.375+0000 7fab70dc1700 1 -- 172.21.15.23:0/1853051790 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab64011610 con 0x7fab6c13b600 2020-11-20T11:52:21.382 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.375+0000 7fab74027700 1 -- 172.21.15.23:0/1853051790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab6c13b600 msgr2=0x7fab6c13ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:21.382 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.375+0000 7fab74027700 1 --2- 172.21.15.23:0/1853051790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab6c13b600 0x7fab6c13ba50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fab6c063310 tx=0x7fab640092d0).stop 2020-11-20T11:52:21.383 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.375+0000 7fab74027700 1 -- 172.21.15.23:0/1853051790 shutdown_connections 2020-11-20T11:52:21.383 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.375+0000 7fab74027700 1 --2- 172.21.15.23:0/1853051790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab6c13b600 0x7fab6c13ba50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:21.383 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.375+0000 7fab74027700 1 --2- 172.21.15.23:0/1853051790 >> v2:172.21.15.155:3300/0 conn(0x7fab6c0ad5b0 0x7fab6c13b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:21.383 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.375+0000 7fab74027700 1 -- 172.21.15.23:0/1853051790 >> 172.21.15.23:0/1853051790 conn(0x7fab6c0683b0 msgr2=0x7fab6c0b12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:21.383 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.375+0000 7fab74027700 1 -- 172.21.15.23:0/1853051790 shutdown_connections 2020-11-20T11:52:21.384 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.375+0000 7fab74027700 1 -- 172.21.15.23:0/1853051790 wait complete. 2020-11-20T11:52:21.385 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.376+0000 7fab74027700 1 Processor -- start 2020-11-20T11:52:21.385 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.376+0000 7fab74027700 1 -- start start 2020-11-20T11:52:21.386 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.377+0000 7fab74027700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fab6c0ad5b0 0x7fab6c149d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:21.386 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.377+0000 7fab74027700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab6c0b1b70 0x7fab6c0a0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:21.386 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.377+0000 7fab74027700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab6c13b600 0x7fab6c09e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:21.386 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.377+0000 7fab74027700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6c13a620 con 0x7fab6c13b600 2020-11-20T11:52:21.387 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.377+0000 7fab74027700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6c0a4400 con 0x7fab6c0b1b70 2020-11-20T11:52:21.387 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.377+0000 7fab74027700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab6c14a280 con 0x7fab6c0ad5b0 2020-11-20T11:52:21.387 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.377+0000 7fab725c4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab6c13b600 0x7fab6c09e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:21.387 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.377+0000 7fab725c4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab6c13b600 0x7fab6c09e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47204/0 (socket says 172.21.15.23:47204) 2020-11-20T11:52:21.387 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.377+0000 7fab725c4700 1 -- 172.21.15.23:0/3254241975 learned_addr learned my addr 172.21.15.23:0/3254241975 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:52:21.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.377+0000 7fab71dc3700 1 --2- 172.21.15.23:0/3254241975 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fab6c0ad5b0 0x7fab6c149d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:21.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.377+0000 7fab715c2700 1 --2- 172.21.15.23:0/3254241975 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab6c0b1b70 0x7fab6c0a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:21.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.377+0000 7fab725c4700 1 -- 172.21.15.23:0/3254241975 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fab6c0ad5b0 msgr2=0x7fab6c149d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:21.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.377+0000 7fab725c4700 1 --2- 172.21.15.23:0/3254241975 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fab6c0ad5b0 0x7fab6c149d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:21.388 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.377+0000 7fab725c4700 1 -- 172.21.15.23:0/3254241975 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab6c0b1b70 msgr2=0x7fab6c0a0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:21.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.377+0000 7fab725c4700 1 --2- 172.21.15.23:0/3254241975 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fab6c0b1b70 0x7fab6c0a0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:21.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.377+0000 7fab725c4700 1 -- 172.21.15.23:0/3254241975 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab6c14a510 con 0x7fab6c13b600 2020-11-20T11:52:21.389 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.378+0000 7fab725c4700 1 --2- 172.21.15.23:0/3254241975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab6c13b600 0x7fab6c09e350 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fab6c068820 tx=0x7fab64016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:21.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.384+0000 7fab59ffb700 1 -- 172.21.15.23:0/3254241975 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6401b5e0 con 0x7fab6c13b600 2020-11-20T11:52:21.390 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.384+0000 7fab59ffb700 1 -- 172.21.15.23:0/3254241975 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab6401b740 con 0x7fab6c13b600 2020-11-20T11:52:21.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.384+0000 7fab59ffb700 1 -- 172.21.15.23:0/3254241975 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6401bac0 con 0x7fab6c13b600 2020-11-20T11:52:21.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.385+0000 7fab74027700 1 -- 172.21.15.23:0/3254241975 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab6c14a7b0 con 0x7fab6c13b600 2020-11-20T11:52:21.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.385+0000 7fab74027700 1 -- 172.21.15.23:0/3254241975 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab6c0a0a80 con 0x7fab6c13b600 2020-11-20T11:52:21.391 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.386+0000 7fab59ffb700 1 -- 172.21.15.23:0/3254241975 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fab64017070 con 0x7fab6c13b600 2020-11-20T11:52:21.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.386+0000 7fab59ffb700 1 --2- 172.21.15.23:0/3254241975 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fab44056d60 0x7fab44059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:21.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.386+0000 7fab59ffb700 1 -- 172.21.15.23:0/3254241975 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fab64071fe0 con 0x7fab6c13b600 2020-11-20T11:52:21.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.388+0000 7fab74027700 1 --2- 172.21.15.23:0/3254241975 >> [v2:172.21.15.23:6800/23083,v1:172.21.15.23:6801/23083] conn(0x7fab6c09e840 0x7fab6c09da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:21.392 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.388+0000 7fab74027700 1 -- 172.21.15.23:0/3254241975 --> [v2:172.21.15.23:6800/23083,v1:172.21.15.23:6801/23083] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab6c14ac50 con 0x7fab6c09e840 2020-11-20T11:52:21.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.388+0000 7fab71dc3700 1 --2- 172.21.15.23:0/3254241975 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fab44056d60 0x7fab44059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:21.393 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.388+0000 7fab71dc3700 1 --2- 172.21.15.23:0/3254241975 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fab44056d60 0x7fab44059220 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fab5c0037d0 tx=0x7fab5c007bc0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:21.423 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.418+0000 7fab715c2700 1 --2- 172.21.15.23:0/3254241975 >> [v2:172.21.15.23:6800/23083,v1:172.21.15.23:6801/23083] conn(0x7fab6c09e840 0x7fab6c09da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:21.423 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.419+0000 7fab715c2700 1 --2- 172.21.15.23:0/3254241975 >> [v2:172.21.15.23:6800/23083,v1:172.21.15.23:6801/23083] conn(0x7fab6c09e840 0x7fab6c09da60 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:21.430 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.427+0000 7fab59ffb700 1 -- 172.21.15.23:0/3254241975 <== osd.0 v2:172.21.15.23:6800/23083 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7fab6800bb60 con 0x7fab6c09e840 2020-11-20T11:52:21.449 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.446+0000 7fab74027700 1 -- 172.21.15.23:0/3254241975 --> [v2:172.21.15.23:6800/23083,v1:172.21.15.23:6801/23083] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fab6c14d620 con 0x7fab6c09e840 2020-11-20T11:52:21.450 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.447+0000 7fab59ffb700 1 -- 172.21.15.23:0/3254241975 <== osd.0 v2:172.21.15.23:6800/23083 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fab6c14d620 con 0x7fab6c09e840 2020-11-20T11:52:21.451 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:52:21.451 INFO:teuthology.orchestra.run.smithi023.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2020-11-20T11:52:21.452 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:52:21.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.449+0000 7fab74027700 1 -- 172.21.15.23:0/3254241975 >> [v2:172.21.15.23:6800/23083,v1:172.21.15.23:6801/23083] conn(0x7fab6c09e840 msgr2=0x7fab6c09da60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:21.453 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.449+0000 7fab74027700 1 --2- 172.21.15.23:0/3254241975 >> [v2:172.21.15.23:6800/23083,v1:172.21.15.23:6801/23083] conn(0x7fab6c09e840 0x7fab6c09da60 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:21.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.452+0000 7fab74027700 1 -- 172.21.15.23:0/3254241975 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fab44056d60 msgr2=0x7fab44059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:21.455 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.452+0000 7fab74027700 1 --2- 172.21.15.23:0/3254241975 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fab44056d60 0x7fab44059220 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fab5c0037d0 tx=0x7fab5c007bc0).stop 2020-11-20T11:52:21.456 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.452+0000 7fab74027700 1 -- 172.21.15.23:0/3254241975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab6c13b600 msgr2=0x7fab6c09e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:21.456 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.452+0000 7fab74027700 1 --2- 172.21.15.23:0/3254241975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab6c13b600 0x7fab6c09e350 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fab6c068820 tx=0x7fab64016000).stop 2020-11-20T11:52:21.456 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.452+0000 7fab725c4700 1 -- 172.21.15.23:0/3254241975 reap_dead start 2020-11-20T11:52:21.456 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.452+0000 7fab74027700 1 -- 172.21.15.23:0/3254241975 shutdown_connections 2020-11-20T11:52:21.457 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.452+0000 7fab74027700 1 -- 172.21.15.23:0/3254241975 >> 172.21.15.23:0/3254241975 conn(0x7fab6c0683b0 msgr2=0x7fab6c0ab090 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:21.457 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.452+0000 7fab74027700 1 -- 172.21.15.23:0/3254241975 shutdown_connections 2020-11-20T11:52:21.457 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.452+0000 7fab74027700 1 -- 172.21.15.23:0/3254241975 wait complete. 2020-11-20T11:52:21.469 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2020-11-20T11:52:21.624 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.621+0000 7f60903fa700 1 Processor -- start 2020-11-20T11:52:21.625 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.621+0000 7f60903fa700 1 -- start start 2020-11-20T11:52:21.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.622+0000 7f60903fa700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6088060a80 0x7f608809d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:21.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.622+0000 7f60903fa700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6088067fc0 0x7f60880abbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:21.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.622+0000 7f60903fa700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60880a9b40 con 0x7f608809ded0 2020-11-20T11:52:21.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.622+0000 7f60903fa700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60880aacb0 con 0x7f6088067fc0 2020-11-20T11:52:21.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.622+0000 7f60903fa700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60880a9da0 con 0x7f6088060a80 2020-11-20T11:52:21.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.622+0000 7f608e997700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6088067fc0 0x7f60880abbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:21.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.622+0000 7f608e196700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6088060a80 0x7f608809d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:21.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.622+0000 7f608e997700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6088067fc0 0x7f60880abbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47214/0 (socket says 172.21.15.23:47214) 2020-11-20T11:52:21.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.622+0000 7f608e997700 1 -- 172.21.15.23:0/1761861093 learned_addr learned my addr 172.21.15.23:0/1761861093 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:52:21.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.622+0000 7f608e196700 1 --2- 172.21.15.23:0/1761861093 >> v2:172.21.15.155:3300/0 conn(0x7f6088060a80 0x7f608809d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:40106/0 (socket says 172.21.15.23:40106) 2020-11-20T11:52:21.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.622+0000 7f608d995700 1 --1- 172.21.15.23:0/1761861093 >> v1:172.21.15.155:6789/0 conn(0x7f608809ded0 0x7f60880678b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:39436/0 (socket says 172.21.15.23:39436) 2020-11-20T11:52:21.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.623+0000 7f608e196700 1 -- 172.21.15.23:0/1761861093 >> v2:172.21.15.23:3300/0 conn(0x7f6088067fc0 msgr2=0x7f60880abbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:52:21.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.623+0000 7f608e196700 1 --2- 172.21.15.23:0/1761861093 >> v2:172.21.15.23:3300/0 conn(0x7f6088067fc0 0x7f60880abbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:21.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.623+0000 7f608e196700 1 -- 172.21.15.23:0/1761861093 >> v1:172.21.15.155:6789/0 conn(0x7f608809ded0 legacy=0x7f60880678b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:21.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.623+0000 7f608d194700 1 -- 172.21.15.23:0/1761861093 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079656797 0 0) 0x7f60880a9b40 con 0x7f608809ded0 2020-11-20T11:52:21.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.623+0000 7f608e196700 1 -- 172.21.15.23:0/1761861093 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60880aa040 con 0x7f6088060a80 2020-11-20T11:52:21.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.623+0000 7f608e196700 1 --2- 172.21.15.23:0/1761861093 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6088060a80 0x7f608809d9e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f60880b01e0 tx=0x7f6084006e30).ready entity=mon.1 client_cookie=9199cecc4b4acd82 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:21.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.623+0000 7f608d194700 1 -- 172.21.15.23:0/1761861093 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60840097b0 con 0x7f6088060a80 2020-11-20T11:52:21.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.623+0000 7f608d194700 1 -- 172.21.15.23:0/1761861093 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6084000e90 con 0x7f6088060a80 2020-11-20T11:52:21.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.623+0000 7f608d194700 1 -- 172.21.15.23:0/1761861093 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6084009b20 con 0x7f6088060a80 2020-11-20T11:52:21.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.624+0000 7f60903fa700 1 -- 172.21.15.23:0/1761861093 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6088060a80 msgr2=0x7f608809d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:21.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.624+0000 7f60903fa700 1 --2- 172.21.15.23:0/1761861093 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6088060a80 0x7f608809d9e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f60880b01e0 tx=0x7f6084006e30).stop 2020-11-20T11:52:21.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.624+0000 7f60903fa700 1 -- 172.21.15.23:0/1761861093 shutdown_connections 2020-11-20T11:52:21.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.624+0000 7f60903fa700 1 --2- 172.21.15.23:0/1761861093 >> v2:172.21.15.23:3300/0 conn(0x7f6088067fc0 0x7f60880abbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:21.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.624+0000 7f60903fa700 1 --2- 172.21.15.23:0/1761861093 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6088060a80 0x7f608809d9e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:21.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.624+0000 7f60903fa700 1 -- 172.21.15.23:0/1761861093 >> 172.21.15.23:0/1761861093 conn(0x7f6088066730 msgr2=0x7f60880601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:21.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.624+0000 7f60903fa700 1 -- 172.21.15.23:0/1761861093 shutdown_connections 2020-11-20T11:52:21.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.624+0000 7f60903fa700 1 -- 172.21.15.23:0/1761861093 wait complete. 2020-11-20T11:52:21.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.625+0000 7f60903fa700 1 Processor -- start 2020-11-20T11:52:21.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.625+0000 7f60903fa700 1 -- start start 2020-11-20T11:52:21.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.625+0000 7f60903fa700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6088060a80 0x7f60881341d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:21.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.625+0000 7f60903fa700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6088067fc0 0x7f60880a7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:21.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.625+0000 7f60903fa700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f608809ded0 0x7f60880a6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:21.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.625+0000 7f60903fa700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60880a90c0 con 0x7f608809ded0 2020-11-20T11:52:21.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.625+0000 7f60903fa700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60880a92f0 con 0x7f6088060a80 2020-11-20T11:52:21.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.625+0000 7f60903fa700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60880a45f0 con 0x7f6088067fc0 2020-11-20T11:52:21.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.626+0000 7f608e997700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f608809ded0 0x7f60880a6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:21.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.626+0000 7f608e997700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f608809ded0 0x7f60880a6520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47220/0 (socket says 172.21.15.23:47220) 2020-11-20T11:52:21.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.626+0000 7f608e997700 1 -- 172.21.15.23:0/83130181 learned_addr learned my addr 172.21.15.23:0/83130181 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:52:21.639 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.626+0000 7f608e196700 1 --2- 172.21.15.23:0/83130181 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6088060a80 0x7f60881341d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:21.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.626+0000 7f608d995700 1 --2- 172.21.15.23:0/83130181 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6088067fc0 0x7f60880a7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:21.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.626+0000 7f608e997700 1 -- 172.21.15.23:0/83130181 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6088067fc0 msgr2=0x7f60880a7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:21.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.626+0000 7f608e997700 1 --2- 172.21.15.23:0/83130181 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6088067fc0 0x7f60880a7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:21.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.626+0000 7f608e997700 1 -- 172.21.15.23:0/83130181 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6088060a80 msgr2=0x7f60881341d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:21.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.626+0000 7f608e997700 1 --2- 172.21.15.23:0/83130181 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6088060a80 0x7f60881341d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:21.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.626+0000 7f608e997700 1 -- 172.21.15.23:0/83130181 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60880a4890 con 0x7f608809ded0 2020-11-20T11:52:21.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.626+0000 7f608e997700 1 --2- 172.21.15.23:0/83130181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f608809ded0 0x7f60880a6520 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f608000ed10 tx=0x7f60800092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:21.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.626+0000 7f60767fc700 1 -- 172.21.15.23:0/83130181 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608000ee70 con 0x7f608809ded0 2020-11-20T11:52:21.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.626+0000 7f60767fc700 1 -- 172.21.15.23:0/83130181 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f608000cb30 con 0x7f608809ded0 2020-11-20T11:52:21.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.626+0000 7f60903fa700 1 -- 172.21.15.23:0/83130181 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60880a4d30 con 0x7f608809ded0 2020-11-20T11:52:21.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.626+0000 7f60767fc700 1 -- 172.21.15.23:0/83130181 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60800123f0 con 0x7f608809ded0 2020-11-20T11:52:21.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.626+0000 7f60903fa700 1 -- 172.21.15.23:0/83130181 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6088134e60 con 0x7f608809ded0 2020-11-20T11:52:21.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.628+0000 7f60903fa700 1 -- 172.21.15.23:0/83130181 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6088132930 con 0x7f608809ded0 2020-11-20T11:52:21.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.628+0000 7f60767fc700 1 -- 172.21.15.23:0/83130181 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6080022070 con 0x7f608809ded0 2020-11-20T11:52:21.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.628+0000 7f60767fc700 1 --2- 172.21.15.23:0/83130181 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6060056d60 0x7f6060059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:21.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.628+0000 7f60767fc700 1 -- 172.21.15.23:0/83130181 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f60800707f0 con 0x7f608809ded0 2020-11-20T11:52:21.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.628+0000 7f608e196700 1 --2- 172.21.15.23:0/83130181 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6060056d60 0x7f6060059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:21.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.629+0000 7f608e196700 1 --2- 172.21.15.23:0/83130181 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6060056d60 0x7f6060059220 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f6084006e70 tx=0x7f6084015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:21.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.632+0000 7f60767fc700 1 -- 172.21.15.23:0/83130181 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f60800459d0 con 0x7f608809ded0 2020-11-20T11:52:21.827 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.824+0000 7f60903fa700 1 -- 172.21.15.23:0/83130181 --> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f60880601b0 con 0x7f6060056d60 2020-11-20T11:52:21.828 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.825+0000 7f60767fc700 1 -- 172.21.15.23:0/83130181 <== mgr.4105 v2:172.21.15.155:6832/22693 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f60880601b0 con 0x7f6060056d60 2020-11-20T11:52:21.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.827+0000 7f60903fa700 1 -- 172.21.15.23:0/83130181 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6060056d60 msgr2=0x7f6060059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:21.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.827+0000 7f60903fa700 1 --2- 172.21.15.23:0/83130181 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6060056d60 0x7f6060059220 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f6084006e70 tx=0x7f6084015000).stop 2020-11-20T11:52:21.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.827+0000 7f60903fa700 1 -- 172.21.15.23:0/83130181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f608809ded0 msgr2=0x7f60880a6520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:21.830 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.827+0000 7f60903fa700 1 --2- 172.21.15.23:0/83130181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f608809ded0 0x7f60880a6520 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f608000ed10 tx=0x7f60800092d0).stop 2020-11-20T11:52:21.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.827+0000 7f60903fa700 1 -- 172.21.15.23:0/83130181 shutdown_connections 2020-11-20T11:52:21.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.827+0000 7f60903fa700 1 --2- 172.21.15.23:0/83130181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f608809ded0 0x7f60880a6520 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:21.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.827+0000 7f60903fa700 1 --2- 172.21.15.23:0/83130181 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6088060a80 0x7f60881341d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:21.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.827+0000 7f60903fa700 1 --2- 172.21.15.23:0/83130181 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6060056d60 0x7f6060059220 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:21.831 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.827+0000 7f60903fa700 1 --2- 172.21.15.23:0/83130181 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6088067fc0 0x7f60880a7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:21.832 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.827+0000 7f60903fa700 1 -- 172.21.15.23:0/83130181 >> 172.21.15.23:0/83130181 conn(0x7f6088066730 msgr2=0x7f6088068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:21.832 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.827+0000 7f60903fa700 1 -- 172.21.15.23:0/83130181 shutdown_connections 2020-11-20T11:52:21.832 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.827+0000 7f60903fa700 1 -- 172.21.15.23:0/83130181 wait complete. 2020-11-20T11:52:21.833 INFO:teuthology.orchestra.run.smithi023.stderr:instructed osd(s) 0 to deep-scrub 2020-11-20T11:52:21.844 INFO:tasks.ceph:Scrubbing osd.1 2020-11-20T11:52:21.845 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2020-11-20T11:52:21.999 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.995+0000 7fe28b487700 1 Processor -- start 2020-11-20T11:52:21.999 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.995+0000 7fe28b487700 1 -- start start 2020-11-20T11:52:22.004 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.996+0000 7fe28b487700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2840aab80 0x7fe2840abad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:22.005 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.996+0000 7fe28b487700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe2840aaf60 0x7fe28413b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:22.005 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.996+0000 7fe28b487700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe28413aad0 con 0x7fe2840a8570 2020-11-20T11:52:22.005 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.996+0000 7fe28b487700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2840a9e00 con 0x7fe2840aaf60 2020-11-20T11:52:22.005 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.996+0000 7fe28b487700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe28413ad70 con 0x7fe2840aab80 2020-11-20T11:52:22.006 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.996+0000 7fe289c84700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe2840aaf60 0x7fe28413b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:22.006 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.996+0000 7fe289c84700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fe2840aaf60 0x7fe28413b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47224/0 (socket says 172.21.15.23:47224) 2020-11-20T11:52:22.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.996+0000 7fe289c84700 1 -- 172.21.15.23:0/1372666384 learned_addr learned my addr 172.21.15.23:0/1372666384 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:52:22.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.997+0000 7fe28a485700 1 --2- 172.21.15.23:0/1372666384 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2840aab80 0x7fe2840abad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:22.007 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.997+0000 7fe289c84700 1 -- 172.21.15.23:0/1372666384 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2840aab80 msgr2=0x7fe2840abad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:52:22.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.997+0000 7fe289c84700 1 --2- 172.21.15.23:0/1372666384 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2840aab80 0x7fe2840abad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:22.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.997+0000 7fe289c84700 1 -- 172.21.15.23:0/1372666384 >> v1:172.21.15.155:6789/0 conn(0x7fe2840a8570 legacy=0x7fe2840ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:22.008 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.997+0000 7fe289c84700 1 -- 172.21.15.23:0/1372666384 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe28413b010 con 0x7fe2840aaf60 2020-11-20T11:52:22.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.997+0000 7fe28a485700 1 --2- 172.21.15.23:0/1372666384 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2840aab80 0x7fe2840abad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:52:22.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.997+0000 7fe289c84700 1 --2- 172.21.15.23:0/1372666384 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2840aaf60 0x7fe28413b4b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fe274009a10 tx=0x7fe274006ef0).ready entity=mon.0 client_cookie=d760637914c71963 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:22.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.997+0000 7fe289483700 1 -- 172.21.15.23:0/1372666384 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27400b860 con 0x7fe2840aaf60 2020-11-20T11:52:22.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.997+0000 7fe289483700 1 -- 172.21.15.23:0/1372666384 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe274000e90 con 0x7fe2840aaf60 2020-11-20T11:52:22.009 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.997+0000 7fe289483700 1 -- 172.21.15.23:0/1372666384 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27400bbd0 con 0x7fe2840aaf60 2020-11-20T11:52:22.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.997+0000 7fe28b487700 1 -- 172.21.15.23:0/1372666384 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2840aaf60 msgr2=0x7fe28413b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:22.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.998+0000 7fe28b487700 1 --2- 172.21.15.23:0/1372666384 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2840aaf60 0x7fe28413b4b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fe274009a10 tx=0x7fe274006ef0).stop 2020-11-20T11:52:22.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.998+0000 7fe28b487700 1 -- 172.21.15.23:0/1372666384 shutdown_connections 2020-11-20T11:52:22.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.998+0000 7fe28b487700 1 --2- 172.21.15.23:0/1372666384 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2840aab80 0x7fe2840abad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:22.010 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.998+0000 7fe28b487700 1 --2- 172.21.15.23:0/1372666384 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2840aaf60 0x7fe28413b4b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:22.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.998+0000 7fe28b487700 1 -- 172.21.15.23:0/1372666384 >> 172.21.15.23:0/1372666384 conn(0x7fe2840678e0 msgr2=0x7fe284067cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:22.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.998+0000 7fe28b487700 1 -- 172.21.15.23:0/1372666384 shutdown_connections 2020-11-20T11:52:22.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.998+0000 7fe28b487700 1 -- 172.21.15.23:0/1372666384 wait complete. 2020-11-20T11:52:22.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.999+0000 7fe28b487700 1 Processor -- start 2020-11-20T11:52:22.011 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.999+0000 7fe28b487700 1 -- start start 2020-11-20T11:52:22.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.999+0000 7fe28b487700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2840a8570 0x7fe2840b0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:22.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:21.999+0000 7fe28b487700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2840aab80 0x7fe28414d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:22.012 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.000+0000 7fe289c84700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2840aab80 0x7fe28414d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:22.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.000+0000 7fe289c84700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2840aab80 0x7fe28414d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47232/0 (socket says 172.21.15.23:47232) 2020-11-20T11:52:22.013 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.000+0000 7fe28b487700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2840aaf60 0x7fe284135080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:22.014 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.000+0000 7fe28a485700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2840a8570 0x7fe2840b0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:22.014 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.000+0000 7fe289c84700 1 -- 172.21.15.23:0/2634312542 learned_addr learned my addr 172.21.15.23:0/2634312542 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:52:22.014 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.000+0000 7fe28b487700 1 -- 172.21.15.23:0/2634312542 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28413c660 con 0x7fe2840aab80 2020-11-20T11:52:22.015 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.000+0000 7fe28b487700 1 -- 172.21.15.23:0/2634312542 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28413cfd0 con 0x7fe2840a8570 2020-11-20T11:52:22.015 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.000+0000 7fe28b487700 1 -- 172.21.15.23:0/2634312542 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2840a4750 con 0x7fe2840aaf60 2020-11-20T11:52:22.016 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.000+0000 7fe28ac86700 1 --2- 172.21.15.23:0/2634312542 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2840aaf60 0x7fe284135080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:22.019 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.000+0000 7fe289c84700 1 -- 172.21.15.23:0/2634312542 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2840aaf60 msgr2=0x7fe284135080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:22.019 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.000+0000 7fe289c84700 1 --2- 172.21.15.23:0/2634312542 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe2840aaf60 0x7fe284135080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:22.020 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.000+0000 7fe289c84700 1 -- 172.21.15.23:0/2634312542 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2840a8570 msgr2=0x7fe2840b0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:22.020 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.000+0000 7fe289c84700 1 --2- 172.21.15.23:0/2634312542 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe2840a8570 0x7fe2840b0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:22.020 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.000+0000 7fe289c84700 1 -- 172.21.15.23:0/2634312542 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe28413c890 con 0x7fe2840aab80 2020-11-20T11:52:22.021 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.009+0000 7fe289c84700 1 --2- 172.21.15.23:0/2634312542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2840aab80 0x7fe28414d940 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fe284068550 tx=0x7fe274016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:22.021 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.012+0000 7fe2727fc700 1 -- 172.21.15.23:0/2634312542 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe274011a70 con 0x7fe2840aab80 2020-11-20T11:52:22.022 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.012+0000 7fe2727fc700 1 -- 172.21.15.23:0/2634312542 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe274011bd0 con 0x7fe2840aab80 2020-11-20T11:52:22.022 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.012+0000 7fe28b487700 1 -- 172.21.15.23:0/2634312542 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe28413cb30 con 0x7fe2840aab80 2020-11-20T11:52:22.022 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.012+0000 7fe2727fc700 1 -- 172.21.15.23:0/2634312542 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27401d450 con 0x7fe2840aab80 2020-11-20T11:52:22.023 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.013+0000 7fe28b487700 1 -- 172.21.15.23:0/2634312542 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2840b0ca0 con 0x7fe2840aab80 2020-11-20T11:52:22.023 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.013+0000 7fe2727fc700 1 -- 172.21.15.23:0/2634312542 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe274017040 con 0x7fe2840aab80 2020-11-20T11:52:22.024 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.014+0000 7fe2727fc700 1 --2- 172.21.15.23:0/2634312542 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe25c056d60 0x7fe25c059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:22.024 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.014+0000 7fe28b487700 1 -- 172.21.15.23:0/2634312542 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe2840acf00 con 0x7fe2840aab80 2020-11-20T11:52:22.024 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.014+0000 7fe2727fc700 1 -- 172.21.15.23:0/2634312542 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fe274071510 con 0x7fe2840aab80 2020-11-20T11:52:22.025 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.014+0000 7fe2727fc700 1 --2- 172.21.15.23:0/2634312542 >> [v2:172.21.15.23:6816/23099,v1:172.21.15.23:6817/23099] conn(0x7fe25c05e8c0 0x7fe25c060ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:22.025 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.014+0000 7fe2727fc700 1 -- 172.21.15.23:0/2634312542 --> [v2:172.21.15.23:6816/23099,v1:172.21.15.23:6817/23099] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe25c061340 con 0x7fe25c05e8c0 2020-11-20T11:52:22.026 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.014+0000 7fe2727fc700 1 -- 172.21.15.23:0/2634312542 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe2740467b0 con 0x7fe2840aab80 2020-11-20T11:52:22.026 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.014+0000 7fe28a485700 1 --2- 172.21.15.23:0/2634312542 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe25c056d60 0x7fe25c059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:22.026 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.014+0000 7fe28a485700 1 --2- 172.21.15.23:0/2634312542 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe25c056d60 0x7fe25c059220 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fe280006fd0 tx=0x7fe280008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:22.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.043+0000 7fe28ac86700 1 --2- 172.21.15.23:0/2634312542 >> [v2:172.21.15.23:6816/23099,v1:172.21.15.23:6817/23099] conn(0x7fe25c05e8c0 0x7fe25c060ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:22.047 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.044+0000 7fe28ac86700 1 --2- 172.21.15.23:0/2634312542 >> [v2:172.21.15.23:6816/23099,v1:172.21.15.23:6817/23099] conn(0x7fe25c05e8c0 0x7fe25c060ce0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:22.069 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.066+0000 7fe2727fc700 1 -- 172.21.15.23:0/2634312542 <== osd.1 v2:172.21.15.23:6816/23099 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7fe25c061340 con 0x7fe25c05e8c0 2020-11-20T11:52:22.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.080+0000 7fe28b487700 1 -- 172.21.15.23:0/2634312542 --> [v2:172.21.15.23:6816/23099,v1:172.21.15.23:6817/23099] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe2840ad100 con 0x7fe25c05e8c0 2020-11-20T11:52:22.083 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.080+0000 7fe2727fc700 1 -- 172.21.15.23:0/2634312542 <== osd.1 v2:172.21.15.23:6816/23099 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fe2840ad100 con 0x7fe25c05e8c0 2020-11-20T11:52:22.084 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:52:22.084 INFO:teuthology.orchestra.run.smithi023.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2020-11-20T11:52:22.084 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:52:22.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.081+0000 7fe28b487700 1 -- 172.21.15.23:0/2634312542 >> [v2:172.21.15.23:6816/23099,v1:172.21.15.23:6817/23099] conn(0x7fe25c05e8c0 msgr2=0x7fe25c060ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:22.085 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.081+0000 7fe28b487700 1 --2- 172.21.15.23:0/2634312542 >> [v2:172.21.15.23:6816/23099,v1:172.21.15.23:6817/23099] conn(0x7fe25c05e8c0 0x7fe25c060ce0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:22.086 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.082+0000 7fe28b487700 1 -- 172.21.15.23:0/2634312542 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe25c056d60 msgr2=0x7fe25c059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:22.086 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.082+0000 7fe28b487700 1 --2- 172.21.15.23:0/2634312542 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe25c056d60 0x7fe25c059220 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fe280006fd0 tx=0x7fe280008040).stop 2020-11-20T11:52:22.086 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.082+0000 7fe28b487700 1 -- 172.21.15.23:0/2634312542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2840aab80 msgr2=0x7fe28414d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:22.086 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.082+0000 7fe28b487700 1 --2- 172.21.15.23:0/2634312542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2840aab80 0x7fe28414d940 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fe284068550 tx=0x7fe274016000).stop 2020-11-20T11:52:22.086 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.082+0000 7fe28ac86700 1 -- 172.21.15.23:0/2634312542 reap_dead start 2020-11-20T11:52:22.087 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.083+0000 7fe28b487700 1 -- 172.21.15.23:0/2634312542 shutdown_connections 2020-11-20T11:52:22.087 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.083+0000 7fe28b487700 1 -- 172.21.15.23:0/2634312542 >> 172.21.15.23:0/2634312542 conn(0x7fe2840678e0 msgr2=0x7fe284068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:22.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.083+0000 7fe28b487700 1 -- 172.21.15.23:0/2634312542 shutdown_connections 2020-11-20T11:52:22.088 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.084+0000 7fe28b487700 1 -- 172.21.15.23:0/2634312542 wait complete. 2020-11-20T11:52:22.102 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2020-11-20T11:52:22.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.233+0000 7efe57c74700 1 Processor -- start 2020-11-20T11:52:22.236 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.233+0000 7efe57c74700 1 -- start start 2020-11-20T11:52:22.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.234+0000 7efe57c74700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7efe500a35f0 0x7efe500a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:22.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.234+0000 7efe57c74700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe5013e870 0x7efe5013ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:22.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.234+0000 7efe57c74700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe500a9b00 con 0x7efe500a3210 2020-11-20T11:52:22.239 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.234+0000 7efe57c74700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe500a1da0 con 0x7efe500a35f0 2020-11-20T11:52:22.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.234+0000 7efe57c74700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe500aa710 con 0x7efe5013e870 2020-11-20T11:52:22.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.234+0000 7efe55a10700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7efe500a3210 0x7efe500a9f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:33258/0 (socket says 172.21.15.23:33258) 2020-11-20T11:52:22.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.234+0000 7efe55a10700 1 -- 172.21.15.23:0/3777492032 learned_addr learned my addr 172.21.15.23:0/3777492032 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:52:22.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.234+0000 7efe56211700 1 --2- 172.21.15.23:0/3777492032 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe5013e870 0x7efe5013ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:22.240 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.234+0000 7efe5520f700 1 --2- 172.21.15.23:0/3777492032 >> v2:172.21.15.155:3300/0 conn(0x7efe500a35f0 0x7efe500a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:22.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.234+0000 7efe54a0e700 1 -- 172.21.15.23:0/3777492032 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3664600541 0 0) 0x7efe500a9b00 con 0x7efe500a3210 2020-11-20T11:52:22.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.235+0000 7efe54a0e700 1 -- 172.21.15.23:0/3777492032 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe38003280 con 0x7efe500a3210 2020-11-20T11:52:22.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.235+0000 7efe5520f700 1 -- 172.21.15.23:0/3777492032 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe5013e870 msgr2=0x7efe5013ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:52:22.241 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.235+0000 7efe5520f700 1 --2- 172.21.15.23:0/3777492032 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe5013e870 0x7efe5013ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:22.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.235+0000 7efe5520f700 1 -- 172.21.15.23:0/3777492032 >> v1:172.21.15.23:6789/0 conn(0x7efe500a3210 legacy=0x7efe500a9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:22.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.235+0000 7efe5520f700 1 -- 172.21.15.23:0/3777492032 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe500aa9b0 con 0x7efe500a35f0 2020-11-20T11:52:22.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.235+0000 7efe56211700 1 --2- 172.21.15.23:0/3777492032 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe5013e870 0x7efe5013ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:52:22.242 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.235+0000 7efe5520f700 1 --2- 172.21.15.23:0/3777492032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe500a35f0 0x7efe500a18b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7efe40006fd0 tx=0x7efe40006e30).ready entity=mon.1 client_cookie=1b86c968ee07513d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:22.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.235+0000 7efe54a0e700 1 -- 172.21.15.23:0/3777492032 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe40009630 con 0x7efe500a35f0 2020-11-20T11:52:22.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.235+0000 7efe54a0e700 1 -- 172.21.15.23:0/3777492032 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe40000e90 con 0x7efe500a35f0 2020-11-20T11:52:22.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.235+0000 7efe54a0e700 1 -- 172.21.15.23:0/3777492032 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe400099a0 con 0x7efe500a35f0 2020-11-20T11:52:22.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.236+0000 7efe57c74700 1 -- 172.21.15.23:0/3777492032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe500a35f0 msgr2=0x7efe500a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:22.243 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.236+0000 7efe57c74700 1 --2- 172.21.15.23:0/3777492032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe500a35f0 0x7efe500a18b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7efe40006fd0 tx=0x7efe40006e30).stop 2020-11-20T11:52:22.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.236+0000 7efe57c74700 1 -- 172.21.15.23:0/3777492032 shutdown_connections 2020-11-20T11:52:22.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.236+0000 7efe57c74700 1 --2- 172.21.15.23:0/3777492032 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe5013e870 0x7efe5013ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:22.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.236+0000 7efe57c74700 1 --2- 172.21.15.23:0/3777492032 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe500a35f0 0x7efe500a18b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:22.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.236+0000 7efe57c74700 1 -- 172.21.15.23:0/3777492032 >> 172.21.15.23:0/3777492032 conn(0x7efe500601b0 msgr2=0x7efe500605c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:22.244 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.236+0000 7efe57c74700 1 -- 172.21.15.23:0/3777492032 shutdown_connections 2020-11-20T11:52:22.245 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.236+0000 7efe57c74700 1 -- 172.21.15.23:0/3777492032 wait complete. 2020-11-20T11:52:22.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.237+0000 7efe57c74700 1 Processor -- start 2020-11-20T11:52:22.247 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.237+0000 7efe57c74700 1 -- start start 2020-11-20T11:52:22.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.237+0000 7efe57c74700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe500a3210 0x7efe50066920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:22.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.237+0000 7efe57c74700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe500a35f0 0x7efe500a6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:22.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.237+0000 7efe57c74700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe5013e870 0x7efe5014d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:22.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.237+0000 7efe57c74700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe500684a0 con 0x7efe5013e870 2020-11-20T11:52:22.248 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.237+0000 7efe57c74700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe500ac640 con 0x7efe500a35f0 2020-11-20T11:52:22.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.237+0000 7efe57c74700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe500ac870 con 0x7efe500a3210 2020-11-20T11:52:22.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.238+0000 7efe56211700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe5013e870 0x7efe5014d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:22.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.238+0000 7efe5520f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe500a35f0 0x7efe500a6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:22.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.238+0000 7efe56211700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe5013e870 0x7efe5014d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47250/0 (socket says 172.21.15.23:47250) 2020-11-20T11:52:22.249 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.238+0000 7efe56211700 1 -- 172.21.15.23:0/804623300 learned_addr learned my addr 172.21.15.23:0/804623300 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:52:22.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.238+0000 7efe55a10700 1 --2- 172.21.15.23:0/804623300 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe500a3210 0x7efe50066920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:22.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.238+0000 7efe56211700 1 -- 172.21.15.23:0/804623300 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe500a3210 msgr2=0x7efe50066920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:22.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.238+0000 7efe56211700 1 --2- 172.21.15.23:0/804623300 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe500a3210 0x7efe50066920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:22.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.238+0000 7efe56211700 1 -- 172.21.15.23:0/804623300 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe500a35f0 msgr2=0x7efe500a6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:22.250 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.238+0000 7efe56211700 1 --2- 172.21.15.23:0/804623300 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe500a35f0 0x7efe500a6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:22.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.238+0000 7efe56211700 1 -- 172.21.15.23:0/804623300 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe501389b0 con 0x7efe5013e870 2020-11-20T11:52:22.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.238+0000 7efe5520f700 1 --2- 172.21.15.23:0/804623300 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe500a35f0 0x7efe500a6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:52:22.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.238+0000 7efe56211700 1 --2- 172.21.15.23:0/804623300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe5013e870 0x7efe5014d8a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7efe4800ef80 tx=0x7efe48009220).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:22.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.238+0000 7efe3dffb700 1 -- 172.21.15.23:0/804623300 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe4800cc00 con 0x7efe5013e870 2020-11-20T11:52:22.251 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.238+0000 7efe3dffb700 1 -- 172.21.15.23:0/804623300 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe4800cd60 con 0x7efe5013e870 2020-11-20T11:52:22.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.238+0000 7efe57c74700 1 -- 172.21.15.23:0/804623300 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe50066e10 con 0x7efe5013e870 2020-11-20T11:52:22.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.238+0000 7efe3dffb700 1 -- 172.21.15.23:0/804623300 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe48012620 con 0x7efe5013e870 2020-11-20T11:52:22.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.238+0000 7efe57c74700 1 -- 172.21.15.23:0/804623300 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe500a6f30 con 0x7efe5013e870 2020-11-20T11:52:22.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.239+0000 7efe3dffb700 1 -- 172.21.15.23:0/804623300 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7efe48022070 con 0x7efe5013e870 2020-11-20T11:52:22.252 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.240+0000 7efe57c74700 1 -- 172.21.15.23:0/804623300 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe50062c40 con 0x7efe5013e870 2020-11-20T11:52:22.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.240+0000 7efe3dffb700 1 --2- 172.21.15.23:0/804623300 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efe2c056d60 0x7efe2c059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:22.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.240+0000 7efe3dffb700 1 -- 172.21.15.23:0/804623300 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7efe480708d0 con 0x7efe5013e870 2020-11-20T11:52:22.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.240+0000 7efe55a10700 1 --2- 172.21.15.23:0/804623300 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efe2c056d60 0x7efe2c059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:22.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.240+0000 7efe55a10700 1 --2- 172.21.15.23:0/804623300 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efe2c056d60 0x7efe2c059220 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7efe4c0055e0 tx=0x7efe4c009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:22.253 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.244+0000 7efe3dffb700 1 -- 172.21.15.23:0/804623300 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7efe48045bc0 con 0x7efe5013e870 2020-11-20T11:52:22.452 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.449+0000 7efe57c74700 1 -- 172.21.15.23:0/804623300 --> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7efe500a7720 con 0x7efe2c056d60 2020-11-20T11:52:22.453 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.450+0000 7efe3dffb700 1 -- 172.21.15.23:0/804623300 <== mgr.4105 v2:172.21.15.155:6832/22693 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7efe500a7720 con 0x7efe2c056d60 2020-11-20T11:52:22.457 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.454+0000 7efe57c74700 1 -- 172.21.15.23:0/804623300 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efe2c056d60 msgr2=0x7efe2c059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:22.457 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.454+0000 7efe57c74700 1 --2- 172.21.15.23:0/804623300 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efe2c056d60 0x7efe2c059220 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7efe4c0055e0 tx=0x7efe4c009040).stop 2020-11-20T11:52:22.458 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.454+0000 7efe57c74700 1 -- 172.21.15.23:0/804623300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe5013e870 msgr2=0x7efe5014d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:22.458 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.454+0000 7efe57c74700 1 --2- 172.21.15.23:0/804623300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe5013e870 0x7efe5014d8a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7efe4800ef80 tx=0x7efe48009220).stop 2020-11-20T11:52:22.459 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.456+0000 7efe57c74700 1 -- 172.21.15.23:0/804623300 shutdown_connections 2020-11-20T11:52:22.459 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.456+0000 7efe57c74700 1 --2- 172.21.15.23:0/804623300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe5013e870 0x7efe5014d8a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:22.459 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.456+0000 7efe57c74700 1 --2- 172.21.15.23:0/804623300 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efe2c056d60 0x7efe2c059220 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:22.459 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.456+0000 7efe57c74700 1 --2- 172.21.15.23:0/804623300 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe500a3210 0x7efe50066920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:22.459 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.456+0000 7efe57c74700 1 --2- 172.21.15.23:0/804623300 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe500a35f0 0x7efe500a6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:22.460 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.456+0000 7efe57c74700 1 -- 172.21.15.23:0/804623300 >> 172.21.15.23:0/804623300 conn(0x7efe500601b0 msgr2=0x7efe50139dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:22.460 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.457+0000 7efe57c74700 1 -- 172.21.15.23:0/804623300 shutdown_connections 2020-11-20T11:52:22.460 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.457+0000 7efe57c74700 1 -- 172.21.15.23:0/804623300 wait complete. 2020-11-20T11:52:22.461 INFO:teuthology.orchestra.run.smithi023.stderr:instructed osd(s) 1 to deep-scrub 2020-11-20T11:52:22.473 INFO:tasks.ceph:Scrubbing osd.2 2020-11-20T11:52:22.473 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2020-11-20T11:52:22.618 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.614+0000 7f79b6aef700 1 Processor -- start 2020-11-20T11:52:22.619 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.616+0000 7f79b6aef700 1 -- start start 2020-11-20T11:52:22.619 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.616+0000 7f79b6aef700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79b00a6450 0x7f79b00a5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:22.620 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.616+0000 7f79b6aef700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b00a7fd0 con 0x7f79b00a9890 2020-11-20T11:52:22.620 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.616+0000 7f79b6aef700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b00a68c0 con 0x7f79b00a86a0 2020-11-20T11:52:22.620 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.616+0000 7f79b6aef700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79b00a5f00 con 0x7f79b00a6450 2020-11-20T11:52:22.624 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.617+0000 7f79b488b700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f79b00a9890 0x7f79b0061400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:33272/0 (socket says 172.21.15.23:33272) 2020-11-20T11:52:22.624 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.617+0000 7f79b508c700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79b00a6450 0x7f79b00a5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:22.624 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.617+0000 7f79b488b700 1 -- 172.21.15.23:0/2691437919 learned_addr learned my addr 172.21.15.23:0/2691437919 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:52:22.624 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.617+0000 7f79af7fe700 1 -- 172.21.15.23:0/2691437919 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2782342974 0 0) 0x7f79b00a7fd0 con 0x7f79b00a9890 2020-11-20T11:52:22.624 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.617+0000 7f79af7fe700 1 -- 172.21.15.23:0/2691437919 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f799c003280 con 0x7f79b00a9890 2020-11-20T11:52:22.625 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.617+0000 7f79af7fe700 1 -- 172.21.15.23:0/2691437919 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3788448110 0 0) 0x7f799c003280 con 0x7f79b00a9890 2020-11-20T11:52:22.625 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.618+0000 7f79af7fe700 1 -- 172.21.15.23:0/2691437919 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79b00a6450 msgr2=0x7f79b00a5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:52:22.625 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.618+0000 7f79af7fe700 1 --2- 172.21.15.23:0/2691437919 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79b00a6450 0x7f79b00a5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:22.625 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.618+0000 7f79af7fe700 1 -- 172.21.15.23:0/2691437919 >> v1:172.21.15.155:6789/0 conn(0x7f79b00a86a0 legacy=0x7f79b0061b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:22.625 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.618+0000 7f79af7fe700 1 -- 172.21.15.23:0/2691437919 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b0062220 con 0x7f79b00a9890 2020-11-20T11:52:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.618+0000 7f79af7fe700 1 -- 172.21.15.23:0/2691437919 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f79a0001420 con 0x7f79b00a9890 2020-11-20T11:52:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.618+0000 7f79af7fe700 1 -- 172.21.15.23:0/2691437919 >> v1:172.21.15.23:6789/0 conn(0x7f79b00a9890 legacy=0x7f79b0061400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.618+0000 7f79af7fe700 1 --2- 172.21.15.23:0/2691437919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f799c0037a0 0x7f799c003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.618+0000 7f79af7fe700 1 --2- 172.21.15.23:0/2691437919 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79b00a6450 0x7f799c004520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.618+0000 7f79af7fe700 1 --2- 172.21.15.23:0/2691437919 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f799c004a10 0x7f799c004e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:22.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.618+0000 7f79af7fe700 1 -- 172.21.15.23:0/2691437919 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b00a5f00 con 0x7f799c0037a0 2020-11-20T11:52:22.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.618+0000 7f79af7fe700 1 -- 172.21.15.23:0/2691437919 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799c004240 con 0x7f799c004a10 2020-11-20T11:52:22.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.618+0000 7f79af7fe700 1 -- 172.21.15.23:0/2691437919 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f799c007750 con 0x7f79b00a6450 2020-11-20T11:52:22.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.618+0000 7f79b508c700 1 --2- 172.21.15.23:0/2691437919 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f799c004a10 0x7f799c004e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:22.627 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.618+0000 7f79b488b700 1 --2- 172.21.15.23:0/2691437919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f799c0037a0 0x7f799c003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:22.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.618+0000 7f79b508c700 1 -- 172.21.15.23:0/2691437919 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79b00a6450 msgr2=0x7f799c004520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:52:22.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.618+0000 7f79b508c700 1 --2- 172.21.15.23:0/2691437919 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79b00a6450 0x7f799c004520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:52:22.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.618+0000 7f79b508c700 1 -- 172.21.15.23:0/2691437919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f799c0037a0 msgr2=0x7f799c003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:52:22.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.618+0000 7f79b508c700 1 --2- 172.21.15.23:0/2691437919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f799c0037a0 0x7f799c003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:22.628 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.618+0000 7f79b508c700 1 -- 172.21.15.23:0/2691437919 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f799c003280 con 0x7f799c004a10 2020-11-20T11:52:22.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.618+0000 7f79b508c700 1 --2- 172.21.15.23:0/2691437919 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f799c004a10 0x7f799c004e90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f79a800cc40 tx=0x7f79a800e040).ready entity=mon.1 client_cookie=1395279b7c625b30 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:22.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.619+0000 7f79af7fe700 1 -- 172.21.15.23:0/2691437919 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a8011040 con 0x7f799c004a10 2020-11-20T11:52:22.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.619+0000 7f79af7fe700 1 -- 172.21.15.23:0/2691437919 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a8009d10 con 0x7f799c004a10 2020-11-20T11:52:22.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.619+0000 7f79af7fe700 1 -- 172.21.15.23:0/2691437919 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a8038610 con 0x7f799c004a10 2020-11-20T11:52:22.629 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.620+0000 7f79b488b700 1 --2- 172.21.15.23:0/2691437919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f799c0037a0 0x7f799c003bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:52:22.630 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.621+0000 7f79b6aef700 1 -- 172.21.15.23:0/2691437919 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f799c004a10 msgr2=0x7f799c004e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:22.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.621+0000 7f79b6aef700 1 --2- 172.21.15.23:0/2691437919 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f799c004a10 0x7f799c004e90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f79a800cc40 tx=0x7f79a800e040).stop 2020-11-20T11:52:22.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.621+0000 7f79b508c700 1 -- 172.21.15.23:0/2691437919 reap_dead start 2020-11-20T11:52:22.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.624+0000 7f79b6aef700 1 -- 172.21.15.23:0/2691437919 shutdown_connections 2020-11-20T11:52:22.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.624+0000 7f79b6aef700 1 -- 172.21.15.23:0/2691437919 >> 172.21.15.23:0/2691437919 conn(0x7f79b00601b0 msgr2=0x7f79b00605c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:22.631 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.627+0000 7f79b6aef700 1 -- 172.21.15.23:0/2691437919 shutdown_connections 2020-11-20T11:52:22.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.627+0000 7f79b6aef700 1 -- 172.21.15.23:0/2691437919 wait complete. 2020-11-20T11:52:22.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.629+0000 7f79b6aef700 1 Processor -- start 2020-11-20T11:52:22.632 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.629+0000 7f79b6aef700 1 -- start start 2020-11-20T11:52:22.633 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.630+0000 7f79b6aef700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f799c004a10 0x7f79b00ab220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:22.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.630+0000 7f79b6aef700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79b00a0680 0x7f79b009d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:22.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.630+0000 7f79b6aef700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79b009e610 0x7f79b009ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:22.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.630+0000 7f79b6aef700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b009df80 con 0x7f79b009e610 2020-11-20T11:52:22.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.630+0000 7f79b6aef700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b009d990 con 0x7f79b00a0680 2020-11-20T11:52:22.634 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.630+0000 7f79b6aef700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79b009dbb0 con 0x7f799c004a10 2020-11-20T11:52:22.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.630+0000 7f79b488b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f799c004a10 0x7f79b00ab220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:22.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.630+0000 7f79b488b700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f799c004a10 0x7f79b00ab220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:57506/0 (socket says 172.21.15.23:57506) 2020-11-20T11:52:22.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.630+0000 7f79b488b700 1 -- 172.21.15.23:0/1427405512 learned_addr learned my addr 172.21.15.23:0/1427405512 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:52:22.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.630+0000 7f79affff700 1 --2- 172.21.15.23:0/1427405512 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79b00a0680 0x7f79b009d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:22.635 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.631+0000 7f79b508c700 1 --2- 172.21.15.23:0/1427405512 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79b009e610 0x7f79b009ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:22.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.631+0000 7f79b488b700 1 -- 172.21.15.23:0/1427405512 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79b00a0680 msgr2=0x7f79b009d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:22.636 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.631+0000 7f79b488b700 1 --2- 172.21.15.23:0/1427405512 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79b00a0680 0x7f79b009d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:22.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.631+0000 7f79b488b700 1 -- 172.21.15.23:0/1427405512 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79b009e610 msgr2=0x7f79b009ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:22.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.631+0000 7f79b488b700 1 --2- 172.21.15.23:0/1427405512 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f79b009e610 0x7f79b009ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:22.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.631+0000 7f79b488b700 1 -- 172.21.15.23:0/1427405512 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b009e170 con 0x7f799c004a10 2020-11-20T11:52:22.637 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.631+0000 7f79affff700 1 --2- 172.21.15.23:0/1427405512 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f79b00a0680 0x7f79b009d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:52:22.638 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.631+0000 7f79b488b700 1 --2- 172.21.15.23:0/1427405512 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f799c004a10 0x7f79b00ab220 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f79a000ea50 tx=0x7f79a0009040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:22.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.637+0000 7f79acff9700 1 -- 172.21.15.23:0/1427405512 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a000ee90 con 0x7f799c004a10 2020-11-20T11:52:22.640 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.637+0000 7f79acff9700 1 -- 172.21.15.23:0/1427405512 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a0010040 con 0x7f799c004a10 2020-11-20T11:52:22.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.637+0000 7f79b6aef700 1 -- 172.21.15.23:0/1427405512 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79b0065c50 con 0x7f799c004a10 2020-11-20T11:52:22.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.637+0000 7f79acff9700 1 -- 172.21.15.23:0/1427405512 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a0007550 con 0x7f799c004a10 2020-11-20T11:52:22.641 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.637+0000 7f79b6aef700 1 -- 172.21.15.23:0/1427405512 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79b0066110 con 0x7f799c004a10 2020-11-20T11:52:22.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.638+0000 7f79b6aef700 1 -- 172.21.15.23:0/1427405512 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f79b0057dc0 con 0x7f799c004a10 2020-11-20T11:52:22.642 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.639+0000 7f79acff9700 1 -- 172.21.15.23:0/1427405512 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f79a0001170 con 0x7f799c004a10 2020-11-20T11:52:22.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.639+0000 7f79acff9700 1 --2- 172.21.15.23:0/1427405512 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7984056db0 0x7f7984059270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:22.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.639+0000 7f79acff9700 1 -- 172.21.15.23:0/1427405512 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f79a006fe80 con 0x7f799c004a10 2020-11-20T11:52:22.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.639+0000 7f79acff9700 1 --2- 172.21.15.23:0/1427405512 >> [v2:172.21.15.23:6808/23084,v1:172.21.15.23:6809/23084] conn(0x7f798405ea10 0x7f7984060e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:22.643 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.639+0000 7f79affff700 1 --2- 172.21.15.23:0/1427405512 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7984056db0 0x7f7984059270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:22.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.639+0000 7f79acff9700 1 -- 172.21.15.23:0/1427405512 --> [v2:172.21.15.23:6808/23084,v1:172.21.15.23:6809/23084] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7984061490 con 0x7f798405ea10 2020-11-20T11:52:22.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.640+0000 7f79acff9700 1 -- 172.21.15.23:0/1427405512 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79a00724f0 con 0x7f799c004a10 2020-11-20T11:52:22.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.640+0000 7f79b508c700 1 --2- 172.21.15.23:0/1427405512 >> [v2:172.21.15.23:6808/23084,v1:172.21.15.23:6809/23084] conn(0x7f798405ea10 0x7f7984060e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:22.644 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.640+0000 7f79affff700 1 --2- 172.21.15.23:0/1427405512 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7984056db0 0x7f7984059270 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f79b00a0ac0 tx=0x7f79a4009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:22.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.640+0000 7f79b508c700 1 --2- 172.21.15.23:0/1427405512 >> [v2:172.21.15.23:6808/23084,v1:172.21.15.23:6809/23084] conn(0x7f798405ea10 0x7f7984060e30 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:22.645 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.641+0000 7f79acff9700 1 -- 172.21.15.23:0/1427405512 <== osd.2 v2:172.21.15.23:6808/23084 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7f7984061490 con 0x7f798405ea10 2020-11-20T11:52:22.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.659+0000 7f79b6aef700 1 -- 172.21.15.23:0/1427405512 --> [v2:172.21.15.23:6808/23084,v1:172.21.15.23:6809/23084] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f79b0044710 con 0x7f798405ea10 2020-11-20T11:52:22.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.659+0000 7f79acff9700 1 -- 172.21.15.23:0/1427405512 <== osd.2 v2:172.21.15.23:6808/23084 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f79b0044710 con 0x7f798405ea10 2020-11-20T11:52:22.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.659+0000 7f79b6aef700 1 -- 172.21.15.23:0/1427405512 >> [v2:172.21.15.23:6808/23084,v1:172.21.15.23:6809/23084] conn(0x7f798405ea10 msgr2=0x7f7984060e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:22.663 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.659+0000 7f79b6aef700 1 --2- 172.21.15.23:0/1427405512 >> [v2:172.21.15.23:6808/23084,v1:172.21.15.23:6809/23084] conn(0x7f798405ea10 0x7f7984060e30 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:22.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.659+0000 7f79b6aef700 1 -- 172.21.15.23:0/1427405512 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7984056db0 msgr2=0x7f7984059270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:22.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.659+0000 7f79b6aef700 1 --2- 172.21.15.23:0/1427405512 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f7984056db0 0x7f7984059270 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f79b00a0ac0 tx=0x7f79a4009040).stop 2020-11-20T11:52:22.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.659+0000 7f79b6aef700 1 -- 172.21.15.23:0/1427405512 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f799c004a10 msgr2=0x7f79b00ab220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:22.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.659+0000 7f79b6aef700 1 --2- 172.21.15.23:0/1427405512 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f799c004a10 0x7f79b00ab220 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f79a000ea50 tx=0x7f79a0009040).stop 2020-11-20T11:52:22.664 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.659+0000 7f79b508c700 1 -- 172.21.15.23:0/1427405512 reap_dead start 2020-11-20T11:52:22.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.659+0000 7f79b6aef700 1 -- 172.21.15.23:0/1427405512 shutdown_connections 2020-11-20T11:52:22.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.659+0000 7f79b6aef700 1 -- 172.21.15.23:0/1427405512 >> 172.21.15.23:0/1427405512 conn(0x7f79b00601b0 msgr2=0x7f79b0061400 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:22.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.659+0000 7f79b6aef700 1 -- 172.21.15.23:0/1427405512 shutdown_connections 2020-11-20T11:52:22.665 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.659+0000 7f79b6aef700 1 -- 172.21.15.23:0/1427405512 wait complete. 2020-11-20T11:52:22.666 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:52:22.666 INFO:teuthology.orchestra.run.smithi023.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2020-11-20T11:52:22.666 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:52:22.676 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2020-11-20T11:52:22.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.840+0000 7f38fc90f700 1 Processor -- start 2020-11-20T11:52:22.844 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.840+0000 7f38fc90f700 1 -- start start 2020-11-20T11:52:22.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.841+0000 7f38fc90f700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38f80aaab0 0x7f38f813ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:22.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.841+0000 7f38fc90f700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f38f80aae90 0x7f38f81381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:22.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.841+0000 7f38fc90f700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f813e600 con 0x7f38f813aeb0 2020-11-20T11:52:22.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.841+0000 7f38fc90f700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38f813e8a0 con 0x7f38f80aae90 2020-11-20T11:52:22.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.841+0000 7f38fc90f700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38f80ab6d0 con 0x7f38f80aaab0 2020-11-20T11:52:22.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.841+0000 7f38f5d9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f38f80aae90 0x7f38f81381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:22.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.841+0000 7f38f6d9d700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f38f813aeb0 0x7f38f80a8c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:33296/0 (socket says 172.21.15.23:33296) 2020-11-20T11:52:22.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.841+0000 7f38f6d9d700 1 -- 172.21.15.23:0/1935617555 learned_addr learned my addr 172.21.15.23:0/1935617555 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:52:22.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.842+0000 7f38f659c700 1 --2- 172.21.15.23:0/1935617555 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38f80aaab0 0x7f38f813ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:22.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.842+0000 7f38f5d9b700 1 -- 172.21.15.23:0/1935617555 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38f80aaab0 msgr2=0x7f38f813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:52:22.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.842+0000 7f38f5d9b700 1 --2- 172.21.15.23:0/1935617555 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38f80aaab0 0x7f38f813ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:22.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.842+0000 7f38f5d9b700 1 -- 172.21.15.23:0/1935617555 >> v1:172.21.15.23:6789/0 conn(0x7f38f813aeb0 legacy=0x7f38f80a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:22.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.842+0000 7f38f559a700 1 -- 172.21.15.23:0/1935617555 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2464347550 0 0) 0x7f38f813e600 con 0x7f38f813aeb0 2020-11-20T11:52:22.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.842+0000 7f38f5d9b700 1 -- 172.21.15.23:0/1935617555 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f80a9370 con 0x7f38f80aae90 2020-11-20T11:52:22.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.842+0000 7f38f659c700 1 --2- 172.21.15.23:0/1935617555 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38f80aaab0 0x7f38f813ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:52:22.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.842+0000 7f38f5d9b700 1 --2- 172.21.15.23:0/1935617555 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38f80aae90 0x7f38f81381a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f38ec006fd0 tx=0x7f38ec006e30).ready entity=mon.0 client_cookie=401bbcc2b99d6014 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:22.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.842+0000 7f38f559a700 1 -- 172.21.15.23:0/1935617555 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ec0097e0 con 0x7f38f80aae90 2020-11-20T11:52:22.850 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.842+0000 7f38f559a700 1 -- 172.21.15.23:0/1935617555 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38ec000e90 con 0x7f38f80aae90 2020-11-20T11:52:22.850 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.842+0000 7f38f559a700 1 -- 172.21.15.23:0/1935617555 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ec009b50 con 0x7f38f80aae90 2020-11-20T11:52:22.850 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.843+0000 7f38fc90f700 1 -- 172.21.15.23:0/1935617555 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38f80aae90 msgr2=0x7f38f81381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:22.850 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.843+0000 7f38fc90f700 1 --2- 172.21.15.23:0/1935617555 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38f80aae90 0x7f38f81381a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f38ec006fd0 tx=0x7f38ec006e30).stop 2020-11-20T11:52:22.850 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.843+0000 7f38fc90f700 1 -- 172.21.15.23:0/1935617555 shutdown_connections 2020-11-20T11:52:22.851 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.843+0000 7f38fc90f700 1 --2- 172.21.15.23:0/1935617555 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38f80aaab0 0x7f38f813ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:22.851 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.843+0000 7f38fc90f700 1 --2- 172.21.15.23:0/1935617555 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38f80aae90 0x7f38f81381a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:22.851 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.843+0000 7f38fc90f700 1 -- 172.21.15.23:0/1935617555 >> 172.21.15.23:0/1935617555 conn(0x7f38f80678e0 msgr2=0x7f38f8067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:22.851 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.843+0000 7f38fc90f700 1 -- 172.21.15.23:0/1935617555 shutdown_connections 2020-11-20T11:52:22.851 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.843+0000 7f38fc90f700 1 -- 172.21.15.23:0/1935617555 wait complete. 2020-11-20T11:52:22.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.844+0000 7f38fc90f700 1 Processor -- start 2020-11-20T11:52:22.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.844+0000 7f38fc90f700 1 -- start start 2020-11-20T11:52:22.854 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.844+0000 7f38fc90f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38f80aaab0 0x7f38f80a6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:22.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.844+0000 7f38fc90f700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38f80aae90 0x7f38f814dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:22.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.844+0000 7f38fc90f700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38f80b03e0 0x7f38f814a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:22.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.844+0000 7f38fc90f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f80a5980 con 0x7f38f80aaab0 2020-11-20T11:52:22.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.844+0000 7f38fc90f700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f813e3f0 con 0x7f38f80b03e0 2020-11-20T11:52:22.855 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.844+0000 7f38fc90f700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38f80b0cf0 con 0x7f38f80aae90 2020-11-20T11:52:22.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.844+0000 7f38f659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38f80aaab0 0x7f38f80a6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:22.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.844+0000 7f38f659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38f80aaab0 0x7f38f80a6600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47280/0 (socket says 172.21.15.23:47280) 2020-11-20T11:52:22.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.844+0000 7f38f659c700 1 -- 172.21.15.23:0/1628878199 learned_addr learned my addr 172.21.15.23:0/1628878199 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:52:22.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.845+0000 7f38f6d9d700 1 --2- 172.21.15.23:0/1628878199 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38f80b03e0 0x7f38f814a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:22.856 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.845+0000 7f38f5d9b700 1 --2- 172.21.15.23:0/1628878199 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38f80aae90 0x7f38f814dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:22.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.845+0000 7f38f659c700 1 -- 172.21.15.23:0/1628878199 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38f80aae90 msgr2=0x7f38f814dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:22.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.845+0000 7f38f659c700 1 --2- 172.21.15.23:0/1628878199 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38f80aae90 0x7f38f814dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:22.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.845+0000 7f38f659c700 1 -- 172.21.15.23:0/1628878199 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38f80b03e0 msgr2=0x7f38f814a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:22.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.845+0000 7f38f659c700 1 --2- 172.21.15.23:0/1628878199 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38f80b03e0 0x7f38f814a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:22.857 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.845+0000 7f38f659c700 1 -- 172.21.15.23:0/1628878199 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f814d7a0 con 0x7f38f80aaab0 2020-11-20T11:52:22.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.845+0000 7f38f6d9d700 1 --2- 172.21.15.23:0/1628878199 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38f80b03e0 0x7f38f814a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:52:22.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.845+0000 7f38f5d9b700 1 --2- 172.21.15.23:0/1628878199 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38f80aae90 0x7f38f814dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:52:22.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.845+0000 7f38f659c700 1 --2- 172.21.15.23:0/1628878199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38f80aaab0 0x7f38f80a6600 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f38e800db60 tx=0x7f38e8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:22.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.845+0000 7f38e67fc700 1 -- 172.21.15.23:0/1628878199 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e8009790 con 0x7f38f80aaab0 2020-11-20T11:52:22.858 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.845+0000 7f38e67fc700 1 -- 172.21.15.23:0/1628878199 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38e80098f0 con 0x7f38f80aaab0 2020-11-20T11:52:22.859 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.845+0000 7f38fc90f700 1 -- 172.21.15.23:0/1628878199 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38f814da40 con 0x7f38f80aaab0 2020-11-20T11:52:22.859 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.845+0000 7f38e67fc700 1 -- 172.21.15.23:0/1628878199 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e8009cb0 con 0x7f38f80aaab0 2020-11-20T11:52:22.859 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.845+0000 7f38fc90f700 1 -- 172.21.15.23:0/1628878199 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38f814a0e0 con 0x7f38f80aaab0 2020-11-20T11:52:22.859 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.846+0000 7f38fc90f700 1 -- 172.21.15.23:0/1628878199 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38f805c7d0 con 0x7f38f80aaab0 2020-11-20T11:52:22.859 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.846+0000 7f38e67fc700 1 -- 172.21.15.23:0/1628878199 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f38e8013690 con 0x7f38f80aaab0 2020-11-20T11:52:22.860 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.846+0000 7f38e67fc700 1 --2- 172.21.15.23:0/1628878199 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f38cc056d60 0x7f38cc059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:22.860 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.847+0000 7f38e67fc700 1 -- 172.21.15.23:0/1628878199 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f38e8070710 con 0x7f38f80aaab0 2020-11-20T11:52:22.860 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.847+0000 7f38f5d9b700 1 --2- 172.21.15.23:0/1628878199 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f38cc056d60 0x7f38cc059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:22.860 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.847+0000 7f38f5d9b700 1 --2- 172.21.15.23:0/1628878199 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f38cc056d60 0x7f38cc059220 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f38f80aa7e0 tx=0x7f38ec015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:22.860 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:22.850+0000 7f38e67fc700 1 -- 172.21.15.23:0/1628878199 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f38e8045a00 con 0x7f38f80aaab0 2020-11-20T11:52:23.052 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:23.049+0000 7f38fc90f700 1 -- 172.21.15.23:0/1628878199 --> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f38f8044710 con 0x7f38cc056d60 2020-11-20T11:52:24.334 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:23.051+0000 7f38e67fc700 1 -- 172.21.15.23:0/1628878199 <== mgr.4105 v2:172.21.15.155:6832/22693 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f38f8044710 con 0x7f38cc056d60 2020-11-20T11:52:24.334 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:23.052+0000 7f38fc90f700 1 -- 172.21.15.23:0/1628878199 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f38cc056d60 msgr2=0x7f38cc059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:24.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:23.052+0000 7f38fc90f700 1 --2- 172.21.15.23:0/1628878199 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f38cc056d60 0x7f38cc059220 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f38f80aa7e0 tx=0x7f38ec015000).stop 2020-11-20T11:52:24.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:23.052+0000 7f38fc90f700 1 -- 172.21.15.23:0/1628878199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38f80aaab0 msgr2=0x7f38f80a6600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:24.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:23.052+0000 7f38fc90f700 1 --2- 172.21.15.23:0/1628878199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38f80aaab0 0x7f38f80a6600 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f38e800db60 tx=0x7f38e8008040).stop 2020-11-20T11:52:24.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:23.052+0000 7f38fc90f700 1 -- 172.21.15.23:0/1628878199 shutdown_connections 2020-11-20T11:52:24.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:23.052+0000 7f38fc90f700 1 --2- 172.21.15.23:0/1628878199 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f38f80b03e0 0x7f38f814a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:24.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:23.052+0000 7f38fc90f700 1 --2- 172.21.15.23:0/1628878199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f38f80aaab0 0x7f38f80a6600 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:24.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:23.052+0000 7f38fc90f700 1 --2- 172.21.15.23:0/1628878199 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f38cc056d60 0x7f38cc059220 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:24.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:23.052+0000 7f38fc90f700 1 --2- 172.21.15.23:0/1628878199 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f38f80aae90 0x7f38f814dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:24.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:23.052+0000 7f38fc90f700 1 -- 172.21.15.23:0/1628878199 >> 172.21.15.23:0/1628878199 conn(0x7f38f80678e0 msgr2=0x7f38f80adcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:24.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:23.052+0000 7f38fc90f700 1 -- 172.21.15.23:0/1628878199 shutdown_connections 2020-11-20T11:52:24.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:23.053+0000 7f38fc90f700 1 -- 172.21.15.23:0/1628878199 wait complete. 2020-11-20T11:52:24.337 INFO:teuthology.orchestra.run.smithi023.stderr:instructed osd(s) 2 to deep-scrub 2020-11-20T11:52:24.337 INFO:tasks.ceph:Scrubbing osd.3 2020-11-20T11:52:24.338 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2020-11-20T11:52:24.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.499+0000 7f23ecbdd700 1 Processor -- start 2020-11-20T11:52:24.502 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.499+0000 7f23ecbdd700 1 -- start start 2020-11-20T11:52:24.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.499+0000 7f23ecbdd700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f23e80aab80 0x7f23e80abad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:24.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.499+0000 7f23ecbdd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f23e80a8570 0x7f23e80ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:24.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.499+0000 7f23ecbdd700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23e813aad0 con 0x7f23e80aaf60 2020-11-20T11:52:24.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.499+0000 7f23ecbdd700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23e80a9e00 con 0x7f23e80aab80 2020-11-20T11:52:24.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.499+0000 7f23ecbdd700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23e813ad70 con 0x7f23e80a8570 2020-11-20T11:52:24.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.500+0000 7f23e77fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f23e80aab80 0x7f23e80abad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:24.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.500+0000 7f23e77fe700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f23e80aab80 0x7f23e80abad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:40186/0 (socket says 172.21.15.23:40186) 2020-11-20T11:52:24.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.500+0000 7f23e77fe700 1 -- 172.21.15.23:0/249191960 learned_addr learned my addr 172.21.15.23:0/249191960 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:52:24.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.500+0000 7f23e7fff700 1 --2- 172.21.15.23:0/249191960 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f23e80a8570 0x7f23e80ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:24.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.500+0000 7f23e77fe700 1 -- 172.21.15.23:0/249191960 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f23e80a8570 msgr2=0x7f23e80ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:52:24.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.500+0000 7f23e77fe700 1 --2- 172.21.15.23:0/249191960 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f23e80a8570 0x7f23e80ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:24.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.500+0000 7f23e77fe700 1 -- 172.21.15.23:0/249191960 >> v1:172.21.15.155:6789/0 conn(0x7f23e80aaf60 legacy=0x7f23e80ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:52:24.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.500+0000 7f23e77fe700 1 -- 172.21.15.23:0/249191960 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e813b010 con 0x7f23e80aab80 2020-11-20T11:52:24.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.500+0000 7f23e77fe700 1 --2- 172.21.15.23:0/249191960 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f23e80aab80 0x7f23e80abad0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f23d0009a10 tx=0x7f23d0006ef0).ready entity=mon.1 client_cookie=663e72f082dffc24 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:24.507 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.500+0000 7f23e67fc700 1 -- 172.21.15.23:0/249191960 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d000b8b0 con 0x7f23e80aab80 2020-11-20T11:52:24.508 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.500+0000 7f23e67fc700 1 -- 172.21.15.23:0/249191960 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23d0000e90 con 0x7f23e80aab80 2020-11-20T11:52:24.508 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.501+0000 7f23e67fc700 1 -- 172.21.15.23:0/249191960 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d000bc20 con 0x7f23e80aab80 2020-11-20T11:52:24.508 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.501+0000 7f23ecbdd700 1 -- 172.21.15.23:0/249191960 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f23e80aab80 msgr2=0x7f23e80abad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:24.508 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.501+0000 7f23ecbdd700 1 --2- 172.21.15.23:0/249191960 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f23e80aab80 0x7f23e80abad0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f23d0009a10 tx=0x7f23d0006ef0).stop 2020-11-20T11:52:24.508 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.501+0000 7f23ecbdd700 1 -- 172.21.15.23:0/249191960 shutdown_connections 2020-11-20T11:52:24.509 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.501+0000 7f23ecbdd700 1 --2- 172.21.15.23:0/249191960 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f23e80aab80 0x7f23e80abad0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:24.509 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.501+0000 7f23ecbdd700 1 --2- 172.21.15.23:0/249191960 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f23e80a8570 0x7f23e80ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:24.509 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.501+0000 7f23ecbdd700 1 -- 172.21.15.23:0/249191960 >> 172.21.15.23:0/249191960 conn(0x7f23e80678e0 msgr2=0x7f23e8067cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:24.509 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.501+0000 7f23ecbdd700 1 -- 172.21.15.23:0/249191960 shutdown_connections 2020-11-20T11:52:24.510 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.501+0000 7f23ecbdd700 1 -- 172.21.15.23:0/249191960 wait complete. 2020-11-20T11:52:24.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.502+0000 7f23ecbdd700 1 Processor -- start 2020-11-20T11:52:24.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.502+0000 7f23ecbdd700 1 -- start start 2020-11-20T11:52:24.512 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.503+0000 7f23ecbdd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23e80a8570 0x7f23e813e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:24.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.503+0000 7f23ecbdd700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f23e80aab80 0x7f23e80a65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:24.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.503+0000 7f23ecbdd700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f23e80aaf60 0x7f23e80a6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:24.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.503+0000 7f23ecbdd700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e80a79d0 con 0x7f23e80a8570 2020-11-20T11:52:24.513 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.503+0000 7f23ecbdd700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e80a7c70 con 0x7f23e80aaf60 2020-11-20T11:52:24.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.503+0000 7f23ecbdd700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23e80a7f10 con 0x7f23e80aab80 2020-11-20T11:52:24.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.503+0000 7f23e77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23e80a8570 0x7f23e813e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:24.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.503+0000 7f23e77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23e80a8570 0x7f23e813e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47294/0 (socket says 172.21.15.23:47294) 2020-11-20T11:52:24.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.503+0000 7f23e77fe700 1 -- 172.21.15.23:0/3009360425 learned_addr learned my addr 172.21.15.23:0/3009360425 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:52:24.514 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.503+0000 7f23e6ffd700 1 --2- 172.21.15.23:0/3009360425 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f23e80aab80 0x7f23e80a65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:24.515 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.503+0000 7f23e77fe700 1 -- 172.21.15.23:0/3009360425 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f23e80aab80 msgr2=0x7f23e80a65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:24.515 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.503+0000 7f23e77fe700 1 --2- 172.21.15.23:0/3009360425 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f23e80aab80 0x7f23e80a65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:24.515 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.503+0000 7f23e77fe700 1 -- 172.21.15.23:0/3009360425 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f23e80aaf60 msgr2=0x7f23e80a6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:24.515 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.503+0000 7f23e77fe700 1 --2- 172.21.15.23:0/3009360425 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f23e80aaf60 0x7f23e80a6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:52:24.515 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.503+0000 7f23e77fe700 1 -- 172.21.15.23:0/3009360425 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e813eb30 con 0x7f23e80a8570 2020-11-20T11:52:24.516 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.503+0000 7f23e77fe700 1 --2- 172.21.15.23:0/3009360425 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23e80a8570 0x7f23e813e130 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f23e8068550 tx=0x7f23d0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:24.516 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.504+0000 7f23df7fe700 1 -- 172.21.15.23:0/3009360425 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d0011dc0 con 0x7f23e80a8570 2020-11-20T11:52:24.516 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.504+0000 7f23df7fe700 1 -- 172.21.15.23:0/3009360425 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23d001d480 con 0x7f23e80a8570 2020-11-20T11:52:24.516 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.504+0000 7f23ecbdd700 1 -- 172.21.15.23:0/3009360425 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23e80a4660 con 0x7f23e80a8570 2020-11-20T11:52:24.516 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.504+0000 7f23df7fe700 1 -- 172.21.15.23:0/3009360425 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d001d820 con 0x7f23e80a8570 2020-11-20T11:52:24.517 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.504+0000 7f23ecbdd700 1 -- 172.21.15.23:0/3009360425 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23e809eea0 con 0x7f23e80a8570 2020-11-20T11:52:24.517 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.505+0000 7f23ecbdd700 1 -- 172.21.15.23:0/3009360425 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f23e8057dc0 con 0x7f23e80a8570 2020-11-20T11:52:24.517 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.505+0000 7f23df7fe700 1 -- 172.21.15.23:0/3009360425 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f23d0017070 con 0x7f23e80a8570 2020-11-20T11:52:24.517 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.505+0000 7f23df7fe700 1 --2- 172.21.15.23:0/3009360425 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f23c4056d60 0x7f23c4059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:24.517 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.505+0000 7f23df7fe700 1 -- 172.21.15.23:0/3009360425 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f23d0071f00 con 0x7f23e80a8570 2020-11-20T11:52:24.518 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.506+0000 7f23df7fe700 1 --2- 172.21.15.23:0/3009360425 >> [v2:172.21.15.23:6823/23114,v1:172.21.15.23:6825/23114] conn(0x7f23c405e940 0x7f23c4060d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:24.518 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.506+0000 7f23df7fe700 1 -- 172.21.15.23:0/3009360425 --> [v2:172.21.15.23:6823/23114,v1:172.21.15.23:6825/23114] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f23c40613c0 con 0x7f23c405e940 2020-11-20T11:52:24.518 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.506+0000 7f23df7fe700 1 -- 172.21.15.23:0/3009360425 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f23d0074570 con 0x7f23e80a8570 2020-11-20T11:52:24.518 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.506+0000 7f23e6ffd700 1 --2- 172.21.15.23:0/3009360425 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f23c4056d60 0x7f23c4059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:24.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.510+0000 7f23e6ffd700 1 --2- 172.21.15.23:0/3009360425 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f23c4056d60 0x7f23c4059220 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f23e809e4a0 tx=0x7f23d8008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:24.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.510+0000 7f23e7fff700 1 --2- 172.21.15.23:0/3009360425 >> [v2:172.21.15.23:6823/23114,v1:172.21.15.23:6825/23114] conn(0x7f23c405e940 0x7f23c4060d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:24.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.511+0000 7f23e7fff700 1 --2- 172.21.15.23:0/3009360425 >> [v2:172.21.15.23:6823/23114,v1:172.21.15.23:6825/23114] conn(0x7f23c405e940 0x7f23c4060d60 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:24.520 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.512+0000 7f23df7fe700 1 -- 172.21.15.23:0/3009360425 <== osd.3 v2:172.21.15.23:6823/23114 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7f23c40613c0 con 0x7f23c405e940 2020-11-20T11:52:24.529 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.526+0000 7f23ecbdd700 1 -- 172.21.15.23:0/3009360425 --> [v2:172.21.15.23:6823/23114,v1:172.21.15.23:6825/23114] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f23e813b4b0 con 0x7f23c405e940 2020-11-20T11:52:24.530 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.526+0000 7f23df7fe700 1 -- 172.21.15.23:0/3009360425 <== osd.3 v2:172.21.15.23:6823/23114 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f23e813b4b0 con 0x7f23c405e940 2020-11-20T11:52:24.530 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.527+0000 7f23ecbdd700 1 -- 172.21.15.23:0/3009360425 >> [v2:172.21.15.23:6823/23114,v1:172.21.15.23:6825/23114] conn(0x7f23c405e940 msgr2=0x7f23c4060d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:24.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.527+0000 7f23ecbdd700 1 --2- 172.21.15.23:0/3009360425 >> [v2:172.21.15.23:6823/23114,v1:172.21.15.23:6825/23114] conn(0x7f23c405e940 0x7f23c4060d60 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:24.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.527+0000 7f23ecbdd700 1 -- 172.21.15.23:0/3009360425 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f23c4056d60 msgr2=0x7f23c4059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:24.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.527+0000 7f23ecbdd700 1 --2- 172.21.15.23:0/3009360425 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f23c4056d60 0x7f23c4059220 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f23e809e4a0 tx=0x7f23d8008040).stop 2020-11-20T11:52:24.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.527+0000 7f23ecbdd700 1 -- 172.21.15.23:0/3009360425 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23e80a8570 msgr2=0x7f23e813e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:24.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.527+0000 7f23ecbdd700 1 --2- 172.21.15.23:0/3009360425 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23e80a8570 0x7f23e813e130 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f23e8068550 tx=0x7f23d0016000).stop 2020-11-20T11:52:24.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.527+0000 7f23e7fff700 1 -- 172.21.15.23:0/3009360425 reap_dead start 2020-11-20T11:52:24.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.527+0000 7f23ecbdd700 1 -- 172.21.15.23:0/3009360425 shutdown_connections 2020-11-20T11:52:24.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.527+0000 7f23ecbdd700 1 -- 172.21.15.23:0/3009360425 >> 172.21.15.23:0/3009360425 conn(0x7f23e80678e0 msgr2=0x7f23e80ac150 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:24.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.527+0000 7f23ecbdd700 1 -- 172.21.15.23:0/3009360425 shutdown_connections 2020-11-20T11:52:24.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.527+0000 7f23ecbdd700 1 -- 172.21.15.23:0/3009360425 wait complete. 2020-11-20T11:52:24.533 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:52:24.533 INFO:teuthology.orchestra.run.smithi023.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2020-11-20T11:52:24.533 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:52:24.543 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2020-11-20T11:52:24.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.685+0000 7f6f7ac77700 1 Processor -- start 2020-11-20T11:52:24.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.685+0000 7f6f7ac77700 1 -- start start 2020-11-20T11:52:24.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.686+0000 7f6f7ac77700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6f740a9040 0x7f6f740aaf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:24.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.686+0000 7f6f7ac77700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6f740ace70 0x7f6f7413de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:24.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.686+0000 7f6f7ac77700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f740ad860 con 0x7f6f740ad2f0 2020-11-20T11:52:24.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.686+0000 7f6f7ac77700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f740aa8a0 con 0x7f6f740a9040 2020-11-20T11:52:24.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.686+0000 7f6f7ac77700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f740aaa90 con 0x7f6f740ace70 2020-11-20T11:52:24.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.686+0000 7f6f79c75700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f6f740a9040 0x7f6f740aaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:24.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.686+0000 7f6f7a476700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f6f740ad2f0 0x7f6f7413e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:33324/0 (socket says 172.21.15.23:33324) 2020-11-20T11:52:24.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.686+0000 7f6f7a476700 1 -- 172.21.15.23:0/4113245948 learned_addr learned my addr 172.21.15.23:0/4113245948 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:52:24.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.686+0000 7f6f79474700 1 --2- 172.21.15.23:0/4113245948 >> v2:172.21.15.155:3300/0 conn(0x7f6f740ace70 0x7f6f7413de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:24.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.686+0000 7f6f79c75700 1 -- 172.21.15.23:0/4113245948 >> v2:172.21.15.155:3300/0 conn(0x7f6f740ace70 msgr2=0x7f6f7413de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:52:24.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.686+0000 7f6f79c75700 1 --2- 172.21.15.23:0/4113245948 >> v2:172.21.15.155:3300/0 conn(0x7f6f740ace70 0x7f6f7413de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:24.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.686+0000 7f6f79c75700 1 -- 172.21.15.23:0/4113245948 >> v1:172.21.15.23:6789/0 conn(0x7f6f740ad2f0 legacy=0x7f6f7413e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:24.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.686+0000 7f6f79c75700 1 -- 172.21.15.23:0/4113245948 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f740a9420 con 0x7f6f740a9040 2020-11-20T11:52:24.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.686+0000 7f6f79c75700 1 --2- 172.21.15.23:0/4113245948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f740a9040 0x7f6f740aaf30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f6f740a9610 tx=0x7f6f64006dd0).ready entity=mon.0 client_cookie=e669cf1cbf953238 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:24.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.686+0000 7f6f78c73700 1 -- 172.21.15.23:0/4113245948 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f64006030 con 0x7f6f740a9040 2020-11-20T11:52:24.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.687+0000 7f6f78c73700 1 -- 172.21.15.23:0/4113245948 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f64006190 con 0x7f6f740a9040 2020-11-20T11:52:24.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.687+0000 7f6f78c73700 1 -- 172.21.15.23:0/4113245948 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f64006510 con 0x7f6f740a9040 2020-11-20T11:52:24.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.687+0000 7f6f7ac77700 1 -- 172.21.15.23:0/4113245948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f740a9040 msgr2=0x7f6f740aaf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:24.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.687+0000 7f6f7ac77700 1 --2- 172.21.15.23:0/4113245948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f740a9040 0x7f6f740aaf30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f6f740a9610 tx=0x7f6f64006dd0).stop 2020-11-20T11:52:24.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.687+0000 7f6f7ac77700 1 -- 172.21.15.23:0/4113245948 shutdown_connections 2020-11-20T11:52:24.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.687+0000 7f6f7ac77700 1 --2- 172.21.15.23:0/4113245948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f740a9040 0x7f6f740aaf30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:24.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.687+0000 7f6f7ac77700 1 --2- 172.21.15.23:0/4113245948 >> v2:172.21.15.155:3300/0 conn(0x7f6f740ace70 0x7f6f7413de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:24.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.687+0000 7f6f7ac77700 1 -- 172.21.15.23:0/4113245948 >> 172.21.15.23:0/4113245948 conn(0x7f6f740683b0 msgr2=0x7f6f740b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:24.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.687+0000 7f6f7ac77700 1 -- 172.21.15.23:0/4113245948 shutdown_connections 2020-11-20T11:52:24.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.687+0000 7f6f7ac77700 1 -- 172.21.15.23:0/4113245948 wait complete. 2020-11-20T11:52:24.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.688+0000 7f6f7ac77700 1 Processor -- start 2020-11-20T11:52:24.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.688+0000 7f6f7ac77700 1 -- start start 2020-11-20T11:52:24.698 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.689+0000 7f6f7ac77700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f740a9040 0x7f6f7409f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:24.699 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.689+0000 7f6f7ac77700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f740ace70 0x7f6f740a09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:24.699 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.689+0000 7f6f7ac77700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f740ad2f0 0x7f6f740a74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:24.699 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.689+0000 7f6f7ac77700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7409f640 con 0x7f6f740ace70 2020-11-20T11:52:24.699 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.689+0000 7f6f7ac77700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7409db40 con 0x7f6f740a9040 2020-11-20T11:52:24.699 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.689+0000 7f6f7ac77700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7409dde0 con 0x7f6f740ad2f0 2020-11-20T11:52:24.700 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.689+0000 7f6f79474700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f740ace70 0x7f6f740a09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:24.700 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.689+0000 7f6f79474700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f740ace70 0x7f6f740a09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47310/0 (socket says 172.21.15.23:47310) 2020-11-20T11:52:24.700 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.689+0000 7f6f79474700 1 -- 172.21.15.23:0/538204842 learned_addr learned my addr 172.21.15.23:0/538204842 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:52:24.700 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.689+0000 7f6f79c75700 1 --2- 172.21.15.23:0/538204842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f740a9040 0x7f6f7409f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:24.700 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.689+0000 7f6f7a476700 1 --2- 172.21.15.23:0/538204842 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f740ad2f0 0x7f6f740a74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:24.702 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.689+0000 7f6f79474700 1 -- 172.21.15.23:0/538204842 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f740ad2f0 msgr2=0x7f6f740a74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:24.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.689+0000 7f6f79474700 1 --2- 172.21.15.23:0/538204842 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f740ad2f0 0x7f6f740a74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:24.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.689+0000 7f6f79474700 1 -- 172.21.15.23:0/538204842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f740a9040 msgr2=0x7f6f7409f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:24.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.689+0000 7f6f79474700 1 --2- 172.21.15.23:0/538204842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f740a9040 0x7f6f7409f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:24.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.689+0000 7f6f79474700 1 -- 172.21.15.23:0/538204842 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f7409e080 con 0x7f6f740ace70 2020-11-20T11:52:24.703 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.689+0000 7f6f79474700 1 --2- 172.21.15.23:0/538204842 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f740ace70 0x7f6f740a09e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f6f7000db30 tx=0x7f6f70008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:24.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.689+0000 7f6f61ffb700 1 -- 172.21.15.23:0/538204842 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7000de40 con 0x7f6f740ace70 2020-11-20T11:52:24.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.690+0000 7f6f61ffb700 1 -- 172.21.15.23:0/538204842 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f700097c0 con 0x7f6f740ace70 2020-11-20T11:52:24.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.690+0000 7f6f7ac77700 1 -- 172.21.15.23:0/538204842 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f7409e520 con 0x7f6f740ace70 2020-11-20T11:52:24.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.690+0000 7f6f61ffb700 1 -- 172.21.15.23:0/538204842 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f70009b40 con 0x7f6f740ace70 2020-11-20T11:52:24.704 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.690+0000 7f6f7ac77700 1 -- 172.21.15.23:0/538204842 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f74140790 con 0x7f6f740ace70 2020-11-20T11:52:24.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.691+0000 7f6f7ac77700 1 -- 172.21.15.23:0/538204842 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f74057dc0 con 0x7f6f740ace70 2020-11-20T11:52:24.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.691+0000 7f6f61ffb700 1 -- 172.21.15.23:0/538204842 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f6f70013480 con 0x7f6f740ace70 2020-11-20T11:52:24.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.691+0000 7f6f61ffb700 1 --2- 172.21.15.23:0/538204842 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6f4c056d90 0x7f6f4c059250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:24.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.691+0000 7f6f61ffb700 1 -- 172.21.15.23:0/538204842 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f6f700702a0 con 0x7f6f740ace70 2020-11-20T11:52:24.705 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.691+0000 7f6f79c75700 1 --2- 172.21.15.23:0/538204842 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6f4c056d90 0x7f6f4c059250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:24.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.692+0000 7f6f79c75700 1 --2- 172.21.15.23:0/538204842 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6f4c056d90 0x7f6f4c059250 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f6f740a9610 tx=0x7f6f64015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:24.706 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.699+0000 7f6f61ffb700 1 -- 172.21.15.23:0/538204842 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f6f70045530 con 0x7f6f740ace70 2020-11-20T11:52:24.911 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.908+0000 7f6f7ac77700 1 -- 172.21.15.23:0/538204842 --> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f7409d9c0 con 0x7f6f4c056d90 2020-11-20T11:52:24.911 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.908+0000 7f6f61ffb700 1 -- 172.21.15.23:0/538204842 <== mgr.4105 v2:172.21.15.155:6832/22693 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6f6400b5d0 con 0x7f6f4c056d90 2020-11-20T11:52:24.914 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.910+0000 7f6f7ac77700 1 -- 172.21.15.23:0/538204842 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6f4c056d90 msgr2=0x7f6f4c059250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:24.914 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.910+0000 7f6f7ac77700 1 --2- 172.21.15.23:0/538204842 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6f4c056d90 0x7f6f4c059250 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f6f740a9610 tx=0x7f6f64015000).stop 2020-11-20T11:52:24.914 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.910+0000 7f6f7ac77700 1 -- 172.21.15.23:0/538204842 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f740ace70 msgr2=0x7f6f740a09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:24.915 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.910+0000 7f6f7ac77700 1 --2- 172.21.15.23:0/538204842 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f740ace70 0x7f6f740a09e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f6f7000db30 tx=0x7f6f70008040).stop 2020-11-20T11:52:24.915 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.910+0000 7f6f7ac77700 1 -- 172.21.15.23:0/538204842 shutdown_connections 2020-11-20T11:52:24.915 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.910+0000 7f6f7ac77700 1 --2- 172.21.15.23:0/538204842 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f6f4c056d90 0x7f6f4c059250 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:24.915 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.910+0000 7f6f7ac77700 1 --2- 172.21.15.23:0/538204842 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6f740ad2f0 0x7f6f740a74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:24.916 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.910+0000 7f6f7ac77700 1 --2- 172.21.15.23:0/538204842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6f740a9040 0x7f6f7409f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:24.916 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.910+0000 7f6f7ac77700 1 --2- 172.21.15.23:0/538204842 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6f740ace70 0x7f6f740a09e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:24.916 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.910+0000 7f6f7ac77700 1 -- 172.21.15.23:0/538204842 >> 172.21.15.23:0/538204842 conn(0x7f6f740683b0 msgr2=0x7f6f740aba40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:24.917 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.910+0000 7f6f7ac77700 1 -- 172.21.15.23:0/538204842 shutdown_connections 2020-11-20T11:52:24.917 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:24.910+0000 7f6f7ac77700 1 -- 172.21.15.23:0/538204842 wait complete. 2020-11-20T11:52:24.917 INFO:teuthology.orchestra.run.smithi023.stderr:instructed osd(s) 3 to deep-scrub 2020-11-20T11:52:24.934 INFO:tasks.ceph:Scrubbing osd.4 2020-11-20T11:52:24.935 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2020-11-20T11:52:25.096 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.089+0000 7f538b976700 1 Processor -- start 2020-11-20T11:52:25.097 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.090+0000 7f538b976700 1 -- start start 2020-11-20T11:52:25.097 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.091+0000 7f538b976700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53840a9ad0 0x7f5384067580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.091+0000 7f538b976700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5384067a70 0x7f5384135610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.091+0000 7f538b976700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53840aaa00 con 0x7f53840a28c0 2020-11-20T11:52:25.098 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.091+0000 7f538b976700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53840aa2c0 con 0x7f5384067a70 2020-11-20T11:52:25.099 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.091+0000 7f538b976700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53840aa560 con 0x7f53840a9ad0 2020-11-20T11:52:25.099 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.091+0000 7f5389f13700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5384067a70 0x7f5384135610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:25.104 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.091+0000 7f5389f13700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5384067a70 0x7f5384135610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:40214/0 (socket says 172.21.15.23:40214) 2020-11-20T11:52:25.104 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.091+0000 7f5389f13700 1 -- 172.21.15.23:0/544908972 learned_addr learned my addr 172.21.15.23:0/544908972 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:52:25.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.091+0000 7f5388f11700 1 --2- 172.21.15.23:0/544908972 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53840a9ad0 0x7f5384067580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:25.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.091+0000 7f5388f11700 1 -- 172.21.15.23:0/544908972 >> v2:172.21.15.155:3300/0 conn(0x7f5384067a70 msgr2=0x7f5384135610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:52:25.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.092+0000 7f5388f11700 1 --2- 172.21.15.23:0/544908972 >> v2:172.21.15.155:3300/0 conn(0x7f5384067a70 0x7f5384135610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:25.105 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.092+0000 7f5388f11700 1 -- 172.21.15.23:0/544908972 >> v1:172.21.15.155:6789/0 conn(0x7f53840a28c0 legacy=0x7f53840a6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:25.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.092+0000 7f5388f11700 1 -- 172.21.15.23:0/544908972 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53840a2ca0 con 0x7f53840a9ad0 2020-11-20T11:52:25.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.092+0000 7f5388f11700 1 --2- 172.21.15.23:0/544908972 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53840a9ad0 0x7f5384067580 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f5378006fd0 tx=0x7f5378006e30).ready entity=mon.2 client_cookie=420743d5cdd1268a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:25.106 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.093+0000 7f5389f13700 1 --2- 172.21.15.23:0/544908972 >> v2:172.21.15.155:3300/0 conn(0x7f5384067a70 0x7f5384135610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:52:25.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.096+0000 7f5377fff700 1 -- 172.21.15.23:0/544908972 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5378009870 con 0x7f53840a9ad0 2020-11-20T11:52:25.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.096+0000 7f5377fff700 1 -- 172.21.15.23:0/544908972 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5378000e90 con 0x7f53840a9ad0 2020-11-20T11:52:25.109 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.096+0000 7f5377fff700 1 -- 172.21.15.23:0/544908972 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5378009be0 con 0x7f53840a9ad0 2020-11-20T11:52:25.110 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.096+0000 7f538b976700 1 -- 172.21.15.23:0/544908972 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53840a9ad0 msgr2=0x7f5384067580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:25.110 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.096+0000 7f538b976700 1 --2- 172.21.15.23:0/544908972 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53840a9ad0 0x7f5384067580 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f5378006fd0 tx=0x7f5378006e30).stop 2020-11-20T11:52:25.110 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.096+0000 7f538b976700 1 -- 172.21.15.23:0/544908972 shutdown_connections 2020-11-20T11:52:25.111 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.096+0000 7f538b976700 1 --2- 172.21.15.23:0/544908972 >> v2:172.21.15.155:3300/0 conn(0x7f5384067a70 0x7f5384135610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:25.111 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.096+0000 7f538b976700 1 --2- 172.21.15.23:0/544908972 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53840a9ad0 0x7f5384067580 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:25.111 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.096+0000 7f538b976700 1 -- 172.21.15.23:0/544908972 >> 172.21.15.23:0/544908972 conn(0x7f53840666e0 msgr2=0x7f5384063d70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:25.111 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.097+0000 7f538b976700 1 -- 172.21.15.23:0/544908972 shutdown_connections 2020-11-20T11:52:25.111 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.097+0000 7f538b976700 1 -- 172.21.15.23:0/544908972 wait complete. 2020-11-20T11:52:25.112 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.098+0000 7f538b976700 1 Processor -- start 2020-11-20T11:52:25.112 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.099+0000 7f538b976700 1 -- start start 2020-11-20T11:52:25.112 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.100+0000 7f538b976700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5384067a70 0x7f53840a83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.112 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.100+0000 7f538b976700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f53840a28c0 0x7f53840a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.113 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.100+0000 7f538b976700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53840a9ad0 0x7f53840a5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.113 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.100+0000 7f538b976700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53840a88d0 con 0x7f5384067a70 2020-11-20T11:52:25.113 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.100+0000 7f538b976700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53840a4bf0 con 0x7f53840a28c0 2020-11-20T11:52:25.113 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.100+0000 7f538b976700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53840a4e90 con 0x7f53840a9ad0 2020-11-20T11:52:25.114 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.105+0000 7f5388f11700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f53840a28c0 0x7f53840a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:25.114 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.105+0000 7f5388f11700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f53840a28c0 0x7f53840a4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:40218/0 (socket says 172.21.15.23:40218) 2020-11-20T11:52:25.114 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.105+0000 7f5388f11700 1 -- 172.21.15.23:0/3911410415 learned_addr learned my addr 172.21.15.23:0/3911410415 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:52:25.115 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.105+0000 7f5388f11700 1 -- 172.21.15.23:0/3911410415 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53840a9ad0 msgr2=0x7f53840a5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:25.115 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.105+0000 7f5388f11700 1 --2- 172.21.15.23:0/3911410415 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f53840a9ad0 0x7f53840a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:52:25.115 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.105+0000 7f5388f11700 1 -- 172.21.15.23:0/3911410415 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5384067a70 msgr2=0x7f53840a83e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:25.115 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.105+0000 7f5388f11700 1 --2- 172.21.15.23:0/3911410415 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5384067a70 0x7f53840a83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:52:25.116 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.105+0000 7f5388f11700 1 -- 172.21.15.23:0/3911410415 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53840a4920 con 0x7f53840a28c0 2020-11-20T11:52:25.116 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.106+0000 7f5388f11700 1 --2- 172.21.15.23:0/3911410415 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f53840a28c0 0x7f53840a4010 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f53840aa6a0 tx=0x7f5378015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:25.116 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.106+0000 7f53757fa700 1 -- 172.21.15.23:0/3911410415 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5378011c60 con 0x7f53840a28c0 2020-11-20T11:52:25.117 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.106+0000 7f53757fa700 1 -- 172.21.15.23:0/3911410415 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5378011dc0 con 0x7f53840a28c0 2020-11-20T11:52:25.117 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.106+0000 7f538b976700 1 -- 172.21.15.23:0/3911410415 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53840a8cf0 con 0x7f53840a28c0 2020-11-20T11:52:25.119 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.107+0000 7f53757fa700 1 -- 172.21.15.23:0/3911410415 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f537801e690 con 0x7f53840a28c0 2020-11-20T11:52:25.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.107+0000 7f538b976700 1 -- 172.21.15.23:0/3911410415 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53840a46c0 con 0x7f53840a28c0 2020-11-20T11:52:25.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.108+0000 7f538b976700 1 -- 172.21.15.23:0/3911410415 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f53840676f0 con 0x7f53840a28c0 2020-11-20T11:52:25.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.108+0000 7f53757fa700 1 -- 172.21.15.23:0/3911410415 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5378025030 con 0x7f53840a28c0 2020-11-20T11:52:25.120 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.109+0000 7f53757fa700 1 --2- 172.21.15.23:0/3911410415 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5360056d60 0x7f5360059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.121 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.109+0000 7f53757fa700 1 -- 172.21.15.23:0/3911410415 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f5378072c60 con 0x7f53840a28c0 2020-11-20T11:52:25.121 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.109+0000 7f53757fa700 1 --2- 172.21.15.23:0/3911410415 >> [v2:172.21.15.155:6800/23280,v1:172.21.15.155:6801/23280] conn(0x7f536005e7d0 0x7f5360060bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.121 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.109+0000 7f53757fa700 1 -- 172.21.15.23:0/3911410415 --> [v2:172.21.15.155:6800/23280,v1:172.21.15.155:6801/23280] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5360061250 con 0x7f536005e7d0 2020-11-20T11:52:25.122 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.109+0000 7f53757fa700 1 -- 172.21.15.23:0/3911410415 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5378071340 con 0x7f53840a28c0 2020-11-20T11:52:25.122 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.109+0000 7f5389712700 1 --2- 172.21.15.23:0/3911410415 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5360056d60 0x7f5360059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:25.122 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.109+0000 7f5389f13700 1 --2- 172.21.15.23:0/3911410415 >> [v2:172.21.15.155:6800/23280,v1:172.21.15.155:6801/23280] conn(0x7f536005e7d0 0x7f5360060bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:25.122 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.110+0000 7f5389712700 1 --2- 172.21.15.23:0/3911410415 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5360056d60 0x7f5360059220 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f53840a3b10 tx=0x7f5380007810).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:25.123 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.110+0000 7f5389f13700 1 --2- 172.21.15.23:0/3911410415 >> [v2:172.21.15.155:6800/23280,v1:172.21.15.155:6801/23280] conn(0x7f536005e7d0 0x7f5360060bf0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:25.123 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.112+0000 7f53757fa700 1 -- 172.21.15.23:0/3911410415 <== osd.4 v2:172.21.15.155:6800/23280 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7f5360061250 con 0x7f536005e7d0 2020-11-20T11:52:25.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.132+0000 7f538b976700 1 -- 172.21.15.23:0/3911410415 --> [v2:172.21.15.155:6800/23280,v1:172.21.15.155:6801/23280] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f53840676f0 con 0x7f536005e7d0 2020-11-20T11:52:25.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.133+0000 7f53757fa700 1 -- 172.21.15.23:0/3911410415 <== osd.4 v2:172.21.15.155:6800/23280 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f53840676f0 con 0x7f536005e7d0 2020-11-20T11:52:25.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.133+0000 7f538b976700 1 -- 172.21.15.23:0/3911410415 >> [v2:172.21.15.155:6800/23280,v1:172.21.15.155:6801/23280] conn(0x7f536005e7d0 msgr2=0x7f5360060bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:25.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.133+0000 7f538b976700 1 --2- 172.21.15.23:0/3911410415 >> [v2:172.21.15.155:6800/23280,v1:172.21.15.155:6801/23280] conn(0x7f536005e7d0 0x7f5360060bf0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:25.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.133+0000 7f538b976700 1 -- 172.21.15.23:0/3911410415 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5360056d60 msgr2=0x7f5360059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:25.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.133+0000 7f538b976700 1 --2- 172.21.15.23:0/3911410415 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5360056d60 0x7f5360059220 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f53840a3b10 tx=0x7f5380007810).stop 2020-11-20T11:52:25.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.133+0000 7f538b976700 1 -- 172.21.15.23:0/3911410415 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f53840a28c0 msgr2=0x7f53840a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:25.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.133+0000 7f538b976700 1 --2- 172.21.15.23:0/3911410415 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f53840a28c0 0x7f53840a4010 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f53840aa6a0 tx=0x7f5378015000).stop 2020-11-20T11:52:25.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.133+0000 7f5389f13700 1 -- 172.21.15.23:0/3911410415 reap_dead start 2020-11-20T11:52:25.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.133+0000 7f538b976700 1 -- 172.21.15.23:0/3911410415 shutdown_connections 2020-11-20T11:52:25.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.133+0000 7f538b976700 1 -- 172.21.15.23:0/3911410415 >> 172.21.15.23:0/3911410415 conn(0x7f53840666e0 msgr2=0x7f5384068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:25.144 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:52:25.144 INFO:teuthology.orchestra.run.smithi023.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2020-11-20T11:52:25.144 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:52:25.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.134+0000 7f538b976700 1 -- 172.21.15.23:0/3911410415 shutdown_connections 2020-11-20T11:52:25.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.134+0000 7f538b976700 1 -- 172.21.15.23:0/3911410415 wait complete. 2020-11-20T11:52:25.163 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2020-11-20T11:52:25.320 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.317+0000 7f36164c6700 1 Processor -- start 2020-11-20T11:52:25.321 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.318+0000 7f36164c6700 1 -- start start 2020-11-20T11:52:25.322 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.318+0000 7f36164c6700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f36100a6e10 0x7f3610061850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.322 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.319+0000 7f36164c6700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3610062ad0 con 0x7f3610137870 2020-11-20T11:52:25.322 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.319+0000 7f36164c6700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f361013b580 con 0x7f3610137410 2020-11-20T11:52:25.322 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.319+0000 7f36164c6700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36100a61f0 con 0x7f36100a6e10 2020-11-20T11:52:25.323 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.319+0000 7f360f7fe700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f3610137410 0x7f361013a760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:39558/0 (socket says 172.21.15.23:39558) 2020-11-20T11:52:25.323 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.319+0000 7f360f7fe700 1 -- 172.21.15.23:0/2375745572 learned_addr learned my addr 172.21.15.23:0/2375745572 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:52:25.323 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.319+0000 7f3614a63700 1 --1- 172.21.15.23:0/2375745572 >> v1:172.21.15.23:6789/0 conn(0x7f3610137870 0x7f361013ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:33354/0 (socket says 172.21.15.23:33354) 2020-11-20T11:52:25.323 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.319+0000 7f360ffff700 1 --2- 172.21.15.23:0/2375745572 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f36100a6e10 0x7f3610061850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:25.325 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.319+0000 7f360effd700 1 -- 172.21.15.23:0/2375745572 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1833955491 0 0) 0x7f361013b580 con 0x7f3610137410 2020-11-20T11:52:25.325 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.319+0000 7f360effd700 1 -- 172.21.15.23:0/2375745572 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35fc003280 con 0x7f3610137410 2020-11-20T11:52:25.326 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.319+0000 7f360effd700 1 -- 172.21.15.23:0/2375745572 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396601348 0 0) 0x7f3610062ad0 con 0x7f3610137870 2020-11-20T11:52:25.326 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.319+0000 7f360effd700 1 -- 172.21.15.23:0/2375745572 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f361013b580 con 0x7f3610137870 2020-11-20T11:52:25.326 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.319+0000 7f360effd700 1 -- 172.21.15.23:0/2375745572 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2389859670 0 0) 0x7f35fc003280 con 0x7f3610137410 2020-11-20T11:52:25.326 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.319+0000 7f360effd700 1 -- 172.21.15.23:0/2375745572 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f36100a6e10 msgr2=0x7f3610061850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:52:25.326 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.319+0000 7f360effd700 1 --2- 172.21.15.23:0/2375745572 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f36100a6e10 0x7f3610061850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:25.327 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.319+0000 7f360effd700 1 -- 172.21.15.23:0/2375745572 >> v1:172.21.15.23:6789/0 conn(0x7f3610137870 legacy=0x7f361013ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:25.327 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.320+0000 7f360effd700 1 -- 172.21.15.23:0/2375745572 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36100a63e0 con 0x7f3610137410 2020-11-20T11:52:25.327 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.320+0000 7f360effd700 1 -- 172.21.15.23:0/2375745572 <== mon.1 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f3604001520 con 0x7f3610137410 2020-11-20T11:52:25.327 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.320+0000 7f360effd700 1 -- 172.21.15.23:0/2375745572 >> v1:172.21.15.155:6789/0 conn(0x7f3610137410 legacy=0x7f361013a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:25.328 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.320+0000 7f360effd700 1 --2- 172.21.15.23:0/2375745572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35fc004990 0x7f35fc004d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.328 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.320+0000 7f360effd700 1 --2- 172.21.15.23:0/2375745572 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35fc0034c0 0x7f35fc003940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.328 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.320+0000 7f360effd700 1 --2- 172.21.15.23:0/2375745572 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35fc003e30 0x7f35fc00b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.328 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.320+0000 7f360effd700 1 -- 172.21.15.23:0/2375745572 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35fc0042b0 con 0x7f35fc004990 2020-11-20T11:52:25.328 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.320+0000 7f360effd700 1 -- 172.21.15.23:0/2375745572 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35fc00bc60 con 0x7f35fc003e30 2020-11-20T11:52:25.329 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.320+0000 7f360effd700 1 -- 172.21.15.23:0/2375745572 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35fc00bda0 con 0x7f35fc0034c0 2020-11-20T11:52:25.329 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.320+0000 7f360ffff700 1 --2- 172.21.15.23:0/2375745572 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f36100a6e10 0x7f3610061850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:52:25.329 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.320+0000 7f3614a63700 1 --2- 172.21.15.23:0/2375745572 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35fc003e30 0x7f35fc00b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:25.329 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.320+0000 7f360ffff700 1 --2- 172.21.15.23:0/2375745572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35fc004990 0x7f35fc004d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:25.329 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.320+0000 7f360f7fe700 1 --2- 172.21.15.23:0/2375745572 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35fc0034c0 0x7f35fc003940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:25.330 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.320+0000 7f3614a63700 1 -- 172.21.15.23:0/2375745572 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35fc0034c0 msgr2=0x7f35fc003940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:52:25.330 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.320+0000 7f3614a63700 1 --2- 172.21.15.23:0/2375745572 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35fc0034c0 0x7f35fc003940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:25.330 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.320+0000 7f3614a63700 1 -- 172.21.15.23:0/2375745572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35fc004990 msgr2=0x7f35fc004d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:52:25.330 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.320+0000 7f3614a63700 1 --2- 172.21.15.23:0/2375745572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35fc004990 0x7f35fc004d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:25.330 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.320+0000 7f3614a63700 1 -- 172.21.15.23:0/2375745572 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36100a63e0 con 0x7f35fc003e30 2020-11-20T11:52:25.331 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.320+0000 7f360f7fe700 1 --2- 172.21.15.23:0/2375745572 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f35fc0034c0 0x7f35fc003940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:52:25.331 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.320+0000 7f3614a63700 1 --2- 172.21.15.23:0/2375745572 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35fc003e30 0x7f35fc00b770 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f360800c490 tx=0x7f36080021f0).ready entity=mon.1 client_cookie=5c19a84840892387 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:25.331 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.320+0000 7f360effd700 1 -- 172.21.15.23:0/2375745572 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360800ce40 con 0x7f35fc003e30 2020-11-20T11:52:25.331 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.321+0000 7f360effd700 1 -- 172.21.15.23:0/2375745572 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3608004ad0 con 0x7f35fc003e30 2020-11-20T11:52:25.331 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.321+0000 7f360effd700 1 -- 172.21.15.23:0/2375745572 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3608004e90 con 0x7f35fc003e30 2020-11-20T11:52:25.332 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.321+0000 7f36164c6700 1 -- 172.21.15.23:0/2375745572 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35fc003e30 msgr2=0x7f35fc00b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:25.332 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.321+0000 7f36164c6700 1 --2- 172.21.15.23:0/2375745572 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35fc003e30 0x7f35fc00b770 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f360800c490 tx=0x7f36080021f0).stop 2020-11-20T11:52:25.332 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.321+0000 7f3614a63700 1 -- 172.21.15.23:0/2375745572 reap_dead start 2020-11-20T11:52:25.332 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.321+0000 7f36164c6700 1 -- 172.21.15.23:0/2375745572 shutdown_connections 2020-11-20T11:52:25.332 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.321+0000 7f36164c6700 1 -- 172.21.15.23:0/2375745572 >> 172.21.15.23:0/2375745572 conn(0x7f3610060c80 msgr2=0x7f36100acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:25.332 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.322+0000 7f36164c6700 1 -- 172.21.15.23:0/2375745572 shutdown_connections 2020-11-20T11:52:25.333 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.322+0000 7f36164c6700 1 -- 172.21.15.23:0/2375745572 wait complete. 2020-11-20T11:52:25.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.323+0000 7f36164c6700 1 Processor -- start 2020-11-20T11:52:25.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.323+0000 7f36164c6700 1 -- start start 2020-11-20T11:52:25.335 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.324+0000 7f36164c6700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35fc003e30 0x7f361009dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.324+0000 7f36164c6700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3610067b70 0x7f361009f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.324+0000 7f36164c6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3610148f40 0x7f36101493c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.324+0000 7f36164c6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36101433d0 con 0x7f3610148f40 2020-11-20T11:52:25.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.324+0000 7f36164c6700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361009e5c0 con 0x7f35fc003e30 2020-11-20T11:52:25.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.324+0000 7f36164c6700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f361009e320 con 0x7f3610067b70 2020-11-20T11:52:25.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.324+0000 7f3614a63700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3610148f40 0x7f36101493c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:25.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.324+0000 7f360ffff700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35fc003e30 0x7f361009dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:25.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.324+0000 7f3614a63700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3610148f40 0x7f36101493c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47348/0 (socket says 172.21.15.23:47348) 2020-11-20T11:52:25.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.324+0000 7f3614a63700 1 -- 172.21.15.23:0/1730891147 learned_addr learned my addr 172.21.15.23:0/1730891147 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:52:25.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.324+0000 7f360f7fe700 1 --2- 172.21.15.23:0/1730891147 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3610067b70 0x7f361009f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:25.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.324+0000 7f3614a63700 1 -- 172.21.15.23:0/1730891147 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3610067b70 msgr2=0x7f361009f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:25.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.324+0000 7f3614a63700 1 --2- 172.21.15.23:0/1730891147 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3610067b70 0x7f361009f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:25.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.324+0000 7f3614a63700 1 -- 172.21.15.23:0/1730891147 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35fc003e30 msgr2=0x7f361009dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:25.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.324+0000 7f3614a63700 1 --2- 172.21.15.23:0/1730891147 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35fc003e30 0x7f361009dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:25.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.324+0000 7f3614a63700 1 -- 172.21.15.23:0/1730891147 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36100676d0 con 0x7f3610148f40 2020-11-20T11:52:25.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.324+0000 7f360ffff700 1 --2- 172.21.15.23:0/1730891147 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35fc003e30 0x7f361009dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:52:25.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.324+0000 7f3614a63700 1 --2- 172.21.15.23:0/1730891147 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3610148f40 0x7f36101493c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f35fc005500 tx=0x7f3608015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:25.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.325+0000 7f35effff700 1 -- 172.21.15.23:0/1730891147 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360800ce40 con 0x7f3610148f40 2020-11-20T11:52:25.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.325+0000 7f35effff700 1 -- 172.21.15.23:0/1730891147 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3608001150 con 0x7f3610148f40 2020-11-20T11:52:25.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.325+0000 7f35effff700 1 -- 172.21.15.23:0/1730891147 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3608021670 con 0x7f3610148f40 2020-11-20T11:52:25.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.325+0000 7f36164c6700 1 -- 172.21.15.23:0/1730891147 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36101383f0 con 0x7f3610148f40 2020-11-20T11:52:25.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.325+0000 7f36164c6700 1 -- 172.21.15.23:0/1730891147 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3610066a00 con 0x7f3610148f40 2020-11-20T11:52:25.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.326+0000 7f36164c6700 1 -- 172.21.15.23:0/1730891147 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3610067f50 con 0x7f3610148f40 2020-11-20T11:52:25.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.326+0000 7f35effff700 1 -- 172.21.15.23:0/1730891147 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f3608010030 con 0x7f3610148f40 2020-11-20T11:52:25.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.327+0000 7f35effff700 1 --2- 172.21.15.23:0/1730891147 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f35e8056d60 0x7f35e8059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.327+0000 7f35effff700 1 -- 172.21.15.23:0/1730891147 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f3608019020 con 0x7f3610148f40 2020-11-20T11:52:25.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.327+0000 7f360ffff700 1 --2- 172.21.15.23:0/1730891147 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f35e8056d60 0x7f35e8059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:25.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.327+0000 7f360ffff700 1 --2- 172.21.15.23:0/1730891147 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f35e8056d60 0x7f35e8059220 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f36100aded0 tx=0x7f3600006e30).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:25.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.331+0000 7f35effff700 1 -- 172.21.15.23:0/1730891147 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f36080464a0 con 0x7f3610148f40 2020-11-20T11:52:25.538 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.535+0000 7f36164c6700 1 -- 172.21.15.23:0/1730891147 --> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f361005c520 con 0x7f35e8056d60 2020-11-20T11:52:25.538 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.535+0000 7f35effff700 1 -- 172.21.15.23:0/1730891147 <== mgr.4105 v2:172.21.15.155:6832/22693 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f361005c520 con 0x7f35e8056d60 2020-11-20T11:52:25.540 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.537+0000 7f36164c6700 1 -- 172.21.15.23:0/1730891147 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f35e8056d60 msgr2=0x7f35e8059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:25.541 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.537+0000 7f36164c6700 1 --2- 172.21.15.23:0/1730891147 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f35e8056d60 0x7f35e8059220 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f36100aded0 tx=0x7f3600006e30).stop 2020-11-20T11:52:25.541 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.537+0000 7f36164c6700 1 -- 172.21.15.23:0/1730891147 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3610148f40 msgr2=0x7f36101493c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:25.541 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.537+0000 7f36164c6700 1 --2- 172.21.15.23:0/1730891147 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3610148f40 0x7f36101493c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f35fc005500 tx=0x7f3608015000).stop 2020-11-20T11:52:25.541 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.537+0000 7f36164c6700 1 -- 172.21.15.23:0/1730891147 shutdown_connections 2020-11-20T11:52:25.542 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.537+0000 7f36164c6700 1 --2- 172.21.15.23:0/1730891147 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3610148f40 0x7f36101493c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:25.542 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.537+0000 7f36164c6700 1 --2- 172.21.15.23:0/1730891147 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f35fc003e30 0x7f361009dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:25.542 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.537+0000 7f36164c6700 1 --2- 172.21.15.23:0/1730891147 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f35e8056d60 0x7f35e8059220 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:25.542 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.537+0000 7f36164c6700 1 --2- 172.21.15.23:0/1730891147 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3610067b70 0x7f361009f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:25.543 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.537+0000 7f36164c6700 1 -- 172.21.15.23:0/1730891147 >> 172.21.15.23:0/1730891147 conn(0x7f3610060c80 msgr2=0x7f3610061850 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:25.543 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.537+0000 7f36164c6700 1 -- 172.21.15.23:0/1730891147 shutdown_connections 2020-11-20T11:52:25.543 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.537+0000 7f36164c6700 1 -- 172.21.15.23:0/1730891147 wait complete. 2020-11-20T11:52:25.544 INFO:teuthology.orchestra.run.smithi023.stderr:instructed osd(s) 4 to deep-scrub 2020-11-20T11:52:25.554 INFO:tasks.ceph:Scrubbing osd.5 2020-11-20T11:52:25.554 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2020-11-20T11:52:25.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.710+0000 7f32f6040700 1 Processor -- start 2020-11-20T11:52:25.713 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.710+0000 7f32f6040700 1 -- start start 2020-11-20T11:52:25.716 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.711+0000 7f32f6040700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f32f00aaf60 0x7f32f00ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.717 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.711+0000 7f32f6040700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32f00a8570 0x7f32f00abad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.717 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.711+0000 7f32f6040700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f013aad0 con 0x7f32f00aab80 2020-11-20T11:52:25.717 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.711+0000 7f32f6040700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32f00a9e00 con 0x7f32f00aaf60 2020-11-20T11:52:25.717 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.711+0000 7f32f6040700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32f013ad70 con 0x7f32f00a8570 2020-11-20T11:52:25.718 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.711+0000 7f32f503e700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f32f00aab80 0x7f32f00ac1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:33372/0 (socket says 172.21.15.23:33372) 2020-11-20T11:52:25.721 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.711+0000 7f32f503e700 1 -- 172.21.15.23:0/589959821 learned_addr learned my addr 172.21.15.23:0/589959821 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:52:25.721 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.711+0000 7f32f483d700 1 --2- 172.21.15.23:0/589959821 >> v2:172.21.15.155:3300/0 conn(0x7f32f00aaf60 0x7f32f00ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:25.721 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.711+0000 7f32dffff700 1 -- 172.21.15.23:0/589959821 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2920845254 0 0) 0x7f32f013aad0 con 0x7f32f00aab80 2020-11-20T11:52:25.722 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.711+0000 7f32dffff700 1 -- 172.21.15.23:0/589959821 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32d8003280 con 0x7f32f00aab80 2020-11-20T11:52:25.722 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.711+0000 7f32dffff700 1 -- 172.21.15.23:0/589959821 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 193344336 0 0) 0x7f32d8003280 con 0x7f32f00aab80 2020-11-20T11:52:25.722 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.711+0000 7f32dffff700 1 -- 172.21.15.23:0/589959821 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32f00a8570 msgr2=0x7f32f00abad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:52:25.722 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.711+0000 7f32dffff700 1 --2- 172.21.15.23:0/589959821 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32f00a8570 0x7f32f00abad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:52:25.723 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.711+0000 7f32dffff700 1 -- 172.21.15.23:0/589959821 >> v2:172.21.15.155:3300/0 conn(0x7f32f00aaf60 msgr2=0x7f32f00ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:52:25.723 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.711+0000 7f32dffff700 1 --2- 172.21.15.23:0/589959821 >> v2:172.21.15.155:3300/0 conn(0x7f32f00aaf60 0x7f32f00ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:25.724 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.711+0000 7f32dffff700 1 -- 172.21.15.23:0/589959821 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f013b010 con 0x7f32f00aab80 2020-11-20T11:52:25.724 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.711+0000 7f32dffff700 1 -- 172.21.15.23:0/589959821 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7f32ec001520 con 0x7f32f00aab80 2020-11-20T11:52:25.724 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.711+0000 7f32dffff700 1 -- 172.21.15.23:0/589959821 >> v1:172.21.15.23:6789/0 conn(0x7f32f00aab80 legacy=0x7f32f00ac1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:25.724 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.711+0000 7f32dffff700 1 --2- 172.21.15.23:0/589959821 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32f00a8570 0x7f32d8003770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.725 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.711+0000 7f32dffff700 1 --2- 172.21.15.23:0/589959821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32d8003c80 0x7f32d8004520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.725 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.711+0000 7f32dffff700 1 --2- 172.21.15.23:0/589959821 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32d8004a10 0x7f32d8004e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.726 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.711+0000 7f32dffff700 1 -- 172.21.15.23:0/589959821 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f00a9e00 con 0x7f32d8003c80 2020-11-20T11:52:25.726 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.711+0000 7f32dffff700 1 -- 172.21.15.23:0/589959821 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f013ad70 con 0x7f32d8004a10 2020-11-20T11:52:25.726 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.711+0000 7f32dffff700 1 -- 172.21.15.23:0/589959821 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d8004240 con 0x7f32f00a8570 2020-11-20T11:52:25.727 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.712+0000 7f32f483d700 1 --2- 172.21.15.23:0/589959821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32d8003c80 0x7f32d8004520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:25.727 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.712+0000 7f32f583f700 1 --2- 172.21.15.23:0/589959821 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32d8004a10 0x7f32d8004e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:25.727 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.712+0000 7f32f503e700 1 --2- 172.21.15.23:0/589959821 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32f00a8570 0x7f32d8003770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:25.728 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.712+0000 7f32f483d700 1 -- 172.21.15.23:0/589959821 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32f00a8570 msgr2=0x7f32d8003770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:52:25.728 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.712+0000 7f32f483d700 1 --2- 172.21.15.23:0/589959821 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32f00a8570 0x7f32d8003770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:25.728 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.712+0000 7f32f483d700 1 -- 172.21.15.23:0/589959821 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32d8004a10 msgr2=0x7f32d8004e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:52:25.729 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.712+0000 7f32f483d700 1 --2- 172.21.15.23:0/589959821 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32d8004a10 0x7f32d8004e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:25.729 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.712+0000 7f32f483d700 1 -- 172.21.15.23:0/589959821 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d8003280 con 0x7f32d8003c80 2020-11-20T11:52:25.729 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.712+0000 7f32f583f700 1 --2- 172.21.15.23:0/589959821 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32d8004a10 0x7f32d8004e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:52:25.730 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.712+0000 7f32f503e700 1 --2- 172.21.15.23:0/589959821 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32f00a8570 0x7f32d8003770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2020-11-20T11:52:25.730 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.712+0000 7f32f483d700 1 --2- 172.21.15.23:0/589959821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32d8003c80 0x7f32d8004520 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f32e000bc40 tx=0x7f32e0006e30).ready entity=mon.0 client_cookie=d58563f0fab7a8d server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:25.731 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.712+0000 7f32dffff700 1 -- 172.21.15.23:0/589959821 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e000b350 con 0x7f32d8003c80 2020-11-20T11:52:25.731 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.712+0000 7f32dffff700 1 -- 172.21.15.23:0/589959821 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e000ea50 con 0x7f32d8003c80 2020-11-20T11:52:25.731 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.712+0000 7f32dffff700 1 -- 172.21.15.23:0/589959821 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e000edd0 con 0x7f32d8003c80 2020-11-20T11:52:25.732 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.713+0000 7f32f6040700 1 -- 172.21.15.23:0/589959821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32d8003c80 msgr2=0x7f32d8004520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:25.732 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.713+0000 7f32f6040700 1 --2- 172.21.15.23:0/589959821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32d8003c80 0x7f32d8004520 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f32e000bc40 tx=0x7f32e0006e30).stop 2020-11-20T11:52:25.732 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.713+0000 7f32f583f700 1 -- 172.21.15.23:0/589959821 reap_dead start 2020-11-20T11:52:25.733 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.713+0000 7f32f6040700 1 -- 172.21.15.23:0/589959821 shutdown_connections 2020-11-20T11:52:25.733 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.713+0000 7f32f6040700 1 -- 172.21.15.23:0/589959821 >> 172.21.15.23:0/589959821 conn(0x7f32f00678e0 msgr2=0x7f32f0067cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:25.733 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.713+0000 7f32f6040700 1 -- 172.21.15.23:0/589959821 shutdown_connections 2020-11-20T11:52:25.734 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.713+0000 7f32f6040700 1 -- 172.21.15.23:0/589959821 wait complete. 2020-11-20T11:52:25.737 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.714+0000 7f32f6040700 1 Processor -- start 2020-11-20T11:52:25.738 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.714+0000 7f32f6040700 1 -- start start 2020-11-20T11:52:25.738 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.715+0000 7f32f6040700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32d8003c80 0x7f32f013e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.738 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.715+0000 7f32f6040700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32f00674c0 0x7f32f00651e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.739 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.716+0000 7f32f6040700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32f00a4f40 0x7f32f00a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.739 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.716+0000 7f32f483d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32f00674c0 0x7f32f00651e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:25.739 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.716+0000 7f32f583f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32f00a4f40 0x7f32f00a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:25.740 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.716+0000 7f32f583f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32f00a4f40 0x7f32f00a53c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47366/0 (socket says 172.21.15.23:47366) 2020-11-20T11:52:25.740 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.716+0000 7f32f483d700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32f00674c0 0x7f32f00651e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:57604/0 (socket says 172.21.15.23:57604) 2020-11-20T11:52:25.740 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.716+0000 7f32f583f700 1 -- 172.21.15.23:0/2255619382 learned_addr learned my addr 172.21.15.23:0/2255619382 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:52:25.741 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.716+0000 7f32f6040700 1 -- 172.21.15.23:0/2255619382 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f0067200 con 0x7f32f00a4f40 2020-11-20T11:52:25.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.716+0000 7f32f6040700 1 -- 172.21.15.23:0/2255619382 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f0066f60 con 0x7f32d8003c80 2020-11-20T11:52:25.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.716+0000 7f32f6040700 1 -- 172.21.15.23:0/2255619382 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32f0063af0 con 0x7f32f00674c0 2020-11-20T11:52:25.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.716+0000 7f32f503e700 1 --2- 172.21.15.23:0/2255619382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32d8003c80 0x7f32f013e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:25.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.716+0000 7f32f583f700 1 -- 172.21.15.23:0/2255619382 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32f00674c0 msgr2=0x7f32f00651e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:25.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.716+0000 7f32f583f700 1 --2- 172.21.15.23:0/2255619382 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32f00674c0 0x7f32f00651e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:25.743 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.716+0000 7f32f583f700 1 -- 172.21.15.23:0/2255619382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32d8003c80 msgr2=0x7f32f013e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:25.744 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.716+0000 7f32f583f700 1 --2- 172.21.15.23:0/2255619382 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f32d8003c80 0x7f32f013e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:25.744 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.716+0000 7f32f583f700 1 -- 172.21.15.23:0/2255619382 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f00a6c00 con 0x7f32f00a4f40 2020-11-20T11:52:25.744 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.716+0000 7f32f483d700 1 --2- 172.21.15.23:0/2255619382 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f32f00674c0 0x7f32f00651e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:52:25.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.717+0000 7f32f583f700 1 --2- 172.21.15.23:0/2255619382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32f00a4f40 0x7f32f00a53c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f32f00acdc0 tx=0x7f32e80013e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:25.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.718+0000 7f32dd7fa700 1 -- 172.21.15.23:0/2255619382 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e800de90 con 0x7f32f00a4f40 2020-11-20T11:52:25.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.718+0000 7f32dd7fa700 1 -- 172.21.15.23:0/2255619382 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e800b7b0 con 0x7f32f00a4f40 2020-11-20T11:52:25.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.718+0000 7f32f6040700 1 -- 172.21.15.23:0/2255619382 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f00a7d20 con 0x7f32f00a4f40 2020-11-20T11:52:25.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.718+0000 7f32dd7fa700 1 -- 172.21.15.23:0/2255619382 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e800bb70 con 0x7f32f00a4f40 2020-11-20T11:52:25.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.718+0000 7f32f6040700 1 -- 172.21.15.23:0/2255619382 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f00a6200 con 0x7f32f00a4f40 2020-11-20T11:52:25.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.719+0000 7f32f6040700 1 -- 172.21.15.23:0/2255619382 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f32f00b1b80 con 0x7f32f00a4f40 2020-11-20T11:52:25.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.720+0000 7f32dd7fa700 1 -- 172.21.15.23:0/2255619382 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f32e8012420 con 0x7f32f00a4f40 2020-11-20T11:52:25.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.720+0000 7f32dd7fa700 1 --2- 172.21.15.23:0/2255619382 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f32c8056db0 0x7f32c8059270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.747 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.720+0000 7f32dd7fa700 1 -- 172.21.15.23:0/2255619382 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f32e806fc00 con 0x7f32f00a4f40 2020-11-20T11:52:25.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.720+0000 7f32f503e700 1 --2- 172.21.15.23:0/2255619382 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f32c8056db0 0x7f32c8059270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:25.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.720+0000 7f32dd7fa700 1 --2- 172.21.15.23:0/2255619382 >> [v2:172.21.15.155:6816/23298,v1:172.21.15.155:6817/23298] conn(0x7f32c805ea10 0x7f32c8060e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.720+0000 7f32dd7fa700 1 -- 172.21.15.23:0/2255619382 --> [v2:172.21.15.155:6816/23298,v1:172.21.15.155:6817/23298] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f32c8061490 con 0x7f32c805ea10 2020-11-20T11:52:25.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.720+0000 7f32dd7fa700 1 -- 172.21.15.23:0/2255619382 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f32e8072270 con 0x7f32f00a4f40 2020-11-20T11:52:25.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.721+0000 7f32f483d700 1 --2- 172.21.15.23:0/2255619382 >> [v2:172.21.15.155:6816/23298,v1:172.21.15.155:6817/23298] conn(0x7f32c805ea10 0x7f32c8060e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:25.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.721+0000 7f32f503e700 1 --2- 172.21.15.23:0/2255619382 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f32c8056db0 0x7f32c8059270 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f32ec0013f0 tx=0x7f32ec009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:25.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.721+0000 7f32f483d700 1 --2- 172.21.15.23:0/2255619382 >> [v2:172.21.15.155:6816/23298,v1:172.21.15.155:6817/23298] conn(0x7f32c805ea10 0x7f32c8060e30 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:25.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.722+0000 7f32dd7fa700 1 -- 172.21.15.23:0/2255619382 <== osd.5 v2:172.21.15.155:6816/23298 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7f32c8061490 con 0x7f32c805ea10 2020-11-20T11:52:25.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.739+0000 7f32f6040700 1 -- 172.21.15.23:0/2255619382 --> [v2:172.21.15.155:6816/23298,v1:172.21.15.155:6817/23298] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f32f0063cb0 con 0x7f32c805ea10 2020-11-20T11:52:25.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.740+0000 7f32dd7fa700 1 -- 172.21.15.23:0/2255619382 <== osd.5 v2:172.21.15.155:6816/23298 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f32f0063cb0 con 0x7f32c805ea10 2020-11-20T11:52:25.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.740+0000 7f32f6040700 1 -- 172.21.15.23:0/2255619382 >> [v2:172.21.15.155:6816/23298,v1:172.21.15.155:6817/23298] conn(0x7f32c805ea10 msgr2=0x7f32c8060e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:25.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.740+0000 7f32f6040700 1 --2- 172.21.15.23:0/2255619382 >> [v2:172.21.15.155:6816/23298,v1:172.21.15.155:6817/23298] conn(0x7f32c805ea10 0x7f32c8060e30 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:25.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.740+0000 7f32f6040700 1 -- 172.21.15.23:0/2255619382 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f32c8056db0 msgr2=0x7f32c8059270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:25.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.740+0000 7f32f6040700 1 --2- 172.21.15.23:0/2255619382 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f32c8056db0 0x7f32c8059270 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f32ec0013f0 tx=0x7f32ec009040).stop 2020-11-20T11:52:25.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.740+0000 7f32f6040700 1 -- 172.21.15.23:0/2255619382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32f00a4f40 msgr2=0x7f32f00a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:25.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.740+0000 7f32f6040700 1 --2- 172.21.15.23:0/2255619382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32f00a4f40 0x7f32f00a53c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f32f00acdc0 tx=0x7f32e80013e0).stop 2020-11-20T11:52:25.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.740+0000 7f32f583f700 1 -- 172.21.15.23:0/2255619382 reap_dead start 2020-11-20T11:52:25.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.740+0000 7f32f6040700 1 -- 172.21.15.23:0/2255619382 shutdown_connections 2020-11-20T11:52:25.756 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.740+0000 7f32f6040700 1 -- 172.21.15.23:0/2255619382 >> 172.21.15.23:0/2255619382 conn(0x7f32f00678e0 msgr2=0x7f32f00aef40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:25.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.740+0000 7f32f6040700 1 -- 172.21.15.23:0/2255619382 shutdown_connections 2020-11-20T11:52:25.757 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.740+0000 7f32f6040700 1 -- 172.21.15.23:0/2255619382 wait complete. 2020-11-20T11:52:25.757 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:52:25.758 INFO:teuthology.orchestra.run.smithi023.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2020-11-20T11:52:25.758 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:52:25.759 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2020-11-20T11:52:25.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.918+0000 7fc96e1bb700 1 Processor -- start 2020-11-20T11:52:25.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.918+0000 7fc96e1bb700 1 -- start start 2020-11-20T11:52:25.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.919+0000 7fc96e1bb700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fc96813aeb0 0x7fc9680a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.919+0000 7fc96e1bb700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9681388b0 con 0x7fc9680aaab0 2020-11-20T11:52:25.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.919+0000 7fc96e1bb700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc96813e600 con 0x7fc9680aae90 2020-11-20T11:52:25.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.919+0000 7fc96e1bb700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc96813e8a0 con 0x7fc96813aeb0 2020-11-20T11:52:25.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.919+0000 7fc966ffd700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fc9680aae90 0x7fc9681381a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:39598/0 (socket says 172.21.15.23:39598) 2020-11-20T11:52:25.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.919+0000 7fc966ffd700 1 -- 172.21.15.23:0/1225902891 learned_addr learned my addr 172.21.15.23:0/1225902891 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:52:25.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.920+0000 7fc9667fc700 1 -- 172.21.15.23:0/1225902891 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2501460386 0 0) 0x7fc96813e600 con 0x7fc9680aae90 2020-11-20T11:52:25.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.920+0000 7fc9667fc700 1 -- 172.21.15.23:0/1225902891 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc954003280 con 0x7fc9680aae90 2020-11-20T11:52:25.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.920+0000 7fc9667fc700 1 -- 172.21.15.23:0/1225902891 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3917172699 0 0) 0x7fc9681388b0 con 0x7fc9680aaab0 2020-11-20T11:52:25.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.920+0000 7fc9667fc700 1 -- 172.21.15.23:0/1225902891 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc96813e600 con 0x7fc9680aaab0 2020-11-20T11:52:25.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.920+0000 7fc9667fc700 1 -- 172.21.15.23:0/1225902891 <== mon.1 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2842019046 0 0) 0x7fc954003280 con 0x7fc9680aae90 2020-11-20T11:52:25.928 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.920+0000 7fc9667fc700 1 -- 172.21.15.23:0/1225902891 >> v2:172.21.15.155:3300/0 conn(0x7fc96813aeb0 msgr2=0x7fc9680a8c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:52:25.928 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.920+0000 7fc9667fc700 1 --2- 172.21.15.23:0/1225902891 >> v2:172.21.15.155:3300/0 conn(0x7fc96813aeb0 0x7fc9680a8c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:52:25.928 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.920+0000 7fc9667fc700 1 -- 172.21.15.23:0/1225902891 >> v1:172.21.15.23:6789/0 conn(0x7fc9680aaab0 legacy=0x7fc96813ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:25.928 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.920+0000 7fc9667fc700 1 -- 172.21.15.23:0/1225902891 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9680ab6d0 con 0x7fc9680aae90 2020-11-20T11:52:25.928 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.920+0000 7fc9667fc700 1 -- 172.21.15.23:0/1225902891 <== mon.1 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7fc95c001520 con 0x7fc9680aae90 2020-11-20T11:52:25.929 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.920+0000 7fc9667fc700 1 -- 172.21.15.23:0/1225902891 >> v1:172.21.15.155:6789/0 conn(0x7fc9680aae90 legacy=0x7fc9681381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:25.929 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.920+0000 7fc9667fc700 1 --2- 172.21.15.23:0/1225902891 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc954004960 0x7fc954004d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.929 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.920+0000 7fc9667fc700 1 --2- 172.21.15.23:0/1225902891 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9540034c0 0x7fc954003940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.929 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.920+0000 7fc9667fc700 1 --2- 172.21.15.23:0/1225902891 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc954003e30 0x7fc95400b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.929 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.920+0000 7fc9667fc700 1 -- 172.21.15.23:0/1225902891 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96813e8a0 con 0x7fc9540034c0 2020-11-20T11:52:25.930 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.920+0000 7fc9667fc700 1 -- 172.21.15.23:0/1225902891 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95400bc60 con 0x7fc954003e30 2020-11-20T11:52:25.930 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.920+0000 7fc9667fc700 1 -- 172.21.15.23:0/1225902891 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc95400bda0 con 0x7fc954004960 2020-11-20T11:52:25.930 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.920+0000 7fc966ffd700 1 --2- 172.21.15.23:0/1225902891 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9540034c0 0x7fc954003940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:25.930 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.920+0000 7fc9677fe700 1 --2- 172.21.15.23:0/1225902891 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc954004960 0x7fc954004d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:25.930 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.921+0000 7fc966ffd700 1 -- 172.21.15.23:0/1225902891 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc954004960 msgr2=0x7fc954004d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:52:25.931 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.921+0000 7fc966ffd700 1 --2- 172.21.15.23:0/1225902891 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc954004960 0x7fc954004d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:25.931 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.921+0000 7fc966ffd700 1 -- 172.21.15.23:0/1225902891 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc954003e30 msgr2=0x7fc95400b770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:52:25.931 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.921+0000 7fc966ffd700 1 --2- 172.21.15.23:0/1225902891 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc954003e30 0x7fc95400b770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:52:25.931 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.921+0000 7fc966ffd700 1 -- 172.21.15.23:0/1225902891 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc954003280 con 0x7fc9540034c0 2020-11-20T11:52:25.931 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.921+0000 7fc9677fe700 1 --2- 172.21.15.23:0/1225902891 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc954004960 0x7fc954004d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:52:25.932 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.921+0000 7fc966ffd700 1 --2- 172.21.15.23:0/1225902891 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9540034c0 0x7fc954003940 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fc95c00c470 tx=0x7fc95c0077c0).ready entity=mon.0 client_cookie=2e90d9779527b61b server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:25.932 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.921+0000 7fc9667fc700 1 -- 172.21.15.23:0/1225902891 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95c00ce10 con 0x7fc9540034c0 2020-11-20T11:52:25.932 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.921+0000 7fc9667fc700 1 -- 172.21.15.23:0/1225902891 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc95c0072e0 con 0x7fc9540034c0 2020-11-20T11:52:25.932 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.922+0000 7fc967fff700 1 -- 172.21.15.23:0/1225902891 reap_dead start 2020-11-20T11:52:25.932 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.922+0000 7fc96e1bb700 1 -- 172.21.15.23:0/1225902891 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9540034c0 msgr2=0x7fc954003940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:25.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.922+0000 7fc96e1bb700 1 --2- 172.21.15.23:0/1225902891 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9540034c0 0x7fc954003940 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fc95c00c470 tx=0x7fc95c0077c0).stop 2020-11-20T11:52:25.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.925+0000 7fc96e1bb700 1 -- 172.21.15.23:0/1225902891 shutdown_connections 2020-11-20T11:52:25.933 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.925+0000 7fc96e1bb700 1 --2- 172.21.15.23:0/1225902891 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9540034c0 0x7fc954003940 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:25.934 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.925+0000 7fc96e1bb700 1 -- 172.21.15.23:0/1225902891 >> 172.21.15.23:0/1225902891 conn(0x7fc9680678e0 msgr2=0x7fc968067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:25.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.932+0000 7fc96e1bb700 1 -- 172.21.15.23:0/1225902891 shutdown_connections 2020-11-20T11:52:25.937 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.932+0000 7fc96e1bb700 1 -- 172.21.15.23:0/1225902891 wait complete. 2020-11-20T11:52:25.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.933+0000 7fc96e1bb700 1 Processor -- start 2020-11-20T11:52:25.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.934+0000 7fc96e1bb700 1 -- start start 2020-11-20T11:52:25.938 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.935+0000 7fc96e1bb700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9540034c0 0x7fc9680a5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.935+0000 7fc96e1bb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc96809eca0 0x7fc9680abf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.935+0000 7fc96e1bb700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc9680a46e0 0x7fc9680a7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.935+0000 7fc96e1bb700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9680a6e80 con 0x7fc96809eca0 2020-11-20T11:52:25.939 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.935+0000 7fc96e1bb700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9680a4df0 con 0x7fc9540034c0 2020-11-20T11:52:25.940 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.935+0000 7fc96e1bb700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9680a70b0 con 0x7fc9680a46e0 2020-11-20T11:52:25.942 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.938+0000 7fc967fff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc9680a46e0 0x7fc9680a7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:25.942 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.938+0000 7fc967fff700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc9680a46e0 0x7fc9680a7550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:57622/0 (socket says 172.21.15.23:57622) 2020-11-20T11:52:25.943 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.938+0000 7fc967fff700 1 -- 172.21.15.23:0/2958855548 learned_addr learned my addr 172.21.15.23:0/2958855548 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:52:25.943 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.938+0000 7fc9677fe700 1 --2- 172.21.15.23:0/2958855548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9540034c0 0x7fc9680a5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:25.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.938+0000 7fc9677fe700 1 -- 172.21.15.23:0/2958855548 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc9680a46e0 msgr2=0x7fc9680a7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:25.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.938+0000 7fc9677fe700 1 --2- 172.21.15.23:0/2958855548 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc9680a46e0 0x7fc9680a7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:25.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.938+0000 7fc9677fe700 1 -- 172.21.15.23:0/2958855548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc96809eca0 msgr2=0x7fc9680abf00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-11-20T11:52:25.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.938+0000 7fc9677fe700 1 --2- 172.21.15.23:0/2958855548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc96809eca0 0x7fc9680abf00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:52:25.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.938+0000 7fc9677fe700 1 -- 172.21.15.23:0/2958855548 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9680a4b40 con 0x7fc9540034c0 2020-11-20T11:52:25.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.939+0000 7fc967fff700 1 --2- 172.21.15.23:0/2958855548 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc9680a46e0 0x7fc9680a7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:52:25.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.939+0000 7fc9677fe700 1 --2- 172.21.15.23:0/2958855548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9540034c0 0x7fc9680a5cf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fc958001690 tx=0x7fc958009040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:25.950 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.947+0000 7fc94f7fe700 1 -- 172.21.15.23:0/2958855548 <== mon.1 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc958007240 con 0x7fc9540034c0 2020-11-20T11:52:25.950 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.947+0000 7fc94f7fe700 1 -- 172.21.15.23:0/2958855548 <== mon.1 v2:172.21.15.155:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9580073a0 con 0x7fc9540034c0 2020-11-20T11:52:25.951 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.947+0000 7fc96e1bb700 1 -- 172.21.15.23:0/2958855548 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9680a7a40 con 0x7fc9540034c0 2020-11-20T11:52:25.951 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.947+0000 7fc94f7fe700 1 -- 172.21.15.23:0/2958855548 <== mon.1 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95800c8b0 con 0x7fc9540034c0 2020-11-20T11:52:25.951 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.947+0000 7fc96e1bb700 1 -- 172.21.15.23:0/2958855548 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9681336f0 con 0x7fc9540034c0 2020-11-20T11:52:25.953 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.948+0000 7fc96e1bb700 1 -- 172.21.15.23:0/2958855548 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9680a4f60 con 0x7fc9540034c0 2020-11-20T11:52:25.953 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.948+0000 7fc94f7fe700 1 -- 172.21.15.23:0/2958855548 <== mon.1 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fc95800cc60 con 0x7fc9540034c0 2020-11-20T11:52:25.953 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.949+0000 7fc94f7fe700 1 --2- 172.21.15.23:0/2958855548 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc938056db0 0x7fc938059270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:25.954 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.949+0000 7fc94f7fe700 1 -- 172.21.15.23:0/2958855548 <== mon.1 v2:172.21.15.155:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fc958070680 con 0x7fc9540034c0 2020-11-20T11:52:25.954 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.949+0000 7fc966ffd700 1 --2- 172.21.15.23:0/2958855548 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc938056db0 0x7fc938059270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:25.954 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.949+0000 7fc966ffd700 1 --2- 172.21.15.23:0/2958855548 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc938056db0 0x7fc938059270 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fc96809f0e0 tx=0x7fc95c015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:25.956 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:25.953+0000 7fc94f7fe700 1 -- 172.21.15.23:0/2958855548 <== mon.1 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fc9680a4f60 con 0x7fc9540034c0 2020-11-20T11:52:26.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.143+0000 7fc96e1bb700 1 -- 172.21.15.23:0/2958855548 --> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fc96805c7d0 con 0x7fc938056db0 2020-11-20T11:52:26.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.144+0000 7fc94f7fe700 1 -- 172.21.15.23:0/2958855548 <== mgr.4105 v2:172.21.15.155:6832/22693 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc96805c7d0 con 0x7fc938056db0 2020-11-20T11:52:26.149 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.146+0000 7fc96e1bb700 1 -- 172.21.15.23:0/2958855548 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc938056db0 msgr2=0x7fc938059270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:26.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.146+0000 7fc96e1bb700 1 --2- 172.21.15.23:0/2958855548 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc938056db0 0x7fc938059270 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fc96809f0e0 tx=0x7fc95c015000).stop 2020-11-20T11:52:26.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.146+0000 7fc96e1bb700 1 -- 172.21.15.23:0/2958855548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9540034c0 msgr2=0x7fc9680a5cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:26.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.146+0000 7fc96e1bb700 1 --2- 172.21.15.23:0/2958855548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9540034c0 0x7fc9680a5cf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fc958001690 tx=0x7fc958009040).stop 2020-11-20T11:52:26.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.146+0000 7fc96e1bb700 1 -- 172.21.15.23:0/2958855548 shutdown_connections 2020-11-20T11:52:26.150 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.146+0000 7fc96e1bb700 1 --2- 172.21.15.23:0/2958855548 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fc938056db0 0x7fc938059270 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:26.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.146+0000 7fc96e1bb700 1 --2- 172.21.15.23:0/2958855548 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fc9680a46e0 0x7fc9680a7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:26.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.146+0000 7fc96e1bb700 1 --2- 172.21.15.23:0/2958855548 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fc9540034c0 0x7fc9680a5cf0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:26.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.146+0000 7fc96e1bb700 1 --2- 172.21.15.23:0/2958855548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc96809eca0 0x7fc9680abf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:52:26.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.146+0000 7fc96e1bb700 1 -- 172.21.15.23:0/2958855548 >> 172.21.15.23:0/2958855548 conn(0x7fc9680678e0 msgr2=0x7fc968068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:26.151 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.146+0000 7fc96e1bb700 1 -- 172.21.15.23:0/2958855548 shutdown_connections 2020-11-20T11:52:26.152 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.146+0000 7fc96e1bb700 1 -- 172.21.15.23:0/2958855548 wait complete. 2020-11-20T11:52:26.152 INFO:teuthology.orchestra.run.smithi023.stderr:instructed osd(s) 5 to deep-scrub 2020-11-20T11:52:26.163 INFO:tasks.ceph:Scrubbing osd.6 2020-11-20T11:52:26.163 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2020-11-20T11:52:26.324 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.321+0000 7f5f7cff0700 1 Processor -- start 2020-11-20T11:52:26.325 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.321+0000 7f5f7cff0700 1 -- start start 2020-11-20T11:52:26.327 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.322+0000 7f5f7cff0700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5f78067580 0x7f5f780679e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:26.328 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.322+0000 7f5f7cff0700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5f78137a00 0x7f5f78137e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:26.328 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.322+0000 7f5f7cff0700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f780a2000 con 0x7f5f780af190 2020-11-20T11:52:26.328 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.322+0000 7f5f7cff0700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f78067ed0 con 0x7f5f78067580 2020-11-20T11:52:26.328 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.322+0000 7f5f7cff0700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f780aaf00 con 0x7f5f78137a00 2020-11-20T11:52:26.329 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.322+0000 7f5f75d9b700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f5f78067580 0x7f5f780679e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:26.329 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.322+0000 7f5f7659c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f5f780af190 0x7f5f780aa7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:33406/0 (socket says 172.21.15.23:33406) 2020-11-20T11:52:26.329 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.322+0000 7f5f7659c700 1 -- 172.21.15.23:0/3063718583 learned_addr learned my addr 172.21.15.23:0/3063718583 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:52:26.329 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.322+0000 7f5f76d9d700 1 --2- 172.21.15.23:0/3063718583 >> v2:172.21.15.155:3300/0 conn(0x7f5f78137a00 0x7f5f78137e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:26.329 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.323+0000 7f5f7559a700 1 -- 172.21.15.23:0/3063718583 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079587115 0 0) 0x7f5f780a2000 con 0x7f5f780af190 2020-11-20T11:52:26.330 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.323+0000 7f5f7559a700 1 -- 172.21.15.23:0/3063718583 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f5c003280 con 0x7f5f780af190 2020-11-20T11:52:26.330 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.323+0000 7f5f75d9b700 1 -- 172.21.15.23:0/3063718583 >> v2:172.21.15.155:3300/0 conn(0x7f5f78137a00 msgr2=0x7f5f78137e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:52:26.330 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.323+0000 7f5f75d9b700 1 --2- 172.21.15.23:0/3063718583 >> v2:172.21.15.155:3300/0 conn(0x7f5f78137a00 0x7f5f78137e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:26.330 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.323+0000 7f5f75d9b700 1 -- 172.21.15.23:0/3063718583 >> v1:172.21.15.23:6789/0 conn(0x7f5f780af190 legacy=0x7f5f780aa7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:26.331 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.323+0000 7f5f75d9b700 1 -- 172.21.15.23:0/3063718583 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f780aa3c0 con 0x7f5f78067580 2020-11-20T11:52:26.331 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.323+0000 7f5f75d9b700 1 --2- 172.21.15.23:0/3063718583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f78067580 0x7f5f780679e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f5f780ab3b0 tx=0x7f5f6c006e30).ready entity=mon.0 client_cookie=a3ebb37c7efd4cd2 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:26.331 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.323+0000 7f5f7559a700 1 -- 172.21.15.23:0/3063718583 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c0095c0 con 0x7f5f78067580 2020-11-20T11:52:26.331 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.323+0000 7f5f7559a700 1 -- 172.21.15.23:0/3063718583 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f6c000e90 con 0x7f5f78067580 2020-11-20T11:52:26.332 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.323+0000 7f5f7559a700 1 -- 172.21.15.23:0/3063718583 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c009930 con 0x7f5f78067580 2020-11-20T11:52:26.332 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.324+0000 7f5f7cff0700 1 -- 172.21.15.23:0/3063718583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f78067580 msgr2=0x7f5f780679e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:26.332 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.324+0000 7f5f7cff0700 1 --2- 172.21.15.23:0/3063718583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f78067580 0x7f5f780679e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f5f780ab3b0 tx=0x7f5f6c006e30).stop 2020-11-20T11:52:26.332 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.324+0000 7f5f7cff0700 1 -- 172.21.15.23:0/3063718583 shutdown_connections 2020-11-20T11:52:26.333 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.324+0000 7f5f7cff0700 1 --2- 172.21.15.23:0/3063718583 >> v2:172.21.15.155:3300/0 conn(0x7f5f78137a00 0x7f5f78137e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:26.333 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.324+0000 7f5f7cff0700 1 --2- 172.21.15.23:0/3063718583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f78067580 0x7f5f780679e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:26.333 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.324+0000 7f5f7cff0700 1 -- 172.21.15.23:0/3063718583 >> 172.21.15.23:0/3063718583 conn(0x7f5f780666e0 msgr2=0x7f5f78063d70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:26.334 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.324+0000 7f5f7cff0700 1 -- 172.21.15.23:0/3063718583 shutdown_connections 2020-11-20T11:52:26.334 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.324+0000 7f5f7cff0700 1 -- 172.21.15.23:0/3063718583 wait complete. 2020-11-20T11:52:26.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.325+0000 7f5f7cff0700 1 Processor -- start 2020-11-20T11:52:26.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.325+0000 7f5f7cff0700 1 -- start start 2020-11-20T11:52:26.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.326+0000 7f5f7cff0700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5f78067580 0x7f5f781317b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:26.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.326+0000 7f5f7cff0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f780af190 0x7f5f78139370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:26.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.326+0000 7f5f7cff0700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5f78137a00 0x7f5f78062920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:26.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.326+0000 7f5f7cff0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7813ae50 con 0x7f5f780af190 2020-11-20T11:52:26.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.326+0000 7f5f7cff0700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f78138900 con 0x7f5f78067580 2020-11-20T11:52:26.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.326+0000 7f5f7cff0700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f78139860 con 0x7f5f78137a00 2020-11-20T11:52:26.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.326+0000 7f5f75d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f780af190 0x7f5f78139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:26.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.326+0000 7f5f75d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f780af190 0x7f5f78139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47394/0 (socket says 172.21.15.23:47394) 2020-11-20T11:52:26.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.326+0000 7f5f75d9b700 1 -- 172.21.15.23:0/3197589662 learned_addr learned my addr 172.21.15.23:0/3197589662 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:52:26.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.326+0000 7f5f76d9d700 1 --2- 172.21.15.23:0/3197589662 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5f78137a00 0x7f5f78062920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:26.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.326+0000 7f5f7659c700 1 --2- 172.21.15.23:0/3197589662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5f78067580 0x7f5f781317b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:26.340 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.326+0000 7f5f75d9b700 1 -- 172.21.15.23:0/3197589662 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5f78137a00 msgr2=0x7f5f78062920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:26.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.326+0000 7f5f75d9b700 1 --2- 172.21.15.23:0/3197589662 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5f78137a00 0x7f5f78062920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:26.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.326+0000 7f5f75d9b700 1 -- 172.21.15.23:0/3197589662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5f78067580 msgr2=0x7f5f781317b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:26.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.326+0000 7f5f75d9b700 1 --2- 172.21.15.23:0/3197589662 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5f78067580 0x7f5f781317b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:26.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.326+0000 7f5f75d9b700 1 -- 172.21.15.23:0/3197589662 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f7813b040 con 0x7f5f780af190 2020-11-20T11:52:26.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.326+0000 7f5f75d9b700 1 --2- 172.21.15.23:0/3197589662 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f780af190 0x7f5f78139370 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f5f6c006fd0 tx=0x7f5f6c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:26.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.327+0000 7f5f667fc700 1 -- 172.21.15.23:0/3197589662 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c011ac0 con 0x7f5f780af190 2020-11-20T11:52:26.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.327+0000 7f5f667fc700 1 -- 172.21.15.23:0/3197589662 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f6c011c20 con 0x7f5f780af190 2020-11-20T11:52:26.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.327+0000 7f5f7cff0700 1 -- 172.21.15.23:0/3197589662 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f7813b2e0 con 0x7f5f780af190 2020-11-20T11:52:26.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.327+0000 7f5f667fc700 1 -- 172.21.15.23:0/3197589662 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c01e460 con 0x7f5f780af190 2020-11-20T11:52:26.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.327+0000 7f5f7cff0700 1 -- 172.21.15.23:0/3197589662 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f78062e10 con 0x7f5f780af190 2020-11-20T11:52:26.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.328+0000 7f5f7cff0700 1 -- 172.21.15.23:0/3197589662 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5f78066af0 con 0x7f5f780af190 2020-11-20T11:52:26.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.328+0000 7f5f667fc700 1 -- 172.21.15.23:0/3197589662 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f5f6c025030 con 0x7f5f780af190 2020-11-20T11:52:26.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.328+0000 7f5f667fc700 1 --2- 172.21.15.23:0/3197589662 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5f4c056d60 0x7f5f4c059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:26.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.328+0000 7f5f667fc700 1 -- 172.21.15.23:0/3197589662 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f5f6c0715b0 con 0x7f5f780af190 2020-11-20T11:52:26.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.328+0000 7f5f667fc700 1 --2- 172.21.15.23:0/3197589662 >> [v2:172.21.15.155:6824/23307,v1:172.21.15.155:6825/23307] conn(0x7f5f4c05e940 0x7f5f4c060d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:26.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.328+0000 7f5f667fc700 1 -- 172.21.15.23:0/3197589662 --> [v2:172.21.15.155:6824/23307,v1:172.21.15.155:6825/23307] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5f4c0613c0 con 0x7f5f4c05e940 2020-11-20T11:52:26.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.328+0000 7f5f667fc700 1 -- 172.21.15.23:0/3197589662 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5f6c073c20 con 0x7f5f780af190 2020-11-20T11:52:26.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.328+0000 7f5f7659c700 1 --2- 172.21.15.23:0/3197589662 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5f4c056d60 0x7f5f4c059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:26.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.329+0000 7f5f7659c700 1 --2- 172.21.15.23:0/3197589662 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5f4c056d60 0x7f5f4c059220 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f5f680055e0 tx=0x7f5f68009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:26.345 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.329+0000 7f5f76d9d700 1 --2- 172.21.15.23:0/3197589662 >> [v2:172.21.15.155:6824/23307,v1:172.21.15.155:6825/23307] conn(0x7f5f4c05e940 0x7f5f4c060d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:26.345 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.330+0000 7f5f76d9d700 1 --2- 172.21.15.23:0/3197589662 >> [v2:172.21.15.155:6824/23307,v1:172.21.15.155:6825/23307] conn(0x7f5f4c05e940 0x7f5f4c060d60 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:26.345 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.333+0000 7f5f667fc700 1 -- 172.21.15.23:0/3197589662 <== osd.6 v2:172.21.15.155:6824/23307 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7f5f4c0613c0 con 0x7f5f4c05e940 2020-11-20T11:52:26.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.354+0000 7f5f7cff0700 1 -- 172.21.15.23:0/3197589662 --> [v2:172.21.15.155:6824/23307,v1:172.21.15.155:6825/23307] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f5f7813b470 con 0x7f5f4c05e940 2020-11-20T11:52:26.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.356+0000 7f5f667fc700 1 -- 172.21.15.23:0/3197589662 <== osd.6 v2:172.21.15.155:6824/23307 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f5f7813b470 con 0x7f5f4c05e940 2020-11-20T11:52:26.359 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:52:26.359 INFO:teuthology.orchestra.run.smithi023.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2020-11-20T11:52:26.360 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:52:26.361 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.356+0000 7f5f7cff0700 1 -- 172.21.15.23:0/3197589662 >> [v2:172.21.15.155:6824/23307,v1:172.21.15.155:6825/23307] conn(0x7f5f4c05e940 msgr2=0x7f5f4c060d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:26.361 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.356+0000 7f5f7cff0700 1 --2- 172.21.15.23:0/3197589662 >> [v2:172.21.15.155:6824/23307,v1:172.21.15.155:6825/23307] conn(0x7f5f4c05e940 0x7f5f4c060d60 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:26.361 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.357+0000 7f5f7cff0700 1 -- 172.21.15.23:0/3197589662 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5f4c056d60 msgr2=0x7f5f4c059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:26.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.357+0000 7f5f7cff0700 1 --2- 172.21.15.23:0/3197589662 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f5f4c056d60 0x7f5f4c059220 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f5f680055e0 tx=0x7f5f68009040).stop 2020-11-20T11:52:26.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.357+0000 7f5f7cff0700 1 -- 172.21.15.23:0/3197589662 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f780af190 msgr2=0x7f5f78139370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:26.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.357+0000 7f5f7cff0700 1 --2- 172.21.15.23:0/3197589662 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f780af190 0x7f5f78139370 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f5f6c006fd0 tx=0x7f5f6c015000).stop 2020-11-20T11:52:26.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.357+0000 7f5f76d9d700 1 -- 172.21.15.23:0/3197589662 reap_dead start 2020-11-20T11:52:26.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.358+0000 7f5f7cff0700 1 -- 172.21.15.23:0/3197589662 shutdown_connections 2020-11-20T11:52:26.363 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.358+0000 7f5f7cff0700 1 -- 172.21.15.23:0/3197589662 >> 172.21.15.23:0/3197589662 conn(0x7f5f780666e0 msgr2=0x7f5f78137e90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:26.363 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.358+0000 7f5f7cff0700 1 -- 172.21.15.23:0/3197589662 shutdown_connections 2020-11-20T11:52:26.363 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.358+0000 7f5f7cff0700 1 -- 172.21.15.23:0/3197589662 wait complete. 2020-11-20T11:52:26.372 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2020-11-20T11:52:26.515 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.512+0000 7fd574b24700 1 Processor -- start 2020-11-20T11:52:26.516 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.512+0000 7fd574b24700 1 -- start start 2020-11-20T11:52:26.518 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.513+0000 7fd574b24700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5700678b0 0x7fd570067d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:26.518 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.513+0000 7fd574b24700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5700a9b40 con 0x7fd570060a80 2020-11-20T11:52:26.518 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.513+0000 7fd574b24700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5700aacb0 con 0x7fd5700abbd0 2020-11-20T11:52:26.518 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.513+0000 7fd574b24700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd570068220 con 0x7fd5700678b0 2020-11-20T11:52:26.518 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.513+0000 7fd56e59c700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7fd570060a80 0x7fd57009d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:33422/0 (socket says 172.21.15.23:33422) 2020-11-20T11:52:26.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.513+0000 7fd56e59c700 1 -- 172.21.15.23:0/3625647507 learned_addr learned my addr 172.21.15.23:0/3625647507 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:52:26.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.513+0000 7fd56dd9b700 1 --2- 172.21.15.23:0/3625647507 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5700678b0 0x7fd570067d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:26.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.513+0000 7fd56d59a700 1 -- 172.21.15.23:0/3625647507 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3357983060 0 0) 0x7fd5700a9b40 con 0x7fd570060a80 2020-11-20T11:52:26.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.513+0000 7fd56d59a700 1 -- 172.21.15.23:0/3625647507 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd554003280 con 0x7fd570060a80 2020-11-20T11:52:26.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.513+0000 7fd56d59a700 1 -- 172.21.15.23:0/3625647507 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1591076662 0 0) 0x7fd5700aacb0 con 0x7fd5700abbd0 2020-11-20T11:52:26.520 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.514+0000 7fd56d59a700 1 -- 172.21.15.23:0/3625647507 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5700a9b40 con 0x7fd5700abbd0 2020-11-20T11:52:26.520 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.514+0000 7fd56dd9b700 1 -- 172.21.15.23:0/3625647507 >> v1:172.21.15.155:6789/0 conn(0x7fd5700abbd0 legacy=0x7fd5700ac060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:26.520 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.514+0000 7fd56dd9b700 1 -- 172.21.15.23:0/3625647507 >> v1:172.21.15.23:6789/0 conn(0x7fd570060a80 legacy=0x7fd57009d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:26.520 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.514+0000 7fd56dd9b700 1 -- 172.21.15.23:0/3625647507 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5700a9da0 con 0x7fd5700678b0 2020-11-20T11:52:26.520 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.514+0000 7fd56dd9b700 1 --2- 172.21.15.23:0/3625647507 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5700678b0 0x7fd570067d30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fd56400ba00 tx=0x7fd564006e30).ready entity=mon.2 client_cookie=e945311f5626d15f server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:26.521 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.514+0000 7fd56d59a700 1 -- 172.21.15.23:0/3625647507 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd564009630 con 0x7fd5700678b0 2020-11-20T11:52:26.521 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.514+0000 7fd56d59a700 1 -- 172.21.15.23:0/3625647507 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd564000e90 con 0x7fd5700678b0 2020-11-20T11:52:26.521 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.514+0000 7fd56d59a700 1 -- 172.21.15.23:0/3625647507 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5640099a0 con 0x7fd5700678b0 2020-11-20T11:52:26.521 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.515+0000 7fd574b24700 1 -- 172.21.15.23:0/3625647507 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5700678b0 msgr2=0x7fd570067d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:26.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.515+0000 7fd574b24700 1 --2- 172.21.15.23:0/3625647507 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5700678b0 0x7fd570067d30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fd56400ba00 tx=0x7fd564006e30).stop 2020-11-20T11:52:26.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.515+0000 7fd574b24700 1 -- 172.21.15.23:0/3625647507 shutdown_connections 2020-11-20T11:52:26.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.515+0000 7fd574b24700 1 --2- 172.21.15.23:0/3625647507 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd5700678b0 0x7fd570067d30 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:26.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.515+0000 7fd574b24700 1 -- 172.21.15.23:0/3625647507 >> 172.21.15.23:0/3625647507 conn(0x7fd570066730 msgr2=0x7fd5700601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:26.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.515+0000 7fd574b24700 1 -- 172.21.15.23:0/3625647507 shutdown_connections 2020-11-20T11:52:26.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.515+0000 7fd574b24700 1 -- 172.21.15.23:0/3625647507 wait complete. 2020-11-20T11:52:26.525 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.516+0000 7fd574b24700 1 Processor -- start 2020-11-20T11:52:26.525 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.516+0000 7fd574b24700 1 -- start start 2020-11-20T11:52:26.525 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.516+0000 7fd574b24700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd570060a80 0x7fd57013e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:26.526 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.516+0000 7fd574b24700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5700678b0 0x7fd57013a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:26.526 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.516+0000 7fd574b24700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5700abbd0 0x7fd57013aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:26.526 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.516+0000 7fd574b24700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57013afe0 con 0x7fd5700678b0 2020-11-20T11:52:26.526 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.516+0000 7fd574b24700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd570136980 con 0x7fd5700abbd0 2020-11-20T11:52:26.526 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.516+0000 7fd574b24700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd57013d6a0 con 0x7fd570060a80 2020-11-20T11:52:26.527 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.516+0000 7fd56dd9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5700678b0 0x7fd57013a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:26.527 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.516+0000 7fd56dd9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5700678b0 0x7fd57013a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47412/0 (socket says 172.21.15.23:47412) 2020-11-20T11:52:26.527 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.516+0000 7fd56e59c700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd570060a80 0x7fd57013e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:26.527 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.516+0000 7fd56dd9b700 1 -- 172.21.15.23:0/2965020898 learned_addr learned my addr 172.21.15.23:0/2965020898 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:52:26.527 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.517+0000 7fd56dd9b700 1 -- 172.21.15.23:0/2965020898 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd570060a80 msgr2=0x7fd57013e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:26.528 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.517+0000 7fd56dd9b700 1 --2- 172.21.15.23:0/2965020898 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd570060a80 0x7fd57013e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:26.528 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.517+0000 7fd56dd9b700 1 -- 172.21.15.23:0/2965020898 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5700abbd0 msgr2=0x7fd57013aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:26.528 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.517+0000 7fd56dd9b700 1 --2- 172.21.15.23:0/2965020898 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5700abbd0 0x7fd57013aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:52:26.528 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.517+0000 7fd56dd9b700 1 -- 172.21.15.23:0/2965020898 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd57013d940 con 0x7fd5700678b0 2020-11-20T11:52:26.528 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.517+0000 7fd56dd9b700 1 --2- 172.21.15.23:0/2965020898 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5700678b0 0x7fd57013a600 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fd56400b150 tx=0x7fd564015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:26.529 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.518+0000 7fd55e7fc700 1 -- 172.21.15.23:0/2965020898 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd564011e50 con 0x7fd5700678b0 2020-11-20T11:52:26.529 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.518+0000 7fd55e7fc700 1 -- 172.21.15.23:0/2965020898 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd56401e460 con 0x7fd5700678b0 2020-11-20T11:52:26.529 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.518+0000 7fd574b24700 1 -- 172.21.15.23:0/2965020898 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd57013dbe0 con 0x7fd5700678b0 2020-11-20T11:52:26.529 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.518+0000 7fd55e7fc700 1 -- 172.21.15.23:0/2965020898 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56401e7e0 con 0x7fd5700678b0 2020-11-20T11:52:26.529 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.518+0000 7fd574b24700 1 -- 172.21.15.23:0/2965020898 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd570134f20 con 0x7fd5700678b0 2020-11-20T11:52:26.530 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.519+0000 7fd574b24700 1 -- 172.21.15.23:0/2965020898 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd57013dd70 con 0x7fd5700678b0 2020-11-20T11:52:26.530 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.519+0000 7fd55e7fc700 1 -- 172.21.15.23:0/2965020898 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd564025030 con 0x7fd5700678b0 2020-11-20T11:52:26.530 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.519+0000 7fd55e7fc700 1 --2- 172.21.15.23:0/2965020898 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd544056d60 0x7fd544059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:26.530 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.519+0000 7fd55e7fc700 1 -- 172.21.15.23:0/2965020898 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fd564071880 con 0x7fd5700678b0 2020-11-20T11:52:26.530 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.520+0000 7fd56e59c700 1 --2- 172.21.15.23:0/2965020898 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd544056d60 0x7fd544059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:26.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.520+0000 7fd56e59c700 1 --2- 172.21.15.23:0/2965020898 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd544056d60 0x7fd544059220 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7fd5600055e0 tx=0x7fd560009040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:26.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.523+0000 7fd55e7fc700 1 -- 172.21.15.23:0/2965020898 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fd564046c20 con 0x7fd5700678b0 2020-11-20T11:52:26.745 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.742+0000 7fd574b24700 1 -- 172.21.15.23:0/2965020898 --> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7fd57013c2f0 con 0x7fd544056d60 2020-11-20T11:52:26.746 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.743+0000 7fd55e7fc700 1 -- 172.21.15.23:0/2965020898 <== mgr.4105 v2:172.21.15.155:6832/22693 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd56000f460 con 0x7fd544056d60 2020-11-20T11:52:26.748 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.744+0000 7fd574b24700 1 -- 172.21.15.23:0/2965020898 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd544056d60 msgr2=0x7fd544059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:26.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.744+0000 7fd574b24700 1 --2- 172.21.15.23:0/2965020898 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd544056d60 0x7fd544059220 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7fd5600055e0 tx=0x7fd560009040).stop 2020-11-20T11:52:26.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.744+0000 7fd574b24700 1 -- 172.21.15.23:0/2965020898 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5700678b0 msgr2=0x7fd57013a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:26.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.744+0000 7fd574b24700 1 --2- 172.21.15.23:0/2965020898 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5700678b0 0x7fd57013a600 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fd56400b150 tx=0x7fd564015000).stop 2020-11-20T11:52:26.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.745+0000 7fd574b24700 1 -- 172.21.15.23:0/2965020898 shutdown_connections 2020-11-20T11:52:26.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.745+0000 7fd574b24700 1 --2- 172.21.15.23:0/2965020898 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd5700abbd0 0x7fd57013aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2020-11-20T11:52:26.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.745+0000 7fd574b24700 1 --2- 172.21.15.23:0/2965020898 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd544056d60 0x7fd544059220 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:26.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.745+0000 7fd574b24700 1 --2- 172.21.15.23:0/2965020898 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd570060a80 0x7fd57013e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:26.750 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.745+0000 7fd574b24700 1 --2- 172.21.15.23:0/2965020898 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5700678b0 0x7fd57013a600 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:26.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.745+0000 7fd574b24700 1 -- 172.21.15.23:0/2965020898 >> 172.21.15.23:0/2965020898 conn(0x7fd570066730 msgr2=0x7fd570068f00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:26.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.745+0000 7fd574b24700 1 -- 172.21.15.23:0/2965020898 shutdown_connections 2020-11-20T11:52:26.751 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.745+0000 7fd574b24700 1 -- 172.21.15.23:0/2965020898 wait complete. 2020-11-20T11:52:26.751 INFO:teuthology.orchestra.run.smithi023.stderr:instructed osd(s) 6 to deep-scrub 2020-11-20T11:52:26.760 INFO:tasks.ceph:Scrubbing osd.7 2020-11-20T11:52:26.761 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2020-11-20T11:52:26.913 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.910+0000 7fd96136e700 1 Processor -- start 2020-11-20T11:52:26.913 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.910+0000 7fd96136e700 1 -- start start 2020-11-20T11:52:26.915 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.911+0000 7fd96136e700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd95c0a5750 0x7fd95c0a3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:26.916 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.911+0000 7fd96136e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd95c0a90c0 0x7fd95c0aa2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:26.916 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.911+0000 7fd96136e700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd95c0a9550 con 0x7fd95c0a45b0 2020-11-20T11:52:26.916 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.911+0000 7fd96136e700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd95c0a4b30 con 0x7fd95c0a5750 2020-11-20T11:52:26.916 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.911+0000 7fd96136e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c0a4dd0 con 0x7fd95c0a90c0 2020-11-20T11:52:26.916 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.911+0000 7fd95bfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd95c0a5750 0x7fd95c0a3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:26.917 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.911+0000 7fd95bfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7fd95c0a5750 0x7fd95c0a3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47420/0 (socket says 172.21.15.23:47420) 2020-11-20T11:52:26.917 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.911+0000 7fd95bfff700 1 -- 172.21.15.23:0/3224357893 learned_addr learned my addr 172.21.15.23:0/3224357893 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:52:26.917 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.911+0000 7fd960b6d700 1 --2- 172.21.15.23:0/3224357893 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd95c0a90c0 0x7fd95c0aa2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:26.917 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.911+0000 7fd960b6d700 1 -- 172.21.15.23:0/3224357893 >> v2:172.21.15.23:3300/0 conn(0x7fd95c0a5750 msgr2=0x7fd95c0a3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:52:26.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.911+0000 7fd960b6d700 1 --2- 172.21.15.23:0/3224357893 >> v2:172.21.15.23:3300/0 conn(0x7fd95c0a5750 0x7fd95c0a3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:26.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.911+0000 7fd960b6d700 1 -- 172.21.15.23:0/3224357893 >> v1:172.21.15.155:6789/0 conn(0x7fd95c0a45b0 legacy=0x7fd95c0a9ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-11-20T11:52:26.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.911+0000 7fd960b6d700 1 -- 172.21.15.23:0/3224357893 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd95c0a5b30 con 0x7fd95c0a90c0 2020-11-20T11:52:26.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.911+0000 7fd95bfff700 1 --2- 172.21.15.23:0/3224357893 >> v2:172.21.15.23:3300/0 conn(0x7fd95c0a5750 0x7fd95c0a3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:52:26.918 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.912+0000 7fd960b6d700 1 --2- 172.21.15.23:0/3224357893 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd95c0a90c0 0x7fd95c0aa2b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fd954009ce0 tx=0x7fd954009220).ready entity=mon.2 client_cookie=c8d8f8cc160bad7a server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:26.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.914+0000 7fd95affd700 1 -- 172.21.15.23:0/3224357893 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95400ba00 con 0x7fd95c0a90c0 2020-11-20T11:52:26.921 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.914+0000 7fd95affd700 1 -- 172.21.15.23:0/3224357893 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd95400bb60 con 0x7fd95c0a90c0 2020-11-20T11:52:26.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.914+0000 7fd95affd700 1 -- 172.21.15.23:0/3224357893 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd954022410 con 0x7fd95c0a90c0 2020-11-20T11:52:26.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.915+0000 7fd96136e700 1 -- 172.21.15.23:0/3224357893 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd95c0a90c0 msgr2=0x7fd95c0aa2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:26.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.915+0000 7fd96136e700 1 --2- 172.21.15.23:0/3224357893 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd95c0a90c0 0x7fd95c0aa2b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fd954009ce0 tx=0x7fd954009220).stop 2020-11-20T11:52:26.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.915+0000 7fd96136e700 1 -- 172.21.15.23:0/3224357893 shutdown_connections 2020-11-20T11:52:26.922 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.915+0000 7fd96136e700 1 --2- 172.21.15.23:0/3224357893 >> v2:172.21.15.23:3300/0 conn(0x7fd95c0a5750 0x7fd95c0a3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:26.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.915+0000 7fd96136e700 1 --2- 172.21.15.23:0/3224357893 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd95c0a90c0 0x7fd95c0aa2b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:26.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.915+0000 7fd96136e700 1 -- 172.21.15.23:0/3224357893 >> 172.21.15.23:0/3224357893 conn(0x7fd95c0601b0 msgr2=0x7fd95c0605c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:26.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.915+0000 7fd96136e700 1 -- 172.21.15.23:0/3224357893 shutdown_connections 2020-11-20T11:52:26.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.915+0000 7fd96136e700 1 -- 172.21.15.23:0/3224357893 wait complete. 2020-11-20T11:52:26.923 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.916+0000 7fd96136e700 1 Processor -- start 2020-11-20T11:52:26.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.916+0000 7fd96136e700 1 -- start start 2020-11-20T11:52:26.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.917+0000 7fd96136e700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd95c0a45b0 0x7fd95c0acfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:26.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.917+0000 7fd96136e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd95c0a5750 0x7fd95c1389b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:26.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.917+0000 7fd96136e700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd95c0a90c0 0x7fd95c149130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:26.924 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.917+0000 7fd96136e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c148a60 con 0x7fd95c0a5750 2020-11-20T11:52:26.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.917+0000 7fd96136e700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c138ea0 con 0x7fd95c0a90c0 2020-11-20T11:52:26.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.917+0000 7fd96136e700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c145490 con 0x7fd95c0a45b0 2020-11-20T11:52:26.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.917+0000 7fd95b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd95c0a5750 0x7fd95c1389b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:26.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.917+0000 7fd95b7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd95c0a5750 0x7fd95c1389b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47422/0 (socket says 172.21.15.23:47422) 2020-11-20T11:52:26.925 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.917+0000 7fd95b7fe700 1 -- 172.21.15.23:0/217224051 learned_addr learned my addr 172.21.15.23:0/217224051 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:52:26.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.917+0000 7fd95bfff700 1 --2- 172.21.15.23:0/217224051 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd95c0a45b0 0x7fd95c0acfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:26.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.917+0000 7fd960b6d700 1 --2- 172.21.15.23:0/217224051 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd95c0a90c0 0x7fd95c149130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:26.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.917+0000 7fd95b7fe700 1 -- 172.21.15.23:0/217224051 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd95c0a45b0 msgr2=0x7fd95c0acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:26.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.917+0000 7fd95b7fe700 1 --2- 172.21.15.23:0/217224051 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd95c0a45b0 0x7fd95c0acfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:26.926 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.917+0000 7fd95b7fe700 1 -- 172.21.15.23:0/217224051 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd95c0a90c0 msgr2=0x7fd95c149130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:26.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.917+0000 7fd95b7fe700 1 --2- 172.21.15.23:0/217224051 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd95c0a90c0 0x7fd95c149130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:26.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.917+0000 7fd95b7fe700 1 -- 172.21.15.23:0/217224051 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd95c139130 con 0x7fd95c0a5750 2020-11-20T11:52:26.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.917+0000 7fd95b7fe700 1 --2- 172.21.15.23:0/217224051 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd95c0a5750 0x7fd95c1389b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fd950009730 tx=0x7fd950006e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:26.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.917+0000 7fd93bfff700 1 -- 172.21.15.23:0/217224051 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95000d4a0 con 0x7fd95c0a5750 2020-11-20T11:52:26.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.917+0000 7fd93bfff700 1 -- 172.21.15.23:0/217224051 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd95000d600 con 0x7fd95c0a5750 2020-11-20T11:52:26.927 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.918+0000 7fd96136e700 1 -- 172.21.15.23:0/217224051 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd95c148c90 con 0x7fd95c0a5750 2020-11-20T11:52:26.928 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.918+0000 7fd93bfff700 1 -- 172.21.15.23:0/217224051 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95000d9a0 con 0x7fd95c0a5750 2020-11-20T11:52:26.932 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.918+0000 7fd96136e700 1 -- 172.21.15.23:0/217224051 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd95c13ad40 con 0x7fd95c0a5750 2020-11-20T11:52:26.932 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.918+0000 7fd96136e700 1 -- 172.21.15.23:0/217224051 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd95c057dc0 con 0x7fd95c0a5750 2020-11-20T11:52:26.934 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.919+0000 7fd93bfff700 1 -- 172.21.15.23:0/217224051 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fd95001b030 con 0x7fd95c0a5750 2020-11-20T11:52:26.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.919+0000 7fd93bfff700 1 --2- 172.21.15.23:0/217224051 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd934056d60 0x7fd934059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:26.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.919+0000 7fd93bfff700 1 -- 172.21.15.23:0/217224051 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fd95006fd90 con 0x7fd95c0a5750 2020-11-20T11:52:26.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.919+0000 7fd93bfff700 1 --2- 172.21.15.23:0/217224051 >> [v2:172.21.15.155:6808/23281,v1:172.21.15.155:6809/23281] conn(0x7fd93405e940 0x7fd934060d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:26.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.919+0000 7fd93bfff700 1 -- 172.21.15.23:0/217224051 --> [v2:172.21.15.155:6808/23281,v1:172.21.15.155:6809/23281] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd9340613c0 con 0x7fd93405e940 2020-11-20T11:52:26.935 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.919+0000 7fd93bfff700 1 -- 172.21.15.23:0/217224051 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd950018020 con 0x7fd95c0a5750 2020-11-20T11:52:26.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.919+0000 7fd95bfff700 1 --2- 172.21.15.23:0/217224051 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd934056d60 0x7fd934059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:26.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.920+0000 7fd960b6d700 1 --2- 172.21.15.23:0/217224051 >> [v2:172.21.15.155:6808/23281,v1:172.21.15.155:6809/23281] conn(0x7fd93405e940 0x7fd934060d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:26.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.920+0000 7fd95bfff700 1 --2- 172.21.15.23:0/217224051 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd934056d60 0x7fd934059220 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fd94c006a30 tx=0x7fd94c013040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:26.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.920+0000 7fd960b6d700 1 --2- 172.21.15.23:0/217224051 >> [v2:172.21.15.155:6808/23281,v1:172.21.15.155:6809/23281] conn(0x7fd93405e940 0x7fd934060d60 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:26.936 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.921+0000 7fd93bfff700 1 -- 172.21.15.23:0/217224051 <== osd.7 v2:172.21.15.155:6808/23281 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22725 (crc 0 0 0) 0x7fd9340613c0 con 0x7fd93405e940 2020-11-20T11:52:26.950 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.948+0000 7fd96136e700 1 -- 172.21.15.23:0/217224051 --> [v2:172.21.15.155:6808/23281,v1:172.21.15.155:6809/23281] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd95c149860 con 0x7fd93405e940 2020-11-20T11:52:26.954 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.950+0000 7fd93bfff700 1 -- 172.21.15.23:0/217224051 <== osd.7 v2:172.21.15.155:6808/23281 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fd95c149860 con 0x7fd93405e940 2020-11-20T11:52:26.954 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.950+0000 7fd96136e700 1 -- 172.21.15.23:0/217224051 >> [v2:172.21.15.155:6808/23281,v1:172.21.15.155:6809/23281] conn(0x7fd93405e940 msgr2=0x7fd934060d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:26.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.950+0000 7fd96136e700 1 --2- 172.21.15.23:0/217224051 >> [v2:172.21.15.155:6808/23281,v1:172.21.15.155:6809/23281] conn(0x7fd93405e940 0x7fd934060d60 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:26.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.951+0000 7fd96136e700 1 -- 172.21.15.23:0/217224051 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd934056d60 msgr2=0x7fd934059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:26.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.951+0000 7fd96136e700 1 --2- 172.21.15.23:0/217224051 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fd934056d60 0x7fd934059220 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fd94c006a30 tx=0x7fd94c013040).stop 2020-11-20T11:52:26.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.951+0000 7fd96136e700 1 -- 172.21.15.23:0/217224051 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd95c0a5750 msgr2=0x7fd95c1389b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:26.955 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.951+0000 7fd96136e700 1 --2- 172.21.15.23:0/217224051 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd95c0a5750 0x7fd95c1389b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fd950009730 tx=0x7fd950006e30).stop 2020-11-20T11:52:26.956 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.951+0000 7fd960b6d700 1 -- 172.21.15.23:0/217224051 reap_dead start 2020-11-20T11:52:26.956 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.951+0000 7fd96136e700 1 -- 172.21.15.23:0/217224051 shutdown_connections 2020-11-20T11:52:26.956 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.951+0000 7fd96136e700 1 -- 172.21.15.23:0/217224051 >> 172.21.15.23:0/217224051 conn(0x7fd95c0601b0 msgr2=0x7fd95c0aa2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:26.956 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.951+0000 7fd96136e700 1 -- 172.21.15.23:0/217224051 shutdown_connections 2020-11-20T11:52:26.957 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:26.951+0000 7fd96136e700 1 -- 172.21.15.23:0/217224051 wait complete. 2020-11-20T11:52:26.957 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2020-11-20T11:52:26.957 INFO:teuthology.orchestra.run.smithi023.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2020-11-20T11:52:26.957 INFO:teuthology.orchestra.run.smithi023.stdout:} 2020-11-20T11:52:26.967 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2020-11-20T11:52:27.129 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.126+0000 7f71bacad700 1 Processor -- start 2020-11-20T11:52:27.129 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.126+0000 7f71bacad700 1 -- start start 2020-11-20T11:52:27.131 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.127+0000 7f71bacad700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f71b40a9040 0x7f71b40aaf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:27.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.127+0000 7f71bacad700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71b40ad2f0 0x7f71b413e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:27.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.127+0000 7f71bacad700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b40ad860 con 0x7f71b40ace70 2020-11-20T11:52:27.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.127+0000 7f71bacad700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71b40aa8a0 con 0x7f71b40a9040 2020-11-20T11:52:27.132 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.127+0000 7f71bacad700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71b40aaa90 con 0x7f71b40ad2f0 2020-11-20T11:52:27.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.127+0000 7f71b9cab700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f71b40a9040 0x7f71b40aaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:27.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.127+0000 7f71b94aa700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f71b40ace70 0x7f71b413de00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:33450/0 (socket says 172.21.15.23:33450) 2020-11-20T11:52:27.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.127+0000 7f71b94aa700 1 -- 172.21.15.23:0/755243993 learned_addr learned my addr 172.21.15.23:0/755243993 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:52:27.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.127+0000 7f71ba4ac700 1 --2- 172.21.15.23:0/755243993 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71b40ad2f0 0x7f71b413e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:27.133 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.127+0000 7f71b9cab700 1 -- 172.21.15.23:0/755243993 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71b40ad2f0 msgr2=0x7f71b413e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:52:27.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.127+0000 7f71b9cab700 1 --2- 172.21.15.23:0/755243993 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71b40ad2f0 0x7f71b413e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:27.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.127+0000 7f71b9cab700 1 -- 172.21.15.23:0/755243993 >> v1:172.21.15.23:6789/0 conn(0x7f71b40ace70 legacy=0x7f71b413de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:27.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.127+0000 7f71b8ca9700 1 -- 172.21.15.23:0/755243993 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1432859854 0 0) 0x7f71b40ad860 con 0x7f71b40ace70 2020-11-20T11:52:27.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.127+0000 7f71b9cab700 1 -- 172.21.15.23:0/755243993 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b40a9420 con 0x7f71b40a9040 2020-11-20T11:52:27.134 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.128+0000 7f71b9cab700 1 --2- 172.21.15.23:0/755243993 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71b40a9040 0x7f71b40aaf30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f71b000b9d0 tx=0x7f71b0006dd0).ready entity=mon.0 client_cookie=a1659af825ae666e server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:27.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.128+0000 7f71b8ca9700 1 -- 172.21.15.23:0/755243993 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b0006030 con 0x7f71b40a9040 2020-11-20T11:52:27.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.128+0000 7f71b8ca9700 1 -- 172.21.15.23:0/755243993 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71b0006190 con 0x7f71b40a9040 2020-11-20T11:52:27.135 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.128+0000 7f71b8ca9700 1 -- 172.21.15.23:0/755243993 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b0006510 con 0x7f71b40a9040 2020-11-20T11:52:27.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.128+0000 7f71bacad700 1 -- 172.21.15.23:0/755243993 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71b40a9040 msgr2=0x7f71b40aaf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:27.138 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.128+0000 7f71bacad700 1 --2- 172.21.15.23:0/755243993 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71b40a9040 0x7f71b40aaf30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f71b000b9d0 tx=0x7f71b0006dd0).stop 2020-11-20T11:52:27.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.129+0000 7f71bacad700 1 -- 172.21.15.23:0/755243993 shutdown_connections 2020-11-20T11:52:27.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.129+0000 7f71bacad700 1 --2- 172.21.15.23:0/755243993 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71b40ad2f0 0x7f71b413e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:27.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.129+0000 7f71bacad700 1 --2- 172.21.15.23:0/755243993 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71b40a9040 0x7f71b40aaf30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:27.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.129+0000 7f71bacad700 1 -- 172.21.15.23:0/755243993 >> 172.21.15.23:0/755243993 conn(0x7f71b40683b0 msgr2=0x7f71b40b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:27.139 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.129+0000 7f71bacad700 1 -- 172.21.15.23:0/755243993 shutdown_connections 2020-11-20T11:52:27.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.129+0000 7f71bacad700 1 -- 172.21.15.23:0/755243993 wait complete. 2020-11-20T11:52:27.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.130+0000 7f71bacad700 1 Processor -- start 2020-11-20T11:52:27.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.130+0000 7f71bacad700 1 -- start start 2020-11-20T11:52:27.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.130+0000 7f71bacad700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71b40a9040 0x7f71b409f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:27.140 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.130+0000 7f71bacad700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71b40ace70 0x7f71b4149cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:27.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.130+0000 7f71bacad700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71b40ad2f0 0x7f71b40a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:27.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.131+0000 7f71bacad700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b414a1b0 con 0x7f71b40ad2f0 2020-11-20T11:52:27.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.131+0000 7f71bacad700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b409ea60 con 0x7f71b40ace70 2020-11-20T11:52:27.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.131+0000 7f71bacad700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71b409ed00 con 0x7f71b40a9040 2020-11-20T11:52:27.141 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.131+0000 7f71b94aa700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71b40ace70 0x7f71b4149cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:27.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.131+0000 7f71ba4ac700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71b40ad2f0 0x7f71b40a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:27.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.131+0000 7f71b94aa700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71b40ace70 0x7f71b4149cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:40336/0 (socket says 172.21.15.23:40336) 2020-11-20T11:52:27.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.131+0000 7f71b94aa700 1 -- 172.21.15.23:0/3858537607 learned_addr learned my addr 172.21.15.23:0/3858537607 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:52:27.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.131+0000 7f71b9cab700 1 --2- 172.21.15.23:0/3858537607 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71b40a9040 0x7f71b409f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:27.142 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.131+0000 7f71b9cab700 1 -- 172.21.15.23:0/3858537607 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71b40ace70 msgr2=0x7f71b4149cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:27.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.131+0000 7f71b9cab700 1 --2- 172.21.15.23:0/3858537607 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71b40ace70 0x7f71b4149cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:27.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.131+0000 7f71b9cab700 1 -- 172.21.15.23:0/3858537607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71b40ad2f0 msgr2=0x7f71b40a0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:27.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.131+0000 7f71b9cab700 1 --2- 172.21.15.23:0/3858537607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71b40ad2f0 0x7f71b40a0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:27.143 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.131+0000 7f71b9cab700 1 -- 172.21.15.23:0/3858537607 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b409efa0 con 0x7f71b40a9040 2020-11-20T11:52:27.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.131+0000 7f71b94aa700 1 --2- 172.21.15.23:0/3858537607 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71b40ace70 0x7f71b4149cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:52:27.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.131+0000 7f71b9cab700 1 --2- 172.21.15.23:0/3858537607 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71b40a9040 0x7f71b409f440 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f71b40a8c40 tx=0x7f71b0015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:27.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.131+0000 7f71a1ffb700 1 -- 172.21.15.23:0/3858537607 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b0005920 con 0x7f71b40a9040 2020-11-20T11:52:27.144 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.131+0000 7f71a1ffb700 1 -- 172.21.15.23:0/3858537607 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71b0005a80 con 0x7f71b40a9040 2020-11-20T11:52:27.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.131+0000 7f71a1ffb700 1 -- 172.21.15.23:0/3858537607 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b0005e00 con 0x7f71b40a9040 2020-11-20T11:52:27.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.131+0000 7f71bacad700 1 -- 172.21.15.23:0/3858537607 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71b4140860 con 0x7f71b40a9040 2020-11-20T11:52:27.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.132+0000 7f71bacad700 1 -- 172.21.15.23:0/3858537607 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71b414a580 con 0x7f71b40a9040 2020-11-20T11:52:27.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.133+0000 7f71bacad700 1 -- 172.21.15.23:0/3858537607 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71b4057dc0 con 0x7f71b40a9040 2020-11-20T11:52:27.145 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.133+0000 7f71a1ffb700 1 -- 172.21.15.23:0/3858537607 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f71b001a040 con 0x7f71b40a9040 2020-11-20T11:52:27.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.133+0000 7f71a1ffb700 1 --2- 172.21.15.23:0/3858537607 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f718c056d60 0x7f718c059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:27.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.133+0000 7f71a1ffb700 1 -- 172.21.15.23:0/3858537607 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f71b0071530 con 0x7f71b40a9040 2020-11-20T11:52:27.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.134+0000 7f71b94aa700 1 --2- 172.21.15.23:0/3858537607 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f718c056d60 0x7f718c059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:27.146 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.134+0000 7f71b94aa700 1 --2- 172.21.15.23:0/3858537607 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f718c056d60 0x7f718c059220 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f71b40b10c0 tx=0x7f71a4007b00).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:27.147 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.138+0000 7f71a1ffb700 1 -- 172.21.15.23:0/3858537607 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f71b00468a0 con 0x7f71b40a9040 2020-11-20T11:52:27.334 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.331+0000 7f71bacad700 1 -- 172.21.15.23:0/3858537607 --> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f71b40b1ee0 con 0x7f718c056d60 2020-11-20T11:52:27.334 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.331+0000 7f71a1ffb700 1 -- 172.21.15.23:0/3858537607 <== mgr.4105 v2:172.21.15.155:6832/22693 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f71b40b1ee0 con 0x7f718c056d60 2020-11-20T11:52:27.336 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.333+0000 7f71bacad700 1 -- 172.21.15.23:0/3858537607 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f718c056d60 msgr2=0x7f718c059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:27.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.333+0000 7f71bacad700 1 --2- 172.21.15.23:0/3858537607 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f718c056d60 0x7f718c059220 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f71b40b10c0 tx=0x7f71a4007b00).stop 2020-11-20T11:52:27.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.333+0000 7f71bacad700 1 -- 172.21.15.23:0/3858537607 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71b40a9040 msgr2=0x7f71b409f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:27.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.333+0000 7f71bacad700 1 --2- 172.21.15.23:0/3858537607 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71b40a9040 0x7f71b409f440 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f71b40a8c40 tx=0x7f71b0015000).stop 2020-11-20T11:52:27.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.333+0000 7f71bacad700 1 -- 172.21.15.23:0/3858537607 shutdown_connections 2020-11-20T11:52:27.337 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.333+0000 7f71bacad700 1 --2- 172.21.15.23:0/3858537607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f71b40ad2f0 0x7f71b40a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:27.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.333+0000 7f71bacad700 1 --2- 172.21.15.23:0/3858537607 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f718c056d60 0x7f718c059220 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:27.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.333+0000 7f71bacad700 1 --2- 172.21.15.23:0/3858537607 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f71b40a9040 0x7f71b409f440 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:27.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.333+0000 7f71bacad700 1 --2- 172.21.15.23:0/3858537607 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f71b40ace70 0x7f71b4149cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:27.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.333+0000 7f71bacad700 1 -- 172.21.15.23:0/3858537607 >> 172.21.15.23:0/3858537607 conn(0x7f71b40683b0 msgr2=0x7f71b40ac330 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:27.338 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.333+0000 7f71bacad700 1 -- 172.21.15.23:0/3858537607 shutdown_connections 2020-11-20T11:52:27.339 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.333+0000 7f71bacad700 1 -- 172.21.15.23:0/3858537607 wait complete. 2020-11-20T11:52:27.339 INFO:teuthology.orchestra.run.smithi023.stderr:instructed osd(s) 7 to deep-scrub 2020-11-20T11:52:27.351 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-20T11:52:27.508 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.505+0000 7efe780fa700 1 Processor -- start 2020-11-20T11:52:27.509 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.505+0000 7efe780fa700 1 -- start start 2020-11-20T11:52:27.517 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.506+0000 7efe780fa700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7efe7013aeb0 0x7efe700a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:27.517 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.506+0000 7efe780fa700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe701388b0 con 0x7efe700aaab0 2020-11-20T11:52:27.517 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.506+0000 7efe780fa700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe7013e600 con 0x7efe700aae90 2020-11-20T11:52:27.517 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.506+0000 7efe780fa700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe7013e8a0 con 0x7efe7013aeb0 2020-11-20T11:52:27.518 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.506+0000 7efe75e96700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7efe700aaab0 0x7efe7013ed40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:33464/0 (socket says 172.21.15.23:33464) 2020-11-20T11:52:27.518 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.506+0000 7efe75e96700 1 -- 172.21.15.23:0/3669347242 learned_addr learned my addr 172.21.15.23:0/3669347242 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:52:27.518 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.506+0000 7efe74e94700 1 -- 172.21.15.23:0/3669347242 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2600312867 0 0) 0x7efe701388b0 con 0x7efe700aaab0 2020-11-20T11:52:27.518 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.506+0000 7efe74e94700 1 -- 172.21.15.23:0/3669347242 --> v1:172.21.15.23:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe58003280 con 0x7efe700aaab0 2020-11-20T11:52:27.518 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.506+0000 7efe74e94700 1 -- 172.21.15.23:0/3669347242 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2877603890 0 0) 0x7efe7013e600 con 0x7efe700aae90 2020-11-20T11:52:27.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.506+0000 7efe74e94700 1 -- 172.21.15.23:0/3669347242 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe701388b0 con 0x7efe700aae90 2020-11-20T11:52:27.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.507+0000 7efe74e94700 1 -- 172.21.15.23:0/3669347242 <== mon.0 v1:172.21.15.23:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1880612909 0 0) 0x7efe58003280 con 0x7efe700aaab0 2020-11-20T11:52:27.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.507+0000 7efe74e94700 1 -- 172.21.15.23:0/3669347242 >> v2:172.21.15.23:3300/0 conn(0x7efe7013aeb0 msgr2=0x7efe700a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:52:27.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.507+0000 7efe74e94700 1 --2- 172.21.15.23:0/3669347242 >> v2:172.21.15.23:3300/0 conn(0x7efe7013aeb0 0x7efe700a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2020-11-20T11:52:27.519 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.507+0000 7efe74e94700 1 -- 172.21.15.23:0/3669347242 >> v1:172.21.15.155:6789/0 conn(0x7efe700aae90 legacy=0x7efe701381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:27.520 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.507+0000 7efe74e94700 1 -- 172.21.15.23:0/3669347242 --> v1:172.21.15.23:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe700ab6d0 con 0x7efe700aaab0 2020-11-20T11:52:27.520 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.507+0000 7efe74e94700 1 -- 172.21.15.23:0/3669347242 <== mon.0 v1:172.21.15.23:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 471396175 0 0) 0x7efe6c001520 con 0x7efe700aaab0 2020-11-20T11:52:27.520 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.507+0000 7efe74e94700 1 -- 172.21.15.23:0/3669347242 >> v1:172.21.15.23:6789/0 conn(0x7efe700aaab0 legacy=0x7efe7013ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:27.520 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.507+0000 7efe74e94700 1 --2- 172.21.15.23:0/3669347242 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe58003900 0x7efe58003d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:27.521 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.507+0000 7efe74e94700 1 --2- 172.21.15.23:0/3669347242 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe58004520 0x7efe580049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:27.521 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.507+0000 7efe74e94700 1 --2- 172.21.15.23:0/3669347242 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe58004e90 0x7efe5800b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:27.521 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.507+0000 7efe74e94700 1 -- 172.21.15.23:0/3669347242 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7013e8a0 con 0x7efe58004520 2020-11-20T11:52:27.521 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.507+0000 7efe74e94700 1 -- 172.21.15.23:0/3669347242 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe5800bc60 con 0x7efe58003900 2020-11-20T11:52:27.521 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.507+0000 7efe74e94700 1 -- 172.21.15.23:0/3669347242 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe5800bda0 con 0x7efe58004e90 2020-11-20T11:52:27.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.507+0000 7efe75695700 1 --2- 172.21.15.23:0/3669347242 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe58004520 0x7efe580049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:27.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.507+0000 7efe75e96700 1 --2- 172.21.15.23:0/3669347242 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe58003900 0x7efe58003d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:27.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.507+0000 7efe76697700 1 --2- 172.21.15.23:0/3669347242 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe58004e90 0x7efe5800b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:27.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.507+0000 7efe75695700 1 -- 172.21.15.23:0/3669347242 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe58004e90 msgr2=0x7efe5800b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:52:27.522 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.507+0000 7efe75695700 1 --2- 172.21.15.23:0/3669347242 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe58004e90 0x7efe5800b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:27.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.507+0000 7efe75695700 1 -- 172.21.15.23:0/3669347242 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe58003900 msgr2=0x7efe58003d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:52:27.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.507+0000 7efe75695700 1 --2- 172.21.15.23:0/3669347242 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe58003900 0x7efe58003d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:27.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.507+0000 7efe75695700 1 -- 172.21.15.23:0/3669347242 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe58003280 con 0x7efe58004520 2020-11-20T11:52:27.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.507+0000 7efe76697700 1 -- 172.21.15.23:0/3669347242 reap_dead start 2020-11-20T11:52:27.523 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.507+0000 7efe75e96700 1 --2- 172.21.15.23:0/3669347242 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe58003900 0x7efe58003d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-11-20T11:52:27.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.507+0000 7efe75695700 1 --2- 172.21.15.23:0/3669347242 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe58004520 0x7efe580049a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7efe60001af0 tx=0x7efe60007bd0).ready entity=mon.0 client_cookie=50fcae90b662932 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:27.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.507+0000 7efe74e94700 1 -- 172.21.15.23:0/3669347242 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe60007020 con 0x7efe58004520 2020-11-20T11:52:27.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.508+0000 7efe74e94700 1 -- 172.21.15.23:0/3669347242 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe60007180 con 0x7efe58004520 2020-11-20T11:52:27.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.508+0000 7efe74e94700 1 -- 172.21.15.23:0/3669347242 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6000c8c0 con 0x7efe58004520 2020-11-20T11:52:27.524 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.508+0000 7efe780fa700 1 -- 172.21.15.23:0/3669347242 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe58004520 msgr2=0x7efe580049a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:27.525 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.508+0000 7efe780fa700 1 --2- 172.21.15.23:0/3669347242 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe58004520 0x7efe580049a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7efe60001af0 tx=0x7efe60007bd0).stop 2020-11-20T11:52:27.525 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.508+0000 7efe780fa700 1 -- 172.21.15.23:0/3669347242 shutdown_connections 2020-11-20T11:52:27.525 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.508+0000 7efe780fa700 1 --2- 172.21.15.23:0/3669347242 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe58004520 0x7efe580049a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:27.526 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.508+0000 7efe780fa700 1 -- 172.21.15.23:0/3669347242 >> 172.21.15.23:0/3669347242 conn(0x7efe700678e0 msgr2=0x7efe70067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:27.526 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.508+0000 7efe780fa700 1 -- 172.21.15.23:0/3669347242 shutdown_connections 2020-11-20T11:52:27.526 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.508+0000 7efe780fa700 1 -- 172.21.15.23:0/3669347242 wait complete. 2020-11-20T11:52:27.527 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.509+0000 7efe780fa700 1 Processor -- start 2020-11-20T11:52:27.527 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.509+0000 7efe780fa700 1 -- start start 2020-11-20T11:52:27.527 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.510+0000 7efe780fa700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe58004520 0x7efe7009eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:27.527 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.510+0000 7efe780fa700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe700b1ca0 0x7efe700a6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:27.527 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.510+0000 7efe780fa700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe700a6500 0x7efe7013d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:27.528 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.510+0000 7efe780fa700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe700b2120 con 0x7efe700b1ca0 2020-11-20T11:52:27.528 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.510+0000 7efe780fa700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe700a6980 con 0x7efe58004520 2020-11-20T11:52:27.528 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.510+0000 7efe780fa700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe700a74f0 con 0x7efe700a6500 2020-11-20T11:52:27.528 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.510+0000 7efe75695700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe700b1ca0 0x7efe700a6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:27.528 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.510+0000 7efe75e96700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe58004520 0x7efe7009eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:27.529 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.510+0000 7efe75695700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe700b1ca0 0x7efe700a6010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47460/0 (socket says 172.21.15.23:47460) 2020-11-20T11:52:27.529 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.510+0000 7efe75695700 1 -- 172.21.15.23:0/2034056589 learned_addr learned my addr 172.21.15.23:0/2034056589 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:52:27.529 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.510+0000 7efe75e96700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe58004520 0x7efe7009eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:40354/0 (socket says 172.21.15.23:40354) 2020-11-20T11:52:27.529 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.510+0000 7efe76697700 1 --2- 172.21.15.23:0/2034056589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe700a6500 0x7efe7013d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:27.529 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.510+0000 7efe75695700 1 -- 172.21.15.23:0/2034056589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe700a6500 msgr2=0x7efe7013d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:27.530 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.510+0000 7efe75695700 1 --2- 172.21.15.23:0/2034056589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe700a6500 0x7efe7013d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:27.530 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.510+0000 7efe75695700 1 -- 172.21.15.23:0/2034056589 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe58004520 msgr2=0x7efe7009eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:27.530 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.510+0000 7efe75695700 1 --2- 172.21.15.23:0/2034056589 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe58004520 0x7efe7009eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:27.530 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.510+0000 7efe75695700 1 -- 172.21.15.23:0/2034056589 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe700a7790 con 0x7efe700b1ca0 2020-11-20T11:52:27.530 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.510+0000 7efe75e96700 1 --2- 172.21.15.23:0/2034056589 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe58004520 0x7efe7009eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:52:27.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.511+0000 7efe75695700 1 --2- 172.21.15.23:0/2034056589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe700b1ca0 0x7efe700a6010 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7efe58005440 tx=0x7efe60016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:27.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.511+0000 7efe5dffb700 1 -- 172.21.15.23:0/2034056589 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe60012a70 con 0x7efe700b1ca0 2020-11-20T11:52:27.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.511+0000 7efe5dffb700 1 -- 172.21.15.23:0/2034056589 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe60012bd0 con 0x7efe700b1ca0 2020-11-20T11:52:27.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.511+0000 7efe780fa700 1 -- 172.21.15.23:0/2034056589 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe700a7a30 con 0x7efe700b1ca0 2020-11-20T11:52:27.531 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.511+0000 7efe5dffb700 1 -- 172.21.15.23:0/2034056589 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6001d440 con 0x7efe700b1ca0 2020-11-20T11:52:27.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.512+0000 7efe780fa700 1 -- 172.21.15.23:0/2034056589 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe700a5370 con 0x7efe700b1ca0 2020-11-20T11:52:27.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.513+0000 7efe780fa700 1 -- 172.21.15.23:0/2034056589 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe7005c7d0 con 0x7efe700b1ca0 2020-11-20T11:52:27.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.513+0000 7efe5dffb700 1 -- 172.21.15.23:0/2034056589 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7efe60024030 con 0x7efe700b1ca0 2020-11-20T11:52:27.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.513+0000 7efe5dffb700 1 --2- 172.21.15.23:0/2034056589 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efe48056db0 0x7efe48059270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:27.532 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.513+0000 7efe5dffb700 1 -- 172.21.15.23:0/2034056589 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7efe60071260 con 0x7efe700b1ca0 2020-11-20T11:52:27.533 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.513+0000 7efe75e96700 1 --2- 172.21.15.23:0/2034056589 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efe48056db0 0x7efe48059270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:27.533 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.514+0000 7efe75e96700 1 --2- 172.21.15.23:0/2034056589 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efe48056db0 0x7efe48059270 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7efe6c001200 tx=0x7efe6c007040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:27.533 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.517+0000 7efe5dffb700 1 -- 172.21.15.23:0/2034056589 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7efe600465d0 con 0x7efe700b1ca0 2020-11-20T11:52:27.742 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.739+0000 7efe780fa700 1 -- 172.21.15.23:0/2034056589 --> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efe700a7bc0 con 0x7efe48056db0 2020-11-20T11:52:27.749 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.746+0000 7efe5dffb700 1 -- 172.21.15.23:0/2034056589 <== mgr.4105 v2:172.21.15.155:6832/22693 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+106833 (secure 0 0 0) 0x7efe700a7bc0 con 0x7efe48056db0 2020-11-20T11:52:27.752 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:52:27.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.748+0000 7efe780fa700 1 -- 172.21.15.23:0/2034056589 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efe48056db0 msgr2=0x7efe48059270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:27.752 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.748+0000 7efe780fa700 1 --2- 172.21.15.23:0/2034056589 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efe48056db0 0x7efe48059270 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7efe6c001200 tx=0x7efe6c007040).stop 2020-11-20T11:52:27.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.748+0000 7efe780fa700 1 -- 172.21.15.23:0/2034056589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe700b1ca0 msgr2=0x7efe700a6010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:27.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.748+0000 7efe780fa700 1 --2- 172.21.15.23:0/2034056589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe700b1ca0 0x7efe700a6010 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7efe58005440 tx=0x7efe60016000).stop 2020-11-20T11:52:27.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.748+0000 7efe780fa700 1 -- 172.21.15.23:0/2034056589 shutdown_connections 2020-11-20T11:52:27.753 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.748+0000 7efe780fa700 1 --2- 172.21.15.23:0/2034056589 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7efe48056db0 0x7efe48059270 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:27.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.748+0000 7efe780fa700 1 --2- 172.21.15.23:0/2034056589 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7efe700a6500 0x7efe7013d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:27.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.748+0000 7efe780fa700 1 --2- 172.21.15.23:0/2034056589 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7efe58004520 0x7efe7009eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:27.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.748+0000 7efe780fa700 1 --2- 172.21.15.23:0/2034056589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe700b1ca0 0x7efe700a6010 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:27.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.748+0000 7efe780fa700 1 -- 172.21.15.23:0/2034056589 >> 172.21.15.23:0/2034056589 conn(0x7efe700678e0 msgr2=0x7efe700abc10 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:27.754 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.748+0000 7efe780fa700 1 -- 172.21.15.23:0/2034056589 shutdown_connections 2020-11-20T11:52:27.755 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:27.748+0000 7efe780fa700 1 -- 172.21.15.23:0/2034056589 wait complete. 2020-11-20T11:52:27.755 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2020-11-20T11:52:27.765 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":625,"stamp":"2020-11-20T11:52:25.793450+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8259474655,"num_objects":2000,"num_object_clones":0,"num_object_copies":7946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2000,"num_whiteouts":0,"num_read":23,"num_read_kb":61,"num_write":254527,"num_write_kb":14114655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82434,"ondisk_log_size":82434,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":66,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":18940136,"kb_used_data":16275240,"kb_used_omap":0,"kb_used_meta":2656640,"kb_avail":736034584,"statfs":{"total":773094113280,"available":753699414016,"internally_reserved":0,"allocated":16665845760,"data_stored":16664548126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2720399360},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,1,6,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":188,"apply_latency_ms":188,"commit_latency_ns":188000000,"apply_latency_ns":188000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3662102688,"num_objects":-897,"num_object_clones":0,"num_object_copies":-3582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":901,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002551"},"pg_stats":[{"pgid":"4.2","version":"25'15336","reported_seq":"15348","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:21.441768+0000","last_change":"2020-11-20T11:32:25.125195+0000","last_active":"2020-11-20T11:52:21.441768+0000","last_peered":"2020-11-20T11:52:21.441768+0000","last_clean":"2020-11-20T11:52:21.441768+0000","last_became_active":"2020-11-20T11:32:25.124627+0000","last_became_peered":"2020-11-20T11:32:25.124627+0000","last_unstale":"2020-11-20T11:52:21.441768+0000","last_undegraded":"2020-11-20T11:52:21.441768+0000","last_fullsized":"2020-11-20T11:52:21.441768+0000","mapping_epoch":22,"log_start":"25'5309","ondisk_log_start":"25'5309","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":10027,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1145082224,"num_objects":276,"num_object_clones":0,"num_object_copies":1104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33278,"num_write_kb":1823424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"16","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.480586+0000","last_change":"2020-11-20T11:32:19.482604+0000","last_active":"2020-11-20T11:32:25.480586+0000","last_peered":"2020-11-20T11:32:25.480586+0000","last_clean":"2020-11-20T11:32:25.480586+0000","last_became_active":"2020-11-20T11:32:19.482197+0000","last_became_peered":"2020-11-20T11:32:19.482197+0000","last_unstale":"2020-11-20T11:32:25.480586+0000","last_undegraded":"2020-11-20T11:32:25.480586+0000","last_fullsized":"2020-11-20T11:32:25.480586+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"25'29","reported_seq":"42","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:12.129105+0000","last_change":"2020-11-20T11:32:22.098576+0000","last_active":"2020-11-20T11:52:12.129105+0000","last_peered":"2020-11-20T11:52:12.129105+0000","last_clean":"2020-11-20T11:52:12.129105+0000","last_became_active":"2020-11-20T11:32:22.098264+0000","last_became_peered":"2020-11-20T11:32:22.098264+0000","last_unstale":"2020-11-20T11:52:12.129105+0000","last_undegraded":"2020-11-20T11:52:12.129105+0000","last_fullsized":"2020-11-20T11:52:12.129105+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"25'14047","reported_seq":"14059","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:21.579171+0000","last_change":"2020-11-20T11:32:25.479776+0000","last_active":"2020-11-20T11:52:21.579171+0000","last_peered":"2020-11-20T11:52:21.579171+0000","last_clean":"2020-11-20T11:52:21.579171+0000","last_became_active":"2020-11-20T11:32:25.479350+0000","last_became_peered":"2020-11-20T11:32:25.479350+0000","last_unstale":"2020-11-20T11:52:21.579171+0000","last_undegraded":"2020-11-20T11:52:21.579171+0000","last_fullsized":"2020-11-20T11:52:21.579171+0000","mapping_epoch":22,"log_start":"25'3991","ondisk_log_start":"25'3991","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":10056,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":985667824,"num_objects":236,"num_object_clones":0,"num_object_copies":944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31450,"num_write_kb":1652120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"16","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.479488+0000","last_change":"2020-11-20T11:32:18.924328+0000","last_active":"2020-11-20T11:32:25.479488+0000","last_peered":"2020-11-20T11:32:25.479488+0000","last_clean":"2020-11-20T11:32:25.479488+0000","last_became_active":"2020-11-20T11:32:18.923879+0000","last_became_peered":"2020-11-20T11:32:18.923879+0000","last_unstale":"2020-11-20T11:32:25.479488+0000","last_undegraded":"2020-11-20T11:32:25.479488+0000","last_fullsized":"2020-11-20T11:32:25.479488+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"25'1083","reported_seq":"1096","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:12.255599+0000","last_change":"2020-11-20T11:32:22.093559+0000","last_active":"2020-11-20T11:52:12.255599+0000","last_peered":"2020-11-20T11:52:12.255599+0000","last_clean":"2020-11-20T11:52:12.255599+0000","last_became_active":"2020-11-20T11:32:22.093123+0000","last_became_peered":"2020-11-20T11:32:22.093123+0000","last_unstale":"2020-11-20T11:52:12.255599+0000","last_undegraded":"2020-11-20T11:52:12.255599+0000","last_fullsized":"2020-11-20T11:52:12.255599+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":1083,"ondisk_log_size":1083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4554460,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1083,"num_write_kb":4776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"25'14382","reported_seq":"14393","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:21.656818+0000","last_change":"2020-11-20T11:32:25.125052+0000","last_active":"2020-11-20T11:52:21.656818+0000","last_peered":"2020-11-20T11:52:21.656818+0000","last_clean":"2020-11-20T11:52:21.656818+0000","last_became_active":"2020-11-20T11:32:25.124401+0000","last_became_peered":"2020-11-20T11:32:25.124401+0000","last_unstale":"2020-11-20T11:52:21.656818+0000","last_undegraded":"2020-11-20T11:52:21.656818+0000","last_fullsized":"2020-11-20T11:52:21.656818+0000","mapping_epoch":22,"log_start":"25'4285","ondisk_log_start":"25'4285","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":10097,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1132876080,"num_objects":271,"num_object_clones":0,"num_object_copies":1084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31071,"num_write_kb":1732953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"15","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.474417+0000","last_change":"2020-11-20T11:32:18.927388+0000","last_active":"2020-11-20T11:32:25.474417+0000","last_peered":"2020-11-20T11:32:25.474417+0000","last_clean":"2020-11-20T11:32:25.474417+0000","last_became_active":"2020-11-20T11:32:18.926981+0000","last_became_peered":"2020-11-20T11:32:18.926981+0000","last_unstale":"2020-11-20T11:32:25.474417+0000","last_undegraded":"2020-11-20T11:32:25.474417+0000","last_fullsized":"2020-11-20T11:32:25.474417+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"25'13","reported_seq":"26","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:12.252954+0000","last_change":"2020-11-20T11:32:22.088128+0000","last_active":"2020-11-20T11:52:12.252954+0000","last_peered":"2020-11-20T11:52:12.252954+0000","last_clean":"2020-11-20T11:52:12.252954+0000","last_became_active":"2020-11-20T11:32:22.087723+0000","last_became_peered":"2020-11-20T11:32:22.087723+0000","last_unstale":"2020-11-20T11:52:12.252954+0000","last_undegraded":"2020-11-20T11:52:12.252954+0000","last_fullsized":"2020-11-20T11:52:12.252954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"25'13547","reported_seq":"13559","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:24.530203+0000","last_change":"2020-11-20T11:32:25.492514+0000","last_active":"2020-11-20T11:52:24.530203+0000","last_peered":"2020-11-20T11:52:24.530203+0000","last_clean":"2020-11-20T11:52:24.530203+0000","last_became_active":"2020-11-20T11:32:25.491657+0000","last_became_peered":"2020-11-20T11:32:25.491657+0000","last_unstale":"2020-11-20T11:52:24.530203+0000","last_undegraded":"2020-11-20T11:52:24.530203+0000","last_fullsized":"2020-11-20T11:52:24.530203+0000","mapping_epoch":22,"log_start":"25'3448","ondisk_log_start":"25'3448","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":10099,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1019215872,"num_objects":243,"num_object_clones":0,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30687,"num_write_kb":1726809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"16","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.479536+0000","last_change":"2020-11-20T11:32:18.927044+0000","last_active":"2020-11-20T11:32:25.479536+0000","last_peered":"2020-11-20T11:32:25.479536+0000","last_clean":"2020-11-20T11:32:25.479536+0000","last_became_active":"2020-11-20T11:32:18.926711+0000","last_became_peered":"2020-11-20T11:32:18.926711+0000","last_unstale":"2020-11-20T11:32:25.479536+0000","last_undegraded":"2020-11-20T11:32:25.479536+0000","last_fullsized":"2020-11-20T11:32:25.479536+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'2","reported_seq":"20","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.488044+0000","last_change":"2020-11-20T11:32:14.446049+0000","last_active":"2020-11-20T11:32:25.488044+0000","last_peered":"2020-11-20T11:32:25.488044+0000","last_clean":"2020-11-20T11:32:25.488044+0000","last_became_active":"2020-11-20T11:32:14.445477+0000","last_became_peered":"2020-11-20T11:32:14.445477+0000","last_unstale":"2020-11-20T11:32:25.488044+0000","last_undegraded":"2020-11-20T11:32:25.488044+0000","last_fullsized":"2020-11-20T11:32:25.488044+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:10.515989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:10.515989+0000","last_clean_scrub_stamp":"2020-11-20T11:32:10.515989+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"25'4","reported_seq":"20","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:25.210203+0000","last_change":"2020-11-20T11:52:25.210262+0000","last_active":"2020-11-20T11:52:25.210203+0000","last_peered":"2020-11-20T11:52:25.210203+0000","last_clean":"2020-11-20T11:52:25.210203+0000","last_became_active":"2020-11-20T11:32:22.087777+0000","last_became_peered":"2020-11-20T11:32:22.087777+0000","last_unstale":"2020-11-20T11:52:25.210203+0000","last_undegraded":"2020-11-20T11:52:25.210203+0000","last_fullsized":"2020-11-20T11:52:25.210203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'4","last_scrub_stamp":"2020-11-20T11:52:25.210199+0000","last_deep_scrub":"25'4","last_deep_scrub_stamp":"2020-11-20T11:52:25.210199+0000","last_clean_scrub_stamp":"2020-11-20T11:52:25.210199+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"25'12290","reported_seq":"12302","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:22.580209+0000","last_change":"2020-11-20T11:32:25.491667+0000","last_active":"2020-11-20T11:52:22.580209+0000","last_peered":"2020-11-20T11:52:22.580209+0000","last_clean":"2020-11-20T11:52:22.580209+0000","last_became_active":"2020-11-20T11:32:25.491100+0000","last_became_peered":"2020-11-20T11:32:25.491100+0000","last_unstale":"2020-11-20T11:52:22.580209+0000","last_undegraded":"2020-11-20T11:52:22.580209+0000","last_fullsized":"2020-11-20T11:52:22.580209+0000","mapping_epoch":22,"log_start":"25'2252","ondisk_log_start":"25'2252","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":10038,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":952130560,"num_objects":228,"num_object_clones":0,"num_object_copies":912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28100,"num_write_kb":1700287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"15","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.488045+0000","last_change":"2020-11-20T11:32:18.924600+0000","last_active":"2020-11-20T11:32:25.488045+0000","last_peered":"2020-11-20T11:32:25.488045+0000","last_clean":"2020-11-20T11:32:25.488045+0000","last_became_active":"2020-11-20T11:32:18.924324+0000","last_became_peered":"2020-11-20T11:32:18.924324+0000","last_unstale":"2020-11-20T11:32:25.488045+0000","last_undegraded":"2020-11-20T11:32:25.488045+0000","last_fullsized":"2020-11-20T11:32:25.488045+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"25'56","reported_seq":"92","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:12.255012+0000","last_change":"2020-11-20T11:32:22.088716+0000","last_active":"2020-11-20T11:52:12.255012+0000","last_peered":"2020-11-20T11:52:12.255012+0000","last_clean":"2020-11-20T11:52:12.255012+0000","last_became_active":"2020-11-20T11:32:22.088414+0000","last_became_peered":"2020-11-20T11:32:22.088414+0000","last_unstale":"2020-11-20T11:52:12.255012+0000","last_undegraded":"2020-11-20T11:52:12.255012+0000","last_fullsized":"2020-11-20T11:52:12.255012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":47377,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":61,"num_write":58,"num_write_kb":143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"25'15030","reported_seq":"15042","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:22.439289+0000","last_change":"2020-11-20T11:32:25.491507+0000","last_active":"2020-11-20T11:52:22.439289+0000","last_peered":"2020-11-20T11:52:22.439289+0000","last_clean":"2020-11-20T11:52:22.439289+0000","last_became_active":"2020-11-20T11:32:25.490917+0000","last_became_peered":"2020-11-20T11:32:25.490917+0000","last_unstale":"2020-11-20T11:52:22.439289+0000","last_undegraded":"2020-11-20T11:52:22.439289+0000","last_fullsized":"2020-11-20T11:52:22.439289+0000","mapping_epoch":22,"log_start":"25'4951","ondisk_log_start":"25'4951","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":10079,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":994050048,"num_objects":237,"num_object_clones":0,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33601,"num_write_kb":1816287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"15","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.488121+0000","last_change":"2020-11-20T11:32:19.482719+0000","last_active":"2020-11-20T11:32:25.488121+0000","last_peered":"2020-11-20T11:32:25.488121+0000","last_clean":"2020-11-20T11:32:25.488121+0000","last_became_active":"2020-11-20T11:32:19.482359+0000","last_became_peered":"2020-11-20T11:32:19.482359+0000","last_unstale":"2020-11-20T11:32:25.488121+0000","last_undegraded":"2020-11-20T11:32:25.488121+0000","last_fullsized":"2020-11-20T11:32:25.488121+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"25'558","reported_seq":"571","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:10.276646+0000","last_change":"2020-11-20T11:32:22.093099+0000","last_active":"2020-11-20T11:52:10.276646+0000","last_peered":"2020-11-20T11:52:10.276646+0000","last_clean":"2020-11-20T11:52:10.276646+0000","last_became_active":"2020-11-20T11:32:22.092619+0000","last_became_peered":"2020-11-20T11:32:22.092619+0000","last_unstale":"2020-11-20T11:52:10.276646+0000","last_undegraded":"2020-11-20T11:52:10.276646+0000","last_fullsized":"2020-11-20T11:52:10.276646+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":558,"ondisk_log_size":558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4194947,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":555,"num_write_kb":4297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"25'14018","reported_seq":"14030","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:25.113776+0000","last_change":"2020-11-20T11:32:25.492150+0000","last_active":"2020-11-20T11:52:25.113776+0000","last_peered":"2020-11-20T11:52:25.113776+0000","last_clean":"2020-11-20T11:52:25.113776+0000","last_became_active":"2020-11-20T11:32:25.491099+0000","last_became_peered":"2020-11-20T11:32:25.491099+0000","last_unstale":"2020-11-20T11:52:25.113776+0000","last_undegraded":"2020-11-20T11:52:25.113776+0000","last_fullsized":"2020-11-20T11:52:25.113776+0000","mapping_epoch":22,"log_start":"25'3929","ondisk_log_start":"25'3929","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":10089,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1010827264,"num_objects":241,"num_object_clones":0,"num_object_copies":964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31649,"num_write_kb":1743046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"25'10","reported_seq":"23","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:09.920224+0000","last_change":"2020-11-20T11:32:22.461697+0000","last_active":"2020-11-20T11:52:09.920224+0000","last_peered":"2020-11-20T11:52:09.920224+0000","last_clean":"2020-11-20T11:52:09.920224+0000","last_became_active":"2020-11-20T11:32:22.461331+0000","last_became_peered":"2020-11-20T11:32:22.461331+0000","last_unstale":"2020-11-20T11:52:09.920224+0000","last_undegraded":"2020-11-20T11:52:09.920224+0000","last_fullsized":"2020-11-20T11:52:09.920224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"16","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.480381+0000","last_change":"2020-11-20T11:32:18.923886+0000","last_active":"2020-11-20T11:32:25.480381+0000","last_peered":"2020-11-20T11:32:25.480381+0000","last_clean":"2020-11-20T11:32:25.480381+0000","last_became_active":"2020-11-20T11:32:18.923448+0000","last_became_peered":"2020-11-20T11:32:18.923448+0000","last_unstale":"2020-11-20T11:32:25.480381+0000","last_undegraded":"2020-11-20T11:32:25.480381+0000","last_fullsized":"2020-11-20T11:32:25.480381+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"25'13930","reported_seq":"13942","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:23.045454+0000","last_change":"2020-11-20T11:32:25.491585+0000","last_active":"2020-11-20T11:52:23.045454+0000","last_peered":"2020-11-20T11:52:23.045454+0000","last_clean":"2020-11-20T11:52:23.045454+0000","last_became_active":"2020-11-20T11:32:25.491038+0000","last_became_peered":"2020-11-20T11:32:25.491038+0000","last_unstale":"2020-11-20T11:52:23.045454+0000","last_undegraded":"2020-11-20T11:52:23.045454+0000","last_fullsized":"2020-11-20T11:52:23.045454+0000","mapping_epoch":22,"log_start":"25'3877","ondisk_log_start":"25'3877","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":10053,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1010827264,"num_objects":241,"num_object_clones":0,"num_object_copies":964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32583,"num_write_kb":1910315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"15","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.473625+0000","last_change":"2020-11-20T11:32:18.924052+0000","last_active":"2020-11-20T11:32:25.473625+0000","last_peered":"2020-11-20T11:32:25.473625+0000","last_clean":"2020-11-20T11:32:25.473625+0000","last_became_active":"2020-11-20T11:32:18.923668+0000","last_became_peered":"2020-11-20T11:32:18.923668+0000","last_unstale":"2020-11-20T11:32:25.473625+0000","last_undegraded":"2020-11-20T11:32:25.473625+0000","last_fullsized":"2020-11-20T11:32:25.473625+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"25'141","reported_seq":"154","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:12.255022+0000","last_change":"2020-11-20T11:32:22.093019+0000","last_active":"2020-11-20T11:52:12.255022+0000","last_peered":"2020-11-20T11:52:12.255022+0000","last_clean":"2020-11-20T11:52:12.255022+0000","last_became_active":"2020-11-20T11:32:22.092604+0000","last_became_peered":"2020-11-20T11:32:22.092604+0000","last_unstale":"2020-11-20T11:52:12.255022+0000","last_undegraded":"2020-11-20T11:52:12.255022+0000","last_fullsized":"2020-11-20T11:52:12.255022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":141,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":355,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":8250677136,"num_objects":1973,"num_object_clones":0,"num_object_copies":7892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":252419,"num_write_kb":14105241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16413310976,"data_stored":16413310976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80538,"ondisk_log_size":80538,"up":32,"acting":32,"num_store_stats":8},{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":8797519,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":23,"num_read_kb":61,"num_write":2106,"num_write_kb":9404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17653760,"data_stored":17595038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1894,"ondisk_log_size":1894,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":6,"up_from":14,"seq":60129542389,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1693900,"kb_used_data":1433860,"kb_used_omap":0,"kb_used_meta":260032,"kb_avail":92677940,"statfs":{"total":96636764160,"available":94902210560,"internally_reserved":0,"allocated":1468272640,"data_stored":1468117776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":266272768},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 11:51:42 2020","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.0109999999999999,"15min":1.4450000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.45700000000000002},"max":{"1min":3.677,"5min":9.6129999999999995,"15min":22.257999999999999},"last":1.4390000000000001},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.96999999999999997,"15min":1.4079999999999999},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":2.1379999999999999,"5min":6.3099999999999996,"15min":19.855},"last":1.1910000000000001}]},{"osd":1,"last update":"Fri Nov 20 11:51:42 2020","interfaces":[{"interface":"back","average":{"1min":5.4039999999999999,"5min":8.8640000000000008,"15min":6.9290000000000003},"min":{"1min":0.70999999999999996,"5min":0.504,"15min":0.504},"max":{"1min":23.738,"5min":39.268999999999998,"15min":42.552999999999997},"last":14.787000000000001},{"interface":"front","average":{"1min":6.1950000000000003,"5min":9.2330000000000005,"15min":6.8099999999999996},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":39.792999999999999,"5min":39.792999999999999,"15min":58.981000000000002},"last":14.484}]},{"osd":2,"last update":"Fri Nov 20 11:51:42 2020","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.194,"15min":1.2050000000000001},"min":{"1min":0.65700000000000003,"5min":0.57999999999999996,"15min":0.57799999999999996},"max":{"1min":3.7090000000000001,"5min":6.0880000000000001,"15min":7.4790000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0640000000000001,"15min":1.135},"min":{"1min":0.63200000000000001,"5min":0.61399999999999999,"15min":0.54200000000000004},"max":{"1min":2.2879999999999998,"5min":8.1519999999999992,"15min":8.1519999999999992},"last":0.82899999999999996}]},{"osd":3,"last update":"Fri Nov 20 11:51:42 2020","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.2470000000000001,"15min":1.4870000000000001},"min":{"1min":0.628,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":3.2400000000000002,"5min":9.7330000000000005,"15min":11.287000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.1870000000000001,"15min":1.4770000000000001},"min":{"1min":0.65800000000000003,"5min":0.65300000000000002,"15min":0.61099999999999999},"max":{"1min":5.3170000000000002,"5min":5.3170000000000002,"15min":11.256},"last":0.755}]},{"osd":4,"last update":"Fri Nov 20 11:51:42 2020","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":1.109,"15min":1.325},"min":{"1min":0.57999999999999996,"5min":0.57699999999999996,"15min":0.53500000000000003},"max":{"1min":1.319,"5min":9.1150000000000002,"15min":10.087999999999999},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.2649999999999999,"15min":1.3540000000000001},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.57399999999999995},"max":{"1min":1.655,"5min":8.9309999999999992,"15min":9.468},"last":0.88700000000000001}]},{"osd":5,"last update":"Fri Nov 20 11:51:42 2020","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.135,"15min":1.274},"min":{"1min":0.68000000000000005,"5min":0.621,"15min":0.59499999999999997},"max":{"1min":1.52,"5min":7.8780000000000001,"15min":7.8780000000000001},"last":1.081},{"interface":"front","average":{"1min":1.097,"5min":1.2609999999999999,"15min":1.494},"min":{"1min":0.69999999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.532,"5min":9.4779999999999998,"15min":9.4779999999999998},"last":1.042}]},{"osd":7,"last update":"Fri Nov 20 11:51:42 2020","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.214,"15min":1.3680000000000001},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.70799999999999996},"max":{"1min":1.7569999999999999,"5min":4.3289999999999997,"15min":6.8760000000000003},"last":1.002},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.258,"15min":1.423},"min":{"1min":0.79000000000000004,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.5549999999999999,"5min":8.0239999999999991,"15min":8.0239999999999991},"last":0.95399999999999996}]}]},{"osd":3,"up_from":14,"seq":60129542390,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2245032,"kb_used_data":1927648,"kb_used_omap":0,"kb_used_meta":317376,"kb_avail":92126808,"statfs":{"total":96636764160,"available":94337851392,"internally_reserved":0,"allocated":1973911552,"data_stored":1973741039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":324993024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 11:52:00 2020","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.98399999999999999,"15min":1.6080000000000001},"min":{"1min":0.48199999999999998,"5min":0.47099999999999997,"15min":0.45400000000000001},"max":{"1min":1.5760000000000001,"5min":6.2949999999999999,"15min":42.424999999999997},"last":1.806},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.93999999999999995,"15min":1.728},"min":{"1min":0.498,"5min":0.49299999999999999,"15min":0.46700000000000003},"max":{"1min":2.1859999999999999,"5min":3.6600000000000001,"15min":41.997999999999998},"last":0.627}]},{"osd":1,"last update":"Fri Nov 20 11:52:05 2020","interfaces":[{"interface":"back","average":{"1min":5.0510000000000002,"5min":10.712999999999999,"15min":7.5990000000000002},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.47799999999999998},"max":{"1min":14.266999999999999,"5min":65.120000000000005,"15min":65.120000000000005},"last":3.1190000000000002},{"interface":"front","average":{"1min":5.3579999999999997,"5min":11.06,"15min":7.9379999999999997},"min":{"1min":0.79200000000000004,"5min":0.78400000000000003,"15min":0.69599999999999995},"max":{"1min":14.33,"5min":65.222999999999999,"15min":65.222999999999999},"last":3.7040000000000002}]},{"osd":2,"last update":"Fri Nov 20 11:52:00 2020","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.0609999999999999,"15min":1.1759999999999999},"min":{"1min":0.57499999999999996,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":2.1699999999999999,"5min":5.2380000000000004,"15min":22.370999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.2929999999999999,"15min":1.4399999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.52000000000000002},"max":{"1min":1.954,"5min":5.2130000000000001,"15min":8.8320000000000007},"last":2.0329999999999999}]},{"osd":4,"last update":"Fri Nov 20 11:52:00 2020","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.786,"15min":1.907},"min":{"1min":0.70799999999999996,"5min":0.70399999999999996,"15min":0.64100000000000001},"max":{"1min":1.8120000000000001,"5min":46.268000000000001,"15min":46.268000000000001},"last":1.105},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3009999999999999,"15min":1.6399999999999999},"min":{"1min":0.91200000000000003,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.306,"5min":5.3659999999999997,"15min":40.478000000000002},"last":0.94399999999999995}]},{"osd":5,"last update":"Fri Nov 20 11:52:00 2020","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.9279999999999999,"15min":1.962},"min":{"1min":0.76400000000000001,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.8939999999999999,"5min":46.314999999999998,"15min":46.314999999999998},"last":3.246},{"interface":"front","average":{"1min":1.21,"5min":1.369,"15min":1.9099999999999999},"min":{"1min":0.83499999999999996,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.9430000000000001,"5min":4.141,"15min":34.972999999999999},"last":1.3540000000000001}]},{"osd":6,"last update":"Fri Nov 20 11:52:00 2020","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.9890000000000001,"15min":1.782},"min":{"1min":1.0149999999999999,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":3.0259999999999998,"5min":46.219000000000001,"15min":46.219000000000001},"last":1.071},{"interface":"front","average":{"1min":1.4339999999999999,"5min":2.0680000000000001,"15min":1.9279999999999999},"min":{"1min":0.996,"5min":0.871,"15min":0.82699999999999996},"max":{"1min":2.085,"5min":46.469000000000001,"15min":46.469000000000001},"last":3.3679999999999999}]},{"osd":7,"last update":"Fri Nov 20 11:52:00 2020","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":2.0619999999999998,"15min":2.1070000000000002},"min":{"1min":1.016,"5min":0.91600000000000004,"15min":0.86799999999999999},"max":{"1min":2.069,"5min":46.432000000000002,"15min":46.432000000000002},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.587,"5min":2.129,"15min":1.9690000000000001},"min":{"1min":1.097,"5min":0.92200000000000004,"15min":0.91500000000000004},"max":{"1min":2.9169999999999998,"5min":46.348999999999997,"15min":46.348999999999997},"last":1.1910000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575092,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1817972,"kb_used_data":1551020,"kb_used_omap":0,"kb_used_meta":266944,"kb_avail":92553868,"statfs":{"total":96636764160,"available":94775160832,"internally_reserved":0,"allocated":1588244480,"data_stored":1588083453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":273350656},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.99199999999999999,"15min":1.6319999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":4.3280000000000003,"5min":9.1920000000000002,"15min":41.939999999999998},"last":2.7799999999999998},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.048,"15min":1.7430000000000001},"min":{"1min":0.57499999999999996,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":4.9480000000000004,"5min":5.5869999999999997,"15min":42.704999999999998},"last":1.0660000000000001}]},{"osd":1,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":9.5839999999999996,"15min":7.2359999999999998},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.52500000000000002},"max":{"1min":21.244,"5min":39.244999999999997,"15min":76.847999999999999},"last":22.484000000000002},{"interface":"front","average":{"1min":7.0979999999999999,"5min":9.9730000000000008,"15min":7.2450000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53200000000000003},"max":{"1min":20.898,"5min":39.472999999999999,"15min":164.745},"last":22.210000000000001}]},{"osd":2,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":1.0600000000000001,"15min":2.2549999999999999},"min":{"1min":0.621,"5min":0.53200000000000003,"15min":0.52800000000000002},"max":{"1min":2.02,"5min":4.681,"15min":269.16800000000001},"last":3.0150000000000001},{"interface":"front","average":{"1min":0.92800000000000005,"5min":1.147,"15min":2.2269999999999999},"min":{"1min":0.69699999999999995,"5min":0.60599999999999998,"15min":0.56299999999999994},"max":{"1min":1.1839999999999999,"5min":5.6289999999999996,"15min":262.65300000000002},"last":1.2490000000000001}]},{"osd":3,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.2010000000000001,"15min":1.742},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":3.427,"5min":10.930999999999999,"15min":40.585000000000001},"last":0.97899999999999998},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1890000000000001,"15min":1.6850000000000001},"min":{"1min":0.75,"5min":0.68200000000000005,"15min":0.59799999999999998},"max":{"1min":1.79,"5min":6.016,"15min":40.259999999999998},"last":2.9020000000000001}]},{"osd":5,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.95299999999999996,"15min":1.3180000000000001},"min":{"1min":0.60399999999999998,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.4079999999999999,"5min":3.7189999999999999,"15min":18.222999999999999},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.2,"15min":1.3560000000000001},"min":{"1min":0.67800000000000005,"5min":0.61299999999999999,"15min":0.60299999999999998},"max":{"1min":1.95,"5min":6.1840000000000002,"15min":8.8239999999999998},"last":3.3879999999999999}]},{"osd":6,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.048,"15min":1.3640000000000001},"min":{"1min":0.78500000000000003,"5min":0.67800000000000005,"15min":0.63300000000000001},"max":{"1min":1.746,"5min":2.298,"15min":7.0819999999999999},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0609999999999999,"15min":1.359},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.67000000000000004},"max":{"1min":1.5369999999999999,"5min":1.7050000000000001,"15min":9.3200000000000003},"last":1.044}]},{"osd":7,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.1970000000000001,"15min":1.417},"min":{"1min":0.85999999999999999,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.8600000000000001,"5min":2.4390000000000001,"15min":6.9089999999999998},"last":3.9620000000000002},{"interface":"front","average":{"1min":1.198,"5min":1.2070000000000001,"15min":1.573},"min":{"1min":0.98999999999999999,"5min":0.82499999999999996,"15min":0.77300000000000002},"max":{"1min":1.712,"5min":2.8700000000000001,"15min":9.4380000000000006},"last":1.7869999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575092,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3042572,"kb_used_data":2628484,"kb_used_omap":0,"kb_used_meta":414080,"kb_avail":91329268,"statfs":{"total":96636764160,"available":93521170432,"internally_reserved":0,"allocated":2691567616,"data_stored":2691406589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":424017920},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,3,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":6.1040000000000001,"5min":11.039999999999999,"15min":8.218},"min":{"1min":0.57499999999999996,"5min":0.497,"15min":0.47099999999999997},"max":{"1min":28.193000000000001,"5min":47.369,"15min":83.134},"last":1.131},{"interface":"front","average":{"1min":8.7210000000000001,"5min":11.254,"15min":8.5129999999999999},"min":{"1min":0.64500000000000002,"5min":0.51400000000000001,"15min":0.50700000000000001},"max":{"1min":39.725999999999999,"5min":47.856999999999999,"15min":82.665000000000006},"last":1.085}]},{"osd":2,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.95599999999999996,"15min":1.393},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.46700000000000003},"max":{"1min":1.901,"5min":4.7800000000000002,"15min":19.231000000000002},"last":5.5789999999999997},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.2310000000000001,"15min":1.6659999999999999},"min":{"1min":0.52300000000000002,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":6.5810000000000004,"5min":11.981,"15min":19.974},"last":1.464}]},{"osd":3,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.139,"15min":1.6910000000000001},"min":{"1min":0.61599999999999999,"5min":0.51300000000000001,"15min":0.51200000000000001},"max":{"1min":2.7229999999999999,"5min":6.3440000000000003,"15min":41.634},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.157,"5min":1.3240000000000001,"15min":2.0289999999999999},"min":{"1min":0.63400000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.657,"5min":6.3380000000000001,"15min":44.423000000000002},"last":26.803000000000001}]},{"osd":4,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.258,"15min":1.8240000000000001},"min":{"1min":0.80100000000000005,"5min":0.73499999999999999,"15min":0.68899999999999995},"max":{"1min":1.782,"5min":4.3300000000000001,"15min":19.471},"last":5.6989999999999998},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.319,"15min":1.665},"min":{"1min":0.91100000000000003,"5min":0.68200000000000005,"15min":0.68100000000000005},"max":{"1min":2.1619999999999999,"5min":6.7450000000000001,"15min":19.809000000000001},"last":1.5589999999999999}]},{"osd":5,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.3460000000000001,"15min":1.889},"min":{"1min":0.91300000000000003,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.7809999999999999,"5min":4.1879999999999997,"15min":20.457000000000001},"last":5.9260000000000002},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.425,"15min":1.853},"min":{"1min":0.82799999999999996,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":2.3839999999999999,"5min":6.633,"15min":20.565999999999999},"last":1.3580000000000001}]},{"osd":6,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.4630000000000001,"15min":1.8979999999999999},"min":{"1min":1.0900000000000001,"5min":0.77200000000000002,"15min":0.71599999999999997},"max":{"1min":2.254,"5min":6.5110000000000001,"15min":20.699000000000002},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.4890000000000001,"15min":1.895},"min":{"1min":1.022,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":2.0670000000000002,"5min":6.6219999999999999,"15min":20.609999999999999},"last":4.6929999999999996}]},{"osd":7,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.587,"15min":2.0489999999999999},"min":{"1min":1.119,"5min":0.90400000000000003,"15min":0.88400000000000001},"max":{"1min":2.1970000000000001,"5min":6.4950000000000001,"15min":20.495000000000001},"last":4.8230000000000004},{"interface":"front","average":{"1min":1.518,"5min":1.5149999999999999,"15min":2.1829999999999998},"min":{"1min":1.0669999999999999,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":2.0190000000000001,"5min":5.0179999999999998,"15min":20.559000000000001},"last":8.0760000000000005}]}]},{"osd":7,"up_from":13,"seq":55834575092,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2984348,"kb_used_data":2562324,"kb_used_omap":0,"kb_used_meta":422016,"kb_avail":91387492,"statfs":{"total":96636764160,"available":93580791808,"internally_reserved":0,"allocated":2623819776,"data_stored":2623657295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":432144384},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 11:51:56 2020","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.92600000000000005,"15min":1.647},"min":{"1min":0.47499999999999998,"5min":0.44900000000000001,"15min":0.41799999999999998},"max":{"1min":1.2809999999999999,"5min":5.8170000000000002,"15min":43.341999999999999},"last":16.305},{"interface":"front","average":{"1min":0.872,"5min":0.96199999999999997,"15min":1.845},"min":{"1min":0.52100000000000002,"5min":0.49099999999999999,"15min":0.47299999999999998},"max":{"1min":1.7929999999999999,"5min":4.2809999999999997,"15min":42.759},"last":16.289999999999999}]},{"osd":1,"last update":"Fri Nov 20 11:51:56 2020","interfaces":[{"interface":"back","average":{"1min":7.3440000000000003,"5min":10.259,"15min":7.4269999999999996},"min":{"1min":0.52300000000000002,"5min":0.52100000000000002,"15min":0.51800000000000002},"max":{"1min":36.731000000000002,"5min":45.369999999999997,"15min":47.488999999999997},"last":5.2190000000000003},{"interface":"front","average":{"1min":7.0300000000000002,"5min":10.272,"15min":7.375},"min":{"1min":0.71799999999999997,"5min":0.69099999999999995,"15min":0.55800000000000005},"max":{"1min":36.750999999999998,"5min":45.343000000000004,"15min":47.521999999999998},"last":5.2519999999999998}]},{"osd":2,"last update":"Fri Nov 20 11:51:56 2020","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":1.0620000000000001,"15min":1.2170000000000001},"min":{"1min":0.58199999999999996,"5min":0.55900000000000005,"15min":0.53400000000000003},"max":{"1min":1.6799999999999999,"5min":3.9199999999999999,"15min":9.2040000000000006},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.96599999999999997,"15min":1.3280000000000001},"min":{"1min":0.59499999999999997,"5min":0.58999999999999997,"15min":0.56200000000000006},"max":{"1min":1.2669999999999999,"5min":4.1859999999999999,"15min":43.755000000000003},"last":0.57499999999999996}]},{"osd":3,"last update":"Fri Nov 20 11:51:56 2020","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.206,"15min":1.837},"min":{"1min":0.79300000000000004,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.7549999999999999,"5min":4.3280000000000003,"15min":43.146999999999998},"last":9.2349999999999994},{"interface":"front","average":{"1min":1.175,"5min":1.224,"15min":1.7849999999999999},"min":{"1min":0.79300000000000004,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":2.7789999999999999,"5min":4.3419999999999996,"15min":44.014000000000003},"last":9.4039999999999999}]},{"osd":4,"last update":"Fri Nov 20 11:51:56 2020","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":0.99299999999999999,"15min":1.2909999999999999},"min":{"1min":0.67700000000000005,"5min":0.60699999999999998,"15min":0.59899999999999998},"max":{"1min":1.643,"5min":3.3460000000000001,"15min":9.9469999999999992},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.04,"5min":0.96399999999999997,"15min":1.417},"min":{"1min":0.746,"5min":0.57499999999999996,"15min":0.57199999999999995},"max":{"1min":1.4379999999999999,"5min":1.8520000000000001,"15min":43.890999999999998},"last":0.95899999999999996}]},{"osd":5,"last update":"Fri Nov 20 11:51:56 2020","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.087,"15min":1.585},"min":{"1min":0.66900000000000004,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":2.327,"5min":4.2370000000000001,"15min":47.914999999999999},"last":0.88400000000000001},{"interface":"front","average":{"1min":1.298,"5min":1.2030000000000001,"15min":1.5269999999999999},"min":{"1min":0.88100000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":2.3530000000000002,"5min":7.1630000000000003,"15min":43.228000000000002},"last":1.21}]},{"osd":6,"last update":"Fri Nov 20 11:51:56 2020","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.1459999999999999,"15min":1.583},"min":{"1min":0.72699999999999998,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.895,"5min":7.0659999999999998,"15min":47.780000000000001},"last":1.079},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.2190000000000001,"15min":1.728},"min":{"1min":0.84799999999999998,"5min":0.76800000000000002,"15min":0.73199999999999998},"max":{"1min":1.958,"5min":3.4620000000000002,"15min":49.289999999999999},"last":1.0449999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575093,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2421876,"kb_used_data":2085356,"kb_used_omap":0,"kb_used_meta":336512,"kb_avail":91949964,"statfs":{"total":96636764160,"available":94156763136,"internally_reserved":0,"allocated":2135404544,"data_stored":2135241648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":344588288},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 11:51:41 2020","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.96099999999999997,"15min":1.657},"min":{"1min":0.47999999999999998,"5min":0.46700000000000003,"15min":0.40300000000000002},"max":{"1min":1.0429999999999999,"5min":5.1299999999999999,"15min":66.593999999999994},"last":11.228999999999999},{"interface":"front","average":{"1min":1.1279999999999999,"5min":0.96999999999999997,"15min":1.6499999999999999},"min":{"1min":0.49199999999999999,"5min":0.47899999999999998,"15min":0.45600000000000002},"max":{"1min":5.6239999999999997,"5min":5.6239999999999997,"15min":64.793000000000006},"last":11.215}]},{"osd":1,"last update":"Fri Nov 20 11:51:49 2020","interfaces":[{"interface":"back","average":{"1min":4.3719999999999999,"5min":12.068,"15min":7.2839999999999998},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.53300000000000003},"max":{"1min":16.989000000000001,"5min":71.489000000000004,"15min":71.489000000000004},"last":14.621},{"interface":"front","average":{"1min":4.3540000000000001,"5min":10.722,"15min":6.4649999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":17.050999999999998,"5min":72.109999999999999,"15min":72.109999999999999},"last":14.324}]},{"osd":2,"last update":"Fri Nov 20 11:51:49 2020","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":1.056,"15min":1.1699999999999999},"min":{"1min":0.63900000000000001,"5min":0.54900000000000004,"15min":0.53900000000000003},"max":{"1min":1.4410000000000001,"5min":4.9690000000000003,"15min":5.0490000000000004},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.1160000000000001,"15min":1.302},"min":{"1min":0.61299999999999999,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.496,"5min":4.9909999999999997,"15min":10.227},"last":0.84799999999999998}]},{"osd":3,"last update":"Fri Nov 20 11:51:49 2020","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":1.0820000000000001,"15min":1.5289999999999999},"min":{"1min":0.63300000000000001,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.431,"5min":4.0579999999999998,"15min":64.049999999999997},"last":0.82399999999999995},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.2110000000000001,"15min":1.4810000000000001},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.59299999999999997},"max":{"1min":3.5059999999999998,"5min":7.7110000000000003,"15min":63.718000000000004},"last":0.71599999999999997}]},{"osd":4,"last update":"Fri Nov 20 11:51:41 2020","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.99299999999999999,"15min":1.2010000000000001},"min":{"1min":0.65200000000000002,"5min":0.59299999999999997,"15min":0.56299999999999994},"max":{"1min":1.359,"5min":6.907,"15min":10.092000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":1.073,"5min":1.103,"15min":1.25},"min":{"1min":0.73499999999999999,"5min":0.66000000000000003,"15min":0.58499999999999996},"max":{"1min":1.5329999999999999,"5min":4.3280000000000003,"15min":6.1070000000000002},"last":0.91100000000000003}]},{"osd":6,"last update":"Fri Nov 20 11:51:49 2020","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.2090000000000001,"15min":1.456},"min":{"1min":0.76100000000000001,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.8120000000000001,"5min":6.3579999999999997,"15min":10.818},"last":0.75900000000000001},{"interface":"front","average":{"1min":1.216,"5min":1.218,"15min":1.347},"min":{"1min":0.81499999999999995,"5min":0.76700000000000002,"15min":0.68700000000000006},"max":{"1min":1.7929999999999999,"5min":4.1879999999999997,"15min":6.3280000000000003},"last":0.98399999999999999}]},{"osd":7,"last update":"Fri Nov 20 11:51:41 2020","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.1599999999999999,"15min":1.3},"min":{"1min":0.90700000000000003,"5min":0.73199999999999998,"15min":0.70599999999999996},"max":{"1min":1.5429999999999999,"5min":3.5539999999999998,"15min":5.2359999999999998},"last":0.94799999999999995},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.337,"15min":1.536},"min":{"1min":0.94899999999999995,"5min":0.876,"15min":0.78400000000000003},"max":{"1min":1.9490000000000001,"5min":4.5739999999999998,"15min":7.1929999999999996},"last":1.238}]}]},{"osd":1,"up_from":14,"seq":60129542385,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3680516,"kb_used_data":3189244,"kb_used_omap":0,"kb_used_meta":483072,"kb_avail":90691324,"statfs":{"total":96636764160,"available":92867915776,"internally_reserved":0,"allocated":3265785856,"data_stored":3265623443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":494665728},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,3,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":72,"apply_latency_ms":72,"commit_latency_ns":72000000,"apply_latency_ns":72000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 11:51:46 2020","interfaces":[{"interface":"back","average":{"1min":4.8540000000000001,"5min":5.6390000000000002,"15min":5.194},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.45900000000000002},"max":{"1min":24.831,"5min":70.593000000000004,"15min":70.593000000000004},"last":30.748000000000001},{"interface":"front","average":{"1min":2.5369999999999999,"5min":4.7039999999999997,"15min":5.4210000000000003},"min":{"1min":0.70399999999999996,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":15.93,"5min":55.527000000000001,"15min":55.527000000000001},"last":30.878}]},{"osd":2,"last update":"Fri Nov 20 11:51:46 2020","interfaces":[{"interface":"back","average":{"1min":2.8199999999999998,"5min":3.9830000000000001,"15min":3.782},"min":{"1min":0.66700000000000004,"5min":0.52600000000000002,"15min":0.47699999999999998},"max":{"1min":15.91,"5min":51.380000000000003,"15min":51.380000000000003},"last":0.94499999999999995},{"interface":"front","average":{"1min":5.569,"5min":8.7390000000000008,"15min":6.1740000000000004},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.56499999999999995},"max":{"1min":25.695,"5min":70.781999999999996,"15min":70.781999999999996},"last":2.867}]},{"osd":3,"last update":"Fri Nov 20 11:51:46 2020","interfaces":[{"interface":"back","average":{"1min":6.0220000000000002,"5min":9.3510000000000009,"15min":6.8819999999999997},"min":{"1min":0.748,"5min":0.748,"15min":0.61799999999999999},"max":{"1min":25.109999999999999,"5min":70.700999999999993,"15min":70.700999999999993},"last":2.609},{"interface":"front","average":{"1min":6.0430000000000001,"5min":6.5910000000000002,"15min":5.6349999999999998},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.629},"max":{"1min":23.783000000000001,"5min":56.049999999999997,"15min":56.049999999999997},"last":2.8180000000000001}]},{"osd":4,"last update":"Fri Nov 20 11:51:46 2020","interfaces":[{"interface":"back","average":{"1min":5.7960000000000003,"5min":6.4189999999999996,"15min":5.165},"min":{"1min":0.90200000000000002,"5min":0.74299999999999999,"15min":0.73799999999999999},"max":{"1min":21.149999999999999,"5min":55.652000000000001,"15min":55.652000000000001},"last":2.5470000000000002},{"interface":"front","average":{"1min":4.5759999999999996,"5min":5.2859999999999996,"15min":4.6829999999999998},"min":{"1min":0.91300000000000003,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":23.812000000000001,"5min":51.926000000000002,"15min":51.926000000000002},"last":4.218}]},{"osd":5,"last update":"Fri Nov 20 11:51:46 2020","interfaces":[{"interface":"back","average":{"1min":6.1139999999999999,"5min":8.0169999999999995,"15min":5.8099999999999996},"min":{"1min":0.996,"5min":0.72199999999999998,"15min":0.69999999999999996},"max":{"1min":23.952999999999999,"5min":52.07,"15min":52.07},"last":4.3689999999999998},{"interface":"front","average":{"1min":7.0149999999999997,"5min":9.0860000000000003,"15min":6.7770000000000001},"min":{"1min":0.875,"5min":0.875,"15min":0.80800000000000005},"max":{"1min":24.981000000000002,"5min":52.262999999999998,"15min":52.262999999999998},"last":2.7389999999999999}]},{"osd":6,"last update":"Fri Nov 20 11:51:46 2020","interfaces":[{"interface":"back","average":{"1min":6.1840000000000002,"5min":7.4880000000000004,"15min":6.3049999999999997},"min":{"1min":1.167,"5min":1.087,"15min":0.88500000000000001},"max":{"1min":21.015999999999998,"5min":55.228000000000002,"15min":55.228000000000002},"last":2.6869999999999998},{"interface":"front","average":{"1min":7.8600000000000003,"5min":9.5489999999999995,"15min":6.8339999999999996},"min":{"1min":1.1639999999999999,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":30.184999999999999,"5min":52.390000000000001,"15min":52.759},"last":4.5199999999999996}]},{"osd":7,"last update":"Fri Nov 20 11:51:46 2020","interfaces":[{"interface":"back","average":{"1min":7.3719999999999999,"5min":11.303000000000001,"15min":8.0090000000000003},"min":{"1min":1.018,"5min":1.018,"15min":0.96299999999999997},"max":{"1min":25.225999999999999,"5min":52.137999999999998,"15min":52.137999999999998},"last":3.008},{"interface":"front","average":{"1min":8.4309999999999992,"5min":10.443,"15min":7.6100000000000003},"min":{"1min":1.2849999999999999,"5min":1.214,"15min":0.97999999999999998},"max":{"1min":26.350000000000001,"5min":55.387999999999998,"15min":55.387999999999998},"last":2.964}]}]},{"osd":2,"up_from":14,"seq":60129542389,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1053920,"kb_used_data":897304,"kb_used_omap":0,"kb_used_meta":156608,"kb_avail":93317920,"statfs":{"total":96636764160,"available":95557550080,"internally_reserved":0,"allocated":918839296,"data_stored":918676883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":160366592},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 11:51:55 2020","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.1639999999999999,"15min":1.7769999999999999},"min":{"1min":0.55900000000000005,"5min":0.503,"15min":0.436},"max":{"1min":4.5540000000000003,"5min":8.4770000000000003,"15min":68.239999999999995},"last":49.945999999999998},{"interface":"front","average":{"1min":1.018,"5min":1.2370000000000001,"15min":2.077},"min":{"1min":0.61099999999999999,"5min":0.52300000000000002,"15min":0.48999999999999999},"max":{"1min":2.3130000000000002,"5min":9.1310000000000002,"15min":68.804000000000002},"last":50.831000000000003}]},{"osd":1,"last update":"Fri Nov 20 11:51:55 2020","interfaces":[{"interface":"back","average":{"1min":6.2859999999999996,"5min":8.3379999999999992,"15min":6.9489999999999998},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.56299999999999994},"max":{"1min":31.527999999999999,"5min":53.654000000000003,"15min":72.744},"last":15.388},{"interface":"front","average":{"1min":6.5579999999999998,"5min":8.5719999999999992,"15min":7.2720000000000002},"min":{"1min":0.94699999999999995,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":31.486000000000001,"5min":53.222999999999999,"15min":73.438000000000002},"last":15.414999999999999}]},{"osd":3,"last update":"Fri Nov 20 11:51:55 2020","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":1.014,"15min":1.3180000000000001},"min":{"1min":0.58099999999999996,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":3.605,"5min":4.0339999999999998,"15min":21.465},"last":1.456},{"interface":"front","average":{"1min":1.413,"5min":1.3660000000000001,"15min":1.6439999999999999},"min":{"1min":0.77800000000000002,"5min":0.67800000000000005,"15min":0.48199999999999998},"max":{"1min":4.7050000000000001,"5min":5.0270000000000001,"15min":21.693000000000001},"last":1.6879999999999999}]},{"osd":4,"last update":"Fri Nov 20 11:51:55 2020","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.4239999999999999,"15min":1.4950000000000001},"min":{"1min":0.80900000000000005,"5min":0.67300000000000004,"15min":0.626},"max":{"1min":1.8759999999999999,"5min":4.6689999999999996,"15min":10.731999999999999},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.419,"5min":1.4179999999999999,"15min":1.5880000000000001},"min":{"1min":0.76200000000000001,"5min":0.72599999999999998,"15min":0.56499999999999995},"max":{"1min":4.5890000000000004,"5min":7.2699999999999996,"15min":9.6519999999999992},"last":1.121}]},{"osd":5,"last update":"Fri Nov 20 11:51:55 2020","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.369,"15min":1.4339999999999999},"min":{"1min":0.71699999999999997,"5min":0.70099999999999996,"15min":0.64100000000000001},"max":{"1min":4.7329999999999997,"5min":6.1859999999999999,"15min":12.045999999999999},"last":0.63},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.5629999999999999,"15min":1.7210000000000001},"min":{"1min":0.90200000000000002,"5min":0.82399999999999995,"15min":0.70299999999999996},"max":{"1min":2.0009999999999999,"5min":5.0620000000000003,"15min":11.856},"last":0.74199999999999999}]},{"osd":6,"last update":"Fri Nov 20 11:51:55 2020","interfaces":[{"interface":"back","average":{"1min":1.629,"5min":1.595,"15min":1.671},"min":{"1min":0.97899999999999998,"5min":0.81999999999999995,"15min":0.65100000000000002},"max":{"1min":5.0759999999999996,"5min":7.4130000000000003,"15min":7.4130000000000003},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.5880000000000001,"5min":1.5449999999999999,"15min":1.5760000000000001},"min":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.83899999999999997},"max":{"1min":4.8929999999999998,"5min":6.3209999999999997,"15min":7.5369999999999999},"last":0.93600000000000005}]},{"osd":7,"last update":"Fri Nov 20 11:51:55 2020","interfaces":[{"interface":"back","average":{"1min":1.675,"5min":1.734,"15min":1.7809999999999999},"min":{"1min":1.079,"5min":0.98999999999999999,"15min":0.85099999999999998},"max":{"1min":5.2240000000000002,"5min":7.54,"15min":7.54},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.6719999999999999,"5min":1.613,"15min":1.708},"min":{"1min":0.97599999999999998,"5min":0.89800000000000002,"15min":0.83799999999999997},"max":{"1min":5.04,"5min":6.4420000000000002,"15min":7.6909999999999998},"last":0.80900000000000005}]}]}]}} 2020-11-20T11:52:27.771 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2020-11-20T11:32:24.089255+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=24, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:27.771 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2020-11-20T11:32:17.903652+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=17, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:27.772 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2020-11-20T11:32:21.059110+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=21, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:27.772 INFO:tasks.ceph:pgid 4.4 last_scrub_stamp 2020-11-20T11:32:24.089255+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=24, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:27.772 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2020-11-20T11:32:17.903652+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=17, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:27.772 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2020-11-20T11:32:21.059110+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=21, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:27.773 INFO:tasks.ceph:pgid 4.7 last_scrub_stamp 2020-11-20T11:32:24.089255+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=24, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:27.773 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2020-11-20T11:32:17.903652+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=17, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:27.773 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2020-11-20T11:32:21.059110+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=21, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:27.774 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2020-11-20T11:32:24.089255+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=24, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:27.774 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2020-11-20T11:32:17.903652+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=17, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:27.774 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2020-11-20T11:32:10.515989+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=10, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:27.775 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2020-11-20T11:32:24.089255+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=24, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:27.775 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2020-11-20T11:32:17.903652+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=17, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:27.775 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2020-11-20T11:32:21.059110+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=21, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:27.776 INFO:tasks.ceph:pgid 4.3 last_scrub_stamp 2020-11-20T11:32:24.089255+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=24, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:27.776 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2020-11-20T11:32:17.903652+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=17, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:27.776 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2020-11-20T11:32:21.059110+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=21, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:27.777 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2020-11-20T11:32:24.089255+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=24, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:27.777 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2020-11-20T11:32:21.059110+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=21, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:27.777 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2020-11-20T11:32:17.903652+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=17, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:27.777 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2020-11-20T11:32:24.089255+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=24, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:27.778 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2020-11-20T11:32:17.903652+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=17, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:27.778 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2020-11-20T11:32:21.059110+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=21, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:27.778 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2020-11-20T11:52:47.780 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-20T11:52:47.943 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.939+0000 7f05c1ad2700 1 Processor -- start 2020-11-20T11:52:47.943 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.939+0000 7f05c1ad2700 1 -- start start 2020-11-20T11:52:47.943 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.940+0000 7f05c1ad2700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05bc0aaab0 0x7f05bc13ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:47.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.940+0000 7f05c1ad2700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f05bc13aeb0 0x7f05bc0a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:47.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.940+0000 7f05c1ad2700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05bc1388b0 con 0x7f05bc0aae90 2020-11-20T11:52:47.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.940+0000 7f05c1ad2700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05bc13e600 con 0x7f05bc13aeb0 2020-11-20T11:52:47.944 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.940+0000 7f05bbfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f05bc13aeb0 0x7f05bc0a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:47.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.940+0000 7f05bbfff700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f05bc13aeb0 0x7f05bc0a8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47470/0 (socket says 172.21.15.23:47470) 2020-11-20T11:52:47.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.940+0000 7f05bbfff700 1 -- 172.21.15.23:0/844564305 learned_addr learned my addr 172.21.15.23:0/844564305 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:52:47.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.940+0000 7f05c1ad2700 1 -- 172.21.15.23:0/844564305 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc13e8a0 con 0x7f05bc0aaab0 2020-11-20T11:52:47.945 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.940+0000 7f05bb7fe700 1 --2- 172.21.15.23:0/844564305 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05bc0aaab0 0x7f05bc13ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:47.946 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.940+0000 7f05bb7fe700 1 -- 172.21.15.23:0/844564305 >> v2:172.21.15.23:3300/0 conn(0x7f05bc13aeb0 msgr2=0x7f05bc0a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:52:47.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.940+0000 7f05bb7fe700 1 --2- 172.21.15.23:0/844564305 >> v2:172.21.15.23:3300/0 conn(0x7f05bc13aeb0 0x7f05bc0a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:47.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.940+0000 7f05bb7fe700 1 -- 172.21.15.23:0/844564305 >> v1:172.21.15.155:6789/0 conn(0x7f05bc0aae90 legacy=0x7f05bc1381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:47.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.940+0000 7f05bb7fe700 1 -- 172.21.15.23:0/844564305 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05bc0ab6d0 con 0x7f05bc0aaab0 2020-11-20T11:52:47.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.941+0000 7f05bbfff700 1 --2- 172.21.15.23:0/844564305 >> v2:172.21.15.23:3300/0 conn(0x7f05bc13aeb0 0x7f05bc0a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:52:47.947 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.941+0000 7f05bb7fe700 1 --2- 172.21.15.23:0/844564305 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05bc0aaab0 0x7f05bc13ed40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f05ac006fd0 tx=0x7f05ac006e30).ready entity=mon.2 client_cookie=4903a09cf63e15e0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:47.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.941+0000 7f05ba7fc700 1 -- 172.21.15.23:0/844564305 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ac0097e0 con 0x7f05bc0aaab0 2020-11-20T11:52:47.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.941+0000 7f05ba7fc700 1 -- 172.21.15.23:0/844564305 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05ac000e90 con 0x7f05bc0aaab0 2020-11-20T11:52:47.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.941+0000 7f05ba7fc700 1 -- 172.21.15.23:0/844564305 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ac009b50 con 0x7f05bc0aaab0 2020-11-20T11:52:47.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.943+0000 7f05c1ad2700 1 -- 172.21.15.23:0/844564305 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05bc0aaab0 msgr2=0x7f05bc13ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:47.948 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.943+0000 7f05c1ad2700 1 --2- 172.21.15.23:0/844564305 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05bc0aaab0 0x7f05bc13ed40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f05ac006fd0 tx=0x7f05ac006e30).stop 2020-11-20T11:52:47.958 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.947+0000 7f05c1ad2700 1 -- 172.21.15.23:0/844564305 shutdown_connections 2020-11-20T11:52:47.959 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.947+0000 7f05c1ad2700 1 --2- 172.21.15.23:0/844564305 >> v2:172.21.15.23:3300/0 conn(0x7f05bc13aeb0 0x7f05bc0a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:47.959 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.947+0000 7f05c1ad2700 1 --2- 172.21.15.23:0/844564305 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05bc0aaab0 0x7f05bc13ed40 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:47.960 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.947+0000 7f05c1ad2700 1 -- 172.21.15.23:0/844564305 >> 172.21.15.23:0/844564305 conn(0x7f05bc0678e0 msgr2=0x7f05bc067cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:47.960 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.947+0000 7f05c1ad2700 1 -- 172.21.15.23:0/844564305 shutdown_connections 2020-11-20T11:52:47.960 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.948+0000 7f05c1ad2700 1 -- 172.21.15.23:0/844564305 wait complete. 2020-11-20T11:52:47.960 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.949+0000 7f05c1ad2700 1 Processor -- start 2020-11-20T11:52:47.961 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.949+0000 7f05c1ad2700 1 -- start start 2020-11-20T11:52:47.961 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.949+0000 7f05c1ad2700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05bc0aaab0 0x7f05bc0a54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:47.961 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.949+0000 7f05c1ad2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05bc0aae90 0x7f05bc0b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:47.961 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.949+0000 7f05c1ad2700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05bc1369a0 0x7f05bc14dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:47.962 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.949+0000 7f05c1ad2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc0b09b0 con 0x7f05bc0aae90 2020-11-20T11:52:47.962 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.949+0000 7f05c1ad2700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc0b0c40 con 0x7f05bc1369a0 2020-11-20T11:52:47.962 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.949+0000 7f05c1ad2700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc0b0ee0 con 0x7f05bc0aaab0 2020-11-20T11:52:47.962 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.950+0000 7f05baffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05bc0aae90 0x7f05bc0b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:47.962 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.950+0000 7f05baffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05bc0aae90 0x7f05bc0b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47474/0 (socket says 172.21.15.23:47474) 2020-11-20T11:52:47.963 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.950+0000 7f05baffd700 1 -- 172.21.15.23:0/2229826749 learned_addr learned my addr 172.21.15.23:0/2229826749 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:52:47.963 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.950+0000 7f05bb7fe700 1 --2- 172.21.15.23:0/2229826749 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05bc0aaab0 0x7f05bc0a54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:47.963 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.950+0000 7f05bbfff700 1 --2- 172.21.15.23:0/2229826749 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05bc1369a0 0x7f05bc14dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:47.963 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.950+0000 7f05baffd700 1 -- 172.21.15.23:0/2229826749 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05bc0aaab0 msgr2=0x7f05bc0a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:47.963 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.950+0000 7f05baffd700 1 --2- 172.21.15.23:0/2229826749 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05bc0aaab0 0x7f05bc0a54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:47.964 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.950+0000 7f05baffd700 1 -- 172.21.15.23:0/2229826749 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05bc1369a0 msgr2=0x7f05bc14dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:47.964 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.950+0000 7f05baffd700 1 --2- 172.21.15.23:0/2229826749 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05bc1369a0 0x7f05bc14dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:47.964 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.950+0000 7f05baffd700 1 -- 172.21.15.23:0/2229826749 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05bc14e180 con 0x7f05bc0aae90 2020-11-20T11:52:47.964 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.950+0000 7f05baffd700 1 --2- 172.21.15.23:0/2229826749 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05bc0aae90 0x7f05bc0b04c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f05b000c710 tx=0x7f05b0007bd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:47.964 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.950+0000 7f059b7fe700 1 -- 172.21.15.23:0/2229826749 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b0007020 con 0x7f05bc0aae90 2020-11-20T11:52:47.965 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.950+0000 7f059b7fe700 1 -- 172.21.15.23:0/2229826749 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b0001e90 con 0x7f05bc0aae90 2020-11-20T11:52:47.965 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.950+0000 7f05c1ad2700 1 -- 172.21.15.23:0/2229826749 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05bc14e430 con 0x7f05bc0aae90 2020-11-20T11:52:47.965 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.950+0000 7f059b7fe700 1 -- 172.21.15.23:0/2229826749 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b0007390 con 0x7f05bc0aae90 2020-11-20T11:52:47.965 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.951+0000 7f05c1ad2700 1 -- 172.21.15.23:0/2229826749 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05bc1326f0 con 0x7f05bc0aae90 2020-11-20T11:52:47.966 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.951+0000 7f05c1ad2700 1 -- 172.21.15.23:0/2229826749 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05bc05c7d0 con 0x7f05bc0aae90 2020-11-20T11:52:47.966 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.952+0000 7f059b7fe700 1 -- 172.21.15.23:0/2229826749 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f05b001a060 con 0x7f05bc0aae90 2020-11-20T11:52:47.966 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.952+0000 7f059b7fe700 1 --2- 172.21.15.23:0/2229826749 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0594056d60 0x7f0594059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:52:47.966 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.952+0000 7f059b7fe700 1 -- 172.21.15.23:0/2229826749 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f05b006f6a0 con 0x7f05bc0aae90 2020-11-20T11:52:47.966 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.952+0000 7f05bb7fe700 1 --2- 172.21.15.23:0/2229826749 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0594056d60 0x7f0594059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:52:47.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.953+0000 7f05bb7fe700 1 --2- 172.21.15.23:0/2229826749 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0594056d60 0x7f0594059220 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f05ac006e70 tx=0x7f05ac015000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:52:47.967 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:47.959+0000 7f059b7fe700 1 -- 172.21.15.23:0/2229826749 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f05b0044990 con 0x7f05bc0aae90 2020-11-20T11:52:48.174 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:48.171+0000 7f05c1ad2700 1 -- 172.21.15.23:0/2229826749 --> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f05bc044710 con 0x7f0594056d60 2020-11-20T11:52:48.181 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:48.178+0000 7f059b7fe700 1 -- 172.21.15.23:0/2229826749 <== mgr.4105 v2:172.21.15.155:6832/22693 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+106752 (secure 0 0 0) 0x7f05bc044710 con 0x7f0594056d60 2020-11-20T11:52:48.182 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:52:48.184 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:48.179+0000 7f05c1ad2700 1 -- 172.21.15.23:0/2229826749 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0594056d60 msgr2=0x7f0594059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:48.184 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:48.179+0000 7f05c1ad2700 1 --2- 172.21.15.23:0/2229826749 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0594056d60 0x7f0594059220 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f05ac006e70 tx=0x7f05ac015000).stop 2020-11-20T11:52:48.184 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:48.179+0000 7f05c1ad2700 1 -- 172.21.15.23:0/2229826749 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05bc0aae90 msgr2=0x7f05bc0b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:52:48.185 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:48.179+0000 7f05c1ad2700 1 --2- 172.21.15.23:0/2229826749 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05bc0aae90 0x7f05bc0b04c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f05b000c710 tx=0x7f05b0007bd0).stop 2020-11-20T11:52:48.185 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:48.180+0000 7f05c1ad2700 1 -- 172.21.15.23:0/2229826749 shutdown_connections 2020-11-20T11:52:48.185 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:48.180+0000 7f05c1ad2700 1 --2- 172.21.15.23:0/2229826749 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f05bc1369a0 0x7f05bc14dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:48.185 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:48.180+0000 7f05c1ad2700 1 --2- 172.21.15.23:0/2229826749 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0594056d60 0x7f0594059220 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:48.186 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:48.180+0000 7f05c1ad2700 1 --2- 172.21.15.23:0/2229826749 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f05bc0aaab0 0x7f05bc0a54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:48.187 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:48.180+0000 7f05c1ad2700 1 --2- 172.21.15.23:0/2229826749 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f05bc0aae90 0x7f05bc0b04c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:52:48.188 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:48.180+0000 7f05c1ad2700 1 -- 172.21.15.23:0/2229826749 >> 172.21.15.23:0/2229826749 conn(0x7f05bc0678e0 msgr2=0x7f05bc1405e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:52:48.188 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:48.180+0000 7f05c1ad2700 1 -- 172.21.15.23:0/2229826749 shutdown_connections 2020-11-20T11:52:48.188 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:52:48.180+0000 7f05c1ad2700 1 -- 172.21.15.23:0/2229826749 wait complete. 2020-11-20T11:52:48.189 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2020-11-20T11:52:48.198 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":636,"stamp":"2020-11-20T11:52:47.800622+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3066445135,"num_objects":756,"num_object_clones":0,"num_object_copies":2970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":756,"num_whiteouts":0,"num_read":23,"num_read_kb":61,"num_write":255773,"num_write_kb":14114657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13231,"num_omap_keys":12,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82438,"ondisk_log_size":82438,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":66,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8677720,"kb_used_data":5989208,"kb_used_omap":0,"kb_used_meta":2678208,"kb_avail":746297000,"statfs":{"total":773094113280,"available":764208128000,"internally_reserved":0,"allocated":6132948992,"data_stored":6131651358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2742484992},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,4,8,14,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":181,"apply_latency_ms":181,"commit_latency_ns":181000000,"apply_latency_ns":181000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2626115472,"num_objects":-632,"num_object_clones":0,"num_object_copies":-2528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":633,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005384"},"pg_stats":[{"pgid":"4.2","version":"25'15646","reported_seq":"15659","reported_epoch":"25","state":"active+clean+scrubbing+deep","last_fresh":"2020-11-20T11:52:46.919057+0000","last_change":"2020-11-20T11:52:40.197222+0000","last_active":"2020-11-20T11:52:46.919057+0000","last_peered":"2020-11-20T11:52:46.919057+0000","last_clean":"2020-11-20T11:52:46.919057+0000","last_became_active":"2020-11-20T11:32:25.124627+0000","last_became_peered":"2020-11-20T11:32:25.124627+0000","last_unstale":"2020-11-20T11:52:46.919057+0000","last_undegraded":"2020-11-20T11:52:46.919057+0000","last_fullsized":"2020-11-20T11:52:46.919057+0000","mapping_epoch":22,"log_start":"25'5609","ondisk_log_start":"25'5609","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":10037,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":436207616,"num_objects":104,"num_object_clones":0,"num_object_copies":416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33450,"num_write_kb":1823424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"19","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:23.044277+0000","last_change":"2020-11-20T11:52:23.044351+0000","last_active":"2020-11-20T11:52:23.044277+0000","last_peered":"2020-11-20T11:52:23.044277+0000","last_clean":"2020-11-20T11:52:23.044277+0000","last_became_active":"2020-11-20T11:32:19.482197+0000","last_became_peered":"2020-11-20T11:32:19.482197+0000","last_unstale":"2020-11-20T11:52:23.044277+0000","last_undegraded":"2020-11-20T11:52:23.044277+0000","last_fullsized":"2020-11-20T11:52:23.044277+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:52:23.044271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:52:23.044271+0000","last_clean_scrub_stamp":"2020-11-20T11:52:23.044271+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"25'31","reported_seq":"44","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:27.169773+0000","last_change":"2020-11-20T11:32:22.098576+0000","last_active":"2020-11-20T11:52:27.169773+0000","last_peered":"2020-11-20T11:52:27.169773+0000","last_clean":"2020-11-20T11:52:27.169773+0000","last_became_active":"2020-11-20T11:32:22.098264+0000","last_became_peered":"2020-11-20T11:32:22.098264+0000","last_unstale":"2020-11-20T11:52:27.169773+0000","last_undegraded":"2020-11-20T11:52:27.169773+0000","last_fullsized":"2020-11-20T11:52:27.169773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:21.059110+0000","last_clean_scrub_stamp":"2020-11-20T11:32:21.059110+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"25'14375","reported_seq":"14387","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:46.789273+0000","last_change":"2020-11-20T11:32:25.479776+0000","last_active":"2020-11-20T11:52:46.789273+0000","last_peered":"2020-11-20T11:52:46.789273+0000","last_clean":"2020-11-20T11:52:46.789273+0000","last_became_active":"2020-11-20T11:32:25.479350+0000","last_became_peered":"2020-11-20T11:32:25.479350+0000","last_unstale":"2020-11-20T11:52:46.789273+0000","last_undegraded":"2020-11-20T11:52:46.789273+0000","last_fullsized":"2020-11-20T11:52:46.789273+0000","mapping_epoch":22,"log_start":"25'4291","ondisk_log_start":"25'4291","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":10084,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":301989888,"num_objects":72,"num_object_clones":0,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31614,"num_write_kb":1652120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"19","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:26.352689+0000","last_change":"2020-11-20T11:52:26.352771+0000","last_active":"2020-11-20T11:52:26.352689+0000","last_peered":"2020-11-20T11:52:26.352689+0000","last_clean":"2020-11-20T11:52:26.352689+0000","last_became_active":"2020-11-20T11:32:18.923879+0000","last_became_peered":"2020-11-20T11:32:18.923879+0000","last_unstale":"2020-11-20T11:52:26.352689+0000","last_undegraded":"2020-11-20T11:52:26.352689+0000","last_fullsized":"2020-11-20T11:52:26.352689+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:52:26.352682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:52:26.352682+0000","last_clean_scrub_stamp":"2020-11-20T11:52:26.352682+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"25'1087","reported_seq":"1103","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:28.276341+0000","last_change":"2020-11-20T11:52:28.276414+0000","last_active":"2020-11-20T11:52:28.276341+0000","last_peered":"2020-11-20T11:52:28.276341+0000","last_clean":"2020-11-20T11:52:28.276341+0000","last_became_active":"2020-11-20T11:32:22.093123+0000","last_became_peered":"2020-11-20T11:32:22.093123+0000","last_unstale":"2020-11-20T11:52:28.276341+0000","last_undegraded":"2020-11-20T11:52:28.276341+0000","last_fullsized":"2020-11-20T11:52:28.276341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'1087","last_scrub_stamp":"2020-11-20T11:52:28.276335+0000","last_deep_scrub":"25'1087","last_deep_scrub_stamp":"2020-11-20T11:52:28.276335+0000","last_clean_scrub_stamp":"2020-11-20T11:52:28.276335+0000","log_size":1087,"ondisk_log_size":1087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4554460,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1083,"num_write_kb":4776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4710,"num_omap_keys":10,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"25'14719","reported_seq":"14731","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:46.679393+0000","last_change":"2020-11-20T11:32:25.125052+0000","last_active":"2020-11-20T11:52:46.679393+0000","last_peered":"2020-11-20T11:52:46.679393+0000","last_clean":"2020-11-20T11:52:46.679393+0000","last_became_active":"2020-11-20T11:32:25.124401+0000","last_became_peered":"2020-11-20T11:32:25.124401+0000","last_unstale":"2020-11-20T11:52:46.679393+0000","last_undegraded":"2020-11-20T11:52:46.679393+0000","last_fullsized":"2020-11-20T11:52:46.679393+0000","mapping_epoch":22,"log_start":"25'4685","ondisk_log_start":"25'4685","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":10034,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":310378496,"num_objects":74,"num_object_clones":0,"num_object_copies":296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31268,"num_write_kb":1732953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"18","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:24.935039+0000","last_change":"2020-11-20T11:52:24.935164+0000","last_active":"2020-11-20T11:52:24.935039+0000","last_peered":"2020-11-20T11:52:24.935039+0000","last_clean":"2020-11-20T11:52:24.935039+0000","last_became_active":"2020-11-20T11:32:18.926981+0000","last_became_peered":"2020-11-20T11:32:18.926981+0000","last_unstale":"2020-11-20T11:52:24.935039+0000","last_undegraded":"2020-11-20T11:52:24.935039+0000","last_fullsized":"2020-11-20T11:52:24.935039+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:52:24.935031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:52:24.935031+0000","last_clean_scrub_stamp":"2020-11-20T11:52:24.935031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"25'17","reported_seq":"33","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:34.238666+0000","last_change":"2020-11-20T11:52:34.238741+0000","last_active":"2020-11-20T11:52:34.238666+0000","last_peered":"2020-11-20T11:52:34.238666+0000","last_clean":"2020-11-20T11:52:34.238666+0000","last_became_active":"2020-11-20T11:32:22.087723+0000","last_became_peered":"2020-11-20T11:32:22.087723+0000","last_unstale":"2020-11-20T11:52:34.238666+0000","last_undegraded":"2020-11-20T11:52:34.238666+0000","last_fullsized":"2020-11-20T11:52:34.238666+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'17","last_scrub_stamp":"2020-11-20T11:52:34.238658+0000","last_deep_scrub":"25'17","last_deep_scrub_stamp":"2020-11-20T11:52:34.238658+0000","last_clean_scrub_stamp":"2020-11-20T11:52:34.238658+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"25'13815","reported_seq":"13830","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:44.542875+0000","last_change":"2020-11-20T11:52:38.489578+0000","last_active":"2020-11-20T11:52:44.542875+0000","last_peered":"2020-11-20T11:52:44.542875+0000","last_clean":"2020-11-20T11:52:44.542875+0000","last_became_active":"2020-11-20T11:32:25.491657+0000","last_became_peered":"2020-11-20T11:32:25.491657+0000","last_unstale":"2020-11-20T11:52:44.542875+0000","last_undegraded":"2020-11-20T11:52:44.542875+0000","last_fullsized":"2020-11-20T11:52:44.542875+0000","mapping_epoch":22,"log_start":"25'3748","ondisk_log_start":"25'3748","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'13748","last_scrub_stamp":"2020-11-20T11:52:38.489500+0000","last_deep_scrub":"25'13748","last_deep_scrub_stamp":"2020-11-20T11:52:38.489500+0000","last_clean_scrub_stamp":"2020-11-20T11:52:38.489500+0000","log_size":10067,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":356515840,"num_objects":85,"num_object_clones":0,"num_object_copies":340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30845,"num_write_kb":1726809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"19","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:37.393053+0000","last_change":"2020-11-20T11:52:37.393142+0000","last_active":"2020-11-20T11:52:37.393053+0000","last_peered":"2020-11-20T11:52:37.393053+0000","last_clean":"2020-11-20T11:52:37.393053+0000","last_became_active":"2020-11-20T11:32:18.926711+0000","last_became_peered":"2020-11-20T11:32:18.926711+0000","last_unstale":"2020-11-20T11:52:37.393053+0000","last_undegraded":"2020-11-20T11:52:37.393053+0000","last_fullsized":"2020-11-20T11:52:37.393053+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:52:37.393046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:52:37.393046+0000","last_clean_scrub_stamp":"2020-11-20T11:52:37.393046+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"25'4","reported_seq":"25","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:27.883942+0000","last_change":"2020-11-20T11:52:27.884019+0000","last_active":"2020-11-20T11:52:27.883942+0000","last_peered":"2020-11-20T11:52:27.883942+0000","last_clean":"2020-11-20T11:52:27.883942+0000","last_became_active":"2020-11-20T11:32:14.445477+0000","last_became_peered":"2020-11-20T11:32:14.445477+0000","last_unstale":"2020-11-20T11:52:27.883942+0000","last_undegraded":"2020-11-20T11:52:27.883942+0000","last_fullsized":"2020-11-20T11:52:27.883942+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'4","last_scrub_stamp":"2020-11-20T11:52:27.883935+0000","last_deep_scrub":"25'4","last_deep_scrub_stamp":"2020-11-20T11:52:27.883935+0000","last_clean_scrub_stamp":"2020-11-20T11:52:27.883935+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8521,"num_omap_keys":2,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"25'4","reported_seq":"20","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:25.210203+0000","last_change":"2020-11-20T11:52:25.210262+0000","last_active":"2020-11-20T11:52:25.210203+0000","last_peered":"2020-11-20T11:52:25.210203+0000","last_clean":"2020-11-20T11:52:25.210203+0000","last_became_active":"2020-11-20T11:32:22.087777+0000","last_became_peered":"2020-11-20T11:32:22.087777+0000","last_unstale":"2020-11-20T11:52:25.210203+0000","last_undegraded":"2020-11-20T11:52:25.210203+0000","last_fullsized":"2020-11-20T11:52:25.210203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'4","last_scrub_stamp":"2020-11-20T11:52:25.210199+0000","last_deep_scrub":"25'4","last_deep_scrub_stamp":"2020-11-20T11:52:25.210199+0000","last_clean_scrub_stamp":"2020-11-20T11:52:25.210199+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"25'12535","reported_seq":"12547","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:42.502822+0000","last_change":"2020-11-20T11:32:25.491667+0000","last_active":"2020-11-20T11:52:42.502822+0000","last_peered":"2020-11-20T11:52:42.502822+0000","last_clean":"2020-11-20T11:52:42.502822+0000","last_became_active":"2020-11-20T11:32:25.491100+0000","last_became_peered":"2020-11-20T11:32:25.491100+0000","last_unstale":"2020-11-20T11:52:42.502822+0000","last_undegraded":"2020-11-20T11:52:42.502822+0000","last_fullsized":"2020-11-20T11:52:42.502822+0000","mapping_epoch":22,"log_start":"25'2452","ondisk_log_start":"25'2452","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":10083,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":402653184,"num_objects":96,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28232,"num_write_kb":1700287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"18","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:28.804184+0000","last_change":"2020-11-20T11:52:28.804248+0000","last_active":"2020-11-20T11:52:28.804184+0000","last_peered":"2020-11-20T11:52:28.804184+0000","last_clean":"2020-11-20T11:52:28.804184+0000","last_became_active":"2020-11-20T11:32:18.924324+0000","last_became_peered":"2020-11-20T11:32:18.924324+0000","last_unstale":"2020-11-20T11:52:28.804184+0000","last_undegraded":"2020-11-20T11:52:28.804184+0000","last_fullsized":"2020-11-20T11:52:28.804184+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:52:28.804180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:52:28.804180+0000","last_clean_scrub_stamp":"2020-11-20T11:52:28.804180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"25'56","reported_seq":"95","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:24.000161+0000","last_change":"2020-11-20T11:52:24.000261+0000","last_active":"2020-11-20T11:52:24.000161+0000","last_peered":"2020-11-20T11:52:24.000161+0000","last_clean":"2020-11-20T11:52:24.000161+0000","last_became_active":"2020-11-20T11:32:22.088414+0000","last_became_peered":"2020-11-20T11:32:22.088414+0000","last_unstale":"2020-11-20T11:52:24.000161+0000","last_undegraded":"2020-11-20T11:52:24.000161+0000","last_fullsized":"2020-11-20T11:52:24.000161+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'56","last_scrub_stamp":"2020-11-20T11:52:24.000149+0000","last_deep_scrub":"25'56","last_deep_scrub_stamp":"2020-11-20T11:52:24.000149+0000","last_clean_scrub_stamp":"2020-11-20T11:52:24.000149+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":47377,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":61,"num_write":58,"num_write_kb":143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"25'15291","reported_seq":"15303","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:42.662033+0000","last_change":"2020-11-20T11:32:25.491507+0000","last_active":"2020-11-20T11:52:42.662033+0000","last_peered":"2020-11-20T11:52:42.662033+0000","last_clean":"2020-11-20T11:52:42.662033+0000","last_became_active":"2020-11-20T11:32:25.490917+0000","last_became_peered":"2020-11-20T11:32:25.490917+0000","last_unstale":"2020-11-20T11:52:42.662033+0000","last_undegraded":"2020-11-20T11:52:42.662033+0000","last_fullsized":"2020-11-20T11:52:42.662033+0000","mapping_epoch":22,"log_start":"25'5251","ondisk_log_start":"25'5251","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":10040,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":448790528,"num_objects":107,"num_object_clones":0,"num_object_copies":428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33731,"num_write_kb":1816287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"15","reported_epoch":"24","state":"active+clean","last_fresh":"2020-11-20T11:32:25.488121+0000","last_change":"2020-11-20T11:32:19.482719+0000","last_active":"2020-11-20T11:32:25.488121+0000","last_peered":"2020-11-20T11:32:25.488121+0000","last_clean":"2020-11-20T11:32:25.488121+0000","last_became_active":"2020-11-20T11:32:19.482359+0000","last_became_peered":"2020-11-20T11:32:19.482359+0000","last_unstale":"2020-11-20T11:32:25.488121+0000","last_undegraded":"2020-11-20T11:32:25.488121+0000","last_fullsized":"2020-11-20T11:32:25.488121+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:17.903652+0000","last_clean_scrub_stamp":"2020-11-20T11:32:17.903652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"25'562","reported_seq":"578","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:39.452009+0000","last_change":"2020-11-20T11:52:39.452233+0000","last_active":"2020-11-20T11:52:39.452009+0000","last_peered":"2020-11-20T11:52:39.452009+0000","last_clean":"2020-11-20T11:52:39.452009+0000","last_became_active":"2020-11-20T11:32:22.092619+0000","last_became_peered":"2020-11-20T11:32:22.092619+0000","last_unstale":"2020-11-20T11:52:39.452009+0000","last_undegraded":"2020-11-20T11:52:39.452009+0000","last_fullsized":"2020-11-20T11:52:39.452009+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'562","last_scrub_stamp":"2020-11-20T11:52:39.452002+0000","last_deep_scrub":"25'562","last_deep_scrub_stamp":"2020-11-20T11:52:39.452002+0000","last_clean_scrub_stamp":"2020-11-20T11:52:39.452002+0000","log_size":562,"ondisk_log_size":562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4194947,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":555,"num_write_kb":4297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"25'14306","reported_seq":"14318","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:45.205440+0000","last_change":"2020-11-20T11:32:25.492150+0000","last_active":"2020-11-20T11:52:45.205440+0000","last_peered":"2020-11-20T11:52:45.205440+0000","last_clean":"2020-11-20T11:52:45.205440+0000","last_became_active":"2020-11-20T11:32:25.491099+0000","last_became_peered":"2020-11-20T11:32:25.491099+0000","last_unstale":"2020-11-20T11:52:45.205440+0000","last_undegraded":"2020-11-20T11:52:45.205440+0000","last_fullsized":"2020-11-20T11:52:45.205440+0000","mapping_epoch":22,"log_start":"25'4229","ondisk_log_start":"25'4229","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":10077,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":390070272,"num_objects":93,"num_object_clones":0,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31797,"num_write_kb":1743046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"25'15","reported_seq":"31","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:40.151034+0000","last_change":"2020-11-20T11:52:40.151093+0000","last_active":"2020-11-20T11:52:40.151034+0000","last_peered":"2020-11-20T11:52:40.151034+0000","last_clean":"2020-11-20T11:52:40.151034+0000","last_became_active":"2020-11-20T11:32:22.461331+0000","last_became_peered":"2020-11-20T11:32:22.461331+0000","last_unstale":"2020-11-20T11:52:40.151034+0000","last_undegraded":"2020-11-20T11:52:40.151034+0000","last_fullsized":"2020-11-20T11:52:40.151034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'15","last_scrub_stamp":"2020-11-20T11:52:40.151029+0000","last_deep_scrub":"25'15","last_deep_scrub_stamp":"2020-11-20T11:52:40.151029+0000","last_clean_scrub_stamp":"2020-11-20T11:52:40.151029+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"19","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:28.147369+0000","last_change":"2020-11-20T11:52:28.147444+0000","last_active":"2020-11-20T11:52:28.147369+0000","last_peered":"2020-11-20T11:52:28.147369+0000","last_clean":"2020-11-20T11:52:28.147369+0000","last_became_active":"2020-11-20T11:32:18.923448+0000","last_became_peered":"2020-11-20T11:32:18.923448+0000","last_unstale":"2020-11-20T11:52:28.147369+0000","last_undegraded":"2020-11-20T11:52:28.147369+0000","last_fullsized":"2020-11-20T11:52:28.147369+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:52:28.147363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:52:28.147363+0000","last_clean_scrub_stamp":"2020-11-20T11:52:28.147363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"25'14175","reported_seq":"14187","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:43.220686+0000","last_change":"2020-11-20T11:32:25.491585+0000","last_active":"2020-11-20T11:52:43.220686+0000","last_peered":"2020-11-20T11:52:43.220686+0000","last_clean":"2020-11-20T11:52:43.220686+0000","last_became_active":"2020-11-20T11:32:25.491038+0000","last_became_peered":"2020-11-20T11:32:25.491038+0000","last_unstale":"2020-11-20T11:52:43.220686+0000","last_undegraded":"2020-11-20T11:52:43.220686+0000","last_fullsized":"2020-11-20T11:52:43.220686+0000","mapping_epoch":22,"log_start":"25'4077","ondisk_log_start":"25'4077","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:32:24.089255+0000","last_clean_scrub_stamp":"2020-11-20T11:32:24.089255+0000","log_size":10098,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":411041792,"num_objects":98,"num_object_clones":0,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32726,"num_write_kb":1910315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"18","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:28.204947+0000","last_change":"2020-11-20T11:52:28.205034+0000","last_active":"2020-11-20T11:52:28.204947+0000","last_peered":"2020-11-20T11:52:28.204947+0000","last_clean":"2020-11-20T11:52:28.204947+0000","last_became_active":"2020-11-20T11:32:18.923668+0000","last_became_peered":"2020-11-20T11:32:18.923668+0000","last_unstale":"2020-11-20T11:52:28.204947+0000","last_undegraded":"2020-11-20T11:52:28.204947+0000","last_fullsized":"2020-11-20T11:52:28.204947+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:52:28.204940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:52:28.204940+0000","last_clean_scrub_stamp":"2020-11-20T11:52:28.204940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"25'142","reported_seq":"158","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:26.096858+0000","last_change":"2020-11-20T11:52:26.096939+0000","last_active":"2020-11-20T11:52:26.096858+0000","last_peered":"2020-11-20T11:52:26.096858+0000","last_clean":"2020-11-20T11:52:26.096858+0000","last_became_active":"2020-11-20T11:32:22.092604+0000","last_became_peered":"2020-11-20T11:32:22.092604+0000","last_unstale":"2020-11-20T11:52:26.096858+0000","last_undegraded":"2020-11-20T11:52:26.096858+0000","last_fullsized":"2020-11-20T11:52:26.096858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'142","last_scrub_stamp":"2020-11-20T11:52:26.096850+0000","last_deep_scrub":"25'142","last_deep_scrub_stamp":"2020-11-20T11:52:26.096850+0000","last_clean_scrub_stamp":"2020-11-20T11:52:26.096850+0000","log_size":142,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":355,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":3057647616,"num_objects":729,"num_object_clones":0,"num_object_copies":2916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":253663,"num_write_kb":14105241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5916065792,"data_stored":5916065792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80520,"ondisk_log_size":80520,"up":32,"acting":32,"num_store_stats":8},{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":8797519,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":23,"num_read_kb":61,"num_write":2108,"num_write_kb":9406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4710,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17653760,"data_stored":17595038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1914,"ondisk_log_size":1914,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8521,"num_omap_keys":2,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":6,"up_from":14,"seq":60129542393,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":808328,"kb_used_data":555264,"kb_used_omap":0,"kb_used_meta":253056,"kb_avail":93563512,"statfs":{"total":96636764160,"available":95809036288,"internally_reserved":0,"allocated":568590336,"data_stored":568435472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":259129344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 11:51:42 2020","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.0109999999999999,"15min":1.4450000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.45700000000000002},"max":{"1min":3.677,"5min":9.6129999999999995,"15min":22.257999999999999},"last":16.411999999999999},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.96999999999999997,"15min":1.4079999999999999},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":2.1379999999999999,"5min":6.3099999999999996,"15min":19.855},"last":16.567}]},{"osd":1,"last update":"Fri Nov 20 11:51:42 2020","interfaces":[{"interface":"back","average":{"1min":5.4039999999999999,"5min":8.8640000000000008,"15min":6.9290000000000003},"min":{"1min":0.70999999999999996,"5min":0.504,"15min":0.504},"max":{"1min":23.738,"5min":39.268999999999998,"15min":42.552999999999997},"last":1.792},{"interface":"front","average":{"1min":6.1950000000000003,"5min":9.2330000000000005,"15min":6.8099999999999996},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":39.792999999999999,"5min":39.792999999999999,"15min":58.981000000000002},"last":2.9489999999999998}]},{"osd":2,"last update":"Fri Nov 20 11:51:42 2020","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.194,"15min":1.2050000000000001},"min":{"1min":0.65700000000000003,"5min":0.57999999999999996,"15min":0.57799999999999996},"max":{"1min":3.7090000000000001,"5min":6.0880000000000001,"15min":7.4790000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0640000000000001,"15min":1.135},"min":{"1min":0.63200000000000001,"5min":0.61399999999999999,"15min":0.54200000000000004},"max":{"1min":2.2879999999999998,"5min":8.1519999999999992,"15min":8.1519999999999992},"last":0.78900000000000003}]},{"osd":3,"last update":"Fri Nov 20 11:51:42 2020","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.2470000000000001,"15min":1.4870000000000001},"min":{"1min":0.628,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":3.2400000000000002,"5min":9.7330000000000005,"15min":11.287000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.1870000000000001,"15min":1.4770000000000001},"min":{"1min":0.65800000000000003,"5min":0.65300000000000002,"15min":0.61099999999999999},"max":{"1min":5.3170000000000002,"5min":5.3170000000000002,"15min":11.256},"last":0.83999999999999997}]},{"osd":4,"last update":"Fri Nov 20 11:51:42 2020","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":1.109,"15min":1.325},"min":{"1min":0.57999999999999996,"5min":0.57699999999999996,"15min":0.53500000000000003},"max":{"1min":1.319,"5min":9.1150000000000002,"15min":10.087999999999999},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.2649999999999999,"15min":1.3540000000000001},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.57399999999999995},"max":{"1min":1.655,"5min":8.9309999999999992,"15min":9.468},"last":0.69399999999999995}]},{"osd":5,"last update":"Fri Nov 20 11:51:42 2020","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.135,"15min":1.274},"min":{"1min":0.68000000000000005,"5min":0.621,"15min":0.59499999999999997},"max":{"1min":1.52,"5min":7.8780000000000001,"15min":7.8780000000000001},"last":0.66900000000000004},{"interface":"front","average":{"1min":1.097,"5min":1.2609999999999999,"15min":1.494},"min":{"1min":0.69999999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.532,"5min":9.4779999999999998,"15min":9.4779999999999998},"last":1.038}]},{"osd":7,"last update":"Fri Nov 20 11:51:42 2020","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.214,"15min":1.3680000000000001},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.70799999999999996},"max":{"1min":1.7569999999999999,"5min":4.3289999999999997,"15min":6.8760000000000003},"last":0.93100000000000005},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.258,"15min":1.423},"min":{"1min":0.79000000000000004,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.5549999999999999,"5min":8.0239999999999991,"15min":8.0239999999999991},"last":0.90000000000000002}]}]},{"osd":3,"up_from":14,"seq":60129542394,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1033816,"kb_used_data":708880,"kb_used_omap":0,"kb_used_meta":324928,"kb_avail":93338024,"statfs":{"total":96636764160,"available":95578136576,"internally_reserved":0,"allocated":725893120,"data_stored":725722607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":332726272},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 11:52:00 2020","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.98399999999999999,"15min":1.6080000000000001},"min":{"1min":0.48199999999999998,"5min":0.47099999999999997,"15min":0.45400000000000001},"max":{"1min":1.5760000000000001,"5min":6.2949999999999999,"15min":42.424999999999997},"last":5.1669999999999998},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.93999999999999995,"15min":1.728},"min":{"1min":0.498,"5min":0.49299999999999999,"15min":0.46700000000000003},"max":{"1min":2.1859999999999999,"5min":3.6600000000000001,"15min":41.997999999999998},"last":5.3780000000000001}]},{"osd":1,"last update":"Fri Nov 20 11:52:05 2020","interfaces":[{"interface":"back","average":{"1min":5.0510000000000002,"5min":10.712999999999999,"15min":7.5990000000000002},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.47799999999999998},"max":{"1min":14.266999999999999,"5min":65.120000000000005,"15min":65.120000000000005},"last":0.93400000000000005},{"interface":"front","average":{"1min":5.3579999999999997,"5min":11.06,"15min":7.9379999999999997},"min":{"1min":0.79200000000000004,"5min":0.78400000000000003,"15min":0.69599999999999995},"max":{"1min":14.33,"5min":65.222999999999999,"15min":65.222999999999999},"last":1.825}]},{"osd":2,"last update":"Fri Nov 20 11:52:00 2020","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":1.0609999999999999,"15min":1.1759999999999999},"min":{"1min":0.57499999999999996,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":2.1699999999999999,"5min":5.2380000000000004,"15min":22.370999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.2929999999999999,"15min":1.4399999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.52000000000000002},"max":{"1min":1.954,"5min":5.2130000000000001,"15min":8.8320000000000007},"last":1.081}]},{"osd":4,"last update":"Fri Nov 20 11:52:00 2020","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.786,"15min":1.907},"min":{"1min":0.70799999999999996,"5min":0.70399999999999996,"15min":0.64100000000000001},"max":{"1min":1.8120000000000001,"5min":46.268000000000001,"15min":46.268000000000001},"last":1.51},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3009999999999999,"15min":1.6399999999999999},"min":{"1min":0.91200000000000003,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.306,"5min":5.3659999999999997,"15min":40.478000000000002},"last":1.111}]},{"osd":5,"last update":"Fri Nov 20 11:52:00 2020","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.9279999999999999,"15min":1.962},"min":{"1min":0.76400000000000001,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.8939999999999999,"5min":46.314999999999998,"15min":46.314999999999998},"last":1.385},{"interface":"front","average":{"1min":1.21,"5min":1.369,"15min":1.9099999999999999},"min":{"1min":0.83499999999999996,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.9430000000000001,"5min":4.141,"15min":34.972999999999999},"last":1.663}]},{"osd":6,"last update":"Fri Nov 20 11:52:00 2020","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.9890000000000001,"15min":1.782},"min":{"1min":1.0149999999999999,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":3.0259999999999998,"5min":46.219000000000001,"15min":46.219000000000001},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.4339999999999999,"5min":2.0680000000000001,"15min":1.9279999999999999},"min":{"1min":0.996,"5min":0.871,"15min":0.82699999999999996},"max":{"1min":2.085,"5min":46.469000000000001,"15min":46.469000000000001},"last":1.22}]},{"osd":7,"last update":"Fri Nov 20 11:52:00 2020","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":2.0619999999999998,"15min":2.1070000000000002},"min":{"1min":1.016,"5min":0.91600000000000004,"15min":0.86799999999999999},"max":{"1min":2.069,"5min":46.432000000000002,"15min":46.432000000000002},"last":1.7969999999999999},{"interface":"front","average":{"1min":1.587,"5min":2.129,"15min":1.9690000000000001},"min":{"1min":1.097,"5min":0.92200000000000004,"15min":0.91500000000000004},"max":{"1min":2.9169999999999998,"5min":46.348999999999997,"15min":46.348999999999997},"last":1.274}]}]},{"osd":4,"up_from":13,"seq":55834575096,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":918044,"kb_used_data":653972,"kb_used_omap":0,"kb_used_meta":264064,"kb_avail":93453796,"statfs":{"total":96636764160,"available":95696687104,"internally_reserved":0,"allocated":669667328,"data_stored":669506301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":270401536},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.99199999999999999,"15min":1.6319999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":4.3280000000000003,"5min":9.1920000000000002,"15min":41.939999999999998},"last":3.831},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.048,"15min":1.7430000000000001},"min":{"1min":0.57499999999999996,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":4.9480000000000004,"5min":5.5869999999999997,"15min":42.704999999999998},"last":3.121}]},{"osd":1,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":9.5839999999999996,"15min":7.2359999999999998},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.52500000000000002},"max":{"1min":21.244,"5min":39.244999999999997,"15min":76.847999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":7.0979999999999999,"5min":9.9730000000000008,"15min":7.2450000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53200000000000003},"max":{"1min":20.898,"5min":39.472999999999999,"15min":164.745},"last":0.74399999999999999}]},{"osd":2,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":1.0600000000000001,"15min":2.2549999999999999},"min":{"1min":0.621,"5min":0.53200000000000003,"15min":0.52800000000000002},"max":{"1min":2.02,"5min":4.681,"15min":269.16800000000001},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.92800000000000005,"5min":1.147,"15min":2.2269999999999999},"min":{"1min":0.69699999999999995,"5min":0.60599999999999998,"15min":0.56299999999999994},"max":{"1min":1.1839999999999999,"5min":5.6289999999999996,"15min":262.65300000000002},"last":1.218}]},{"osd":3,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.2010000000000001,"15min":1.742},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":3.427,"5min":10.930999999999999,"15min":40.585000000000001},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.1890000000000001,"15min":1.6850000000000001},"min":{"1min":0.75,"5min":0.68200000000000005,"15min":0.59799999999999998},"max":{"1min":1.79,"5min":6.016,"15min":40.259999999999998},"last":0.86899999999999999}]},{"osd":5,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.95299999999999996,"15min":1.3180000000000001},"min":{"1min":0.60399999999999998,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.4079999999999999,"5min":3.7189999999999999,"15min":18.222999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.2,"15min":1.3560000000000001},"min":{"1min":0.67800000000000005,"5min":0.61299999999999999,"15min":0.60299999999999998},"max":{"1min":1.95,"5min":6.1840000000000002,"15min":8.8239999999999998},"last":0.97999999999999998}]},{"osd":6,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.048,"15min":1.3640000000000001},"min":{"1min":0.78500000000000003,"5min":0.67800000000000005,"15min":0.63300000000000001},"max":{"1min":1.746,"5min":2.298,"15min":7.0819999999999999},"last":1.288},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0609999999999999,"15min":1.359},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.67000000000000004},"max":{"1min":1.5369999999999999,"5min":1.7050000000000001,"15min":9.3200000000000003},"last":1.0289999999999999}]},{"osd":7,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.1970000000000001,"15min":1.417},"min":{"1min":0.85999999999999999,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.8600000000000001,"5min":2.4390000000000001,"15min":6.9089999999999998},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.198,"5min":1.2070000000000001,"15min":1.573},"min":{"1min":0.98999999999999999,"5min":0.82499999999999996,"15min":0.77300000000000002},"max":{"1min":1.712,"5min":2.8700000000000001,"15min":9.4380000000000006},"last":1.171}]}]},{"osd":0,"up_from":13,"seq":55834575096,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1584732,"kb_used_data":1151636,"kb_used_omap":0,"kb_used_meta":422848,"kb_avail":92787108,"statfs":{"total":96636764160,"available":95013998592,"internally_reserved":0,"allocated":1179275264,"data_stored":1179114237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":432996352},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,0,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":6.1040000000000001,"5min":11.039999999999999,"15min":8.218},"min":{"1min":0.57499999999999996,"5min":0.497,"15min":0.47099999999999997},"max":{"1min":28.193000000000001,"5min":47.369,"15min":83.134},"last":1.4790000000000001},{"interface":"front","average":{"1min":8.7210000000000001,"5min":11.254,"15min":8.5129999999999999},"min":{"1min":0.64500000000000002,"5min":0.51400000000000001,"15min":0.50700000000000001},"max":{"1min":39.725999999999999,"5min":47.856999999999999,"15min":82.665000000000006},"last":2.0139999999999998}]},{"osd":2,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.95599999999999996,"15min":1.393},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.46700000000000003},"max":{"1min":1.901,"5min":4.7800000000000002,"15min":19.231000000000002},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.2310000000000001,"15min":1.6659999999999999},"min":{"1min":0.52300000000000002,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":6.5810000000000004,"5min":11.981,"15min":19.974},"last":1.6379999999999999}]},{"osd":3,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.139,"15min":1.6910000000000001},"min":{"1min":0.61599999999999999,"5min":0.51300000000000001,"15min":0.51200000000000001},"max":{"1min":2.7229999999999999,"5min":6.3440000000000003,"15min":41.634},"last":0.92900000000000005},{"interface":"front","average":{"1min":1.157,"5min":1.3240000000000001,"15min":2.0289999999999999},"min":{"1min":0.63400000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.657,"5min":6.3380000000000001,"15min":44.423000000000002},"last":0.86299999999999999}]},{"osd":4,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.258,"15min":1.8240000000000001},"min":{"1min":0.80100000000000005,"5min":0.73499999999999999,"15min":0.68899999999999995},"max":{"1min":1.782,"5min":4.3300000000000001,"15min":19.471},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.319,"15min":1.665},"min":{"1min":0.91100000000000003,"5min":0.68200000000000005,"15min":0.68100000000000005},"max":{"1min":2.1619999999999999,"5min":6.7450000000000001,"15min":19.809000000000001},"last":1.5820000000000001}]},{"osd":5,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.3460000000000001,"15min":1.889},"min":{"1min":0.91300000000000003,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.7809999999999999,"5min":4.1879999999999997,"15min":20.457000000000001},"last":1.175},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.425,"15min":1.853},"min":{"1min":0.82799999999999996,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":2.3839999999999999,"5min":6.633,"15min":20.565999999999999},"last":2.2749999999999999}]},{"osd":6,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.4630000000000001,"15min":1.8979999999999999},"min":{"1min":1.0900000000000001,"5min":0.77200000000000002,"15min":0.71599999999999997},"max":{"1min":2.254,"5min":6.5110000000000001,"15min":20.699000000000002},"last":2.536},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.4890000000000001,"15min":1.895},"min":{"1min":1.022,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":2.0670000000000002,"5min":6.6219999999999999,"15min":20.609999999999999},"last":2.6709999999999998}]},{"osd":7,"last update":"Fri Nov 20 11:51:48 2020","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.587,"15min":2.0489999999999999},"min":{"1min":1.119,"5min":0.90400000000000003,"15min":0.88400000000000001},"max":{"1min":2.1970000000000001,"5min":6.4950000000000001,"15min":20.495000000000001},"last":2.411},{"interface":"front","average":{"1min":1.518,"5min":1.5149999999999999,"15min":2.1829999999999998},"min":{"1min":1.0669999999999999,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":2.0190000000000001,"5min":5.0179999999999998,"15min":20.559000000000001},"last":1.3029999999999999}]}]},{"osd":7,"up_from":13,"seq":55834575097,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1237712,"kb_used_data":809224,"kb_used_omap":0,"kb_used_meta":428480,"kb_avail":93134128,"statfs":{"total":96636764160,"available":95369347072,"internally_reserved":0,"allocated":828645376,"data_stored":828482895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":438763520},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 11:51:56 2020","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.92600000000000005,"15min":1.647},"min":{"1min":0.47499999999999998,"5min":0.44900000000000001,"15min":0.41799999999999998},"max":{"1min":1.2809999999999999,"5min":5.8170000000000002,"15min":43.341999999999999},"last":20.696000000000002},{"interface":"front","average":{"1min":0.872,"5min":0.96199999999999997,"15min":1.845},"min":{"1min":0.52100000000000002,"5min":0.49099999999999999,"15min":0.47299999999999998},"max":{"1min":1.7929999999999999,"5min":4.2809999999999997,"15min":42.759},"last":20.521999999999998}]},{"osd":1,"last update":"Fri Nov 20 11:51:56 2020","interfaces":[{"interface":"back","average":{"1min":7.3440000000000003,"5min":10.259,"15min":7.4269999999999996},"min":{"1min":0.52300000000000002,"5min":0.52100000000000002,"15min":0.51800000000000002},"max":{"1min":36.731000000000002,"5min":45.369999999999997,"15min":47.488999999999997},"last":6.7350000000000003},{"interface":"front","average":{"1min":7.0300000000000002,"5min":10.272,"15min":7.375},"min":{"1min":0.71799999999999997,"5min":0.69099999999999995,"15min":0.55800000000000005},"max":{"1min":36.750999999999998,"5min":45.343000000000004,"15min":47.521999999999998},"last":6.7590000000000003}]},{"osd":2,"last update":"Fri Nov 20 11:51:56 2020","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":1.0620000000000001,"15min":1.2170000000000001},"min":{"1min":0.58199999999999996,"5min":0.55900000000000005,"15min":0.53400000000000003},"max":{"1min":1.6799999999999999,"5min":3.9199999999999999,"15min":9.2040000000000006},"last":1.23},{"interface":"front","average":{"1min":0.88300000000000001,"5min":0.96599999999999997,"15min":1.3280000000000001},"min":{"1min":0.59499999999999997,"5min":0.58999999999999997,"15min":0.56200000000000006},"max":{"1min":1.2669999999999999,"5min":4.1859999999999999,"15min":43.755000000000003},"last":0.67100000000000004}]},{"osd":3,"last update":"Fri Nov 20 11:51:56 2020","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.206,"15min":1.837},"min":{"1min":0.79300000000000004,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.7549999999999999,"5min":4.3280000000000003,"15min":43.146999999999998},"last":1.143},{"interface":"front","average":{"1min":1.175,"5min":1.224,"15min":1.7849999999999999},"min":{"1min":0.79300000000000004,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":2.7789999999999999,"5min":4.3419999999999996,"15min":44.014000000000003},"last":0.98199999999999998}]},{"osd":4,"last update":"Fri Nov 20 11:51:56 2020","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":0.99299999999999999,"15min":1.2909999999999999},"min":{"1min":0.67700000000000005,"5min":0.60699999999999998,"15min":0.59899999999999998},"max":{"1min":1.643,"5min":3.3460000000000001,"15min":9.9469999999999992},"last":1.0409999999999999},{"interface":"front","average":{"1min":1.04,"5min":0.96399999999999997,"15min":1.417},"min":{"1min":0.746,"5min":0.57499999999999996,"15min":0.57199999999999995},"max":{"1min":1.4379999999999999,"5min":1.8520000000000001,"15min":43.890999999999998},"last":0.74399999999999999}]},{"osd":5,"last update":"Fri Nov 20 11:51:56 2020","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.087,"15min":1.585},"min":{"1min":0.66900000000000004,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":2.327,"5min":4.2370000000000001,"15min":47.914999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":1.298,"5min":1.2030000000000001,"15min":1.5269999999999999},"min":{"1min":0.88100000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":2.3530000000000002,"5min":7.1630000000000003,"15min":43.228000000000002},"last":0.95599999999999996}]},{"osd":6,"last update":"Fri Nov 20 11:51:56 2020","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.1459999999999999,"15min":1.583},"min":{"1min":0.72699999999999998,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.895,"5min":7.0659999999999998,"15min":47.780000000000001},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.2190000000000001,"15min":1.728},"min":{"1min":0.84799999999999998,"5min":0.76800000000000002,"15min":0.73199999999999998},"max":{"1min":1.958,"5min":3.4620000000000002,"15min":49.289999999999999},"last":1.3300000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575098,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1017360,"kb_used_data":676104,"kb_used_omap":0,"kb_used_meta":341248,"kb_avail":93354480,"statfs":{"total":96636764160,"available":95594987520,"internally_reserved":0,"allocated":692330496,"data_stored":692167600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":349437952},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 11:52:41 2020","interfaces":[{"interface":"back","average":{"1min":4.8120000000000003,"5min":1.7410000000000001,"15min":1.9299999999999999},"min":{"1min":0.60399999999999998,"5min":0.46700000000000003,"15min":0.40300000000000002},"max":{"1min":46.997999999999998,"5min":46.997999999999998,"15min":66.593999999999994},"last":0.60499999999999998},{"interface":"front","average":{"1min":4.6539999999999999,"5min":1.696,"15min":1.9079999999999999},"min":{"1min":0.501,"5min":0.47899999999999998,"15min":0.45600000000000002},"max":{"1min":45.832000000000001,"5min":45.832000000000001,"15min":64.793000000000006},"last":0.79500000000000004}]},{"osd":1,"last update":"Fri Nov 20 11:51:49 2020","interfaces":[{"interface":"back","average":{"1min":4.3719999999999999,"5min":12.068,"15min":7.2839999999999998},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.53300000000000003},"max":{"1min":16.989000000000001,"5min":71.489000000000004,"15min":71.489000000000004},"last":14.66},{"interface":"front","average":{"1min":4.3540000000000001,"5min":10.722,"15min":6.4649999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":17.050999999999998,"5min":72.109999999999999,"15min":72.109999999999999},"last":14.837}]},{"osd":2,"last update":"Fri Nov 20 11:51:49 2020","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":1.056,"15min":1.1699999999999999},"min":{"1min":0.63900000000000001,"5min":0.54900000000000004,"15min":0.53900000000000003},"max":{"1min":1.4410000000000001,"5min":4.9690000000000003,"15min":5.0490000000000004},"last":1.329},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.1160000000000001,"15min":1.302},"min":{"1min":0.61299999999999999,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.496,"5min":4.9909999999999997,"15min":10.227},"last":1.0429999999999999}]},{"osd":3,"last update":"Fri Nov 20 11:51:49 2020","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":1.0820000000000001,"15min":1.5289999999999999},"min":{"1min":0.63300000000000001,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.431,"5min":4.0579999999999998,"15min":64.049999999999997},"last":1.786},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.2110000000000001,"15min":1.4810000000000001},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.59299999999999997},"max":{"1min":3.5059999999999998,"5min":7.7110000000000003,"15min":63.718000000000004},"last":1.2070000000000001}]},{"osd":4,"last update":"Fri Nov 20 11:52:41 2020","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":0.99199999999999999,"15min":1.222},"min":{"1min":0.64600000000000002,"5min":0.60599999999999998,"15min":0.56299999999999994},"max":{"1min":6.0179999999999998,"5min":6.0179999999999998,"15min":10.092000000000001},"last":0.92300000000000004},{"interface":"front","average":{"1min":1.464,"5min":1.1659999999999999,"15min":1.272},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.58499999999999996},"max":{"1min":6.742,"5min":6.742,"15min":6.742},"last":1.494}]},{"osd":6,"last update":"Fri Nov 20 11:51:49 2020","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.2090000000000001,"15min":1.456},"min":{"1min":0.76100000000000001,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.8120000000000001,"5min":6.3579999999999997,"15min":10.818},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.216,"5min":1.218,"15min":1.347},"min":{"1min":0.81499999999999995,"5min":0.76700000000000002,"15min":0.68700000000000006},"max":{"1min":1.7929999999999999,"5min":4.1879999999999997,"15min":6.3280000000000003},"last":1.7030000000000001}]},{"osd":7,"last update":"Fri Nov 20 11:52:41 2020","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.1870000000000001,"15min":1.3089999999999999},"min":{"1min":0.91500000000000004,"5min":0.77600000000000002,"15min":0.70599999999999996},"max":{"1min":2.4769999999999999,"5min":3.5539999999999998,"15min":5.2359999999999998},"last":1.407},{"interface":"front","average":{"1min":1.982,"5min":1.466,"15min":1.583},"min":{"1min":1.028,"5min":0.94299999999999995,"15min":0.78400000000000003},"max":{"1min":6.6200000000000001,"5min":6.6200000000000001,"15min":7.1929999999999996},"last":1.621}]}]},{"osd":1,"up_from":14,"seq":60129542390,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1542928,"kb_used_data":1059080,"kb_used_omap":0,"kb_used_meta":483840,"kb_avail":92828912,"statfs":{"total":96636764160,"available":95056805888,"internally_reserved":0,"allocated":1084497920,"data_stored":1084335507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":495452160},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,4,4,12,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":65,"apply_latency_ms":65,"commit_latency_ns":65000000,"apply_latency_ns":65000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 11:51:46 2020","interfaces":[{"interface":"back","average":{"1min":4.8540000000000001,"5min":5.6390000000000002,"15min":5.194},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.45900000000000002},"max":{"1min":24.831,"5min":70.593000000000004,"15min":70.593000000000004},"last":2.0960000000000001},{"interface":"front","average":{"1min":2.5369999999999999,"5min":4.7039999999999997,"15min":5.4210000000000003},"min":{"1min":0.70399999999999996,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":15.93,"5min":55.527000000000001,"15min":55.527000000000001},"last":2.351}]},{"osd":2,"last update":"Fri Nov 20 11:51:46 2020","interfaces":[{"interface":"back","average":{"1min":2.8199999999999998,"5min":3.9830000000000001,"15min":3.782},"min":{"1min":0.66700000000000004,"5min":0.52600000000000002,"15min":0.47699999999999998},"max":{"1min":15.91,"5min":51.380000000000003,"15min":51.380000000000003},"last":2.1339999999999999},{"interface":"front","average":{"1min":5.569,"5min":8.7390000000000008,"15min":6.1740000000000004},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.56499999999999995},"max":{"1min":25.695,"5min":70.781999999999996,"15min":70.781999999999996},"last":2.2509999999999999}]},{"osd":3,"last update":"Fri Nov 20 11:51:46 2020","interfaces":[{"interface":"back","average":{"1min":6.0220000000000002,"5min":9.3510000000000009,"15min":6.8819999999999997},"min":{"1min":0.748,"5min":0.748,"15min":0.61799999999999999},"max":{"1min":25.109999999999999,"5min":70.700999999999993,"15min":70.700999999999993},"last":3.976},{"interface":"front","average":{"1min":6.0430000000000001,"5min":6.5910000000000002,"15min":5.6349999999999998},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.629},"max":{"1min":23.783000000000001,"5min":56.049999999999997,"15min":56.049999999999997},"last":3.5190000000000001}]},{"osd":4,"last update":"Fri Nov 20 11:51:46 2020","interfaces":[{"interface":"back","average":{"1min":5.7960000000000003,"5min":6.4189999999999996,"15min":5.165},"min":{"1min":0.90200000000000002,"5min":0.74299999999999999,"15min":0.73799999999999999},"max":{"1min":21.149999999999999,"5min":55.652000000000001,"15min":55.652000000000001},"last":3.6560000000000001},{"interface":"front","average":{"1min":4.5759999999999996,"5min":5.2859999999999996,"15min":4.6829999999999998},"min":{"1min":0.91300000000000003,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":23.812000000000001,"5min":51.926000000000002,"15min":51.926000000000002},"last":3.46}]},{"osd":5,"last update":"Fri Nov 20 11:51:46 2020","interfaces":[{"interface":"back","average":{"1min":6.1139999999999999,"5min":8.0169999999999995,"15min":5.8099999999999996},"min":{"1min":0.996,"5min":0.72199999999999998,"15min":0.69999999999999996},"max":{"1min":23.952999999999999,"5min":52.07,"15min":52.07},"last":3.7490000000000001},{"interface":"front","average":{"1min":7.0149999999999997,"5min":9.0860000000000003,"15min":6.7770000000000001},"min":{"1min":0.875,"5min":0.875,"15min":0.80800000000000005},"max":{"1min":24.981000000000002,"5min":52.262999999999998,"15min":52.262999999999998},"last":5.3810000000000002}]},{"osd":6,"last update":"Fri Nov 20 11:51:46 2020","interfaces":[{"interface":"back","average":{"1min":6.1840000000000002,"5min":7.4880000000000004,"15min":6.3049999999999997},"min":{"1min":1.167,"5min":1.087,"15min":0.88500000000000001},"max":{"1min":21.015999999999998,"5min":55.228000000000002,"15min":55.228000000000002},"last":3.8109999999999999},{"interface":"front","average":{"1min":7.8600000000000003,"5min":9.5489999999999995,"15min":6.8339999999999996},"min":{"1min":1.1639999999999999,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":30.184999999999999,"5min":52.390000000000001,"15min":52.759},"last":3.6139999999999999}]},{"osd":7,"last update":"Fri Nov 20 11:51:46 2020","interfaces":[{"interface":"back","average":{"1min":7.3719999999999999,"5min":11.303000000000001,"15min":8.0090000000000003},"min":{"1min":1.018,"5min":1.018,"15min":0.96299999999999997},"max":{"1min":25.225999999999999,"5min":52.137999999999998,"15min":52.137999999999998},"last":6.4130000000000003},{"interface":"front","average":{"1min":8.4309999999999992,"5min":10.443,"15min":7.6100000000000003},"min":{"1min":1.2849999999999999,"5min":1.214,"15min":0.97999999999999998},"max":{"1min":26.350000000000001,"5min":55.387999999999998,"15min":55.387999999999998},"last":3.948}]}]},{"osd":2,"up_from":14,"seq":60129542393,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":534800,"kb_used_data":375048,"kb_used_omap":0,"kb_used_meta":159744,"kb_avail":93837040,"statfs":{"total":96636764160,"available":96089128960,"internally_reserved":0,"allocated":384049152,"data_stored":383886739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":163577856},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 11:51:55 2020","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.1639999999999999,"15min":1.7769999999999999},"min":{"1min":0.55900000000000005,"5min":0.503,"15min":0.436},"max":{"1min":4.5540000000000003,"5min":8.4770000000000003,"15min":68.239999999999995},"last":0.94799999999999995},{"interface":"front","average":{"1min":1.018,"5min":1.2370000000000001,"15min":2.077},"min":{"1min":0.61099999999999999,"5min":0.52300000000000002,"15min":0.48999999999999999},"max":{"1min":2.3130000000000002,"5min":9.1310000000000002,"15min":68.804000000000002},"last":0.91500000000000004}]},{"osd":1,"last update":"Fri Nov 20 11:51:55 2020","interfaces":[{"interface":"back","average":{"1min":6.2859999999999996,"5min":8.3379999999999992,"15min":6.9489999999999998},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.56299999999999994},"max":{"1min":31.527999999999999,"5min":53.654000000000003,"15min":72.744},"last":4.4130000000000003},{"interface":"front","average":{"1min":6.5579999999999998,"5min":8.5719999999999992,"15min":7.2720000000000002},"min":{"1min":0.94699999999999995,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":31.486000000000001,"5min":53.222999999999999,"15min":73.438000000000002},"last":4.4580000000000002}]},{"osd":3,"last update":"Fri Nov 20 11:51:55 2020","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":1.014,"15min":1.3180000000000001},"min":{"1min":0.58099999999999996,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":3.605,"5min":4.0339999999999998,"15min":21.465},"last":0.877},{"interface":"front","average":{"1min":1.413,"5min":1.3660000000000001,"15min":1.6439999999999999},"min":{"1min":0.77800000000000002,"5min":0.67800000000000005,"15min":0.48199999999999998},"max":{"1min":4.7050000000000001,"5min":5.0270000000000001,"15min":21.693000000000001},"last":1.5189999999999999}]},{"osd":4,"last update":"Fri Nov 20 11:51:55 2020","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.4239999999999999,"15min":1.4950000000000001},"min":{"1min":0.80900000000000005,"5min":0.67300000000000004,"15min":0.626},"max":{"1min":1.8759999999999999,"5min":4.6689999999999996,"15min":10.731999999999999},"last":1.742},{"interface":"front","average":{"1min":1.419,"5min":1.4179999999999999,"15min":1.5880000000000001},"min":{"1min":0.76200000000000001,"5min":0.72599999999999998,"15min":0.56499999999999995},"max":{"1min":4.5890000000000004,"5min":7.2699999999999996,"15min":9.6519999999999992},"last":1.5529999999999999}]},{"osd":5,"last update":"Fri Nov 20 11:51:55 2020","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.369,"15min":1.4339999999999999},"min":{"1min":0.71699999999999997,"5min":0.70099999999999996,"15min":0.64100000000000001},"max":{"1min":4.7329999999999997,"5min":6.1859999999999999,"15min":12.045999999999999},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.5629999999999999,"15min":1.7210000000000001},"min":{"1min":0.90200000000000002,"5min":0.82399999999999995,"15min":0.70299999999999996},"max":{"1min":2.0009999999999999,"5min":5.0620000000000003,"15min":11.856},"last":1.9379999999999999}]},{"osd":6,"last update":"Fri Nov 20 11:51:55 2020","interfaces":[{"interface":"back","average":{"1min":1.629,"5min":1.595,"15min":1.671},"min":{"1min":0.97899999999999998,"5min":0.81999999999999995,"15min":0.65100000000000002},"max":{"1min":5.0759999999999996,"5min":7.4130000000000003,"15min":7.4130000000000003},"last":1.972},{"interface":"front","average":{"1min":1.5880000000000001,"5min":1.5449999999999999,"15min":1.5760000000000001},"min":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.83899999999999997},"max":{"1min":4.8929999999999998,"5min":6.3209999999999997,"15min":7.5369999999999999},"last":1.645}]},{"osd":7,"last update":"Fri Nov 20 11:51:55 2020","interfaces":[{"interface":"back","average":{"1min":1.675,"5min":1.734,"15min":1.7809999999999999},"min":{"1min":1.079,"5min":0.98999999999999999,"15min":0.85099999999999998},"max":{"1min":5.2240000000000002,"5min":7.54,"15min":7.54},"last":1.802},{"interface":"front","average":{"1min":1.6719999999999999,"5min":1.613,"15min":1.708},"min":{"1min":0.97599999999999998,"5min":0.89800000000000002,"15min":0.83799999999999997},"max":{"1min":5.04,"5min":6.4420000000000002,"15min":7.6909999999999998},"last":1.853}]}]}]}} 2020-11-20T11:52:48.203 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2020-11-20T11:32:24.089255+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=24, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:48.204 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2020-11-20T11:32:21.059110+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=21, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:48.204 INFO:tasks.ceph:pgid 4.4 last_scrub_stamp 2020-11-20T11:32:24.089255+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=24, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:48.204 INFO:tasks.ceph:pgid 4.7 last_scrub_stamp 2020-11-20T11:32:24.089255+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=24, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:48.205 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2020-11-20T11:32:24.089255+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=24, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:48.205 INFO:tasks.ceph:pgid 4.3 last_scrub_stamp 2020-11-20T11:32:24.089255+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=24, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:48.206 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2020-11-20T11:32:17.903652+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=17, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:48.206 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2020-11-20T11:32:24.089255+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=24, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:48.206 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2020-11-20T11:32:24.089255+0000 time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=32, tm_sec=24, tm_wday=4, tm_yday=325, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=11, tm_mday=20, tm_hour=11, tm_min=52, tm_sec=20, tm_wday=4, tm_yday=325, tm_isdst=0) 2020-11-20T11:52:48.206 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2020-11-20T11:53:08.207 INFO:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-11-20T11:53:08.341 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.337+0000 7f01894de700 1 Processor -- start 2020-11-20T11:53:08.342 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.337+0000 7f01894de700 1 -- start start 2020-11-20T11:53:08.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.338+0000 7f01894de700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f01840ad600 0x7f018413af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:53:08.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.338+0000 7f01894de700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f01840ad9e0 0x7f01840a5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:53:08.343 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.338+0000 7f01894de700 1 -- --> v1:172.21.15.23:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01840a6f10 con 0x7f01840a6a70 2020-11-20T11:53:08.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.338+0000 7f01894de700 1 -- --> v2:172.21.15.23:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01840a71b0 con 0x7f01840ad9e0 2020-11-20T11:53:08.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.338+0000 7f01894de700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01840a4da0 con 0x7f01840ad600 2020-11-20T11:53:08.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.338+0000 7f01827fc700 1 --2- >> v2:172.21.15.23:3300/0 conn(0x7f01840ad9e0 0x7f01840a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:53:08.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.338+0000 7f01837fe700 1 --1- >> v1:172.21.15.23:6789/0 conn(0x7f01840a6a70 0x7f01840a90a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.23:6789/0 says I am v1:172.21.15.23:33502/0 (socket says 172.21.15.23:33502) 2020-11-20T11:53:08.344 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.338+0000 7f01837fe700 1 -- 172.21.15.23:0/3190889175 learned_addr learned my addr 172.21.15.23:0/3190889175 (peer_addr_for_me v1:172.21.15.23:0/0) 2020-11-20T11:53:08.345 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.338+0000 7f0182ffd700 1 --2- 172.21.15.23:0/3190889175 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f01840ad600 0x7f018413af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:53:08.345 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.338+0000 7f01827fc700 1 -- 172.21.15.23:0/3190889175 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f01840ad600 msgr2=0x7f018413af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:53:08.345 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.339+0000 7f01827fc700 1 --2- 172.21.15.23:0/3190889175 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f01840ad600 0x7f018413af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:53:08.345 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.339+0000 7f01827fc700 1 -- 172.21.15.23:0/3190889175 >> v1:172.21.15.23:6789/0 conn(0x7f01840a6a70 legacy=0x7f01840a90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:53:08.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.339+0000 7f0181ffb700 1 -- 172.21.15.23:0/3190889175 <== mon.0 v1:172.21.15.23:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3100214051 0 0) 0x7f01840a6f10 con 0x7f01840a6a70 2020-11-20T11:53:08.352 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.339+0000 7f01827fc700 1 -- 172.21.15.23:0/3190889175 --> v2:172.21.15.23:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01841352f0 con 0x7f01840ad9e0 2020-11-20T11:53:08.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.339+0000 7f01827fc700 1 --2- 172.21.15.23:0/3190889175 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01840ad9e0 0x7f01840a5240 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f016c006fd0 tx=0x7f016c006e30).ready entity=mon.0 client_cookie=a7b67fe2de355d3 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:53:08.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.339+0000 7f0181ffb700 1 -- 172.21.15.23:0/3190889175 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016c0097e0 con 0x7f01840ad9e0 2020-11-20T11:53:08.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.339+0000 7f0181ffb700 1 -- 172.21.15.23:0/3190889175 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f016c000e90 con 0x7f01840ad9e0 2020-11-20T11:53:08.353 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.339+0000 7f0181ffb700 1 -- 172.21.15.23:0/3190889175 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016c009b50 con 0x7f01840ad9e0 2020-11-20T11:53:08.354 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.339+0000 7f01894de700 1 -- 172.21.15.23:0/3190889175 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01840ad9e0 msgr2=0x7f01840a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:53:08.354 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.339+0000 7f01894de700 1 --2- 172.21.15.23:0/3190889175 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01840ad9e0 0x7f01840a5240 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f016c006fd0 tx=0x7f016c006e30).stop 2020-11-20T11:53:08.354 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.339+0000 7f01894de700 1 -- 172.21.15.23:0/3190889175 shutdown_connections 2020-11-20T11:53:08.354 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.339+0000 7f01894de700 1 --2- 172.21.15.23:0/3190889175 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f01840ad600 0x7f018413af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:53:08.354 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.339+0000 7f01894de700 1 --2- 172.21.15.23:0/3190889175 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01840ad9e0 0x7f01840a5240 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:53:08.355 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.339+0000 7f01894de700 1 -- 172.21.15.23:0/3190889175 >> 172.21.15.23:0/3190889175 conn(0x7f0184060c80 msgr2=0x7f01840acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:53:08.355 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.339+0000 7f01894de700 1 -- 172.21.15.23:0/3190889175 shutdown_connections 2020-11-20T11:53:08.355 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.339+0000 7f01894de700 1 -- 172.21.15.23:0/3190889175 wait complete. 2020-11-20T11:53:08.355 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.341+0000 7f01894de700 1 Processor -- start 2020-11-20T11:53:08.355 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.341+0000 7f01894de700 1 -- start start 2020-11-20T11:53:08.356 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.341+0000 7f01894de700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f01840a6a70 0x7f0184061aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:53:08.356 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.341+0000 7f01894de700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f01840ad600 0x7f0184137ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:53:08.356 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.341+0000 7f01894de700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01840ad9e0 0x7f01841381d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:53:08.357 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.341+0000 7f01894de700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018409f4d0 con 0x7f01840ad9e0 2020-11-20T11:53:08.357 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.341+0000 7f01894de700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01841386c0 con 0x7f01840ad600 2020-11-20T11:53:08.357 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.341+0000 7f01894de700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f018409f6c0 con 0x7f01840a6a70 2020-11-20T11:53:08.357 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.341+0000 7f01837fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01840ad9e0 0x7f01841381d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:53:08.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.341+0000 7f01837fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01840ad9e0 0x7f01841381d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47490/0 (socket says 172.21.15.23:47490) 2020-11-20T11:53:08.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.341+0000 7f01837fe700 1 -- 172.21.15.23:0/4075285738 learned_addr learned my addr 172.21.15.23:0/4075285738 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:53:08.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.341+0000 7f0182ffd700 1 --2- 172.21.15.23:0/4075285738 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f01840a6a70 0x7f0184061aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:53:08.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.341+0000 7f01827fc700 1 --2- 172.21.15.23:0/4075285738 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f01840ad600 0x7f0184137ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:53:08.358 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.342+0000 7f01837fe700 1 -- 172.21.15.23:0/4075285738 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f01840a6a70 msgr2=0x7f0184061aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:53:08.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.342+0000 7f01837fe700 1 --2- 172.21.15.23:0/4075285738 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f01840a6a70 0x7f0184061aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:53:08.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.342+0000 7f01837fe700 1 -- 172.21.15.23:0/4075285738 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f01840ad600 msgr2=0x7f0184137ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:53:08.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.342+0000 7f01837fe700 1 --2- 172.21.15.23:0/4075285738 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f01840ad600 0x7f0184137ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:53:08.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.342+0000 7f01837fe700 1 -- 172.21.15.23:0/4075285738 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f018409f960 con 0x7f01840ad9e0 2020-11-20T11:53:08.359 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.342+0000 7f01837fe700 1 --2- 172.21.15.23:0/4075285738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01840ad9e0 0x7f01841381d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f017c00cae0 tx=0x7f017c007dc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:53:08.360 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.342+0000 7f017affd700 1 -- 172.21.15.23:0/4075285738 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017c005bc0 con 0x7f01840ad9e0 2020-11-20T11:53:08.360 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.342+0000 7f017affd700 1 -- 172.21.15.23:0/4075285738 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f017c005d20 con 0x7f01840ad9e0 2020-11-20T11:53:08.360 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.342+0000 7f01894de700 1 -- 172.21.15.23:0/4075285738 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f018409fc00 con 0x7f01840ad9e0 2020-11-20T11:53:08.360 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.342+0000 7f017affd700 1 -- 172.21.15.23:0/4075285738 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017c007320 con 0x7f01840ad9e0 2020-11-20T11:53:08.361 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.342+0000 7f01894de700 1 -- 172.21.15.23:0/4075285738 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0184062670 con 0x7f01840ad9e0 2020-11-20T11:53:08.361 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.343+0000 7f01894de700 1 -- 172.21.15.23:0/4075285738 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f018405c520 con 0x7f01840ad9e0 2020-11-20T11:53:08.361 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.343+0000 7f017affd700 1 -- 172.21.15.23:0/4075285738 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7f017c010030 con 0x7f01840ad9e0 2020-11-20T11:53:08.361 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.344+0000 7f017affd700 1 --2- 172.21.15.23:0/4075285738 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0160056d60 0x7f0160059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:53:08.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.344+0000 7f017affd700 1 -- 172.21.15.23:0/4075285738 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7f017c070650 con 0x7f01840ad9e0 2020-11-20T11:53:08.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.344+0000 7f0182ffd700 1 --2- 172.21.15.23:0/4075285738 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0160056d60 0x7f0160059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:53:08.362 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.345+0000 7f0182ffd700 1 --2- 172.21.15.23:0/4075285738 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0160056d60 0x7f0160059220 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f017400ba40 tx=0x7f0174008040).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:53:08.363 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.347+0000 7f017affd700 1 -- 172.21.15.23:0/4075285738 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7f017c045960 con 0x7f01840ad9e0 2020-11-20T11:53:08.496 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.493+0000 7f01894de700 1 -- 172.21.15.23:0/4075285738 --> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f018409fd90 con 0x7f0160056d60 2020-11-20T11:53:08.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.497+0000 7f017affd700 1 -- 172.21.15.23:0/4075285738 <== mgr.4105 v2:172.21.15.155:6832/22693 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+106239 (secure 0 0 0) 0x7f018409fd90 con 0x7f0160056d60 2020-11-20T11:53:08.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.499+0000 7f01894de700 1 -- 172.21.15.23:0/4075285738 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0160056d60 msgr2=0x7f0160059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:53:08.503 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.499+0000 7f01894de700 1 --2- 172.21.15.23:0/4075285738 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0160056d60 0x7f0160059220 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f017400ba40 tx=0x7f0174008040).stop 2020-11-20T11:53:08.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.499+0000 7f01894de700 1 -- 172.21.15.23:0/4075285738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01840ad9e0 msgr2=0x7f01841381d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:53:08.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.499+0000 7f01894de700 1 --2- 172.21.15.23:0/4075285738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01840ad9e0 0x7f01841381d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f017c00cae0 tx=0x7f017c007dc0).stop 2020-11-20T11:53:08.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.499+0000 7f01894de700 1 -- 172.21.15.23:0/4075285738 shutdown_connections 2020-11-20T11:53:08.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.499+0000 7f01894de700 1 --2- 172.21.15.23:0/4075285738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01840ad9e0 0x7f01841381d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:53:08.504 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.499+0000 7f01894de700 1 --2- 172.21.15.23:0/4075285738 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7f0160056d60 0x7f0160059220 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:53:08.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.499+0000 7f01894de700 1 --2- 172.21.15.23:0/4075285738 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f01840a6a70 0x7f0184061aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:53:08.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.499+0000 7f01894de700 1 --2- 172.21.15.23:0/4075285738 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f01840ad600 0x7f0184137ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:53:08.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.499+0000 7f01894de700 1 -- 172.21.15.23:0/4075285738 >> 172.21.15.23:0/4075285738 conn(0x7f0184060c80 msgr2=0x7f01840aced0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:53:08.505 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.499+0000 7f01894de700 1 -- 172.21.15.23:0/4075285738 shutdown_connections 2020-11-20T11:53:08.506 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.499+0000 7f01894de700 1 -- 172.21.15.23:0/4075285738 wait complete. 2020-11-20T11:53:08.506 INFO:teuthology.orchestra.run.smithi023.stdout: 2020-11-20T11:53:08.507 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2020-11-20T11:53:08.514 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":646,"stamp":"2020-11-20T11:53:07.804851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8797519,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":23,"num_read_kb":61,"num_write":256504,"num_write_kb":14114659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13231,"num_omap_keys":12,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82350,"ondisk_log_size":82350,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":66,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2738136,"kb_used_data":19288,"kb_used_omap":0,"kb_used_meta":2718784,"kb_avail":752236584,"statfs":{"total":773094113280,"available":770290262016,"internally_reserved":0,"allocated":19750912,"data_stored":18453278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2784034816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":95,"apply_latency_ms":95,"commit_latency_ns":95000000,"apply_latency_ns":95000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1371537408,"num_objects":-327,"num_object_clones":0,"num_object_copies":-1308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":329,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002610"},"pg_stats":[{"pgid":"4.2","version":"25'15846","reported_seq":"15861","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:59.773375+0000","last_change":"2020-11-20T11:52:52.882916+0000","last_active":"2020-11-20T11:52:59.773375+0000","last_peered":"2020-11-20T11:52:59.773375+0000","last_clean":"2020-11-20T11:52:59.773375+0000","last_became_active":"2020-11-20T11:32:25.124627+0000","last_became_peered":"2020-11-20T11:32:25.124627+0000","last_unstale":"2020-11-20T11:52:59.773375+0000","last_undegraded":"2020-11-20T11:52:59.773375+0000","last_fullsized":"2020-11-20T11:52:59.773375+0000","mapping_epoch":22,"log_start":"25'5809","ondisk_log_start":"25'5809","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'15737","last_scrub_stamp":"2020-11-20T11:52:52.882791+0000","last_deep_scrub":"25'15737","last_deep_scrub_stamp":"2020-11-20T11:52:52.882791+0000","last_clean_scrub_stamp":"2020-11-20T11:52:52.882791+0000","log_size":10037,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33554,"num_write_kb":1823424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"19","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:23.044277+0000","last_change":"2020-11-20T11:52:23.044351+0000","last_active":"2020-11-20T11:52:23.044277+0000","last_peered":"2020-11-20T11:52:23.044277+0000","last_clean":"2020-11-20T11:52:23.044277+0000","last_became_active":"2020-11-20T11:32:19.482197+0000","last_became_peered":"2020-11-20T11:32:19.482197+0000","last_unstale":"2020-11-20T11:52:23.044277+0000","last_undegraded":"2020-11-20T11:52:23.044277+0000","last_fullsized":"2020-11-20T11:52:23.044277+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:52:23.044271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:52:23.044271+0000","last_clean_scrub_stamp":"2020-11-20T11:52:23.044271+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"25'36","reported_seq":"52","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:57.828903+0000","last_change":"2020-11-20T11:52:53.373241+0000","last_active":"2020-11-20T11:52:57.828903+0000","last_peered":"2020-11-20T11:52:57.828903+0000","last_clean":"2020-11-20T11:52:57.828903+0000","last_became_active":"2020-11-20T11:32:22.098264+0000","last_became_peered":"2020-11-20T11:32:22.098264+0000","last_unstale":"2020-11-20T11:52:57.828903+0000","last_undegraded":"2020-11-20T11:52:57.828903+0000","last_fullsized":"2020-11-20T11:52:57.828903+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'34","last_scrub_stamp":"2020-11-20T11:52:53.373154+0000","last_deep_scrub":"25'34","last_deep_scrub_stamp":"2020-11-20T11:52:53.373154+0000","last_clean_scrub_stamp":"2020-11-20T11:52:53.373154+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"25'14541","reported_seq":"14556","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:53:00.372944+0000","last_change":"2020-11-20T11:53:00.373098+0000","last_active":"2020-11-20T11:53:00.372944+0000","last_peered":"2020-11-20T11:53:00.372944+0000","last_clean":"2020-11-20T11:53:00.372944+0000","last_became_active":"2020-11-20T11:32:25.479350+0000","last_became_peered":"2020-11-20T11:32:25.479350+0000","last_unstale":"2020-11-20T11:53:00.372944+0000","last_undegraded":"2020-11-20T11:53:00.372944+0000","last_fullsized":"2020-11-20T11:53:00.372944+0000","mapping_epoch":22,"log_start":"25'4491","ondisk_log_start":"25'4491","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'14541","last_scrub_stamp":"2020-11-20T11:53:00.372937+0000","last_deep_scrub":"25'14541","last_deep_scrub_stamp":"2020-11-20T11:53:00.372937+0000","last_clean_scrub_stamp":"2020-11-20T11:53:00.372937+0000","log_size":10050,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31686,"num_write_kb":1652120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"19","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:26.352689+0000","last_change":"2020-11-20T11:52:26.352771+0000","last_active":"2020-11-20T11:52:26.352689+0000","last_peered":"2020-11-20T11:52:26.352689+0000","last_clean":"2020-11-20T11:52:26.352689+0000","last_became_active":"2020-11-20T11:32:18.923879+0000","last_became_peered":"2020-11-20T11:32:18.923879+0000","last_unstale":"2020-11-20T11:52:26.352689+0000","last_undegraded":"2020-11-20T11:52:26.352689+0000","last_fullsized":"2020-11-20T11:52:26.352689+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:52:26.352682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:52:26.352682+0000","last_clean_scrub_stamp":"2020-11-20T11:52:26.352682+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"25'1087","reported_seq":"1103","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:28.276341+0000","last_change":"2020-11-20T11:52:28.276414+0000","last_active":"2020-11-20T11:52:28.276341+0000","last_peered":"2020-11-20T11:52:28.276341+0000","last_clean":"2020-11-20T11:52:28.276341+0000","last_became_active":"2020-11-20T11:32:22.093123+0000","last_became_peered":"2020-11-20T11:32:22.093123+0000","last_unstale":"2020-11-20T11:52:28.276341+0000","last_undegraded":"2020-11-20T11:52:28.276341+0000","last_fullsized":"2020-11-20T11:52:28.276341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'1087","last_scrub_stamp":"2020-11-20T11:52:28.276335+0000","last_deep_scrub":"25'1087","last_deep_scrub_stamp":"2020-11-20T11:52:28.276335+0000","last_clean_scrub_stamp":"2020-11-20T11:52:28.276335+0000","log_size":1087,"ondisk_log_size":1087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4554460,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1083,"num_write_kb":4776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4710,"num_omap_keys":10,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"25'14879","reported_seq":"14894","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:53:03.359028+0000","last_change":"2020-11-20T11:53:03.359130+0000","last_active":"2020-11-20T11:53:03.359028+0000","last_peered":"2020-11-20T11:53:03.359028+0000","last_clean":"2020-11-20T11:53:03.359028+0000","last_became_active":"2020-11-20T11:32:25.124401+0000","last_became_peered":"2020-11-20T11:32:25.124401+0000","last_unstale":"2020-11-20T11:53:03.359028+0000","last_undegraded":"2020-11-20T11:53:03.359028+0000","last_fullsized":"2020-11-20T11:53:03.359028+0000","mapping_epoch":22,"log_start":"25'4785","ondisk_log_start":"25'4785","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'14879","last_scrub_stamp":"2020-11-20T11:53:03.359024+0000","last_deep_scrub":"25'14879","last_deep_scrub_stamp":"2020-11-20T11:53:03.359024+0000","last_clean_scrub_stamp":"2020-11-20T11:53:03.359024+0000","log_size":10094,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31342,"num_write_kb":1732953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"18","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:24.935039+0000","last_change":"2020-11-20T11:52:24.935164+0000","last_active":"2020-11-20T11:52:24.935039+0000","last_peered":"2020-11-20T11:52:24.935039+0000","last_clean":"2020-11-20T11:52:24.935039+0000","last_became_active":"2020-11-20T11:32:18.926981+0000","last_became_peered":"2020-11-20T11:32:18.926981+0000","last_unstale":"2020-11-20T11:52:24.935039+0000","last_undegraded":"2020-11-20T11:52:24.935039+0000","last_fullsized":"2020-11-20T11:52:24.935039+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:52:24.935031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:52:24.935031+0000","last_clean_scrub_stamp":"2020-11-20T11:52:24.935031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"25'17","reported_seq":"33","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:34.238666+0000","last_change":"2020-11-20T11:52:34.238741+0000","last_active":"2020-11-20T11:52:34.238666+0000","last_peered":"2020-11-20T11:52:34.238666+0000","last_clean":"2020-11-20T11:52:34.238666+0000","last_became_active":"2020-11-20T11:32:22.087723+0000","last_became_peered":"2020-11-20T11:32:22.087723+0000","last_unstale":"2020-11-20T11:52:34.238666+0000","last_undegraded":"2020-11-20T11:52:34.238666+0000","last_fullsized":"2020-11-20T11:52:34.238666+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'17","last_scrub_stamp":"2020-11-20T11:52:34.238658+0000","last_deep_scrub":"25'17","last_deep_scrub_stamp":"2020-11-20T11:52:34.238658+0000","last_clean_scrub_stamp":"2020-11-20T11:52:34.238658+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"25'13999","reported_seq":"14014","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:59.657373+0000","last_change":"2020-11-20T11:52:38.489578+0000","last_active":"2020-11-20T11:52:59.657373+0000","last_peered":"2020-11-20T11:52:59.657373+0000","last_clean":"2020-11-20T11:52:59.657373+0000","last_became_active":"2020-11-20T11:32:25.491657+0000","last_became_peered":"2020-11-20T11:32:25.491657+0000","last_unstale":"2020-11-20T11:52:59.657373+0000","last_undegraded":"2020-11-20T11:52:59.657373+0000","last_fullsized":"2020-11-20T11:52:59.657373+0000","mapping_epoch":22,"log_start":"25'3948","ondisk_log_start":"25'3948","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'13748","last_scrub_stamp":"2020-11-20T11:52:38.489500+0000","last_deep_scrub":"25'13748","last_deep_scrub_stamp":"2020-11-20T11:52:38.489500+0000","last_clean_scrub_stamp":"2020-11-20T11:52:38.489500+0000","log_size":10051,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30930,"num_write_kb":1726809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"19","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:37.393053+0000","last_change":"2020-11-20T11:52:37.393142+0000","last_active":"2020-11-20T11:52:37.393053+0000","last_peered":"2020-11-20T11:52:37.393053+0000","last_clean":"2020-11-20T11:52:37.393053+0000","last_became_active":"2020-11-20T11:32:18.926711+0000","last_became_peered":"2020-11-20T11:32:18.926711+0000","last_unstale":"2020-11-20T11:52:37.393053+0000","last_undegraded":"2020-11-20T11:52:37.393053+0000","last_fullsized":"2020-11-20T11:52:37.393053+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:52:37.393046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:52:37.393046+0000","last_clean_scrub_stamp":"2020-11-20T11:52:37.393046+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"25'4","reported_seq":"25","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:27.883942+0000","last_change":"2020-11-20T11:52:27.884019+0000","last_active":"2020-11-20T11:52:27.883942+0000","last_peered":"2020-11-20T11:52:27.883942+0000","last_clean":"2020-11-20T11:52:27.883942+0000","last_became_active":"2020-11-20T11:32:14.445477+0000","last_became_peered":"2020-11-20T11:32:14.445477+0000","last_unstale":"2020-11-20T11:52:27.883942+0000","last_undegraded":"2020-11-20T11:52:27.883942+0000","last_fullsized":"2020-11-20T11:52:27.883942+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'4","last_scrub_stamp":"2020-11-20T11:52:27.883935+0000","last_deep_scrub":"25'4","last_deep_scrub_stamp":"2020-11-20T11:52:27.883935+0000","last_clean_scrub_stamp":"2020-11-20T11:52:27.883935+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8521,"num_omap_keys":2,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"25'4","reported_seq":"20","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:25.210203+0000","last_change":"2020-11-20T11:52:25.210262+0000","last_active":"2020-11-20T11:52:25.210203+0000","last_peered":"2020-11-20T11:52:25.210203+0000","last_clean":"2020-11-20T11:52:25.210203+0000","last_became_active":"2020-11-20T11:32:22.087777+0000","last_became_peered":"2020-11-20T11:32:22.087777+0000","last_unstale":"2020-11-20T11:52:25.210203+0000","last_undegraded":"2020-11-20T11:52:25.210203+0000","last_fullsized":"2020-11-20T11:52:25.210203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'4","last_scrub_stamp":"2020-11-20T11:52:25.210199+0000","last_deep_scrub":"25'4","last_deep_scrub_stamp":"2020-11-20T11:52:25.210199+0000","last_clean_scrub_stamp":"2020-11-20T11:52:25.210199+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"25'12736","reported_seq":"12751","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:53:01.620797+0000","last_change":"2020-11-20T11:53:01.620897+0000","last_active":"2020-11-20T11:53:01.620797+0000","last_peered":"2020-11-20T11:53:01.620797+0000","last_clean":"2020-11-20T11:53:01.620797+0000","last_became_active":"2020-11-20T11:32:25.491100+0000","last_became_peered":"2020-11-20T11:32:25.491100+0000","last_unstale":"2020-11-20T11:53:01.620797+0000","last_undegraded":"2020-11-20T11:53:01.620797+0000","last_fullsized":"2020-11-20T11:53:01.620797+0000","mapping_epoch":22,"log_start":"25'2652","ondisk_log_start":"25'2652","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'12736","last_scrub_stamp":"2020-11-20T11:53:01.620791+0000","last_deep_scrub":"25'12736","last_deep_scrub_stamp":"2020-11-20T11:53:01.620791+0000","last_clean_scrub_stamp":"2020-11-20T11:53:01.620791+0000","log_size":10084,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28328,"num_write_kb":1700287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"18","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:28.804184+0000","last_change":"2020-11-20T11:52:28.804248+0000","last_active":"2020-11-20T11:52:28.804184+0000","last_peered":"2020-11-20T11:52:28.804184+0000","last_clean":"2020-11-20T11:52:28.804184+0000","last_became_active":"2020-11-20T11:32:18.924324+0000","last_became_peered":"2020-11-20T11:32:18.924324+0000","last_unstale":"2020-11-20T11:52:28.804184+0000","last_undegraded":"2020-11-20T11:52:28.804184+0000","last_fullsized":"2020-11-20T11:52:28.804184+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:52:28.804180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:52:28.804180+0000","last_clean_scrub_stamp":"2020-11-20T11:52:28.804180+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"25'56","reported_seq":"95","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:24.000161+0000","last_change":"2020-11-20T11:52:24.000261+0000","last_active":"2020-11-20T11:52:24.000161+0000","last_peered":"2020-11-20T11:52:24.000161+0000","last_clean":"2020-11-20T11:52:24.000161+0000","last_became_active":"2020-11-20T11:32:22.088414+0000","last_became_peered":"2020-11-20T11:32:22.088414+0000","last_unstale":"2020-11-20T11:52:24.000161+0000","last_undegraded":"2020-11-20T11:52:24.000161+0000","last_fullsized":"2020-11-20T11:52:24.000161+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'56","last_scrub_stamp":"2020-11-20T11:52:24.000149+0000","last_deep_scrub":"25'56","last_deep_scrub_stamp":"2020-11-20T11:52:24.000149+0000","last_clean_scrub_stamp":"2020-11-20T11:52:24.000149+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":47377,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":61,"num_write":58,"num_write_kb":143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"25'15514","reported_seq":"15529","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:59.783995+0000","last_change":"2020-11-20T11:52:58.811579+0000","last_active":"2020-11-20T11:52:59.783995+0000","last_peered":"2020-11-20T11:52:59.783995+0000","last_clean":"2020-11-20T11:52:59.783995+0000","last_became_active":"2020-11-20T11:32:25.490917+0000","last_became_peered":"2020-11-20T11:32:25.490917+0000","last_unstale":"2020-11-20T11:52:59.783995+0000","last_undegraded":"2020-11-20T11:52:59.783995+0000","last_fullsized":"2020-11-20T11:52:59.783995+0000","mapping_epoch":22,"log_start":"25'5451","ondisk_log_start":"25'5451","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'15501","last_scrub_stamp":"2020-11-20T11:52:58.811488+0000","last_deep_scrub":"25'15501","last_deep_scrub_stamp":"2020-11-20T11:52:58.811488+0000","last_clean_scrub_stamp":"2020-11-20T11:52:58.811488+0000","log_size":10063,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33838,"num_write_kb":1816287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"18","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:53:02.626313+0000","last_change":"2020-11-20T11:53:02.626397+0000","last_active":"2020-11-20T11:53:02.626313+0000","last_peered":"2020-11-20T11:53:02.626313+0000","last_clean":"2020-11-20T11:53:02.626313+0000","last_became_active":"2020-11-20T11:32:19.482359+0000","last_became_peered":"2020-11-20T11:32:19.482359+0000","last_unstale":"2020-11-20T11:53:02.626313+0000","last_undegraded":"2020-11-20T11:53:02.626313+0000","last_fullsized":"2020-11-20T11:53:02.626313+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:53:02.626308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:53:02.626308+0000","last_clean_scrub_stamp":"2020-11-20T11:53:02.626308+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"25'562","reported_seq":"578","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:39.452009+0000","last_change":"2020-11-20T11:52:39.452233+0000","last_active":"2020-11-20T11:52:39.452009+0000","last_peered":"2020-11-20T11:52:39.452009+0000","last_clean":"2020-11-20T11:52:39.452009+0000","last_became_active":"2020-11-20T11:32:22.092619+0000","last_became_peered":"2020-11-20T11:32:22.092619+0000","last_unstale":"2020-11-20T11:52:39.452009+0000","last_undegraded":"2020-11-20T11:52:39.452009+0000","last_fullsized":"2020-11-20T11:52:39.452009+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'562","last_scrub_stamp":"2020-11-20T11:52:39.452002+0000","last_deep_scrub":"25'562","last_deep_scrub_stamp":"2020-11-20T11:52:39.452002+0000","last_clean_scrub_stamp":"2020-11-20T11:52:39.452002+0000","log_size":562,"ondisk_log_size":562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4194947,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":555,"num_write_kb":4297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"25'14472","reported_seq":"14487","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:59.734446+0000","last_change":"2020-11-20T11:52:56.800058+0000","last_active":"2020-11-20T11:52:59.734446+0000","last_peered":"2020-11-20T11:52:59.734446+0000","last_clean":"2020-11-20T11:52:59.734446+0000","last_became_active":"2020-11-20T11:32:25.491099+0000","last_became_peered":"2020-11-20T11:32:25.491099+0000","last_unstale":"2020-11-20T11:52:59.734446+0000","last_undegraded":"2020-11-20T11:52:59.734446+0000","last_fullsized":"2020-11-20T11:52:59.734446+0000","mapping_epoch":22,"log_start":"25'4429","ondisk_log_start":"25'4429","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'14440","last_scrub_stamp":"2020-11-20T11:52:56.799964+0000","last_deep_scrub":"25'14440","last_deep_scrub_stamp":"2020-11-20T11:52:56.799964+0000","last_clean_scrub_stamp":"2020-11-20T11:52:56.799964+0000","log_size":10043,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31890,"num_write_kb":1743046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"25'15","reported_seq":"31","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:40.151034+0000","last_change":"2020-11-20T11:52:40.151093+0000","last_active":"2020-11-20T11:52:40.151034+0000","last_peered":"2020-11-20T11:52:40.151034+0000","last_clean":"2020-11-20T11:52:40.151034+0000","last_became_active":"2020-11-20T11:32:22.461331+0000","last_became_peered":"2020-11-20T11:32:22.461331+0000","last_unstale":"2020-11-20T11:52:40.151034+0000","last_undegraded":"2020-11-20T11:52:40.151034+0000","last_fullsized":"2020-11-20T11:52:40.151034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'15","last_scrub_stamp":"2020-11-20T11:52:40.151029+0000","last_deep_scrub":"25'15","last_deep_scrub_stamp":"2020-11-20T11:52:40.151029+0000","last_clean_scrub_stamp":"2020-11-20T11:52:40.151029+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"19","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:28.147369+0000","last_change":"2020-11-20T11:52:28.147444+0000","last_active":"2020-11-20T11:52:28.147369+0000","last_peered":"2020-11-20T11:52:28.147369+0000","last_clean":"2020-11-20T11:52:28.147369+0000","last_became_active":"2020-11-20T11:32:18.923448+0000","last_became_peered":"2020-11-20T11:32:18.923448+0000","last_unstale":"2020-11-20T11:52:28.147369+0000","last_undegraded":"2020-11-20T11:52:28.147369+0000","last_fullsized":"2020-11-20T11:52:28.147369+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:52:28.147363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:52:28.147363+0000","last_clean_scrub_stamp":"2020-11-20T11:52:28.147363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"25'14382","reported_seq":"14397","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:59.752230+0000","last_change":"2020-11-20T11:52:59.514192+0000","last_active":"2020-11-20T11:52:59.752230+0000","last_peered":"2020-11-20T11:52:59.752230+0000","last_clean":"2020-11-20T11:52:59.752230+0000","last_became_active":"2020-11-20T11:32:25.491038+0000","last_became_peered":"2020-11-20T11:32:25.491038+0000","last_unstale":"2020-11-20T11:52:59.752230+0000","last_undegraded":"2020-11-20T11:52:59.752230+0000","last_fullsized":"2020-11-20T11:52:59.752230+0000","mapping_epoch":22,"log_start":"25'4377","ondisk_log_start":"25'4377","created":22,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'14380","last_scrub_stamp":"2020-11-20T11:52:59.514109+0000","last_deep_scrub":"25'14380","last_deep_scrub_stamp":"2020-11-20T11:52:59.514109+0000","last_clean_scrub_stamp":"2020-11-20T11:52:59.514109+0000","log_size":10005,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32824,"num_write_kb":1910315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"18","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:28.204947+0000","last_change":"2020-11-20T11:52:28.205034+0000","last_active":"2020-11-20T11:52:28.204947+0000","last_peered":"2020-11-20T11:52:28.204947+0000","last_clean":"2020-11-20T11:52:28.204947+0000","last_became_active":"2020-11-20T11:32:18.923668+0000","last_became_peered":"2020-11-20T11:32:18.923668+0000","last_unstale":"2020-11-20T11:52:28.204947+0000","last_undegraded":"2020-11-20T11:52:28.204947+0000","last_fullsized":"2020-11-20T11:52:28.204947+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-11-20T11:52:28.204940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-11-20T11:52:28.204940+0000","last_clean_scrub_stamp":"2020-11-20T11:52:28.204940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"25'142","reported_seq":"158","reported_epoch":"25","state":"active+clean","last_fresh":"2020-11-20T11:52:26.096858+0000","last_change":"2020-11-20T11:52:26.096939+0000","last_active":"2020-11-20T11:52:26.096858+0000","last_peered":"2020-11-20T11:52:26.096858+0000","last_clean":"2020-11-20T11:52:26.096858+0000","last_became_active":"2020-11-20T11:32:22.092604+0000","last_became_peered":"2020-11-20T11:32:22.092604+0000","last_unstale":"2020-11-20T11:52:26.096858+0000","last_undegraded":"2020-11-20T11:52:26.096858+0000","last_fullsized":"2020-11-20T11:52:26.096858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'142","last_scrub_stamp":"2020-11-20T11:52:26.096850+0000","last_deep_scrub":"25'142","last_deep_scrub_stamp":"2020-11-20T11:52:26.096850+0000","last_clean_scrub_stamp":"2020-11-20T11:52:26.096850+0000","log_size":142,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":355,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":254392,"num_write_kb":14105241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80427,"ondisk_log_size":80427,"up":32,"acting":32,"num_store_stats":8},{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":8797519,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":23,"num_read_kb":61,"num_write":2110,"num_write_kb":9408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":4710,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17653760,"data_stored":17595038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1919,"ondisk_log_size":1919,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8521,"num_omap_keys":2,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":6,"up_from":14,"seq":60129542397,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":253896,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":253632,"kb_avail":94117944,"statfs":{"total":96636764160,"available":96376774656,"internally_reserved":0,"allocated":262144,"data_stored":107280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":259719168},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 11:52:45 2020","interfaces":[{"interface":"back","average":{"1min":3.3620000000000001,"5min":1.492,"15min":1.6220000000000001},"min":{"1min":0.58499999999999996,"5min":0.46500000000000002,"15min":0.45700000000000002},"max":{"1min":16.411999999999999,"5min":16.411999999999999,"15min":22.257999999999999},"last":1.133},{"interface":"front","average":{"1min":3.0099999999999998,"5min":1.3420000000000001,"15min":1.5529999999999999},"min":{"1min":0.51600000000000001,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":16.567,"5min":16.567,"15min":19.855},"last":1.0940000000000001}]},{"osd":1,"last update":"Fri Nov 20 11:52:45 2020","interfaces":[{"interface":"back","average":{"1min":6.3099999999999996,"5min":8.2319999999999993,"15min":7.2880000000000003},"min":{"1min":0.60399999999999998,"5min":0.504,"15min":0.504},"max":{"1min":28.545000000000002,"5min":39.268999999999998,"15min":42.552999999999997},"last":1.331},{"interface":"front","average":{"1min":8.0399999999999991,"5min":8.9469999999999992,"15min":7.2809999999999997},"min":{"1min":0.65900000000000003,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":28.32,"5min":39.792999999999999,"15min":58.981000000000002},"last":1.3089999999999999}]},{"osd":2,"last update":"Fri Nov 20 11:52:45 2020","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.139,"15min":1.2150000000000001},"min":{"1min":0.63100000000000001,"5min":0.57999999999999996,"15min":0.57799999999999996},"max":{"1min":1.6459999999999999,"5min":5.71,"15min":7.4790000000000001},"last":1.512},{"interface":"front","average":{"1min":1.25,"5min":1.0249999999999999,"15min":1.1559999999999999},"min":{"1min":0.624,"5min":0.61399999999999999,"15min":0.54200000000000004},"max":{"1min":3.9900000000000002,"5min":3.9900000000000002,"15min":8.1519999999999992},"last":1.4239999999999999}]},{"osd":3,"last update":"Fri Nov 20 11:52:45 2020","interfaces":[{"interface":"back","average":{"1min":1.8160000000000001,"5min":1.325,"15min":1.532},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":12.337999999999999,"5min":12.337999999999999,"15min":12.337999999999999},"last":1.476},{"interface":"front","average":{"1min":1.9199999999999999,"5min":1.325,"15min":1.5369999999999999},"min":{"1min":0.69299999999999995,"5min":0.65300000000000002,"15min":0.61099999999999999},"max":{"1min":12.629,"5min":12.629,"15min":12.629},"last":1.355}]},{"osd":4,"last update":"Fri Nov 20 11:52:45 2020","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.1240000000000001,"15min":1.3340000000000001},"min":{"1min":0.65100000000000002,"5min":0.57699999999999996,"15min":0.53500000000000003},"max":{"1min":4.8310000000000004,"5min":9.1150000000000002,"15min":10.087999999999999},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.115,"5min":1.105,"15min":1.3540000000000001},"min":{"1min":0.69399999999999995,"5min":0.65800000000000003,"15min":0.57399999999999995},"max":{"1min":1.8200000000000001,"5min":5.2080000000000002,"15min":9.468},"last":1.226}]},{"osd":5,"last update":"Fri Nov 20 11:52:45 2020","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.0309999999999999,"15min":1.2749999999999999},"min":{"1min":0.66900000000000004,"5min":0.621,"15min":0.59499999999999997},"max":{"1min":1.774,"5min":1.998,"15min":7.8780000000000001},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.2390000000000001,"15min":1.49},"min":{"1min":0.81200000000000006,"5min":0.69999999999999996,"15min":0.65700000000000003},"max":{"1min":1.9330000000000001,"5min":9.4779999999999998,"15min":9.4779999999999998},"last":1.6379999999999999}]},{"osd":7,"last update":"Fri Nov 20 11:52:45 2020","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.264,"15min":1.381},"min":{"1min":0.93100000000000005,"5min":0.81599999999999995,"15min":0.70799999999999996},"max":{"1min":3.4159999999999999,"5min":4.3289999999999997,"15min":6.8760000000000003},"last":1.663},{"interface":"front","average":{"1min":1.8,"5min":1.2929999999999999,"15min":1.4570000000000001},"min":{"1min":0.86499999999999999,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":8.9979999999999993,"5min":8.9979999999999993,"15min":8.9979999999999993},"last":1.554}]}]},{"osd":3,"up_from":14,"seq":60129542398,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":339288,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":339008,"kb_avail":94032552,"statfs":{"total":96636764160,"available":96289333248,"internally_reserved":0,"allocated":278528,"data_stored":108015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":347144192},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 11:53:01 2020","interfaces":[{"interface":"back","average":{"1min":11.214,"5min":3.0209999999999999,"15min":2.3109999999999999},"min":{"1min":0.52900000000000003,"5min":0.47099999999999997,"15min":0.46300000000000002},"max":{"1min":43.817,"5min":43.817,"15min":43.817},"last":0.63},{"interface":"front","average":{"1min":10.891,"5min":2.9060000000000001,"15min":2.3999999999999999},"min":{"1min":0.504,"5min":0.49299999999999999,"15min":0.46700000000000003},"max":{"1min":43.786999999999999,"5min":43.786999999999999,"15min":43.786999999999999},"last":0.71599999999999997}]},{"osd":1,"last update":"Fri Nov 20 11:52:05 2020","interfaces":[{"interface":"back","average":{"1min":5.0510000000000002,"5min":10.712999999999999,"15min":7.5990000000000002},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.47799999999999998},"max":{"1min":14.266999999999999,"5min":65.120000000000005,"15min":65.120000000000005},"last":0.66600000000000004},{"interface":"front","average":{"1min":5.3579999999999997,"5min":11.06,"15min":7.9379999999999997},"min":{"1min":0.79200000000000004,"5min":0.78400000000000003,"15min":0.69599999999999995},"max":{"1min":14.33,"5min":65.222999999999999,"15min":65.222999999999999},"last":0.88200000000000001}]},{"osd":2,"last update":"Fri Nov 20 11:53:01 2020","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.1040000000000001,"15min":1.208},"min":{"1min":0.54000000000000004,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":5.1680000000000001,"5min":5.2380000000000004,"15min":22.370999999999999},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.6879999999999999,"5min":1.3140000000000001,"15min":1.4850000000000001},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.52000000000000002},"max":{"1min":6.5259999999999998,"5min":6.5259999999999998,"15min":8.8320000000000007},"last":0.96599999999999997}]},{"osd":4,"last update":"Fri Nov 20 11:53:01 2020","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.7929999999999999,"15min":1.929},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":4.4630000000000001,"5min":46.268000000000001,"15min":46.268000000000001},"last":1.244},{"interface":"front","average":{"1min":1.6719999999999999,"5min":1.3500000000000001,"15min":1.6639999999999999},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":5.298,"5min":5.3659999999999997,"15min":40.478000000000002},"last":1.1759999999999999}]},{"osd":5,"last update":"Fri Nov 20 11:53:01 2020","interfaces":[{"interface":"back","average":{"1min":3.024,"5min":2.2120000000000002,"15min":2.0790000000000002},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":21.879000000000001,"5min":46.314999999999998,"15min":46.314999999999998},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.9970000000000001,"5min":1.4370000000000001,"15min":1.954},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":8.1509999999999998,"5min":8.1509999999999998,"15min":34.972999999999999},"last":1.147}]},{"osd":6,"last update":"Fri Nov 20 11:53:01 2020","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.0950000000000002,"15min":1.8260000000000001},"min":{"1min":0.82899999999999996,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":5.7800000000000002,"5min":46.219000000000001,"15min":46.219000000000001},"last":1.278},{"interface":"front","average":{"1min":2.919,"5min":2.3039999999999998,"15min":2.0329999999999999},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82699999999999996},"max":{"1min":22.053999999999998,"5min":46.469000000000001,"15min":46.469000000000001},"last":1.403}]},{"osd":7,"last update":"Fri Nov 20 11:53:01 2020","interfaces":[{"interface":"back","average":{"1min":1.71,"5min":2.0510000000000002,"15min":2.1269999999999998},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.86799999999999999},"max":{"1min":4.6870000000000003,"5min":46.432000000000002,"15min":46.432000000000002},"last":1.4399999999999999},{"interface":"front","average":{"1min":3.157,"5min":2.4239999999999999,"15min":2.0840000000000001},"min":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":21.968,"5min":46.348999999999997,"15min":46.348999999999997},"last":1.3620000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575100,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":269980,"kb_used_data":4756,"kb_used_omap":0,"kb_used_meta":265216,"kb_avail":94101860,"statfs":{"total":96636764160,"available":96360304640,"internally_reserved":0,"allocated":4870144,"data_stored":4709117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":271581184},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 11:52:49 2020","interfaces":[{"interface":"back","average":{"1min":2.984,"5min":1.2969999999999999,"15min":1.7729999999999999},"min":{"1min":0.46300000000000002,"5min":0.433,"15min":0.433},"max":{"1min":29.323,"5min":29.323,"15min":41.939999999999998},"last":0.86799999999999999},{"interface":"front","average":{"1min":3.3820000000000001,"5min":1.4339999999999999,"15min":1.9099999999999999},"min":{"1min":0.505,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":29.591999999999999,"5min":29.591999999999999,"15min":42.704999999999998},"last":0.76300000000000001}]},{"osd":1,"last update":"Fri Nov 20 11:52:49 2020","interfaces":[{"interface":"back","average":{"1min":6.9160000000000004,"5min":9.1539999999999999,"15min":7.6299999999999999},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":29.309999999999999,"5min":39.244999999999997,"15min":76.847999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":6.976,"5min":9.8089999999999993,"15min":7.6479999999999997},"min":{"1min":0.74399999999999999,"5min":0.53400000000000003,"15min":0.53200000000000003},"max":{"1min":27.805,"5min":39.472999999999999,"15min":164.745},"last":1.1579999999999999}]},{"osd":2,"last update":"Fri Nov 20 11:52:49 2020","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.044,"15min":2.2690000000000001},"min":{"1min":0.63300000000000001,"5min":0.60599999999999998,"15min":0.52800000000000002},"max":{"1min":6.399,"5min":6.399,"15min":269.16800000000001},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.482,"5min":1.1359999999999999,"15min":2.258},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":5.2990000000000004,"5min":5.2990000000000004,"15min":262.65300000000002},"last":1.2709999999999999}]},{"osd":3,"last update":"Fri Nov 20 11:52:49 2020","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.0209999999999999,"15min":1.7450000000000001},"min":{"1min":0.63100000000000001,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":3.0680000000000001,"5min":3.427,"15min":40.585000000000001},"last":1.24},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.125,"15min":1.694},"min":{"1min":0.76000000000000001,"5min":0.68200000000000005,"15min":0.59799999999999998},"max":{"1min":3.2269999999999999,"5min":3.2269999999999999,"15min":40.259999999999998},"last":1.1879999999999999}]},{"osd":5,"last update":"Fri Nov 20 11:52:49 2020","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":0.91700000000000004,"15min":1.3180000000000001},"min":{"1min":0.60799999999999998,"5min":0.60399999999999998,"15min":0.58199999999999996},"max":{"1min":2.802,"5min":2.802,"15min":18.222999999999999},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.518,"5min":1.145,"15min":1.3720000000000001},"min":{"1min":0.78200000000000003,"5min":0.61299999999999999,"15min":0.60299999999999998},"max":{"1min":3.794,"5min":3.794,"15min":8.8239999999999998},"last":1.0489999999999999}]},{"osd":6,"last update":"Fri Nov 20 11:52:49 2020","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.147,"15min":1.387},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.63300000000000001},"max":{"1min":5.0590000000000002,"5min":5.0590000000000002,"15min":7.0819999999999999},"last":1.008},{"interface":"front","average":{"1min":1.129,"5min":1.079,"15min":1.3620000000000001},"min":{"1min":0.82899999999999996,"5min":0.71399999999999997,"15min":0.67000000000000004},"max":{"1min":1.4810000000000001,"5min":1.6020000000000001,"15min":9.3200000000000003},"last":1.3999999999999999}]},{"osd":7,"last update":"Fri Nov 20 11:52:49 2020","interfaces":[{"interface":"back","average":{"1min":1.671,"5min":1.29,"15min":1.444},"min":{"1min":0.80700000000000005,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":5.4160000000000004,"5min":5.4160000000000004,"15min":6.9089999999999998},"last":1.3799999999999999},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.2350000000000001,"15min":1.581},"min":{"1min":0.97099999999999997,"5min":0.872,"15min":0.77300000000000002},"max":{"1min":3.4100000000000001,"5min":3.4100000000000001,"15min":9.4380000000000006},"last":1.6459999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575100,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":428444,"kb_used_data":4756,"kb_used_omap":0,"kb_used_meta":423680,"kb_avail":93943396,"statfs":{"total":96636764160,"available":96198037504,"internally_reserved":0,"allocated":4870144,"data_stored":4709117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":433848320},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 20 11:52:49 2020","interfaces":[{"interface":"back","average":{"1min":8.3840000000000003,"5min":10.541,"15min":8.7159999999999993},"min":{"1min":0.64800000000000002,"5min":0.57499999999999996,"15min":0.47099999999999997},"max":{"1min":37.985999999999997,"5min":47.369,"15min":83.134},"last":1.0169999999999999},{"interface":"front","average":{"1min":9.0939999999999994,"5min":10.763,"15min":9.0559999999999992},"min":{"1min":0.83599999999999997,"5min":0.51400000000000001,"15min":0.50700000000000001},"max":{"1min":38.127000000000002,"5min":47.856999999999999,"15min":82.665000000000006},"last":1.2909999999999999}]},{"osd":2,"last update":"Fri Nov 20 11:52:49 2020","interfaces":[{"interface":"back","average":{"1min":3.153,"5min":1.3600000000000001,"15min":1.55},"min":{"1min":0.57699999999999996,"5min":0.52400000000000002,"15min":0.46700000000000003},"max":{"1min":31.538,"5min":31.538,"15min":31.538},"last":1.337},{"interface":"front","average":{"1min":2.952,"5min":1.4870000000000001,"15min":1.794},"min":{"1min":0.629,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":30.809999999999999,"5min":30.809999999999999,"15min":30.809999999999999},"last":1.361}]},{"osd":3,"last update":"Fri Nov 20 11:52:49 2020","interfaces":[{"interface":"back","average":{"1min":5.4640000000000004,"5min":1.9339999999999999,"15min":1.98},"min":{"1min":0.624,"5min":0.51300000000000001,"15min":0.51200000000000001},"max":{"1min":33.506,"5min":33.506,"15min":41.634},"last":1.052},{"interface":"front","average":{"1min":6.0679999999999996,"5min":2.1819999999999999,"15min":2.3610000000000002},"min":{"1min":0.70199999999999996,"5min":0.54700000000000004,"15min":0.45300000000000001},"max":{"1min":31.658999999999999,"5min":31.658999999999999,"15min":44.423000000000002},"last":1.0800000000000001}]},{"osd":4,"last update":"Fri Nov 20 11:52:49 2020","interfaces":[{"interface":"back","average":{"1min":3.6459999999999999,"5min":1.7290000000000001,"15min":1.9810000000000001},"min":{"1min":0.77600000000000002,"5min":0.73499999999999999,"15min":0.68899999999999995},"max":{"1min":31.946000000000002,"5min":31.946000000000002,"15min":31.946000000000002},"last":1.5129999999999999},{"interface":"front","average":{"1min":4.0499999999999998,"5min":1.7949999999999999,"15min":1.845},"min":{"1min":0.90800000000000003,"5min":0.70199999999999996,"15min":0.68100000000000005},"max":{"1min":33.713999999999999,"5min":33.713999999999999,"15min":33.713999999999999},"last":1.397}]},{"osd":5,"last update":"Fri Nov 20 11:52:49 2020","interfaces":[{"interface":"back","average":{"1min":4.5880000000000001,"5min":1.962,"15min":2.1099999999999999},"min":{"1min":0.95499999999999996,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":32.058999999999997,"5min":32.058999999999997,"15min":32.058999999999997},"last":1.714},{"interface":"front","average":{"1min":3.4929999999999999,"5min":1.794,"15min":2},"min":{"1min":1.0920000000000001,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":31.885999999999999,"5min":31.885999999999999,"15min":31.885999999999999},"last":1.6359999999999999}]},{"osd":6,"last update":"Fri Nov 20 11:52:49 2020","interfaces":[{"interface":"back","average":{"1min":3.8780000000000001,"5min":1.9179999999999999,"15min":2.073},"min":{"1min":0.95399999999999996,"5min":0.77200000000000002,"15min":0.71599999999999997},"max":{"1min":32.011000000000003,"5min":32.011000000000003,"15min":32.011000000000003},"last":1.8},{"interface":"front","average":{"1min":4.5510000000000002,"5min":1.9910000000000001,"15min":2.1000000000000001},"min":{"1min":1.212,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":33.838999999999999,"5min":33.838999999999999,"15min":33.838999999999999},"last":1.6639999999999999}]},{"osd":7,"last update":"Fri Nov 20 11:52:49 2020","interfaces":[{"interface":"back","average":{"1min":5.2240000000000002,"5min":2.206,"15min":2.298},"min":{"1min":1.2130000000000001,"5min":0.90400000000000003,"15min":0.88400000000000001},"max":{"1min":33.960999999999999,"5min":33.960999999999999,"15min":33.960999999999999},"last":1.7549999999999999},{"interface":"front","average":{"1min":4.3440000000000003,"5min":2.052,"15min":2.3759999999999999},"min":{"1min":1.302,"5min":0.89500000000000002,"15min":0.84599999999999997},"max":{"1min":32.213000000000001,"5min":32.213000000000001,"15min":32.213000000000001},"last":1.875}]}]},{"osd":7,"up_from":13,"seq":55834575101,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":427152,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":426880,"kb_avail":93944688,"statfs":{"total":96636764160,"available":96199360512,"internally_reserved":0,"allocated":270336,"data_stored":107855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":437125120},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 11:52:58 2020","interfaces":[{"interface":"back","average":{"1min":6.6379999999999999,"5min":1.994,"15min":2.0459999999999998},"min":{"1min":0.48699999999999999,"5min":0.44900000000000001,"15min":0.41799999999999998},"max":{"1min":20.696000000000002,"5min":20.696000000000002,"15min":43.341999999999999},"last":0.83799999999999997},{"interface":"front","average":{"1min":6.5979999999999999,"5min":2.0110000000000001,"15min":2.2320000000000002},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.47299999999999998},"max":{"1min":20.521999999999998,"5min":20.521999999999998,"15min":42.759},"last":1.2849999999999999}]},{"osd":1,"last update":"Fri Nov 20 11:52:58 2020","interfaces":[{"interface":"back","average":{"1min":7.3419999999999996,"5min":9.4280000000000008,"15min":7.859},"min":{"1min":0.53000000000000003,"5min":0.52100000000000002,"15min":0.51800000000000002},"max":{"1min":32.491999999999997,"5min":45.369999999999997,"15min":47.488999999999997},"last":1.254},{"interface":"front","average":{"1min":8.8360000000000003,"5min":9.7850000000000001,"15min":7.9050000000000002},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.55800000000000005},"max":{"1min":33.262,"5min":45.343000000000004,"15min":47.521999999999998},"last":1.1890000000000001}]},{"osd":2,"last update":"Fri Nov 20 11:52:58 2020","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":0.98999999999999999,"15min":1.2270000000000001},"min":{"1min":0.57899999999999996,"5min":0.55900000000000005,"15min":0.53400000000000003},"max":{"1min":5.0579999999999998,"5min":5.0579999999999998,"15min":9.2040000000000006},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.2330000000000001,"5min":0.97699999999999998,"15min":1.3520000000000001},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.56200000000000006},"max":{"1min":2.6850000000000001,"5min":3.7480000000000002,"15min":43.755000000000003},"last":1.129}]},{"osd":3,"last update":"Fri Nov 20 11:52:58 2020","interfaces":[{"interface":"back","average":{"1min":4.444,"5min":1.835,"15min":2.0619999999999998},"min":{"1min":0.64000000000000001,"5min":0.61299999999999999,"15min":0.60999999999999999},"max":{"1min":16.442,"5min":16.442,"15min":43.146999999999998},"last":1.032},{"interface":"front","average":{"1min":4.5579999999999998,"5min":1.907,"15min":2.0150000000000001},"min":{"1min":0.97599999999999998,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":16.48,"5min":16.48,"15min":44.014000000000003},"last":1.391}]},{"osd":4,"last update":"Fri Nov 20 11:52:58 2020","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.052,"15min":1.3069999999999999},"min":{"1min":0.64500000000000002,"5min":0.61399999999999999,"15min":0.59899999999999998},"max":{"1min":6.056,"5min":6.056,"15min":9.9469999999999992},"last":0.69699999999999995},{"interface":"front","average":{"1min":1.629,"5min":1.1000000000000001,"15min":1.4570000000000001},"min":{"1min":0.621,"5min":0.57499999999999996,"15min":0.57199999999999995},"max":{"1min":6.2060000000000004,"5min":6.2060000000000004,"15min":43.890999999999998},"last":0.97899999999999998}]},{"osd":5,"last update":"Fri Nov 20 11:52:58 2020","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.1559999999999999,"15min":1.6140000000000001},"min":{"1min":0.749,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":4.476,"5min":4.476,"15min":47.914999999999999},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.212,"15min":1.544},"min":{"1min":0.77400000000000002,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":4.2320000000000002,"5min":7.1630000000000003,"15min":43.228000000000002},"last":1.3220000000000001}]},{"osd":6,"last update":"Fri Nov 20 11:52:58 2020","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.171,"15min":1.591},"min":{"1min":0.68700000000000006,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":4.3550000000000004,"5min":7.0659999999999998,"15min":47.780000000000001},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.6599999999999999,"5min":1.2929999999999999,"15min":1.75},"min":{"1min":0.84199999999999997,"5min":0.79800000000000004,"15min":0.73199999999999998},"max":{"1min":5.1639999999999997,"5min":5.1639999999999997,"15min":49.289999999999999},"last":1.1599999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575102,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":351184,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":350912,"kb_avail":94020656,"statfs":{"total":96636764160,"available":96277151744,"internally_reserved":0,"allocated":270336,"data_stored":107440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":359333888},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 11:52:41 2020","interfaces":[{"interface":"back","average":{"1min":4.8120000000000003,"5min":1.7410000000000001,"15min":1.9299999999999999},"min":{"1min":0.60399999999999998,"5min":0.46700000000000003,"15min":0.40300000000000002},"max":{"1min":46.997999999999998,"5min":46.997999999999998,"15min":66.593999999999994},"last":1.087},{"interface":"front","average":{"1min":4.6539999999999999,"5min":1.696,"15min":1.9079999999999999},"min":{"1min":0.501,"5min":0.47899999999999998,"15min":0.45600000000000002},"max":{"1min":45.832000000000001,"5min":45.832000000000001,"15min":64.793000000000006},"last":0.753}]},{"osd":1,"last update":"Fri Nov 20 11:52:50 2020","interfaces":[{"interface":"back","average":{"1min":11.244999999999999,"5min":11.135,"15min":7.968},"min":{"1min":0.68100000000000005,"5min":0.60199999999999998,"15min":0.53300000000000003},"max":{"1min":41.683999999999997,"5min":41.683999999999997,"15min":71.489000000000004},"last":0.90900000000000003},{"interface":"front","average":{"1min":8.1929999999999996,"5min":9.3219999999999992,"15min":6.9560000000000004},"min":{"1min":0.56100000000000005,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":23.135000000000002,"5min":39.064,"15min":72.109999999999999},"last":1.395}]},{"osd":2,"last update":"Fri Nov 20 11:52:50 2020","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":0.98699999999999999,"15min":1.171},"min":{"1min":0.59999999999999998,"5min":0.55700000000000005,"15min":0.53900000000000003},"max":{"1min":2.1000000000000001,"5min":4.9690000000000003,"15min":5.0490000000000004},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.476,"5min":1.179,"15min":1.331},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":6.2530000000000001,"5min":6.2530000000000001,"15min":10.227},"last":1.1799999999999999}]},{"osd":3,"last update":"Fri Nov 20 11:52:50 2020","interfaces":[{"interface":"back","average":{"1min":2.1419999999999999,"5min":1.2430000000000001,"15min":1.597},"min":{"1min":0.624,"5min":0.624,"15min":0.59599999999999997},"max":{"1min":7.431,"5min":7.431,"15min":64.049999999999997},"last":1.123},{"interface":"front","average":{"1min":1.7609999999999999,"5min":1.179,"15min":1.532},"min":{"1min":0.67700000000000005,"5min":0.64600000000000002,"15min":0.59299999999999997},"max":{"1min":7.2389999999999999,"5min":7.2389999999999999,"15min":63.718000000000004},"last":1.4850000000000001}]},{"osd":4,"last update":"Fri Nov 20 11:52:41 2020","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":0.99199999999999999,"15min":1.222},"min":{"1min":0.64600000000000002,"5min":0.60599999999999998,"15min":0.56299999999999994},"max":{"1min":6.0179999999999998,"5min":6.0179999999999998,"15min":10.092000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.464,"5min":1.1659999999999999,"15min":1.272},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.58499999999999996},"max":{"1min":6.742,"5min":6.742,"15min":6.742},"last":1.377}]},{"osd":6,"last update":"Fri Nov 20 11:52:50 2020","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.2190000000000001,"15min":1.474},"min":{"1min":0.75900000000000001,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":6.1449999999999996,"5min":6.3579999999999997,"15min":10.818},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.7669999999999999,"5min":1.2929999999999999,"15min":1.3879999999999999},"min":{"1min":0.90600000000000003,"5min":0.81499999999999995,"15min":0.68700000000000006},"max":{"1min":6.3440000000000003,"5min":6.3440000000000003,"15min":6.3440000000000003},"last":1.2589999999999999}]},{"osd":7,"last update":"Fri Nov 20 11:52:41 2020","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.1870000000000001,"15min":1.3089999999999999},"min":{"1min":0.91500000000000004,"5min":0.77600000000000002,"15min":0.70599999999999996},"max":{"1min":2.4769999999999999,"5min":3.5539999999999998,"15min":5.2359999999999998},"last":1.5629999999999999},{"interface":"front","average":{"1min":1.982,"5min":1.466,"15min":1.583},"min":{"1min":1.028,"5min":0.94299999999999995,"15min":0.78400000000000003},"max":{"1min":6.6200000000000001,"5min":6.6200000000000001,"15min":7.1929999999999996},"last":1.4650000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542394,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505104,"kb_used_data":4360,"kb_used_omap":0,"kb_used_meta":500736,"kb_avail":93866736,"statfs":{"total":96636764160,"available":96119537664,"internally_reserved":0,"allocated":4464640,"data_stored":4302227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":512753664},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 11:52:50 2020","interfaces":[{"interface":"back","average":{"1min":9.125,"5min":6.3029999999999999,"15min":5.7560000000000002},"min":{"1min":0.54600000000000004,"5min":0.48699999999999999,"15min":0.45900000000000002},"max":{"1min":30.748000000000001,"5min":70.593000000000004,"15min":70.593000000000004},"last":0.85799999999999998},{"interface":"front","average":{"1min":8.952,"5min":5.4359999999999999,"15min":5.9640000000000004},"min":{"1min":0.505,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":33.924999999999997,"5min":55.527000000000001,"15min":55.527000000000001},"last":0.88200000000000001}]},{"osd":2,"last update":"Fri Nov 20 11:52:50 2020","interfaces":[{"interface":"back","average":{"1min":2.9329999999999998,"5min":3.4910000000000001,"15min":3.9279999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.47699999999999998},"max":{"1min":31.283000000000001,"5min":41.609000000000002,"15min":51.380000000000003},"last":0.82599999999999996},{"interface":"front","average":{"1min":4.6539999999999999,"5min":7.7069999999999999,"15min":6.4219999999999997},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.65700000000000003},"max":{"1min":31.091999999999999,"5min":70.781999999999996,"15min":70.781999999999996},"last":1.224}]},{"osd":3,"last update":"Fri Nov 20 11:52:50 2020","interfaces":[{"interface":"back","average":{"1min":9.5630000000000006,"5min":9.4459999999999997,"15min":7.4580000000000002},"min":{"1min":0.97599999999999998,"5min":0.748,"15min":0.61799999999999999},"max":{"1min":36.503,"5min":70.700999999999993,"15min":70.700999999999993},"last":1.0900000000000001},{"interface":"front","average":{"1min":7.173,"5min":6.4269999999999996,"15min":6.0439999999999996},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":35.826000000000001,"5min":56.049999999999997,"15min":56.049999999999997},"last":1.0629999999999999}]},{"osd":4,"last update":"Fri Nov 20 11:52:50 2020","interfaces":[{"interface":"back","average":{"1min":7.452,"5min":6.234,"15min":5.5839999999999996},"min":{"1min":0.754,"5min":0.754,"15min":0.74299999999999999},"max":{"1min":49.488999999999997,"5min":55.652000000000001,"15min":55.652000000000001},"last":1.367},{"interface":"front","average":{"1min":6.1379999999999999,"5min":4.867,"15min":5.0119999999999996},"min":{"1min":0.68500000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":46.639000000000003,"5min":46.639000000000003,"15min":51.926000000000002},"last":0.99099999999999999}]},{"osd":5,"last update":"Fri Nov 20 11:52:50 2020","interfaces":[{"interface":"back","average":{"1min":4.5579999999999998,"5min":7.1109999999999998,"15min":6.0289999999999999},"min":{"1min":0.82099999999999995,"5min":0.79700000000000004,"15min":0.69999999999999996},"max":{"1min":31.065000000000001,"5min":47.073999999999998,"15min":52.07},"last":1.145},{"interface":"front","average":{"1min":6.8109999999999999,"5min":8.5090000000000003,"15min":7.1440000000000001},"min":{"1min":1.0469999999999999,"5min":0.875,"15min":0.80800000000000005},"max":{"1min":31.218,"5min":44.316000000000003,"15min":52.262999999999998},"last":1.3360000000000001}]},{"osd":6,"last update":"Fri Nov 20 11:52:50 2020","interfaces":[{"interface":"back","average":{"1min":5.7649999999999997,"5min":6.9379999999999997,"15min":6.5999999999999996},"min":{"1min":0.872,"5min":0.872,"15min":0.872},"max":{"1min":33.582000000000001,"5min":55.228000000000002,"15min":55.228000000000002},"last":1.5189999999999999},{"interface":"front","average":{"1min":4.7279999999999998,"5min":8.7059999999999995,"15min":7.0579999999999998},"min":{"1min":0.94899999999999995,"5min":0.91200000000000003,"15min":0.91200000000000003},"max":{"1min":31.175999999999998,"5min":46.930999999999997,"15min":52.759},"last":1.254}]},{"osd":7,"last update":"Fri Nov 20 11:52:50 2020","interfaces":[{"interface":"back","average":{"1min":7.8659999999999997,"5min":10.76,"15min":8.4399999999999995},"min":{"1min":1.052,"5min":1.018,"15min":0.96299999999999997},"max":{"1min":31.338000000000001,"5min":46.868000000000002,"15min":52.137999999999998},"last":1.4390000000000001},{"interface":"front","average":{"1min":6.0389999999999997,"5min":9.7059999999999995,"15min":7.9169999999999998},"min":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"max":{"1min":35.957999999999998,"5min":55.387999999999998,"15min":55.387999999999998},"last":1.655}]}]},{"osd":2,"up_from":14,"seq":60129542397,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":163088,"kb_used_data":4360,"kb_used_omap":0,"kb_used_meta":158720,"kb_avail":94208752,"statfs":{"total":96636764160,"available":96469762048,"internally_reserved":0,"allocated":4464640,"data_stored":4302227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":162529280},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 20 11:52:58 2020","interfaces":[{"interface":"back","average":{"1min":8.3610000000000007,"5min":2.4910000000000001,"15min":2.2869999999999999},"min":{"1min":0.55200000000000005,"5min":0.503,"15min":0.436},"max":{"1min":49.945999999999998,"5min":49.945999999999998,"15min":68.239999999999995},"last":0.83999999999999997},{"interface":"front","average":{"1min":10.755000000000001,"5min":3.0110000000000001,"15min":2.7410000000000001},"min":{"1min":0.56699999999999995,"5min":0.52300000000000002,"15min":0.48999999999999999},"max":{"1min":50.831000000000003,"5min":50.831000000000003,"15min":68.804000000000002},"last":0.88200000000000001}]},{"osd":1,"last update":"Fri Nov 20 11:52:58 2020","interfaces":[{"interface":"back","average":{"1min":5.7089999999999996,"5min":8.0890000000000004,"15min":7.258},"min":{"1min":1.079,"5min":0.63500000000000001,"15min":0.56299999999999994},"max":{"1min":22.349,"5min":37.204000000000001,"15min":72.744},"last":1.0289999999999999},{"interface":"front","average":{"1min":6.0199999999999996,"5min":8.2759999999999998,"15min":7.5990000000000002},"min":{"1min":1.135,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":22.866,"5min":37.238999999999997,"15min":73.438000000000002},"last":1.21}]},{"osd":3,"last update":"Fri Nov 20 11:52:58 2020","interfaces":[{"interface":"back","average":{"1min":2.8700000000000001,"5min":1.341,"15min":1.4470000000000001},"min":{"1min":0.61399999999999999,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":21.428000000000001,"5min":21.428000000000001,"15min":21.465},"last":0.95699999999999996},{"interface":"front","average":{"1min":2.952,"5min":1.613,"15min":1.762},"min":{"1min":0.70199999999999996,"5min":0.67800000000000005,"15min":0.48199999999999998},"max":{"1min":21.661999999999999,"5min":21.661999999999999,"15min":21.693000000000001},"last":1.3320000000000001}]},{"osd":4,"last update":"Fri Nov 20 11:52:58 2020","interfaces":[{"interface":"back","average":{"1min":1.6040000000000001,"5min":1.3700000000000001,"15min":1.518},"min":{"1min":0.79400000000000004,"5min":0.67300000000000004,"15min":0.626},"max":{"1min":3.9319999999999999,"5min":4.6689999999999996,"15min":10.731999999999999},"last":1.24},{"interface":"front","average":{"1min":1.871,"5min":1.3700000000000001,"15min":1.627},"min":{"1min":0.77200000000000002,"5min":0.72599999999999998,"15min":0.56499999999999995},"max":{"1min":5.4189999999999996,"5min":5.4189999999999996,"15min":9.6519999999999992},"last":1.353}]},{"osd":5,"last update":"Fri Nov 20 11:52:58 2020","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.288,"15min":1.4510000000000001},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":7.0439999999999996,"5min":7.0439999999999996,"15min":12.045999999999999},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.7370000000000001,"5min":1.508,"15min":1.742},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.70299999999999996},"max":{"1min":3.8050000000000002,"5min":4.7960000000000003,"15min":11.856},"last":1.5309999999999999}]},{"osd":6,"last update":"Fri Nov 20 11:52:58 2020","interfaces":[{"interface":"back","average":{"1min":2.1480000000000001,"5min":1.5720000000000001,"15min":1.7190000000000001},"min":{"1min":0.86599999999999999,"5min":0.81999999999999995,"15min":0.65100000000000002},"max":{"1min":9.6880000000000006,"5min":9.6880000000000006,"15min":9.6880000000000006},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.831,"5min":1.512,"15min":1.6180000000000001},"min":{"1min":0.93600000000000005,"5min":0.85299999999999998,"15min":0.83899999999999997},"max":{"1min":9.1720000000000006,"5min":9.1720000000000006,"15min":9.1720000000000006},"last":1.782}]},{"osd":7,"last update":"Fri Nov 20 11:52:58 2020","interfaces":[{"interface":"back","average":{"1min":2.1899999999999999,"5min":1.6599999999999999,"15min":1.8320000000000001},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":5.8170000000000002,"5min":5.8170000000000002,"15min":7.54},"last":1.5820000000000001},{"interface":"front","average":{"1min":2.3210000000000002,"5min":1.6739999999999999,"15min":1.7749999999999999},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":7.1829999999999998,"5min":7.1829999999999998,"15min":7.6909999999999998},"last":1.625}]}]}]}} 2020-11-20T11:53:08.558 INFO:teuthology.orchestra.run.smithi023:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2020-11-20T11:53:08.675 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.671+0000 7fe5938bc700 1 Processor -- start 2020-11-20T11:53:08.675 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.671+0000 7fe5938bc700 1 -- start start 2020-11-20T11:53:08.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.671+0000 7fe5938bc700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe58c061380 0x7fe58c0aaac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:53:08.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.671+0000 7fe5938bc700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe58c0a9dd0 0x7fe58c0a3a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2020-11-20T11:53:08.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.672+0000 7fe5938bc700 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe58c0aafb0 con 0x7fe58c0a3f80 2020-11-20T11:53:08.677 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.672+0000 7fe5938bc700 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe58c0a3050 con 0x7fe58c061380 2020-11-20T11:53:08.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.672+0000 7fe5938bc700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe58c0a32f0 con 0x7fe58c0a9dd0 2020-11-20T11:53:08.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.672+0000 7fe590e57700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe58c0a9dd0 0x7fe58c0a3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:53:08.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.672+0000 7fe591658700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe58c061380 0x7fe58c0aaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:53:08.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.672+0000 7fe590e57700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe58c0a9dd0 0x7fe58c0a3a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.23:57734/0 (socket says 172.21.15.23:57734) 2020-11-20T11:53:08.678 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.672+0000 7fe591658700 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe58c061380 0x7fe58c0aaac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.23:40390/0 (socket says 172.21.15.23:40390) 2020-11-20T11:53:08.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.672+0000 7fe591e59700 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fe58c0a3f80 0x7fe58c13e3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.23:39718/0 (socket says 172.21.15.23:39718) 2020-11-20T11:53:08.679 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.672+0000 7fe590e57700 1 -- 172.21.15.23:0/2530577678 learned_addr learned my addr 172.21.15.23:0/2530577678 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:53:08.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.672+0000 7fe590e57700 1 -- 172.21.15.23:0/2530577678 >> v2:172.21.15.155:3300/0 conn(0x7fe58c061380 msgr2=0x7fe58c0aaac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-11-20T11:53:08.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.672+0000 7fe590e57700 1 --2- 172.21.15.23:0/2530577678 >> v2:172.21.15.155:3300/0 conn(0x7fe58c061380 0x7fe58c0aaac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:53:08.681 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.672+0000 7fe590e57700 1 -- 172.21.15.23:0/2530577678 >> v1:172.21.15.155:6789/0 conn(0x7fe58c0a3f80 legacy=0x7fe58c13e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:53:08.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.672+0000 7fe590e57700 1 -- 172.21.15.23:0/2530577678 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe58c0a3590 con 0x7fe58c0a9dd0 2020-11-20T11:53:08.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.672+0000 7fe57bfff700 1 -- 172.21.15.23:0/2530577678 <== mon.1 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2658386709 0 0) 0x7fe58c0aafb0 con 0x7fe58c0a3f80 2020-11-20T11:53:08.682 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.672+0000 7fe591658700 1 --2- 172.21.15.23:0/2530577678 >> v2:172.21.15.155:3300/0 conn(0x7fe58c061380 0x7fe58c0aaac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2020-11-20T11:53:08.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.673+0000 7fe590e57700 1 --2- 172.21.15.23:0/2530577678 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe58c0a9dd0 0x7fe58c0a3a90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fe588009a10 tx=0x7fe588006ef0).ready entity=mon.2 client_cookie=1cb6505497cf16ad server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:53:08.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.673+0000 7fe57bfff700 1 -- 172.21.15.23:0/2530577678 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe58800b860 con 0x7fe58c0a9dd0 2020-11-20T11:53:08.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.673+0000 7fe57bfff700 1 -- 172.21.15.23:0/2530577678 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe588000e90 con 0x7fe58c0a9dd0 2020-11-20T11:53:08.683 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.673+0000 7fe57bfff700 1 -- 172.21.15.23:0/2530577678 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe58800bbd0 con 0x7fe58c0a9dd0 2020-11-20T11:53:08.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.673+0000 7fe5938bc700 1 -- 172.21.15.23:0/2530577678 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe58c0a9dd0 msgr2=0x7fe58c0a3a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:53:08.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.673+0000 7fe5938bc700 1 --2- 172.21.15.23:0/2530577678 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe58c0a9dd0 0x7fe58c0a3a90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fe588009a10 tx=0x7fe588006ef0).stop 2020-11-20T11:53:08.687 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.673+0000 7fe5938bc700 1 -- 172.21.15.23:0/2530577678 shutdown_connections 2020-11-20T11:53:08.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.673+0000 7fe5938bc700 1 --2- 172.21.15.23:0/2530577678 >> v2:172.21.15.155:3300/0 conn(0x7fe58c061380 0x7fe58c0aaac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2020-11-20T11:53:08.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.673+0000 7fe5938bc700 1 --2- 172.21.15.23:0/2530577678 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe58c0a9dd0 0x7fe58c0a3a90 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:53:08.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.673+0000 7fe5938bc700 1 -- 172.21.15.23:0/2530577678 >> 172.21.15.23:0/2530577678 conn(0x7fe58c0ae280 msgr2=0x7fe58c0aa290 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:53:08.688 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.673+0000 7fe5938bc700 1 -- 172.21.15.23:0/2530577678 shutdown_connections 2020-11-20T11:53:08.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.673+0000 7fe5938bc700 1 -- 172.21.15.23:0/2530577678 wait complete. 2020-11-20T11:53:08.690 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.674+0000 7fe5938bc700 1 Processor -- start 2020-11-20T11:53:08.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.674+0000 7fe5938bc700 1 -- start start 2020-11-20T11:53:08.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.675+0000 7fe5938bc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe58c061380 0x7fe58c13e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:53:08.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.675+0000 7fe5938bc700 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe58c0a3f80 0x7fe58c13cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:53:08.691 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.675+0000 7fe5938bc700 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe58c0a9dd0 0x7fe58c1422f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:53:08.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.675+0000 7fe5938bc700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58c13f2b0 con 0x7fe58c061380 2020-11-20T11:53:08.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.675+0000 7fe5938bc700 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58c13d390 con 0x7fe58c0a3f80 2020-11-20T11:53:08.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.675+0000 7fe5938bc700 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe58c1427e0 con 0x7fe58c0a9dd0 2020-11-20T11:53:08.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.675+0000 7fe591658700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe58c061380 0x7fe58c13e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:53:08.692 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.675+0000 7fe591658700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe58c061380 0x7fe58c13e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47500/0 (socket says 172.21.15.23:47500) 2020-11-20T11:53:08.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.675+0000 7fe591658700 1 -- 172.21.15.23:0/935254453 learned_addr learned my addr 172.21.15.23:0/935254453 (peer_addr_for_me v2:172.21.15.23:0/0) 2020-11-20T11:53:08.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.675+0000 7fe590e57700 1 --2- 172.21.15.23:0/935254453 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe58c0a3f80 0x7fe58c13cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:53:08.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.675+0000 7fe591e59700 1 --2- 172.21.15.23:0/935254453 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe58c0a9dd0 0x7fe58c1422f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:53:08.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.675+0000 7fe591658700 1 -- 172.21.15.23:0/935254453 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe58c0a9dd0 msgr2=0x7fe58c1422f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:53:08.693 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.675+0000 7fe591658700 1 --2- 172.21.15.23:0/935254453 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe58c0a9dd0 0x7fe58c1422f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:53:08.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.675+0000 7fe591658700 1 -- 172.21.15.23:0/935254453 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe58c0a3f80 msgr2=0x7fe58c13cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:53:08.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.675+0000 7fe591658700 1 --2- 172.21.15.23:0/935254453 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe58c0a3f80 0x7fe58c13cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:53:08.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.676+0000 7fe591658700 1 -- 172.21.15.23:0/935254453 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe58c13c4c0 con 0x7fe58c061380 2020-11-20T11:53:08.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.676+0000 7fe591658700 1 --2- 172.21.15.23:0/935254453 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe58c061380 0x7fe58c13e9a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fe57c00ef80 tx=0x7fe57c006e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:53:08.694 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.676+0000 7fe5797fa700 1 -- 172.21.15.23:0/935254453 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57c00cb30 con 0x7fe58c061380 2020-11-20T11:53:08.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.676+0000 7fe5797fa700 1 -- 172.21.15.23:0/935254453 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe57c00cc90 con 0x7fe58c061380 2020-11-20T11:53:08.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.676+0000 7fe5938bc700 1 -- 172.21.15.23:0/935254453 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe58c13c760 con 0x7fe58c061380 2020-11-20T11:53:08.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.676+0000 7fe5797fa700 1 -- 172.21.15.23:0/935254453 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57c0123f0 con 0x7fe58c061380 2020-11-20T11:53:08.695 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.676+0000 7fe5938bc700 1 -- 172.21.15.23:0/935254453 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe58c13ca50 con 0x7fe58c061380 2020-11-20T11:53:08.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.677+0000 7fe5938bc700 1 -- 172.21.15.23:0/935254453 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe58c136390 con 0x7fe58c061380 2020-11-20T11:53:08.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.677+0000 7fe5797fa700 1 -- 172.21.15.23:0/935254453 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 67517+0+0 (secure 0 0 0) 0x7fe57c0127a0 con 0x7fe58c061380 2020-11-20T11:53:08.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.677+0000 7fe5797fa700 1 --2- 172.21.15.23:0/935254453 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe564056d60 0x7fe564059220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2020-11-20T11:53:08.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.677+0000 7fe5797fa700 1 -- 172.21.15.23:0/935254453 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fe57c0705c0 con 0x7fe58c061380 2020-11-20T11:53:08.696 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.678+0000 7fe590e57700 1 --2- 172.21.15.23:0/935254453 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe564056d60 0x7fe564059220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2020-11-20T11:53:08.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.678+0000 7fe590e57700 1 --2- 172.21.15.23:0/935254453 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe564056d60 0x7fe564059220 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fe588006f30 tx=0x7fe588016000).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-11-20T11:53:08.697 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.681+0000 7fe5797fa700 1 -- 172.21.15.23:0/935254453 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+143281 (secure 0 0 0) 0x7fe57c0458b0 con 0x7fe58c061380 2020-11-20T11:53:08.835 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.831+0000 7fe5938bc700 1 -- 172.21.15.23:0/935254453 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"} v 0) v1 -- 0x7fe58c0a3790 con 0x7fe58c061380 2020-11-20T11:53:08.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.839+0000 7fe5797fa700 1 -- 172.21.15.23:0/935254453 <== mon.0 v2:172.21.15.23:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fe57c0458b0 con 0x7fe58c061380 2020-11-20T11:53:08.843 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.839+0000 7fe5797fa700 1 -- 172.21.15.23:0/935254453 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mon_command_ack([{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}]=0 v2) v1 ==== 123+0+0 (secure 0 0 0) 0x7fe57c048ed0 con 0x7fe58c061380 2020-11-20T11:53:08.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.841+0000 7fe5938bc700 1 -- 172.21.15.23:0/935254453 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe564056d60 msgr2=0x7fe564059220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:53:08.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.841+0000 7fe5938bc700 1 --2- 172.21.15.23:0/935254453 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe564056d60 0x7fe564059220 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fe588006f30 tx=0x7fe588016000).stop 2020-11-20T11:53:08.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.841+0000 7fe5938bc700 1 -- 172.21.15.23:0/935254453 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe58c061380 msgr2=0x7fe58c13e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-11-20T11:53:08.846 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.841+0000 7fe5938bc700 1 --2- 172.21.15.23:0/935254453 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe58c061380 0x7fe58c13e9a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fe57c00ef80 tx=0x7fe57c006e30).stop 2020-11-20T11:53:08.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.841+0000 7fe5938bc700 1 -- 172.21.15.23:0/935254453 shutdown_connections 2020-11-20T11:53:08.847 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.841+0000 7fe5938bc700 1 --2- 172.21.15.23:0/935254453 >> [v2:172.21.15.155:6832/22693,v1:172.21.15.155:6833/22693] conn(0x7fe564056d60 0x7fe564059220 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:53:08.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.841+0000 7fe5938bc700 1 --2- 172.21.15.23:0/935254453 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe58c0a9dd0 0x7fe58c1422f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:53:08.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.841+0000 7fe5938bc700 1 --2- 172.21.15.23:0/935254453 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe58c061380 0x7fe58c13e9a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:53:08.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.841+0000 7fe5938bc700 1 --2- 172.21.15.23:0/935254453 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe58c0a3f80 0x7fe58c13cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2020-11-20T11:53:08.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.841+0000 7fe5938bc700 1 -- 172.21.15.23:0/935254453 >> 172.21.15.23:0/935254453 conn(0x7fe58c0ae280 msgr2=0x7fe58c0aa940 unknown :-1 s=STATE_NONE l=0).mark_down 2020-11-20T11:53:08.848 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.842+0000 7fe5938bc700 1 -- 172.21.15.23:0/935254453 shutdown_connections 2020-11-20T11:53:08.849 INFO:teuthology.orchestra.run.smithi023.stderr:2020-11-20T11:53:08.842+0000 7fe5938bc700 1 -- 172.21.15.23:0/935254453 wait complete. 2020-11-20T11:53:08.857 INFO:teuthology.misc:Shutting down mds daemons... 2020-11-20T11:53:08.858 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2020-11-20T11:53:08.858 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T11:53:08.868 INFO:tasks.ceph.mds.a:Stopped 2020-11-20T11:53:08.869 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2020-11-20T11:53:08.869 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T11:53:14.772 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2020-11-20T11:53:14.971 INFO:tasks.ceph.mds.b:Stopped 2020-11-20T11:53:14.971 INFO:teuthology.misc:Shutting down osd daemons... 2020-11-20T11:53:14.972 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2020-11-20T11:53:14.972 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T11:53:20.981 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2020-11-20T11:53:20.981 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2020-11-20T11:53:21.074 INFO:tasks.ceph.osd.0:Stopped 2020-11-20T11:53:21.075 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2020-11-20T11:53:21.075 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T11:53:27.090 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2020-11-20T11:53:27.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2020-11-20T11:53:27.177 INFO:tasks.ceph.osd.1:Stopped 2020-11-20T11:53:27.178 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2020-11-20T11:53:27.178 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T11:53:27.253 INFO:tasks.ceph.osd.2:Stopped 2020-11-20T11:53:27.254 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2020-11-20T11:53:27.254 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T11:53:27.353 INFO:tasks.ceph.osd.3:Stopped 2020-11-20T11:53:27.354 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2020-11-20T11:53:27.354 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T11:53:27.442 INFO:tasks.ceph.osd.4:Stopped 2020-11-20T11:53:27.442 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2020-11-20T11:53:27.443 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T11:53:27.538 INFO:tasks.ceph.osd.5:Stopped 2020-11-20T11:53:27.538 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2020-11-20T11:53:27.539 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T11:53:27.617 INFO:tasks.ceph.osd.6:Stopped 2020-11-20T11:53:27.618 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2020-11-20T11:53:27.618 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T11:53:27.717 INFO:tasks.ceph.osd.7:Stopped 2020-11-20T11:53:27.718 INFO:teuthology.misc:Shutting down mgr daemons... 2020-11-20T11:53:27.718 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2020-11-20T11:53:27.718 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T11:53:27.738 INFO:tasks.ceph.mgr.y:Stopped 2020-11-20T11:53:27.739 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2020-11-20T11:53:27.739 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T11:53:27.757 INFO:tasks.ceph.mgr.x:Stopped 2020-11-20T11:53:27.758 INFO:teuthology.misc:Shutting down mon daemons... 2020-11-20T11:53:27.758 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2020-11-20T11:53:27.759 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T11:53:27.793 INFO:tasks.ceph.mon.a:Stopped 2020-11-20T11:53:27.794 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2020-11-20T11:53:27.794 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T11:53:27.809 INFO:tasks.ceph.mon.b:Stopped 2020-11-20T11:53:27.809 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2020-11-20T11:53:27.809 INFO:teuthology.orchestra.run:waiting for 300 2020-11-20T11:53:27.861 INFO:tasks.ceph.mon.c:Stopped 2020-11-20T11:53:27.862 INFO:tasks.ceph:Checking cluster log for badness... 2020-11-20T11:53:27.862 INFO:teuthology.orchestra.run.smithi023:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2020-11-20T11:53:28.905 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi023.front.sepia.ceph.com 2020-11-20T11:53:28.905 INFO:teuthology.orchestra.run.smithi023:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2020-11-20T11:53:32.092 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2020-11-20T11:53:34.348 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi023.front.sepia.ceph.com 2020-11-20T11:53:34.349 INFO:teuthology.orchestra.run.smithi023:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2020-11-20T11:53:34.538 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi023.front.sepia.ceph.com 2020-11-20T11:53:34.538 INFO:teuthology.orchestra.run.smithi023:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2020-11-20T11:53:34.703 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi023.front.sepia.ceph.com 2020-11-20T11:53:34.704 INFO:teuthology.orchestra.run.smithi023:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2020-11-20T11:53:34.906 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi155.front.sepia.ceph.com 2020-11-20T11:53:34.907 INFO:teuthology.orchestra.run.smithi155:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2020-11-20T11:53:35.442 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi155.front.sepia.ceph.com 2020-11-20T11:53:35.442 INFO:teuthology.orchestra.run.smithi155:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2020-11-20T11:53:35.579 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi155.front.sepia.ceph.com 2020-11-20T11:53:35.579 INFO:teuthology.orchestra.run.smithi155:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2020-11-20T11:53:35.667 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi155.front.sepia.ceph.com 2020-11-20T11:53:35.668 INFO:teuthology.orchestra.run.smithi155:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2020-11-20T11:53:35.755 INFO:tasks.ceph:Archiving mon data... 2020-11-20T11:53:35.757 DEBUG:teuthology.misc:Transferring archived files from smithi023:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi/5640259/data/mon.a.tgz 2020-11-20T11:53:35.757 INFO:teuthology.orchestra.run.smithi023:> mktemp 2020-11-20T11:53:35.795 INFO:teuthology.orchestra.run.smithi023.stdout:/tmp/tmp.KEb7OTmJKx 2020-11-20T11:53:35.795 INFO:teuthology.orchestra.run.smithi023:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.KEb7OTmJKx 2020-11-20T11:53:36.908 INFO:teuthology.orchestra.run.smithi023:> sudo chmod 0666 /tmp/tmp.KEb7OTmJKx 2020-11-20T11:53:37.159 DEBUG:teuthology.orchestra.remote:smithi023:/tmp/tmp.KEb7OTmJKx is 7MB 2020-11-20T11:53:37.684 INFO:teuthology.orchestra.run.smithi023:> rm -fr /tmp/tmp.KEb7OTmJKx 2020-11-20T11:53:37.701 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi/5640259/data/mon.b.tgz 2020-11-20T11:53:37.701 INFO:teuthology.orchestra.run.smithi155:> mktemp 2020-11-20T11:53:37.733 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.itEvkKuOzr 2020-11-20T11:53:37.733 INFO:teuthology.orchestra.run.smithi155:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.itEvkKuOzr 2020-11-20T11:53:38.854 INFO:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.itEvkKuOzr 2020-11-20T11:53:38.993 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.itEvkKuOzr is 7MB 2020-11-20T11:53:39.292 INFO:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.itEvkKuOzr 2020-11-20T11:53:39.316 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi/5640259/data/mon.c.tgz 2020-11-20T11:53:39.317 INFO:teuthology.orchestra.run.smithi155:> mktemp 2020-11-20T11:53:39.378 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.c140Osnr0S 2020-11-20T11:53:39.379 INFO:teuthology.orchestra.run.smithi155:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.c140Osnr0S 2020-11-20T11:53:40.468 INFO:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.c140Osnr0S 2020-11-20T11:53:40.549 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.c140Osnr0S is 7MB 2020-11-20T11:53:40.757 INFO:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.c140Osnr0S 2020-11-20T11:53:40.772 INFO:tasks.ceph:Cleaning ceph cluster... 2020-11-20T11:53:40.773 INFO:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2020-11-20T11:53:40.775 INFO:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2020-11-20T11:53:40.884 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2020-11-20T11:53:40.885 INFO:teuthology.orchestra.run.smithi023:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2020-11-20T11:53:40.887 INFO:teuthology.orchestra.run.smithi155:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2020-11-20T11:53:40.912 INFO:teuthology.orchestra.run.smithi155.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2020-11-20T11:53:40.918 INFO:teuthology.orchestra.run.smithi023.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2020-11-20T11:53:40.924 INFO:tasks.ceph:Archiving crash dumps... 2020-11-20T11:53:40.929 DEBUG:teuthology.misc:Transferring archived files from smithi023:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi/5640259/remote/smithi023/crash 2020-11-20T11:53:40.930 INFO:teuthology.orchestra.run.smithi023:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2020-11-20T11:53:40.992 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi/5640259/remote/smithi155/crash 2020-11-20T11:53:40.993 INFO:teuthology.orchestra.run.smithi155:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2020-11-20T11:53:41.020 INFO:tasks.ceph:Shutting down logrotate 2020-11-20T11:56:40.832 INFO:teuthology.orchestra.run.smithi023:> sudo rm /etc/logrotate.d/ceph-test.conf 2020-11-20T11:56:40.971 INFO:teuthology.orchestra.run.smithi155:> sudo rm /etc/logrotate.d/ceph-test.conf 2020-11-20T11:56:41.036 INFO:tasks.ceph:Compressing logs... 2020-11-20T11:56:41.036 INFO:teuthology.orchestra.run.smithi023:> sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2020-11-20T11:56:41.039 INFO:teuthology.orchestra.run.smithi155:> sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2020-11-20T11:56:41.042 INFO:tasks.ceph:Archiving logs... 2020-11-20T11:56:41.043 DEBUG:teuthology.misc:Transferring archived files from smithi023:/var/log/ceph to /home/teuthworker/archive/pdonnell-2020-11-20_08:39:42-fs-wip-pdonnell-testing-20201120.053228-distro-basic-smithi/5640259/remote/smithi023/log 2020-11-20T11:56:41.044 INFO:teuthology.orchestra.run.smithi023:> sudo tar cz -f - -C /var/log/ceph -- . 2020-11-20T11:57:35.722 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.36207.log: File removed before we read it 2020-11-20T11:57:35.838 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.26643.log: File removed before we read it 2020-11-20T11:57:35.839 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.35896.log: File removed before we read it 2020-11-20T11:57:35.839 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.43980.log: File removed before we read it 2020-11-20T11:57:35.839 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.42536.log: File removed before we read it 2020-11-20T11:57:35.840 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.47278.log: File removed before we read it 2020-11-20T11:57:35.840 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.36341.log: File removed before we read it 2020-11-20T11:57:35.840 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.32063.log: File removed before we read it 2020-11-20T11:57:35.840 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.33926.log: File removed before we read it 2020-11-20T11:57:35.840 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.37026.log: File removed before we read it 2020-11-20T11:57:35.856 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.27021.log: File removed before we read it 2020-11-20T11:57:35.856 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-mon.a.log: File removed before we read it 2020-11-20T11:57:35.856 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.36706.log: File removed before we read it 2020-11-20T11:57:35.857 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.42428.log: File removed before we read it 2020-11-20T11:57:35.857 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.47143.log: File removed before we read it 2020-11-20T11:57:35.857 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.32924.log: File removed before we read it 2020-11-20T11:57:35.857 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.28093.log: File removed before we read it 2020-11-20T11:57:35.857 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.47017.log: File removed before we read it 2020-11-20T11:57:35.858 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.29340.log: File removed before we read it 2020-11-20T11:57:35.858 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.25584.log: File removed before we read it 2020-11-20T11:57:35.858 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.40027.log: File removed before we read it 2020-11-20T11:57:35.858 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.35275.log: File removed before we read it 2020-11-20T11:57:35.858 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.32413.log: File removed before we read it 2020-11-20T11:57:35.859 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.37068.log: File removed before we read it 2020-11-20T11:57:35.859 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.40156.log: File removed before we read it 2020-11-20T11:57:35.859 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.43408.log: File removed before we read it 2020-11-20T11:57:35.859 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.44794.log: File removed before we read it 2020-11-20T11:57:35.859 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.26853.log: File removed before we read it 2020-11-20T11:57:35.860 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.32840.log: File removed before we read it 2020-11-20T11:57:35.860 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.34348.log: File removed before we read it 2020-11-20T11:57:35.860 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.27315.log: File removed before we read it 2020-11-20T11:57:35.860 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.34576.log: File removed before we read it 2020-11-20T11:57:35.860 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.26937.log: File removed before we read it 2020-11-20T11:57:35.861 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.33615.log: File removed before we read it 2020-11-20T11:57:35.861 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.32506.log: File removed before we read it 2020-11-20T11:57:35.861 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.28177.log: File removed before we read it 2020-11-20T11:57:35.861 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.42906.log: File removed before we read it 2020-11-20T11:57:35.862 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.41856.log: File removed before we read it 2020-11-20T11:57:35.862 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.27147.log: File removed before we read it 2020-11-20T11:57:35.862 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.23169.log: File removed before we read it 2020-11-20T11:57:35.862 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.41721.log: File removed before we read it 2020-11-20T11:57:35.862 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.36039.log: File removed before we read it 2020-11-20T11:57:35.863 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.27567.log: File removed before we read it 2020-11-20T11:57:35.863 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.40313.log: File removed before we read it 2020-11-20T11:57:35.863 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.33884.log: File removed before we read it 2020-11-20T11:57:35.863 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.32189.log: File removed before we read it 2020-11-20T11:57:35.863 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.40700.log: File removed before we read it 2020-11-20T11:57:35.864 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.41268.log: File removed before we read it 2020-11-20T11:57:35.864 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.32371.log: File removed before we read it 2020-11-20T11:57:35.864 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.45075.log: File removed before we read it 2020-11-20T11:57:35.864 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.42057.log: File removed before we read it 2020-11-20T11:57:35.864 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.28051.log: File removed before we read it 2020-11-20T11:57:35.865 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.26248.log: File removed before we read it 2020-11-20T11:57:35.865 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.44251.log: File removed before we read it 2020-11-20T11:57:35.865 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.33968.log: File removed before we read it 2020-11-20T11:57:35.865 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.29424.log: File removed before we read it 2020-11-20T11:57:35.866 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.44034.log: File removed before we read it 2020-11-20T11:57:35.866 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.45465.log: File removed before we read it 2020-11-20T11:57:35.866 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.40875.log: File removed before we read it 2020-11-20T11:57:35.866 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-mds.a.log: File removed before we read it 2020-11-20T11:57:35.866 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.29713.log: File removed before we read it 2020-11-20T11:57:35.867 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.39764.log: File removed before we read it 2020-11-20T11:57:35.867 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.42385.log: File removed before we read it 2020-11-20T11:57:35.867 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.25816.log: File removed before we read it 2020-11-20T11:57:35.867 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.36299.log: File removed before we read it 2020-11-20T11:57:35.867 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.26517.log: File removed before we read it 2020-11-20T11:57:35.867 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.46681.log: File removed before we read it 2020-11-20T11:57:35.868 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.29755.log: File removed before we read it 2020-11-20T11:57:35.868 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.45865.log: File removed before we read it 2020-11-20T11:57:35.868 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.32147.log: File removed before we read it 2020-11-20T11:57:35.868 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.27231.log: File removed before we read it 2020-11-20T11:57:35.868 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.40071.log: File removed before we read it 2020-11-20T11:57:35.871 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.45698.log: File removed before we read it 2020-11-20T11:57:35.871 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.41109.log: File removed before we read it 2020-11-20T11:57:35.872 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.35938.log: File removed before we read it 2020-11-20T11:57:35.872 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.46076.log: File removed before we read it 2020-11-20T11:57:35.872 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.37244.log: File removed before we read it 2020-11-20T11:57:35.872 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.41551.log: File removed before we read it 2020-11-20T11:57:35.872 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.46424.log: File removed before we read it 2020-11-20T11:57:35.873 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.45423.log: File removed before we read it 2020-11-20T11:57:35.873 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.45381.log: File removed before we read it 2020-11-20T11:57:35.873 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.44845.log: File removed before we read it 2020-11-20T11:57:35.873 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.34492.log: File removed before we read it 2020-11-20T11:57:35.873 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.34305.log: File removed before we read it 2020-11-20T11:57:35.874 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.42729.log: File removed before we read it 2020-11-20T11:57:35.874 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.35317.log: File removed before we read it 2020-11-20T11:57:35.874 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.45159.log: File removed before we read it 2020-11-20T11:57:35.874 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.28648.log: File removed before we read it 2020-11-20T11:57:35.874 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.38430.log: File removed before we read it 2020-11-20T11:57:35.875 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.26475.log: File removed before we read it 2020-11-20T11:57:35.875 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.32329.log: File removed before we read it 2020-11-20T11:57:35.875 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.35627.log: File removed before we read it 2020-11-20T11:57:35.875 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.33201.log: File removed before we read it 2020-11-20T11:57:35.876 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.41676.log: File removed before we read it 2020-11-20T11:57:35.876 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.43641.log: File removed before we read it 2020-11-20T11:57:35.876 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.27651.log: File removed before we read it 2020-11-20T11:57:35.876 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.35854.log: File removed before we read it 2020-11-20T11:57:35.876 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.33842.log: File removed before we read it 2020-11-20T11:57:35.877 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.44616.log: File removed before we read it 2020-11-20T11:57:35.877 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.26601.log: File removed before we read it 2020-11-20T11:57:35.877 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.44975.log: File removed before we read it 2020-11-20T11:57:35.877 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.33243.log: File removed before we read it 2020-11-20T11:57:35.878 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.26433.log: File removed before we read it 2020-11-20T11:57:35.878 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.42290.log: File removed before we read it 2020-11-20T11:57:35.878 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.41417.log: File removed before we read it 2020-11-20T11:57:35.879 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.36858.log: File removed before we read it 2020-11-20T11:57:35.879 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.29049.log: File removed before we read it 2020-11-20T11:57:35.879 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.34010.log: File removed before we read it 2020-11-20T11:57:35.880 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.39614.log: File removed before we read it 2020-11-20T11:57:35.880 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.45992.log: File removed before we read it 2020-11-20T11:57:35.880 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.32798.log: File removed before we read it 2020-11-20T11:57:35.880 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.40632.log: File removed before we read it 2020-11-20T11:57:35.880 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.28690.log: File removed before we read it 2020-11-20T11:57:35.881 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.30149.log: File removed before we read it 2020-11-20T11:57:35.881 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.34144.log: File removed before we read it 2020-11-20T11:57:35.881 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.35070.log: File removed before we read it 2020-11-20T11:57:35.881 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.25858.log: File removed before we read it 2020-11-20T11:57:35.882 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.27063.log: File removed before we read it 2020-11-20T11:57:35.882 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.35812.log: File removed before we read it 2020-11-20T11:57:35.882 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.26080.log: File removed before we read it 2020-11-20T11:57:35.882 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.35485.log: File removed before we read it 2020-11-20T11:57:35.882 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.36622.log: File removed before we read it 2020-11-20T11:57:35.883 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.36815.log: File removed before we read it 2020-11-20T11:57:35.883 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.42864.log: File removed before we read it 2020-11-20T11:57:35.883 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.36467.log: File removed before we read it 2020-11-20T11:57:35.883 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.47101.log: File removed before we read it 2020-11-20T11:57:35.884 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.43595.log: File removed before we read it 2020-11-20T11:57:35.884 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.46555.log: File removed before we read it 2020-11-20T11:57:35.884 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.44431.log: File removed before we read it 2020-11-20T11:57:35.884 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.45907.log: File removed before we read it 2020-11-20T11:57:35.884 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.46118.log: File removed before we read it 2020-11-20T11:57:35.885 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.36942.log: File removed before we read it 2020-11-20T11:57:35.885 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.28858.log: File removed before we read it 2020-11-20T11:57:35.885 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.41503.log: File removed before we read it 2020-11-20T11:57:35.885 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.46891.log: File removed before we read it 2020-11-20T11:57:35.886 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.42333.log: File removed before we read it 2020-11-20T11:57:35.886 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.42682.log: File removed before we read it 2020-11-20T11:57:35.886 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.33758.log: File removed before we read it 2020-11-20T11:57:35.886 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.41941.log: File removed before we read it 2020-11-20T11:57:35.887 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.40786.log: File removed before we read it 2020-11-20T11:57:35.887 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.34752.log: File removed before we read it 2020-11-20T11:57:35.887 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.35233.log: File removed before we read it 2020-11-20T11:57:35.887 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.38319.log: File removed before we read it 2020-11-20T11:57:35.887 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.42627.log: File removed before we read it 2020-11-20T11:57:35.888 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.34836.log: File removed before we read it 2020-11-20T11:57:35.888 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.36123.log: File removed before we read it 2020-11-20T11:57:35.889 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.41193.log: File removed before we read it 2020-11-20T11:57:35.889 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.26895.log: File removed before we read it 2020-11-20T11:57:35.889 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.32105.log: File removed before we read it 2020-11-20T11:57:35.890 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.40742.log: File removed before we read it 2020-11-20T11:57:35.890 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.47360.log: File removed before we read it 2020-11-20T11:57:35.891 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.33487.log: File removed before we read it 2020-11-20T11:57:35.891 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.29525.log: File removed before we read it 2020-11-20T11:57:35.891 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.26206.log: File removed before we read it 2020-11-20T11:57:35.892 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.44574.log: File removed before we read it 2020-11-20T11:57:35.892 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.29483.log: File removed before we read it 2020-11-20T11:57:35.894 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.33377.log: File removed before we read it 2020-11-20T11:57:35.894 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.29295.log: File removed before we read it 2020-11-20T11:57:35.895 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.36081.log: File removed before we read it 2020-11-20T11:57:35.895 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.39659.log: File removed before we read it 2020-11-20T11:57:35.896 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.25944.log: File removed before we read it 2020-11-20T11:57:35.896 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.33293.log: File removed before we read it 2020-11-20T11:57:35.897 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.46160.log: File removed before we read it 2020-11-20T11:57:35.897 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.26685.log: File removed before we read it 2020-11-20T11:57:35.898 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.40423.log: File removed before we read it 2020-11-20T11:57:35.900 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.27357.log: File removed before we read it 2020-11-20T11:57:35.900 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.27609.log: File removed before we read it 2020-11-20T11:57:35.900 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.46220.log: File removed before we read it 2020-11-20T11:57:35.901 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.33531.log: File removed before we read it 2020-11-20T11:57:35.901 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.34222.log: File removed before we read it 2020-11-20T11:57:35.902 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.26979.log: File removed before we read it 2020-11-20T11:57:35.902 INFO:teuthology.orchestra.run.smithi023.stderr:tar: ./ceph-client.admin.45027.log: File removed before we read it